LZ4 — Extremely Fast Lossless Compression Algorithm

Introduction

LZ4 is a lossless data compression algorithm that prioritizes speed above all else. It delivers compression speeds exceeding 780 MB/s and decompression speeds near 5 GB/s per core on modern hardware, making it the go-to choice for scenarios where latency matters more than file size.

What LZ4 Does

• Compresses data at over 780 MB/s per core while achieving reasonable compression ratios
• Decompresses at nearly 5 GB/s per core, fast enough to be transparent in data pipelines
• Provides an HC (High Compression) variant that trades compression speed for 20-30% better ratios
• Offers both a block-level API for embedding in storage engines and a frame format for standalone files
• Supports dictionary compression for improving ratios on small, repetitive data like log entries

Architecture Overview

LZ4 is an LZ77-family byte-oriented compressor that uses a hash table to find matches in a 64KB sliding window. The format encodes literal runs and match references in a simple token-based scheme with minimal branching, enabling the decompressor to run almost entirely in L1 cache. The frame format (.lz4) adds checksums, content size hints, and dictionary IDs on top of the raw block format. LZ4-HC uses the same decompression format but employs a more exhaustive match search at compression time.

Self-Hosting & Configuration

• Install via package manager: apt install liblz4-dev or brew install lz4
• Build from source: make in the project root produces the CLI tool and shared library
• CMake build available: mkdir build && cd build && cmake .. && make
• C API: include lz4.h and link -llz4; two functions: LZ4_compress_default() and LZ4_decompress_safe()
• Available via vcpkg, Conan, and built into many Linux distributions

Key Features

• Decompression speed of ~5 GB/s per core, among the fastest of any general-purpose compressor
• Codec simplicity enables hardware-friendly implementation with minimal branch misprediction
• LZ4-HC mode produces output compatible with the fast decompressor while achieving better ratios
• Frame format supports streaming compression with configurable block sizes (64KB to 4MB)
• Dictionary mode improves compression of small messages by seeding the compressor with shared context

Comparison with Similar Tools

• Snappy (Google) — similar speed class but LZ4 achieves slightly better ratios and faster decompression
• Zstandard (zstd) — better compression ratios with configurable speed; LZ4 is faster at the lowest latency levels
• gzip/zlib — much higher compression ratios but 5-10x slower to decompress; LZ4 wins when speed is critical
• Brotli — optimized for web content with superior ratios; LZ4 targets low-latency storage and networking
• lzo — comparable speed philosophy but LZ4 has a simpler API and better community support

FAQ

Q: When should I use LZ4 vs Zstandard? A: Use LZ4 when decompression latency is the top priority (storage engines, network protocols). Use Zstandard when you need a balance of speed and compression ratio.

Q: Is the LZ4 format standardized? A: The LZ4 frame format is documented in a public specification. The block format is defined by the reference implementation.

Q: What systems use LZ4 internally? A: Linux kernel (transparent filesystem compression), ZFS, MySQL, ClickHouse, Apache Kafka, and many databases use LZ4 for on-disk and in-transit compression.

Q: Can I use LZ4 in a multi-threaded pipeline? A: The CLI tool supports multi-frame parallel compression. The library itself is thread-safe for independent compression contexts.

Sources

• https://github.com/lz4/lz4
• https://lz4.org/