# LZ4 — Extremely Fast Lossless Compression Algorithm

> LZ4 is a lossless compression algorithm focused on compression and decompression speed. It achieves over 500 MB/s compression and multiple GB/s decompression per core, making it the standard choice for real-time data processing and storage systems.

## Install

Save as a script file and run:

# LZ4 — Extremely Fast Lossless Compression Algorithm

## Quick Use
```bash
# Install
sudo apt install lz4
# Compress a file
lz4 data.bin data.bin.lz4
# Decompress
lz4 -d data.bin.lz4 data.bin
# High compression mode
lz4 -9 data.bin data.bin.lz4
# Benchmark
lz4 -b data.bin
```

## Introduction
LZ4 is a lossless data compression algorithm that prioritizes speed above all else. It delivers compression speeds exceeding 780 MB/s and decompression speeds near 5 GB/s per core on modern hardware, making it the go-to choice for scenarios where latency matters more than file size.

## What LZ4 Does
- Compresses data at over 780 MB/s per core while achieving reasonable compression ratios
- Decompresses at nearly 5 GB/s per core, fast enough to be transparent in data pipelines
- Provides an HC (High Compression) variant that trades compression speed for 20-30% better ratios
- Offers both a block-level API for embedding in storage engines and a frame format for standalone files
- Supports dictionary compression for improving ratios on small, repetitive data like log entries

## Architecture Overview
LZ4 is an LZ77-family byte-oriented compressor that uses a hash table to find matches in a 64KB sliding window. The format encodes literal runs and match references in a simple token-based scheme with minimal branching, enabling the decompressor to run almost entirely in L1 cache. The frame format (`.lz4`) adds checksums, content size hints, and dictionary IDs on top of the raw block format. LZ4-HC uses the same decompression format but employs a more exhaustive match search at compression time.

## Self-Hosting & Configuration
- Install via package manager: `apt install liblz4-dev` or `brew install lz4`
- Build from source: `make` in the project root produces the CLI tool and shared library
- CMake build available: `mkdir build && cd build && cmake .. && make`
- C API: include `lz4.h` and link `-llz4`; two functions: `LZ4_compress_default()` and `LZ4_decompress_safe()`
- Available via vcpkg, Conan, and built into many Linux distributions

## Key Features
- Decompression speed of ~5 GB/s per core, among the fastest of any general-purpose compressor
- Codec simplicity enables hardware-friendly implementation with minimal branch misprediction
- LZ4-HC mode produces output compatible with the fast decompressor while achieving better ratios
- Frame format supports streaming compression with configurable block sizes (64KB to 4MB)
- Dictionary mode improves compression of small messages by seeding the compressor with shared context

## Comparison with Similar Tools
- **Snappy (Google)** — similar speed class but LZ4 achieves slightly better ratios and faster decompression
- **Zstandard (zstd)** — better compression ratios with configurable speed; LZ4 is faster at the lowest latency levels
- **gzip/zlib** — much higher compression ratios but 5-10x slower to decompress; LZ4 wins when speed is critical
- **Brotli** — optimized for web content with superior ratios; LZ4 targets low-latency storage and networking
- **lzo** — comparable speed philosophy but LZ4 has a simpler API and better community support

## FAQ
**Q: When should I use LZ4 vs Zstandard?**
A: Use LZ4 when decompression latency is the top priority (storage engines, network protocols). Use Zstandard when you need a balance of speed and compression ratio.

**Q: Is the LZ4 format standardized?**
A: The LZ4 frame format is documented in a public specification. The block format is defined by the reference implementation.

**Q: What systems use LZ4 internally?**
A: Linux kernel (transparent filesystem compression), ZFS, MySQL, ClickHouse, Apache Kafka, and many databases use LZ4 for on-disk and in-transit compression.

**Q: Can I use LZ4 in a multi-threaded pipeline?**
A: The CLI tool supports multi-frame parallel compression. The library itself is thread-safe for independent compression contexts.

## Sources
- https://github.com/lz4/lz4
- https://lz4.org/

---
Source: https://tokrepo.com/en/workflows/asset-23033b77
Author: Script Depot