Zstandard (zstd) — Fast Real-Time Compression Algorithm by Meta

Introduction

Zstandard (zstd) is a real-time compression algorithm developed by Yann Collet at Meta. It provides compression ratios comparable to zlib at speeds approaching lz4, making it suitable for both storage and network use cases where both speed and ratio matter.

What Zstandard Does

• Compresses data at levels 1-22, covering the full spectrum from speed-optimized to ratio-optimized
• Decompresses at over 1500 MB/s on a single core regardless of compression level
• Trains dictionaries on small data samples for superior compression of small messages
• Supports streaming compression with configurable memory limits
• Provides a CLI tool compatible with gzip-style workflows

Architecture Overview

Zstandard uses a combination of LZ77 matching with a fast entropy coder (FSE — Finite State Entropy, a tANS implementation). At low levels it favors speed with hash-based match finding. At high levels it employs optimal parsing with suffix arrays. Dictionary compression prepends learned patterns to the frame, dramatically improving ratios on small payloads like JSON or log lines.

Self-Hosting & Configuration

• Build from source: make or cmake -B build && cmake --build build
• Link against libzstd (shared/static) via pkg-config or CMake find_package
• CLI supports gzip-compatible -c, -d, -k flags for drop-in replacement
• Train dictionaries with zstd --train on representative samples (ideal for small records)
• Tunable memory usage via ZSTD_c_windowLog for constrained environments

Key Features

• Used in the Linux kernel (btrfs, squashfs), FreeBSD, MySQL, PostgreSQL, and Hadoop
• Adaptive compression adjusts level in real time based on I/O speed
• Long-range matching mode for backup-style workloads with repeated patterns
• Multi-threaded compression via pzstd or built-in ZSTD_c_nbWorkers
• BSD licensed with no dependencies beyond a C99 compiler

Comparison with Similar Tools

• lz4 — faster compression and decompression but lower ratio; ideal for caching
• gzip/zlib — ubiquitous but 3-5x slower at similar ratios
• brotli — better ratio for web content but slower compression; designed for HTTP
• xz/lzma — highest ratios but 10-50x slower; suited for archival only
• snappy — Google's fast codec; similar speed to zstd level 1 but worse ratio

FAQ

Q: When should I use zstd over gzip? A: Almost always. zstd compresses faster and produces smaller output at comparable levels. Use gzip only when interoperability with legacy systems is required.

Q: How does dictionary compression work? A: Train a dictionary on sample data with zstd --train. The dictionary captures common patterns and is prepended to each frame, allowing better compression of small records (under 4 KB).

Q: Is zstd supported in my programming language? A: Yes. Official bindings exist for C/C++. Community bindings cover Python, Rust, Go, Java, Node.js, and more.

Q: Can I use zstd for real-time network traffic? A: Yes. At level 1 it compresses faster than network throughput on most links, making it ideal for RPC and log shipping.

Sources

• https://github.com/facebook/zstd
• https://facebook.github.io/zstd/