# Zstandard (zstd) — Fast Real-Time Compression Algorithm by Meta

> A fast lossless compression algorithm targeting real-time scenarios with high compression ratios, used in the Linux kernel, databases, and package managers.

## Install

Save as a script file and run:

# Zstandard (zstd) — Fast Real-Time Compression Algorithm by Meta

## Quick Use
```bash
# Install
sudo apt install zstd          # Debian/Ubuntu
brew install zstd              # macOS

# Compress and decompress
zstd myfile.txt                # produces myfile.txt.zst
zstd -d myfile.txt.zst        # decompress
zstd -19 myfile.txt           # max compression level
```

## Introduction
Zstandard (zstd) is a real-time compression algorithm developed by Yann Collet at Meta. It provides compression ratios comparable to zlib at speeds approaching lz4, making it suitable for both storage and network use cases where both speed and ratio matter.

## What Zstandard Does
- Compresses data at levels 1-22, covering the full spectrum from speed-optimized to ratio-optimized
- Decompresses at over 1500 MB/s on a single core regardless of compression level
- Trains dictionaries on small data samples for superior compression of small messages
- Supports streaming compression with configurable memory limits
- Provides a CLI tool compatible with gzip-style workflows

## Architecture Overview
Zstandard uses a combination of LZ77 matching with a fast entropy coder (FSE — Finite State Entropy, a tANS implementation). At low levels it favors speed with hash-based match finding. At high levels it employs optimal parsing with suffix arrays. Dictionary compression prepends learned patterns to the frame, dramatically improving ratios on small payloads like JSON or log lines.

## Self-Hosting & Configuration
- Build from source: `make` or `cmake -B build && cmake --build build`
- Link against `libzstd` (shared/static) via pkg-config or CMake `find_package`
- CLI supports gzip-compatible `-c`, `-d`, `-k` flags for drop-in replacement
- Train dictionaries with `zstd --train` on representative samples (ideal for small records)
- Tunable memory usage via `ZSTD_c_windowLog` for constrained environments

## Key Features
- Used in the Linux kernel (btrfs, squashfs), FreeBSD, MySQL, PostgreSQL, and Hadoop
- Adaptive compression adjusts level in real time based on I/O speed
- Long-range matching mode for backup-style workloads with repeated patterns
- Multi-threaded compression via `pzstd` or built-in `ZSTD_c_nbWorkers`
- BSD licensed with no dependencies beyond a C99 compiler

## Comparison with Similar Tools
- **lz4** — faster compression and decompression but lower ratio; ideal for caching
- **gzip/zlib** — ubiquitous but 3-5x slower at similar ratios
- **brotli** — better ratio for web content but slower compression; designed for HTTP
- **xz/lzma** — highest ratios but 10-50x slower; suited for archival only
- **snappy** — Google's fast codec; similar speed to zstd level 1 but worse ratio

## FAQ
**Q: When should I use zstd over gzip?**
A: Almost always. zstd compresses faster and produces smaller output at comparable levels. Use gzip only when interoperability with legacy systems is required.

**Q: How does dictionary compression work?**
A: Train a dictionary on sample data with `zstd --train`. The dictionary captures common patterns and is prepended to each frame, allowing better compression of small records (under 4 KB).

**Q: Is zstd supported in my programming language?**
A: Yes. Official bindings exist for C/C++. Community bindings cover Python, Rust, Go, Java, Node.js, and more.

**Q: Can I use zstd for real-time network traffic?**
A: Yes. At level 1 it compresses faster than network throughput on most links, making it ideal for RPC and log shipping.

## Sources
- https://github.com/facebook/zstd
- https://facebook.github.io/zstd/

---
Source: https://tokrepo.com/en/workflows/asset-748e0a04
Author: Script Depot