WAL-G — Cloud-Native Database Backup and Archival Tool

Introduction

WAL-G is an open-source archival and restoration tool for databases in the cloud. Originally built as a successor to WAL-E for PostgreSQL, it has expanded to support MySQL, MongoDB, SQL Server, and more. WAL-G efficiently compresses and uploads base backups and write-ahead log segments to S3, GCS, Azure, or any S3-compatible storage.

What WAL-G Does

• Creates and manages full and incremental base backups for PostgreSQL, MySQL, and MongoDB
• Archives write-ahead log (WAL) segments continuously for point-in-time recovery
• Compresses backups using LZ4, LZMA, Brotli, or zstd to minimize storage costs
• Encrypts backups at rest using AES-256 or GPG for security compliance
• Supports retention policies to automatically delete old backups

Architecture Overview

WAL-G is a single Go binary that integrates with the database's native backup mechanisms. For PostgreSQL, it hooks into pg_basebackup and the WAL archiving system. When a backup is triggered, WAL-G reads the data directory, compresses it in parallel using multiple CPU cores, and uploads chunks to the configured object storage. WAL segments are archived individually as they are produced. For restoration, WAL-G downloads the base backup, then replays WAL segments up to the desired point in time. Delta backups store only changed pages since the last full backup, reducing upload size.

Self-Hosting & Configuration

• Install as a single binary — no daemon or service infrastructure required
• Configure storage credentials and paths through environment variables
• Set WAL archiving in postgresql.conf to call wal-g wal-push automatically
• Define retention policies (keep N backups or backups newer than N days)
• Schedule full backups via cron and let WAL archiving handle continuous protection

Key Features

• Parallel compression and upload saturates both CPU and network bandwidth
• Delta (incremental) backups reduce backup size by 80-90% for large databases
• Point-in-time recovery to any moment between the oldest backup and the latest WAL
• Supports S3, GCS, Azure Blob, Swift, and any S3-compatible storage (MinIO, Ceph)
• Copy-on-write page tracking for PostgreSQL enables efficient delta backups

Comparison with Similar Tools

• pg_dump — logical backup only, no WAL archiving; WAL-G provides physical backups with PITR
• pgBackRest — similar feature set for PostgreSQL; WAL-G is lighter and supports more databases
• Barman — PostgreSQL backup manager by EDB; WAL-G is simpler to configure with cloud storage
• BorgBackup — general-purpose dedup backup; WAL-G understands database internals for consistent backups
• Restic — encrypted cloud backup tool; WAL-G integrates with WAL archiving for zero-data-loss guarantees

FAQ

Q: Can WAL-G do point-in-time recovery? A: Yes. WAL-G restores a base backup and replays WAL segments to any timestamp you specify, giving you point-in-time recovery with second-level granularity.

Q: How much storage do delta backups save? A: For databases with moderate write rates, delta backups are typically 80-90% smaller than full backups. The exact savings depend on your change rate.

Q: Does WAL-G work with managed PostgreSQL services? A: WAL-G requires access to the data directory and WAL archiving configuration. It works with self-managed PostgreSQL but not with fully managed services like RDS where those are not exposed.

Q: Can I use WAL-G with MinIO instead of AWS S3? A: Yes. Set the WALG_S3_PREFIX and AWS_ENDPOINT environment variables to point at your MinIO instance. WAL-G treats it the same as S3.

Sources

• https://github.com/wal-g/wal-g
• https://wal-g.readthedocs.io