Duplicacy — Lock-Free Deduplication Cloud Backup

Introduction

Duplicacy is a cross-platform backup tool that uses lock-free deduplication to allow multiple machines to share the same cloud storage without conflicts. Unlike traditional backup tools, it requires no central server — each client backs up independently while still deduplicating data across all clients.

What Duplicacy Does

• Deduplicates data at the chunk level across multiple machines sharing the same storage
• Backs up to 30+ cloud storage backends including S3, Backblaze B2, Azure, GCS, SFTP, and local disks
• Uses lock-free design so multiple clients can back up simultaneously without coordination
• Encrypts all data client-side with AES-256-GCM before uploading
• Supports incremental backups with configurable retention policies and pruning

Architecture Overview

Duplicacy splits files into variable-length chunks using a rolling hash, then deduplicates by content hash. Chunks are uploaded to the chosen storage backend individually. A snapshot file records which chunks compose each backup revision. The lock-free design avoids centralized locking by using optimistic concurrency — each client reads the latest snapshot index, uploads new chunks, and appends its snapshot. Conflicts are resolved by chunk-level content addressing.

Self-Hosting & Configuration

• Download the single CLI binary (Go, no dependencies) for Linux, macOS, or Windows
• Initialize a repository with duplicacy init <snapshot-id> <storage-url>
• Configure encryption, chunk size, and threading in the .duplicacy/preferences file
• Schedule backups via cron or systemd timers on Linux, Task Scheduler on Windows
• Use the optional Web GUI (commercial license) for managing multiple repositories and schedules

Key Features

• Lock-free deduplication allows concurrent backups from multiple machines to one storage
• Variable-length chunking provides high deduplication ratios across file modifications
• Client-side AES-256-GCM encryption with optional RSA key wrapping
• Supports backup copy to a secondary storage for 3-2-1 strategy
• Fast restore with parallel chunk downloads and in-place file reconstruction

Comparison with Similar Tools

• Restic — Similar chunk-based dedup but uses repository locking; single-repo-per-machine is simpler
• BorgBackup — Mature dedup backup with compression; requires SSH access to the backup server
• Kopia — Modern backup tool with snapshot-based dedup; strong encryption, growing ecosystem
• Duplicati — GUI-friendly backup to cloud; uses block-level dedup but can be slower on large datasets
• rclone — Cloud sync tool without deduplication; good for mirroring but not incremental backup

FAQ

Q: Is Duplicacy free? A: The CLI is open source (personal use). The Web GUI requires a commercial license for non-personal use.

Q: Can two computers back up to the same S3 bucket? A: Yes. Each computer uses a unique snapshot ID. Duplicacy deduplicates common data across all snapshots without coordination locks.

Q: How do I prune old backups? A: Use duplicacy prune -keep 30:180 -keep 7:30 -keep 1:7 to define retention tiers by age.

Q: Does it support compression? A: Yes. Enable zstd or lz4 compression per chunk during initialization with -compression-level.

Sources

• https://github.com/gilbertchen/duplicacy
• https://duplicacy.com