LMDB — Lightning-Fast Memory-Mapped Key-Value Store

Introduction

LMDB (Lightning Memory-Mapped Database) is a compact, high-performance embedded key-value store originally developed for OpenLDAP. It uses memory-mapped files to provide zero-copy reads, MVCC concurrency with a single writer and unlimited readers, and full ACID transactions — all in roughly 10,000 lines of C with no external dependencies.

What LMDB Does

• Stores sorted key-value pairs in a B+ tree backed by memory-mapped files
• Provides fully ACID transactions with copy-on-write MVCC for crash safety
• Allows unlimited concurrent readers with zero locking overhead on read paths
• Delivers zero-copy reads — data is accessed directly from the memory map without deserialization
• Supports multiple named databases within a single environment

Architecture Overview

LMDB maps the entire database file into the process address space using mmap. Reads return pointers directly into the mapped region, avoiding any copy. Writes use copy-on-write on B+ tree pages and a single-writer lock. A two-slot meta-page scheme provides atomic commits without write-ahead logs or compaction. The file format is the native memory layout, making it platform-specific but extremely fast.

Self-Hosting & Configuration

• Available in most Linux distributions as liblmdb-dev; compile from source with a single make
• Open an environment with mdb_env_open() specifying the directory path and max database size
• Set MDB_MAPSIZE to the maximum expected database size — LMDB does not grow the file dynamically
• Use MDB_NOLOCK for single-process access to skip the reader lock table
• Bindings exist for Python (lmdb), Go (lmdb-go), Rust (heed), Node.js, and most other languages

Key Features

• Entire codebase is ~10K lines of C with zero external dependencies
• Read performance approaches raw memory bandwidth with zero-copy access
• Crash-proof — no WAL or recovery process needed after unexpected shutdown
• Readers never block writers and writers never block readers via MVCC
• Database file is a single regular file, easy to back up with a simple file copy

Comparison with Similar Tools

• RocksDB — LSM-tree design optimized for write-heavy workloads; LMDB excels at read-heavy patterns
• LevelDB — Google's LSM store; LMDB offers stronger consistency and zero-copy reads
• BoltDB/bbolt — Go B+ tree database inspired by LMDB; LMDB is the original C implementation
• SQLite — Full SQL database; LMDB is lower-level but faster for pure key-value access
• BadgerDB — Go key-value store with value separation; LMDB is simpler and battle-tested over 10+ years

FAQ

Q: When should I use LMDB over RocksDB? A: Use LMDB when reads dominate writes and you want zero-copy access. Use RocksDB for write-intensive workloads.

Q: Does LMDB support compression? A: No built-in compression. The memory-mapped design prioritizes access speed over storage efficiency.

Q: Is LMDB thread-safe? A: Yes. Multiple threads can read concurrently. Writes are serialized via a single-writer lock.

Q: What is the maximum database size? A: Set via MDB_MAPSIZE at environment creation. On 64-bit systems, values up to several terabytes are practical.

Sources

• https://github.com/LMDB/lmdb
• http://www.lmdb.tech/doc/