FoundationDB — Distributed Key-Value Store by Apple

Introduction

FoundationDB is a distributed, ordered key-value store built by Apple that provides strict ACID transactions across a cluster. It serves as the foundation layer for higher-level data models, including the document layer, record layer, and Apple's CloudKit infrastructure powering billions of requests daily.

What FoundationDB Does

• Provides strictly serializable ACID transactions across distributed nodes
• Stores ordered key-value pairs with keys and values up to 10KB and 100KB respectively
• Scales linearly by adding nodes, handling millions of operations per second
• Supports multi-model layers: document, record, graph, and spatial built on top
• Uses deterministic simulation testing to verify correctness under failure scenarios

Architecture Overview

FoundationDB separates the transaction system (coordinators, proxies, resolvers) from the storage system (storage servers, log servers). The coordination layer uses Paxos for leader election. Transaction proxies accept client requests, assign read versions, and send mutations to resolvers that check for conflicts. Committed mutations flow through log servers to durable storage. This separation allows independent scaling of compute and storage.

Self-Hosting & Configuration

• Install server and client packages from GitHub releases for Ubuntu, CentOS, or macOS
• Single-node setup works out of the box; configure fdb.cluster file for multi-node
• Set data directory and log directory in foundationdb.conf
• Use the fdbcli tool to configure redundancy mode (single, double, triple)
• Monitor cluster health with fdbcli status and machine-readable status JSON endpoint

Key Features

• Deterministic simulation testing catches bugs that traditional testing misses
• Automatic data distribution and rebalancing across nodes
• Watch API for reactive applications that need to observe key changes
• Backup and restore tools with continuous backup to blob storage
• Multi-region support with configurable replication policies

Comparison with Similar Tools

• etcd — focused on configuration and service discovery; FoundationDB handles general-purpose data workloads
• CockroachDB — SQL layer built-in; FoundationDB provides a key-value primitive for building custom layers
• TiKV — similar distributed KV store; FoundationDB has stronger simulation testing guarantees
• Redis — in-memory with optional persistence; FoundationDB is disk-first with full ACID
• Cassandra — eventually consistent; FoundationDB provides strict serializability

FAQ

Q: Why does Apple use FoundationDB? A: Apple uses it as the backbone for iCloud and CloudKit because its ACID guarantees and linear scalability handle billions of operations reliably.

Q: What is the Record Layer? A: It is a structured data layer built on FoundationDB that provides schema management, indexing, and query planning, used extensively inside Apple.

Q: Can FoundationDB replace my SQL database? A: Not directly. FoundationDB is a key-value store. You need a layer like the Record Layer or Document Layer to get SQL-like features.

Q: How does simulation testing work? A: FoundationDB compiles into a deterministic simulation that can test millions of failure scenarios (network partitions, disk failures, crashes) in minutes.

Sources

• https://github.com/apple/foundationdb
• https://apple.github.io/foundationdb/