Jepsen — Distributed Systems Verification with Fault Injection

Introduction

Jepsen is a framework for testing the correctness of distributed systems under fault conditions. Created by Kyle Kingsbury, it has uncovered critical bugs in databases such as PostgreSQL, MongoDB, CockroachDB, Redis, and many others. Each Jepsen analysis is published openly, making it a de facto audit standard for distributed data systems.

What Jepsen Does

• Injects network partitions, process crashes, and clock skew into live clusters
• Runs concurrent client operations against the system under test
• Records a history of all operations and their outcomes
• Checks the recorded history against formal consistency models (linearizability, serializability, etc.)
• Generates detailed timeline visualizations and HTML reports

Architecture Overview

Jepsen orchestrates a cluster of Docker containers or VMs via SSH. A control node runs the test harness written in Clojure, which spawns client threads to issue operations against the system. A separate nemesis thread introduces faults on a schedule. After the test, a checker module feeds the operation history into a consistency model verifier (typically Elle or Knossos) to detect violations.

Self-Hosting & Configuration

• Requires a set of Debian-based nodes reachable via SSH (Docker Compose setup included)
• Tests are written in Clojure and run with Leiningen
• Configure fault scenarios (partitions, kills, clock skew) via nemesis composition
• Results are stored locally with HTML reports and SVG timelines
• Each database under test needs a dedicated namespace with setup and teardown logic

Key Features

• Formal consistency checking with Elle (fast) and Knossos (thorough) verifiers
• Composable nemesis system for combining fault types
• Detailed HTML reports with latency plots and timeline visualizations
• Published analyses covering dozens of production databases
• Supports testing queues, registers, sets, and append workloads

Comparison with Similar Tools

• Chaos Mesh — Injects faults in Kubernetes but does not verify correctness; Jepsen checks consistency
• Litmus — Chaos engineering platform; Jepsen is a correctness verification framework
• TLA+ — Model checks specifications; Jepsen tests actual running software
• FoundationDB Simulation — Built-in deterministic testing; Jepsen is system-agnostic

FAQ

Q: Do I need to know Clojure to use Jepsen? A: Basic Clojure knowledge helps, but the framework provides templates and examples to get started quickly.

Q: Can Jepsen test my custom application? A: Yes. You write a client namespace that maps your application's operations to Jepsen's operation model.

Q: How long does a typical test run take? A: Most tests run for a few minutes, though longer durations increase the chance of catching rare bugs.

Q: Are Jepsen analyses available publicly? A: Yes. Kyle Kingsbury publishes detailed reports at jepsen.io for many popular databases.

Sources

• https://github.com/jepsen-io/jepsen
• https://jepsen.io