# Jepsen — Distributed Systems Verification with Fault Injection

> A Clojure framework for testing the safety and consistency guarantees of distributed databases and queues under real-world failures.

## Install

Save in your project root:

# Jepsen — Distributed Systems Verification with Fault Injection

## Quick Use
```bash
# Clone and run a test against etcd
git clone https://github.com/jepsen-io/jepsen.git
cd jepsen/jepsen
lein run test --nodes n1,n2,n3,n4,n5 --workload etcdemo
# Results and timeline plots are written to store/
```

## Introduction
Jepsen is a framework for testing the correctness of distributed systems under fault conditions. Created by Kyle Kingsbury, it has uncovered critical bugs in databases such as PostgreSQL, MongoDB, CockroachDB, Redis, and many others. Each Jepsen analysis is published openly, making it a de facto audit standard for distributed data systems.

## What Jepsen Does
- Injects network partitions, process crashes, and clock skew into live clusters
- Runs concurrent client operations against the system under test
- Records a history of all operations and their outcomes
- Checks the recorded history against formal consistency models (linearizability, serializability, etc.)
- Generates detailed timeline visualizations and HTML reports

## Architecture Overview
Jepsen orchestrates a cluster of Docker containers or VMs via SSH. A control node runs the test harness written in Clojure, which spawns client threads to issue operations against the system. A separate nemesis thread introduces faults on a schedule. After the test, a checker module feeds the operation history into a consistency model verifier (typically Elle or Knossos) to detect violations.

## Self-Hosting & Configuration
- Requires a set of Debian-based nodes reachable via SSH (Docker Compose setup included)
- Tests are written in Clojure and run with Leiningen
- Configure fault scenarios (partitions, kills, clock skew) via nemesis composition
- Results are stored locally with HTML reports and SVG timelines
- Each database under test needs a dedicated namespace with setup and teardown logic

## Key Features
- Formal consistency checking with Elle (fast) and Knossos (thorough) verifiers
- Composable nemesis system for combining fault types
- Detailed HTML reports with latency plots and timeline visualizations
- Published analyses covering dozens of production databases
- Supports testing queues, registers, sets, and append workloads

## Comparison with Similar Tools
- **Chaos Mesh** — Injects faults in Kubernetes but does not verify correctness; Jepsen checks consistency
- **Litmus** — Chaos engineering platform; Jepsen is a correctness verification framework
- **TLA+** — Model checks specifications; Jepsen tests actual running software
- **FoundationDB Simulation** — Built-in deterministic testing; Jepsen is system-agnostic

## FAQ
**Q: Do I need to know Clojure to use Jepsen?**
A: Basic Clojure knowledge helps, but the framework provides templates and examples to get started quickly.

**Q: Can Jepsen test my custom application?**
A: Yes. You write a client namespace that maps your application's operations to Jepsen's operation model.

**Q: How long does a typical test run take?**
A: Most tests run for a few minutes, though longer durations increase the chance of catching rare bugs.

**Q: Are Jepsen analyses available publicly?**
A: Yes. Kyle Kingsbury publishes detailed reports at jepsen.io for many popular databases.

## Sources
- https://github.com/jepsen-io/jepsen
- https://jepsen.io

---
Source: https://tokrepo.com/en/workflows/asset-16df6ba1
Author: AI Open Source