Ceph — Unified Distributed Storage at Scale

What it is

Ceph is the reference open-source distributed storage system, originally developed at UC Santa Cruz and now stewarded by the Ceph Foundation (Linux Foundation). It presents a single cluster as block storage (RBD), S3/Swift-compatible object storage (RGW), and POSIX file access (CephFS).

Ceph powers OpenStack, Rook on Kubernetes, and thousands of private clouds. It scales from three nodes to exabytes with no single point of failure, self-healing through replication or erasure coding.

How it saves time or tokens

Ceph unifies three storage interfaces in one cluster. Instead of managing separate systems for VM block devices, object storage, and shared filesystems, you deploy Ceph once and expose all three. The CRUSH algorithm distributes data pseudo-randomly across OSDs without a central metadata server, eliminating bottlenecks. Self-healing means that when a disk or node fails, Ceph automatically re-replicates data to maintain the configured redundancy level.

How to use

1. Bootstrap a cluster with cephadm:

curl --silent --remote-name --location \
  https://github.com/ceph/ceph/raw/main/src/cephadm/cephadm
chmod +x cephadm
./cephadm add-repo --release reef
./cephadm install
cephadm bootstrap --mon-ip 192.168.1.10

1. Create a block device and mount it:

ceph osd pool create rbd 32
rbd create demo --size 1G --pool rbd
rbd map demo -p rbd
mkfs.ext4 /dev/rbd0 && mount /dev/rbd0 /mnt/demo

1. Enable S3-compatible object storage:

ceph orch apply rgw mystore
# Access via S3 API at http://node:80

Example

Using Ceph RBD as persistent volumes in Kubernetes via Rook:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
  storage:
    useAllNodes: true
    useAllDevices: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  pool: replicapool
  clusterID: rook-ceph
reclaimPolicy: Delete

Related on TokRepo

• Self-hosted tools — More self-hosted infrastructure tools on TokRepo.
• DevOps tools — Browse infrastructure and storage tools.

Common pitfalls

• Running fewer than 3 monitor daemons risks losing quorum. Always deploy an odd number of monitors (3 or 5).
• Not sizing OSD journals on SSDs causes poor write performance. Place OSD journals on fast NVMe drives for HDD-backed clusters.
• Enabling erasure coding without understanding the recovery overhead. Erasure coding saves storage space but increases CPU usage during recovery compared to replication.