Cap'n Proto — Insanely Fast Data Serialization and RPC

Introduction

Cap'n Proto is a binary serialization format and RPC framework created by Kenton Varda, who previously designed Protocol Buffers v2 at Google. Its core innovation is that the wire format is identical to the in-memory layout, meaning data can be read directly from a buffer with no parsing or deserialization step, achieving effectively infinite decode speed.

What Cap'n Proto Does

• Serializes structured data to a binary format that requires zero deserialization to read
• Provides a capability-based RPC system with promise pipelining to reduce network round trips
• Generates type-safe code for C++, Rust, Go, Python, Java, and other languages from schema files
• Supports schema evolution with backward and forward compatibility through field numbering
• Enables memory-mapped file access where serialized data is read directly without copying

Architecture Overview

Cap'n Proto defines messages as a sequence of fixed-width segments with pointer-based references between objects. Each struct field is stored at a fixed offset, so reading a field is a single pointer dereference with no parsing. The RPC layer builds on this with an object-capability security model where references to remote objects are unforgeable tokens, and promise pipelining allows calling methods on not-yet-returned results to collapse multiple round trips into one.

Self-Hosting & Configuration

• Install via package manager: apt install capnproto or brew install capnp
• Build from source with CMake for the latest features and optimizations
• Schema files (.capnp) define data types and are compiled with capnp compile
• C++ integration requires linking libcapnp and libkj (Cap'n Proto's async I/O library)
• The RPC system uses KJ async framework for event-driven networking

Key Features

• Zero-copy deserialization eliminates parsing latency; data is used directly from the wire buffer
• Promise pipelining in the RPC layer reduces latency by issuing dependent calls before prior results arrive
• Canonical encoding ensures consistent byte output for content hashing and comparison
• Built-in support for memory-mapped files as a zero-cost persistent storage format
• Time-travel RPC allows calling methods on future results, collapsing multi-hop interactions

Comparison with Similar Tools

• Protocol Buffers — requires decode/encode step; Cap'n Proto reads data in-place with no parsing cost
• FlatBuffers — also zero-copy but uses a table-based layout; Cap'n Proto has pointer-based layout and integrated RPC
• MessagePack — compact binary format needing full deserialization; Cap'n Proto trades compactness for zero-copy access
• Apache Thrift — provides serialization and RPC but without zero-copy reads or promise pipelining
• gRPC — HTTP/2-based RPC on Protobuf; Cap'n Proto RPC uses its own protocol with capability security

FAQ

Q: Is Cap'n Proto smaller than Protobuf on the wire? A: Not necessarily. Cap'n Proto messages include padding for alignment, so they can be slightly larger. The benefit is zero-cost deserialization, not compression.

Q: Can I use Cap'n Proto for persistent storage? A: Yes. The format is stable and can be memory-mapped directly from disk files for instant access.

Q: What languages have Cap'n Proto support? A: C++ (reference), Rust, Go, Python, Java, C#, JavaScript, and others via community-maintained libraries.

Q: How does schema evolution work? A: Fields are numbered. New fields can be added at the end, old fields remain readable, and removed fields are simply ignored. This provides forward and backward compatibility.

Sources

• https://github.com/capnproto/capnproto
• https://capnproto.org/