Redpanda — Kafka-Compatible Streaming Platform Without JVM

What it is

Redpanda is a streaming data platform written in C++ that provides full Kafka API compatibility. It is a drop-in replacement for Apache Kafka that eliminates the JVM, ZooKeeper, and the operational complexity they bring. Redpanda ships as a single binary with a built-in schema registry, HTTP proxy, and admin API.

Redpanda is designed for teams running Kafka workloads who want simpler operations, lower tail latency, and fewer dependencies.

How it saves time or tokens

Kafka deployments require tuning JVM garbage collection, managing ZooKeeper ensembles, and handling the operational overhead of a multi-process system. Redpanda eliminates all of that. A single binary replaces the Kafka broker, ZooKeeper, schema registry, and REST proxy. Deployment goes from managing 4+ processes to 1. The C++ implementation also provides lower p99 latency because there are no GC pauses.

How to use

1. Install the rpk CLI and start a single-node cluster:

# macOS
brew install redpanda-data/tap/redpanda

# Docker single-node
docker run -d --name redpanda \
  -p 9092:9092 -p 8081:8081 -p 8082:8082 -p 9644:9644 \
  docker.redpanda.com/redpandadata/redpanda:latest \
  redpanda start --smp 1 --memory 1G --overprovisioned

1. Create a topic and produce messages:

rpk topic create my-topic --partitions 3
echo 'hello redpanda' | rpk topic produce my-topic
rpk topic consume my-topic

1. Point your existing Kafka clients at localhost:9092. No client library changes needed since Redpanda speaks the Kafka wire protocol natively.

Example

Migrating from Kafka to Redpanda typically requires only changing the broker address:

from kafka import KafkaProducer, KafkaConsumer

# Before (Kafka)
# producer = KafkaProducer(bootstrap_servers='kafka:9092')

# After (Redpanda) — same client library, just change the address
producer = KafkaProducer(bootstrap_servers='redpanda:9092')
producer.send('my-topic', b'hello from redpanda')
producer.flush()

consumer = KafkaConsumer('my-topic', bootstrap_servers='redpanda:9092')
for msg in consumer:
    print(msg.value)

Related on TokRepo

• DevOps tools — Browse infrastructure and streaming tools
• Self-hosted tools — Explore self-hosted data infrastructure

Common pitfalls

• Assuming Redpanda supports every Kafka feature identically. While the Kafka API compatibility is high, some edge cases around transactions or specific client configurations may differ. Test your exact workload before migrating production.
• Over-provisioning resources for development. Redpanda's single-binary design means a development instance needs far less memory and CPU than Kafka. Use --smp 1 --memory 1G for local development.
• Not using the built-in schema registry. Redpanda includes a schema registry at port 8081 by default. There is no need to deploy a separate Confluent Schema Registry.