Apache Kafka — Distributed Event Streaming Platform

What it is

Apache Kafka is an open-source distributed event streaming platform originally created at LinkedIn and donated to the Apache Software Foundation. It handles publish-subscribe messaging, durable storage, and stream processing in a single platform. Written in Java and Scala, Kafka organizes data into topics split across partitions for horizontal scalability.

Kafka is used by backend engineers building data pipelines, streaming analytics platforms, and event-driven microservice architectures. If your system needs to move large volumes of events reliably between producers and consumers, Kafka is the standard choice.

How it saves time or tokens

Kafka eliminates the need to build custom message queuing and replay infrastructure. Its partitioned log model means consumers can rewind and replay events without the producer doing extra work. The KRaft mode (available since v3.5) removes the ZooKeeper dependency, cutting operational overhead by eliminating a separate coordination cluster. A single Kafka broker handles millions of messages per second with sub-millisecond latency on commodity hardware.

How to use

1. Download and extract the Kafka binary distribution, then format storage with a generated cluster ID using KRaft mode.
2. Start the broker with bin/kafka-server-start.sh config/kraft/server.properties -- no ZooKeeper required.
3. Create a topic, produce messages via the console producer, and read them back with the console consumer.

Example

# Download Kafka
curl -O https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
tar xzf kafka_2.13-3.7.0.tgz && cd kafka_2.13-3.7.0

# Format storage with KRaft
KAFKA_CLUSTER_ID=$(bin/kafka-storage.sh random-uuid)
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

# Start broker
bin/kafka-server-start.sh config/kraft/server.properties

# Create topic and produce
bin/kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092
bin/kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092

Related on TokRepo

• Automation tools -- browse other event-driven and pipeline automation assets
• Self-hosted tools -- Kafka pairs well with self-hosted data infrastructure

Common pitfalls

• Running ZooKeeper when KRaft mode is available wastes resources and adds operational complexity.
• Setting partition count too low at topic creation time limits consumer parallelism and cannot be decreased later.
• Ignoring consumer group lag monitoring leads to silent data processing delays that surface only during incidents.