Sarama — Pure Go Client Library for Apache Kafka

Introduction

Sarama is the most established Go client library for Apache Kafka. Originally created by Shopify and now maintained under IBM, it provides a pure Go implementation covering the full Kafka protocol—producing, consuming, consumer groups, and cluster administration—without requiring CGO or external C libraries.

What Sarama Does

• Implements the Kafka wire protocol in pure Go for all broker API versions
• Provides sync and async producers with configurable batching and compression
• Supports consumer groups with automatic partition rebalancing
• Includes admin client for topic creation, deletion, and configuration management
• Handles SASL authentication (PLAIN, SCRAM, OAUTHBEARER) and TLS encryption

Architecture Overview

Sarama maintains persistent TCP connections to Kafka brokers, multiplexing requests across them. The AsyncProducer batches messages by topic-partition and flushes based on size or time thresholds, achieving high throughput. The ConsumerGroup implementation coordinates partition assignment through the Kafka group protocol, handling heartbeats, rebalances, and offset commits. All protocol encoding is done in pure Go using binary readers/writers matching the Kafka wire format.

Self-Hosting & Configuration

• Install: go get github.com/IBM/sarama
• Configure via sarama.NewConfig() setting version, producer acks, and consumer offsets
• Set config.Version to match your Kafka cluster version for feature compatibility
• Enable TLS and SASL via config.Net.TLS and config.Net.SASL fields
• Use config.Producer.RequiredAcks = sarama.WaitForAll for durable writes

Key Features

• Pure Go with no CGO dependencies, simplifying cross-compilation and containerization
• Supports Kafka protocol versions from 0.8 through 3.x+
• Async producer achieves high throughput with configurable batching and compression (gzip, snappy, lz4, zstd)
• Consumer group implementation with pluggable rebalance strategies
• Built-in metrics via a pluggable interface compatible with go-metrics

Comparison with Similar Tools

• confluent-kafka-go — wraps librdkafka (C library) for higher raw throughput; Sarama is pure Go with simpler deployment
• franz-go — newer pure Go client with modern API; Sarama has longer track record and broader community adoption
• segmentio/kafka-go — simpler API for basic use cases; Sarama covers more protocol features and admin operations
• Watermill — event-driven framework that can use Sarama as a backend; Sarama is lower-level
• librdkafka — C client with bindings; Sarama avoids CGO complexity in Go deployments

FAQ

Q: Should I use Sarama or confluent-kafka-go? A: Use Sarama if you want pure Go with no CGO. Use confluent-kafka-go if you need maximum throughput and can accept C library dependencies.

Q: How do I handle consumer group rebalances? A: Implement the ConsumerGroupHandler interface with Setup, Cleanup, and ConsumeClaim methods to manage partition assignment lifecycle.

Q: Does Sarama support exactly-once semantics? A: Sarama supports idempotent producers and transactional APIs for exactly-once within Kafka, but end-to-end exactly-once depends on your consumer logic.

Q: How do I monitor Sarama in production? A: Register a metrics registry via config.MetricRegistry to export producer/consumer metrics to Prometheus or other systems.

Sources

• https://github.com/IBM/sarama
• https://pkg.go.dev/github.com/IBM/sarama