Redpanda Connect (Benthos) — Declarative Stream Processing Engine

Introduction

Redpanda Connect (formerly Benthos) makes it trivial to wire together data streams with YAML. Instead of writing custom connectors in code, you declare inputs, processors, and outputs in a config file. It handles backpressure, retries, and batching so engineers focus on the transformation logic rather than plumbing.

What Redpanda Connect Does

• Reads from 60+ inputs including Kafka, AMQP, HTTP, S3, NATS, and files
• Applies transformations using Bloblang, a purpose-built mapping language
• Writes to 60+ outputs with automatic batching, retries, and backpressure
• Chains processors for filtering, enrichment, deduplication, and branching
• Runs as a single binary with no external dependencies

Architecture Overview

Redpanda Connect is a single Go binary that reads a YAML config describing an input, a pipeline of processors, and an output. The engine manages internal buffering and backpressure between stages. Bloblang is the built-in functional mapping language for transformations. Multiple streams can run in a single process with the streams mode API. Resources like caches and rate limits are shared across the config.

Self-Hosting & Configuration

• Download a single binary or use Docker: docker run redpandadata/connect run config.yaml
• Define input, pipeline, and output sections in a YAML config file
• Use Bloblang mappings for transformations without writing Go code
• Enable streams mode to manage multiple pipelines via a REST API
• Monitor with built-in Prometheus metrics and structured JSON logging

Key Features

• Zero-dependency single binary deployment
• Bloblang: expressive mapping language for complex transformations
• 60+ inputs and outputs with automatic retries and backpressure
• Streams mode for running many pipelines in one process
• Built-in caching, rate limiting, and circuit-breaking

Comparison with Similar Tools

• Apache Kafka Connect — JVM-based, requires Kafka; Redpanda Connect is a standalone binary that works with any source or sink
• Apache NiFi — visual flow-based tool for enterprise ETL; Redpanda Connect is config-as-code and much lighter
• Logstash — log-focused pipeline; Redpanda Connect handles structured data transformations with Bloblang
• Fluentd/Fluent Bit — log collection agents; Redpanda Connect is a general-purpose stream processor
• Apache Flink — distributed stream engine for complex analytics; Redpanda Connect is for lightweight routing and transformation

FAQ

Q: Is Benthos the same as Redpanda Connect? A: Yes. Benthos was renamed to Redpanda Connect after Redpanda acquired the project. The core functionality is identical.

Q: What is Bloblang? A: Bloblang is a purpose-built functional mapping language for data transformation. It supports conditionals, string manipulation, JSON navigation, and function composition.

Q: Can it handle high throughput? A: Yes. Redpanda Connect is written in Go with efficient buffering and can process hundreds of thousands of messages per second on a single core.

Q: Does it require Redpanda (the Kafka alternative)? A: No. Redpanda Connect works with any input and output. It is a standalone tool that does not require Redpanda or Kafka.

Sources

• https://github.com/redpanda-data/benthos
• https://docs.redpanda.com/redpanda-connect/