Redpanda Connect (Benthos) — Declarative Stream Processing Engine

What it is

Redpanda Connect (formerly Benthos) is a high-performance declarative stream processor written in Go. It connects, transforms, and routes data between 200+ sources and sinks using YAML configuration files. You define inputs, processors, and outputs without writing custom code.

The tool serves data engineers, backend developers, and DevOps teams who need to move and transform data between systems like Kafka, databases, APIs, and file stores.

The project is actively maintained with regular releases and a growing user community. Documentation covers common use cases, and the open-source nature means you can inspect the source code, contribute fixes, and adapt the tool to your specific requirements.

How it saves time or tokens

Instead of writing custom ETL scripts for each data pipeline, Redpanda Connect lets you define the entire flow in YAML. Adding a new source or transformation is a config change, not a code change. The single-binary deployment eliminates dependency management and simplifies CI/CD.

How to use

1. Install the binary: brew install redpanda-data/tap/connect or download from GitHub releases.
2. Create a YAML config defining your input, pipeline processors, and output.
3. Run with rpk connect run config.yml.

Example

# config.yml - Read from Kafka, transform JSON, write to PostgreSQL
input:
  kafka:
    addresses: ['localhost:9092']
    topics: ['events']
    consumer_group: 'connect-group'

pipeline:
  processors:
    - mapping: |
        root.user_id = this.user.id
        root.event = this.type
        root.timestamp = this.created_at

output:
  sql_insert:
    driver: postgres
    dsn: 'postgres://user:pass@localhost:5432/analytics'
    table: events
    columns: ['user_id', 'event', 'timestamp']

Related on TokRepo

• AI Tools for Automation -- automation and data pipeline tools
• AI Tools for DevOps -- infrastructure and operations tooling

Common pitfalls

• The mapping processor uses Bloblang, a domain-specific language. Read the Bloblang docs before writing complex transformations.
• Redpanda Connect buffers messages in memory by default. For high-throughput pipelines, configure explicit buffer limits to avoid OOM.
• The rename from Benthos to Redpanda Connect means older tutorials reference the benthos binary. Use rpk connect for current versions.

Before adopting this tool, evaluate whether it fits your team's existing workflow. Read the official documentation thoroughly, and start with a small proof-of-concept rather than a full migration. Community forums, GitHub issues, and Stack Overflow are valuable resources when you encounter edge cases not covered in the documentation.