# Redpanda Connect (Benthos) — Declarative Stream Processing Engine

> Redpanda Connect, formerly known as Benthos, is a high-performance declarative stream processor written in Go. It connects, transforms, and routes data between any combination of sources and sinks using simple YAML configuration files.

## Install

Save as a script file and run:

# Redpanda Connect (Benthos) — Declarative Stream Processing Engine

## Quick Use
```bash
# Install via Homebrew or download binary
brew install redpanda-data/tap/redpanda-connect
# Run with a YAML config
rpk connect run config.yaml
# Or pipe data inline
echo '{"msg":"hello"}' | rpk connect run -r "mapping: root.upper = this.msg.uppercase()"
```

## Introduction
Redpanda Connect (formerly Benthos) makes it trivial to wire together data streams with YAML. Instead of writing custom connectors in code, you declare inputs, processors, and outputs in a config file. It handles backpressure, retries, and batching so engineers focus on the transformation logic rather than plumbing.

## What Redpanda Connect Does
- Reads from 60+ inputs including Kafka, AMQP, HTTP, S3, NATS, and files
- Applies transformations using Bloblang, a purpose-built mapping language
- Writes to 60+ outputs with automatic batching, retries, and backpressure
- Chains processors for filtering, enrichment, deduplication, and branching
- Runs as a single binary with no external dependencies

## Architecture Overview
Redpanda Connect is a single Go binary that reads a YAML config describing an input, a pipeline of processors, and an output. The engine manages internal buffering and backpressure between stages. Bloblang is the built-in functional mapping language for transformations. Multiple streams can run in a single process with the streams mode API. Resources like caches and rate limits are shared across the config.

## Self-Hosting & Configuration
- Download a single binary or use Docker: docker run redpandadata/connect run config.yaml
- Define input, pipeline, and output sections in a YAML config file
- Use Bloblang mappings for transformations without writing Go code
- Enable streams mode to manage multiple pipelines via a REST API
- Monitor with built-in Prometheus metrics and structured JSON logging

## Key Features
- Zero-dependency single binary deployment
- Bloblang: expressive mapping language for complex transformations
- 60+ inputs and outputs with automatic retries and backpressure
- Streams mode for running many pipelines in one process
- Built-in caching, rate limiting, and circuit-breaking

## Comparison with Similar Tools
- **Apache Kafka Connect** — JVM-based, requires Kafka; Redpanda Connect is a standalone binary that works with any source or sink
- **Apache NiFi** — visual flow-based tool for enterprise ETL; Redpanda Connect is config-as-code and much lighter
- **Logstash** — log-focused pipeline; Redpanda Connect handles structured data transformations with Bloblang
- **Fluentd/Fluent Bit** — log collection agents; Redpanda Connect is a general-purpose stream processor
- **Apache Flink** — distributed stream engine for complex analytics; Redpanda Connect is for lightweight routing and transformation

## FAQ
**Q: Is Benthos the same as Redpanda Connect?**
A: Yes. Benthos was renamed to Redpanda Connect after Redpanda acquired the project. The core functionality is identical.

**Q: What is Bloblang?**
A: Bloblang is a purpose-built functional mapping language for data transformation. It supports conditionals, string manipulation, JSON navigation, and function composition.

**Q: Can it handle high throughput?**
A: Yes. Redpanda Connect is written in Go with efficient buffering and can process hundreds of thousands of messages per second on a single core.

**Q: Does it require Redpanda (the Kafka alternative)?**
A: No. Redpanda Connect works with any input and output. It is a standalone tool that does not require Redpanda or Kafka.

## Sources
- https://github.com/redpanda-data/benthos
- https://docs.redpanda.com/redpanda-connect/

---
Source: https://tokrepo.com/en/workflows/e95cabb2-39eb-11f1-9bc6-00163e2b0d79
Author: Script Depot