OpenTelemetry Collector — Vendor-Neutral Telemetry Pipeline

Introduction

OpenTelemetry Collector is the swiss-army-knife agent of the OpenTelemetry project. Instead of running a Datadog agent, a New Relic agent, a Prometheus node exporter, and a Fluent Bit side-by-side, teams deploy one Collector and configure receivers, processors, and exporters per signal type. It graduated to CNCF-incubating then to the stable trace and metrics pipelines, and is the recommended collection layer in almost every modern observability stack.

What the Collector Does

• Receives telemetry in OTLP, Jaeger, Zipkin, Prometheus, statsd, Fluent Forward, Kafka, and many more.
• Processes data through batching, attribute editing, sampling, redaction, and resource enrichment.
• Exports to vendors (Datadog, Honeycomb, Splunk, New Relic) and OSS stores (Tempo, Jaeger, Prometheus, Loki).
• Acts as a sidecar (agent mode) for low-latency collection or as a standalone gateway for fan-in and routing.
• Generates internal metrics and zpages so you can observe your observability pipeline.

Architecture Overview

The Collector is a Go binary with a component-based plugin model: every receiver, processor, exporter, extension, and connector is a compiled-in module. opentelemetry-collector is the minimal core; opentelemetry-collector-contrib adds the full catalog. Pipelines are directed graphs declared in YAML — a receiver hands data to a sequence of processors, which fan out to exporters. The OpenTelemetry Collector Builder (ocb) lets you build a custom distribution with only the components you need, shrinking image size and attack surface.

Self-Hosting & Configuration

• Pick otelcol (core) for stability or otelcol-contrib for breadth; build your own distro with ocb.
• Deploy agents as a DaemonSet on Kubernetes, gateway pods as a Deployment behind a Service.
• Use memory_limiter, batch, and retry_on_failure processors in every production pipeline.
• Secure OTLP endpoints with mTLS and enforce auth via the bearertokenauth or oidc extensions.
• Enable zpages on a sidecar port for live pipeline inspection without scraping metrics.

Key Features

• Single binary for metrics, logs, and traces — no language-specific agent sprawl.
• Hot-reloadable YAML config; validation via --dry-run.
• Tail-sampling and probabilistic-sampling processors for cost control.
• Transform processor with OTTL (OpenTelemetry Transformation Language) for surgical edits.
• First-class Prometheus compatibility: scrape classic /metrics endpoints and remote-write to any TSDB.

Comparison with Similar Tools

• Fluent Bit — Great for logs, limited for metrics/traces; OTel Collector covers all three signals.
• Vector — Very fast, Rust-based, excellent for logs; OTel wins on trace/metrics ecosystem breadth.
• Telegraf — Metric-centric with Influx lineage; lacks first-class trace support.
• Jaeger Agent — Legacy trace-only forwarder; the Jaeger project itself now recommends the Collector.
• Vendor agents (Datadog, New Relic) — Single-vendor; the Collector keeps your pipeline portable.

FAQ

Q: Agent or gateway mode? A: Usually both. Agents run on every node (low latency, cheap fan-in), gateway pods centralize egress, auth, and sampling.

Q: Will it replace Prometheus? A: Not the storage. It can scrape Prometheus and remote-write elsewhere, but you still need a TSDB for querying.

Q: How do I keep the image small? A: Use ocb to build a custom distro with only the handful of components your pipeline actually uses.

Q: Does it support profiling signals? A: Yes — the profiles signal is in development behind feature flags and is converging with OpenTelemetry SDKs.

Sources

• https://github.com/open-telemetry/opentelemetry-collector
• https://opentelemetry.io/docs/collector/