OpenTelemetry Collector — Unified Telemetry Data Pipeline

Introduction

The OpenTelemetry Collector is a vendor-neutral proxy that sits between your applications and observability backends. It receives telemetry in any format (OTLP, Jaeger, Prometheus, Zipkin), applies processing like filtering and batching, and exports to one or many destinations — eliminating vendor lock-in and simplifying instrumentation.

What OpenTelemetry Collector Does

• Receives traces, metrics, and logs via OTLP, Jaeger, Prometheus, Zipkin, and 80+ receivers
• Processes telemetry with batching, sampling, attribute manipulation, and tail-based sampling
• Exports data to any backend including Grafana, Datadog, New Relic, Jaeger, and cloud providers
• Runs as an agent (sidecar/DaemonSet) or gateway (centralized) deployment
• Provides built-in health checks, zpages, and Prometheus self-monitoring

Architecture Overview

The Collector follows a pipeline model with three stages: receivers ingest data, processors transform it, and exporters send it to backends. Multiple pipelines can run in parallel for traces, metrics, and logs. Extensions add capabilities like health checking, bearer token auth, and zPages debugging. The contrib distribution bundles 200+ community components beyond the core.

Self-Hosting & Configuration

• Define pipelines in YAML specifying receivers, processors, and exporters per signal type
• Deploy as a Kubernetes DaemonSet for node-level collection or a Deployment for gateway mode
• Use the Helm chart (open-telemetry/opentelemetry-helm-charts) for production Kubernetes installs
• Scale horizontally behind a load balancer using the loadbalancing exporter for trace-aware routing
• Secure with mTLS between collectors and backends using the configtls settings

Key Features

• Single binary handles traces, metrics, and logs without running separate agents
• 200+ components in the contrib distribution covering virtually every observability vendor
• Sub-millisecond per-span processing overhead in production benchmarks
• Built-in tail-based sampling reduces storage costs while preserving interesting traces
• OpenTelemetry Operator for Kubernetes automates sidecar injection and collector management

Comparison with Similar Tools

• Fluentd/Fluent Bit — log-focused; OTel Collector handles traces, metrics, and logs in one agent
• Grafana Alloy — Grafana's all-in-one agent; OTel Collector is vendor-neutral with broader ecosystem
• Datadog Agent — vendor-specific; OTel Collector exports to any backend
• Vector — high-performance log/metrics pipeline; OTel Collector adds native distributed tracing
• Jaeger Agent — trace-only; OTel Collector is the recommended replacement

FAQ

Q: Should I use the core or contrib distribution? A: Start with contrib for maximum compatibility. Switch to a custom build with the OCB builder to include only what you need in production.

Q: Can it replace Prometheus scraping? A: Yes. The Prometheus receiver scrapes targets and the Collector can export to any metrics backend.

Q: What is the performance overhead? A: Minimal. The Collector processes millions of spans per second on modest hardware with sub-millisecond latency per span.

Q: How does it differ from OpenTelemetry SDKs? A: SDKs instrument your application code. The Collector is an infrastructure component that receives, processes, and routes the telemetry SDKs produce.

Sources

• https://github.com/open-telemetry/opentelemetry-collector
• https://opentelemetry.io/docs/collector/