# OpenTelemetry Collector — Vendor-Neutral Telemetry Pipeline

> The OpenTelemetry Collector is the CNCF-graduated pipeline for receiving, processing, and exporting metrics, logs, and traces across any observability backend, replacing per-vendor agents with one portable binary.

## Install

Save in your project root:

# OpenTelemetry Collector — Vendor-Neutral Telemetry Pipeline

## Quick Use
```bash
# Run the contrib distribution with a minimal config
cat > otel.yaml <<''EOF''
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  batch: {}
exporters:
  debug: { verbosity: basic }
  otlphttp:
    endpoint: https://api.honeycomb.io
    headers: { x-honeycomb-team: "${HONEYCOMB_API_KEY}" }
service:
  pipelines:
    traces:  { receivers: [otlp], processors: [batch], exporters: [debug, otlphttp] }
    metrics: { receivers: [otlp], processors: [batch], exporters: [debug, otlphttp] }
EOF

docker run --rm -p 4317:4317 -p 4318:4318 
  -v "$PWD/otel.yaml:/etc/otel/config.yaml" 
  otel/opentelemetry-collector-contrib:latest 
  --config=/etc/otel/config.yaml
```

## Introduction
OpenTelemetry Collector is the swiss-army-knife agent of the OpenTelemetry project. Instead of running a Datadog agent, a New Relic agent, a Prometheus node exporter, and a Fluent Bit side-by-side, teams deploy one Collector and configure receivers, processors, and exporters per signal type. It graduated to CNCF-incubating then to the stable trace and metrics pipelines, and is the recommended collection layer in almost every modern observability stack.

## What the Collector Does
- Receives telemetry in OTLP, Jaeger, Zipkin, Prometheus, statsd, Fluent Forward, Kafka, and many more.
- Processes data through batching, attribute editing, sampling, redaction, and resource enrichment.
- Exports to vendors (Datadog, Honeycomb, Splunk, New Relic) and OSS stores (Tempo, Jaeger, Prometheus, Loki).
- Acts as a sidecar (agent mode) for low-latency collection or as a standalone gateway for fan-in and routing.
- Generates internal metrics and zpages so you can observe your observability pipeline.

## Architecture Overview
The Collector is a Go binary with a component-based plugin model: every receiver, processor, exporter, extension, and connector is a compiled-in module. `opentelemetry-collector` is the minimal core; `opentelemetry-collector-contrib` adds the full catalog. Pipelines are directed graphs declared in YAML — a receiver hands data to a sequence of processors, which fan out to exporters. The OpenTelemetry Collector Builder (`ocb`) lets you build a custom distribution with only the components you need, shrinking image size and attack surface.

## Self-Hosting & Configuration
- Pick `otelcol` (core) for stability or `otelcol-contrib` for breadth; build your own distro with `ocb`.
- Deploy agents as a DaemonSet on Kubernetes, gateway pods as a Deployment behind a Service.
- Use memory_limiter, batch, and retry_on_failure processors in every production pipeline.
- Secure OTLP endpoints with mTLS and enforce auth via the bearertokenauth or oidc extensions.
- Enable `zpages` on a sidecar port for live pipeline inspection without scraping metrics.

## Key Features
- Single binary for metrics, logs, and traces — no language-specific agent sprawl.
- Hot-reloadable YAML config; validation via `--dry-run`.
- Tail-sampling and probabilistic-sampling processors for cost control.
- Transform processor with OTTL (OpenTelemetry Transformation Language) for surgical edits.
- First-class Prometheus compatibility: scrape classic `/metrics` endpoints and remote-write to any TSDB.

## Comparison with Similar Tools
- **Fluent Bit** — Great for logs, limited for metrics/traces; OTel Collector covers all three signals.
- **Vector** — Very fast, Rust-based, excellent for logs; OTel wins on trace/metrics ecosystem breadth.
- **Telegraf** — Metric-centric with Influx lineage; lacks first-class trace support.
- **Jaeger Agent** — Legacy trace-only forwarder; the Jaeger project itself now recommends the Collector.
- **Vendor agents (Datadog, New Relic)** — Single-vendor; the Collector keeps your pipeline portable.

## FAQ
**Q:** Agent or gateway mode?
A: Usually both. Agents run on every node (low latency, cheap fan-in), gateway pods centralize egress, auth, and sampling.

**Q:** Will it replace Prometheus?
A: Not the storage. It can scrape Prometheus and remote-write elsewhere, but you still need a TSDB for querying.

**Q:** How do I keep the image small?
A: Use `ocb` to build a custom distro with only the handful of components your pipeline actually uses.

**Q:** Does it support profiling signals?
A: Yes — the profiles signal is in development behind feature flags and is converging with OpenTelemetry SDKs.

## Sources
- https://github.com/open-telemetry/opentelemetry-collector
- https://opentelemetry.io/docs/collector/


---
Source: https://tokrepo.com/en/workflows/1e161adc-3929-11f1-9bc6-00163e2b0d79
Author: AI Open Source