Vector — High-Performance Observability Data Pipeline

Introduction

Vector is the modern observability data pipeline, built by Timber (acquired by Datadog) and now the most performant log/metric router in the space. It replaces Logstash, Fluentd, and Filebeat with a single Rust binary that's simpler to configure, uses a fraction of the memory, and handles 10–100x more events per core.

With over 21,000 GitHub stars, Vector is used by Datadog (internally), Fastly, Discord, and companies ingesting petabytes of logs. It sits between data sources (servers, apps) and destinations (Loki, ES, S3, Kafka, Datadog, any SaaS).

What Vector Does

Vector runs as an agent or aggregator. Its config has three parts: sources (where data comes from), transforms (parse, enrich, filter using VRL — Vector Remap Language), and sinks (where data goes). It supports 40+ sources, 20+ transforms, and 40+ sinks — plus hot config reloads, back-pressure, and disk buffering.

Architecture Overview

Sources:                        Sinks:
  files, journald, docker,        Loki, Elasticsearch,
  kubernetes logs, syslog,        ClickHouse, Kafka, S3,
  HTTP, AWS kinesis, kafka,       Datadog, Honeycomb,
  prometheus scrape, OTLP, ...    Splunk, Axiom, any OTLP
      \                              /
       \                            /
        V                          V
                 [Vector]
                    |
               [Transforms]
                 remap (VRL),
                 filter, route,
                 aggregate, sample,
                 throttle, dedupe
                    |
            [Buffer / Back-pressure]
                 in-memory or disk
                    |
            [Multi-sink fan-out]

Self-Hosting & Configuration

# Multi-environment log routing
[sources.docker]
type = "docker_logs"

[transforms.enrich]
type = "remap"
inputs = ["docker"]
source = """
  .service = .container_name
  .level = if exists(.level) { .level } else { "info" }
  if contains(string!(.message), "ERROR") { .level = "error" }
"""

[transforms.route]
type = "route"
inputs = ["enrich"]
route.errors = '.level == "error"'
route.rest   = "true"

[sinks.pagerduty]
type = "http"
inputs = ["route.errors"]
uri  = "https://events.pagerduty.com/v2/enqueue"
encoding.codec = "json"

[sinks.s3_archive]
type = "aws_s3"
inputs = ["route.rest"]
bucket = "logs-archive"
key_prefix = "{{ service }}/{{ %Y-%m-%d }}/"
compression = "gzip"
encoding.codec = "json"
buffer.type = "disk"
buffer.max_size = 1073741824

Key Features

• Rust performance — tiny memory footprint, massive throughput
• VRL (Vector Remap Language) — simple, fast, type-safe transforms
• 100+ components — sources, transforms, sinks for every backend
• Hot reload — update config without losing buffered events
• Disk buffers — survive destination outages without data loss
• Kubernetes-native — DaemonSet install, auto-discovers pod logs
• OpenTelemetry — OTLP source and sink, interop with OTel Collector
• Observability — built-in metrics for your pipeline itself

Comparison with Similar Tools

FAQ

Q: Vector vs OpenTelemetry Collector? A: Vector is faster and has richer sources/sinks; OTel Collector is the standard for traces/metrics/logs in OTLP. Many teams run both: Vector for legacy log shipping, OTel Collector for traces.

Q: What is VRL? A: Vector Remap Language — a tiny, strongly typed DSL for transforming events. Faster and safer than Lua/Ruby alternatives, with excellent error messages.

Q: Can Vector replace Fluent Bit as a node agent? A: Yes. Vector's agent mode is designed for DaemonSet use with pod-log collection, Kubernetes metadata enrichment, and local buffering.

Q: Is Vector truly open source? A: Yes, MPL-2.0 licensed. Datadog sponsors development but Vector works with any backend, not just Datadog.

Sources

• GitHub: https://github.com/vectordotdev/vector
• Docs: https://vector.dev
• Company: Datadog (ex-Timber)
• License: MPL-2.0