Fluentd — Unified Logging Layer for Cloud-Native Infrastructure

Introduction

Fluentd, born at Treasure Data in 2011, pioneered the idea of a "unified logging layer": a single daemon that collects logs from anywhere, parses them into structured JSON, and routes them everywhere. Its plugin ecosystem (1000+) is unmatched.

Graduated from the CNCF in 2019, Fluentd is used by Red Hat OpenShift, IBM, Nintendo, and thousands of Kubernetes operators. It pairs with Fluent Bit (lightweight agent) for the canonical "Fluent Bit on every node, Fluentd as the aggregator" pattern.

What Fluentd Does

Fluentd has three core concepts: inputs (plugins that collect data from sources), filters (transform/enrich records), and outputs (send to destinations). Configuration is XML-like directives. Buffers persist events during destination outages. Tags route events through the pipeline.

Architecture Overview

Inputs                 Outputs
  tail (files)           elasticsearch
  forward (TCP)          s3
  http                   kafka
  syslog                 bigquery
  kubernetes             loki
  systemd                splunk
  dstat, tcp, exec,      mongodb, mysql,
  windows_eventlog, ...  http, file, stdout, ...
      |                      ^
      v                      |
    [Tags + Events]           
       |                      
    [Filters]                 
      record_transformer,     
      parser, grep, geoip     
       |                      
    [Buffer]                  
      memory or file          
      flush_interval, retry   

Self-Hosting & Configuration

# Kubernetes DaemonSet aggregator pattern
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /fluentd/log/fluentd-containers.log.pos
  tag kubernetes.*
  <parse>
    @type json
    time_key time
  </parse>
</source>

<filter kubernetes.**>
  @type kubernetes_metadata
  kubernetes_url https://kubernetes.default
  verify_ssl true
</filter>

<filter kubernetes.**>
  @type grep
  <regexp>
    key $.kubernetes.container_name
    pattern /^(?!healthcheck|sidecar).*$/
  </regexp>
</filter>

<match kubernetes.**>
  @type elasticsearch_dynamic
  host ${ENV["ES_HOST"]}
  port 9200
  logstash_prefix ${record["kubernetes"]["namespace_name"]}
  <buffer>
    @type file
    path /fluentd/buffer/es
    chunk_limit_size 8MB
    flush_interval 5s
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

Key Features

• 1000+ plugins — any source, any destination via the plugin ecosystem
• Structured JSON events — unified format across log pipelines
• Kubernetes metadata — auto-enrich with pod, namespace, container labels
• Buffer + retry — disk/memory buffers survive destination outages
• Tag-based routing — flexible match/filter syntax by tag patterns
• CNCF graduated — stable governance, vendor-neutral
• Fluent Bit pairing — lightweight agent forwards to Fluentd aggregator
• Language support — Ruby-based plugins, C extensions for hot paths

Comparison with Similar Tools

FAQ

Q: Fluentd vs Fluent Bit — use both? A: Yes, canonical pattern: Fluent Bit as lightweight node agent collecting logs (minimal CPU/memory), Fluentd as aggregator with richer transforms and buffer features. Both from the Fluent project.

Q: Why pick Fluentd over Vector in 2026? A: Ecosystem breadth (1000+ plugins), stability (CNCF graduated), and Kubernetes integrations. Vector has better performance but smaller plugin set. Many teams keep Fluentd for legacy pipelines and adopt Vector for new ones.

Q: How do I write custom plugins? A: Fluentd plugins are Ruby gems. Subclass Fluent::Plugin::Input/Filter/Output and publish to RubyGems. Most needs are already covered by existing plugins.

Q: Is Fluentd in decline? A: Still actively developed and widely used, but some teams migrate to Vector or OTel Collector. Fluentd remains the standard for mature Kubernetes logging stacks.

Sources

• GitHub: https://github.com/fluent/fluentd
• Docs: https://www.fluentd.org
• Foundation: CNCF (Graduated)
• License: Apache-2.0