# Fluentd — Unified Logging Layer for Cloud-Native Infrastructure

> Fluentd is a CNCF-graduated open-source data collector that unifies log collection and routing. With 1000+ plugins, it connects any source to any destination — the standard log layer for Kubernetes alongside Fluent Bit.

## Install

Save as a script file and run:

# Fluentd — Unified Logging Layer for Cloud Native

## Quick Use
```bash
# Run Fluentd in Docker
docker run -d -p 24224:24224 -v $(pwd)/conf:/fluentd/etc fluent/fluentd:v1.17
```

```xml
# fluentd.conf — collect JSON over TCP and ship to Elasticsearch
<source>
  @type forward
  port 24224
</source>

<filter app.**>
  @type record_transformer
  <record>
    host "#{Socket.gethostname}"
    env  "prod"
  </record>
</filter>

<match **>
  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true
  logstash_prefix app-logs
  <buffer>
    @type file
    path /fluentd/buffer/es
    flush_interval 5s
  </buffer>
</match>
```

## Introduction
Fluentd, born at Treasure Data in 2011, pioneered the idea of a "unified logging layer": a single daemon that collects logs from anywhere, parses them into structured JSON, and routes them everywhere. Its plugin ecosystem (1000+) is unmatched.

Graduated from the CNCF in 2019, Fluentd is used by Red Hat OpenShift, IBM, Nintendo, and thousands of Kubernetes operators. It pairs with Fluent Bit (lightweight agent) for the canonical "Fluent Bit on every node, Fluentd as the aggregator" pattern.

## What Fluentd Does
Fluentd has three core concepts: **inputs** (plugins that collect data from sources), **filters** (transform/enrich records), and **outputs** (send to destinations). Configuration is XML-like directives. Buffers persist events during destination outages. Tags route events through the pipeline.

## Architecture Overview
```
Inputs                 Outputs
  tail (files)           elasticsearch
  forward (TCP)          s3
  http                   kafka
  syslog                 bigquery
  kubernetes             loki
  systemd                splunk
  dstat, tcp, exec,      mongodb, mysql,
  windows_eventlog, ...  http, file, stdout, ...
      |                      ^
      v                      |
    [Tags + Events]           
       |                      
    [Filters]                 
      record_transformer,     
      parser, grep, geoip     
       |                      
    [Buffer]                  
      memory or file          
      flush_interval, retry   
```

## Self-Hosting & Configuration
```xml
# Kubernetes DaemonSet aggregator pattern
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /fluentd/log/fluentd-containers.log.pos
  tag kubernetes.*
  <parse>
    @type json
    time_key time
  </parse>
</source>

<filter kubernetes.**>
  @type kubernetes_metadata
  kubernetes_url https://kubernetes.default
  verify_ssl true
</filter>

<filter kubernetes.**>
  @type grep
  <regexp>
    key $.kubernetes.container_name
    pattern /^(?!healthcheck|sidecar).*$/
  </regexp>
</filter>

<match kubernetes.**>
  @type elasticsearch_dynamic
  host ${ENV["ES_HOST"]}
  port 9200
  logstash_prefix ${record["kubernetes"]["namespace_name"]}
  <buffer>
    @type file
    path /fluentd/buffer/es
    chunk_limit_size 8MB
    flush_interval 5s
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>
```

## Key Features
- **1000+ plugins** — any source, any destination via the plugin ecosystem
- **Structured JSON events** — unified format across log pipelines
- **Kubernetes metadata** — auto-enrich with pod, namespace, container labels
- **Buffer + retry** — disk/memory buffers survive destination outages
- **Tag-based routing** — flexible match/filter syntax by tag patterns
- **CNCF graduated** — stable governance, vendor-neutral
- **Fluent Bit pairing** — lightweight agent forwards to Fluentd aggregator
- **Language support** — Ruby-based plugins, C extensions for hot paths

## Comparison with Similar Tools
| Feature | Fluentd | Fluent Bit | Vector | Logstash | OTel Collector |
|---|---|---|---|---|---|
| Language | Ruby (C core) | C | Rust | JRuby (JVM) | Go |
| Footprint | Moderate | Very Low | Very Low | High | Low |
| Plugin count | 1000+ | Growing | 100+ | 200+ | Growing |
| Config | Directive XML-like | Directive | TOML/YAML + VRL | Ruby DSL | YAML |
| Kubernetes | Strong | Very strong | Strong | Moderate | Strong |
| Best For | Aggregator role | Node agent | Modern greenfield | Elastic-centric | Traces + modern |

## FAQ
**Q: Fluentd vs Fluent Bit — use both?**
A: Yes, canonical pattern: Fluent Bit as lightweight node agent collecting logs (minimal CPU/memory), Fluentd as aggregator with richer transforms and buffer features. Both from the Fluent project.

**Q: Why pick Fluentd over Vector in 2026?**
A: Ecosystem breadth (1000+ plugins), stability (CNCF graduated), and Kubernetes integrations. Vector has better performance but smaller plugin set. Many teams keep Fluentd for legacy pipelines and adopt Vector for new ones.

**Q: How do I write custom plugins?**
A: Fluentd plugins are Ruby gems. Subclass Fluent::Plugin::Input/Filter/Output and publish to RubyGems. Most needs are already covered by existing plugins.

**Q: Is Fluentd in decline?**
A: Still actively developed and widely used, but some teams migrate to Vector or OTel Collector. Fluentd remains the standard for mature Kubernetes logging stacks.

## Sources
- GitHub: https://github.com/fluent/fluentd
- Docs: https://www.fluentd.org
- Foundation: CNCF (Graduated)
- License: Apache-2.0

---
Source: https://tokrepo.com/en/workflows/63e017ab-37c8-11f1-9bc6-00163e2b0d79
Author: Script Depot