Fluent Bit — Lightweight High-Performance Log and Metrics Processor

Introduction

Fluent Bit is the successor to Fluentd for environments where a small footprint matters most: Kubernetes DaemonSets, embedded devices and edge gateways. Written in C with a tiny runtime footprint (roughly 1 MB binary, a few MB RAM), it speaks the Fluent protocol, OpenTelemetry, Prometheus and dozens of native outputs.

What Fluent Bit Does

• Tails files, listens on TCP/UDP, scrapes Prometheus endpoints or reads from systemd
• Parses logs with built-in and user-defined parsers (JSON, regex, multiline, CRI)
• Enriches records with Kubernetes metadata, GeoIP, Lua scripts and WASM filters
• Forwards to Fluentd, Loki, Elastic, OpenSearch, Kafka, S3, Datadog, Splunk, NATS, etc.
• Exposes OpenTelemetry traces and metrics over OTLP

Architecture Overview

A Fluent Bit pipeline has four stages: Inputs → Parsers → Filters → Outputs. Records flow through an event engine backed by Chunk I/O, which uses memory and disk buffers to guarantee durability under backpressure. The single-process runtime uses an async I/O loop with minimal locking, so a single Pod can handle hundreds of thousands of events per second. Plugins live in the same process; you can also compile community plugins or write your own in Go/C.

Self-Hosting & Configuration

• Deploy as a DaemonSet on Kubernetes using the official Helm chart
• Mount /var/log, /var/lib/docker/containers read-only to tail container logs
• Use the kubernetes filter to attach Pod labels, annotations and container metadata
• Configure Mem_Buf_Limit and filesystem storage to prevent unbounded growth
• Use Hot Reload (SIGHUP) to update the config without restarts

Key Features

• Tiny footprint compared with JVM-based agents
• Native Kubernetes metadata filter and CRI parser
• 100+ plugins spanning log, metric and trace backends
• Disk-backed buffers for at-least-once delivery
• Stream processing queries over the event stream

Comparison with Similar Tools

• Fluentd — Ruby-based predecessor; heavier, richer plugin ecosystem
• Logstash — JVM-based, powerful filters, but memory-hungry
• Vector — Rust agent with similar goals; strong observability story
• Promtail — Loki-focused only; Fluent Bit ships to more backends
• rsyslog / syslog-ng — great for classic Unix logging, weaker on cloud outputs

FAQ

Q: Is Fluent Bit a replacement for Fluentd? A: For edge/agent scenarios yes. Many deployments still use Fluent Bit as a forwarder and Fluentd as an aggregator for complex pipelines.

Q: Can Fluent Bit handle metrics and traces? A: Yes, via OpenTelemetry OTLP inputs/outputs and Prometheus scrape support. Full OTel parity is ongoing.

Q: How do multiline logs work? A: Use the built-in multiline filter or Multiline.Parser, with presets for Java, Python, Go, Ruby and Kubernetes CRI logs.

Q: What about backpressure? A: Configure filesystem buffering and storage.type filesystem so events persist on disk until the output catches up.

Sources

• https://github.com/fluent/fluent-bit
• https://docs.fluentbit.io