Fluent Bit — Lightweight High-Performance Log and Metrics Processor

What it is

Fluent Bit is a fast, lightweight telemetry agent from the Fluentd ecosystem. It collects logs, metrics, and traces from any source, processes them with built-in filters and parsers, and routes the output to multiple destinations (Elasticsearch, Splunk, Datadog, S3, Kafka, and 80+ others). Written in C with a small memory footprint, it runs efficiently on embedded systems, containers, and cloud infrastructure.

SRE teams building observability pipelines, DevOps engineers shipping logs from Kubernetes clusters, and platform teams needing a lightweight data collector use Fluent Bit as their telemetry agent. It is a CNCF graduated project.

How it saves time or tokens

Traditional log shipping requires running heavy agents (Logstash, Fluentd) on every node, consuming significant CPU and memory. Fluent Bit achieves the same functionality with a ~450KB binary and minimal resource usage. Its plugin-based architecture lets you add inputs, filters, and outputs through configuration rather than custom code. Hot-reloading configuration changes avoids downtime during pipeline updates.

How to use

1. Run with Docker:

docker run --rm -it fluent/fluent-bit:latest \
  /fluent-bit/bin/fluent-bit \
  -i cpu -o stdout -f 1

1. Configure a pipeline with a configuration file:

[INPUT]
    Name   tail
    Path   /var/log/syslog
    Tag    syslog

[FILTER]
    Name   grep
    Match  syslog
    Regex  log error

[OUTPUT]
    Name   es
    Match  *
    Host   elasticsearch.example.com
    Port   9200
    Index  logs

1. Deploy to Kubernetes as a DaemonSet to collect logs from every node.

Example

# Kubernetes DaemonSet for Fluent Bit
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:latest
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: config
              mountPath: /fluent-bit/etc/
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: config
          configMap:
            name: fluent-bit-config

Related on TokRepo

• Monitoring Tools -- explore monitoring and observability tools
• DevOps Tools -- discover DevOps infrastructure tools

Common pitfalls

• Backpressure handling is critical. If the output destination is slow, Fluent Bit buffers data in memory. Configure storage.total_limit_size to prevent unbounded memory growth.
• Log parsing consumes CPU. Use the parser filter only when structured log extraction is needed. For simple forwarding, skip parsing to reduce overhead.
• The tail input plugin tracks file positions in a database. If you delete the database file, Fluent Bit re-reads log files from the beginning, causing duplicate entries.