Filebeat — Lightweight Log Shipper from the Elastic Stack

Introduction

Filebeat is a lightweight, open-source log shipper maintained by Elastic as part of the Beats family. It tails log files, collects container logs, and reads cloud audit trails, then delivers structured events to Elasticsearch, Logstash, Kafka, or Redis with minimal resource overhead.

What Filebeat Does

• Tails log files and tracks read offsets for reliable at-least-once delivery
• Ships container logs from Docker and Kubernetes via autodiscovery
• Provides 60-plus pre-built modules for Nginx, Apache, MySQL, AWS, and more
• Applies multiline joining, JSON parsing, and field enrichment before output
• Integrates with Elastic Agent for unified fleet management

Architecture Overview

Filebeat runs one or more harvesters, each responsible for reading a single file and sending events to a central spooler. The spooler batches events and forwards them to the configured output (Elasticsearch, Logstash, Kafka). A registry file persists read offsets so Filebeat resumes from the correct position after restarts. Back-pressure handling pauses harvesting when the output is saturated.

Self-Hosting and Configuration

• Install via package managers (apt, yum), Docker images, or standalone archives
• Edit filebeat.yml to define inputs, modules, and output destinations
• Enable modules with filebeat modules enable <name> for pre-configured dashboards
• Use Autodiscover with Docker or Kubernetes for dynamic container log collection
• Deploy as a DaemonSet in Kubernetes for per-node log collection

Key Features

• Single-binary Go application with a small memory footprint (typically 30-50 MB)
• Built-in modules provide ready-made parsing, dashboards, and ML jobs
• Supports multiple outputs simultaneously via Logstash fan-out
• Processors pipeline allows filtering, enrichment, and conditional routing
• Internal queue with configurable memory or disk-backed buffering

Comparison with Similar Tools

• Fluentd — more flexible with a plugin ecosystem but heavier on resources; Filebeat is lighter for pure log shipping
• Fluent Bit — similarly lightweight with broader input/output options but less integrated with Elastic dashboards
• Promtail — Loki-focused log agent; Filebeat targets Elasticsearch and the broader Elastic ecosystem
• Vector — high-performance Rust-based pipeline; Filebeat has tighter Elastic integration and pre-built modules

FAQ

Q: Does Filebeat guarantee no log loss? A: Filebeat provides at-least-once delivery by persisting file offsets in a registry. Duplicate events are possible on unclean shutdowns.

Q: Can I use Filebeat without Elasticsearch? A: Yes. Filebeat can output to Logstash, Kafka, Redis, or any compatible endpoint.

Q: How does Filebeat handle log rotation? A: It detects renamed or truncated files and opens new harvesters as needed, continuing to track offsets by file identity (inode).

Q: What is the difference between Filebeat and Elastic Agent? A: Elastic Agent is a unified agent that can run Filebeat, Metricbeat, and other Beats internally, managed via Fleet. Standalone Filebeat is simpler for single-purpose log collection.

Sources

• https://github.com/elastic/beats
• https://www.elastic.co/beats/filebeat