Elastic Beats — Lightweight Data Shippers for Observability

Introduction

Beats are purpose-built Go binaries from Elastic that sit on your servers and ship operational data to the Elastic Stack. Each Beat focuses on one data type—Filebeat for logs, Metricbeat for metrics, Packetbeat for network traffic—keeping resource usage minimal.

What Elastic Beats Does

• Filebeat: tails log files, handles multiline, and tracks reading position
• Metricbeat: collects system and service metrics from modules (Docker, Kubernetes, MySQL)
• Packetbeat: captures network protocols (HTTP, DNS, MySQL, TLS) as structured events
• Auditbeat: collects Linux audit framework data and file integrity events
• Heartbeat: monitors service uptime via ICMP, TCP, and HTTP checks

Architecture Overview

Each Beat is built on libbeat, a shared Go framework that provides configuration, output management, and a publisher pipeline. Inputs collect raw data and create events. Processors enrich or filter events inline. The output stage sends events to Elasticsearch, Logstash, Kafka, or Redis. A registry file tracks input state for at-least-once delivery.

Self-Hosting & Configuration

• Each Beat has its own YAML config file (e.g., filebeat.yml)
• Configure output.elasticsearch or output.logstash for destination
• Enable built-in modules with <beat> modules enable <module> for pre-configured collection
• Use autodiscover with Docker or Kubernetes labels for dynamic input configuration
• Deploy as DaemonSets in Kubernetes using Elastic's official Helm charts

Key Features

• Module system: pre-packaged configs, parsers, and dashboards for 60+ technologies
• Autodiscover: automatically detect and monitor new containers and pods
• Backpressure handling: in-memory and disk-based spooling prevent data loss
• Processors: add fields, drop events, decode JSON, resolve DNS inline
• Elastic Common Schema (ECS) compliance for consistent field naming across Beats

Comparison with Similar Tools

• Fluent Bit — single binary for logs and metrics; Beats offer purpose-built agents per data type
• Telegraf — push-based with 300+ plugins; Beats integrate tighter with Elastic Stack dashboards
• Grafana Alloy — OpenTelemetry-native; Beats predate OTel and have deeper Elastic modules
• rsyslog — traditional syslog daemon; Beats add structured parsing and direct ES output

FAQ

Q: Can I use Beats without Elasticsearch? A: Yes. Beats can output to Logstash, Kafka, Redis, or files. However, module dashboards require Kibana and Elasticsearch.

Q: How much memory does Filebeat use? A: Typically 30-80 MB RSS depending on the number of harvesters and processing pipeline complexity.

Q: What is the difference between Beats and Elastic Agent? A: Elastic Agent is a unified agent that bundles multiple Beats under a single binary managed by Fleet. Standalone Beats are still supported.

Q: Can I write a custom Beat? A: Yes. The libbeat framework and Beat generator scaffold let you create custom Beats in Go.

Sources

• https://github.com/elastic/beats
• https://www.elastic.co/beats