# Elastic Beats — Lightweight Data Shippers for Observability

> Elastic Beats is a family of lightweight, single-purpose agents that collect logs, metrics, network data, and audit events from servers and containers and ship them to Elasticsearch or Logstash.

## Install

Save as a script file and run:

# Elastic Beats — Lightweight Data Shippers for Observability

## Quick Use
```bash
# Install Filebeat on Debian/Ubuntu
sudo apt-get install filebeat

# Enable a module (e.g., nginx)
sudo filebeat modules enable nginx

# Set up dashboards and index templates
sudo filebeat setup

# Start the service
sudo systemctl start filebeat
```

## Introduction
Beats are purpose-built Go binaries from Elastic that sit on your servers and ship operational data to the Elastic Stack. Each Beat focuses on one data type—Filebeat for logs, Metricbeat for metrics, Packetbeat for network traffic—keeping resource usage minimal.

## What Elastic Beats Does
- Filebeat: tails log files, handles multiline, and tracks reading position
- Metricbeat: collects system and service metrics from modules (Docker, Kubernetes, MySQL)
- Packetbeat: captures network protocols (HTTP, DNS, MySQL, TLS) as structured events
- Auditbeat: collects Linux audit framework data and file integrity events
- Heartbeat: monitors service uptime via ICMP, TCP, and HTTP checks

## Architecture Overview
Each Beat is built on libbeat, a shared Go framework that provides configuration, output management, and a publisher pipeline. Inputs collect raw data and create events. Processors enrich or filter events inline. The output stage sends events to Elasticsearch, Logstash, Kafka, or Redis. A registry file tracks input state for at-least-once delivery.

## Self-Hosting & Configuration
- Each Beat has its own YAML config file (e.g., `filebeat.yml`)
- Configure `output.elasticsearch` or `output.logstash` for destination
- Enable built-in modules with `<beat> modules enable <module>` for pre-configured collection
- Use autodiscover with Docker or Kubernetes labels for dynamic input configuration
- Deploy as DaemonSets in Kubernetes using Elastic's official Helm charts

## Key Features
- Module system: pre-packaged configs, parsers, and dashboards for 60+ technologies
- Autodiscover: automatically detect and monitor new containers and pods
- Backpressure handling: in-memory and disk-based spooling prevent data loss
- Processors: add fields, drop events, decode JSON, resolve DNS inline
- Elastic Common Schema (ECS) compliance for consistent field naming across Beats

## Comparison with Similar Tools
- **Fluent Bit** — single binary for logs and metrics; Beats offer purpose-built agents per data type
- **Telegraf** — push-based with 300+ plugins; Beats integrate tighter with Elastic Stack dashboards
- **Grafana Alloy** — OpenTelemetry-native; Beats predate OTel and have deeper Elastic modules
- **rsyslog** — traditional syslog daemon; Beats add structured parsing and direct ES output

## FAQ
**Q: Can I use Beats without Elasticsearch?**
A: Yes. Beats can output to Logstash, Kafka, Redis, or files. However, module dashboards require Kibana and Elasticsearch.

**Q: How much memory does Filebeat use?**
A: Typically 30-80 MB RSS depending on the number of harvesters and processing pipeline complexity.

**Q: What is the difference between Beats and Elastic Agent?**
A: Elastic Agent is a unified agent that bundles multiple Beats under a single binary managed by Fleet. Standalone Beats are still supported.

**Q: Can I write a custom Beat?**
A: Yes. The libbeat framework and Beat generator scaffold let you create custom Beats in Go.

## Sources
- https://github.com/elastic/beats
- https://www.elastic.co/beats

---
Source: https://tokrepo.com/en/workflows/0eadbc4f-3f31-11f1-9bc6-00163e2b0d79
Author: Script Depot