Grafana Loki — Prometheus-Inspired Log Aggregation System

What Loki Does

• Log Aggregation: Collect and store logs from all your services in one place
• LogQL: Prometheus-inspired query language for searching and aggregating logs
• Label-Based Indexing: Index only metadata (labels), not log content — 10x cheaper storage
• Grafana Integration: Native integration with Grafana for visualization
• Multi-Tenancy: Separate logs per tenant/team/environment
• Horizontal Scaling: Scale read/write paths independently
• Cloud Native: Designed for Kubernetes and cloud environments
• Compression: Gzip/LZ4/Snappy log compression
• Retention: Configurable retention periods per label stream

Architecture

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Applications│────▶│  Promtail /  │────▶│  Loki        │
│  Containers  │     │  Fluent Bit /│     │  Distributor │
│  Systemd     │     │  Vector /    │     │  Ingester    │
└──────────────┘     │  OTel        │     │  Querier     │
                     └──────────────┘     └──────┬───────┘
                                                 │
                                          ┌──────┴───────┐
                                          │  Object      │
                                          │  Storage     │
                                          │  (S3/GCS/    │
                                          │  MinIO/local)│
                                          └──────────────┘

Self-Hosting

Docker Compose (Simple Setup)

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - loki-data:/loki

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin

volumes:
  loki-data:

Promtail Config

# promtail-config.yml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          host: myserver
          __path__: /var/log/*log

  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        target_label: container

LogQL (Query Language)

Basic Queries

# All logs from nginx
{container="nginx"}

# Logs containing "error" (case insensitive)
{container="nginx"} |~ "(?i)error"

# JSON logs with specific field
{job="api"} | json | level="error"

# Exclude healthchecks
{container="nginx"} != "/health"

# Multiple filters
{namespace="production", app="web"} |= "500" != "healthcheck"

Metric Queries

# Count errors per minute
count_over_time({container="api"} |= "ERROR" [1m])

# Rate of requests
rate({container="nginx"}[5m])

# Error rate percentage
sum(rate({app="api"} |= "ERROR" [5m]))
  / sum(rate({app="api"}[5m])) * 100

# Top 10 hosts by log volume
topk(10, sum(rate({job="varlogs"}[5m])) by (host))

Structured Logs

# Parse JSON and filter
{app="api"}
  | json
  | status >= 500
  | duration > 1000

# Extract labels from logs
{app="web"}
  | regexp `(?P<method>w+) (?P<path>/S+)`
  | method="POST"

Key Features

Cost Efficiency

ElasticSearch indexes every word in logs:
  → 100GB logs → 200-400GB storage
  → High CPU for indexing
  → Expensive RAM requirements

Loki indexes only labels:
  → 100GB logs → 50-100GB storage (compressed)
  → Low CPU for indexing
  → Minimal RAM (only for query time)

Label-Based Sharding

Log stream = unique combination of labels
{namespace="prod", app="api", pod="api-abc123"}
{namespace="prod", app="web", pod="web-xyz789"}

Labels become index keys
Log content is only scanned during queries

Integration with Metrics

Grafana Dashboard:
├── CPU Usage (Prometheus metric)
├── Error Rate (LogQL count_over_time)
├── Recent Errors (Loki logs)
└── Link errors to trace in Tempo

Loki vs Alternatives

FAQ

Q: Loki or ElasticSearch — which should I pick? A: If you mainly want to filter logs by time range and labels (container, namespace, pod), Loki is cheaper and more efficient. If you need complex full-text search and analysis over log contents, ElasticSearch is more powerful.

Q: Why does it only index labels? A: It's Loki's core design. Most log queries are "give me the logs for service X during time window Y" — label indexing covers that. Content matching is handled by grep-style filtering at query time. The result is 10×+ lower storage costs.

Q: What scale is it suitable for? A: Everything from single-machine logs (a few GB/day) to large production clusters (several TB/day). Single-instance deployments handle small-to-medium scale; distributed deployments scale linearly up to the PB range.

Sources & Credits

• GitHub: grafana/loki — 28K+ ⭐ | AGPL-3.0
• Website: grafana.com/loki