Prometheus — Open Source Monitoring & Alerting Toolkit

What Prometheus Does

• Metrics Collection: Pull-based metrics scraping from instrumented applications and exporters
• Time Series DB: Efficient local storage optimized for time series data with compression
• PromQL: Powerful query language for slicing, dicing, and aggregating time series data
• Alerting: Alert rules with Alertmanager for routing, grouping, and notification
• Service Discovery: Auto-discover targets from Kubernetes, Consul, DNS, EC2, and more
• Exporters: 500+ exporters for databases, hardware, messaging, storage, and cloud services
• Federation: Hierarchical federation for scaling across multiple Prometheus instances

Architecture

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│ Targets      │◀────│  Prometheus  │────▶│  Alertmanager│
│ (Exporters)  │pull │  Server      │     │  (Notify)    │
│              │     │  TSDB + Rules│     └──────────────┘
└──────────────┘     └──────┬───────┘
                            │
                     ┌──────┴───────┐
                     │  Grafana     │
                     │  (Visualize) │
                     └──────────────┘

Key design principle: Pull-based — Prometheus scrapes metrics from HTTP endpoints, rather than having applications push metrics. This makes it easier to detect when a target is down.

Self-Hosting

Docker Compose (Full Stack)

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert.rules.yml:/etc/prometheus/alert.rules.yml
      - prometheus-data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=30d"

  alertmanager:
    image: prom/alertmanager:latest
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro

volumes:
  prometheus-data:

Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - "alert.rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

PromQL Essentials

# Instant vector — current value
up{job="node"}

# Range vector — values over time
node_cpu_seconds_total[5m]

# Rate — per-second rate of increase
rate(http_requests_total[5m])

# Aggregation — sum across instances
sum(rate(http_requests_total[5m])) by (method, status)

# Histogram quantile — P95 latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Arithmetic — error percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m])) * 100

# Prediction — disk full in 4 hours?
predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600) < 0

Instrumenting Your App

Go

import "github.com/prometheus/client_golang/prometheus"

var httpRequests = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests",
    },
    []string{"method", "status"},
)

func init() { prometheus.MustRegister(httpRequests) }

// In handler:
httpRequests.WithLabelValues("GET", "200").Inc()

Python

from prometheus_client import Counter, start_http_server

REQUEST_COUNT = Counter('http_requests_total', 'Total requests', ['method', 'status'])

REQUEST_COUNT.labels(method='GET', status='200').inc()
start_http_server(8000)  # Expose /metrics on port 8000

Popular Exporters

Prometheus vs Alternatives

FAQ

Q: How long can Prometheus retain data? A: 15 days by default. Adjust via --storage.tsdb.retention.time. For long-term storage, use remote storage solutions like Thanos or Cortex.

Q: Is Prometheus good for log collection? A: No. Prometheus is specifically for numeric metrics. For logs, use Loki (also from Grafana Labs) — it pairs perfectly with Prometheus.

Q: How many metrics can a single Prometheus instance handle? A: A single instance handles millions of active time series. For very large scale, use federation or Thanos/Mimir for horizontal scaling.