Prometheus Alertmanager — Alert Routing and Notification Hub

Introduction

Alertmanager is the notification layer of the Prometheus ecosystem. While Prometheus evaluates alerting rules and fires alerts, Alertmanager receives them, groups related alerts together, suppresses duplicates, and dispatches notifications through the configured channels.

What Alertmanager Does

• Receives firing and resolved alerts from one or more Prometheus servers
• Groups related alerts by configurable labels to reduce notification noise
• Deduplicates identical alerts from HA Prometheus pairs
• Routes alerts to receivers based on label matchers and severity
• Supports silences and inhibition rules to suppress known or dependent alerts

Architecture Overview

Alertmanager is a single Go binary that exposes an HTTP API on port 9093. Incoming alerts enter a dispatcher that evaluates routing rules organized as a tree. Matching alerts are grouped by configured labels and held for a group_wait interval before notification. The notification pipeline sends to configured receivers. A gossip protocol (via Hashicorp Memberlist) synchronizes silences and notification state across a cluster of Alertmanager instances.

Self-Hosting & Configuration

• Config lives in alertmanager.yml with route, receivers, and inhibit_rules sections
• Define receivers for email, Slack, PagerDuty, OpsGenie, Webhook, and others
• Use group_by, group_wait, and group_interval to tune notification batching
• Run multiple instances with --cluster.peer flags for high availability
• Manage silences via the web UI at http://localhost:9093/#/silences

Key Features

• Routing tree: hierarchical label-based routing with match and match_re conditions
• Inhibition: automatically mute lower-severity alerts when a critical alert fires
• Silences: time-bound muting with label matchers, manageable via UI or API
• Clustering: HA mode with mesh gossip ensures no duplicate notifications
• Templating: customize notification messages with Go templates and alert data

Comparison with Similar Tools

• Grafana Alerting — built into Grafana, multi-datasource; Alertmanager is Prometheus-native
• PagerDuty — SaaS incident management; Alertmanager is self-hosted and free
• Opsgenie — SaaS alert routing; Alertmanager integrates as a notification target
• Robusta — Kubernetes-focused enrichment; Alertmanager is protocol-agnostic

FAQ

Q: Can I use Alertmanager without Prometheus? A: Yes. Any system that POSTs alerts in the Alertmanager API format can use it as a notification router.

Q: How do silences work? A: A silence matches alerts by label matchers and suppresses notifications for a defined time window. Silences are managed via the web UI or the v2 API.

Q: What happens if Alertmanager restarts? A: Active alerts are re-sent by Prometheus on its next evaluation cycle. Silences and notification log are persisted to disk by default.

Q: How do I test my routing configuration? A: Use amtool config routes test with sample labels or the routing tree visualizer in the web UI.

Sources

• https://github.com/prometheus/alertmanager
• https://prometheus.io/docs/alerting/latest/alertmanager/