kube-prometheus — Production-Ready Kubernetes Monitoring Stack

Introduction

kube-prometheus is an opinionated collection of Kubernetes manifests, Grafana dashboards, and Prometheus recording and alerting rules that deploys a complete monitoring stack for Kubernetes clusters. Maintained by the Prometheus Operator team, it bundles Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, and blackbox-exporter with sensible defaults for immediate production use.

What kube-prometheus Does

• Deploys a complete Prometheus-based monitoring stack with a single kubectl apply
• Includes 100+ pre-configured alerting rules for Kubernetes components and node health
• Ships with curated Grafana dashboards for cluster, node, pod, and workload monitoring
• Manages Prometheus and Alertmanager lifecycle via the Prometheus Operator CRDs
• Collects metrics from kubelets, kube-state-metrics, node-exporter, and CoreDNS automatically

Architecture Overview

kube-prometheus generates its manifests using jsonnet and the kube-prometheus jsonnet library. The Prometheus Operator watches for ServiceMonitor and PrometheusRule custom resources to configure Prometheus scrape targets and alerting rules declaratively. Two Prometheus replicas run for high availability, Alertmanager handles deduplication and routing of alerts, and Grafana visualizes metrics with pre-loaded dashboards. All components deploy in the monitoring namespace with RBAC policies and network policies pre-configured.

Self-Hosting & Configuration

• Requires a Kubernetes cluster (1.25+) with kubectl configured
• Apply manifests/setup first to install CRDs, then manifests/ for the full stack
• Customize the stack by forking the jsonnet source and regenerating manifests with jsonnet-bundler
• Add application monitoring by creating ServiceMonitor resources pointing to your service endpoints
• Configure Alertmanager routing for Slack, PagerDuty, or email via the alertmanager-secret

Key Features

• Batteries-included deployment with Prometheus, Grafana, Alertmanager, and exporters
• Pre-built alerting rules following Kubernetes monitoring best practices and SLO methodology
• Grafana dashboards for USE method analysis of nodes and RED method analysis of services
• Jsonnet-based customization for reproducible, version-controlled monitoring configuration
• Network policies and RBAC included for secure multi-tenant monitoring

Comparison with Similar Tools

• Prometheus Operator alone — provides CRDs and lifecycle management; kube-prometheus adds dashboards, alerts, and a full stack
• Helm kube-prometheus-stack — Helm chart wrapping similar components; kube-prometheus uses jsonnet for more composable customization
• Datadog / New Relic — commercial SaaS monitoring; kube-prometheus is self-hosted and free
• VictoriaMetrics stack — alternative metrics backend; kube-prometheus is Prometheus-native with broader community adoption
• Grafana Cloud — managed Prometheus and Grafana; kube-prometheus runs entirely within your cluster

FAQ

Q: How does kube-prometheus differ from the kube-prometheus-stack Helm chart? A: Both deploy similar components. kube-prometheus uses jsonnet for manifest generation and is the upstream reference. The Helm chart (maintained by the community) wraps similar resources with Helm values for easier customization.

Q: Can I add custom dashboards and alerting rules? A: Yes. Add ConfigMaps for Grafana dashboards and PrometheusRule resources for custom alerts. With the jsonnet approach, you can compose custom rules into the generated manifests.

Q: How much resource does the monitoring stack consume? A: Resource usage depends on cluster size and metric cardinality. A typical small cluster uses 2-4 GB RAM for Prometheus. The manifests include resource requests and limits for all components.

Q: Does it support long-term metric storage? A: kube-prometheus uses local Prometheus storage by default. For long-term retention, integrate with Thanos or Cortex as a remote write backend.

Sources

• https://github.com/prometheus-operator/kube-prometheus
• https://prometheus-operator.dev/docs/getting-started/introduction/