kube-prometheus-stack is the de facto Kubernetes monitoring bundle: Prometheus for metrics collection, Alertmanager for routing alerts, Grafana for dashboards, and pre-built dashboards/recording rules/alert rules for cluster components. The team that installs it via Helm has production-grade monitoring with sensible defaults in 30 minutes; the team that rolls its own from scratch spends weeks tuning scrape configs, alert rules, and dashboard panels. The stack is the right starting point for most clusters.
Table of contents
- What kube-prometheus-stack includes
- Installation via Helm
- Configuring alerts
- Storage and retention
- What kube-prometheus-stack does not do
- FAQ
What kube-prometheus-stack includes
Prometheus server with Kubernetes service discovery. Prometheus automatically discovers pods, services, and endpoints via the Kubernetes API and scrapes them based on annotations. The team that runs the stack has metrics flowing from every workload without configuring individual scrape jobs.
Alertmanager for alert routing and deduplication. Alertmanager groups related alerts, deduplicates, and routes to receivers (Slack, PagerDuty, email, webhook). The team that has Alertmanager integrated has alerts that don’t page 5 times for the same issue.
Grafana with pre-built dashboards. The stack ships ~50 dashboards covering node metrics, pod metrics, cluster components, ingress, and more. The team that has these dashboards out of the box has immediate visibility without building dashboards from scratch.
kube-state-metrics for cluster-state metrics. Exposes metrics about Kubernetes objects (Deployment status, Node conditions, Pod phases) that Prometheus can scrape. The team that has kube-state-metrics can alert on ‘Deployment has 0 ready replicas’ or ‘Node is NotReady’.
Installation via Helm
Add the Helm repo: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts. Update repos: helm repo update. Install: helm install kube-prom prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace.
Customize via values.yaml for production. Set prometheus.prometheusSpec.retention: 30d (default 10d), set prometheus.prometheusSpec.storageSpec to a PersistentVolumeClaim (the default uses emptyDir, lost on pod restart), set grafana.adminPassword (default ‘prom-operator’).
Resource sizing. Prometheus needs 2-4GB RAM for a small cluster (50 nodes, 500 pods); 8GB+ for larger clusters. Grafana needs 256MB-1GB. Alertmanager needs 128MB-512MB. The team that runs on small nodes (2GB RAM) tunes resource limits down; the team that has headroom lets the stack consume what it needs.
Configuring alerts
Default alert rules cover the basics. Node down, Deployment replicas missing, persistent volume filling, pod crash looping, container OOM killed, etc. The team that has these defaults catches the common issues; the team that needs application-specific alerts adds custom rules.
Custom rules via PrometheusRule CRD. Define a PrometheusRule resource; kube-prometheus-stack picks it up automatically. Example: alert on HTTP 5xx rate above 1% for 5 minutes. The team that has custom rules tailored to their services catches application-specific issues.
Alert routing via AlertmanagerConfig. Route alerts by team, severity, or namespace to different receivers. The team that has the database team get database alerts and the application team get application alerts has a sustainable on-call rotation.
Storage and retention
Use a PersistentVolumeClaim for Prometheus storage. The default emptyDir loses all metrics on pod restart. The team that runs a PVC-backed Prometheus has metrics survive pod restarts and node failures.
Tune retention based on disk and query needs. retention: 30d keeps 30 days of high-resolution metrics; older data is dropped. The team that needs longer retention uses Thanos or Cortex for remote storage; the team that does not needs PVC size = (ingestion rate * retention period * 1.5 overhead).
WAL (Write-Ahead Log) for crash safety. Prometheus writes to the WAL before flushing to TSDB; on crash, the WAL is replayed to recover unflushed data. The team that runs kube-prometheus-stack has WAL enabled by default.
What kube-prometheus-stack does not do
Log aggregation. Use Loki, Elasticsearch, or a managed service for logs. The team that wants logs alongside metrics adds Loki to the stack or runs a separate log stack.
Distributed tracing. Use Jaeger, Tempo, or a managed service for traces. The team that needs tracing for latency debugging adds OpenTelemetry collectors and a tracing backend.
Long-term metrics storage. Prometheus is not a long-term store. The team that needs months or years of metrics for compliance or trend analysis uses Thanos, Cortex, or a managed service (Datadog, Grafana Cloud, Chronosphere).
Application-level instrumentation. kube-prometheus-stack scrapes cluster and infrastructure metrics; the team that wants application metrics (request rate, error rate, latency per endpoint) instruments the application with Prometheus client libraries or OpenTelemetry exporters.
FAQ
What is the difference between Prometheus and kube-prometheus-stack?
Prometheus is the metrics server. kube-prometheus-stack is a Helm chart that bundles Prometheus, Alertmanager, Grafana, kube-state-metrics, node-exporter, and pre-built dashboards/recording rules/alert rules. The team that installs kube-prometheus-stack gets everything needed for Kubernetes monitoring in one chart.
How much does kube-prometheus-stack cost?
Free - all components are open source. The cost is the resources consumed: 2-8GB RAM for Prometheus, 1GB for Grafana, plus disk for metrics storage. The team that runs on a small cluster (10 nodes, 100 pods) needs ~4GB RAM for the stack. Larger clusters need more.
Should I use Prometheus Operator or kube-prometheus-stack?
kube-prometheus-stack uses Prometheus Operator as a dependency. The Operator manages Prometheus, Alertmanager, and ThanosRuler custom resources. The team that uses kube-prometheus-stack gets the Operator’s CRDs (ServiceMonitor, PrometheusRule, AlertmanagerConfig) plus the bundled components.
How do I add custom dashboards?
Create a ConfigMap with the dashboard JSON in the grafana label and the dashboard label. The team that follows the kube-prometheus-stack convention has Grafana pick up dashboards automatically. Alternatively, import via the Grafana UI.
What is the difference between recording rules and alert rules?
Recording rules precompute frequently-used queries (e.g., rate(http_requests_total[5m])) and store the result as a new metric. Alert rules evaluate queries and fire alerts. The team that uses recording rules has faster dashboards; the team that uses alert rules has actionable notifications.
How do I migrate from kube-prometheus-stack to Grafana Cloud or Datadog?
Install the relevant agent (Grafana Agent, Datadog Agent) in the cluster. Configure it to scrape the same metrics kube-prometheus-stack does. Disable kube-prometheus-stack’s Prometheus (or remove it entirely). The team that migrates keeps the dashboards but uses a managed backend.
How this fits the rest of the stack
For a sense of what the full project costs before it commits, the RunxBuild hosting calculator shows the line items together. The API, the database, the storage, the worker, the bandwidth - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers.
Useful related references: