How to Monitor Kubernetes Clusters Using Prometheus and Grafana

Running production workloads on Kubernetes is great, but without proper visibility you’re flying blind. Prometheus and Grafana together form a powerful, open‑source monitoring stack that can surface cluster health, application performance, and resource utilization in real time. In this guide we’ll walk through every step required to get a production‑grade monitoring solution up and running, from provisioning a cluster to creating alerts that keep you ahead of problems.

What You’ll Need

• A running Kubernetes cluster (1.21+ recommended)
• kubectl configured for cluster access
• Helm 3 installed on your workstation
• Basic knowledge of YAML and Kubernetes objects
• Access to a container registry (Docker Hub works fine)

Step 1: Set Up a Kubernetes Cluster

If you already have a cluster, skip this section. For testing, you can spin up a local cluster with kind or k3d. Below is a quick kind example that creates a three‑node cluster:

cat > kind-config.yaml <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker
EOF
kind create cluster --config=kind-config.yaml

Verify the cluster is reachable:

kubectl get nodes

You should see three nodes in the Ready state. For cloud providers, follow the vendor‑specific provisioning steps and ensure kubectl points to the new context.

Step 2: Install Helm

Helm is the de‑facto package manager for Kubernetes and makes installing Prometheus and Grafana painless. Install Helm (v3.x) with the official script:

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Confirm the installation:

helm version

Next, add the official prometheus-community and grafana repositories:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

These repos host the charts we’ll use in the next steps.

Step 3: Deploy Prometheus Using the kube‑prometheus‑stack Chart

The kube-prometheus-stack chart bundles Prometheus, Alertmanager, node‑exporter, and a set of default recording and alerting rules. Deploy it into a dedicated monitoring namespace:

kubectl create namespace monitoring
helm install prometheus prometheus-community/kube-prometheus-stack 
  --namespace monitoring 
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

The --set flag ensures that any ServiceMonitor objects you create later will be picked up automatically. Wait a couple of minutes for all components to become ready, then check:

kubectl get pods -n monitoring

You should see pods for prometheus, alertmanager, grafana (installed later), and various exporters.

Step 4: Deploy Grafana and Connect It to Prometheus

Grafana is included in the kube-prometheus-stack chart, but we’ll expose it via a LoadBalancer (or Ingress) for easy access. First, retrieve the autogenerated admin password:

kubectl get secret --namespace monitoring prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Now expose Grafana. For cloud clusters, a LoadBalancer service works well:

kubectl patch svc prometheus-grafana -n monitoring 
  -p '{"spec": {"type": "LoadBalancer"}}'

Find the external IP:

kubectl get svc prometheus-grafana -n monitoring

Open the IP in a browser, log in with admin and the password you just decoded. Grafana should already have a pre‑configured Prometheus data source named Prometheus. If not, add it manually:

URL: http://prometheus-operated.monitoring.svc:9090
Access: Server (default)

Once the data source is verified, you’re ready to build dashboards.

Step 5: Create Dashboards for Cluster Metrics

Grafana ships with a handful of community dashboards. Import the popular “Kubernetes Cluster Monitoring (via Prometheus)” dashboard (ID 315). In Grafana:

1. Click the “+” icon → “Import”.
2. Enter 315 and click “Load”.
3. Select the Prometheus data source you configured.
4. Click “Import”.

The dashboard will instantly display node CPU, memory, pod status, and more. For custom needs, you can clone the dashboard JSON and edit queries. Here’s a quick example of a PromQL query that shows the top 5 namespaces by CPU usage:

sum(rate(container_cpu_usage_seconds_total{image!="",container!="POD"}[5m])) by (namespace) 
  | topk(5, __value__)

Save the panel, give it a meaningful title, and you’ve just extended your observability stack.

Step 6: Configure Alerting Rules and Notification Channels

Alertmanager, bundled with the stack, handles routing and silencing. First, create a simple alert rule that fires when any node’s CPU usage exceeds 80% for 5 minutes. Save the following as cpu-high.yaml and apply it:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: node-cpu-high
  namespace: monitoring
spec:
  groups:
  - name: node.rules
    rules:
    - alert: NodeCPUHigh
      expr: sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance) 
            / sum(rate(node_cpu_seconds_total[5m])) by (instance) > 0.8
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High CPU usage on {{ $labels.instance }}"
        description: "CPU usage is above 80% for more than 5 minutes."

Apply the rule:

kubectl apply -f cpu-high.yaml

Next, configure a notification channel (e.g., Slack). In Grafana, go to “Alerting → Notification channels → New channel”, choose “Slack”, paste the webhook URL, and set the “Send on all alerts” toggle. Then, in the Alertmanager config (automatically generated by the chart), map the Grafana‑created route to the Slack receiver. Restart the Alertmanager pod if needed:

kubectl rollout restart deployment prometheus-alertmanager -n monitoring

Trigger the alert by running a CPU‑intensive pod, and you should receive a Slack message within minutes.

Common Mistakes to Avoid

1. Forgetting to label ServiceMonitors. Prometheus only scrapes services that have a matching serviceMonitorSelector. Ensure your services carry the label release: prometheus (or whatever you set in the chart).

2. Over‑scraping. Adding every namespace to the scrape list can explode storage usage. Scope your Prometheus spec with namespaceSelector and serviceMonitorNamespaceSelector to limit the scope.

3. Ignoring resource limits. Prometheus itself can consume a lot of CPU and RAM. Set resources.limits in the Helm values file to prevent it from starving other workloads.

4. Using default passwords. The Helm chart creates a random admin password, but many tutorials override it with “admin”. Always keep the secret or change it after first login.

5. Not persisting data. By default the chart uses an emptyDir volume, which is lost on pod restart. Enable a PersistentVolumeClaim by adding prometheus.prometheusSpec.storageSpec.volumeClaimTemplate to your values.

Tips and Tricks

• Leverage recording rules. Pre‑compute expensive queries (e.g., node_cpu:rate5m) to speed up dashboards.

• Use Grafana’s provisioning. Store dashboard JSON files in a ConfigMap and mount them, so dashboards survive Grafana upgrades.

• Enable Thanos sidecar. If you need long‑term storage or global querying across clusters, add the Thanos sidecar to the Prometheus pods.

• Tag alerts with severity. Alertmanager routing can silence “info” alerts during business hours while still notifying on “critical” ones.

• Export metrics from your own apps. Include the Prometheus client library for your language (Go, Java, Python) and expose /metrics endpoints.

Frequently Asked Questions

Can I use Prometheus without Helm?

Yes. You can apply the raw manifests from the prometheus-operator GitHub repo, but Helm simplifies version upgrades, value overrides, and dependency management.

How much storage does Prometheus need?

It depends on scrape interval, retention period, and the number of metrics. A common baseline is 2 GB per 10 k time series for a 15‑day retention. Start with a 20‑30 GB PVC and monitor prometheus_tsdb_storage_blocks_bytes to adjust.

Is Grafana free for commercial use?

Grafana OSS is licensed under Apache 2.0 and can be used commercially at no cost. Enterprises often opt for Grafana Enterprise for added plugins, support, and reporting features.

Conclusion

By following this guide you now have a fully functional monitoring stack that gives you deep insight into Kubernetes cluster health, alerts you before issues become outages, and provides a flexible platform for custom dashboards. Keep refining your scrape targets, recording rules, and alert thresholds as your workloads evolve, and you’ll stay ahead of the curve in any production environment.

Photo by Eduardo Pastor on Unsplash