How to Monitor Server Health with Prometheus and Grafana: A Step‑by‑Step Guide

Keeping an eye on your servers is essential for any modern operation. When a CPU spikes, memory leaks, or a disk fills up, you need to know instantly—and you need a reliable way to visualize those metrics. Prometheus and Grafana together provide a powerful, open‑source monitoring stack that can be deployed in minutes and scaled to thousands of nodes. In this guide we’ll walk through every step required to get a basic, production‑ready monitoring solution up and running on a Linux server. By the end you’ll have a live dashboard, alert rules, and a solid foundation for expanding your observability toolkit.

What You’ll Need

• A Linux server (Ubuntu 22.04, Debian 12, or CentOS 9) with sudo access.
• Root or sudo privileges to install packages.
• Basic familiarity with systemd services and networking.
• Port 9090 (Prometheus) and 3000 (Grafana) open in your firewall.
• A web browser to access the Grafana UI.

Step 1: Install Prometheus

First, download the latest stable release from the official GitHub repository. Replace 2.53.0 with the current version if needed.

sudo apt-get update && sudo apt-get install -y wget tar
wget https://github.com/prometheus/prometheus/releases/download/v2.53.0/prometheus-2.53.0.linux-amd64.tar.gz
sudo tar -xzf prometheus-2.53.0.linux-amd64.tar.gz -C /opt/
sudo mv /opt/prometheus-2.53.0.linux-amd64 /opt/prometheus
sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus
sudo chown -R prometheus:prometheus /opt/prometheus

Create a minimal prometheus.yml configuration file.

cat <<'EOF' | sudo tee /etc/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']
EOF
sudo chown prometheus:prometheus /etc/prometheus.yml

Now set up a systemd service.

cat <<'EOF' | sudo tee /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/prometheus/prometheus 
  --config.file=/etc/prometheus.yml 
  --storage.tsdb.path=/opt/prometheus/data 
  --web.console.templates=/opt/prometheus/consoles 
  --web.console.libraries=/opt/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

Verify it’s running: curl http://localhost:9090/metrics. You should see a long list of internal Prometheus metrics.

Step 2: Install Node Exporter (Metrics Collector)

Node Exporter gathers system‑level metrics (CPU, memory, disk, network) and exposes them for Prometheus to scrape.

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.1/node_exporter-1.8.1.linux-amd64.tar.gz
sudo tar -xzf node_exporter-1.8.1.linux-amd64.tar.gz -C /opt/
sudo mv /opt/node_exporter-1.8.1.linux-amd64 /opt/node_exporter
sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter
sudo chown -R node_exporter:node_exporter /opt/node_exporter

Create a service file:

cat <<'EOF' | sudo tee /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/opt/node_exporter/node_exporter

[Install]
WantedBy=default.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Check the endpoint: curl http://localhost:9100/metrics. You should see metrics like node_cpu_seconds_total.

Step 3: Verify Prometheus Scraping

Open http://localhost:9090/targets. The node_exporter job should be listed as UP. If it shows DOWN, double‑check the targets address and firewall rules.

Step 4: Install Grafana

Grafana provides the visual layer. Use the official APT repository for automatic updates.

sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt-get update
sudo apt-get install -y grafana
sudo systemctl enable --now grafana-server

Grafana runs on port 3000. Browse to http://localhost:3000. The default credentials are admin / admin. You’ll be prompted to change the password immediately.

Step 5: Add Prometheus as a Data Source

In Grafana’s UI:

1. Click the gear icon → Data Sources.
2. Click Add data source and select Prometheus.
3. Set the URL to http://localhost:9090.
4. Leave the default Access as Server and click Save & Test.

If the test succeeds, Grafana can now query Prometheus.

Step 6: Build Your First Dashboard

Grafana ships with a ready‑made “Node Exporter Full” dashboard (ID 1860). Import it to avoid reinventing the wheel.

1. Click the +** icon → Import.
2. Enter 1860 in the Grafana.com Dashboard field and click Load.
3. Select the Prometheus data source you just added.
4. Click Import.

The dashboard now displays CPU usage, memory pressure, disk I/O, and network traffic in real time. Feel free to duplicate panels, adjust time ranges, or add custom alerts.

Step 7: Configure Alerting (Prometheus Alertmanager)

While Grafana can send alerts, the Prometheus ecosystem recommends using Alertmanager for routing, silencing, and inhibition.

wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
sudo tar -xzf alertmanager-0.27.0.linux-amd64.tar.gz -C /opt/
sudo mv /opt/alertmanager-0.27.0.linux-amd64 /opt/alertmanager
sudo useradd --no-create-home --shell /usr/sbin/nologin alertmanager
sudo chown -R alertmanager:alertmanager /opt/alertmanager

Create a simple alertmanager.yml that sends email via SMTP (replace placeholders).

cat <<'EOF' | sudo tee /etc/alertmanager.yml
global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'YOUR_PASSWORD'
route:
  receiver: email-notifications
receivers:
- name: email-notifications
  email_configs:
  - to: '[email protected]'
EOF
sudo chown alertmanager:alertmanager /etc/alertmanager.yml

Systemd service for Alertmanager:

cat <<'EOF' | sudo tee /etc/systemd/system/alertmanager.service
[Unit]
Description=Prometheus Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/opt/alertmanager/alertmanager 
  --config.file=/etc/alertmanager.yml

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager

Add Alertmanager to Prometheus configuration (append to prometheus.yml):

alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']

Reload Prometheus: sudo systemctl reload prometheus. Then create a sample rule file cpu_alert.yml:

cat <<'EOF' | sudo tee /etc/prometheus/rules.yml
groups:
- name: cpu_alerts
  rules:
  - alert: HighCpuUsage
    expr: avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)  80% on {{ $labels.instance }}"
      description: "CPU idle time is below 20% for more than 2 minutes."
EOF
sudo chown prometheus:prometheus /etc/prometheus/rules.yml

Reference the rule file in prometheus.yml under rule_files: and reload again. When the condition triggers, Alertmanager will email the ops team.

Step 8: Secure Your Stack

Running services on default ports without authentication is fine for a lab, but production demands TLS and auth.

• Prometheus: Use a reverse proxy (NGINX or Caddy) with basic auth or client certificates. Example NGINX snippet:

server {
    listen 443 ssl;
    server_name prometheus.example.com;
    ssl_certificate /etc/ssl/certs/example.crt;
    ssl_certificate_key /etc/ssl/private/example.key;
    auth_basic "Prometheus";
    auth_basic_user_file /etc/nginx/.htpasswd;
    location / {
        proxy_pass http://localhost:9090;
    }
}

• Grafana: Enable HTTPS in /etc/grafana/grafana.ini and enforce strong passwords. Also consider LDAP or OAuth integration.

Finally, restrict firewall access to only trusted IP ranges:

sudo ufw allow from 203.0.113.0/24 to any port 3000
sudo ufw allow from 203.0.113.0/24 to any port 9090

Common Mistakes to Avoid

1. Forgetting to reload services after config changes. Prometheus and Alertmanager won’t pick up new files until you run systemctl reload or restart.

2. Mismatched target addresses. Using localhost works for a single‑node setup, but in a multi‑node environment you must reference each node’s IP or DNS name.

3. Exposing ports to the internet. Never leave 9090 or 3000 open to the world without TLS and auth.

4. Over‑scraping. The default 15‑second interval is fine for most servers, but setting a very low interval (e.g., 1s) can overload both the exporter and Prometheus storage.

5. Ignoring retention settings. By default Prometheus keeps data for 15 days. Adjust --storage.tsdb.retention.time if you need longer history, but be aware of disk usage.

Tips and Tricks

• Use recording rules. Pre‑aggregate expensive queries (e.g., average CPU per host) to speed up dashboards.

• Leverage Grafana variables. Create a $instance dropdown to switch between servers on a single dashboard.

• Export dashboards as JSON. Version‑control them in Git for reproducible environments.

• Enable Prometheus remote write. Forward metrics to a long‑term storage solution like Cortex or Thanos for high‑availability queries.

• Monitor Prometheus itself. Add the prometheus job to scrape its own /metrics endpoint; Grafana has a built‑in “Prometheus Overview” dashboard.

Frequently Asked Questions

Can I monitor Windows servers with this stack?

Yes. Install the Windows exporter (formerly wmi_exporter) on each Windows host and add it as a scrape target in prometheus.yml. The same Grafana dashboards can be adapted to display Windows‑specific metrics.

Do I need to install Prometheus on every server I want to monitor?

No. Prometheus follows a pull model; you only need one central Prometheus server that scrapes exporters running on each monitored host.

How much disk space does Prometheus need?

It depends on retention, scrape interval, and the number of metrics. As a rule of thumb, a 1‑CPU, 2‑GB server with a 15‑second interval and 15‑day retention consumes ~2‑3 GB of storage for ~100 metrics per node.

Conclusion

Prometheus and Grafana together give you a flexible, open‑source monitoring solution that scales from a single VM to a sprawling data‑center. By following the eight steps above—installing exporters, wiring Prometheus, visualizing with Grafana, and adding alerting—you’ll gain real‑time insight into CPU, memory, disk, and network health while avoiding common pitfalls. Keep security in mind, fine‑tune scrape intervals, and treat your dashboards as code. With that foundation, you’re ready to expand into service‑level objectives, custom exporters, and long‑term storage, turning raw metrics into actionable intelligence.

Photo by Kevin Ache on Unsplash