Prometheus in Linux: metrics collection

Q: A target is DOWN

Open /targets — the reason is right there (connection refused / timeout / 404). Check that the exporter is listening (ss -tlnp | grep 9100) and the port is open (firewall). curl http://target:9100/metrics to see if it serves metrics.

Q: No data for a metric

Check the metric name and labels in the UI (autocomplete). up{job="x"} tells you whether the job is being scraped.

Q: Prometheus is eating disk space

That's retention and the number of series. Use --storage.tsdb.retention.time=15d; cut down high-cardinality labels.

Q: Alerts aren't arriving

Check the rules (promtool check rules), the connection to Alertmanager, and its routes.

Overview

Prometheus collects metrics using a pull model: it periodically scrapes HTTP /metrics endpoints on its targets. It stores time series, lets you query them with PromQL, and sends alerts.

exporters (node_exporter, app /metrics) ← scrape ← Prometheus → PromQL / alerts → Alertmanager

Exporters expose services' metrics: node_exporter (the host), nginx-exporter, and so on.

Configuration

prometheus.yml — what to scrape and how often:

yaml

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['10.0.0.1:9100', '10.0.0.2:9100']
  - job_name: app
    metrics_path: /metrics
    static_configs:
      - targets: ['app:8000']

bash

promtool check config prometheus.yml      # check the config
curl -s localhost:9090/-/reload -X POST   # reload it (if web.enable-lifecycle is on)

Recipes

1

PromQL and checking status

The web UI is on :9090. Example queries:

promql

up                                  # 1/0 — is the target alive
rate(http_requests_total[5m])       # requests per second over 5 minutes
node_filesystem_avail_bytes / node_filesystem_size_bytes   # fraction of free disk space
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m]))*100)  # %CPU

http://<prom>:9090/targets shows the status of every target (UP/DOWN and scrape errors).

Troubleshooting

4

A target is DOWN

Open /targets — the reason is right there (connection refused / timeout / 404). Check that the exporter is listening (ss -tlnp | grep 9100) and the port is open (firewall). curl http://target:9100/metrics to see if it serves metrics.

No data for a metric

Check the metric name and labels in the UI (autocomplete). up{job="x"} tells you whether the job is being scraped.

Prometheus is eating disk space

That's retention and the number of series. Use --storage.tsdb.retention.time=15d; cut down high-cardinality labels.

Alerts aren't arriving

Check the rules (promtool check rules), the connection to Alertmanager, and its routes.

Prometheus — metrics collection

Cheat sheet