Observability: metrics, alerts, logs / Stage 4 of 9 · step 5 of 6

The full-volume alert stays silent

~30 min

Not running Starting usually ~20 s

Situation

The reports server has a /srv/data volume where the nightly sales exports go. Two weeks ago it ran out of space at night, the reports were not built in the morning, and the team learned about it from accounting. A colleague added the DiskAlmostFull alert rule for an almost full volume to Prometheus — the rule files live in /etc/prometheus/rules/ — reloaded the configuration and went on holiday. Yesterday the volume filled up again. There was no alert.

Prometheus and node_exporter are running on the machine; metrics are scraped and rules are evaluated every 5 seconds. There is no browser: you see Prometheus only from the terminal — curl -s localhost:9090/api/v1/… | jq, promtool. sudo works without a password.

Task

The DiskAlmostFull rule is loaded into the running Prometheus from the rule files, and promtool check config /etc/prometheus/prometheus.yml passes.
When less than 10 % of /srv/data is free, the DiskAlmostFull alert fires — goes to the firing state — within a minute: the nightly export fills the volume quickly.
When more than 10 % is free again, the alert clears.
Prometheus and node_exporter are enabled and run with the settings from their files.

The check fills the volume with a temporary file itself, waits for the alert and deletes the file. You can test your rule the same way: sudo fallocate -l 50M /srv/data/test, and afterwards sudo rm /srv/data/test. Leave the exports in /srv/data/reports alone.

Lab goals

0 of 4

The DiskAlmostFull rule is loaded into the running Prometheus from the rule files, promtool check config passes — not checked yet
When less than 10 % of /srv/data is free, the DiskAlmostFull alert fires — firing — within a minute — not checked yet
When more than 10 % is free again, the alert clears — not checked yet
Prometheus and node_exporter are enabled and run with the settings from their files — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Observability · picked for this lab

Prometheus: metric collection and alerting rules

Cheat sheet

$_ — insert into the terminal

sudo systemctl enable --now prometheus turn it on
promtool check config /etc/prometheus/prometheus.yml validate
sudo systemctl reload prometheus reread the config
curl -s localhost:9090/-/ready is the server ready
promtool query instant http://localhost:9090 up what it collects
curl -s localhost:9090/api/v1/targets | jq .data targets, errors
curl -s localhost:9090/api/v1/rules | jq .data rules
curl -s localhost:9090/api/v1/alerts | jq .data alerts
promtool check rules /etc/prometheus/disk.rules.yml a rule file

prometheus.yml

The packaged config on the training machine, without comments:

yaml

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
      monitor: 'example'
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']
rule_files:
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    scrape_timeout: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']

Section	What it sets
`global`	default scrape and rule evaluation intervals; `external_labels` are labels every alert gets
`alerting`	the Alertmanager address
`rule_files`	rule files; the package leaves the list empty, so there are no rules
`scrape_configs`	what to collect: jobs and their targets

Jobs and targets

A job is a group of similar targets with shared scrape settings:

yaml

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100', 'node2:9100']
  - job_name: app
    scrape_interval: 5s
    metrics_path: /metrics
    static_configs:
      - targets: ['127.0.0.1:8000']
        labels:
          team: web

Prometheus adds the job label (the job name) and the instance label (the address from targets) to every series of a target; the labels from labels go on every series of that block's targets too: up{instance="127.0.0.1:8000", job="app", team="web"}. metrics_path defaults to /metrics, scrape_timeout to 10 seconds. An interval set in a job overrides the global one.

labels goes inside a static_configs item, next to targets. A job has no such field: promtool check config answers field labels not found in type config.ScrapeConfig. Labels from global.external_labels are something else: only data that leaves Prometheus gets them (alerts sent to Alertmanager, remote storage), and queries to Prometheus itself do not show them.

Reread the config

bash

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

reload sends the service a HUP signal, and Prometheus rereads the config and the rule files without a restart. The HTTP way, curl -X POST localhost:9090/-/reload, is off on the training machine: the answer is Lifecycle API is not enabled..

Whether the server accepted the config:

the journal, journalctl -u prometheus -o cat | tail -3, has either Completed loading of configuration file or Error reloading config;
the metric prometheus_config_last_reload_successful is 1 or 0 (it updates with the next scrape of the prometheus job, a few seconds later);
/api/v1/status/config and /api/v1/rules show what is actually loaded.

systemctl reload exits with code 0 even with a broken file: the signal was delivered, and whether to accept the config is up to Prometheus. A bad config is not applied on reload: the server keeps running with the old one. On restart with a bad config the service will not start.

bash

curl -s localhost:9090/metrics | grep '^prometheus_config_last_reload'

Startup flags

The service flags are the ARGS variable in /etc/default/prometheus, empty in the package. The ones needed most often:

Flag	What it sets
`--storage.tsdb.retention.time=30d`	how long to keep data (15 days by default)
`--web.enable-lifecycle`	turns on `/-/reload` and `/-/quit` over HTTP
`--web.listen-address=127.0.0.1:9090`	address and port (`0.0.0.0:9090` by default)

sudo systemctl restart prometheus applies the flags. Which flags the server is running with now: /api/v1/status/flags.

Add a target and make sure it is collected

bash

sudoedit /etc/prometheus/prometheus.yml   # the target in targets
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
sleep 20
promtool query instant http://localhost:9090 'up{job="app"}'

Wait at least one scrape interval. => 1 means the target is being collected, => 0 means Prometheus could not reach it, and the reason is in lastError.

Try a threshold before writing the rule

bash

promtool query instant http://localhost:9090 \
  'node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.05'

A rule expression is an ordinary query. An empty answer means the alert would not fire right now; series in the answer are the alerts that would fire, one per series.

Compare what is loaded with the file

bash

curl -s localhost:9090/api/v1/status/config \
  | jq -r .data.yaml | grep -A6 'job_name: app'
curl -s localhost:9090/api/v1/rules \
  | jq -r '.data.groups[].rules[].name'

The server prints the loaded config with default values filled in (honor_timestamps, scheme), so compare the fields you need rather than whole files.

A target is down

The target's lastError names the cause; what each error means is in the table of prometheus. Compare scrapeUrl with what the service listens on: ss -tlnp on the target machine and curl to the same address. More about the exporter: node-exporter, node-exporter and node-exporter.

Error reloading config

text

level=error msg="Error reloading config" err="couldn't load
  configuration (--config.file=\"/etc/prometheus/prometheus.yml\"):
  parsing YAML file
  /etc/prometheus/prometheus.yml: yaml: line 44: did not find expected
  ',' or ']'"

reload did not apply the new config; the server runs with the old one, and prometheus_config_last_reload_successful is 0. Fix the error, check with promtool check config and reload again.

error loading rules, previous rule set restored

A rule file has an error, and after reload the previous rules stay: /api/v1/rules shows the old set. What is wrong: promtool check rules <file>:

text

  FAILED:
/tmp/bad.yml: 5:15: group "demo", rule 1, "DiskAlmostFull": could not
  parse expression: 1:47: parse error: unexpected <op:<>

The service does not start after a restart

text

level=error msg="Error loading config (--config.file=…)"
  err="parsing YAML file /etc/prometheus/prometheus.yml: …"

Prometheus does not start with a bad config, and systemctl is-active says failed. Fix it, check with promtool check config, then run sudo systemctl reset-failed prometheus and start.

field … not found in type config.plain

text

  FAILED: parsing YAML file /tmp/p.yml: yaml: unmarshal errors:
  line 26: field scrape_config not found in type config.plain

An unknown key: here scrape_config instead of scrape_configs. Other common typos are static_config, target, and job instead of job_name. The key can also be right but at the wrong level: field labels not found in type config.ScrapeConfig means labels was written in the job, while it belongs inside static_configs (prometheus).

did not find expected ',' or ']'

A YAML syntax error: an unclosed bracket in targets: ['…', a quote or an indent. The line number is in the message.

The rule is in the file, but /api/v1/rules is empty

the file is not in rule_files; the packaged config leaves that list empty;
the path is wrong: a relative path counts from the directory of prometheus.yml, and promtool check config tells how many rule files it found (SUCCESS: 1 rule files found);
there was no reload after the edit, or it failed, see prometheus.

The alert does not fire

the rule is loaded and its state is inactive: run the expr with promtool query instant. Empty means the condition is false right now: a reversed sign (> instead of <), a threshold in percent against a ratio, or a label with a typo;
the alert is stuck in pending: for has not passed yet, or the condition comes and goes and the countdown starts over;
the alert is firing but no notification arrives: continue with alertmanager.

Lifecycle API is not enabled

The answer to curl -X POST localhost:9090/-/reload: Prometheus runs without --web.enable-lifecycle. Reload with sudo systemctl reload prometheus.

query error: connection refused

text

query error: Post "http://localhost:9090/api/v1/query": dial tcp
  127.0.0.1:9090: connect: connection refused

promtool query could not reach the server: Prometheus is not running, see systemctl status prometheus.

Restart with an unchecked config

A reload with an error keeps the old config; a restart with an error leaves the machine without monitoring: the service does not come up, metrics are not collected, alerts are not evaluated. Before any restart, run promtool check config.

The full-volume alert stays silent

Situation

Task

Signing in is needed to start the machine

Add a target and make sure it is collected

Try a threshold before writing the rule

Compare what is loaded with the file

A target is down

Error reloading config

error loading rules, previous rule set restored

The service does not start after a restart

field … not found in type config.plain

did not find expected ',' or ']'

The rule is in the file, but /api/v1/rules is empty

The alert does not fire

Lifecycle API is not enabled

query error: connection refused

Restart with an unchecked config