Last night's order export filled almost the whole /srv/data volume. The machine is monitored: Prometheus scrapes node_exporter metrics every 5 seconds, and /etc/prometheus/rules/disk.yml holds the DiskAlmostFull rule — exactly about a full disk. See how much space is left on the volume and which alerts Prometheus considers firing.
Compare the volume and the alert list
Check how much space is left on the /srv/data volume, then which alerts Prometheus has firing and which rules it has loaded.
First, predict
The volume is 95% full, and the DiskAlmostFull rule for a full disk is written. What will the Prometheus alert list return?
Now start the exercise, run the command and compare your prediction with the output.
The volume is 95% full, the file with DiskAlmostFull sits in /etc/prometheus/rules, and the Prometheus rule list is empty. Why is there no alert?
-
The metrics are there:
promtool query instant http://localhost:9090 'node_filesystem_avail_bytes'prints the /srv/data series. It is the rule list that is empty. -
/api/v1/alertslists Prometheus's own alerts, including the ones passed on. Here Prometheus does not even have the rule:curl -s localhost:9090/api/v1/rules | jq .data. -
A waiting alert would be listed with
"state": "pending", and a loaded rule would have a state of its own. Here both lists are empty.
The rule list is empty: Prometheus has not loaded disk.yml, so DiskAlmostFull does not exist for it.