Threshold and duration

The rule is loaded, yet its state is inactive: the expression is evaluated every 5 seconds and returns nothing.

expr is an expression with a comparison. The comparison works as a filter: it keeps the series for which the condition holds, and each remaining series becomes an alert with its own labels. An empty result means no alert. The left side of the expression without the comparison shows what the rule sees:

bash

promtool query instant http://localhost:9090 \
  'node_filesystem_avail_bytes{mountpoint="/srv/data"}
   / node_filesystem_size_bytes{mountpoint="/srv/data"}'
# {…, mountpoint="/srv/data"} => 0.0499…

The expression computes the free share of the volume: 0.05, that is 5%. The rule's condition > 0.95 keeps volumes with more than 95% free — nearly empty ones. A threshold is compared with what the expression computes: for the free share, a full disk is a small number.

for is how long the condition must hold before the alert fires: a short spike will not raise it. Until the time is up, the alert waits.

State	Meaning
`inactive`	the expression returned nothing
`pending`	it returned something, held for less than `for`
`firing`	held for longer than `for` — the alert fires

/api/v1/rules shows the rule states, /api/v1/alerts the waiting and firing alerts.

Exercise · do it

Fix the DiskAlmostFull threshold

The DiskAlmostFull rule in /etc/prometheus/rules/disk.yml is loaded but silent. Fix it so it fires when less than 10% of the volume is free, keeping the duration as it is. Check the file with promtool, reread the configuration and wait for the alert to fire.

Goals · 0 of 2

Fix the threshold in the rule not done yet
Bring the alert to firing not done yet

Check yourself 1 question

DiskAlmostFull is firing. You delete /srv/data/export.bin, and the volume has plenty of space again. What will /api/v1/alerts show half a minute later?

for delays only the firing. Once the expression returns nothing the alert goes away: check curl -s localhost:9090/api/v1/alerts | jq .data after the delete.
Alertmanager sends notifications, while whether an alert fires is decided by the expression in Prometheus. It re-evaluates the rule every 5 seconds.
No restart is needed: the rule is evaluated every 5 seconds, and an empty result removes the alert at the next evaluation.

The expression went empty, and at the next evaluation the alert left the list.

An alert rule: threshold and duration