The rule decides when the alert fires, while labels and a description tell whoever receives it what happened.
labels adds labels to the alert on top of the series labels. severity: warning or critical — Alertmanager uses such labels to decide whom to notify; that is the next lesson. annotations is text for a person: summary can include the series labels and value.
On the firing DiskAlmostFull this line becomes /srv/data: free space 4.994%.
sudo systemctl reload prometheus succeeds even with a mistake in a rule file: Prometheus logs "previous rule set restored" and keeps working with the old rules. Check the file with promtool check rules before the reread, and the result with curl -s localhost:9090/api/v1/status/runtimeinfo | jq .data.reloadConfigSuccess.
Your own rule follows the same technique on other metrics. The volume's metrics are in the exporter output: curl -s localhost:9100/metrics | grep 'mountpoint="/srv/data"'.
Write a rule for inodes
The /srv/data volume has space, yet sudo touch /srv/data/reports/new.csv answers “No space left on device” — the inodes ran out. Add an InodesAlmostFull rule to /etc/prometheus/rules: it fires when less than 10% of the inodes on /srv/data are free for longer than 15 seconds, and gives the alert a severity label set to warning. Load it into Prometheus and wait for firing.
Goals · 0 of 2
- Write the InodesAlmostFull rule and load it not done yet
- Bring the alert to firing not done yet