Linux Labs

Observability: metrics, alerts, logs / Stage 4 of 9

An alert rule: threshold and duration

Not running Starting usually ~20 s
Leave the lesson

Step 6 of 6 · summary · 2 min

Summary

  • Prometheus checks only the rules from the files listed in rule_files; the package's empty rule_files: means "no rules".
  • promtool check rules and promtool check config catch a mistake before the reread, and a failed reload keeps the previous rules.
  • A rule expression is a filter: each series in the result becomes an alert, an empty result means no alert. The threshold is compared with what the expression computes.
  • for keeps an alert pending until the condition has held for the set time, then it is firing; as soon as the expression goes empty, the alert leaves.
  • DiskAlmostFull from the start of the lesson was silent for two reasons: Prometheus had not loaded its file, and the > 0.95 threshold sat on the free share.
Ubuntu 24.04
Loading…