An alert rule: threshold and duration

Prometheus checks only the rules it has loaded from the files listed in its configuration.

An alert rule is an entry in a YAML file: the alert name, a PromQL expression and the conditions under which the alert fires. Rules are collected into groups, and one file can hold several:

groups:
  - name: machines      # a rule group
    rules:
      - alert: InstanceDown
        expr: up == 0   # which series "fire"
        for: 1m         # how long to hold

Prometheus learns which files to read from the rule_files list in /etc/prometheus/prometheus.yml: a file path or a pattern that picks up the whole directory.

rule_files:
  - /etc/prometheus/rules/machines.yml

In the package configuration rule_files: is empty — Prometheus has no rules, however many files lie next to it.

A mistake in a file is caught before the reread: promtool check rules <file> parses the YAML and the rule expressions, and promtool check config checks the configuration together with the connected files and prints how many it found. sudo systemctl reload prometheus rereads the configuration — just as after editing the scrape jobs in the lesson on scraping metrics.