Alertmanager: who gets alerts and how often

cheat sheet · Ubuntu 24.04 · Alertmanager 0.26 https://lab.dimsanych.ru/en/wiki/alertmanager/

Cheat sheet

amtool check-config /etc/prometheus/alertmanager.yml validate
sudo systemctl reload prometheus-alertmanager reread the config
amtool alert active alerts
amtool alert query severity=critical filter by labels
amtool alert query -o extended all labels
amtool silence add instance=web01 -d 2h -c 'disk swap' silence
amtool silence query active silences
amtool silence expire 6c91da38-1b58-4c6f-835e-3b504d18dd7b lift
amtool alert add alertname=Test severity=warning a test alert

Configuration

alertmanager.yml

A config with two webhook receivers: everything goes to chat, critical alerts to pager.

yaml

route:
  receiver: chat
  group_by: [alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="critical"
      receiver: pager
receivers:
  - name: chat
    webhook_configs:
      - url: http://127.0.0.1:5001/chat
  - name: pager
    webhook_configs:
      - url: http://127.0.0.1:5001/pager

The webhook gets a POST with JSON: receiver, status (firing or resolved), groupLabels, commonLabels and the list of alerts with the labels, annotations and status of each. For a webhook, send_resolved is on by default: when an alert is resolved, a notification about that arrives too.

Check and apply:

bash

amtool check-config /etc/prometheus/alertmanager.yml
sudo systemctl reload prometheus-alertmanager
journalctl -u prometheus-alertmanager -o cat | tail -3

check-config prints SUCCESS and what it found: the route, the number of receivers and inhibit rules. After reload the journal says Completed loading of configuration file.

Startup flags and the cluster

The service flags are the ARGS variable in /etc/default/prometheus-alertmanager (empty in the package). By default Alertmanager starts as a cluster node: it listens on port 9094 and looks for the machine's private address to announce to its peers. On a machine without one (the training VM has only lo) the service does not start (alertmanager). A single node without a cluster:

bash

# /etc/default/prometheus-alertmanager
ARGS="--cluster.listen-address="

Flags are read at start: after an edit, run sudo systemctl restart prometheus-alertmanager. curl -s localhost:9093/api/v2/status | jq .cluster.status then says "disabled".