A group and three timers

Alertmanager sends a notification per group of alerts, and group_by decides what ends up in one group.

Prometheus decides that an alert fires and hands it to Alertmanager — a service that decides whom to tell, when and how many times. Its configuration is /etc/prometheus/alertmanager.yml:

yaml

route:                  # the root route
  receiver: chat
  group_by: ['...']
  group_wait: 5s
  group_interval: 10s
  repeat_interval: 4h
receivers:              # receivers
  - name: chat
    webhook_configs:
      - url: http://127.0.0.1:5001/chat

A receiver is a name plus delivery addresses: a webhook, mail, a messenger. group_by is a list of labels: alerts with equal values of these labels go into one group, and one notification with all its alerts goes out per group. ['...'] means "all labels" — every alert is a group of its own, hence 50 notifications. To keep the alerts of one event together, group_by keeps the labels they share and leaves out the ones that tell them apart, such as instance.

Three timers set when to send:

Timer	What it sets
`group_wait`	how long a new group waits for neighbours before the first notification
`group_interval`	how long until reporting that new alerts joined the group
`repeat_interval`	how long until repeating a notification about an unchanged group

The defaults are 30 seconds, 5 minutes and 4 hours; the first two are shortened here to save waiting. amtool check-config <file> checks the file, sudo systemctl reload prometheus-alertmanager rereads it — after that Alertmanager builds the groups anew.

Exercise · do it

Fold the event into one notification

Make the 50 InstanceDown alerts reach the webhook as one notification: fix the grouping in /etc/prometheus/alertmanager.yml (with sudo), check the file with amtool and make Alertmanager reread its configuration.

Goals · 0 of 2

Fix the grouping in alertmanager.yml not done yet
Get one notification for all 50 alerts not done yet

Check yourself 1 question

All 50 InstanceDown alerts are now in one group, and the notification went out. The hosts stay down, the alerts do not change. When will the webhook get the next notification about this group?

Alertmanager reminds you of a long-running problem: it repeats an unchanged group once per repeat_interval, and sends resolved when the alerts go away.
group_wait is waited out before the first notification of a new group, and this group has already been sent.
group_interval is waited out before a message about changes in the group, and nothing changes here. An unchanged group waits for repeat_interval.

Alertmanager repeats an unchanged group once per repeat_interval — here once every 4 hours.

Alertmanager: to whom and how many times