At night a switch rebooted, and 50 web servers dropped off the network for a couple of minutes. The on-call engineer woke up to 50 chat messages in a row. This machine has the same: Alertmanager received 50 InstanceDown alerts in one request, and the training webhook, standing in for the chat here, wrote down everything it was sent — one line per notification. Here the lesson setup sends the alerts through the Alertmanager API, just as Prometheus does.
Count the notifications
Count how many notifications the training webhook wrote to /var/lib/lab-webhook/notifications.jsonl, and look at groupLabels of the first three.
First, predict
Alertmanager received 50 InstanceDown alerts in one request. How many notifications did the webhook write — one line per notification?
Now start the exercise, run the command and compare your prediction with the output.
50 alerts reached Alertmanager in one request, yet there are 50 notifications, each with its own instance in groupLabels. Why?
-
All 50 came in one request to the Alertmanager API. Grouping is what split them:
jq -c .groupLabelsshows each notification with its own label set. -
A line is a whole request from Alertmanager:
jq '.alerts | length'over the file prints 1 on every line, because each notification holds one alert. -
A resend would carry the same group labels, while here each line has its own instance.
sudo journalctl -u lab-webhookshows 50 different groups.
Each notification has its own groupLabels — every alert ended up as a separate group.