One receiver for everything is noise for the on-call engineer. A routing tree splits alerts between receivers by their labels.
The root route takes every alert. Inside it, routes is a list of child routes: each has matchers — conditions on labels — and its own receiver. Alertmanager checks the child routes in order and hands the alert to the first one that matches; if none does, the alert stays with the root receiver. continue: true on a route lets the search go on after a match.
A child route takes group_by and the timers from its parent unless it sets its own. amtool config routes test --config.file=<file> <label>=<value> prints where an alert with the given labels will go — the receiver name, sending nothing. The Warning: --config.file flag overrides the --alertmanager.url line before it means the file is checked: the command does not ask the running service.
Send the critical ones to the on-call
Right now every alert goes to the chat. Make the alerts labelled severity=critical go to the on-call — a pager receiver with the address http://127.0.0.1:5001/pager — while the rest still go to chat. Check the route with amtool and make Alertmanager reread its configuration.
Goals · 0 of 2
- Describe the pager receiver and a route to it not done yet
- Deliver the critical alerts to pager not done yet