Linux Labs

Observability: metrics, alerts, logs / Stage 5 of 9

Alertmanager: to whom and how many times

Not running Starting usually ~20 s
Leave the lesson

Step 4 of 6 · on a machine · 7 min

Whom to send to: routes

One receiver for everything is noise for the on-call engineer. A routing tree splits alerts between receivers by their labels.

The root route takes every alert. Inside it, routes is a list of child routes: each has matchers — conditions on labels — and its own receiver. Alertmanager checks the child routes in order and hands the alert to the first one that matches; if none does, the alert stays with the root receiver. continue: true on a route lets the search go on after a match.

yaml
route:
  receiver: chat
  routes:
    - matchers: [team="db"]
      receiver: dba
receivers:
  - name: chat
    # …
  - name: dba
    webhook_configs:
      - url: http://127.0.0.1:5001/dba

A child route takes group_by and the timers from its parent unless it sets its own. amtool config routes test --config.file=<file> <label>=<value> prints where an alert with the given labels will go — the receiver name, sending nothing. The Warning: --config.file flag overrides the --alertmanager.url line before it means the file is checked: the command does not ask the running service.

Exercise · do it

Send the critical ones to the on-call

Right now every alert goes to the chat. Make the alerts labelled severity=critical go to the on-call — a pager receiver with the address http://127.0.0.1:5001/pager — while the rest still go to chat. Check the route with amtool and make Alertmanager reread its configuration.

Sign in and start The machine is available after signing in

Goals · 0 of 2

  • Describe the pager receiver and a route to it not done yet
  • Deliver the critical alerts to pager not done yet

Step goals

0 of 2
  • Describe the pager receiver and a route to it — not done yet
  • Deliver the critical alerts to pager — not done yet
Ubuntu 24.04

Step goals

0 of 2
  • Describe the pager receiver and a route to it — not done yet
  • Deliver the critical alerts to pager — not done yet
Loading…