Alertmanager: who gets alerts and how often

Alertmanager is the service that receives alerts from Prometheus and decides whom to tell, when and how many times. Prometheus decides by a rule that an alert is firing (prometheus) and sends it to Alertmanager. From there Alertmanager:

1. groups alerts by labels, so one event gives one notification rather than one per machine;
2. picks a receiver through the routing tree;
3. skips alerts that are silenced or inhibited by another alert;
4. sends the notification (an email, a webhook, a chat message) and repeats it while the alert keeps firing.

Prometheus ──alerts──▶ Alertmanager :9093 ──notifications──▶ receivers
 (rules)               groups, routes,                    (webhook,
                       silences, inhibition                email…)

In Ubuntu this is the prometheus-alertmanager package: a service of the same name, the config /etc/prometheus/alertmanager.yml, and data (silences and the log of what was sent) in /var/lib/prometheus/alertmanager. Port 9093 is the web UI and the API, 9094 links cluster nodes. The amtool utility comes in the same package. On the training machine the package is installed and the service is off, and the packaged config is an example with email and PagerDuty: with no network it is useless, so you write your own.

The path of an alert

Where Prometheus sends alerts is the alerting section of prometheus.yml; the package sets localhost:9093. Whether Prometheus has found Alertmanager:

$ curl -s localhost:9090/api/v1/alertmanagers | jq -c .data
{"activeAlertmanagers":[{"url":"http://localhost:9093/api/v2/alerts"}],
  "droppedAlertmanagers":[]}

Prometheus builds the sending address from scheme (http by default), the address from targets, path_prefix (empty by default) and /api/v2/alerts; the full address is the url in the answer above. An extra path_prefix, such as /alertmanager copied from a server behind a proxy, gives an address Alertmanager does not have (alertmanager).

Prometheus sends only alerts in the firing state and keeps resending them while they fire; once the condition no longer holds, it sends the alert once more with an end time, and Alertmanager marks it resolved. Prometheus adds the external_labels from its config to the alert's labels: the packaged config has monitor: 'example', so every alert carries monitor="example".

An alert in Alertmanager is labels (they tell alerts apart, group them and pick the route), annotations (text: summary, description) and a start and end time.

Grouping and intervals

group_by lists labels: alerts with the same values of these labels fall into one group, and a group is one notification with a list of alerts. On the training machine, three InstanceDown alerts for web01, web02 and web03:

Without group_by all alerts of the route are one group; group_by: ['...'] puts every alert on its own.

Fifty notifications for one event almost always come from grouping: group_by has a label that is unique to each alert (instance, pod, path), or '...'. The same notification every few minutes means repeat_interval is too short.

Routes

Routes form a tree. The root route takes every alert and must have a receiver. Child routes under routes are checked top to bottom; the alert goes to the first one whose matchers fit and goes deeper if that one has routes of its own. When none fits, the parent's receiver gets the notification. continue: true on a route means "check the following ones too": that is how one notification reaches several receivers. A child route inherits the parent's group_by and intervals unless it sets its own; a group_by it sets replaces the parent's entirely, the labels do not add up.

Critical alerts to both the on-call person and the team:

route:
  receiver: team
  routes:
    - matchers: [severity="critical"]
      receiver: oncall
      continue: true
    - matchers: [team="web"]
      receiver: team-web

Without continue: true, a critical alert of the web team would go to the on-call person only: checking would stop at the first route that matched.

Conditions in matchers look like promql selectors: severity="critical", team!="db", severity=~"critical|page". Every condition in the list has to hold. The old match and match_re forms are still read.

route:
  receiver: chat
  group_by: [alertname]
  routes:
    - matchers:
        - severity="critical"
      receiver: pager

amtool draws the tree and tells where an alert with given labels will go (the --config.file flag is required, see alertmanager):

$ amtool config routes show \
    --config.file=/etc/prometheus/alertmanager.yml
Routing tree:
.
└── default-route  receiver: chat
    └── {severity="critical"}  receiver: pager

$ amtool config routes test \
    --config.file=/etc/prometheus/alertmanager.yml severity=critical
pager

Silences and inhibition

A silence is a temporary ban on notifications for alerts whose labels match its conditions. A person sets it, with amtool silence add or the web UI, with a comment and a duration. A silenced alert stays in the list with the suppressed state, and no notifications go out for it. Silences are stored in /var/lib/prometheus/alertmanager and survive a restart.

Inhibition is a rule in the config: while a source alert fires, target alerts with the same values of the equal labels are not sent. When a machine is down, there is no point in a separate message that its disk is filling up:

inhibit_rules:
  - source_matchers: [alertname="NodeDown"]
    target_matchers: [severity="warning"]
    equal: [instance]

On the training machine: a silence on alertname=DiskFull instance=web02 means no notification about web02 while web04 still gets one; NodeDown on web03 puts DiskFull for the same web03 into the suppressed state.

amtool

amtool is Alertmanager from the terminal: it lists alerts, sets and lifts silences, checks the config and the routes. The alert and silence commands go to the Alertmanager API. On the training machine amtool goes to http://localhost:9093 by default; another address is --alertmanager.url=… or a line in /etc/amtool/config.yml (for everyone) or ~/.config/amtool/config.yml (for you):

alertmanager.url: http://localhost:9093

Alerts:

$ amtool alert
Alertname  Starts At                Summary      State
DiskFull   2026-10-22 04:44:53 UTC  disk is low  active
DiskFull   2026-10-22 04:44:53 UTC               active

Summary is the summary annotation. Silenced and inhibited alerts are not in the amtool alert output; -s (--silenced) and -i (--inhibited) show them. amtool alert query instance=web01 filters by labels, -o extended prints every label and annotation, -s shows silenced alerts, -i inhibited ones, -o json gives everything as JSON for jq.

Silences:

$ amtool silence add alertname=DiskFull instance=web01 \
    -d 30m -c 'disk swap'
6c91da38-1b58-4c6f-835e-3b504d18dd7b
$ amtool silence query
ID                                    Matchers                       …
6c91da38-1b58-4c6f-835e-3b504d18dd7b  alertname="DiskFull" instance=…

The silence query output is cut here: the Ends At, Created By and Comment columns follow. The command prints the new silence's ID; amtool silence expire <ID> lifts it, and amtool silence query -q lists the IDs of all silences. -d is the duration (30m, 2h), -c the comment; without one the silence is not set.

A test alert without Prometheus, to check the route and the receiver:

amtool alert add alertname=Test severity=warning \
  --annotation=summary='delivery check'

• prometheus — the rules that produce alerts.
• promql — rule expressions and the syntax of label conditions.
• node-exporter — machine metrics that rules are written on.
• grafana — Grafana's own alerts and notifications.
• jq — parse amtool alert query -o json and the webhook body.
• curl — the Alertmanager API from the terminal.
• systemctl — reload, restart, reset-failed.
• journalctl — the service journal: config and delivery errors.