Observability: metrics, alerts, logs / Stage 5 of 9

Alertmanager: to whom and how many times

Not running Starting usually ~20 s

Step 5 of 6 · lab · 30 min

Alerts do not reach the receiver

Two instances of the online shop were down for twenty minutes last night. Prometheus saw it and the alert fired, yet the team in charge of the shop received not a single notification. Find where the alerts get lost and make the team learn about the instances that went down in one message.

Situation

The online shop runs on three instances of the application — the lab-shop@1, lab-shop@2 and lab-shop@3 services at 127.0.0.1:8001–8003. Prometheus scrapes their metrics, and when an instance stops answering, the ShopInstanceDown rule fires an alert with the team="web" and severity="critical" labels. Prometheus hands its alerts to Alertmanager on the same machine (/etc/prometheus/alertmanager.yml), which sends out the notifications.

The receivers are a training webhook at 127.0.0.1:5001; the path in the URL tells the receivers apart: /web is the web team's channel (the web-team receiver), /pager is the on-call engineer's phone, /ops is the common admin channel. Every notification is a JSON line in /var/lib/lab-webhook/notifications.jsonl.

Last night instances 2 and 3 were down for twenty minutes. The web team received nothing.

You can replay the night: sudo systemctl stop lab-shop@2 lab-shop@3, wait forty seconds or so and bring them back — sudo systemctl start lab-shop@2 lab-shop@3. amtool already knows the Alertmanager address. sudo works without a password.

Task

The ShopInstanceDown alerts with the team="web" and severity="critical" labels reach the web team — /web.
When two instances go down at once, the web team gets them in one notification — one group.
Other critical alerts still go to the on-call engineer — the pager receiver.
The settings are saved in /etc/prometheus/alertmanager.yml, Alertmanager has reloaded the file since the last edit, and amtool check-config passes. The Alertmanager service is enabled.

The check stops two shop instances itself, waits up to a minute for the notifications and starts the instances again. The ShopInstanceDown rule and the instances themselves need no fixing.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 4

The ShopInstanceDown alerts with the team="web" and severity="critical" labels reach the web team — /web of the training webhook — not done yet
When two instances go down at once, the web team gets them in one notification — one group — not done yet
Other critical alerts still go to the on-call engineer — the pager receiver — not done yet
The settings are saved in /etc/prometheus/alertmanager.yml, Alertmanager has reloaded the file since the last edit, amtool check-config passes; the service is enabled — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 4

The ShopInstanceDown alerts with the team="web" and severity="critical" labels reach the web team — /web of the training webhook — not done yet
When two instances go down at once, the web team gets them in one notification — one group — not done yet
Other critical alerts still go to the on-call engineer — the pager receiver — not done yet
The settings are saved in /etc/prometheus/alertmanager.yml, Alertmanager has reloaded the file since the last edit, amtool check-config passes; the service is enabled — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 4

The ShopInstanceDown alerts with the team="web" and severity="critical" labels reach the web team — /web of the training webhook — not done yet
When two instances go down at once, the web team gets them in one notification — one group — not done yet
Other critical alerts still go to the on-call engineer — the pager receiver — not done yet
The settings are saved in /etc/prometheus/alertmanager.yml, Alertmanager has reloaded the file since the last edit, amtool check-config passes; the service is enabled — not done yet

Observability · picked for this lab

Prometheus: metric collection and alerting rules

Cheat sheet

$_ — insert into the terminal

sudo systemctl enable --now prometheus turn it on
promtool check config /etc/prometheus/prometheus.yml validate
sudo systemctl reload prometheus reread the config
curl -s localhost:9090/-/ready is the server ready
promtool query instant http://localhost:9090 up what it collects
curl -s localhost:9090/api/v1/targets | jq .data targets, errors
curl -s localhost:9090/api/v1/rules | jq .data rules
curl -s localhost:9090/api/v1/alerts | jq .data alerts
promtool check rules /etc/prometheus/disk.rules.yml a rule file

prometheus.yml

The packaged config on the training machine, without comments:

yaml

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
      monitor: 'example'
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']
rule_files:
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    scrape_timeout: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']

Section	What it sets
`global`	default scrape and rule evaluation intervals; `external_labels` are labels every alert gets
`alerting`	the Alertmanager address
`rule_files`	rule files; the package leaves the list empty, so there are no rules
`scrape_configs`	what to collect: jobs and their targets

Jobs and targets

A job is a group of similar targets with shared scrape settings:

yaml

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100', 'node2:9100']
  - job_name: app
    scrape_interval: 5s
    metrics_path: /metrics
    static_configs:
      - targets: ['127.0.0.1:8000']
        labels:
          team: web

Prometheus adds the job label (the job name) and the instance label (the address from targets) to every series of a target; the labels from labels go on every series of that block's targets too: up{instance="127.0.0.1:8000", job="app", team="web"}. metrics_path defaults to /metrics, scrape_timeout to 10 seconds. An interval set in a job overrides the global one.

labels goes inside a static_configs item, next to targets. A job has no such field: promtool check config answers field labels not found in type config.ScrapeConfig. Labels from global.external_labels are something else: only data that leaves Prometheus gets them (alerts sent to Alertmanager, remote storage), and queries to Prometheus itself do not show them.

Reread the config

bash

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

reload sends the service a HUP signal, and Prometheus rereads the config and the rule files without a restart. The HTTP way, curl -X POST localhost:9090/-/reload, is off on the training machine: the answer is Lifecycle API is not enabled..

Whether the server accepted the config:

the journal, journalctl -u prometheus -o cat | tail -3, has either Completed loading of configuration file or Error reloading config;
the metric prometheus_config_last_reload_successful is 1 or 0 (it updates with the next scrape of the prometheus job, a few seconds later);
/api/v1/status/config and /api/v1/rules show what is actually loaded.

systemctl reload exits with code 0 even with a broken file: the signal was delivered, and whether to accept the config is up to Prometheus. A bad config is not applied on reload: the server keeps running with the old one. On restart with a bad config the service will not start.

bash

curl -s localhost:9090/metrics | grep '^prometheus_config_last_reload'

Startup flags

The service flags are the ARGS variable in /etc/default/prometheus, empty in the package. The ones needed most often:

Flag	What it sets
`--storage.tsdb.retention.time=30d`	how long to keep data (15 days by default)
`--web.enable-lifecycle`	turns on `/-/reload` and `/-/quit` over HTTP
`--web.listen-address=127.0.0.1:9090`	address and port (`0.0.0.0:9090` by default)

sudo systemctl restart prometheus applies the flags. Which flags the server is running with now: /api/v1/status/flags.

Add a target and make sure it is collected

bash

sudoedit /etc/prometheus/prometheus.yml   # the target in targets
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
sleep 20
promtool query instant http://localhost:9090 'up{job="app"}'

Wait at least one scrape interval. => 1 means the target is being collected, => 0 means Prometheus could not reach it, and the reason is in lastError.

Try a threshold before writing the rule

bash

promtool query instant http://localhost:9090 \
  'node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.05'

A rule expression is an ordinary query. An empty answer means the alert would not fire right now; series in the answer are the alerts that would fire, one per series.

Compare what is loaded with the file

bash

curl -s localhost:9090/api/v1/status/config \
  | jq -r .data.yaml | grep -A6 'job_name: app'
curl -s localhost:9090/api/v1/rules \
  | jq -r '.data.groups[].rules[].name'

The server prints the loaded config with default values filled in (honor_timestamps, scheme), so compare the fields you need rather than whole files.

A target is down

The target's lastError names the cause; what each error means is in the table of prometheus. Compare scrapeUrl with what the service listens on: ss -tlnp on the target machine and curl to the same address. More about the exporter: node-exporter, node-exporter and node-exporter.

Error reloading config

text

level=error msg="Error reloading config" err="couldn't load
  configuration (--config.file=\"/etc/prometheus/prometheus.yml\"):
  parsing YAML file
  /etc/prometheus/prometheus.yml: yaml: line 44: did not find expected
  ',' or ']'"

reload did not apply the new config; the server runs with the old one, and prometheus_config_last_reload_successful is 0. Fix the error, check with promtool check config and reload again.

error loading rules, previous rule set restored

A rule file has an error, and after reload the previous rules stay: /api/v1/rules shows the old set. What is wrong: promtool check rules <file>:

text

  FAILED:
/tmp/bad.yml: 5:15: group "demo", rule 1, "DiskAlmostFull": could not
  parse expression: 1:47: parse error: unexpected <op:<>

The service does not start after a restart

text

level=error msg="Error loading config (--config.file=…)"
  err="parsing YAML file /etc/prometheus/prometheus.yml: …"

Prometheus does not start with a bad config, and systemctl is-active says failed. Fix it, check with promtool check config, then run sudo systemctl reset-failed prometheus and start.

field … not found in type config.plain

text

  FAILED: parsing YAML file /tmp/p.yml: yaml: unmarshal errors:
  line 26: field scrape_config not found in type config.plain

An unknown key: here scrape_config instead of scrape_configs. Other common typos are static_config, target, and job instead of job_name. The key can also be right but at the wrong level: field labels not found in type config.ScrapeConfig means labels was written in the job, while it belongs inside static_configs (prometheus).

did not find expected ',' or ']'

A YAML syntax error: an unclosed bracket in targets: ['…', a quote or an indent. The line number is in the message.

The rule is in the file, but /api/v1/rules is empty

the file is not in rule_files; the packaged config leaves that list empty;
the path is wrong: a relative path counts from the directory of prometheus.yml, and promtool check config tells how many rule files it found (SUCCESS: 1 rule files found);
there was no reload after the edit, or it failed, see prometheus.

The alert does not fire

the rule is loaded and its state is inactive: run the expr with promtool query instant. Empty means the condition is false right now: a reversed sign (> instead of <), a threshold in percent against a ratio, or a label with a typo;
the alert is stuck in pending: for has not passed yet, or the condition comes and goes and the countdown starts over;
the alert is firing but no notification arrives: continue with alertmanager.

Lifecycle API is not enabled

The answer to curl -X POST localhost:9090/-/reload: Prometheus runs without --web.enable-lifecycle. Reload with sudo systemctl reload prometheus.

query error: connection refused

text

query error: Post "http://localhost:9090/api/v1/query": dial tcp
  127.0.0.1:9090: connect: connection refused

promtool query could not reach the server: Prometheus is not running, see systemctl status prometheus.

Restart with an unchecked config

A reload with an error keeps the old config; a restart with an error leaves the machine without monitoring: the service does not come up, metrics are not collected, alerts are not evaluated. Before any restart, run promtool check config.

Alertmanager: to whom and how many times

Situation

Task

Signing in is needed to start the machine

Add a target and make sure it is collected

Try a threshold before writing the rule

Compare what is loaded with the file

A target is down

Error reloading config

error loading rules, previous rule set restored

The service does not start after a restart

field … not found in type config.plain

did not find expected ',' or ']'

The rule is in the file, but /api/v1/rules is empty

The alert does not fire

Lifecycle API is not enabled

query error: connection refused

Restart with an unchecked config