Observability: metrics, alerts, logs / Final lab

The second machine is unwatched

~75 min

Not running Starting usually ~20 s

Situation

A week ago the second machine, node2, was put under node1's watch and the ticket was closed. node1 runs Prometheus, Alertmanager and an rsyslog log collector. Alertmanager sends notifications for the on-call team to the lab-webhook training webhook: it appends every notification it receives as a line to /var/lib/lab-webhook/notifications.jsonl.

Yesterday node2 was down for half an hour, and nothing reached the on-call team. Digging in showed more: Prometheus on node1 does not see node2's metrics, and there are no fresh node2 logs on the collector.

node1 is your terminal. Run commands on node2 over SSH: ssh node2 'command', or enter its shell with ssh node2 and leave with exit. sudo works without a password on both machines. There is no browser to the machines: look at Prometheus and Alertmanager from the terminal — curl -s localhost:9090/api/v1/… | jq, promtool query instant http://localhost:9090 '<expression>', amtool alert.

Task

Prometheus on node1 scrapes node2: the node2:9100 target in the node job is up.
When node2 stops answering, the NodeDown alert about it (instance="node2:9100") becomes firing within a minute. While node2 works, NodeDown about it stays quiet.
The notification about a firing NodeDown reaches the on-call team — as a new line in /var/lib/lab-webhook/notifications.jsonl.
node2's logs are collected on node1: a line sent on node2 with logger shows up in a file in the /var/log/remote/node2/ directory on node1.

"Check" briefly stops the exporter on node2 (prometheus-node-exporter) by itself, sends a logger -t lab-verify … line from node2 and puts everything back. The check waits for the alert and the line and takes up to a minute and a half.

Constraints

Everything must survive a reboot of both machines: the services are enabled, the settings live in files, and the running services have re-read those files after your edits.
Do not turn off the firewall on node2 — open in it only what monitoring needs. Do not close SSH from node1 to node2: there is no other way into node2.
Leave the lab-webhook training webhook and its file alone.

There is no internet in the lab — only the network between node1 and node2, and everything you need is installed. Follow each chain from start to end and after every edit look at the result the way the check sees it: targets — curl -s localhost:9090/api/v1/targets | jq, alerts — curl -s localhost:9090/api/v1/alerts | jq and amtool alert, notifications — tail /var/lib/lab-webhook/notifications.jsonl, logs — logger on node2 and sudo ls -R /var/log/remote/ on node1.

Lab goals

0 of 4

Prometheus on node1 scrapes node2 — the node2:9100 target in the node job is up; node2's firewall stays on, and all of it survives a reboot — not checked yet
When node2 stops answering, the NodeDown alert about it is firing within a minute, and while node2 works it stays quiet; the rule is loaded from its file — not checked yet
The notification about a firing NodeDown reaches the on-call team — the training webhook on node1; Prometheus and Alertmanager run with the settings from their files — not checked yet
A logger line from node2 shows up on node1 in /var/log/remote/node2/; rsyslog on both machines runs with the settings from its files and starts on its own — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Observability · picked for this lab

Prometheus: metric collection and alerting rules

Cheat sheet

$_ — insert into the terminal

sudo systemctl enable --now prometheus turn it on
promtool check config /etc/prometheus/prometheus.yml validate
sudo systemctl reload prometheus reread the config
curl -s localhost:9090/-/ready is the server ready
promtool query instant http://localhost:9090 up what it collects
curl -s localhost:9090/api/v1/targets | jq .data targets, errors
curl -s localhost:9090/api/v1/rules | jq .data rules
curl -s localhost:9090/api/v1/alerts | jq .data alerts
promtool check rules /etc/prometheus/disk.rules.yml a rule file

prometheus.yml

The packaged config on the training machine, without comments:

yaml

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
      monitor: 'example'
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']
rule_files:
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    scrape_timeout: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']

Section	What it sets
`global`	default scrape and rule evaluation intervals; `external_labels` are labels every alert gets
`alerting`	the Alertmanager address
`rule_files`	rule files; the package leaves the list empty, so there are no rules
`scrape_configs`	what to collect: jobs and their targets

Jobs and targets

A job is a group of similar targets with shared scrape settings:

yaml

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100', 'node2:9100']
  - job_name: app
    scrape_interval: 5s
    metrics_path: /metrics
    static_configs:
      - targets: ['127.0.0.1:8000']
        labels:
          team: web

Prometheus adds the job label (the job name) and the instance label (the address from targets) to every series of a target; the labels from labels go on every series of that block's targets too: up{instance="127.0.0.1:8000", job="app", team="web"}. metrics_path defaults to /metrics, scrape_timeout to 10 seconds. An interval set in a job overrides the global one.

labels goes inside a static_configs item, next to targets. A job has no such field: promtool check config answers field labels not found in type config.ScrapeConfig. Labels from global.external_labels are something else: only data that leaves Prometheus gets them (alerts sent to Alertmanager, remote storage), and queries to Prometheus itself do not show them.

Reread the config

bash

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

reload sends the service a HUP signal, and Prometheus rereads the config and the rule files without a restart. The HTTP way, curl -X POST localhost:9090/-/reload, is off on the training machine: the answer is Lifecycle API is not enabled..

Whether the server accepted the config:

the journal, journalctl -u prometheus -o cat | tail -3, has either Completed loading of configuration file or Error reloading config;
the metric prometheus_config_last_reload_successful is 1 or 0 (it updates with the next scrape of the prometheus job, a few seconds later);
/api/v1/status/config and /api/v1/rules show what is actually loaded.

systemctl reload exits with code 0 even with a broken file: the signal was delivered, and whether to accept the config is up to Prometheus. A bad config is not applied on reload: the server keeps running with the old one. On restart with a bad config the service will not start.

bash

curl -s localhost:9090/metrics | grep '^prometheus_config_last_reload'

Startup flags

The service flags are the ARGS variable in /etc/default/prometheus, empty in the package. The ones needed most often:

Flag	What it sets
`--storage.tsdb.retention.time=30d`	how long to keep data (15 days by default)
`--web.enable-lifecycle`	turns on `/-/reload` and `/-/quit` over HTTP
`--web.listen-address=127.0.0.1:9090`	address and port (`0.0.0.0:9090` by default)

sudo systemctl restart prometheus applies the flags. Which flags the server is running with now: /api/v1/status/flags.

Add a target and make sure it is collected

bash

sudoedit /etc/prometheus/prometheus.yml   # the target in targets
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
sleep 20
promtool query instant http://localhost:9090 'up{job="app"}'

Wait at least one scrape interval. => 1 means the target is being collected, => 0 means Prometheus could not reach it, and the reason is in lastError.

Try a threshold before writing the rule

bash

promtool query instant http://localhost:9090 \
  'node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.05'

A rule expression is an ordinary query. An empty answer means the alert would not fire right now; series in the answer are the alerts that would fire, one per series.

Compare what is loaded with the file

bash

curl -s localhost:9090/api/v1/status/config \
  | jq -r .data.yaml | grep -A6 'job_name: app'
curl -s localhost:9090/api/v1/rules \
  | jq -r '.data.groups[].rules[].name'

The server prints the loaded config with default values filled in (honor_timestamps, scheme), so compare the fields you need rather than whole files.

A target is down

The target's lastError names the cause; what each error means is in the table of prometheus. Compare scrapeUrl with what the service listens on: ss -tlnp on the target machine and curl to the same address. More about the exporter: node-exporter, node-exporter and node-exporter.

Error reloading config

text

level=error msg="Error reloading config" err="couldn't load
  configuration (--config.file=\"/etc/prometheus/prometheus.yml\"):
  parsing YAML file
  /etc/prometheus/prometheus.yml: yaml: line 44: did not find expected
  ',' or ']'"

reload did not apply the new config; the server runs with the old one, and prometheus_config_last_reload_successful is 0. Fix the error, check with promtool check config and reload again.

error loading rules, previous rule set restored

A rule file has an error, and after reload the previous rules stay: /api/v1/rules shows the old set. What is wrong: promtool check rules <file>:

text

  FAILED:
/tmp/bad.yml: 5:15: group "demo", rule 1, "DiskAlmostFull": could not
  parse expression: 1:47: parse error: unexpected <op:<>

The service does not start after a restart

text

level=error msg="Error loading config (--config.file=…)"
  err="parsing YAML file /etc/prometheus/prometheus.yml: …"

Prometheus does not start with a bad config, and systemctl is-active says failed. Fix it, check with promtool check config, then run sudo systemctl reset-failed prometheus and start.

field … not found in type config.plain

text

  FAILED: parsing YAML file /tmp/p.yml: yaml: unmarshal errors:
  line 26: field scrape_config not found in type config.plain

An unknown key: here scrape_config instead of scrape_configs. Other common typos are static_config, target, and job instead of job_name. The key can also be right but at the wrong level: field labels not found in type config.ScrapeConfig means labels was written in the job, while it belongs inside static_configs (prometheus).

did not find expected ',' or ']'

A YAML syntax error: an unclosed bracket in targets: ['…', a quote or an indent. The line number is in the message.

The rule is in the file, but /api/v1/rules is empty

the file is not in rule_files; the packaged config leaves that list empty;
the path is wrong: a relative path counts from the directory of prometheus.yml, and promtool check config tells how many rule files it found (SUCCESS: 1 rule files found);
there was no reload after the edit, or it failed, see prometheus.

The alert does not fire

the rule is loaded and its state is inactive: run the expr with promtool query instant. Empty means the condition is false right now: a reversed sign (> instead of <), a threshold in percent against a ratio, or a label with a typo;
the alert is stuck in pending: for has not passed yet, or the condition comes and goes and the countdown starts over;
the alert is firing but no notification arrives: continue with alertmanager.

Lifecycle API is not enabled

The answer to curl -X POST localhost:9090/-/reload: Prometheus runs without --web.enable-lifecycle. Reload with sudo systemctl reload prometheus.

query error: connection refused

text

query error: Post "http://localhost:9090/api/v1/query": dial tcp
  127.0.0.1:9090: connect: connection refused

promtool query could not reach the server: Prometheus is not running, see systemctl status prometheus.

Restart with an unchecked config

A reload with an error keeps the old config; a restart with an error leaves the machine without monitoring: the service does not come up, metrics are not collected, alerts are not evaluated. Before any restart, run promtool check config.

The second machine is unwatched

Situation

Task

Constraints

Signing in is needed to start the machine

Add a target and make sure it is collected

Try a threshold before writing the rule

Compare what is loaded with the file

A target is down

Error reloading config

error loading rules, previous rule set restored

The service does not start after a restart

field … not found in type config.plain

did not find expected ',' or ']'

The rule is in the file, but /api/v1/rules is empty

The alert does not fire

Lifecycle API is not enabled

query error: connection refused

Restart with an unchecked config