Observability: metrics, alerts, logs / Stage 1 of 9 · step 6 of 7

A machine dropped out of monitoring

~20 min

Not running Starting usually ~20 s

Situation

This server is watched by the office monitoring. Prometheus runs right here: every 5 seconds it scrapes the metrics of the machine itself — CPU, memory, disks, network — from the node_exporter exporter. In the Prometheus configuration this is the node job.

Since last night the machine's graphs have been empty, and the on-call summary shows its up as 0. Meanwhile Prometheus keeps scraping its own metrics just fine.

There is no browser to the machine, so look at Prometheus from the terminal: queries — promtool query instant http://localhost:9090 '<query>', its API — curl -s localhost:9090/api/v1/… and jq. sudo works without a password.

Task

The node job's target in Prometheus is up: promtool query instant http://localhost:9090 'up{job="node"}' prints => 1.
node_exporter runs as the prometheus-node-exporter service; the service is enabled and listens on port 9100.
The settings are saved in files: promtool check config /etc/prometheus/prometheus.yml passes, Prometheus runs with what the file says, the prometheus service is enabled — after the services restart the machine stays in monitoring.

Constraints

node_exporter listens on port 9100 on every office server — that is how the other Prometheus servers find the machine.
The node job stays in /etc/prometheus/prometheus.yml under its own name.

What Prometheus knows about each of its targets — the scrape address, the state and the last error — is shown by curl -s localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {scrapeUrl, health, lastError}'

Lab goals

0 of 3

The node job's target in Prometheus is up — not checked yet
node_exporter runs as the prometheus-node-exporter service, is enabled and listens on port 9100 — not checked yet
The settings live in files; promtool check config passes, Prometheus runs with what is in prometheus.yml, the prometheus service is enabled — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Observability · picked for this lab

Prometheus: metric collection and alerting rules

Cheat sheet

$_ — insert into the terminal

sudo systemctl enable --now prometheus turn it on
promtool check config /etc/prometheus/prometheus.yml validate
sudo systemctl reload prometheus reread the config
curl -s localhost:9090/-/ready is the server ready
promtool query instant http://localhost:9090 up what it collects
curl -s localhost:9090/api/v1/targets | jq .data targets, errors
curl -s localhost:9090/api/v1/rules | jq .data rules
curl -s localhost:9090/api/v1/alerts | jq .data alerts
promtool check rules /etc/prometheus/disk.rules.yml a rule file

prometheus.yml

The packaged config on the training machine, without comments:

yaml

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
      monitor: 'example'
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']
rule_files:
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    scrape_timeout: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']

Section	What it sets
`global`	default scrape and rule evaluation intervals; `external_labels` are labels every alert gets
`alerting`	the Alertmanager address
`rule_files`	rule files; the package leaves the list empty, so there are no rules
`scrape_configs`	what to collect: jobs and their targets

Jobs and targets

A job is a group of similar targets with shared scrape settings:

yaml

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100', 'node2:9100']
  - job_name: app
    scrape_interval: 5s
    metrics_path: /metrics
    static_configs:
      - targets: ['127.0.0.1:8000']
        labels:
          team: web

Prometheus adds the job label (the job name) and the instance label (the address from targets) to every series of a target; the labels from labels go on every series of that block's targets too: up{instance="127.0.0.1:8000", job="app", team="web"}. metrics_path defaults to /metrics, scrape_timeout to 10 seconds. An interval set in a job overrides the global one.

labels goes inside a static_configs item, next to targets. A job has no such field: promtool check config answers field labels not found in type config.ScrapeConfig. Labels from global.external_labels are something else: only data that leaves Prometheus gets them (alerts sent to Alertmanager, remote storage), and queries to Prometheus itself do not show them.

Reread the config

bash

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

reload sends the service a HUP signal, and Prometheus rereads the config and the rule files without a restart. The HTTP way, curl -X POST localhost:9090/-/reload, is off on the training machine: the answer is Lifecycle API is not enabled..

Whether the server accepted the config:

the journal, journalctl -u prometheus -o cat | tail -3, has either Completed loading of configuration file or Error reloading config;
the metric prometheus_config_last_reload_successful is 1 or 0 (it updates with the next scrape of the prometheus job, a few seconds later);
/api/v1/status/config and /api/v1/rules show what is actually loaded.

systemctl reload exits with code 0 even with a broken file: the signal was delivered, and whether to accept the config is up to Prometheus. A bad config is not applied on reload: the server keeps running with the old one. On restart with a bad config the service will not start.

bash

curl -s localhost:9090/metrics | grep '^prometheus_config_last_reload'

Startup flags

The service flags are the ARGS variable in /etc/default/prometheus, empty in the package. The ones needed most often:

Flag	What it sets
`--storage.tsdb.retention.time=30d`	how long to keep data (15 days by default)
`--web.enable-lifecycle`	turns on `/-/reload` and `/-/quit` over HTTP
`--web.listen-address=127.0.0.1:9090`	address and port (`0.0.0.0:9090` by default)

sudo systemctl restart prometheus applies the flags. Which flags the server is running with now: /api/v1/status/flags.

Add a target and make sure it is collected

bash

sudoedit /etc/prometheus/prometheus.yml   # the target in targets
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
sleep 20
promtool query instant http://localhost:9090 'up{job="app"}'

Wait at least one scrape interval. => 1 means the target is being collected, => 0 means Prometheus could not reach it, and the reason is in lastError.

Try a threshold before writing the rule

bash

promtool query instant http://localhost:9090 \
  'node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.05'

A rule expression is an ordinary query. An empty answer means the alert would not fire right now; series in the answer are the alerts that would fire, one per series.

Compare what is loaded with the file

bash

curl -s localhost:9090/api/v1/status/config \
  | jq -r .data.yaml | grep -A6 'job_name: app'
curl -s localhost:9090/api/v1/rules \
  | jq -r '.data.groups[].rules[].name'

The server prints the loaded config with default values filled in (honor_timestamps, scheme), so compare the fields you need rather than whole files.

A target is down

The target's lastError names the cause; what each error means is in the table of prometheus. Compare scrapeUrl with what the service listens on: ss -tlnp on the target machine and curl to the same address. More about the exporter: node-exporter, node-exporter and node-exporter.

Error reloading config

text

level=error msg="Error reloading config" err="couldn't load
  configuration (--config.file=\"/etc/prometheus/prometheus.yml\"):
  parsing YAML file
  /etc/prometheus/prometheus.yml: yaml: line 44: did not find expected
  ',' or ']'"

reload did not apply the new config; the server runs with the old one, and prometheus_config_last_reload_successful is 0. Fix the error, check with promtool check config and reload again.

error loading rules, previous rule set restored

A rule file has an error, and after reload the previous rules stay: /api/v1/rules shows the old set. What is wrong: promtool check rules <file>:

text

  FAILED:
/tmp/bad.yml: 5:15: group "demo", rule 1, "DiskAlmostFull": could not
  parse expression: 1:47: parse error: unexpected <op:<>

The service does not start after a restart

text

level=error msg="Error loading config (--config.file=…)"
  err="parsing YAML file /etc/prometheus/prometheus.yml: …"

Prometheus does not start with a bad config, and systemctl is-active says failed. Fix it, check with promtool check config, then run sudo systemctl reset-failed prometheus and start.

field … not found in type config.plain

text

  FAILED: parsing YAML file /tmp/p.yml: yaml: unmarshal errors:
  line 26: field scrape_config not found in type config.plain

An unknown key: here scrape_config instead of scrape_configs. Other common typos are static_config, target, and job instead of job_name. The key can also be right but at the wrong level: field labels not found in type config.ScrapeConfig means labels was written in the job, while it belongs inside static_configs (prometheus).

did not find expected ',' or ']'

A YAML syntax error: an unclosed bracket in targets: ['…', a quote or an indent. The line number is in the message.

The rule is in the file, but /api/v1/rules is empty

the file is not in rule_files; the packaged config leaves that list empty;
the path is wrong: a relative path counts from the directory of prometheus.yml, and promtool check config tells how many rule files it found (SUCCESS: 1 rule files found);
there was no reload after the edit, or it failed, see prometheus.

The alert does not fire

the rule is loaded and its state is inactive: run the expr with promtool query instant. Empty means the condition is false right now: a reversed sign (> instead of <), a threshold in percent against a ratio, or a label with a typo;
the alert is stuck in pending: for has not passed yet, or the condition comes and goes and the countdown starts over;
the alert is firing but no notification arrives: continue with alertmanager.

Lifecycle API is not enabled

The answer to curl -X POST localhost:9090/-/reload: Prometheus runs without --web.enable-lifecycle. Reload with sudo systemctl reload prometheus.

query error: connection refused

text

query error: Post "http://localhost:9090/api/v1/query": dial tcp
  127.0.0.1:9090: connect: connection refused

promtool query could not reach the server: Prometheus is not running, see systemctl status prometheus.

Restart with an unchecked config

A reload with an error keeps the old config; a restart with an error leaves the machine without monitoring: the service does not come up, metrics are not collected, alerts are not evaluated. Before any restart, run promtool check config.

A machine dropped out of monitoring

Situation

Task

Constraints

Signing in is needed to start the machine

Add a target and make sure it is collected

Try a threshold before writing the rule

Compare what is loaded with the file

A target is down

Error reloading config

error loading rules, previous rule set restored

The service does not start after a restart

field … not found in type config.plain

did not find expected ',' or ']'

The rule is in the file, but /api/v1/rules is empty

The alert does not fire

Lifecycle API is not enabled

query error: connection refused

Restart with an unchecked config