Observability: metrics, alerts, logs / Stage 2 of 9 · step 5 of 6

The app's metrics are not in Prometheus

~20 min

Not running Starting usually ~20 s

Situation

This machine runs a shop: the lab-app application behind nginx, the site is http://shop.lab.test/. The office Prometheus runs here too: every 5 seconds it scrapes its own metrics (the prometheus job) and the machine's (the node job).

The application serves its metrics in the Prometheus format. The web team wants to see them in Prometheus with the label team="web" — its graphs and queries are built on that label. Yesterday a colleague took this on and wrote before leaving: "Added the job, reloaded Prometheus." Yet promtool query instant http://localhost:9090 'up{team="web"}' prints nothing.

There is no browser to the machine: queries — promtool query instant http://localhost:9090 '<query>', the Prometheus API — curl -s localhost:9090/api/v1/… and jq. sudo works without a password.

Task

Prometheus scrapes the metrics of the shop application itself: its target has the label team="web", the target is up, and promtool query instant http://localhost:9090 'labapp_build_info{team="web"}' prints a series.
The shop site works as before: http://shop.lab.test/ answers 200, and http://shop.lab.test/metrics answers 403.
The setting lives in /etc/prometheus/prometheus.yml: promtool check config passes, Prometheus runs with what the file says; the prometheus and node jobs are in place and working.

Constraints

The application's metrics are not served to the outside: the shop's nginx closes /metrics on purpose — do not change its configuration.
Do not change the lab-app application or its service.

Half a minute after the metrics start flowing, the web team will see its requests by response code like this: promtool query instant http://localhost:9090 'sum by (code) (rate(labapp_http_requests_total{team="web"}[1m]))'

Lab goals

0 of 3

Prometheus scrapes the shop app's metrics — the target with the label team="web" is up, and labapp_* series with this label exist — not checked yet
The shop site works as before — / answers 200, and /metrics stays closed to the outside (403) — not checked yet
The setting lives in prometheus.yml; promtool check config passes, Prometheus runs with what the file says; the prometheus and node jobs are in place — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Observability · picked for this lab

Prometheus: metric collection and alerting rules

Cheat sheet

$_ — insert into the terminal

sudo systemctl enable --now prometheus turn it on
promtool check config /etc/prometheus/prometheus.yml validate
sudo systemctl reload prometheus reread the config
curl -s localhost:9090/-/ready is the server ready
promtool query instant http://localhost:9090 up what it collects
curl -s localhost:9090/api/v1/targets | jq .data targets, errors
curl -s localhost:9090/api/v1/rules | jq .data rules
curl -s localhost:9090/api/v1/alerts | jq .data alerts
promtool check rules /etc/prometheus/disk.rules.yml a rule file

prometheus.yml

The packaged config on the training machine, without comments:

yaml

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
      monitor: 'example'
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']
rule_files:
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    scrape_timeout: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']

Section	What it sets
`global`	default scrape and rule evaluation intervals; `external_labels` are labels every alert gets
`alerting`	the Alertmanager address
`rule_files`	rule files; the package leaves the list empty, so there are no rules
`scrape_configs`	what to collect: jobs and their targets

Jobs and targets

A job is a group of similar targets with shared scrape settings:

yaml

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100', 'node2:9100']
  - job_name: app
    scrape_interval: 5s
    metrics_path: /metrics
    static_configs:
      - targets: ['127.0.0.1:8000']
        labels:
          team: web

Prometheus adds the job label (the job name) and the instance label (the address from targets) to every series of a target; the labels from labels go on every series of that block's targets too: up{instance="127.0.0.1:8000", job="app", team="web"}. metrics_path defaults to /metrics, scrape_timeout to 10 seconds. An interval set in a job overrides the global one.

labels goes inside a static_configs item, next to targets. A job has no such field: promtool check config answers field labels not found in type config.ScrapeConfig. Labels from global.external_labels are something else: only data that leaves Prometheus gets them (alerts sent to Alertmanager, remote storage), and queries to Prometheus itself do not show them.

Reread the config

bash

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

reload sends the service a HUP signal, and Prometheus rereads the config and the rule files without a restart. The HTTP way, curl -X POST localhost:9090/-/reload, is off on the training machine: the answer is Lifecycle API is not enabled..

Whether the server accepted the config:

the journal, journalctl -u prometheus -o cat | tail -3, has either Completed loading of configuration file or Error reloading config;
the metric prometheus_config_last_reload_successful is 1 or 0 (it updates with the next scrape of the prometheus job, a few seconds later);
/api/v1/status/config and /api/v1/rules show what is actually loaded.

systemctl reload exits with code 0 even with a broken file: the signal was delivered, and whether to accept the config is up to Prometheus. A bad config is not applied on reload: the server keeps running with the old one. On restart with a bad config the service will not start.

bash

curl -s localhost:9090/metrics | grep '^prometheus_config_last_reload'

Startup flags

The service flags are the ARGS variable in /etc/default/prometheus, empty in the package. The ones needed most often:

Flag	What it sets
`--storage.tsdb.retention.time=30d`	how long to keep data (15 days by default)
`--web.enable-lifecycle`	turns on `/-/reload` and `/-/quit` over HTTP
`--web.listen-address=127.0.0.1:9090`	address and port (`0.0.0.0:9090` by default)

sudo systemctl restart prometheus applies the flags. Which flags the server is running with now: /api/v1/status/flags.

Add a target and make sure it is collected

bash

sudoedit /etc/prometheus/prometheus.yml   # the target in targets
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
sleep 20
promtool query instant http://localhost:9090 'up{job="app"}'

Wait at least one scrape interval. => 1 means the target is being collected, => 0 means Prometheus could not reach it, and the reason is in lastError.

Try a threshold before writing the rule

bash

promtool query instant http://localhost:9090 \
  'node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.05'

A rule expression is an ordinary query. An empty answer means the alert would not fire right now; series in the answer are the alerts that would fire, one per series.

Compare what is loaded with the file

bash

curl -s localhost:9090/api/v1/status/config \
  | jq -r .data.yaml | grep -A6 'job_name: app'
curl -s localhost:9090/api/v1/rules \
  | jq -r '.data.groups[].rules[].name'

The server prints the loaded config with default values filled in (honor_timestamps, scheme), so compare the fields you need rather than whole files.

A target is down

The target's lastError names the cause; what each error means is in the table of prometheus. Compare scrapeUrl with what the service listens on: ss -tlnp on the target machine and curl to the same address. More about the exporter: node-exporter, node-exporter and node-exporter.

Error reloading config

text

level=error msg="Error reloading config" err="couldn't load
  configuration (--config.file=\"/etc/prometheus/prometheus.yml\"):
  parsing YAML file
  /etc/prometheus/prometheus.yml: yaml: line 44: did not find expected
  ',' or ']'"

reload did not apply the new config; the server runs with the old one, and prometheus_config_last_reload_successful is 0. Fix the error, check with promtool check config and reload again.

error loading rules, previous rule set restored

A rule file has an error, and after reload the previous rules stay: /api/v1/rules shows the old set. What is wrong: promtool check rules <file>:

text

  FAILED:
/tmp/bad.yml: 5:15: group "demo", rule 1, "DiskAlmostFull": could not
  parse expression: 1:47: parse error: unexpected <op:<>

The service does not start after a restart

text

level=error msg="Error loading config (--config.file=…)"
  err="parsing YAML file /etc/prometheus/prometheus.yml: …"

Prometheus does not start with a bad config, and systemctl is-active says failed. Fix it, check with promtool check config, then run sudo systemctl reset-failed prometheus and start.

field … not found in type config.plain

text

  FAILED: parsing YAML file /tmp/p.yml: yaml: unmarshal errors:
  line 26: field scrape_config not found in type config.plain

An unknown key: here scrape_config instead of scrape_configs. Other common typos are static_config, target, and job instead of job_name. The key can also be right but at the wrong level: field labels not found in type config.ScrapeConfig means labels was written in the job, while it belongs inside static_configs (prometheus).

did not find expected ',' or ']'

A YAML syntax error: an unclosed bracket in targets: ['…', a quote or an indent. The line number is in the message.

The rule is in the file, but /api/v1/rules is empty

the file is not in rule_files; the packaged config leaves that list empty;
the path is wrong: a relative path counts from the directory of prometheus.yml, and promtool check config tells how many rule files it found (SUCCESS: 1 rule files found);
there was no reload after the edit, or it failed, see prometheus.

The alert does not fire

the rule is loaded and its state is inactive: run the expr with promtool query instant. Empty means the condition is false right now: a reversed sign (> instead of <), a threshold in percent against a ratio, or a label with a typo;
the alert is stuck in pending: for has not passed yet, or the condition comes and goes and the countdown starts over;
the alert is firing but no notification arrives: continue with alertmanager.

Lifecycle API is not enabled

The answer to curl -X POST localhost:9090/-/reload: Prometheus runs without --web.enable-lifecycle. Reload with sudo systemctl reload prometheus.

query error: connection refused

text

query error: Post "http://localhost:9090/api/v1/query": dial tcp
  127.0.0.1:9090: connect: connection refused

promtool query could not reach the server: Prometheus is not running, see systemctl status prometheus.

Restart with an unchecked config

A reload with an error keeps the old config; a restart with an error leaves the machine without monitoring: the service does not come up, metrics are not collected, alerts are not evaluated. Before any restart, run promtool check config.

The app's metrics are not in Prometheus

Situation

Task

Constraints

Signing in is needed to start the machine

Add a target and make sure it is collected

Try a threshold before writing the rule

Compare what is loaded with the file

A target is down

Error reloading config

error loading rules, previous rule set restored

The service does not start after a restart

field … not found in type config.plain

did not find expected ',' or ']'

The rule is in the file, but /api/v1/rules is empty

The alert does not fire

Lifecycle API is not enabled

query error: connection refused

Restart with an unchecked config