Prometheus starts scraping a new target once the config is edited and reread.
A job is added to the scrape_configs list as the same kind of block: job_name, with static_configs and targets under it. Indentation in YAML sets nesting, and one extra space breaks the whole file. So after an edit the config is checked: promtool check config /etc/prometheus/prometheus.yml prints SUCCESS or the number of the broken line (in the Wiki).
To reread the config without a restart, run sudo systemctl reload prometheus: systemd sends Prometheus a HUP signal, and history and scraping of the other targets go on. A broken config is not applied by a reload: the command still exits with 0, Prometheus keeps working from the previous config, and an Error reloading config line goes to its journal (in the Wiki).
In prometheus.yml the node job's target was moved to port 9101, promtool check config printed SUCCESS, and nobody ran a reload. What address will /api/v1/targets show for the node target?
-
A running Prometheus knows only the loaded config and has not seen the new file yet. There is still one target — with the old address.
-
promtool only checks the file: Prometheus reads its config at start and on reload. The loaded config is shown by
/api/v1/status/config.
Without a reload Prometheus works from the config it loaded at start and scrapes the old address. It reads the new one on sudo systemctl reload prometheus — if the file has no errors.
After a successful reload the new target shows up in /api/v1/targets straight away, and its up after the first scrape, within scrape_interval; here that is 5 seconds. If a target needs labels of its own — say, the team responsible for it — a labels: line next to targets sets them: Prometheus adds them to every series of that target. The config loaded right now is shown by /api/v1/status/config.
Connect the app to Prometheus
The labapp training app is running and serves metrics at 127.0.0.1:8000/metrics, but Prometheus does not scrape them. Make it scrape them; call the job labapp. The config is /etc/prometheus/prometheus.yml.
Goals · 0 of 2
- Describe the labapp job in Prometheus's config not done yet
- Get Prometheus to scrape the app not done yet