Observability: metrics, alerts, logs
Machine and application metrics in Prometheus and PromQL queries; alert rules and notification delivery through Alertmanager; logs from several machines on one collector and investigating with logs — up to watching a second machine from the first.
9 stages6 labs~6 hours medium
The track has not been started. The first stage is «Metrics, exporters and scraping», ~50 min.
Module 1
Metrics and Prometheus
What an exporter serves, how Prometheus scrapes targets and what up means; PromQL queries: a counter's rate, a sum by labels and a comparison with a threshold.
3 stages · ~2 hours
-
Stage 1 of 9
1 · Metrics, exporters and scrapingMetrics, exporters and scraping
7 steps · 1 lab · ~50 min 7 steps · 1 lab · ~50 min- Where is the free memory exercises 0 of 1 on a machine 4 min
- 2 A metric line theory 5 min
- 3 A counter and a current value theory 5 min
- 4 Prometheus scrapes targets on a machine 7 min
- 5 A new job on a machine 7 min
- 6 A machine dropped out of monitoring lab 20 min
- 7 Summary summary 2 min
-
Stage 2 of 9
2 · PromQL: rate, sum and comparisonPromQL: rate, sum and comparison
6 steps · 1 lab · ~50 min 6 steps · 1 lab · ~50 min- How much traffic right now exercises 0 of 1 on a machine 5 min
- 2 A counter's rate on a machine 7 min
- 3 Summing by labels on a machine 7 min
- 4 A share and a threshold on a machine 9 min
- 5 The app's metrics are not in Prometheus lab 20 min
- 6 Summary summary 2 min
-
3 · Module test 9 questions · pass mark 70%
Module 2
Alerts
An alert rule with a threshold and a duration, checking it with promtool and the way from pending to firing; Alertmanager: routing, grouping, repeats and silences.
3 stages · ~2 hours
-
Stage 4 of 9
4 · An alert rule: threshold and durationAn alert rule: threshold and duration
6 steps · 1 lab · ~1 hour 6 steps · 1 lab · ~1 hour- The disk is full and there is no alert exercises 0 of 1 on a machine 6 min
- 2 The rule file and rule_files on a machine 7 min
- 3 Threshold and duration on a machine 8 min
- 4 Labels, description and a failed reload on a machine 7 min
- 5 The full-volume alert stays silent lab 30 min
- 6 Summary summary 2 min
-
Stage 5 of 9
5 · Alertmanager: to whom and how many timesAlertmanager: to whom and how many times
6 steps · 1 lab · ~1 hour 6 steps · 1 lab · ~1 hour- Fifty notifications for one event exercises 0 of 1 on a machine 6 min
- 2 A group and three timers on a machine 9 min
- 3 A silence for maintenance on a machine 6 min
- 4 Whom to send to: routes on a machine 7 min
- 5 Alerts do not reach the receiver lab 30 min
- 6 Summary summary 2 min
-
6 · Module test 9 questions · pass mark 70%
Module 3
Logs from many machines
Forwarding syslog to a collector and sorting it by host; investigating with logs: nginx, application and kernel events on one timeline.
3 stages · ~2 hours
-
Stage 7 of 9
7 · Logs in one placeLogs in one place
7 steps · 1 lab · ~1 hour 7 steps · 1 lab · ~1 hour- No lines on the collector exercises 0 of 1 on a machine 4 min
- 2 rsyslog rules on a machine 5 min
- 3 Forwarding to the collector on a machine 5 min
- 4 TCP and UDP on a machine 7 min
- 5 Sorting by machine on a machine 7 min
- 6 The second machine's logs are not on the collector lab 30 min
- 7 Summary summary 2 min
-
Stage 8 of 9
8 · Investigating with logsInvestigating with logs
6 steps · 1 lab · ~55 min 6 steps · 1 lab · ~55 min- The site answers 500 exercises 0 of 1 on a machine 4 min
- 2 Who writes what on a machine 7 min
- 3 Record fields on a machine 8 min
- 4 The kernel on the same timeline on a machine 9 min
- 5 The site returns 500 lab 25 min
- 6 Summary summary 2 min
-
9 · Module test 10 questions · pass mark 70%
-
-
Final test 18 questions · pass mark 80%