Linux Labs

Observability: metrics, alerts, logs / Stage 2 of 9

PromQL: rate, sum and comparison

Not running Starting usually ~20 s
Leave the lesson

Step 3 of 6 · on a machine · 7 min

Summing by labels

One application metric gives many series; to answer “how many errors”, you add them up.

The labapp training app counts requests with the labapp_http_requests_total metric with the code, method and path labels — every combination is its own series. A selector picks the series you need — label conditions in curly braces: {code="500"} is equal, {code!="200"} is not equal, {path=~"/api/.*"} matches a regular expression.

Aggregation adds series up. sum by (code) (…) sums all series with the same code and keeps only that label; sum without (method) (…) drops the named label and keeps the rest. avg, max and count work the same way. You add up rates — sum by (code) (rate(…[1m])): a sum of the counters themselves would give a total since start again.

Exercise · look

Sum the requests by code

Prometheus scrapes the metrics of the labapp training app. Look at the request rate of every labapp_http_requests_total series, then at the sum of the rates by response code.

First, predict

How many lines will the query with sum by (code) print?

Now start the exercise, run the command and compare your prediction with the output.

Sign in and start The machine is available after signing in

Labels not listed in by are gone from the answer: lines with different paths and methods were added into one. Without by — sum(rate(…)) — a single line would remain: all of the app's requests per second.

Ubuntu 24.04
Loading…