Linux Labs

Observability: metrics, alerts, logs / Stage 2 of 9

PromQL: rate, sum and comparison

Not running Starting usually ~20 s
Leave the lesson

Step 2 of 6 · on a machine · 7 min

A counter's rate

PromQL computes the difference of two numbers divided by the time by itself — over all samples in a window.

PromQL is the Prometheus query language. A query by metric name returns the latest value of each series — an instant vector. Add a window in square brackets, node_network_receive_bytes_total[1m], and you get all samples of the last minute — a range vector.

The <code>rate</code> function takes a range vector and computes the average growth per second over that window:

bash
promtool query instant http://localhost:9090 \
  'rate(node_network_receive_bytes_total[1m])'

You get about 200,000 — the same 200 KB per second you worked out from the two queries. If less than a minute has passed since the setup, the window is not full yet and the number is smaller. The answer has no metric name: a rate is a different quantity, while the series labels stay. increase(…[1m]) computes the same as a total over the window — how many bytes arrived in a minute.

A counter drops to zero when its service restarts. rate and increase see the drop and treat it as a reset: counting goes on from zero, and the rate never turns negative. A gauge needs no rate — its value is already “now”.

Check yourself 1 question

labapp was restarted, and its request counter dropped from 5000 to 30. What will rate(labapp_http_requests_total[1m]) show for this series 10 seconds later?

  • rate treats a drop in the counter as a reset and goes on counting from zero: the rate is never negative.

  • rate counts the growth after a reset from zero: 30 new requests are already growth, there is no need to wait for 5000.

rate and increase see that the counter fell and treat it as a restart: growth after the reset counts from zero. So the rate stays as usual, and a service restart does not break the graphs.

Key points

  • A counter's number is a total since start; the load shows in its rate: rate(metric[1m]) is the growth per second over the window.
  • increase is the same growth as a total over the window; both functions handle a counter reset on restart.
  • The window in square brackets fills with samples over time: in the first minute after start the rate is understated.
Ubuntu 24.04
Loading…