An error share is one sum divided by another, and a threshold cuts off what is normal.
Arithmetic in PromQL works between series: a / b divides the series on the left by the series on the right with the same labels. That is why you divide sums — for example, the share of 500 answers among all of the app's requests:
To get the share for each path or method, take both sums with the same by: the labels on the left and on the right must match. A series with no pair on the other side gives no result. A comparison > 0.05 at the end works as a filter: only series above the threshold remain, the rest disappear, and an empty answer means “nobody went over”. With the bool modifier — > bool 0.05 — the comparison returns 1 or 0 for every series.
The query rate(labapp_http_requests_total{code="500"}[1m]) / rate(labapp_http_requests_total[1m]) returned one line with the value 1. Why?
-
Without the division
rateprints fractional rates like 0.52. The 1 is a series divided by itself. -
The path has 200 answers too —
rate(labapp_http_requests_total[1m])shows them. The division just did not see them.
The division looks on the right for a series with the same labels, code included — and finds the same error series. That is why sums with the same by and no code label are divided: then errors are divided by all requests.
From a script the same query is easier to send to the API: curl -s localhost:9090/api/v1/query --data-urlencode 'query=…' | jq '.data.result' (in the Wiki). In the answer every series has its labels and a [time, "value"] pair: the time is seconds since 1970, the value is a string. An expression like this with a threshold is the draft of an alert rule: Prometheus will check it by itself every few seconds and raise an alarm whenever the answer is not empty. That is the next module.