PromQL: the Prometheus query language

PromQL is the Prometheus query language. A query selects time series from the Prometheus database and computes over them: a growth rate, a sum, a ratio, a comparison with a threshold. Alerting rule expressions (prometheus) and grafana panels are written in PromQL.

The Prometheus web UI on :9090 draws the result as a graph. From the terminal, promtool from the prometheus package runs the query:

$ promtool query instant http://localhost:9090 'up'
up{instance="localhost:9090", job="prometheus"} => 1 @[1792643816.192]
up{instance="localhost:9100", job="node"} => 1 @[1792643816.192]

Each line is one series: the metric name, labels in curly braces, the value after =>, and a timestamp in Unix seconds after @ (date -d @1792643816 turns it into a date, see date). Empty output with exit code 0 means "no such series", which is not an error.

Metric, labels and series

A metric is a named quantity: node_memory_MemAvailable_bytes is the available memory in bytes. Labels are name="value" pairs that say what the number is about: mountpoint="/", cpu="0", mode="idle". The name together with the full label set defines a time series, a sequence of values with timestamps. There are as many series as there are distinct label sets: on the training machine node_filesystem_avail_bytes is three series, for /, /boot and /boot/efi.

Prometheus adds the job and instance labels itself when it collects metrics: job is the job name from scrape_configs, instance is the target address (prometheus).

Selector

A selector picks series by name and labels:

node_filesystem_avail_bytes                  # all series of a metric
node_filesystem_avail_bytes{mountpoint="/"}  # label equals
up{job!="prometheus"}                        # label differs
node_cpu_seconds_total{mode=~"user|system"}  # matches a regex
node_cpu_seconds_total{mode!~"idle|iowait"}  # does not match it
{__name__=~"node_memory_.*"}                 # by part of the name

A label value goes in double quotes. The regular expression after =~ has to match the whole value: job=~"node" will not select node2, you need job=~"node.*" (regex).

Instant vector and range vector

up is an instant vector: one latest value per series. A window in square brackets, up{job="node"}[1m], gives a range vector: every point of every series over the last minute. With a scrape every 15 seconds that is four points. Window units are s, m, h, d.

A range vector cannot be graphed on its own: it is passed to a function that reduces the window's points to one number, such as rate, increase, avg_over_time or max_over_time.

Counter and gauge

The metric type shows in the exporter output, on the # TYPE line (node-exporter):

# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 23.82
# TYPE node_memory_MemAvailable_bytes gauge
node_memory_MemAvailable_bytes 2.95616512e+08

The raw value of a counter says nothing about load: 23.82 seconds of CPU idle time since boot could be a lot or a little. What matters is how fast it grows.

rate and increase

rate(counter[window]) is the average growth rate over the window, per second. increase(counter[window]) is the growth over the window, which is rate multiplied by the window length.

rate(node_cpu_seconds_total{mode="idle"}[1m])  # idle share, 0…1
increase(prometheus_http_requests_total[1m])   # requests a minute

CPU seconds per second is a share of time: 0.9 for mode="idle" means the core was idle 90% of the time. Both functions handle counter resets: when the value drops, the process has restarted, and growth is counted from zero without a negative rate.

The window has to hold at least two points; four scrape intervals is the usual choice: [1m] with scrape_interval: 15s. [10s] with a scrape every 15 seconds returns nothing. rate is also empty during the first tens of seconds after Prometheus starts: there is no second point yet.

rate and increase are for counters only. For a gauge, the growth rate is deriv, the change over the window is delta, the average and the maximum are avg_over_time and max_over_time.

Aggregation: sum by and without

sum, avg, min, max and count reduce series to one or to groups by label:

sum(rate(node_cpu_seconds_total[1m]))             # all in one series
sum by (mode) (rate(node_cpu_seconds_total[1m]))  # one per mode
sum without (cpu) (rate(node_cpu_seconds_total[1m]))  # drop cpu
count by (job) (up)                               # targets per job
topk(3, node_filesystem_size_bytes)               # the three largest

After by (…) the result keeps only the listed labels; after without (…) it keeps all but the listed ones:

$ promtool query instant http://localhost:9090 'count by (job) (up)'
{job="prometheus"} => 1 @[1792644275.563]
{job="node"} => 1 @[1792644275.563]

rate first, then sum: a sum of counters from different processes resets whenever any one of them restarts.

Arithmetic and comparison

+ - * / between two sets of series work pairwise, for series with the same labels (the metric name does not count). Between a number and a series they apply to every series:

node_filesystem_avail_bytes{mountpoint="/"}
  / node_filesystem_size_bytes{mountpoint="/"}  # free share
node_memory_MemAvailable_bytes / 1024 / 1024    # in MiB

After arithmetic the result has no metric name, only labels:

$ promtool query instant http://localhost:9090 \
    'node_memory_MemAvailable_bytes / 1024 / 1024'
{instance="localhost:9100", job="node"} => 292.01953125
  @[1792644275.651]

The comparisons >, <, >=, <=, == and != act as a filter: they keep the series for which the condition holds, with their own value, and drop the rest. An empty answer means the condition holds for no series. With the word bool a comparison returns every series with the value 1 or 0:

up == 0                       # only the targets that are down
up == bool 1                  # every target: 1 alive, 0 not
node_load1 > 4                # series where the load is above 4

% in PromQL is the remainder of a division (7 % 3 is 1); the language has no percent. "Below 10%" is written as a ratio, < 0.1, or the ratio is multiplied by 100 and compared with 10. … * 100 < 10% does not parse: parse error: unexpected end of input.

An alerting rule expression is the same kind of comparison: the alert fires for every series that passes the filter (prometheus).

When the label sets of the two sides differ, no pairs are found and the answer is empty: node_filesystem_avail_bytes has mountpoint, device and fstype, node_memory_MemTotal_bytes does not. on (instance) compares only the listed labels, ignoring (…) all but those. When several series on one side match one series on the other, you need group_left:

node_filesystem_avail_bytes
  / on (instance) group_left node_memory_MemTotal_bytes

and, or and unless work on sets of series: a unless b is the series of a that have no series with the same labels in b. absent(…) returns 1 when there are no series at all: absent(up{job="app"}) means there is no app job.

Instant query and range query

promtool query instant evaluates the expression at one moment, now. promtool query range evaluates it at every point of a period with a --step, the way Grafana draws a graph:

$ promtool query range --start=$(date -d '-1 min' +%s) \
    --end=$(date +%s) --step=20s http://localhost:9090 'up{job="node"}'
up{instance="localhost:9100", job="node"} =>
1 @[1792644042]
1 @[1792644062]

The API does the same: /api/v1/query and /api/v1/query_range. Pass the expression with --data-urlencode: it contains curly braces, quotes and spaces. The value in the answer is a string next to the timestamp:

$ curl -s localhost:9090/api/v1/query \
    --data-urlencode 'query=up{job="node"}' | jq -c '.data.result[]'
{"metric":{"__name__":"up","instance":"localhost:9100","job":"node"},
  "value":[1792643816.429,"1"]}

Long output lines are wrapped here. To get the value as a number in jq, use .value[1] | tonumber.

• prometheus — metric collection, config, rules and the API.
• node-exporter — machine metrics: CPU, memory, disks.
• alertmanager — what happens to an alert after the rule.
• grafana — graphs from the same queries.
• date — turn an @[…] timestamp into a date.
• jq — parse the JSON answer of the API.
• regex — syntax for =~.