The exporter's thousands of lines all read the same way: name, labels, number.
Every line that does not start with # is one metric with its value. Here are two CPU lines together with their description:
First comes the name, then the labels in curly braces, and the number at the end. Labels say what the number is about: here, which core spent these seconds and in which mode. The # HELP and # TYPE lines describe the metric: what it counts and what type it is.
A name together with a set of labels is a time series: Prometheus keeps its values over time, like a column of numbers with timestamps. A metric has as many series as there are different label combinations: cores times modes for the CPU, a series per disk for disks, one per mount point for file systems. Hence thousands of lines for a single machine.
The exporter writes numbers as floating point: 2.88198656e+08 is 288,198,656. The unit is at the end of the name: _bytes, _seconds. So the memory lines are found by the start of the name and compared with free -b directly in bytes.
On a machine with two cores node_cpu_seconds_total counts time in 8 modes (idle, user, system…). How many value lines will the exporter return for this metric?
-
One line per core would be the case with the cpu label alone. The second label is mode, so there are 2 × 8 lines.
-
The metric has one name, but every combination of labels is its own series and its own line: a cpu and mode pair.
-
8 lines is a one-core machine like this one:
curl -s localhost:9100/metrics | grep -c '^node_cpu_seconds_total'. A second core adds 8 more.
Every combination of the cpu and mode labels is a separate time series and a separate line: 2 cores × 8 modes = 16. That is why a machine with dozens of cores, disks and partitions has thousands of lines.
Key points
-
A metric line is a name, labels in curly braces and a number;
# HELPand# TYPEdescribe the metric. - A name with a set of labels is a separate time series: a metric has as many lines as label combinations.
-
The unit is at the end of the name (
_bytes,_seconds), and a number may be written as2.88e+08.