node_exporter: machine metrics for Prometheus

node_exporter is the Prometheus exporter for the machine itself. It reads /proc, /sys and the systemd state and serves current values over HTTP on port 9100, path /metrics: CPU, memory, disks, file systems, network, load and service state. It stores nothing and sends nothing anywhere: Prometheus comes for the metrics on a schedule (prometheus).

In Ubuntu this is the prometheus-node-exporter package: a service of the same name, the program /usr/bin/prometheus-node-exporter, and the prometheus user. On the training machine the package is installed and the service is off:

sudo systemctl enable --now prometheus-node-exporter

The exporter answers a second or two after it starts.

Exposition format

The /metrics answer is text. Each metric has two comment lines and value lines:

$ curl -s localhost:9100/metrics | grep MemAvailable
# HELP node_memory_MemAvailable_bytes Memory information field
  MemAvailable_bytes.
# TYPE node_memory_MemAvailable_bytes gauge
node_memory_MemAvailable_bytes 2.95616512e+08

The first line of the output is a single line, wrapped here.

The value is a floating-point number, often in exponent notation: 2.95616512e+08 is 295,616,512 bytes, about 282 MiB. A metric with labels has several value lines:

node_filesystem_avail_bytes{device="/dev/vda4",fstype="ext4",
  mountpoint="/"} 5.361012736e+09

The name tells the units: _bytes, _seconds; _total marks a counter (promql). The node_ prefix is machine metrics, go_ and process_ are about the exporter itself. On the training machine the answer is about 1900 lines: roughly 270 metrics and 1300 node_* values.

Main metrics

How much memory can be taken is MemAvailable. MemFree is memory used by nothing at all; the file cache is not in it, although the kernel gives the cache back on demand. On the training machine MemFree is about 13 MiB and MemAvailable about 280. More on load and memory: load-average and memory-swap.

Collectors

A collector is the part of the exporter that reads one source: meminfo reads /proc/meminfo, filesystem the mounted file systems, netdev the interfaces, systemd the units, textfile the files with your own metrics. More than forty are on on the training machine. Which ones, and whether the last read worked, is in the node_scrape_collector_success metric:

curl -s localhost:9100/metrics | grep '^node_scrape_collector_success'

0 for a collector means its source could not be read; the reason is in the service journal. The --collector.<name> and --no-collector.<name> flags turn a collector on or off (see "Configuration").

The filesystem collector does not serve node_filesystem_* for every mount point. It skips whatever matches --collector.filesystem.mount-points-exclude, which on the training machine defaults to:

^/(dev|proc|run|sys|mnt|media|var/lib/docker/.+|
  var/lib/containers/storage/.+)($|/)

The expression is wrapped here. A volume mounted under /mnt/… or /media/… does not get into the metrics at all: it has no series, and a rule with its mountpoint is empty (node-exporter). File system types are cut by --collector.filesystem.fs-types-exclude (tmpfs is not in that list; proc, sysfs, overlay, squashfs are). A flag in ARGS sets your own list and replaces the default.

textfile: your own metrics in a file

The textfile collector reads the *.prom files in /var/lib/prometheus/node-exporter on every request to /metrics and serves their lines along with its own. This is how a script with no HTTP server of its own tells Prometheus, for example, the time of the last successful backup:

$ echo 'lab_backup_last_success_timestamp_seconds 1792640000' \
    | sudo tee /var/lib/prometheus/node-exporter/backup.prom
$ curl -s localhost:9100/metrics | grep lab_backup
# HELP lab_backup_last_success_timestamp_seconds Metric read from
  /var/lib/prometheus/node-exporter/backup.prom
# TYPE lab_backup_last_success_timestamp_seconds untyped
lab_backup_last_success_timestamp_seconds 1.79264e+09

# HELP and # TYPE lines in the file are optional: without them the exporter writes its own, and the type is untyped. The directory belongs to root, so writing to it takes sudo.

• prometheus — who collects these metrics and how.
• promql — how to turn them into rates, ratios and thresholds.
• system-monitoring — the same quantities with top, free, iostat.
• firewall — open port 9100 to Prometheus only.
• ss — which address and port the exporter listens on.
• systemctl — enable --now, restart, status.
• load-average — what node_load1 and its neighbours mean.
• curl — request /metrics from the terminal.