Linux Labs

Observability: metrics, alerts, logs / Stage 1 of 9

Metrics, exporters and scraping

Not running Starting usually ~20 s
Leave the lesson

Step 3 of 7 · theory · 5 min

A counter and a current value

The type in the # TYPE line tells you how to read the number: as a total since start or as the state right now.

A counter only grows from the moment the exporter started: CPU seconds, bytes read from disk or received over the network. Such names end in _total. A counter's number is a total over the whole run time, and on its own it says nothing about what is happening now. The next lesson turns it into a rate.

A gauge is a current value that goes up and down: free memory, the node_load1 load, disk space. You can read it as it is.

The memory in the opening is a gauge. MemFree is memory nothing uses, not even the file cache; there is always little of it, because the kernel keeps its cache in free memory. MemAvailable is how much programs can get, including the cache the kernel will hand over on demand. The available column of free counts the same way (memory and swap). The on-call engineer needs the MemAvailable figure.

Check yourself 1 question

Which exporter line may have a smaller value a minute from now?

  • Bytes read from the disk are a counter: what was read cannot be “unread”, so the value only grows.

  • This is a counter (# TYPE … counter, the _total ending): idle seconds since the exporter started only add up.

MemAvailable is a gauge: free memory goes up and down with the load. Counters ending in _total only grow while the exporter runs; each metric's type is in its # TYPE line.

Key points

  • A counter ending in _total only grows from start — it is a total over the whole run time.
  • A gauge shows the current value and is read as it is.
  • Memory free for programs is node_memory_MemAvailable_bytes; MemFree leaves out the cache the kernel will hand over.
Ubuntu 24.04
Loading…