Load average: how many tasks wait for the CPU and the disk

Load average is the average number of tasks that want to run: running or waiting for a free CPU (state R) and waiting for the kernel to finish an operation for them (state D). uptime, top and w print three numbers — the average over 1, 5 and 15 minutes:

student@ubuntu:~$ uptime
 23:38:05 up 3 min,  2 users,  load average: 2.90, 1.43, 0.57
student@ubuntu:~$ nproc
1

Load is counted in tasks; it has no percent. Compare it with the number of CPUs from nproc. On one CPU, 1.00 means "busy all the time, nobody queued", while 2.90 means that almost three tasks wanted to run at once on average. On eight CPUs the same 2.90 is an idle machine.

Three numbers

They are moving averages: the 1-minute one reacts to changes quickly, the 15-minute one smooths them out. The three together show the direction: 2.90, 1.43, 0.57 — the load grew in the last minutes, 0.30, 1.50, 2.80 — it is falling.

The number catches up with a new level gradually. Two tasks that keep the CPU busy for exactly one minute raise the 1-minute number to about 1.3; it reaches 2 after several minutes. A short spike barely shows in the 15-minute number.

What counts: R and D

Every 5 seconds the kernel recounts tasks in two states. A task is each thread: a program with eight busy threads adds 8.

The state shows in the STAT column of ps and the S column of top (see processes).

State D

A process in uninterruptible sleep waits for the kernel to finish an operation for it: a disk read, a write to a filesystem, a reply from an NFS server. It gets signals only when the wait is over — even kill -9:

student@ubuntu:~$ ps -eo stat,pid,wchan:30,cmd | awk '$1 ~ /^D/'
D       2147 percpu_rwsem_wait              touch /mnt/frz/f1
D       2148 percpu_rwsem_wait              touch /mnt/frz/f2
D       2149 percpu_rwsem_wait              touch /mnt/frz/f3
student@ubuntu:~$ kill -9 2147; sleep 1; ps -o stat,pid,cmd -p 2147
STAT     PID CMD
D       2147 touch /mnt/frz/f1

Here the filesystem was frozen with fsfreeze -f, and every write to it waits for the thaw. The WCHAN column is the kernel function the process sleeps in. Such tasks use no CPU, yet each one adds one to the load. Once the filesystem was thawed (fsfreeze -u), process 2147 exited at once: the pending SIGKILL took effect.

Load and CPU usage

Load tells you how many tasks want to run. The %Cpu(s) line in top tells you what the CPU is busy with. Together they tell two problems apart on a one-CPU machine:

The fields of %Cpu(s) are shares of CPU time over the last interval:

A process priority is lowered with nice and renice (see the nice value); its share goes to ni.

wa counts only waiting for a device. In the example above the three D processes wait for the filesystem to thaw and never reach the disk, so wa is 0.0. When a process waits for the disk itself, wa grows: during a dd … oflag=direct write on the same machine, vmstat showed wa 82. Yet wa is still idle time: a task that needs the CPU takes that time, and wa drops towards zero while the disk stays just as busy.

Where the numbers come from

uptime, top and monitoring systems read /proc/loadavg:

student@ubuntu:~$ cat /proc/loadavg
1.72 0.68 0.25 1/125 2118

The first three fields are the load average. 1/125 is how many tasks are in R right now and how many tasks exist. The last field is the PID of the most recently created process.

Pressure: /proc/pressure

Load adds up waiting for the CPU and waiting for the kernel into one number. The files /proc/pressure/cpu, io and memory (PSI, pressure stall information) keep them apart. This is cpu while two stress-ng --cpu 2 tasks share one CPU:

student@ubuntu:~$ cat /proc/pressure/cpu
some avg10=64.78 avg60=18.21 avg300=4.14 total=12842145
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

some avg10=64.78 means that during 64.78 % of the last 10 seconds at least one task waited for a CPU; avg60 and avg300 are the same over a minute and over 5 minutes. full in io and memory is the share of time when all active tasks waited at once. For cpu at the whole-machine level full is always 0.

• uptime-who — uptime, w and who is logged in.
• top — load, the %Cpu(s) line and processes in one window.
• processes — states, signals and the nice priority.
• system-monitoring — the order of checks for "the server is slow".
• kill — signals that a D process will receive later.
• docker — CPU and memory limits of a container.
• stress-ng — CPU load for experiments with load.