iostat and pidstat — which disk is busy and which process loads it

iostat and pidstat come from the sysstat package. iostat looks at devices: how many requests, how much data, how long the answers took. pidstat looks at processes: who spends CPU, memory and disk.

Both take an interval and a report count: iostat 1 5 is five reports one second apart. Without an interval there is one report with averages since system boot. The first iostat report with an interval also holds averages since boot; -y skips it.

iostat -x: device fields

iostat -x prints a couple of dozen columns. On the training machine dd was writing to disk in 4 KiB blocks bypassing the cache (oflag=direct):

The same fields with r are for reads (r/s, rkB/s, r_await), with d for discards, with f for device cache flushes.

iostat -dxs: short form

-s squeezes -x into one narrow line, -d drops the CPU block:

tps is requests per second (reads and writes together), rqm/s is requests merged with neighbours before reaching the device.

Reading %util and await

%util 100 on a disk that serves one request at a time (an HDD) means the limit. Disks that take many requests at once (SSD, NVMe, virtual disks) can do more at 100 %. The real sign of overload is growing await and aqu-sz: requests wait in a queue. On the training machine at %util 84 a request waited 0.13 ms, and the queue was below one.

pidstat -d: who loads the disk

sudo pidstat -d 2 1

Only root can read another user's I/O counters. The same dd running as root is invisible without sudo: pidstat -d prints only headers.

ps shows which service the process belongs to:

student@ubuntu:~$ ps -o pid,unit,comm -p 1211
    PID UNIT                            COMMAND
   1211 ssh.service                     sshd

pidstat -u: who waits for the CPU

pidstat -u 1 2      # the Average lines, some columns omitted

Two processes on one CPU: each worked half the time (%usr) and waited for its turn the other half (%wait). High %wait means more R processes than CPUs (see load-average).

pidstat -r shows memory per process: minflt/s and majflt/s are page accesses served without and with the disk, plus VSZ, RSS, %MEM. pidstat -w shows task switches: cswch/s (the process gave up the CPU itself, waiting for I/O or an event) and nvcswch/s (the CPU was taken away).

ionice: disk priority

ionice sets a process's disk access class: realtime (-c1), best-effort (-c2, the default) and idle (-c3 — only when nobody else needs the disk).

student@ubuntu:~$ ionice -p 1
none: prio 0
student@ubuntu:~$ sudo ionice -c3 -p $$; ionice -p $$
idle

none means no class was set, and the process runs as best-effort. The class is honoured by the disk's I/O scheduler. On the training machine vda uses mq-deadline, loop devices use none, and there priorities do nothing:

student@ubuntu:~$ cat /sys/block/vda/queue/scheduler
none [mq-deadline]

• system-monitoring — the order of checks and vmstat.
• load-average — state D, wa and the CPU queue.
• processes — nice, renice, ionice.
• disk-usage — disk space: df and du.
• stress-ng — disk load for experiments.
• top — processes in one window.