vmstat speaks about the machine as a whole. iostat and pidstat from the sysstat package show which disk is busy and which process holds it.
Find who is writing
Look at two iostat snapshots, then at the I/O of processes in pidstat — first as yourself, then with sudo. The last command shows the service of the exporter process.
First, predict
What will pidstat -d show when you run it as yourself, without sudo?
Now start the exercise, run the command and compare your prediction with the output.
In iostat the first snapshot is the average since the machine booted, the second covers the last second: read the second one. The vda line is the machine's disk:
| Column | Meaning |
|---|---|
| w/s, wkB/s | writes per second and KiB per second |
| w_await | how many milliseconds one write waits on average, queue included |
| aqu-sz | how many requests wait in the disk queue on average |
| %util | the share of time the disk had at least one request |
pidstat -d breaks the I/O down by process: kB_rd/s and kB_wr/s are the process's reads and writes in KiB per second. A process's counters live in /proc/<PID>/io, which only the owner and root can read, so without sudo pidstat sees only your processes. pidstat counts what the process itself writes, while iostat counts everything that reached the disk, the filesystem journal included. That is why exporter shows hundreds of KiB per second and vda shows thousands.
The UNIT column of ps -o pid,unit,comm -p <PID> gives the service of a PID; systemctl status <PID> shows the same.
In the second iostat snapshot vda's %util is 95. What does that mean?
-
%util does not measure speed; it counts only the time the disk had requests. At 95% vda writes just a few MB/s — the wkB/s column.
-
%util counts the time the disk had requests.
df -hshows space: less than half of the lesson machine's disk is used.
%util is the share of time the disk had at least one request. 95% means the disk is busy almost all the time, even though it writes little data — a few MB/s. Disk space is a different number, shown by df -h.