Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

System monitoring: where to start when the server is slow

8 min Ubuntu 24.04 · procps 4.0 · sysstat 12.6

The order of checks for "the server is slow": CPU, memory, disk, kernel and network — what shows each resource.

Cheat sheet

8 commands Printable version
  • uptime; nproc load and CPU count
  • vmstat 1 5 queue, memory, swap, disk, CPU
  • top processes: P — by CPU, M — by memory
  • free -h memory: look at available
  • iostat -dxs 1 3 disks: await, aqu-sz, %util
  • sudo pidstat -d 1 3 who reads and writes the disk
  • sudo journalctl -k -p err -b kernel errors of this boot
  • mpstat -P ALL 1 3 load of each CPU

Overview

"The server is slow" means some resource has run out: CPU, memory, disk or network. First find which one with a command or two, then look into it in detail.

vmstat comes from the procps package; iostat, pidstat, mpstat and sar from sysstat. The training machine has both.

Order of checks

Resource Command Sign of trouble
CPU uptime, top load above nproc, id near 0
memory free -h, vmstat 1 little available, si and so grow, OOM in the log
disk vmstat 1, iostat -dxs 1 high wa, b above 0, %util near 100
kernel sudo journalctl -k -p err -b filesystem and disk errors, OOM, hung tasks
network ss -s, sar -n DEV 1 queues and errors on an interface

Each resource in detail: the CPU queue and state D — load-average, memory and swap — memory-swap, OOM — oom-killer, disks and the processes using them — pidstat-iostat, the kernel log — journalctl, sockets and connections — ss.

vmstat

vmstat 1 prints a line every second: the task queue, memory, swap, disk and CPU. The first line holds averages since boot; the following lines show the current picture.

bash 
vmstat 1 5              # 5 lines one second apart
vmstat -w 1             # wide columns for large numbers
vmstat -s               # summary of memory and CPU counters

Two lines from the training machine — two different problems:

Picture r b free cache si so bi bo us sy id wa
two tasks share one CPU 2 0 19180 289936 0 0 0 8 100 0 0 0
dd writes to disk 0 1 45064 268508 0 0 0 32716 1 18 0 81
Column Meaning
r tasks in state R: running and waiting for a CPU
b tasks waiting for disk I/O
swpd used swap, KiB
free, buff, cache free memory, buffers and page cache, KiB
si, so reading from swap and writing to swap, KiB/s
bi, bo reading from disks and writing to disks, KiB/s
in, cs interrupts and task switches per second
us, sy, id, wa, st shares of CPU time, as in top
gu guest VM time, if the machine is a hypervisor itself

b and wa count only waiting for a device. Processes waiting for a frozen filesystem show in load average but not in b (see load-average).

mpstat

mpstat -P ALL 1 shows each CPU separately: %usr, %sys, %iowait, %steal, %idle. On a two-CPU machine an average id of 50 % can mean one CPU fully busy and the other idle. A single-threaded program never uses more than one CPU.

History: sar

sar -u 1 3 shows the CPU right now, like mpstat. The point of sar is the history of past hours, but Ubuntu ships with collection off:

text 
student@ubuntu:~$ grep ENABLED /etc/default/sysstat
ENABLED="false"
student@ubuntu:~$ sar
Linux 6.8.0-124-generic (ubuntu)    10/20/26    _x86_64_    (1 CPU)

23:50:55     LINUX RESTART  (1 CPU)

To turn collection on, set ENABLED="true" in /etc/default/sysstat, then run sudo systemctl restart sysstat. After that sar -u, sar -r (memory) and sar -d (disks) show today by intervals.

Recipes

2

Who eats the CPU and memory

bash 
ps -eo pid,ni,pcpu,comm --sort=-pcpu | head -5   # by CPU
ps -eo pid,rss,comm --sort=-rss | head -5        # by memory, KiB
pidstat -u 1 3          # %CPU per process over time
pidstat -r 1 3          # memory and page faults per process

ps takes a snapshot, and its %CPU is the average over the whole life of the process. Who loads the CPU right now shows in top or pidstat -u 1.

A snapshot for a report

bash 
{ date; uptime; free -h; vmstat 1 3; top -bn1 | head -15; } \
  > ~/snapshot.txt

One file with the time, load, memory and process list — attach it to a ticket or compare it with a similar snapshot later.

Troubleshooting

5

High load, idle CPU

Tasks are in state D. High wa and b above 0 — they wait for the disk; continue with iostat -dxs 1 and sudo pidstat -d 1. wa and b at zero — they wait for something other than a device; see load-average.

High wa: who writes to the disk

bash 
sudo pidstat -d 1 3     # kB_rd/s and kB_wr/s per process

Without sudo, pidstat -d shows only your own processes: only root can read the I/O counters of other users' processes. iotop is not installed on the training machine. The fields of iostat and pidstat are in pidstat-iostat.

free is almost zero

The kernel keeps free memory busy with the page cache and gives it to programs on demand. The memory reserve is the available column of free -h; see memory-swap.

sar prints only LINUX RESTART

History collection is off: ENABLED="false" in /etc/default/sysstat (section "History: sar" above). Once turned on, collection shows data from that moment — the past cannot be recovered.

pidstat shows nothing

pidstat prints only processes that did something during the interval. Other users' processes appear in pidstat -d only for root — run it with sudo.

See also

  • load-average — load, states R and D, the %Cpu(s) line.
  • pidstat-iostat — disks and processes: iostat, pidstat.
  • oom-killer — a process ended when memory ran out.
  • top — processes in one window.
  • memory-swap — memory, cache and swap.
  • processes — states, signals, priority.
  • disk-usage — disk space: df and du.
  • journalctl — the journal of services and the kernel.
  • ss — sockets and connections.

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Topics
Loading…