Inside the system: resources, kernel, processes
Load average and the CPU, memory and the OOM killer, process limits and kernel parameters, disk load, strace and /proc, kernel messages, the system boot and time synchronisation — up to triaging a slow server.
12 stages7 labs~7 hours medium
The track has not been started. The first stage is «Load average and CPU load», ~30 min.
Module 1
CPU, memory and OOM
Who waits for the CPU and who for the disk; how used memory differs from available and why the OOM killer killed a process.
3 stages · ~2 hours
-
Stage 1 of 12
1 · Load average and CPU loadLoad average and CPU load
7 steps · ~30 min 7 steps · ~30 min- An alarm with no load exercises 0 of 1 on a machine 4 min
- 2 What the load average counts theory 5 min
- 3 Processes in the D state on a machine 6 min
- 4 The queue for the CPU on a machine 6 min
- 5 The nice priority theory 4 min
- 6 The CPU goes to the reports on a machine 3 min
- 7 Summary summary 2 min
-
Stage 2 of 12
2 · Memory, swap and the OOM killerMemory, swap and the OOM killer
9 steps · 2 labs · ~1 hour 9 steps · 2 labs · ~1 hour- A service restarts by itself exercises 0 of 1 on a machine 4 min
- 2 Used and available on a machine 6 min
- 3 Swap on a machine 4 min
- 4 The service ceiling on a machine 7 min
- 5 A unit drop-in on a machine 4 min
- 6 A ceiling for the cache on a machine 3 min
- 7 A service restarts in a loop lab 25 min
- 8 A build runs out of memory lab 20 min
- 9 Summary summary 2 min
-
3 · Module test 8 questions · pass mark 70%
Module 2
Limits and kernel parameters
A process limit and a kernel parameter changed so that the change survives a restart; the process that holds the disk.
3 stages · ~2 hours
-
Stage 4 of 12
4 · ulimit and sysctl: system ceilingsulimit and sysctl: system ceilings
10 steps · 2 labs · ~1 hour 10 steps · 2 labs · ~1 hour- The shipper loses logs exercises 0 of 1 on a machine 4 min
- 2 Process limits on a machine 4 min
- 3 Soft and hard on a machine 3 min
- 4 A service's limit on a machine 5 min
- 5 Kernel parameters on a machine 5 min
- 6 A change for boot time theory 4 min
- 7 More watches on a machine 3 min
- 8 A service fails under load lab 25 min
- 9 A kernel setting disappears lab 20 min
- 10 Summary summary 2 min
-
Stage 5 of 12
5 · Who loads the diskWho loads the disk
6 steps · ~30 min 6 steps · ~30 min- The CPU is free, the machine is waiting exercises 0 of 1 on a machine 4 min
- 2 Waiting for the disk theory 6 min
- 3 Who writes to the disk on a machine 8 min
- 4 Giving way on the disk theory 6 min
- 5 The orders wait for the disk on a machine 3 min
- 6 Summary summary 3 min
-
6 · Module test 10 questions · pass mark 70%
Module 3
What a process is doing
The cause of a silent failure from system calls and /proc; kernel messages about OOM, filesystem errors and crashes.
3 stages · ~2 hours
-
Stage 7 of 12
7 · strace and /proc: what a process is doingstrace and /proc: what a process is doing
8 steps · 1 lab · ~55 min 8 steps · 1 lab · ~55 min- The export is silent exercises 0 of 1 on a machine 5 min
- 2 System calls on a machine 6 min
- 3 Misses and filters theory 4 min
- 4 A process that waits on a machine 4 min
- 5 /proc: a process from the inside theory 6 min
- 6 The mail does not go out on a machine 3 min
- 7 An application exits silently lab 25 min
- 8 Summary summary 2 min
-
Stage 8 of 12
8 · dmesg and the kernel logdmesg and the kernel log
8 steps · ~30 min 8 steps · ~30 min- The archive does not respond exercises 0 of 1 on a machine 4 min
- 2 The kernel buffer on a machine 4 min
- 3 The ext4 error line theory 5 min
- 4 A program crashed on a machine 3 min
- 5 The segfault line and importance theory 4 min
- 6 The kernel log in journald theory 5 min
- 7 The nightly export broke off on a machine 3 min
- 8 Summary summary 2 min
-
9 · Module test 10 questions · pass mark 70%
Module 4
Boot and time
The boot chain and what slows it down; UTC, the time zone and time synchronisation.
3 stages · ~2 hours
-
Stage 10 of 12
10 · From GRUB to a target: how the system bootsFrom GRUB to a target: how the system boots
8 steps · 1 lab · ~55 min 8 steps · 1 lab · ~55 min- How long the boot took exercises 0 of 1 on a machine 4 min
- 2 The boot relay on a machine 5 min
- 3 Who held the boot up on a machine 4 min
- 4 Ordering and dependency on a machine 6 min
- 5 Boot targets on a machine 6 min
- 6 A service in the boot on a machine 3 min
- 7 A service holds up the boot lab 25 min
- 8 Summary summary 2 min
-
Stage 11 of 12
11 · Time, time zone and synchronisationTime, time zone and synchronisation
7 steps · 1 lab · ~55 min 7 steps · 1 lab · ~55 min- The export arrives at noon exercises 0 of 1 on a machine 4 min
- 2 The clock and the zone on a machine 5 min
- 3 The schedule follows the machine's zone on a machine 5 min
- 4 Clocks drift on a machine 6 min
- 5 Slewing and stepping on a machine 8 min
- 6 The clocks of two machines drift apart lab 25 min
- 7 Summary summary 2 min
-
12 · Module test 10 questions · pass mark 70%
-
-
Final test 20 questions · pass mark 80%