Troubleshooting

Network diagnostics

A ladder of connectivity checks — from the gateway to DNS and the destination port.

System monitoring: where to start when the server is slow

The order of checks for "the server is slow": CPU, memory, disk, kernel and network — what shows each resource.

Load average: how many tasks wait for the CPU and the disk

The average number of tasks running or waiting for a CPU (R) or for the kernel (D) over 1, 5 and 15 minutes.

ping — check whether packets reach a host

The ping command sends ICMP echo requests to a host and prints replies, latency and loss — the first network check.

stress-ng — timed load on the CPU, memory and disk

stress-ng puts load on the CPU, memory and disk for a set time — for experiments with load, OOM and disk I/O.

iostat and pidstat — which disk is busy and which process loads it

iostat shows disk load — requests, latency, %util; pidstat shows CPU, memory and disk usage per process.

/proc: processes and the kernel as files

/proc is a filesystem where the kernel shows processes (/proc/<pid>) and its own state: memory, load, parameters.

System boot and GRUB

How Linux boots: firmware, the GRUB bootloader, the kernel, systemd targets and emergency mode.

Memory and swap: free, available and the page cache

How to read free: used and free memory, the page cache and available; swap, a swap file and vm.swappiness.

Processes

Viewing processes, signals and priorities; finding the processes that hog CPU or memory.

OOM killer: when memory runs out

A kernel mechanism: when memory runs out on the machine or in a cgroup with a limit, it kills a process with SIGKILL.

shellcheck — checking scripts before they run

shellcheck reads a bash or sh script without running it and finds bugs: unquoted variables, unchecked cd.

Boot targets: default, multi-user, rescue and emergency

A target is a group of units systemd brings the boot up to; rescue and emergency are the targets for repairs.

systemd-analyze: what slows the boot and what is wrong in a unit

Breaks the boot down by time and by the chain of unit dependencies, and checks a unit file before the service runs.

Exit codes: $?, && and ||

An exit code is a number from 0 to 255 that a command reports: 0 means success, anything else a failure or a "no".

Disk space: df, du, inode

Where the space and inodes went — df, du, finding large files, pitfalls.

dmesg — kernel messages: OOM, disk and filesystem errors, segfaults

dmesg prints the kernel ring buffer: messages about disks, filesystems, OOM, crashed programs and hardware.

lsof — open files and ports of processes

lsof shows which process holds a file, a port, or a deleted file that is still open.

Unix sockets — processes talking through a file

A unix socket is a file through which processes on one machine exchange data; connecting needs write permission on it.

strace — which system calls a process makes

strace shows the system calls of a program and their results: a file that did not open, a failed connect, a wait.

Filesystems: ext4, xfs, fsck

Filesystems: creating one with mkfs, labels and UUIDs, fsck checks, choosing between ext4 and xfs.

ulimit and process limits: open files, processes, core

Process resource limits — open files, processes, core: ulimit, /proc/<pid>/limits and a service LimitNOFILE.

DNS and name resolution

How a name becomes an IP — resolv.conf, systemd-resolved, /etc/hosts, dig.

nmap: which ports are visible from outside

A port scanner: shows which ports of a machine are open, closed or filtered, as another machine sees them.

Software RAID with mdadm

Software RAID on Linux — creating an array, checking status, replacing a disk.

HTTP — requests, responses, status codes and headers

HTTP is the web protocol: a client sends a method, path and headers; the server replies with a code, headers, body.

curl — HTTP requests from the console

curl sends HTTP and HTTPS requests from the console: status code, headers, the whole exchange, certificate checks.

tcpdump — capturing and analyzing network traffic

Captures packets to diagnose "is traffic even arriving"; without -n it resolves every IP through DNS and slows down.

ethtool — network card settings

Link speed and duplex, error statistics, cable type — NIC diagnostics.

nc — checking ports and transferring data

Checking whether ports are reachable, simple data transfer over TCP/UDP.

Dockerfile — an image recipe and docker build

A Dockerfile is an image recipe: base, files, build and start commands; docker build turns it into an image.

docker compose — a container stack in one file

docker compose describes a container stack with network, volumes and variables in compose.yaml and starts it at once.

conntrack — connection tracking

The kernel's connection table — what NAT/firewall see, overflow, debugging.