Processes

System monitoring: where to start when the server is slow

The order of checks for "the server is slow": CPU, memory, disk, kernel and network — what shows each resource.

Load average: how many tasks wait for the CPU and the disk

The average number of tasks running or waiting for a CPU (R) or for the kernel (D) over 1, 5 and 15 minutes.

stress-ng — timed load on the CPU, memory and disk

stress-ng puts load on the CPU, memory and disk for a set time — for experiments with load, OOM and disk I/O.

iostat and pidstat — which disk is busy and which process loads it

iostat shows disk load — requests, latency, %util; pidstat shows CPU, memory and disk usage per process.

/proc: processes and the kernel as files

/proc is a filesystem where the kernel shows processes (/proc/<pid>) and its own state: memory, load, parameters.

systemd

The init system and service manager (PID 1): units, targets, dependencies, journals.

Memory and swap: free, available and the page cache

How to read free: used and free memory, the page cache and available; swap, a swap file and vm.swappiness.

Processes

Viewing processes, signals and priorities; finding the processes that hog CPU or memory.

OOM killer: when memory runs out

A kernel mechanism: when memory runs out on the machine or in a cgroup with a limit, it kills a process with SIGKILL.

ps — list processes

The ps command lists running processes with PID, owner, state, CPU and memory use, and the command line.

top — processes and load in real time

The top program refreshes the process list, CPU load, memory and load average every few seconds.

kill — send a signal to a process

The kill command sends a signal to a PID — ask it to exit (TERM), reload settings (HUP), or end it at once (KILL).

pgrep — find a process PID by name

The pgrep command prints the PIDs of processes by name or command line, owner and parent — no ps or grep needed.

ss — sockets and ports

Shows who is listening on ports and which connections are open; a netstat replacement, faster with many connections.

Kernel modules

Kernel modules: loading and unloading with modprobe, kernel messages in dmesg, module parameters.

Background jobs in the shell — &, jobs, nohup

Running things in the background, shell job control, surviving logout.

sysctl — kernel parameters at runtime

sysctl reads and changes kernel parameters; -w lasts until reboot, a file in /etc/sysctl.d/ makes it permanent.

lsof — open files and ports of processes

lsof shows which process holds a file, a port, or a deleted file that is still open.

Unix sockets — processes talking through a file

A unix socket is a file through which processes on one machine exchange data; connecting needs write permission on it.

strace — which system calls a process makes

strace shows the system calls of a program and their results: a file that did not open, a failed connect, a wait.

Capabilities: root split into parts

Parts of root's rights: a process or a file gets only the one it needs, e.g. CAP_NET_BIND_SERVICE for a low port.

Cron — the task scheduler

Running tasks on a schedule: crontab syntax, classic cron and modern systemd timers.

ulimit and process limits: open files, processes, core

Process resource limits — open files, processes, core: ulimit, /proc/<pid>/limits and a service LimitNOFILE.

cgroups and namespaces — how containers work

Isolation (namespaces) and resource limits (cgroups) — the foundation of containers and systemd.

tmux — terminal multiplexer

Keeps a terminal session alive on the server after an SSH drop; sessions live in memory and don't survive a reboot.

screen — terminal multiplexer

Keeps a terminal session alive on the server after an SSH drop; sessions live in memory and don't survive a reboot.

at — a one-time scheduled job

Run a command once at a given time — unlike cron's recurring schedule.

flock — preventing a second run

flock holds a file lock while a command runs, so a second copy of the script waits for it or exits at once.

sleep — delay execution

Pauses execution for a specified duration; does not test whether another program is ready.

wait — wait for a child task in Bash

A Bash builtin that waits for a child task and returns its completion status.