Task → recipe

Basic commands

• Examples alias — command shortcuts
• Commands at — a one-time scheduled job
• Time formats Access is controlled by /etc/at.allow / /etc/at.deny (who can queue jobs). at — a one-time scheduled job
• Conditions and loops The [ -f ], [ -z ] and -eq checks are in test-conditions. Bash scripts: running, variables, arguments
• Default values and parts of a path Bash scripts: running, variables, arguments
• Check the syntax without running shellcheck goes deeper: Bash scripts: running, variables, arguments
• File names with spaces A loop over ls or find output inside $( ) splits names on spaces. Bash scripts: running, variables, arguments
• Find what will execute {#resolve-command} Inspect all definitions of the name. bash — the command shell
• Inspect command-line editing mode {#interactive} set -o lists option states, including emacs and vi. bash — the command shell
• Edit the command line with keys {#line-editing} In emacs mode (on by default), you edit the line before Enter with these keys: bash — the command shell
• Understand startup files {#startup} An interactive non-login shell reads ~/.bashrc. bash — the command shell
• Examples cd - is the most common trick: cd — changing directory
• Tricks In bash you can turn on shopt -s autocd — then typing a bare path without cd goes there directly. cd — changing directory
• Formats Specifiers: date — date and time in the shell
• Timestamps and arithmetic date — date and time in the shell
• Writing to a file echo — printing text
• When to use printf echo behaves differently across shells (especially -e/-n). echo — printing text
• Where to save it (login vs. non-login) File When it's read ~/.bashrc interactive non-login shells (a new terminal/tab) ~/.profile / ~/.bash_profile login shells (SSH login/console login)… Environment variables and PATH
• Test a command in if {#if-command} if reads the command's code itself; $? Exit codes: $?, && and ||
• Save the code before it is overwritten {#save-code} Every command overwrites $? Exit codes: $?, && and ||
• Exit with an error message The message goes to standard error (>&2) and the code is non-zero, so both a person and the calling program understand the script. Exit codes: $?, && and ||
• curl's code and the HTTP response {#curl-http-code} curl's code answers whether an exchange with the server took place. Exit codes: $?, && and ||
• Your own codes for different failures The usual convention: Exit codes: $?, && and ||
• The code of every pipeline stage Exit codes: $?, && and ||
• A crontab line that never overlaps A run that lands on a still-running previous one is quietly skipped. flock — preventing a second run
• A lock inside the script exec 9>file opens descriptor 9 for the whole script, and flock -n 9 locks it. flock — preventing a second run
• A systemd service systemd never starts a second copy of the same unit on its own: flock — preventing a second run
• Wait, but not forever flock — preventing a second run
• Who holds the lock lsof also shows the child process: flock — preventing a second run
• man man sections: Getting help at the terminal
• Other places to look Getting help at the terminal
• Patterns Pattern Meaning * any string (including empty), except / ? Glob patterns and brace expansion
• Glob vs. regex glob is for filenames in the shell; * means "anything". Glob patterns and brace expansion
• Identify the server before making changes {#identify} Check the node name alongside the user identity. hostname — machine names and resolution
• Check resolution of the current name {#resolution} getent queries the configured system name sources. hostname — machine names and resolution
• Compare resolved and interface addresses {#addresses} Different name and interface configuration can legitimately produce different results. hostname — machine names and resolution
• A file name with a space or special character The easiest way is to type the start of the name and press Tab: How a command is built — options and arguments
• An argument that starts with a dash How a command is built — options and arguments
• Several commands on one line How a command is built — options and arguments
• A long command over several lines A \ at the end of a line continues the command on the next one. How a command is built — options and arguments
• Navigation Tab — switch between the left and right panel. mc — a terminal file manager
• F-keys Key Action F3 view a file (mcview) F4 edit it (mcedit) F5 copy the selection to the other panel F6 move / rename F7 create a directory F8 delete F9 the top… mc — a terminal file manager
• Editor and SFTP Access a remote machine right from a panel (an SSH shell link): mc — a terminal file manager
• Keyboard shortcuts ^O = Ctrl+O, M-W = Alt+W. nano — a simple console editor
• Output and errors in separate files Output redirection and pipelines
• The order of 2>&1 matters The shell reads redirections left to right. Output redirection and pipelines
• Process output with a chain of commands Each stage does one thing: Output redirection and pipelines
• Write to a file that needs root sudo cmd > /etc/file won't work: Output redirection and pipelines
• Multi-line input right into a command Output redirection and pipelines
• Empty a log without deleting the file If a process keeps the log open, rm won't free the space, but emptying will. Output redirection and pipelines
• Find out which stage of a pipeline failed Output redirection and pipelines
• Format Specifier Meaning %s %d %f string / integer / float %-Ns %Nd width N (- = left-aligned) %.Nf N digits after the decimal point %05d leading zeros %x %o hex /… printf and echo — printing text in scripts
• More recipes printf and echo — printing text in scripts
• Check location before file operations {#locate} Inspect location and contents separately: pwd — the working directory
• Resolve a physical path {#resolve} Compare both results after entering through an existing symbolic link. pwd — the working directory
• Identify which pwd is selected {#implementation} Bash may select a builtin, function, or alias. pwd — the working directory
• Hotkeys The prefix is Ctrl+a, then a key: screen — terminal multiplexer
• Useful ~/.screenrc (common settings): screen — terminal multiplexer
• A check before running or in CI With -S warning, info and style findings are neither printed nor counted: shellcheck — checking scripts before they run
• Silence one finding in one place The comment covers the next command. shellcheck — checking scripts before they run
• Silence a finding for a whole directory A .shellcheckrc file in the script's directory or above it: shellcheck — checking scripts before they run
• The script sources another file Without -x, shellcheck prints SC1091 … Not following and does not see the functions from the sourced file. shellcheck — checking scripts before they run
• Look at the ready-made fixes Only simple findings come with a fix: shellcheck — checking scripts before they run
• Validate a script argument $# is the number of arguments. test, [ ] and [[ ]] — checking conditions
• Compare a number taken from command output -ge wants a bare number. test, [ ] and [[ ]] — checking conditions
• Patterns and regular expressions With day=2026-10-20 the second condition prints year 2026, month 10: test, [ ] and [[ ]] — checking conditions
• Several conditions at once test, [ ] and [[ ]] — checking conditions
• Hotkeys tmux — terminal multiplexer
• Report the running kernel release {#kernel-release} Record the actual result rather than copying a documentation example. uname — kernel and system information
• Separate kernel and distribution identity {#distribution} Compare independent sources. uname — kernel and system information
• Inspect machine architecture and node name {#machine} Architecture and node name are separate properties; neither is the user name. uname — kernel and system information
• Load average uptime shows three numbers — the average length of the run queue over 1, 5, 15 minutes: uptime, who, w — who is on the system and the load
• Who and from where uptime, who, w — who is on the system and the load
• Quitting and saving vim — editing files in the terminal
• Navigation and editing vim — editing files in the terminal
• Useful bits vim — editing files in the terminal
• Examples Quote a command that has a pipeline/wildcard (watch 'ss | grep x'), otherwise | and * apply to watch itself, not the command. watch — periodically repeat a command
• Examples A reliable "does this command exist" check in a script: which / type / whereis — where a command lives
• Check the current identity {#current-user} Compare the name with the numeric identity. whoami — identify the effective user
• Inspect a sudo invocation {#sudo-context} The training VM permits sudo. whoami — identify the effective user
• Inspect another account {#other-account} Use id to inspect student; whoami does not accept a user operand. whoami — identify the effective user
• Safe with file names Spaces and newlines in file names break a naive xargs. xargs — building arguments from a stream

Files and directories

• Navigation Paths: Basic file commands
• Files and directories rm has no trash can. Basic file commands
• Viewing text Pipes and redirection: Basic file commands
• Terminal and copy-paste Basic file commands
• Back up a config before editing -a keeps the owner and permissions, so the backup can be put back as is. cp — copy files and directories
• A directory into an existing directory, and "into" it The same command gives different results depending on whether dst exists. cp — copy files and directories
• Copy many files into one directory cp — copy files and directories
• Keep the previous version cp — copy files and directories
• Between machines cp works within one machine. cp — copy files and directories
• Examples file — determining file type
• Conditions find — searching for files
• Actions {} + batches arguments (faster), {} \; runs one at a time. find — searching for files
• History: log -p, show, blame {#history} The /etc/demo repository with three commits by two authors: git: change history, undo and a remote repository
• Undo a change and a commit: restore, revert, reset {#revert} What to undo Command History an uncommitted change to a file git restore site.conf unchanged one commit git revert --no-edit <commit> a new commit on top… git: change history, undo and a remote repository
• A root-owned repository: sudo git {#sudo-git} Files in /etc belong to root, and so does a repository in /etc/nginx. git: change history, undo and a remote repository
• A remote repository: fetch, pull, push {#remote} A bare repository is history only, with no working files: git: change history, undo and a remote repository
• Shelve a change: stash {#stash} git stash moves uncommitted changes to a separate store and brings the files back to the last commit — so you can get other people's commits when pull refuses… git: change history, undo and a remote repository
• Conflict {#conflict} A conflict means both sides changed the same lines, and git does not know which version to keep. git: change history, undo and a remote repository
• Branches git: change history, undo and a remote repository
• Examples Argument order is like cp: ln — symbolic and hard links
• Symlink vs hard Symlink (-s) Hard link Points to a directory yes no Across filesystems yes no Target deleted a "broken" symlink data survives as long as any name remains Shown… ln — symbolic and hard links
• Find the most recently changed file ls — list directory contents
• See a directory's own permissions, not its files Without -d the command lists what is inside, and the directory's own permissions stay out of view. ls — list directory contents
• Only directories or only files ls — list directory contents
• Numeric UID and GID instead of names Useful when a file belongs to a deleted user: ls — list directory contents
• Ubuntu's short aliases Ubuntu's ~/.bashrc already defines alias ll='ls -alF', alias la='ls -A' and alias l='ls -CF'. ls — list directory contents
• rmdir rmdir deliberately only removes empty directories — that's a safety net. mkdir — creating directories
• Rename a file in another directory without going there mv — move or rename
• Move all files matching a pattern The shell expands *.log (see globbing). mv — move or rename
• Rename many files: .txt → .md ${f%.txt} cuts off the extension; the quotes keep names with spaces intact. mv — move or rename
• Replace a live file safely Within one filesystem mv replaces the file atomically: mv — move or rename
• In scripts The classic case is finding the script's own directory, so paths don't depend on where it was run from: Paths — basename, dirname, realpath
• Backup and restore restic — deduplicated backups
• Maintenance and scheduling In cron or a systemd timer — backup plus forget --prune on a schedule. restic — deduplicated backups
• Check a pattern before deleting The shell expands the pattern (see globbing), so ls with the same pattern shows exactly the list rm will get. rm — remove files and directories
• Delete old files by age rm can't select by date or size — that's find's job. rm — remove files and directories
• Guard rm -rf against an empty variable ${VAR:?} stops the command if the variable is empty or unset. rm — remove files and directories
• Delete a file with a strange name rm — remove files and directories
• The trailing slash matters! A trailing slash on the source means "contents". rsync — file synchronization and copying
• Commands rsync — file synchronization and copying
• Examples Common -c format fields: stat — file metadata
• Three timestamps atime (%x) — last access (read). stat — file metadata
• Commands -sfn matters when replacing a symlink to a directory: Symbolic and hard links
• Symlink vs hard Symlink (-s) Hard Points to a directory yes no Across filesystems yes no Target deleted "broken" data survives as long as any name remains Shown in ls -l -&gt… Symbolic and hard links
• Commands tar — archiving and extracting files

Text and search

• Write a short file without an editor Quoting EOF turns off expansion: cat — print a whole file
• Find invisible characters ^M$ at line ends means Windows line breaks (\r\n). cat — print a whole file
• Join parts and check the result cat — print a whole file
• Search a file without cat cat file | grep error works, but the extra process isn't needed: cat — print a whole file
• User names and their shells cut — cut fields out of lines
• Addresses from a web server log The first word of an nginx log line is the client address. cut — cut fields out of lines
• A column from a CSV file If CSV values contain commas inside quotes, cut will not understand them: cut — cut fields out of lines
• Everything except one field cut — cut fields out of lines
• Skip lines without the delimiter Without -s, a line with no delimiter is printed whole. cut — cut fields out of lines
• Commands diff and patch — comparing and applying changes
• Examples diff and patch — comparing and applying changes
• Examples grep — searching text
• Investigate stale command resolution {#stale-path} Compare the results. hash — Bash command path cache
• Remove just one entry {#one-entry} The first call remembers the discovered path, the second displays it, and the third removes the entry. hash — Bash command path cache
• Fix the correct shell context {#current-shell} Clear the cache in the shell where the problem occurs. hash — Bash command path cache
• The five largest files in a directory head — the first lines of a file
• The processes using the most memory head — the first lines of a file
• Peek at a file without opening it all head — the first lines of a file
• Lines N to M head — the first lines of a file
• Filters jq — processing JSON on the command line
• Examples and assembly jq — processing JSON on the command line
• Find an error in a long log Search is case-sensitive. less — page through a long file with search
• Keep only the lines you need on screen Inside less, type &Failed and Enter: less — page through a long file with search
• A table with long lines less — page through a long file with search
• Colored output less — page through a long file with search
• Open what you're reading in an editor The v key opens the current file in the default editor — on Ubuntu that's usually nano. less — page through a long file with search
• A word in any case and spelling Regular expressions — patterns for grep, sed and awk
• An IP address Regular expressions — patterns for grep, sed and awk
• A config without comments and empty lines Regular expressions — patterns for grep, sed and awk
• Swap parts of a line Groups ( ) remember parts of the match; \1 and \2 put them into the replacement. Regular expressions — patterns for grep, sed and awk
• Preview a replacement, then edit {#preview} diff shows every changed line before and after: sed and awk — editing and fields in a text stream
• Delete lines sed and awk — editing and fields in a text stream
• Replace a whole line or add one next to it sed and awk — editing and fields in a text stream
• One edit across many files The list of files first, the edit second: sed and awk — editing and fields in a text stream
• Filter and count in awk sed and awk — editing and fields in a text stream
• Which lines occur most often The first sort groups equal addresses together, uniq -c counts them, the second sort -rn puts the largest counts on top, and head keeps ten. sort — sort lines
• A table by a numeric field sort — sort lines
• Sizes with K, M, G -n reads 512M as 512 and 1G as 1, which gives the wrong order. sort — sort lines
• Several keys sort — sort lines
• Versions and numbered file names sort — sort lines
• Write the result back to the same file sort names.txt > names.txt leaves an empty file; -o writes only after reading. sort — sort lines
• Check whether a file is sorted sort — sort lines
• sort / uniq sort, uniq, cut, wc, tr — text stream processing
• cut / tr / wc / more sort, uniq, cut, wc, tr — text stream processing
• Parsing logs and tables sort, uniq, cut, wc, tr — text stream processing
• Follow a log and see only errors --line-buffered makes grep print matches right away instead of in batches. tail — the last lines and following a log
• Skip the header line tail — the last lines and following a log
• The last lines of several logs at once tail — the last lines and following a log
• Stop following when a process ends tail — the last lines and following a log
• Examples tee — output to a file and the screen
• Why sudo tee This is the main reason to know tee. tee — output to a file and the screen
• Top addresses in a web server log This answers "who sent the most requests": uniq — remove or count repeated lines
• How many distinct values uniq — remove or count repeated lines
• Find duplicates in a list uniq — remove or count repeated lines
• Compare only part of the line -f N skips N blank-separated fields; -w N compares the first N characters. uniq — remove or count repeated lines
• How many lines contain an error wc — count lines, words and bytes
• How many files are in a directory ls | wc -l counts too, but skips hidden files, and ls -l | wc -l adds the total line. wc — count lines, words and bytes
• How many distinct values wc — count lines, words and bytes
• Several files at once wc — count lines, words and bytes
• The longest line wc — count lines, words and bytes

Users and permissions

• Separate observation from modification {#inspect-first} First confirm that the name identifies the intended user. chage — password and account expiry
• Set a past date in a training lab {#expire-training-account} Only after labpaused has been prepared and the exercise requests it: chage — password and account expiry
• Check the actual access path {#check-access-path} The chage report shows local settings. chage — password and account expiry
• chmod Two syntaxes: chmod / chown — permissions and ownership
• chown chmod / chown — permissions and ownership
• chgrp chgrp is the same as chown :group, but shorter and more explicit about intent: chmod / chown — permissions and ownership
• umask umask is a mask whose bits the kernel clears from the default permissions when creating a file (usually 666) or a directory (777). chmod / chown — permissions and ownership
• Commands Extended permissions (ACL)
• mask In getfacl output, the mask:: Extended permissions (ACL)
• Numbers and symbols Octal Permissions Typically for 644 rw-r--r-- regular files 600 rw------- secrets, keys 755 rwxr-xr-x scripts, binaries, directories 700 rwx------ private… File permissions
• Special bits The three special bits and a team's shared directory are covered in Special permission bits. File permissions
• Find where a service loses access Run as the service, namei stops exactly at the part it can't pass: namei — permissions at every step of a path
• Compare the parts of a path by hand ls with -d and stat show the same one by one — handy when there are few parts. namei — permissions at every step of a path
• Where a path really leads Without -l, namei shows only types and expands each link with an indent: namei — permissions at every step of a path
• Assign the training account's shell {#set-training-shell} Only for a prepared labsvc, when the exercise requests this change: nologin — a shell that refuses login
• Demonstrate the program's refusal {#demonstrate-refusal} The message may depend on locale and system configuration. nologin — a shell that refuses login
• Test the restriction's boundary {#uid-still-usable} When permitted by sudo policy: nologin — a shell that refuses login
• A temporary password for a new colleague passwd — change or lock a password
• Block login without deleting the account passwd -l closes only password login: passwd — change or lock a password
• Passwords for several users at once chpasswd reads name:password pairs. passwd — change or lock a password
• A team directory Check: Special permission bits: setuid, setgid, sticky
• Fix files that are already there setgid works on new files. Special permission bits: setuid, setgid, sticky
• Find setuid and setgid programs A clean system has a dozen or so: Special permission bits: setuid, setgid, sticky
• Change your mask for good umask — the mask for new files and directories
• A team's shared directory For files to also get the directory's group instead of yours, the directory needs the setgid bit. umask — the mask for new files and directories
• A service's mask A line in the unit or in a drop-in (sudo systemctl edit app) — see systemd. umask — the mask for new files and directories
• Create one file with the right permissions umask — the mask for new files and directories
• A user who can administer the machine On Ubuntu the sudo group grants running commands as root. useradd — create a user
• A system account for a service -r picks a UID below 1000, -M skips the home directory, nologin blocks interactive login. useradd — create a user
• A user with an SSH key instead of a password No password is set, so password login is closed while key login works — see ssh-keys. useradd — create a user
• What ends up in the new home directory With -m the contents of /etc/skel are copied in: useradd — create a user
• Add to a group and check that nothing was lost usermod — change an account and its groups
• Remove from one group The same without gpasswd: usermod — change an account and its groups
• A new group doesn't work in an open session A process gets its groups when it starts. usermod — change an account and its groups
• Close login for a while -L closes only password login: usermod — change an account and its groups
• Management A common mistake. Users and groups
• Passwords, locking, su System users (daemons) usually have uid < 1000, nologin, no password (useradd -r). Users and groups

Processes

• What is this process {#identify} Name, parent, user, full command, working directory, program file and service — enough to tell where the process came from. /proc: processes and the kernel as files
• Find and recover a deleted but open file {#deleted} find prints links like /proc/956/fd/3: /proc: processes and the kernel as files
• A service's environment {#service-env} /proc: processes and the kernel as files
• Shell commands Background jobs in the shell — &, jobs, nohup
• Surviving logout For services, use a systemd unit: Background jobs in the shell — &, jobs, nohup
• Running in parallel For serious parallelism, use xargs -P N (see xargs) or GNU parallel. Background jobs in the shell — &, jobs, nohup
• namespaces Types: cgroups and namespaces — how containers work
• cgroups (v2) CPUQuota=50% is half of one core; MemoryMax= is a hard ceiling: cgroups and namespaces — how containers work
• Slices and CPU sharing {#cpu-weight} systemd places processes in a tree of groups. cgroups and namespaces — how containers work
• Find the process that loads the disk {#find-io-hog} The general order for all resources is in system-monitoring. iostat and pidstat — which disk is busy and which process loads it
• Lower a process's disk priority {#lower-io} nice lowers the CPU share, ionice the disk share; a backup on a busy server needs both. iostat and pidstat — which disk is busy and which process loads it
• Watch one process {#watch-pid} iostat and pidstat — which disk is busy and which process loads it
• End a process properly TERM lets the program close files and remove temporary ones. kill — send a signal to a process
• End by name pkill matches part of the name: kill — send a signal to a process
• A service — through systemctl systemd restarts a service killed with kill if it has Restart= — see systemctl. kill — send a signal to a process
• A process hangs in the terminal Ctrl+C sends INT, Ctrl+Z sends TSTP (stops it and returns the prompt). kill — send a signal to a process
• Load above the CPU count: who is queued {#who-is-running} Two R tasks at 48.8 % share one CPU. Load average: how many tasks wait for the CPU and the disk
• Find what D processes wait for {#find-d} Read the stack bottom up: Load average: how many tasks wait for the CPU and the disk
• Watch the load over time In vmstat, column r is tasks in R (running and waiting for a CPU), b is tasks waiting for the disk. Load average: how many tasks wait for the CPU and the disk
• Common tasks lsof — open files and ports of processes
• The FD column and locks {#fd-column} FD is the descriptor number, then the mode: lsof — open files and ports of processes
• Deleted files: +L1 and DEL {#deleted} A file that was deleted or replaced with a new one lives on while a process holds it — an open unlinked file. lsof — open files and ports of processes
• Add a swap file {#swap-file} To turn swap on at boot, add a line to /etc/fstab: Memory and swap: free, available and the page cache
• Turn swap off {#swapoff} Then remove the line from /etc/fstab and delete the file. Memory and swap: free, available and the page cache
• Drop the cache before a measurement {#drop-caches} This is only for experiments: Memory and swap: free, available and the page cache
• Find whom OOM killed and when {#find} Match the time against the service's journal (journalctl -u <service>) to see what the service was doing before it died. OOM killer: when memory runs out
• A memory limit for a service {#memorymax} The limit goes into a drop-in, a file that extends the unit (see drop-in): OOM killer: when memory runs out
• Protect an important process {#protect} Only root can set a negative adjustment. OOM killer: when memory runs out
• Reproduce OOM safely {#reproduce} tail waits for a newline and keeps all input in memory until it hits the 64 MiB of its group. OOM killer: when memory runs out
• Check whether a program is running -x requires the whole name to match: pgrep — find a process PID by name
• Find the script, not the interpreter All Python scripts share the process name python3; they differ by command line. pgrep — find a process PID by name
• Look at the details of what you found -d, joins the PIDs with commas — the form ps -p accepts, see ps. pgrep — find a process PID by name
• List before pkill pkill takes the same selection options as pgrep but sends a signal — see kill. pgrep — find a process PID by name
• Viewing Processes
• Signals Signal # Meaning TERM 15 graceful shutdown (the default), the process can clean up KILL 9 kill immediately, can't be caught (data may be lost) HUP 1 "hangup"… Processes
• Priority and states {#priority} The STAT column in ps: Processes
• What the STAT letters mean Letter State R running or waiting for a CPU S sleeping, waiting for an event D waiting on disk or network, signals can't interrupt T stopped (Ctrl+Z or a STOP… ps — list processes
• Find a process by name without grep itself ps aux | grep nginx also finds the line of grep nginx itself — hence the "extra" process in the output. ps — list processes
• Custom columns and the parent etime is how long the process has run, ppid is the parent's PID: ps — list processes
• A service's process tree ps — list processes
• Separate two observations {#two-observations} This provides a convenient pause between manual measurements. sleep — delay execution
• Start a short background task {#background-delay} Run the entire example in the same shell: sleep — delay execution
• Interrupt a manual delay {#interrupt-delay} Start sleep 30 in the foreground and press Ctrl+C. sleep — delay execution
• A program exits silently with code 1 {#silent-exit} -o writes the trace to a file, and the program's output stays on screen. strace — which system calls a process makes
• Run as the service user {#as-service-user} root can read any file, so sudo strace shows no permission failures. strace — which system calls a process makes
• Which files a program reads {#which-files} strace — which system calls a process makes
• Where it connects {#network} The address and port are in the connect arguments. strace — which system calls a process makes
• What a process is stuck on {#stuck} A single unfinished line means the process waits in that call: strace — which system calls a process makes
• Summary: which calls and how many {#summary} calls is how many times, errors how many failed, % time the share of time in the kernel. strace — which system calls a process makes
• Load that does not hurt the neighbours {#safe} With nice -n 19 the process takes all the free CPU (97.4 %) and yields it to any process with the normal nice value. stress-ng — timed load on the CPU, memory and disk
• Raise the load average {#raise-load} Three workers on one CPU push the load towards 3. stress-ng — timed load on the CPU, memory and disk
• Get an OOM in a cgroup {#memory-oom} systemd stopped the whole service at the very first OOM (OOMPolicy=stop). stress-ng — timed load on the CPU, memory and disk
• In the background, under control {#background} A shell job dies with the session. stress-ng — timed load on the CPU, memory and disk
• Who eats the CPU and memory ps takes a snapshot, and its %CPU is the average over the whole life of the process. System monitoring: where to start when the server is slow
• A snapshot for a report One file with the time, load, memory and process list — attach it to a ticket or compare it with a similar snapshot later. System monitoring: where to start when the server is slow
• Find what is loading the CPU Open top, press P (sort by CPU) and c (full command line). top — processes and load in real time
• Find what ate the memory Press M to sort by %MEM. top — processes and load in real time
• Stop a process right from top Press k, enter the PID (the top process is offered by default) and a signal: top — processes and load in real time
• A snapshot for a report or ticket -b (batch) prints plain text without redrawing the screen; -n 1 takes one snapshot. top — processes and load in real time
• htop htop is the same idea with color, mouse and scrolling. top — processes and load in real time
• How many files are open and how many are allowed {#count-fds} A descriptor count close to the soft limit means the service will soon get Too many open files. ulimit and process limits: open files, processes, core
• Raise a limit in your shell {#raise-shell} This affects the shell and everything started from it. ulimit and process limits: open files, processes, core
• Who listens on a socket u_str is a stream socket (like TCP), u_dgr a datagram one (like UDP). Unix sockets — processes talking through a file
• Test a socket by hand curl sends an HTTP request into the socket; the name in the URL only goes into the Host header. Unix sockets — processes talking through a file
• Give nginx access This lasts until the application restarts: Unix sockets — processes talking through a file
• The PostgreSQL socket A directory in -h means a socket connection, the same as no -h at all. Unix sockets — processes talking through a file
• Save status before another command {#save-status} Run this in one shell: wait — wait for a child task in Bash
• Distinguish suspension from termination {#termination-only} For your saved PID, use wait -f "$child_pid" when job control is enabled and termination is required. wait — wait for a child task in Bash
• Keep launch and collection together {#same-parent} Start the task and wait for it in the same shell. wait — wait for a child task in Bash

Services and systemd

• Checking BIND9 — DNS server
• A client of your own server {#client} iburst makes the first four polls back to back, so the sync takes seconds instead of minutes. chrony: time synchronization over NTP
• A server for your machines {#server} On node2: chrony: time synchronization over NTP
• Check the sync from a script {#check-script} waitsync <tries> <offset> waits until the clock is in sync and the correction is below the given one (0.5 s here), polling every 10 seconds. chrony: time synchronization over NTP
• Correct the clock now {#step-now} makestep without a fresh measurement changes nothing: chrony: time synchronization over NTP
• cron A crontab line has five time fields plus a command: Cron — the task scheduler
• Other users' crontabs {#user-crontabs} Personal crontabs live in /var/spool/cron/crontabs/, one file per user. Cron — the task scheduler
• Schedule and time zone {#timezone} cron reads the schedule in the machine's time zone: Cron — the task scheduler
• systemd timers A pair of files: Cron — the task scheduler
• Leases and diagnostics DHCP server
• Provisioning (as code) Data sources, dashboards, and alerts are defined in files — reproducible, in git, not through clicking: Grafana — dashboards
• Balancing and routing balance When roundrobin evenly, in turn (the default) leastconn the least-loaded server (long sessions, databases) source sticky by the client's IP Routing by… HAProxy — load balancer
• Three kinds of name systemd distinguishes three host names: hostnamectl — host name and machine info
• Checking a service So the VIP moves when the application fails, not only when the whole node does: keepalived — high availability with VRRP
• Managing VMs KVM and libvirt
• Networks and storage Snapshots: KVM and libvirt
• Who will answer a name The first command shows the answer as a client sees it, the second which listen and server_name nginx actually read, in file order. nginx — web server and reverse proxy
• Find where the proxy breaks The error.log line names the upstream — where nginx went. nginx — web server and reverse proxy
• Relaying through a smarthost The typical "the server sends notifications through an external SMTP" setup (main.cf): Postfix — a mail server
• A role and a database for an application A new password — ALTER ROLE appuser PASSWORD '…';; list roles with \du in psql, databases with \l. PostgreSQL — access, pg_hba.conf, dump and restore
• A password without typing: ~/.pgpass {#pgpass} A line is host:port:database:role:password; * matches any value. PostgreSQL — access, pg_hba.conf, dump and restore
• Dump formats {#dump-formats} pg_dump takes a logical dump — the commands and data the database is rebuilt from. PostgreSQL — access, pg_hba.conf, dump and restore
• Restore into a new database {#restore-new-db} A new empty database has no conflicts: PostgreSQL — access, pg_hba.conf, dump and restore
• Compare a copy with the source {#verify-copy} The same row count, selection checksum and counter value in the source and the copy mean the data matches. PostgreSQL — access, pg_hba.conf, dump and restore
• PromQL and checking status The web UI is on :9090. Prometheus — metrics collection
• redis-cli Redis — in-memory key-value store
• Server Samba — file sharing over SMB
• Client (Linux) Samba — file sharing over SMB
• Check whether a service is running Service
• Start, stop, and enable at boot Service
• See what the service has logged Service
• Control reload vs restart: systemctl — managing systemd services
• Viewing and units edit creates a drop-in at /etc/systemd/system/<unit>.d/override.conf — the packaged unit is untouched, updates won't overwrite it. systemctl — managing systemd services
• Targets A target is a group of units systemd brings the boot up to: systemd
• Move a job from cron to a timer The service: systemd timers — scheduled jobs
• Run the job now and check the result For oneshot, systemctl start waits for the script to finish and returns 0 or 1. systemd timers — scheduled jobs
• An hour after the previous run systemd timers — scheduled jobs
• A temporary timer without files Such a timer lasts until reboot; stop it with systemctl stop tick.timer. systemd timers — scheduled jobs
• Try a script in the service environment The script runs as a temporary service: systemd timers — scheduled jobs
• Commands timedatectl — time, time zone, and synchronization
• Time zone {#timezone} The zone changes only how the time is shown: timedatectl — time, time zone, and synchronization
• chrony instead of timesyncd {#chrony} Installing chrony removes systemd-timesyncd. timedatectl — time, time zone, and synchronization

Network

• Commands conntrack — connection tracking
• A status code for checks The first command prints the code; 000 means there was no HTTP response. curl — HTTP requests from the console
• A site by name without DNS -H 'Host: curl — HTTP requests from the console
• HTTPS with your own CA --cacert adds a trusted root for one request. curl — HTTP requests from the console
• A request into a unix socket The connection goes to the socket file; the name localhost only goes into Host. curl — HTTP requests from the console
• Data and headers With -d curl picks POST itself; -X POST is optional. curl — HTTP requests from the console
• dig / host / nslookup DNS and name resolution
• Commands The interface name comes from ip -br l. ethtool — network card settings
• Just the status code -o /dev/null throws the body away, -w prints the code. HTTP — requests, responses, status codes and headers
• A site by name when DNS does not know it The first command changes only the Host header. HTTP — requests, responses, status codes and headers
• A request by hand This shows HTTP is plain text: HTTP — requests, responses, status codes and headers
• Who answered: nginx or the application An access.log line holds the client address, time, request line, code and response size: HTTP — requests, responses, status codes and headers
• ip objects Object About Common ip address (a) IP addresses on interfaces ip -br a ip link (l) the interfaces themselves (up/down, MTU, MAC) ip -br l ip route (r) the… ip — addresses, interfaces and routes
• Commands Changes made with ip ... ip — addresses, interfaces and routes
• Address, prefix and subnet 192.0.2.10/24 is an address and a prefix: ip — addresses, interfaces and routes
• Network diagnostics The natural ladder: ip — addresses, interfaces and routes
• Checking ports Distinguish the responses: nc — checking ports and transferring data
• Data and transfer nc — checking ports and transferring data
• ping In the output, look at: Network diagnostics
• traceroute / mtr mtr is the main tool: Network diagnostics
• What the machine shows to the outside {#audit} A port that listens on 0.0.0.0 and shows as open from outside is part of the attack surface. nmap: which ports are visible from outside
• Output for a script {#grepable} Each port is number/state/protocol//name///: nmap: which ports are visible from outside
• Check one port {#one-port} nmap: which ports are visible from outside
• Applying changes Over SSH on a remote machine, change network configuration with netplan try: Persistent networking with netplan
• Several files under /etc/netplan All the files are applied together, in name order (01-..., 50-...); a later file can override an earlier one's settings for the same interface. Persistent networking with netplan
• A permanent route to a network behind a neighbour {#static-route} A route added with ip route add lasts until a reboot. Persistent networking with netplan
• Tell "no path" from "no reply" Network is unreachable is printed at once: ping — check whether packets reach a host
• Destination Host Unreachable A host in your subnet didn't answer ARP — it's missing or off. ping — check whether packets reach a host
• A check in a script ping — check whether packets reach a host
• Inside the training network The training machine has no network except lo: ping — check whether packets reach a host
• Commands ss — sockets and ports
• States and filters Filter by TCP state — state <name> (or state connected/listening): ss — sockets and ports
• Generation Installing on the server: SSH keys and ssh-agent
• File permissions sshd enables StrictModes by default: SSH keys and ssh-agent
• ssh-agent The agent keeps the decrypted key in memory so you don't have to type the key's passphrase every time: SSH keys and ssh-agent
• Copying and tunnels ssh, scp — connecting and copying over SSH
• Populate known_hosts for a new server -H hashes the hostname in the file — the same thing ssh itself does on a first connection (see ssh-client). ssh-keyscan — collecting host keys
• Several hosts from a list Handy before rolling out a group of VMs, or in cloud-init/an Ansible playbook, when known_hosts is built in one step for a whole fleet. ssh-keyscan — collecting host keys
• Verify a fetched key against a fingerprint ssh-keyscan — collecting host keys
• Filters BPF syntax: tcpdump — capturing and analyzing network traffic
• Build a two-node network The ends' addresses come from one subnet; otherwise a route is needed (see ip). The veth pair and network namespaces
• Find the other end of a pair The veth pair and network namespaces
• Remove the training network A veth pair doesn't live alone: The veth pair and network namespaces
• netplan: VLAN and bond VLANs and interface bonding
• Checking VLANs and interface bonding
• Commands wget — downloading files over the network
• Common tasks Paired with DNS: whois — domain and IP data
• Investigating an attack A typical path for "where is this coming from" (combined with fail2ban/journalctl logs): whois — domain and IP data

Storage and file systems

• A line for fstab blkid — filesystem UUID, label and type
• Compare an fstab line with the device If it does not, fix fstab. blkid — filesystem UUID, label and type
• Find a device by label or UUID blkid — filesystem UUID, label and type
• What is on a disk with no filesystem blkid — filesystem UUID, label and type
• Limits and reports Disk quotas
• xfs quotas xfs works differently — quotas are turned on with a mount option, not quotacheck: Disk quotas
• df / du / ncdu -x/-xdev is the key to a clean result: Disk space: df, du, inode
• What to clean up Disk space: df, du, inode
• Partition an empty disk in one command The partition starts at 1 MiB: fdisk and parted — partition a disk
• Two partitions on one disk fdisk and parted — partition a disk
• Add a partition in free space No mklabel here: fdisk and parted — partition a disk
• The same in fdisk, step by step For the end you can type a size such as +10G instead of Enter. fdisk and parted — partition a disk
• An MBR table for an older system On MBR the first word after mkpart is the partition type (primary), not a name. fdisk and parted — partition a disk
• The kernel does not see the new partitions fdisk and parted — partition a disk
• Creating and labeling Mounting via /etc/fstab is best done by UUID (/dev/sdX names can shift). Filesystems: ext4, xfs, fsck
• Tuning and info (ext4) Filesystems: ext4, xfs, fsck
• Checking (fsck) Never run fsck/xfs_repair on a mounted filesystem — it can corrupt data. Filesystems: ext4, xfs, fsck
• A practice disk with a partition Loop device — a file as a disk
• A filesystem image without partitions Loop device — a file as a disk
• Clean up afterwards Loop device — a file as a disk
• Attach an image at boot An /etc/fstab line with the loop option — mount attaches the file at boot. Loop device — a file as a disk
• Find a new disk before partitioning it lsblk — block devices
• Mountpoints and UUIDs for fstab lsblk — block devices
• Custom columns for scripts lsblk — block devices
• Commands by layer Layer Create Show Extend PV pvcreate /dev/sdb pvs — VG vgcreate data /dev/sdb vgs vgextend data /dev/sdc LV lvcreate -L 10G -n vol1 data lvs lvextend -r -L +5G… LVM logical volumes
• Creating and extending LVM logical volumes
• Snapshots and moving data LVM logical volumes
• A new data partition mkfs — create a filesystem
• Find a partition by its label mkfs — create a filesystem
• Less root-only space on a big disk By default ext4 keeps 5% of the space for root only. mkfs — create a filesystem
• Lots of small files The number of inodes is fixed at creation. mkfs — create a filesystem
• A filesystem inside a file More in Loop device. mkfs — create a filesystem
• Mounting Common -o options: Mounting and fstab
• Bind mounts and tmpfs Mounting and fstab
• A new disk in fstab: UUID and nofail {#nofail} nofail is for data disks. Mounting and fstab
• Server /etc/exports (one line per export): NFS network filesystem
• Client NFS network filesystem
• Overlays and conversion qemu-img — virtual machine disk images
• Snapshots (internal) For running VMs, take snapshots through virsh snapshot-* (see kvm-libvirt), not qemu-img against a live image. qemu-img — virtual machine disk images
• ext4: resize2fs resize2fs / xfs_growfs — growing a filesystem
• xfs: xfs_growfs resize2fs / xfs_growfs — growing a filesystem
• Both steps at once resize2fs / xfs_growfs — growing a filesystem
• Creating an array Software RAID with mdadm
• Replacing a disk and growing the array Software RAID with mdadm
• Monitoring Software RAID with mdadm

Logs and audit

• Commands journalctl — reading the systemd journal
• Your own lines in the journal: logger and -t {#logger} logger -t tag writes a line to the journal under that tag, and journalctl -t tag shows only those lines. journalctl — reading the systemd journal
• Commands Log rotation with logrotate
• Centralization and rate limiting Receiver (central log server) — enable the input module: rsyslog — system logs

Boot

• Change the default target {#set-default} The target changes at the next boot. Boot targets: default, multi-user, rescue and emergency
• Who starts a unit at boot {#who-wants} Boot targets: default, multi-user, rescue and emergency
• Boot into rescue or emergency {#boot-into-rescue} In the GRUB menu select an entry and press e. Boot targets: default, multi-user, rescue and emergency
• Fix things from emergency and boot on {#fix-from-emergency} mount -a without errors means fstab is fine, and systemctl default takes the boot to the default target. Boot targets: default, multi-user, rescue and emergency
• Debugging cloud-init — first-boot VM setup
• Kernel errors of this boot {#errors-this-boot} dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• Follow the messages {#follow} -w first prints the whole buffer — hundreds of boot lines — and then waits. dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• A crashed program: segfault {#segfault} The program touched memory that is not its own. dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• An ext4 filesystem error {#ext4-error} ext4 found damage: dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• OOM: a process ended for lack of memory {#oom} The lines Out of memory: dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• A task hung in D {#hung-task} The process stayed in state D longer than kernel.hung_task_timeout_secs (120 seconds on Ubuntu; here the threshold was lowered to 20). dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• Commands Kernel modules
• Autoload and blacklist Kernel modules
• dmesg / kernel messages This is where driver and hardware problems show up. Kernel modules
• Find a parameter {#find} 2>/dev/null hides the permission denied on key lines; with sudo all parameters are visible. sysctl — kernel parameters at runtime
• Check that a value survives a reboot {#check-persistent} If sysctl --system gives the right value, boot will give the same: sysctl — kernel parameters at runtime
• Who set the value {#who-set} sysctl --system first prints all the * Applying … lines in order, then all the values. sysctl — kernel parameters at runtime
• Return the default value {#revert} Remove your file from /etc/sysctl.d/ and set the value by hand (sudo sysctl -w vm.swappiness=60) or reboot — sysctl --system does not restore old values, it… sysctl — kernel parameters at runtime
• GRUB The config is /etc/default/grub (don't edit the generated grub.cfg directly): System boot and GRUB
• Targets and timing Target What it is multi-user.target a server with no graphics (normal) graphical.target + GUI rescue.target single-user, root mounted, minimal services… System boot and GRUB
• Find what delays the boot {#slow-boot} systemd-analyze — how long the boot took and to which target. systemd-analyze: what slows the boot and what is wrong in a unit
• Check a unit before starting it {#check-unit} The last line suits a script: systemd-analyze: what slows the boot and what is wrong in a unit
• Did the service come up at boot by itself {#started-at-boot} ActiveEnterTimestampMonotonic is the microseconds from the kernel start to the moment the service became active: systemd-analyze: what slows the boot and what is wrong in a unit
• Check the version and origin The kernel
• Parameters the kernel booted with The kernel
• Kernel messages at boot and at runtime The kernel

Packages

• apt apt / dpkg — package management
• dpkg and repositories apt / dpkg — package management
• Commands snap — Ubuntu packages
• Snap services snap — Ubuntu packages

Security

• Modes and commands {#modes} Profiles live in /etc/apparmor.d/. AppArmor
• Profiles and rollout A profile's structure (simplified): AppArmor
• Rules: watch and syscall auditd — security auditing
• Searching events auditd — security auditing
• Reading an event {#event} On the lab machine the auditd service is off: auditd — security auditing
• Permanent rules: rules.d {#rules-d} augenrules builds /etc/audit/audit.rules from all the rules.d/*.rules files and loads it into the kernel; auditd does the same at start. auditd — security auditing
• A port below 1024 without root {#low-port} A service gets the part through its unit; a program outside systemd gets it through its file: Capabilities: root split into parts
• Find files with capabilities {#find} A clean machine shows the three files from the block above. Capabilities: root split into parts
• Which parts a service has {#service-caps} Capabilities: root split into parts
• Management fail2ban
• ufw Firewall — nftables / ufw
• A persistent nftables set {#nftables-persistent} Rules added with nft add live until a reboot. Firewall — nftables / ufw
• Encryption and signing GPG — encryption and signatures
• Password policy {#password-policy} Test it by action: PAM: login checks, password policy and lockout
• Lockout after three failures {#lockout} The file is replaced whole — compare its old stack with this one (grep -v '^#' /etc/pam.d/common-auth | grep .) and carry over any lines missing here. PAM: login checks, password policy and lockout
• Edit PAM without locking yourself out {#safe-edit} Open a second terminal with sudo -i and keep it until the check is done. PAM: login checks, password policy and lockout
• Restrict a service with a drop-in {#drop-in} Check every setting by the service's work: Service hardening: User=, ProtectSystem, NoNewPrivileges
• Try a setting on one command {#systemd-run} systemd-run runs a command as a temporary service with the same settings: Service hardening: User=, ProtectSystem, NoNewPrivileges
• Hardening Parameter Why PermitRootLogin no don't allow root to log in directly PasswordAuthentication no keys only (the main defense against brute force) MaxAuthTries 3… sshd — the SSH server
• Commands sshd — the SSH server
• Reading and rules iptables rules don't persist on their own. The classic iptables firewall
• What the server sends {#server-certificate} The first certificate in the output is the site certificate. TLS and certificates — openssl, the chain, verification
• Check a chain locally The first command, without the intermediate, says error 20 at 0 depth lookup: TLS and certificates — openssl, the chain, verification
• Build a full chain file {#fullchain} The order is the site certificate first, the intermediate after it. TLS and certificates — openssl, the chain, verification
• Do the key and certificate match? Equal sums mean a pair. TLS and certificates — openssl, the chain, verification
• Issue a certificate with the old key {#renew-same-key} -copy_extensions copy carries SAN from the request into the certificate. TLS and certificates — openssl, the chain, verification
• How long until it expires -checkend N asks whether the certificate expires within N seconds (2592000 is 30 days): TLS and certificates — openssl, the chain, verification
• Trust your own CA system-wide {#trust-own-ca} The file must be named *.crt. TLS and certificates — openssl, the chain, verification
• Let's Encrypt (certbot) A public CA for sites on the internet: TLS and certificates — openssl, the chain, verification
• Is it on and when does it run {#check} Both values "1" and both timers enabled mean updates run once a day. unattended-upgrades: security updates with no one at the keyboard
• Hold a package back {#blacklist} The names are regular expressions: unattended-upgrades: security updates with no one at the keyboard
• Find services running old files {#find-stale} unattended-upgrades: security updates with no one at the keyboard
• Grant one command in a separate file A separate file is easy to see and easy to remove, and the main /etc/sudoers stays untouched. visudo — edit sudoers safely
• Check a rule before installing it Check the grammar first, then the meaning: visudo — edit sudoers safely
• Check as the person the rule is for -n forbids the password prompt: visudo — edit sudoers safely

Automation

• Check a playbook before running it {#before-run} On a correct playbook --syntax-check prints only playbook: ansible-playbook: running a playbook
• To another machine over SSH {#remote} The inventory node2.ini: ansible-playbook: running a playbook
• Part of a playbook {#partial} With -t page the fact gathering, the tagged task and its handler ran — ok=3 changed=2. ansible-playbook: running a playbook
• A variable from the command line {#extra-vars} -e site_title=Changed overrides the playbook's vars:: ansible-playbook: running a playbook
• Inventory {#inventory} What Where Means machine name node2 how the machine is named in the output; Ansible also connects by it if there is no ansible_host ansible_host the machine's… Ansible: inventory, modules and one-off commands
• One-off commands {#adhoc} ansible <machines> -m <module> -a '<arguments>' runs one module on all machines of a group. Ansible: inventory, modules and one-off commands
• become: through sudo {#become} Modules run as the user Ansible logged in as. Ansible: inventory, modules and one-off commands
• Playbooks and roles Variables go in group_vars/, host_vars/; secrets in ansible-vault (ansible-vault encrypt secrets.yml, run with --ask-vault-pass). Ansible: inventory, modules and one-off commands

Containers

• Which image a container runs The first inspect field is the name the container was created with, the second is the image ID. Container images — layers, tags, registries and digests
• Check a digest by hand index.json points to the manifest by digest, the manifest file is named with that digest, and sha256sum of it gives the same number. Container images — layers, tags, registries and digests
• Move an image to a host without network Container images — layers, tags, registries and digests
• Image list for a script Container images — layers, tags, registries and digests
• Free up space Container images — layers, tags, registries and digests
• Validate the file before starting docker compose — a container stack in one file
• Stack state for a script --format json prints one JSON object per line for each container — jq parses them one by one. docker compose — a container stack in one file
• Stop one service docker compose — a container stack in one file
• Logs of all services at once Each line starts with the service name. docker compose — a container stack in one file
• Why the container stopped Code 0 means the command simply finished: Docker — containers: running, ports, volumes, networks
• Re-create a container with new options Data in the volume survives re-creation; data in the container layer does not. Docker — containers: running, ports, volumes, networks
• Find a container among host processes Docker — containers: running, ports, volumes, networks
• A command in a container from a script docker exec -it needs a terminal. Docker — containers: running, ports, volumes, networks
• A site on nginx:alpine CMD is inherited from the base: Dockerfile — an image recipe and docker build
• A process that is not root adduser -D is the busybox version: Dockerfile — an image recipe and docker build
• Check the image before running the service Dockerfile — an image recipe and docker build
• Rebuild and swap the container A new image does not reach a running container by itself — the container has to be re-created. Dockerfile — an image recipe and docker build