Error → fix

Terminal messages

• nginx rejects the configuration nginx: configuration file … test failed nginx -t prints the full error with the file and the line number. nginx — web server and reverse proxy
• No permission for the file or directory permission denied Check the owner and mode with ls -l, and use sudo when root is required. chmod / chown — permissions and ownership
• systemd does not know this unit could not be found Check the name with systemctl list-units --type=service; a changed file needs daemon-reload. systemctl — managing systemd services
• Command not found command not found Check for a typo and the path with which or type. The package may be absent. which / type / whereis — where a command lives
• No such file or directory no such file or directory Check the path with ls and pwd. A relative path starts from the current directory. Basic file commands
• The service failed to start job for … failed because The reason is in systemctl status <service> and journalctl -u <service> -n 50. journalctl — reading the systemd journal
• The port is already in use address already in use ss -tlnp shows who holds the port. Stop it or take another port. ss — sockets and ports
• No space left on the device no space left on device df -h shows free space, du -xh --max-depth=1 / shows the big directories. Disk space: df, du, inode
• mount does not recognise the file system wrong fs type Most often the device has no file system yet, and lsblk -f shows an empty FSTYPE. Mount the partition, not the whole disk. Mounting and fstab
• The disk has no partition table unrecognised disk label This is how a new disk looks. parted mklabel creates the table and erases any old one, so check the disk name with lsblk. fdisk and parted — partition a disk
• The file was not attached as a loop device failed to set … loop device Only root can attach loop devices, so run it with sudo. losetup -a shows what is already attached. Loop device — a file as a disk
• No access to the Docker service permission denied while trying to connect to the docker The Docker socket is open to root and the docker group. Run the command as sudo docker …. Docker — containers: running, ports, volumes, networks
• The Docker service is not running cannot connect to the docker daemon Start it with sudo systemctl enable --now docker; systemctl status docker shows its state. Docker — containers: running, ports, volumes, networks
• The host port is already taken port is already allocated sudo ss -tlnp shows who holds the port. The container was already created — remove it with docker rm and start again with another left-hand port in -p. Docker — containers: running, ports, volumes, networks
• The old Docker builder runs the build legacy builder is deprecated It is a warning; the build goes on. The error, if any, is in the last lines of the output. Dockerfile — an image recipe and docker build
• The image has no such program to run executable file not found in $path A common cause is an exec-form CMD written as one string. Split the words — ["httpd", "-f"] — and check that the image has the program. Dockerfile — an image recipe and docker build
• The kernel buffer is closed to regular users read kernel buffer failed Ubuntu sets kernel.dmesg_restrict = 1 — read sudo dmesg -T or sudo journalctl -k. dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• The process hit its open files limit too many open files Compare Max open files in /proc/<PID>/limits with the files in /proc/<PID>/fd; a service gets a higher limit from LimitNOFILE in a drop-in. ulimit and process limits: open files, processes, core
• The kernel parameter was not changed — root is required permission denied on key The line with the new value was printed, but the value is unchanged. Repeat with sudo and check sysctl <name>. sysctl — kernel parameters at runtime
• Any user can read the swap file insecure permissions Swap turns on, but narrow the permissions — sudo chmod 600 <file>. Memory and swap: free, available and the page cache
• strace cannot attach to the process attach: ptrace On Ubuntu only your own children can be traced without root — sudo strace -p <PID>. strace — which system calls a process makes
• systemd skipped a unit line unknown key name The key is in the wrong section or misspelt — check systemd-analyze verify <unit>. systemd
• The new password failed the quality check bad password: The reason follows the colon (length, character classes, dictionary); pwscore helps pick a password first. PAM: login checks, password policy and lockout
• The password was not changed — no tries left have exhausted maximum number of retries The quality check rejected every try (retry=3). Read the BAD PASSWORD reasons above and check a password with pwscore. PAM: login checks, password policy and lockout
• The account is locked after failed logins the account is locked due to See the counter with sudo faillock --user <name>; lift the lock with --reset or wait for unlock_time. PAM: login checks, password policy and lockout
• pam-auth-update leaves hand-edited files alone local modifications to /etc/pam.d The profile is not enabled. --force rebuilds common-* and wipes manual lines — add them again afterwards. PAM: login checks, password policy and lockout
• sudo does not work inside a service with NoNewPrivileges "no new privileges" flag is set The no_new_privs flag forbids raising rights. Give the service what it needs through User=, a group or a capability. Service hardening: User=, ProtectSystem, NoNewPrivileges
• The service did not start — the User= user does not exist status=217/user Create the user with useradd --system or set DynamicUser=yes. Service hardening: User=, ProtectSystem, NoNewPrivileges
• setcap ran without root unable to set cap_setfcap Only root can give capabilities to files — repeat with sudo. Capabilities: root split into parts
• This scan type requires root scan type which requires root SYN, UDP and OS scans send raw packets — run with sudo or keep the plain scan without -sS. nmap: which ports are visible from outside
• nmap thinks the host is down host seems down The host did not answer the discovery probes. Check the address and repeat with -Pn to scan ports without the check. nmap: which ports are visible from outside
• AppArmor denied the program access apparmor="denied" The line names the profile (profile), the path (name) and what was missing (denied_mask). Add the rule to the profile's local file. AppArmor
• The PostgreSQL setting applies only after a restart cannot be changed without restarting the server After reload the old value stays (pending_restart = t). Run sudo systemctl restart postgresql. PostgreSQL — access, pg_hba.conf, dump and restore
• Another user's crontab needs root must be privileged to use -u crontab -u <user> works only with sudo: sudo crontab -l -u <user>. Cron — the task scheduler
• apt waits for the scheduled update to finish (unattended-upgr) unattended-upgrade holds the dpkg lock. Wait for it (systemctl status apt-daily-upgrade); do not kill it. unattended-upgrades: security updates with no one at the keyboard
• The boot has not finished yet bootup is not yet finished Some boot job is still running — systemctl list-jobs shows which. systemd-analyze: what slows the boot and what is wrong in a unit
• Units wait for each other in a circle transaction order is cyclic Two units have After= on each other — the journal shows Found ordering cycle; remove the extra After=. systemd-analyze: what slows the boot and what is wrong in a unit
• The machine booted into emergency mode you are in emergency mode Most often an /etc/fstab line with a missing device. The cause is in journalctl -xb. Boot targets: default, multi-user, rescue and emergency
• chronyc cannot reach chronyd 506 cannot talk to daemon The service is not running — systemctl status chrony. chrony: time synchronization over NTP
• This chronyc command is for root only 501 not authorised makestep, burst, reload sources change the service — run them with sudo. chrony: time synchronization over NTP
• The time cannot be set by hand while sync is on automatic time synchronization is enabled First sudo timedatectl set-ntp false, after the experiment — set-ntp true. timedatectl — time, time zone, and synchronization
• systemd-timesyncd is not running org.freedesktop.timesync1 If chrony is installed, timesyncd is removed — chronyc tracking shows the state. chrony: time synchronization over NTP
• The repository belongs to another user detected dubious ownership For a root-owned repository use sudo git. git: change history, undo and a remote repository
• git has no author name and email author identity unknown Set them in the repository — git config user.name and git config user.email (with sudo for a root-owned one). git: change history, undo and a remote repository
• pull refused because of a local change would be overwritten by merge Commit the change or shelve it with git stash, then git pull and git stash pop. git: change history, undo and a remote repository
• push rejected — the remote has commits you do not (fetch first) … [rejected] git pull first, then git push again. git: change history, undo and a remote repository
• The branches diverged, git does not know how to reconcile them reconcile divergent branches Merge with a merge commit — git pull --no-rebase. git: change history, undo and a remote repository
• A conflict — both sides changed the same lines conflict (content) Keep what is needed between the <<<<<<</>>>>>>> markers, remove the markers and run git add <file>. git: change history, undo and a remote repository
• The inventory has no such machines could not match supplied host pattern The inventory was not read or lacks the group from hosts: — check the -i path and --list-hosts. ansible-playbook: running a playbook
• Ansible cannot find the module couldn't resolve module/action A typo in the module name — ansible-doc <module> shows whether it exists. ansible-playbook: running a playbook
• notify calls a handler that does not exist the requested handler … was not found The name in notify must match the handler's name letter for letter. ansible-playbook: running a playbook
• A template or task uses a variable that does not exist ansibleundefinedvariable Set it in vars:, the inventory or -e, or add | default(...). ansible-playbook: running a playbook
• Ansible could not connect to the machine over SSH unreachable! … failed to connect to the host via ssh The cause is the ssh text in msg; check by hand with ssh <machine> true as the same user. Ansible: inventory, modules and one-off commands
• The module lacks the rights destination … not writable The module is not running as root — add -b or become: true. Ansible: inventory, modules and one-off commands
• systemd dropped a job to break a cycle deleted to break ordering cycle Two units wait for each other through After= — one service does not start. Jobs are in systemctl list-jobs. systemd-analyze: what slows the boot and what is wrong in a unit

Basic commands

• The alias is gone after reopening the terminal You created it on the command line but never saved it to ~/.bashrc. alias — command shortcuts
• alias does not work in a script / under sudo That is expected: alias — command shortcuts
• I need to temporarily run the real command The \ prefix (\rm file) or command rm file runs the original, bypassing the alias. alias — command shortcuts
• The job did not run The atd daemon must be running (systemctl status atd). at — a one-time scheduled job
• The command "was not found" inside at Like cron, at has a minimal environment (though at does save the current env at the time the job is queued — at -c ID shows it). at — a one-time scheduled job
• I need to cancel it atq (find the id) → atrm <id>. at — a one-time scheduled job
• at: command not found / no atd The at package isn't installed or the daemon isn't running: at — a one-time scheduled job
• One-off vs. recurring vs. a service One-off — at; on a schedule — cron/systemd timers; a permanent service — a systemd unit. at — a one-time scheduled job
• Permission denied {#permission-denied} bash: Bash scripts: running, variables, arguments
• command not found {#command-not-found} backup.sh: Bash scripts: running, variables, arguments
• cannot execute: required file not found {#crlf} bash: Bash scripts: running, variables, arguments
• [[: not found, Syntax error: "(" unexpected {#sh-vs-bash} A bash script was run by sh (dash): Bash scripts: running, variables, arguments
• name: command not found on a line with = {#assignment-spaces} Spaces around =: Bash scripts: running, variables, arguments
• unbound variable {#unbound-variable} backup.sh: Bash scripts: running, variables, arguments
• syntax error: unexpected end of file {#unexpected-eof} A block or a quote is not closed: Bash scripts: running, variables, arguments
• The script breaks on a name with a space {#spaces-in-names} An unquoted variable: Bash scripts: running, variables, arguments
• Works by hand, fails on a schedule {#schedule} Cron and systemd services have a different environment: Bash scripts: running, variables, arguments
• command not found {#missing-command} Check spelling and command lookup. bash — the command shell
• A setting exists in a file but is not active {#settings-missing} Identify the session mode and relevant startup file. bash — the command shell
• History is empty or differs between sessions {#history-empty} History belongs to a shell session and its configuration. bash — the command shell
• cd: too many arguments cd takes one path. cd — changing directory
• cd: not a directory The target path is a file, not a directory. cd — changing directory
• cd: permission denied No x (enter) permission on the directory. cd — changing directory
• "I cd'd inside a script, and afterward I'm back in the old directory" Every process/script has its own working directory. cd — changing directory
• Spaces/colons ended up in a filename date prints with spaces/: date — date and time in the shell
• The time is wrong / the wrong timezone date uses the system timezone. date — date and time in the shell
• Month/weekday names are in English but I need Russian It depends on the locale (LC_TIME). date — date and time in the shell
• I need a timestamp in a script's logs printf '%s %s\n' "$(date -Iseconds)" "message", or use the systemd journal (journalctl adds timestamps itself). date — date and time in the shell
• -e printed as literal text The shell isn't bash (behavior differs in sh/dash), or echo is an external binary. echo — printing text
• An asterisk turned into a file list echo * without quotes is subject to globbing. echo — printing text
• The newline isn't where I want it echo always adds a trailing \n; to skip it, use -n; to insert one in the middle, use -e "...\n..." or printf. echo — printing text
• command not found, even though the binary exists Its directory isn't in $PATH. Environment variables and PATH
• export worked, but disappeared in a new terminal export only lives in the current session. Environment variables and PATH
• The variable is set in the terminal but not over SSH (or vice versa) Different files for login/non-login (see the table). Environment variables and PATH
• A script/cron/systemd doesn't see the variable cron and systemd units do not read your ~/.bashrc. Environment variables and PATH
• sudo "loses" variables sudo resets the environment (env_reset). Environment variables and PATH
• The script prints OK while the service is down {#ok-but-down} The check reads the code of the wrong command: Exit codes: $?, && and ||
• A line in the middle failed, yet the script returned 0 {#last-command} A script returns the code of its last command. Exit codes: $?, && and ||
• The script did its job but returned 1 {#last-test} The same cause from the other side. Exit codes: $?, && and ||
• Code 127 {#code-127} command not found: Exit codes: $?, && and ||
• Code 126 {#code-126} The file was found but cannot run: Exit codes: $?, && and ||
• Code 130, 137 or 143 {#signal-codes} A signal stopped the command: Exit codes: $?, && and ||
• grep stops a script that uses set -e {#grep-no-match} grep returns 1 when nothing matches. Exit codes: $?, && and ||
• cannot open lock file: Permission denied {#permission-denied} The lock file cannot be opened. flock — preventing a second run
• The second run silently vanished {#silent-skip} That is how -n behaves: flock — preventing a second run
• failed to execute: No such file or directory {#failed-to-execute} flock could not find the command (code 69). flock — preventing a second run
• The lock does nothing, runs still overlap {#no-effect} The runs lock different files, or one of them bypasses flock. flock — preventing a second run
• man: the command exists, but there's no page It might be a bash builtin — try help cmd. Getting help at the terminal
• apropos/man -k: "nothing appropriate" The index hasn't been built. Getting help at the terminal
• The man page is too long, you need one flag man cmd, then /-X (search for the flag, n for the next match). Getting help at the terminal
• You don't remember the command's name, only the task apropos <topic> / man -k <topic> searches every page's description. Getting help at the terminal
• You want help with bash syntax (if/for/test) Those are shell builtin constructs: Getting help at the terminal
• The pattern found no files — the command got the literal pattern By default bash leaves an unmatched glob as-is (ls *.xyz → "no such file: Glob patterns and brace expansion
• Hidden files didn't get caught by * * doesn't include files starting with .. Glob patterns and brace expansion
• Spaces/special characters in names break things Quote your variables; for bulk processing, use find -print0 | xargs -0 (see xargs). Glob patterns and brace expansion
• {a,b} didn't expand There must be no spaces inside: Glob patterns and brace expansion
• A short name works but the FQDN does not {#fqdn-fails} Check how the system resolver handles the current name. hostname — machine names and resolution
• hostname -I returned several addresses {#many-addresses} A machine may have several interfaces and addresses. hostname — machine names and resolution
• The old name returned after reboot {#name-reverts} A temporary change does not replace persistent configuration. hostname — machine names and resolution
• invalid option -- 'z' / unrecognized option '--foo' The command has no such option. How a command is built — options and arguments
• missing operand The command lacks an argument: How a command is built — options and arguments
• No such file or directory for half a name The shell split a name with a space into two arguments. How a command is built — options and arguments
• command not found The first word isn't a command the shell knows: How a command is built — options and arguments
• The quotes aren't closed — the prompt changed to > The shell waits for the closing quote. How a command is built — options and arguments
• Garbled Cyrillic/characters A non-UTF-8 locale or terminal. Locale and encoding
• warning: setlocale: LC_*: cannot change locale The requested locale hasn't been generated. Locale and encoding
• A script sorts/parses differently on different machines The locale affects sort, numbers, dates. Locale and encoding
• Dates/numbers in an "unexpected" format That's LC_TIME/LC_NUMERIC. Locale and encoding
• perl: warning setting locale failed Usually harmless but annoying: Locale and encoding
• mc: command not found It's not installed: mc — a terminal file manager
• Garbled panel borders The terminal isn't UTF-8, or the font is missing the glyphs. mc — a terminal file manager
• Ctrl+O does nothing / the shell is empty Ctrl+O shows the subshell. mc — a terminal file manager
• The mouse selects text instead of the panels Hold Shift while selecting with the mouse — that hands the selection to the terminal instead of mc. mc — a terminal file manager
• You saved, but the change didn't take effect for the service The editor only writes the file. nano — a simple console editor
• No permission to save You didn't open it with sudo. nano — a simple console editor
• Long-line wrapping is confusing (configs) nano -w file (or set nowrap) turns off soft wrapping, so long config lines don't look visually "broken". nano — a simple console editor
• Mouse paste breaks / duplicates Terminal paste can be slow with large blocks; for configs this is usually fine. nano — a simple console editor
• More used to vim For servers, know at least the minimum of vim — it's always there; nano sometimes isn't installed (sudo apt install nano). nano — a simple console editor
• Permission denied with sudo … > file Your shell opens the file, not sudo. Output redirection and pipelines
• bash: $F: ambiguous redirect The file name in the variable has a space, or the variable is empty. Output redirection and pipelines
• bash: out.txt: cannot overwrite existing file noclobber mode is on (set -o noclobber); it forbids > onto an existing file. Output redirection and pipelines
• Errors didn't end up in the file > redirects stdout only. Output redirection and pipelines
• grep in a pipeline doesn't find the error text The error went to stderr, and a pipe passes only stdout. Output redirection and pipelines
• The file is empty after sort file > file The shell truncated file for writing before sort read it. Output redirection and pipelines
• echo mangled a string with a dash or a backslash echo "-n", or strings with \, behave unpredictably between shells. printf and echo — printing text in scripts
• printf truncated/repeated the output printf applies the format to all arguments, cycling through it. printf and echo — printing text in scripts
• You need to build a string from variables printf -v var '%s-%s' "$a" "$b" writes it into a variable with no output. printf and echo — printing text in scripts
• %d: invalid number %d received something that isn't a digit. printf and echo — printing text in scripts
• pwd: --: invalid option {#invalid-option} The snapshot shows this message for pwd --version: pwd — the working directory
• The path differs from what you expected {#unexpected-path} A symbolic link can explain the difference without a different directory being involved. pwd — the working directory
• The current directory was removed {#deleted-directory} If the directory path cannot be recovered, change to an existing directory and check again. pwd — the working directory
• screen -r: There is no screen to be resumed No such session (closed/rebooted). screen — terminal multiplexer
• Attached, can't detach ("There is a screen on ... Attached") The session is "stuck" to another terminal: screen — terminal multiplexer
• Garbled/broken display Ctrl+a then C (clear) or reconnect. screen — terminal multiplexer
• Mouse scrolling doesn't work screen scrolls through copy mode by default (Ctrl+a [). screen — terminal multiplexer
• SC2148: Tips depend on target shell {#sc2148} The file has no #! shellcheck — checking scripts before they run
• openBinaryFile: does not exist {#no-file} There is no file at that path: shellcheck — checking scripts before they run
• A finding that does not apply {#false-positive} shellcheck does not know your intent: shellcheck — checking scripts before they run
• shellcheck is quiet, yet the script fails {#silent} shellcheck checks only the text. shellcheck — checking scripts before they run
• [: =: unary operator expected {#unary-operator} The variable is empty and unquoted: test, [ ] and [[ ]] — checking conditions
• [: too many arguments {#too-many-arguments} The value contains a space: test, [ ] and [[ ]] — checking conditions
• [: a: integer expression expected {#integer-expected} -eq, -gt and the rest got something that is not a number: test, [ ] and [[ ]] — checking conditions
• [-f: command not found and missing `]' {#spaces} A space is missing after [ or before ]. test, [ ] and [[ ]] — checking conditions
• [: a: unexpected operator {#unexpected-operator} This is how sh (dash) reacts to == inside [ ]. test, [ ] and [[ ]] — checking conditions
• [ -n $x ] is true for an empty variable {#n-unquoted} Unquoted, the empty value disappears and [ -n ] is left — a check that the string -n is not empty, which is true. test, [ ] and [[ ]] — checking conditions
• value too great for base {#octal} [[ 08 -eq 8 ]] and (( 08 )) read a number with a leading zero as octal, and octal has no 8 or 9. test, [ ] and [[ ]] — checking conditions
• The link is there, but -e says no {#dangling-link} -e and -f look at the file the link points to. test, [ ] and [[ ]] — checking conditions
• I can't exit / everything is being "captured" That's the Ctrl+b prefix. tmux — terminal multiplexer
• sessions should be nested with care You ran tmux inside tmux. tmux — terminal multiplexer
• Sessions disappeared after a reboot tmux keeps sessions in memory — a reboot kills them. tmux — terminal multiplexer
• Mouse scrolling doesn't work Enable set -g mouse on in ~/.tmux.conf and tmux source ~/.tmux.conf (or start a new session). tmux — terminal multiplexer
• Processor information is unknown {#unknown} The implementation did not provide this field. uname — kernel and system information
• The output has no Ubuntu release {#not-ubuntu} uname does not report the distribution release. uname — kernel and system information
• A parser for uname -a stopped working {#summary-parsing} Avoid word positions in the combined summary. uname — kernel and system information
• The server "feels slow" — where to start uptime → if load ≫ core count, find the culprit: uptime, who, w — who is on the system and the load
• last is empty or looks odd Login logs rotate (wtmp/btmp); older entries are in wtmp.1. uptime, who, w — who is on the system and the load
• Someone else is editing the config w shows other sessions and their WHAT. uptime, who, w — who is on the system and the load
• I'm stuck, everything breaks as I type You're in Normal mode, where letters are commands. vim — editing files in the terminal
• Can't save: "E212: Can't open file for writing" No permission (you didn't open it with sudo). vim — editing files in the terminal
• Pasting with the mouse "staircases" the indentation :set paste before pasting (Insert's auto-indent adds extra indentation otherwise), then :set nopaste. vim — editing files in the terminal
• Accidentally made a mess of edits u undoes step by step; :q! vim — editing files in the terminal
• Need something simpler nano file — hints at the bottom (see nano). vim — editing files in the terminal
• A pipeline/grep "doesn't work" inside watch Wrap the whole command in quotes: watch — periodically repeat a command
• Colors disappeared By default watch strips ANSI color. watch — periodically repeat a command
• Need to follow a log line by line For logs, tail -f (see files-basics) or journalctl -f (see journalctl) are better; watch is for a "snapshot of state", not a stream. watch — periodically repeat a command
• Want to stop on a change / on readiness watch -g cmd exits once the output changes (handy as "wait until ready"); watch -e cmd stops if the command errors out. watch — periodically repeat a command
• which found it, but the command says "not found" (or the reverse) which doesn't know about aliases/builtins/functions and reads its own $PATH, which may differ from the current shell's. which / type / whereis — where a command lives
• The "wrong" version runs type -a name shows every match along PATH in search order — the first one is what runs. which / type / whereis — where a command lives
• command not found even though the package is installed The binary isn't in $PATH (often /usr/local/sbin, ~/.local/bin). which / type / whereis — where a command lives
• After installing a new version, the old one still runs Bash caches the resolved path to a command in a hash table for the session. which / type / whereis — where a command lives
• Passing a user name fails {#extra-name} whoami has no target-user argument. whoami — identify the effective user
• Expected student but got root {#wrong-identity} Check whether you are in a privileged shell. whoami — identify the effective user
• The UID exists but its name cannot be resolved {#missing-name} The numeric identity may lack an accessible account entry. whoami — identify the effective user
• Names with spaces get cut up find ... xargs — building arguments from a stream
• xargs ran on empty input and made a mess Add -r (--no-run-if-empty) — don't run the command with no arguments. xargs — building arguments from a stream
• "Argument list too long" / one call with a huge number of arguments xargs splits into batches itself (a system limit); to control it, use -n N. xargs — building arguments from a stream
• Need to insert the argument somewhere other than the end xargs -I{} cmd ... xargs — building arguments from a stream
• Parallel output gets interleaved With -P, lines from different processes interleave. xargs — building arguments from a stream

Files and directories

• rm deleted the wrong thing There's no recovery: Basic file commands
• No such file or directory, even though the file is right there Usually it's a path issue: Basic file commands
• Permission denied while copying No read permission on the source, or no write permission on the destination directory. Basic file commands
• cp didn't copy a directory fully A directory needs recursive copying: Basic file commands
• The file is busy, space wasn't freed after deleting it As long as a process keeps a deleted file open, the space isn't reclaimed. Basic file commands
• cp: -r not specified; omitting directory The source is a directory and -r is missing. cp — copy files and directories
• cp: cannot stat 'x': No such file or directory There's no source at that path. cp — copy files and directories
• cp: cannot create regular file: Permission denied You can't write to the destination directory. cp — copy files and directories
• The copy is owned by root, not the user sudo cp creates new files as root. cp — copy files and directories
• The copy has different permissions or times Without -p or -a the copy gets the current time and permissions from the umask. cp — copy files and directories
• No space left on device The destination filesystem is full. cp — copy files and directories
• "Script won't run: cannot execute binary file" file script shows the real type. file — determining file type
• The text encoding is in question file -i file shows the charset (utf-8 / iso-8859-1). file — determining file type
• Determining the filesystem type on a partition file -s /dev/sdXN (as root) is a quick way without mounting; for partitioning, see mount. file — determining file type
• file says "data" instead of a specific type No signature matched in the /etc/magic database — either the file is corrupted, or it's an unknown/raw format (for instance, encrypted data looks like a random… file — determining file type
• "Permission denied" clutters the output Drop the errors: find — searching for files
• It searches too deep / across other filesystems -maxdepth N limits the depth (put it BEFORE other conditions); -xdev avoids crossing mount points (see mount). find — searching for files
• Accidentally deleted the wrong thing with -delete Always run it without an action first (just the listing), confirm the list, then add -delete/-exec rm. find — searching for files
• find with -name finds nothing The pattern must be quoted ("*.log"), otherwise the shell expands it before find sees it. find — searching for files
• -exec is slow {} \; runs the command once per file. find — searching for files
• detected dubious ownership {#dubious-ownership} The repository belongs to another user (root here), and git will not work in it under your name: git: change history, undo and a remote repository
• Author identity unknown {#identity} The user making the commit has no name and email. git: change history, undo and a remote repository
• Your local changes … would be overwritten by merge {#local-changes} The working copy has an uncommitted change to a file the incoming commits change. git: change history, undo and a remote repository
• Need to specify how to reconcile divergent branches {#divergent} Both you and origin have commits the other side lacks (git status: git: change history, undo and a remote repository
• rejected … (fetch first) {#push-rejected} origin has someone else's commits. git: change history, undo and a remote repository
• refusing to update checked out branch {#non-bare-push} A push into an ordinary repository with working files: git: change history, undo and a remote repository
• Permission denied (publickey) on push The SSH key is not set up or not on the server — see ssh-keys. git: change history, undo and a remote repository
• Accidentally ran reset --hard git reflog remembers where HEAD was — you can go back: git: change history, undo and a remote repository
• A broken (dangling) symlink The target was moved or deleted. ln — symbolic and hard links
• A relative symlink "broke" after a move A relative path is resolved from the link's own directory, not the current one. ln — symbolic and hard links
• ln -s into an existing directory created the link INSIDE it If name is an existing directory, ln -s target dir drops the link at dir/target. ln — symbolic and hard links
• ls: cannot access 'x': No such file or directory There is no such path relative to the current directory. ls — list directory contents
• ls: cannot open directory: Permission denied Reading the listing needs r on the directory, entering it needs x. ls — list directory contents
• The file isn't in the list, but it exists Its name starts with a dot — add -a. ls — list directory contents
• The directory shows 4096 bytes but holds gigabytes ls -l shows the size of the directory entry, not of its contents. ls — list directory contents
• The name is printed in quotes: 'my file' That's how ls shows names with spaces and special characters so you can paste them into a command. ls — list directory contents
• A script parses ls output and breaks on spaces ls output is meant for people. ls — list directory contents
• mkdir: cannot create directory 'x': File exists The directory already exists. mkdir — creating directories
• mkdir: cannot create directory 'a/b/c': No such file or directory The intermediate directories (a, a/b) don't exist. mkdir — creating directories
• Permission denied No w+x permission on the parent directory. mkdir — creating directories
• The permissions aren't what -m specified -m sets the resulting directory's permissions directly (umask doesn't apply to it), but the intermediate directories created by -p still get the regular umask. mkdir — creating directories
• mv: cannot stat 'x': No such file or directory There's no source at that path. mv — move or rename
• mv: cannot move 'src' to 'dst/src': Directory not empty dst already holds a non-empty directory with the same name. mv — move or rename
• Permission denied mv needs write permission on both directories — source and destination; the file's own permissions don't matter. mv — move or rename
• The file is gone after mv The destination was not a directory but a file with that name, and mv overwrote it. mv — move or rename
• mv: invalid option -- 'l' with the name -file.txt mv reads a name starting with a dash as a bundle of options -f -i -l … and trips on the first unknown letter. mv — move or rename
• A path with spaces "fell apart" Always quote it: Paths — basename, dirname, realpath
• realpath: No such file or directory By default, realpath requires the path to exist. Paths — basename, dirname, realpath
• You need the path without resolving symlinks realpath follows symlinks. Paths — basename, dirname, realpath
• Forgot RESTIC_PASSWORD The repository is encrypted — without the password the data cannot be restored. restic — deduplicated backups
• The first backup is slow/large The first snapshot is full, later ones only carry changes (deduplication). restic — deduplicated backups
• The repository has bloated restic forget --keep-* --prune applies the retention policy and actually deletes unused blocks (--prune). restic — deduplicated backups
• locked repository An interrupted operation left a lock behind: restic — deduplicated backups
• Verifying a backup restic check (plus --read-data periodically) and a test restore — a backup without a verified restore is not a backup. restic — deduplicated backups
• rm: cannot remove 'dir': Is a directory The argument is a directory. rm — remove files and directories
• rm: cannot remove: Permission denied You can't write to the directory holding the file. rm — remove files and directories
• rm: remove write-protected regular file? The file lacks w. rm — remove files and directories
• rm: cannot remove: Operation not permitted The file has the immutable attribute (lsattr file shows i), or it's someone else's file in a sticky directory like /tmp. rm — remove files and directories
• Disk space wasn't freed A process still has the file open: rm — remove files and directories
• Argument list too long The pattern matched too many files for one command. rm — remove files and directories
• --delete removed something needed --delete makes dst an exact copy of src. rsync — file synchronization and copying
• Copied to the wrong place / a duplicated directory The trailing slash on the source (see the slash section above). rsync — file synchronization and copying
• It copies everything again every time Comparison is by mtime and size; if time "drifts" (different filesystems or time zones), add -c (checksum comparison) or --modify-window=1. rsync — file synchronization and copying
• Permissions/owners weren't preserved You need -a (or -o/-g/-p individually); for foreign owners, root rights on the receiving side. rsync — file synchronization and copying
• Slow with lots of small files Normal (rsync checks every file). rsync — file synchronization and copying
• "I didn't touch the file, but mtime is fresh" The file was rewritten (an editor saves in place) or a process appended to it. stat — file metadata
• ls -l shows a large size but no space is used A sparse file. stat — file metadata
• A symlink's permissions don't match expectations Symlinks on Linux almost always show 777 — these permissions don't affect access to the target, which is checked on the target file itself. stat — file metadata
• A broken symlink (points "nowhere") The target was deleted or moved. Symbolic and hard links
• You copied something and got the link itself (or the other way around) cp follows a symlink by default (it copies the target). Symbolic and hard links
• du/df disagree because of a hard link Hard links share the same blocks — du counts them once (which is confusing). Symbolic and hard links
• rsync "bloated" the backup Without -H, rsync doesn't preserve hard links (it copies the data again for each name). Symbolic and hard links
• tar: Removing leading '/' Normal: tar — archiving and extracting files
• Extracted it and scattered files into the current directory (a "tar bomb") An archive with no top-level directory. tar — archiving and extracting files
• Permission denied / owners were lost Extract system backups as root and with -p (permissions) / --same-owner (owners, the default for root). tar — archiving and extracting files
• Very slow / a large archive xz (J) compresses harder but is slower; gzip (z) is a balance; for multi-core, use pigz (tar -c dir | pigz > a.tgz). tar — archiving and extracting files
• Don't overwrite existing files on extraction --keep-old-files (error if it exists) or --skip-old-files (skip it). tar — archiving and extracting files

Text and search

• cat: /etc/shadow: Permission denied Only root or the file's group can read it. cat — print a whole file
• Permission denied with sudo cat > /etc/file The > redirection is done by your shell, without root rights, before sudo even starts. cat — print a whole file
• cat: /etc: Is a directory The argument is a directory. cat — print a whole file
• The terminal prints garbage after cat You printed a binary file, and control bytes switched the terminal's mode. cat — print a whole file
• The file flew by, only the end is visible cat doesn't stop at each screen. cat — print a whole file
• Empty output with a space delimiter There is more than one space between words, so the field you asked for is empty. cut — cut fields out of lines
• cut prints the whole line The line has no -d delimiter, and cut prints it unchanged. cut — cut fields out of lines
• cut: you must specify a list of bytes, characters, or fields You did not say what to cut. cut — cut fields out of lines
• cut: the delimiter must be a single character -d got more than one character. cut — cut fields out of lines
• cut: fields are numbered from 1 Fields are numbered from one: cut — cut fields out of lines
• cut: only one type of list may be specified The command has both -f and -c. cut — cut fields out of lines
• Cyrillic letters are cut in half On Ubuntu cut -c counts bytes, like -b, and a Cyrillic letter takes two bytes in UTF-8. cut — cut fields out of lines
• patch: Hunk FAILED / .rej files The context in the target file has already changed. diff and patch — comparing and applying changes
• I don't know which -p level to use git patches usually need patch -p1 (strips a/, b/). diff and patch — comparing and applying changes
• Too much noise in the diff (whitespace/CRLF/case) diff -w (ignore whitespace), -i (case), -B (blank lines). diff and patch — comparing and applying changes
• Compare command output, not files diff <(cmd1) <(cmd2) — process substitution (bash). diff and patch — comparing and applying changes
• grep itself shows up in ps aux | grep A classic. grep — searching text
• Special characters in the pattern get treated as regex A dot, brackets, + are regex. grep — searching text
• \d, \w, lookahead don't work Basic/extended grep doesn't know them — you need -P (PCRE): grep — searching text
• I expect matches, but grep is silent Check the case (-i), the right file, and that the data is actually arriving. grep — searching text
• Too slow over a large tree Exclude directories (--exclude-dir), or use ripgrep (rg) — noticeably faster. grep — searching text
• An unexpected command still runs after clearing {#other-definition} The name may identify a function or alias. hash — Bash command path cache
• The table is empty {#empty-table} This is valid, for example after clearing it. hash — Bash command path cache
• Another terminal did not change {#other-shell} Another shell has its own context. hash — Bash command path cache
• head: cannot open 'x' for reading: No such file or directory There's no file at that path. head — the first lines of a file
• head: invalid number of lines -n must be followed by a number: head — the first lines of a file
• The output stops in the middle of a line You used -c (bytes) instead of -n (lines). head — the first lines of a file
• The line you need isn't in the output It's past the first ten. head — the first lines of a file
• jq: error: Cannot index ... The structure isn't what you expect. jq — processing JSON on the command line
• There are extra quotes in the output jq prints strings with quotes. jq — processing JSON on the command line
• Invalid JSON on the input jq fails on non-JSON. jq — processing JSON on the command line
• You need to substitute a shell variable Don't paste it into the filter text — use --arg name val (a string) or --argjson (a number/JSON), and reference it inside as $name. jq — processing JSON on the command line
• How to quit less q. less — page through a long file with search
• Pattern not found The word isn't below the current position. less — page through a long file with search
• "file" may be a binary file. See it anyway? The file is binary, and reading it as text makes no sense. less — page through a long file with search
• ESC[31m and similar garbage instead of colors The program printed color codes, and less shows them as characters. less — page through a long file with search
• Missing filename less was started with no file and no pipe. less — page through a long file with search
• grep 'a|b' finds nothing {#bre-alternation} Without -E, | is a plain character and grep looks for the string a|b literally. Regular expressions — patterns for grep, sed and awk
• \d matches the letter d {#backslash-d} \d exists only in PCRE. Regular expressions — patterns for grep, sed and awk
• Unmatched ( or ( {#unmatched} A bracket in the pattern is not closed: Regular expressions — patterns for grep, sed and awk
• The dot matches too much {#dot} grep -o '10.20.0.15' also finds the start of 10.20.0.150: Regular expressions — patterns for grep, sed and awk
• The shell expanded the asterisk {#quote-pattern} Unquoted, grep error.* app.log may let the shell replace error.* with matching file names. Regular expressions — patterns for grep, sed and awk
• sed -i changed nothing {#no-change} The pattern matched no line. sed and awk — editing and fields in a text stream
• More was replaced than intended {#too-much} The pattern is wider than needed: sed and awk — editing and fields in a text stream
• unterminated `s' command {#unterminated} sed 's/a/b' is missing the last delimiter. sed and awk — editing and fields in a text stream
• unknown option to `s' {#unknown-option} The pattern or the replacement contains /, and sed took the rest for flags. sed and awk — editing and fields in a text stream
• invalid reference \1 on `s' command's RHS {#invalid-reference} The replacement has \1, but the pattern has no group: sed and awk — editing and fields in a text stream
• awk: unexpected newline or end of string {#awk-syntax} A brace or a quote in the program is not closed: sed and awk — editing and fields in a text stream
• awk prints empty fields {#awk-empty} The line is split with the wrong separator: sed and awk — editing and fields in a text stream
• 10 comes before 9 The comparison is textual. sort — sort lines
• sort -k2 does not sort by the second field only -k2 means "from field 2 to the end of the line", so ties are decided by the following fields. sort — sort lines
• Upper and lower case are not in the order you expected The locale decides the text order. sort — sort lines
• sort: cannot read: x: No such file or directory There is no file at that path. sort — sort lines
• sort: invalid number at field start -k needs a field number: sort — sort lines
• Numbers with leading spaces sort oddly Without -n or -b, leading blanks take part in the comparison. sort — sort lines
• After sort file > file the file is empty The shell truncated the file for writing before sort read it. sort — sort lines
• sort doesn't sort numbers by magnitude By default, sorting is lexicographic: sort, uniq, cut, wc, tr — text stream processing
• uniq didn't remove duplicates uniq only collapses adjacent identical lines, so it needs sort before it: sort, uniq, cut, wc, tr — text stream processing
• cut doesn't see the columns The default delimiter is a tab. sort, uniq, cut, wc, tr — text stream processing
• wc -l shows one line fewer A file's last line without a trailing newline isn't counted. sort, uniq, cut, wc, tr — text stream processing
• Cyrillic breaks in tr tr works on bytes, not characters: sort, uniq, cut, wc, tr — text stream processing
• tail -f went quiet after log rotation logrotate renamed the file and created a new one, while tail -f keeps reading the old one. tail — the last lines and following a log
• tail: cannot open: Permission denied System logs are readable by root and the adm group. tail — the last lines and following a log
• tail: cannot open 'x' for reading: No such file or directory The file doesn't exist — maybe the service logs to journald instead of a file. tail — the last lines and following a log
• New lines show up late The program buffers its output, not tail. tail — the last lines and following a log
• Ctrl+C doesn't bring the prompt back Click the terminal so it has focus and press Ctrl+C again: tail — the last lines and following a log
• tee overwrote the file instead of appending Without -a, tee truncates the file. tee — output to a file and the screen
• The exit code is always 0, even though the command failed In a cmd | tee pipeline, the exit code comes from the last command (tee). tee — output to a file and the screen
• Writing to a root-owned file fails Use | sudo tee, not sudo cmd >. tee — output to a file and the screen
• uniq did not remove the repeats The repeats are not adjacent. uniq — remove or count repeated lines
• Lines look identical but are not collapsed They differ in invisible characters: uniq — remove or count repeated lines
• cut does not pick the number from uniq -c output The number is padded with spaces, and cut -d' ' sees empty fields. uniq — remove or count repeated lines
• uniq: x: No such file or directory There is no file at that path. uniq — remove or count repeated lines
• wc -l shows one line fewer wc counts newlines. wc — count lines, words and bytes
• There is an extra file name in the output The name is printed when the file is an argument. wc — count lines, words and bytes
• wc -c and wc -m give different numbers -c counts bytes, -m counts characters according to the locale. wc — count lines, words and bytes
• ls | wc -l gives the wrong number of files ls without -A skips hidden files, and ls -l adds a total line. wc — count lines, words and bytes
• wc: x: Is a directory wc reads files, not directories. wc — count lines, words and bytes

Users and permissions

• Password expiry changed instead of account expiry {#wrong-field} Compare the fields in sudo chage -l labpaused. chage — password and account expiry
• The user resolves but a local change fails {#nonlocal-account} The record may come from an external NSS source. chage — password and account expiry
• chmod 777 "just in case" Almost always wrong and dangerous (especially on directories/web roots). chmod / chown — permissions and ownership
• A shared directory for a team is not 1777 1777 (sticky bit, world-writable) is the mode for public directories like /tmp: chmod / chown — permissions and ownership
• Recursion broke executability chmod -R 644 dir strips x from directories → you can no longer enter them. chmod / chown — permissions and ownership
• Changed the owner — the service stopped reading the files Many daemons run as their own user (www-data, postgres). chmod / chown — permissions and ownership
• Permissions look right, but access is missing / extra ACL is in play. Extended permissions (ACL)
• I granted rw to a named user, but they still cannot write The mask is capping it: Extended permissions (ACL)
• ACL entries are not inherited by new files You need default ACL on the directory (setfacl -d -m ...). Extended permissions (ACL)
• ACL "disappeared" after a copy or backup cp does not carry ACL over by default — use cp -a/--preserve=all, rsync -A, or tar --acls. Extended permissions (ACL)
• The filesystem does not support ACL It needs the acl mount option (usually on by default on ext4). Extended permissions (ACL)
• Permission denied when running a script The x bit is missing: File permissions
• Can't write to a directory, even though you have w on the file Creating/deleting/renaming a file depends on permissions on the directory (w+x), not on the file itself. File permissions
• The web server can't read files (403) nginx runs as www-data — it needs r on the files and x on every directory along the path. File permissions
• Access should work, but it doesn't ACLs may be in play (a + in ls -l), or AppArmor (look for apparmor ... File permissions
• An SGID directory: new files have the wrong group SGID sets the group for new files, but existing ones and files moved in with mv need chgrp -R by hand. File permissions
• name - No such file or directory There's no part with that name. namei — permissions at every step of a path
• name - Permission denied The directory on the line above has no x for whoever ran namei. namei — permissions at every step of a path
• namei shows everything open, but access fails namei doesn't show ACLs, AppArmor, or the groups a process got when it started. namei — permissions at every step of a path
• The file is 644, yet it can't be read The file's own permissions are checked last. namei — permissions at every step of a path
• The service runs although the shell is blocked {#service-still-runs} This can be expected: nologin — a shell that refuses login
• sudo still permits a command as that user {#sudo-still-works} Inspect sudo rules. nologin — a shell that refuses login
• passwd: Authentication token manipulation error passwd couldn't change the password. passwd — change or lock a password
• passwd: password unchanged The closing line after any such failure. passwd — change or lock a password
• passwd: You may not view or modify password information A regular user is trying to see or change someone else's password. passwd — change or lock a password
• The user still logs in after passwd -l They log in with an SSH key: passwd — change or lock a password
• I type the password and nothing appears That's by design: passwd — change or lock a password
• A new file in the shared directory has a personal group The directory has no setgid (ls -ld shows x rather than s for the group) — run chmod g+s. Special permission bits: setuid, setgid, sticky
• A colleague deleted my file The directory has no sticky bit, and the group has w. Special permission bits: setuid, setgid, sticky
• rm: cannot remove 'x': Operation not permitted The file is someone else's, and the directory has the sticky bit (t or T at the end of ls -ld). Special permission bits: setuid, setgid, sticky
• chmod 775 did not clear setgid on a directory GNU chmod with a three- or four-digit number does not clear setuid and setgid on directories. Special permission bits: setuid, setgid, sticky
• chmod: changing permissions of 'x': Operation not permitted Only the file's owner and root can change its permissions. Special permission bits: setuid, setgid, sticky
• setuid on a script does nothing The Linux kernel ignores setuid on #! Special permission bits: setuid, setgid, sticky
• Files in a shared directory are not group-writable The creator's mask is 022: umask — the mask for new files and directories
• Which mask a command run as another user gets sudo -u alice opens a PAM session, and pam_umask sets alice's mask: umask — the mask for new files and directories
• The mask changed, but old files are the same The mask applies only at creation. umask — the mask for new files and directories
• A new script is not executable The mask never gives files x. umask — the mask for new files and directories
• bash: umask: 9: octal number out of range The mask uses octal digits 0–7: umask — the mask for new files and directories
• useradd: Permission denied / cannot lock /etc/passwd The command ran without root. useradd — create a user
• useradd: user 'alice' already exists The name is taken. useradd — create a user
• useradd: group 'devs' does not exist A group in -G or -g must exist beforehand: useradd — create a user
• The user exists but can't log in No password is set — run sudo passwd alice. useradd — create a user
• A strange shell instead of bash: no history or completion The user was created without -s and got /bin/sh. useradd — create a user
• Could not chdir to home directory on login The user was created without -m, so there's no home directory. useradd — create a user
• usermod: user alice is currently used by process 1234 The name, UID or home directory can't change while the user is logged in. usermod — change an account and its groups
• usermod: group 'devs' does not exist A group given to -g or -G must exist. usermod — change an account and its groups
• Added to a group, still no access The session and services hold the old group list. usermod — change an account and its groups
• Groups vanished after usermod -G That's -G without -a: usermod — change an account and its groups
• gpasswd: user 'alice' is not a member of 'docker' The user was never in that group — check id -nG alice. usermod — change an account and its groups
• Added to a group — it has no effect Group membership applies on the next login. Users and groups
• Accidentally replaced the groups (-G without -a) Restore the needed ones: Users and groups
• Can't delete a user — "currently used by process" End its processes (pkill -u alice, see processes) or use userdel -f. Users and groups
• Login denied, even though the password is correct The account/password is locked (passwd -l/chage -E in the past) or the shell is nologin. Users and groups
• Files show the "wrong" group An SGID directory only assigns its group to NEW files; recursively chgrp -R the old ones (see permissions, chmod-chown). Users and groups

Processes

• Permission denied on another user's process {#permission-denied} fd, environ, cwd, exe, io, maps of another user's process are readable by root only. /proc: processes and the kernel as files
• sudo is there, yet Permission denied {#sudo-redirect} The < and > redirections are done by your own shell without root rights; sudo applies to the command only. /proc: processes and the kernel as files
• No such file or directory {#no-such-pid} The process has ended, and its directory went with it. /proc: processes and the kernel as files
• environ lacks a variable that was set {#environ-stale} The variable was set after the process started. /proc: processes and the kernel as files
• The process died after closing SSH Launched with &, it is tied to the session and gets SIGHUP on logout. Background jobs in the shell — &, jobs, nohup
• A background job's output clutters the terminal Redirect stdout and stderr: Background jobs in the shell — &, jobs, nohup
• Ctrl+Z "froze" a process It wasn't killed — the process is suspended (stopped, state T). Background jobs in the shell — &, jobs, nohup
• The script finished, but background jobs are still running/orphaned Add wait at the end, or trap 'kill 0' EXIT to kill the process group on exit. Background jobs in the shell — &, jobs, nohup
• A process was "killed", logs mention memory cgroup / oom-kill The service hit its cgroup's MemoryMax — the kernel killed it (a cgroup-OOM, not a global one). cgroups and namespaces — how containers work
• A service is "eating" all the CPU/disk Limit it through systemd: cgroups and namespaces — how containers work
• A container can't see other processes/network — that's normal It's by design (namespaces isolate). cgroups and namespaces — how containers work
• Where to see actual usage by service systemd-cgtop (like top, but by cgroup/service) — shows right away which service is loading the system. cgroups and namespaces — how containers work
• pidstat -d shows nothing {#pidstat-empty} Two reasons. iostat and pidstat — which disk is busy and which process loads it
• The first iostat line looks different {#first-report} It holds averages since boot, including the whole boot of the system. iostat and pidstat — which disk is busy and which process loads it
• %util is 100, yet everything works {#util-100} The device serves requests in parallel, and %util counts only the time when anything at all was busy. iostat and pidstat — which disk is busy and which process loads it
• ionice does not help {#ionice-none} The none disk scheduler ignores classes: iostat and pidstat — which disk is busy and which process loads it
• iotop: command not found {#no-iotop} iotop is not installed on the training machine. iostat and pidstat — which disk is busy and which process loads it
• kill: (12345) - No such process There's no process with that PID anymore: kill — send a signal to a process
• kill: (1) - Operation not permitted The process belongs to another user or root. kill — send a signal to a process
• The process is still listed after kill State Z (zombie): kill — send a signal to a process
• The process came back after kill systemd or another supervisor restarts it. kill — send a signal to a process
• kill: invalid signal specification There's no such signal — usually a typo in the name. kill — send a signal to a process
• High load, but the CPU is idle {#high-load-idle-cpu} top shows 90.9 id, and the load on one CPU is 2.90. Load average: how many tasks wait for the CPU and the disk
• A D process ignores kill -9 {#kill-d} The signal waits for the kernel operation to end. Load average: how many tasks wait for the CPU and the disk
• Is a load of 1.00 on one CPU a lot? {#one-per-cpu} There is no queue: Load average: how many tasks wait for the CPU and the disk
• Inside a container the load is the whole machine's {#containers} There is one /proc/loadavg per kernel: Load average: how many tasks wait for the CPU and the disk
• High st on a virtual machine {#steal} st (steal) means the VM needed the CPU, but the hypervisor gave it to other machines. Load average: how many tasks wait for the CPU and the disk
• df shows "full", du doesn't A file that's deleted but still held open by a process doesn't free its space. lsof — open files and ports of processes
• umount: target is busy Something is holding the filesystem: lsof — open files and ports of processes
• A port is in use, but by what? sudo lsof -i :PORT -nP or ss -tlnp. lsof — open files and ports of processes
• A process is "leaking" file descriptors sudo lsof -p PID | wc -l shows how many are open; compare against the ulimit -n limit. lsof — open files and ports of processes
• lsof is slow It's resolving DNS/ports/mounts — add -nP; with a stuck NFS mount it can hang on the mount itself. lsof — open files and ports of processes
• free is almost zero {#low-free} Normal: Memory and swap: free, available and the page cache
• The server is slow, si and so grow {#swapping} Memory is short, and the kernel shuffles pages between memory and swap. Memory and swap: free, available and the page cache
• A process ended, the log says Killed process {#oom} Memory ran out and the kernel ended the process — see oom-killer. Memory and swap: free, available and the page cache
• mkswap or swapon: insecure permissions 0644 {#insecure-permissions} mkswap prints insecure permissions 0644, fix with: Memory and swap: free, available and the page cache
• swapon failed: Device or resource busy {#busy} The file is already on. Memory and swap: free, available and the page cache
• Swap is gone after a reboot {#swap-after-reboot} swapon lasts until a reboot. Memory and swap: free, available and the page cache
• swap.img.swap in the failed list {#swap-unit-failed} systemd makes a <path>.swap unit for every swap line in fstab. Memory and swap: free, available and the page cache
• vm.swappiness is back to 60 {#swappiness-reset} The value was set with sysctl vm.swappiness=… — it lives until a reboot. Memory and swap: free, available and the page cache
• A service restarts in a loop {#restart-loop} systemctl status shows activating (auto-restart) (Result: OOM killer: when memory runs out
• A process vanished without a line in its log {#silent-death} SIGKILL lets the program write nothing. OOM killer: when memory runs out
• Result=exit-code, status=137 {#exit-code-137} The main shell process managed to exit by itself — with code 137 after OOM killed its child — before systemd received the OOM event: OOM killer: when memory runs out
• OOM with free memory {#oom-free-memory} free -h shows the memory of the whole machine, while the OOM happened in a cgroup. OOM killer: when memory runs out
• OOM killed the wrong process {#wrong-victim} The process in the invoked oom-killer line only asked for memory at the moment of shortage. OOM killer: when memory runs out
• pattern that searches for process name longer than 15 characters The process name is cut to 15 characters, so a longer pattern matches nothing. pgrep — find a process PID by name
• pgrep -f finds an extra process The pattern turned up in someone else's command line — for example, a bash -c shell that itself runs pgrep -f …. pgrep — find a process PID by name
• pgrep prints nothing Nothing matched; the exit status is 1. pgrep — find a process PID by name
• pgrep: invalid user name There's no user with that name. pgrep — find a process PID by name
• A process is hogging CPU top (or ps aux --sort=-%cpu | head) → find the PID → wait it out / renice / kill. Processes
• A memory leak / OOM ps aux --sort=-%mem | head; in the journal — journalctl -k | grep -i oom (the kernel killed something under memory pressure, see oom-killer). Processes
• A zombie process (Z/defunct) Harmless by itself — it's waiting for its parent to reap it (wait). Processes
• kill doesn't kill it (even -9) The process is in state D (uninterruptible sleep, waiting on I/O) — kill can't help; wait for the I/O to finish or investigate the disk/NFS. Processes
• Which process opened a port/file For a port — ss -tlnp; for a file/directory — lsof. Processes
• A process eats CPU, but ps shows little %CPU %CPU in ps is the average over the process's lifetime, not "right now". ps — list processes
• error: user name does not exist ps -u expects an existing name or UID. ps — list processes
• The process is listed, but kill doesn't help State Z — the process has already exited and can't be killed: ps — list processes
• The command line is cut off ps fits output to the window width. ps — list processes
• The terminal has not displayed a prompt {#no-prompt} The foreground command is still running. sleep — delay execution
• An error appears instead of a delay {#invalid-duration} Check the operand: sleep — delay execution
• attach: ptrace(PTRACE_SEIZE, …): Operation not permitted {#attach-denied} ptrace_scope = 1 forbids attaching to a process that is not your child. strace — which system calls a process makes
• Program output is mixed with strace {#mixed-output} Both write to the terminal. strace — which system calls a process makes
• Strings are cut: "hello-wo"... {#truncated} strace shows 32 characters of a string, the rest is .... strace — which system calls a process makes
• No calls from children {#no-children} A child process does the work, and strace without -f does not see it. strace — which system calls a process makes
• temp-path '.' must be readable and writeable {#temp-path} --hdd writes files to the current directory, and the student cannot write to /. stress-ng — timed load on the CPU, memory and disk
• gave up trying to mmap, no available memory {#no-memory} The full line is vm: stress-ng — timed load on the CPU, memory and disk
• stress-ng runs and never ends {#runs-forever} --timeout is missing: stress-ng — timed load on the CPU, memory and disk
• note and cpu-method messages in the output {#notes} note: stress-ng — timed load on the CPU, memory and disk
• High load, idle CPU {#high-load-idle-cpu} Tasks are in state D. System monitoring: where to start when the server is slow
• High wa: who writes to the disk {#high-wa} Without sudo, pidstat -d shows only your own processes: System monitoring: where to start when the server is slow
• free is almost zero {#low-free} The kernel keeps free memory busy with the page cache and gives it to programs on demand. System monitoring: where to start when the server is slow
• sar prints only LINUX RESTART {#sar-empty} History collection is off: System monitoring: where to start when the server is slow
• pidstat shows nothing {#pidstat-empty} pidstat prints only processes that did something during the interval. System monitoring: where to start when the server is slow
• Load average is high, but the CPU is nearly idle The queue holds processes in state D waiting on the disk. top — processes and load in real time
• free is almost zero — out of memory? No, if buff/cache and avail Mem are large: top — processes and load in real time
• A process shows 150 %CPU The percentage is per core: top — processes and load in real time
• How to quit top q. top — processes and load in real time
• Garbage and terminal codes when piping Without -b, top draws the screen with control codes. top — processes and load in real time
• Too many open files {#emfile} The process hit its soft nofile limit (error code EMFILE, number 24): ulimit and process limits: open files, processes, core
• cannot modify limit: Operation not permitted {#cannot-modify} The soft limit is being raised above the hard one, or the hard one above its current value without root. ulimit and process limits: open files, processes, core
• Raised it in limits.conf, the service did not notice {#limits-conf-service} limits.conf affects user logins through PAM. ulimit and process limits: open files, processes, core
• systemctl show prints the new limit, the service fails as before {#needs-restart} The service was not restarted after daemon-reload: ulimit and process limits: open files, processes, core
• fork: retry: Resource temporarily unavailable {#nproc} The user hit the nproc process limit (ulimit -u). ulimit and process limits: open files, processes, core
• Permission denied on connect nginx logs connect() to unix:/run/app/app.sock failed (13: Unix sockets — processes talking through a file
• No such file or directory nginx — (2: Unix sockets — processes talking through a file
• Connection refused although the file is there The file is left over from a stopped process — ss -xl does not list it. Unix sockets — processes talking through a file
• Address already in use The server cannot create the socket: Unix sockets — processes talking through a file
• The PID is not a child of this shell {#not-a-child} Check where the task was started. wait — wait for a child task in Bash
• The status belongs to an unexpected command {#overwritten-status} Another command ran between waiting and reading $?. wait — wait for a child task in Bash

Services and systemd

• The server doesn't answer queries Run named-checkconf (any errors?), check the service is active, listening on 53 (ss -tulnp), and the port is open in firewall (53/udp and 53/tcp). BIND9 — DNS server
• I changed a zone — still old data You didn't bump the serial in the SOA, or didn't reload (rndc reload). BIND9 — DNS server
• SERVFAIL An error in the zone (named-checkzone), a broken delegation, or DNSSEC. BIND9 — DNS server
• It only resolves locally Check allow-query / listen-on in named.conf.options — is it listening on the right address and letting clients in. BIND9 — DNS server
• 506 Cannot talk to daemon {#cannot-talk} chronyd is not running: chrony: time synchronization over NTP
• 501 Not authorised {#not-authorised} Commands that change the service state — makestep, burst, reload sources — are for root: chrony: time synchronization over NTP
• sources is empty {#no-sources} The server name does not resolve: chrony: time synchronization over NTP
• Source ^? {#unreachable} The server has never answered (Reach 0): chrony: time synchronization over NTP
• Leap status: Not synchronised {#not-synchronised} There is no usable source: chrony: time synchronization over NTP
• Set the clock by hand, and it went back {#clock-reverted} chronyd brings the time back to the source with a step, as in chrony. chrony: time synchronization over NTP
• timedatectl timesync-status: Failed to query server {#timedatectl} timesync-status asks systemd-timesyncd, and the chrony install removed it. chrony: time synchronization over NTP
• A cron job does not run Cron has a minimal $PATH and no interactive-session environment (aliases, variables from ~/.bashrc) — use absolute paths to commands and files, do not rely on… Cron — the task scheduler
• A timer did not fire while the machine was off Add Persistent=true — it runs the missed job after the machine comes back up. Cron — the task scheduler
• I don't know when the next run is systemctl list-timers --all shows every timer, including inactive ones, with the last and next run times. Cron — the task scheduler
• A user's crontab job is not visible in the system crontab crontab -e edits the current user's crontab (stored under /var/spool/cron/crontabs/) — separate from the system files /etc/crontab and /etc/cron.d/*, where an… Cron — the task scheduler
• Clients aren't getting an address Is the service listening on the right interface (INTERFACESv4)? DHCP server
• Address conflict / two DHCP servers on the network Two servers are handing out from overlapping pools. DHCP server
• A MAC reservation isn't taking effect Check the exact MAC (hardware ethernet) and that the address is outside the dynamic range. DHCP server
• Panel: "No data" Did the data source pass Save & Test? Grafana — dashboards
• Data source: bad gateway / connection refused Grafana can't reach the data source. Grafana — dashboards
• Forgot the admin password sudo grafana-cli admin reset-admin-password <new-password> (or an environment variable at startup). Grafana — dashboards
• The dashboard disappeared after redeploying If it wasn't provisioned, dashboards live in the database (/var/lib/grafana/grafana.db). Grafana — dashboards
• 502 behind a reverse proxy If Grafana sits behind nginx/haproxy on a subpath, set root_url and serve_from_sub_path in grafana.ini, or links/login break. Grafana — dashboards
• 503 Service Unavailable No "live" backends: HAProxy — load balancer
• All requests go to one server balance isn't set, or cookie/source stickiness is enabled. HAProxy — load balancer
• The config doesn't apply haproxy -c -f ... HAProxy — load balancer
• Can't see the client's real IP on the backend Add option forwardfor and read X-Forwarded-For on the backend/nginx (real_ip). HAProxy — load balancer
• Timeouts/disconnects under load Tune timeout connect/client/server in defaults and maxconn. HAProxy — load balancer
• sudo hangs for a moment after changing the name The new name doesn't resolve locally. hostnamectl — host name and machine info
• The name "reverted" after a reboot cloud-init or DHCP is resetting it. hostnamectl — host name and machine info
• hostname -f doesn't return an FQDN There's no domain-qualified record in /etc/hosts/DNS. hostnamectl — host name and machine info
• Same machine-id on cloned VMs When cloning an image, machine-id must be unique, or DHCP leases and journals get confused. hostnamectl — host name and machine info
• VIP is up on both nodes at once (split-brain) The nodes don't see each other's VRRP advertisements. keepalived — high availability with VRRP
• The VIP doesn't move when the application fails There's no track_script — the VIP stays as long as the node is alive, even if the service died. keepalived — high availability with VRRP
• The VIP "flaps" back and forth The vrrp_script is too sensitive, or a recovered MASTER takes the VIP back. keepalived — high availability with VRRP
• virtual_router_id conflict Within one L2 segment, virtual_router_id must be unique per group; a clash with another group breaks the election. keepalived — high availability with VRRP
• Where's the VIP right now, what's happening ip a on each node (the VIP only shows on the MASTER), journalctl -u keepalived for the state transitions (see ip). keepalived — high availability with VRRP
• virsh: failed to connect to the hypervisor libvirtd isn't running (systemctl status libvirtd), or you lack permissions — the user must be in the libvirt group. KVM and libvirt
• The VM won't start virsh start vm1 prints an error; check journalctl -u libvirtd and the log at /var/log/libvirt/qemu/vm1.log for details. KVM and libvirt
• The VM has no network The virtual network isn't running: KVM and libvirt
• Can't get into the VM virsh console vm1 (needs serial enabled in the guest OS) — exit with Ctrl+]. KVM and libvirt
• Someone else's site answers {#wrong-site} The request's Host matched no server_name, so the default server answered. nginx — web server and reverse proxy
• 404 Not Found {#not-found} error.log has the path nginx built: nginx — web server and reverse proxy
• 403 Forbidden (13: nginx — web server and reverse proxy
• 502 Bad Gateway {#bad-gateway} nginx did not open a connection to the backend. nginx — web server and reverse proxy
• 504 Gateway Time-out {#gateway-timeout} There is a connection but no answer: nginx — web server and reverse proxy
• nginx -t: test failed unexpected "}" in …:5 — the line before } has no ;; a duplicate default server for 0.0.0.0:80 — default_server is set in two blocks on one port; open()… nginx — web server and reverse proxy
• conflicting server name {#conflicting-server-name} [warn] conflicting server name "docs.lab.test" on 0.0.0.0:80, ignored — one name sits in two blocks on one port. nginx — web server and reverse proxy
• key values mismatch {#key-mismatch} SSL_CTX_use_PrivateKey(…) failed (… key values mismatch) — the first certificate in the ssl_certificate file does not match the key. nginx — web server and reverse proxy
• The new certificate is not served {#reload} nginx reads certificates at start and on reload. nginx — web server and reverse proxy
• The service does not start: port in use sudo ss -tlnp | grep ':80 ' shows who holds the port. nginx — web server and reverse proxy
• A new listen did not apply {#listen-change} Changing the address on the same port (listen 80 → listen 127.0.0.1:80) does not work through reload: nginx — web server and reverse proxy
• Mail isn't going out / it's stuck in the queue mailq shows the reason next to each message. Postfix — a mail server
• Relay access denied Postfix doesn't forward mail for strangers (anti-relay). Postfix — a mail server
• Mail is landing in spam You need SPF/DKIM/DMARC records in DNS and a correct PTR (rDNS). Postfix — a mail server
• You changed a map, but it didn't take effect After editing sasl_passwd/transport/virtual, always run postmap file (rebuild the .db) and reload. Postfix — a mail server
• Connection refused on port 25 It's only listening on localhost (inet_interfaces = loopback-only), or the port is blocked (firewall). Postfix — a mail server
• Connection refused {#connection-refused} connection to server at "127.0.0.1", port 5432 failed: PostgreSQL — access, pg_hba.conf, dump and restore
• no pg_hba.conf entry {#no-pg-hba-entry} FATAL: PostgreSQL — access, pg_hba.conf, dump and restore
• password authentication failed {#password-failed} A rule matched, the password did not. PostgreSQL — access, pg_hba.conf, dump and restore
• Peer authentication failed {#peer-failed} psql -U appuser without -h goes through the socket, where the rule is peer: PostgreSQL — access, pg_hba.conf, dump and restore
• A pg_hba.conf change has no effect {#hba-not-reloaded} The file was not re-read, or was re-read with an error — then the old rules apply, and pg_reload_conf() still returns t. PostgreSQL — access, pg_hba.conf, dump and restore
• pg_restore: relation "…" already exists {#restore-errors} A restore into a database that already has the objects: PostgreSQL — access, pg_hba.conf, dump and restore
• input file appears to be a text format dump {#text-format-dump} pg_restore got plain SQL — restore it with psql -f. PostgreSQL — access, pg_hba.conf, dump and restore
• A target is DOWN Open /targets — the reason is right there (connection refused / timeout / 404). Prometheus — metrics collection
• No data for a metric Check the metric name and labels in the UI (autocomplete). Prometheus — metrics collection
• Prometheus is eating disk space That's retention and the number of series. Prometheus — metrics collection
• Alerts aren't arriving Check the rules (promtool check rules), the connection to Alertmanager, and its routes. Prometheus — metrics collection
• Could not connect to Redis (6379) The service isn't running (systemctl status redis-server) or it only listens on 127.0.0.1 while the client connects over the network. Redis — in-memory key-value store
• OOM / memory usage grew redis-cli info memory, --bigkeys. Redis — in-memory key-value store
• Data disappeared after a restart By default data lives only in memory. Redis — in-memory key-value store
• Slow / high latency redis-cli --latency, slowlog get. Redis — in-memory key-value store
• NOAUTH Authentication required requirepass is set — connect with redis-cli -a SECRET or send AUTH as the first command. Redis — in-memory key-value store
• NT_STATUS_LOGON_FAILURE Wrong SMB password/user. Samba — file sharing over SMB
• NT_STATUS_ACCESS_DENIED System permissions on the directory (path) or valid users/read only in smb.conf. Samba — file sharing over SMB
• Server/share not visible The SMB port 445/tcp is closed (firewall) or the service isn't running. Samba — file sharing over SMB
• Mount: "host is down" / protocol negotiation failed An SMB version mismatch. Samba — file sharing over SMB
• Cyrillic/permissions in file names unix charset = UTF-8; permissions for new files — create mask/directory mask, force user/force group for a single owner. Samba — file sharing over SMB
• The service "crashed", but systemctl status shows active You may be looking at a different service or a renamed/aliased unit (a .socket instead of a .service). Service
• enable didn't start the service right away enable only sets up autostart for the future (it creates a symlink). Service
• The service keeps restarting on its own (a restart loop) It's crashing after starting, and systemd is configured to restart it (Restart= in the unit file). Service
• Warning: some journal files were not opened due to insufficient permissions. Journal lines in systemctl status are visible only with sudo or to members of the adm and systemd-journal groups. Service
• What's the difference between "service", "daemon", and "unit"? A daemon is a background process in the general sense (the term predates systemd). Service
• is-active lies after starting the binary manually systemctl only sees what it started itself. systemctl — managing systemd services
• Changes to the unit aren't applied After editing a .service file, you need sudo systemctl daemon-reload, then restart. systemctl — managing systemd services
• The service is stuck "activating" Check Type= in the unit and journalctl -xeu <unit>: systemctl — managing systemd services
• The service "doesn't start and doesn't say why" sudo systemctl status -l --no-pager <unit> plus sudo journalctl -u <unit> -b (for the current boot). systemctl — managing systemd services
• Warning: some journal files were not opened due to insufficient permissions. systemctl status showed the service state but no journal lines. systemctl — managing systemd services
• The output opened in a pager, no prompt Long output of status, cat, list-units doesn't fit the terminal window, so systemctl shows it through less: systemctl — managing systemd services
• enable doesn't work — "no installed section" The unit has no [Install] section. systemctl — managing systemd services
• The service doesn't start systemctl status <unit> — exit code and the last journal lines. systemd
• "Unit not found" Check the name and path: systemd
• Slow boot systemd-analyze critical-chain — who held the boot; the order of the investigation is in systemd-analyze. systemd
• Works by hand, fails from the timer {#works-by-hand} It is almost always the service environment (see "The service environment" above). systemd timers — scheduled jobs
• Job for clean.service failed {#job-failed} The script returned non-zero. systemd timers — scheduled jobs
• The timer exists, but the job never runs {#not-firing} systemctl list-timers --all: systemd timers — scheduled jobs
• Refusing to start, unit … to trigger not loaded {#no-service} The timer has nothing to point at: systemd timers — scheduled jobs
• Failed to parse calendar specification {#bad-calendar} OnCalendar= has an error, and the timer will not start. systemd timers — scheduled jobs
• Warning: The unit file … changed on disk {#daemon-reload} The unit file was edited, but systemd still runs the old version — and the old schedule. systemd timers — scheduled jobs
• The unit files have no installation config {#no-install} enable was called on the service. systemd timers — scheduled jobs
• The timer fired once and then went quiet {#remain-after-exit} The service has RemainAfterExit=yes. systemd timers — scheduled jobs
• The time has "drifted", logs/certificates look wrong timedatectl → System clock synchronized: timedatectl — time, time zone, and synchronization
• Failed to set time: Automatic time synchronization is enabled {#set-time} Turn off auto-sync first: timedatectl — time, time zone, and synchronization
• Different times in logs and date Logs are often in UTC, while date is in the local zone. timedatectl — time, time zone, and synchronization
• The clock "jumps" after a reboot The RTC is in the wrong mode (set-local-rtc). timedatectl — time, time zone, and synchronization
• Wrong zone inside a container The host's time zone is not the container's; it's set via the TZ variable or by mounting /etc/localtime. timedatectl — time, time zone, and synchronization

Network

• nf_conntrack: table full, dropping packet The table is full (a busy gateway/NAT) — new connections get dropped. conntrack — connection tracking
• NAT/port forwarding works "on and off" conntrack -L shows whether there's a connection entry and its state. conntrack — connection tracking
• A connection "stuck" after changing firewall rules Existing connections are already in the table (ESTABLISHED) and bypass the new rules. conntrack — connection tracking
• High insert_failed in conntrack -S Races/overflow under a large number of concurrent connections; check the limit and timeouts (/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_*). conntrack — connection tracking
• Where to see current connections without conntrack Local application sockets — ss; the kernel's tracking itself (for NAT/firewall) — conntrack -L. conntrack — connection tracking
• curl: (7) Failed to connect … Couldn't connect to server Nothing listens on the address and port, or a firewall drops the connection. curl — HTTP requests from the console
• curl: (6) Could not resolve host The name was not found: curl — HTTP requests from the console
• curl: (60) SSL certificate problem The certificate failed one of three checks: curl — HTTP requests from the console
• curl: (35) … wrong version number https:// went to a port where the server speaks plain HTTP. curl — HTTP requests from the console
• Exit code 0, yet the site returns an error Without -f a 404 or 502 is a normal response for curl. curl — HTTP requests from the console
• curl: (52) Empty reply from server The connection opened, but the server closed it without a response: curl — HTTP requests from the console
• IPs ping, but names don't DNS isn't working. DNS and name resolution
• A name resolves to the wrong IP An entry in /etc/hosts overrides DNS — check it (getent hosts name). DNS and name resolution
• resolv.conf gets overwritten after a reboot Don't edit /etc/resolv.conf directly (it's a symlink to resolved). DNS and name resolution
• Internal names don't resolve, external ones do There's no search domain or the right DNS for the internal zone. DNS and name resolution
• dig works, but the application "can't find" the host dig goes to DNS bypassing NSS. DNS and name resolution
• "The network is slow" at the physical layer sudo ethtool eth0 → Speed and Duplex. ethtool — network card settings
• Growing errors/drops ethtool -S eth0 → rx_errors, rx_dropped, rx_crc_errors. ethtool — network card settings
• I don't know which port this is physically sudo ethtool -p eth0 10 blinks the port's LED for 10 seconds. ethtool — network card settings
• Speed "jumps around" / half-duplex An auto-negotiation problem with the switch. ethtool — network card settings
• Odd latency/timeouts under load Sometimes offload (GRO/TSO) is the culprit with a specific driver. ethtool — network card settings
• 400 Bad Request The request breaks HTTP rules. HTTP — requests, responses, status codes and headers
• 403 Forbidden The server understood the request and refused. HTTP — requests, responses, status codes and headers
• 404 Not Found There is no file at the path the server built. HTTP — requests, responses, status codes and headers
• 405 Not Allowed The method does not fit the resource: HTTP — requests, responses, status codes and headers
• 413 Request Entity Too Large The request body is larger than allowed. HTTP — requests, responses, status codes and headers
• 502 Bad Gateway, 504 Gateway Timeout A proxy response: HTTP — requests, responses, status codes and headers
• 301 when you expect 200 The server sends you to the address in the Location header. HTTP — requests, responses, status codes and headers
• curl: (35) … wrong version number The reverse of 400 … sent to HTTPS port: HTTP — requests, responses, status codes and headers
• curl: (7) Failed to connect … Couldn't connect to server There is no HTTP response: HTTP — requests, responses, status codes and headers
• No address (ip -br a empty/down) The interface is down (sudo ip link set <if> up), or DHCP/netplan didn't run (sudo netplan apply, journalctl -u systemd-networkd). ip — addresses, interfaces and routes
• NO-CARRIER in ip -br l No physical link: ip — addresses, interfaces and routes
• There's an address, but "Network is unreachable" No default route: ip — addresses, interfaces and routes
• Two interfaces, traffic goes "the wrong way" ip route get <IP> shows the actual choice. ip — addresses, interfaces and routes
• IP pings, names don't That's DNS, not ip: ip — addresses, interfaces and routes
• "Is this port reachable from here?" nc -zvw3 host port. nc — checking ports and transferring data
• The port is open locally, but not from outside On the server, nc -zv 127.0.0.1 8080 checks whether it's listening. nc — checking ports and transferring data
• You need a quick L4 connectivity check between hosts nc -l 8080 on one side, nc -zv host 8080 on the other — does TCP/UDP get through on that exact port (useful when debugging a firewall). nc — checking ports and transferring data
• nc "hangs" Without -w, a connection can wait forever. nc — checking ports and transferring data
• ping by IP works, ping by name doesn't The network is fine, DNS isn't working. Network diagnostics
• Not even the gateway pings A local problem: Network diagnostics
• Ping works, but the service is unreachable ICMP gets through, but the port doesn't. Network diagnostics
• Loss/high latency in the middle of the path mtr -rwc 20 shows which hop the loss grows at. Network diagnostics
• ICMP is blocked — ping isn't meaningful Many hosts don't answer ping on purpose. Network diagnostics
• "It works some of the time" Intermittent loss: Network diagnostics
• Host seems down {#host-down} The host did not answer the discovery probes: nmap: which ports are visible from outside
• The port is closed, but the service is running {#closed-but-running} The service listens on another address. nmap: which ports are visible from outside
• The port is filtered, but there is no rule {#filtered} The packets are dropped by the chain policy (policy drop) or by a rule in another table — sudo nft list ruleset shows every table, including fail2ban's and… nmap: which ports are visible from outside
• You requested a scan type which requires root privileges {#needs-root} A SYN scan, UDP (-sU) and OS detection send raw packets. nmap: which ports are visible from outside
• The route is gone after a reboot {#route-lost} It was added with ip route add, and it is not in the files under /etc/netplan/. Persistent networking with netplan
• Network access disappeared after apply There's an error in the config: Persistent networking with netplan
• netplan apply: nothing changed Check the syntax (sudo netplan generate will show the error and the file), whether you edited the right file, and whether the interface name (ip -br a) matches… Persistent networking with netplan
• No DNS after configuring it Add a nameservers: Persistent networking with netplan
• netplan get shows something different from what's on the system netplan get only reads the files in /etc/netplan/, not the actual state of the interfaces. Persistent networking with netplan
• ping: connect: Network is unreachable There's no route to the address: ping — check whether packets reach a host
• Temporary failure in name resolution The name didn't turn into an address: ping — check whether packets reach a host
• 100% packet loss, though the host surely works A firewall drops ICMP — on the host or on the way. ping — check whether packets reach a host
• ping doesn't stop There's no -c: ping — check whether packets reach a host
• 10–30 % loss, replies all over the place The path is overloaded or unstable. ping — check whether packets reach a host
• "Address already in use" when starting a service The port is already taken. ss — sockets and ports
• The service is running but unreachable from outside Is it listening at all: ss — sockets and ports
• The number of CLOSE-WAIT sockets keeps growing The app isn't closing connections (a descriptor leak) — sooner or later you'll hit ulimit. ss — sockets and ports
• Lots of TIME-WAIT Normal with many short-lived client connections; usually not a problem. ss — sockets and ports
• Not sure which process holds a connection ss -tnp shows users:(("nginx",pid=1234,...)). ss — sockets and ports
• Permission denied (publickey) Is the public key present in ~/.ssh/authorized_keys on the server? SSH keys and ssh-agent
• It asks for the key's passphrase every time The key isn't in the agent: SSH keys and ssh-agent
• The server doesn't accept keys at all On the server, in sshd_config: SSH keys and ssh-agent
• Several keys — the wrong one is offered / "Too many authentication failures" The agent tries every key and hits the server's limit. SSH keys and ssh-agent
• Lost the private key It cannot be recovered — generate a new one and update authorized_keys on the servers. SSH keys and ssh-agent
• Permission denied (publickey) The key isn't on the server or permissions on the client/server are wrong. ssh, scp — connecting and copying over SSH
• Host key verification failed The host's key changed (a reinstall/IP change). ssh, scp — connecting and copying over SSH
• Connection timed out vs refused timed out — never arrived (firewall/route/wrong network); refused — arrived, but the port is closed (sshd isn't running / a different port). ssh, scp — connecting and copying over SSH
• The session drops after a timeout ServerAliveInterval 60 in the client config. ssh, scp — connecting and copying over SSH
• ssh still asks about a new host key A hashed name (-H) in known_hosts won't match if you connect using a different name/alias or port — known_hosts entries are tied to a specific host:port pair. ssh-keyscan — collecting host keys
• ssh warns "REMOTE HOST IDENTIFICATION HAS CHANGED" after reinstalling a server Expected: ssh-keyscan — collecting host keys
• Empty output / timeout The host isn't reachable on the SSH port (down, firewalled, wrong port — check -p), or DNS doesn't resolve the name. ssh-keyscan — collecting host keys
• "Does the request even reach the server?" On the server, sudo tcpdump -ni any port 80, then make a request from the client. tcpdump — capturing and analyzing network traffic
• SYN is visible, but no SYN-ACK Packets arrive but the server doesn't reply: tcpdump — capturing and analyzing network traffic
• SYN-ACK is visible, but the connection doesn't come up An asymmetric route or a firewall on the return path/client side. tcpdump — capturing and analyzing network traffic
• Too much output Narrow it with a filter (host/port), limit with -c N, exclude your own session (not port 22), write to .pcap and analyze in Wireshark. tcpdump — capturing and analyzing network traffic
• No permissions / no interface tcpdump requires root (raw socket access). tcpdump — capturing and analyzing network traffic
• Cannot find device "lab0" The command runs in the wrong namespace. The veth pair and network namespaces
• Cannot open network namespace: No such file or directory There's no namespace with that name. The veth pair and network namespaces
• RTNETLINK answers: File exists An interface or address with that name already exists. The veth pair and network namespaces
• The interface is UP, but ping fails Check the other end: The veth pair and network namespaces
• Operation not permitted Only root can change namespaces or look inside them. The veth pair and network namespaces
• There's a bond, but traffic goes over one NIC only That's normal for active-backup (the second is a spare). VLANs and interface bonding
• The VLAN doesn't work The id tag must match the switch port's configuration (a trunk with this VLAN). VLANs and interface bonding
• The network went down after netplan apply A typo or wrong link. VLANs and interface bonding
• A link keeps "flapping" in the bond cat /proc/net/bonding/bond0 shows which slave is dropping; check the cable/port (ethtool). VLANs and interface bonding
• Changed the bond mode, and the network went down completely The switch port is still set for the old mode. VLANs and interface bonding
• ERROR: cannot verify ... certificate A TLS certificate/chain problem. wget — downloading files over the network
• The download broke off wget -c URL resumes from where it stopped, if the server supports Range. wget — downloading files over the network
• Got HTML instead of the file The link points to a page/redirect/login, not the file. wget — downloading files over the network
• It downloads endlessly / mirrors too much Recursion -r without -np/-l has no brakes. wget — downloading files over the network
• Slow / saturating the link --limit-rate= to avoid saturating the link; -q to keep a script's log clean. wget — downloading files over the network
• Empty/sparse answer for a domain The data is hidden by privacy protection (GDPR/WHOIS privacy). whois — domain and IP data
• The answer is "truncated" or redirects to another server The top-level whois server points to the registrar/registry. whois — domain and IP data
• Want to trace an attack and block the network whois 1.2.3.4 → abuse/OrgAbuseEmail (a complaint), NetRange/CIDR (the whole block, for blocking in firewall). whois — domain and IP data
• whois is slow/times out The registry rate-limits queries. whois — domain and IP data

Storage and file systems

• blkid prints nothing The device holds neither a filesystem nor a partition table — a new disk. blkid — filesystem UUID, label and type
• The output has only PARTLABEL and PARTUUID The partition is in the table, but there is no filesystem on it: blkid — filesystem UUID, label and type
• The UUID in fstab is not found After a new mkfs the filesystem has a new UUID. blkid — filesystem UUID, label and type
• Two partitions have the same UUID The partition was copied whole (dd, a cloned disk). blkid — filesystem UUID, label and type
• fstab has PARTUUID instead of UUID Both work, but they are different values: blkid — filesystem UUID, label and type
• setquota doesn't work / "not found" The filesystem is mounted without usrquota/grpquota — check mount | grep -i quota. Disk quotas
• A user hit their limit quota -u user / repquota show the current usage and the ceiling. Disk quotas
• Quotas "reset" after a reboot quotaon doesn't run itself — make sure the options are in /etc/fstab and the quotaon unit is enabled (on ext filesystems). Disk quotas
• quotacheck wants to unmount the filesystem It's best run on an idle/read-only filesystem. Disk quotas
• df says "full", but du can't find that much A file was deleted but is still open by a process — the space isn't freed until it closes: Disk space: df, du, inode
• No space left on device, even though df -h shows free space You're out of inodes: Disk space: df, du, inode
• Logs filled up the disk du -sh /var/log/*; set up rotation (logrotate) and the journal (journalctl --vacuum-time=, see journalctl). Disk space: df, du, inode
• Deleted files, but the space didn't come back Either a deleted-but-open file (see lsof +L1 above), or it's a different partition — check df -h path. Disk space: df, du, inode
• Error: /dev/vdb: unrecognised disk label The disk has no partition table yet — the normal state of a new disk: fdisk and parted — partition a disk
• mount: wrong fs type, bad option, bad superblock You are mounting a disk or partition that has no filesystem. fdisk and parted — partition a disk
• The partition was created but there is no /dev/vdb1 The kernel has not re-read the table: fdisk and parted — partition a disk
• The kernel still uses the old table fdisk wrote the table, but the kernel keeps the old one because a partition of this disk is mounted. fdisk and parted — partition a disk
• Warning: The resulting partition is not properly aligned The start is given in sectors or bytes that do not match the disk's blocks. fdisk and parted — partition a disk
• An ext4 error, the filesystem went read-only {#ext4-errors} ext4 found damage and wrote EXT4-fs error to the kernel log (see dmesg). Filesystems: ext4, xfs, fsck
• Can't shrink xfs xfs cannot be shrunk at all. Filesystems: ext4, xfs, fsck
• "Structure needs cleaning" (xfs) Corruption: Filesystems: ext4, xfs, fsck
• "Not enough space" on a big disk that has free space 5% is reserved for root by default. Filesystems: ext4, xfs, fsck
• Two partitions with the same UUID after cloning Change one of them: Filesystems: ext4, xfs, fsck
• failed to set up loop device: Permission denied losetup: Loop device — a file as a disk
• There is no /dev/loop0p1 The device was attached without -P, so the kernel did not read the table. Loop device — a file as a disk
• losetup: cannot find an unused loop device No free devices are left, usually because of forgotten attachments. Loop device — a file as a disk
• The file is deleted but the space did not come back The file is still attached to a loop device. Loop device — a file as a disk
• losetup -d did not detach the device The device is mounted. Loop device — a file as a disk
• Plugged in a disk, but lsblk doesn't show it lsblk doesn't rescan the bus — the kernel has to see the device first. lsblk — block devices
• The device is in lsblk, but not mounted lsblk -f with an empty MOUNTPOINT column — the partition exists but isn't mounted and/or isn't in /etc/fstab. lsblk — block devices
• Device names shift between reboots /dev/sdX can change when disks are reordered. lsblk — block devices
• Extended the LV, but there's no extra space The LV isn't the filesystem. LVM logical volumes
• vgs shows little free space (VFree=0) The group is full — add a PV: LVM logical volumes
• A snapshot "overflowed" and became invalid A snapshot stores changes; if the original volume is written to heavily and the snapshot is too small, it overflows (lvs shows 100%). LVM logical volumes
• The volume is missing after a reboot sudo vgchange -ay (activate it), and check auto-activation and /etc/fstab (mount by /dev/<vg>/<lv> or UUID). LVM logical volumes
• Accidentally ran lvremove on the wrong volume Data is gone. LVM logical volumes
• contains a ext4 file system … Proceed anyway? (y,N) The partition already has a filesystem, and mkfs.ext4 asks again. mkfs — create a filesystem
• /dev/vdb1 is mounted; will not make a filesystem here! The partition is mounted and in use. mkfs — create a filesystem
• appears to contain an existing filesystem (mkfs.xfs) mkfs.xfs found another filesystem and refused. mkfs — create a filesystem
• The file /dev/vdb1 does not exist and no size was specified There is no such partition: mkfs — create a filesystem
• After mkfs the partition does not mount from fstab mkfs produced a new UUID, and fstab still has the old one. mkfs — create a filesystem
• mkfs on the whole disk instead of a partition mkfs.ext4 /dev/vdb (no digit) puts the filesystem on the whole disk, over the partition table. mkfs — create a filesystem
• wrong fs type, bad option, bad superblock {#wrong-fs-type} mount answers wrong fs type, bad option, bad superblock on /dev/vdb, missing codepage or helper program, or other error and suggests looking at dmesg. Mounting and fstab
• your fstab has been modified, but systemd still uses the old version {#daemon-reload} mount prints this hint after fstab is edited: Mounting and fstab
• "target is busy" on umount Something is using the filesystem. Mounting and fstab
• An error in fstab → the system didn't boot A bad line can drop the boot into emergency mode. Mounting and fstab
• You mounted it, but the space isn't where you expected If the mount point already had files in it, they're "hidden" under the mounted filesystem (not deleted). Mounting and fstab
• A network filesystem hangs the boot Without _netdev, the system waits for the filesystem before networking is up. Mounting and fstab
• The wrong filesystem/device /dev/sdX names can shift between reboots — mount by UUID (blkid), not by /dev/sdX. Mounting and fstab
• mount.nfs: access denied by server The client's IP doesn't match the mask in /etc/exports, or exportfs -ra wasn't run after editing it. NFS network filesystem
• Ownership/permissions on the mount look "wrong" NFS maps by UID/GID — the client and server need matching ones (or set up idmapping for NFSv4). NFS network filesystem
• Connection timed out / mount hangs The firewall or the service. NFS network filesystem
• Processes hang when the server is unreachable (D state) A "hard" mount (the default) hangs processes indefinitely. NFS network filesystem
• Stale file handle The export was recreated, or the filesystem was remounted on the server. NFS network filesystem
• The image is larger than it should be qcow2 doesn't shrink itself after data is deleted inside it. qemu-img — virtual machine disk images
• Backing file has changed / the overlay is broken An overlay is tightly bound to its backing image. qemu-img — virtual machine disk images
• You resized the disk, but the VM has no extra space qemu-img resize only changes the container. qemu-img — virtual machine disk images
• Don't touch a mounted/in-use image Never run qemu-img against a running VM's disk — you'll corrupt it. qemu-img — virtual machine disk images
• The image is "corrupted" after a crash qemu-img check disk.qcow2 (and -r all to attempt a repair). qemu-img — virtual machine disk images
• xfs_growfs: "specified file is not on device" You passed the device name (/dev/labvg/data) instead of the mountpoint. resize2fs / xfs_growfs — growing a filesystem
• df -h doesn't show the new size after lvextend The volume grew, but the filesystem didn't yet: resize2fs / xfs_growfs — growing a filesystem
• resize2fs: "New size smaller than minimum" The target size is smaller than the data already on the filesystem. resize2fs / xfs_growfs — growing a filesystem
• The resize hangs or complains about space Free space is needed at the volume group level, not just the logical volume: resize2fs / xfs_growfs — growing a filesystem
• The array is degraded ([_U]) A disk dropped out. Software RAID with mdadm
• It doesn't assemble after a reboot The config wasn't saved. Software RAID with mdadm
• The rebuild takes forever / is slow Normal for large disks (hours). Software RAID with mdadm
• RAID is not a backup! RAID protects against a disk failure, but not against deletion, ransomware, or a mistake. Software RAID with mdadm
• A disk from an old array won't let you recreate on it Leftover metadata: Software RAID with mdadm

Logs and audit

• The journal is empty after a reboot journald isn't persistent. journalctl — reading the systemd journal
• The journal is filling up the disk journalctl --disk-usage, then --vacuum-time=7d or --vacuum-size=500M. journalctl — reading the systemd journal
• You need logs for a specific failure journalctl -xeu <unit> --since "5 min ago" is the most common diagnostic query (see systemctl). journalctl — reading the systemd journal
• The time in the journal looks "wrong" journald stores entries in UTC internally and shows the local timezone. journalctl — reading the systemd journal
• A service's logs show up in /var/log too (duplicates) journald and rsyslog can both log the same thing. journalctl — reading the systemd journal
• The log keeps growing, no rotation happens Is there a config in /etc/logrotate.d/? Log rotation with logrotate
• After rotation the service keeps writing to the old (deleted) file There's no postrotate with a reload, and no copytruncate. Log rotation with logrotate
• "error: skipping … because parent directory has insecure permissions" logrotate is complaining about permissions/ownership of the log directory. Log rotation with logrotate
• The disk is already full of logs Urgent fix: Log rotation with logrotate
• The log isn't being written Is the service active (systemctl status rsyslog)? rsyslog — system logs
• Logs grow and fill the disk Rotation is logrotate's job (rsyslog never cleans up on its own). rsyslog — system logs
• Logs aren't arriving at the central server Is the input module enabled on the receiver (imtcp/imudp, port 514) and the port open (firewall, nc -zv host 514)? rsyslog — system logs
• Duplicate lines / "logged twice" journald forwards to rsyslog (imjournal) and the same facility gets written to several files. rsyslog — system logs
• journald vs rsyslog — where to look Structured search by service — journalctl (-u, _COMM=); familiar text files and forwarding — /var/log/* (rsyslog). rsyslog — system logs

Boot

• You are in emergency mode {#emergency-mode} The boot was stopped by a dependency of local-fs.target: Boot targets: default, multi-user, rescue and emergency
• The service works, but is not running after a reboot {#not-enabled} It was not enabled: Boot targets: default, multi-user, rescue and emergency
• The service starts before the network and fails {#before-network} The unit has no After=network-online.target, or only After=network.target. Boot targets: default, multi-user, rescue and emergency
• Dependency failed for … {#dependency-failed} A unit with Requires= (or BindsTo=) did not start because its dependency failed to start. Boot targets: default, multi-user, rescue and emergency
• is-system-running: degraded {#degraded} The boot finished, but some unit failed. Boot targets: default, multi-user, rescue and emergency
• user-data was not applied The first line must be exactly #cloud-config (no leading spaces/BOM), and valid YAML (cloud-init schema --system). cloud-init — first-boot VM setup
• Packages didn't install / runcmd didn't run /var/log/cloud-init-output.log has the command output and errors (no network for apt? cloud-init — first-boot VM setup
• Can't SSH in after provisioning The key never reached the right user, or the network (netplan) never came up. cloud-init — first-boot VM setup
• Hostname/network "revert" to default cloud-init resets them on every boot unless you disable that: cloud-init — first-boot VM setup
• Disable cloud-init entirely sudo touch /etc/cloud/cloud-init.disabled (or remove the package). cloud-init — first-boot VM setup
• read kernel buffer failed: Operation not permitted {#not-permitted} kernel.dmesg_restrict = 1: dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• journalctl -k prints -- No entries -- {#no-entries} Without sudo, a user outside the adm and systemd-journal groups sees only their own entries. dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• The buffer does not start at boot, or is empty {#buffer-wrapped} The buffer overflowed and old lines were pushed out, or someone cleared it (dmesg -C). dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• No journal boot entry found {#no-previous-boot} The journal has no previous boot: dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• Time in dmesg -T and the journal differ {#time-mismatch} -T counts time from boot and ignores suspends. dmesg — kernel messages: OOM, disk and filesystem errors, segfaults
• A device/driver isn't working dmesg | grep -i <device> for initialization errors. Kernel modules
• The OOM-killer killed a process dmesg -T | grep -i oom or journalctl -k | grep -i oom — the kernel killed a process under memory pressure (see oom-killer). Kernel modules
• A module doesn't load at boot Add it to /etc/modules-load.d/*.conf; check modinfo and dependencies (depmod -a). Kernel modules
• modprobe -r: "Module is in use" Something is using the module (lsmod's "Used by" column). Kernel modules
• The disk is throwing I/O errors dmesg showing "I/O error, dev sdX" is a sign of a failing disk; check SMART (smartctl -a) and make a backup (see filesystems). Kernel modules
• Changed it, reset after a reboot {#reset-after-reboot} sysctl -w and writes to /proc/sys last until reboot. sysctl — kernel parameters at runtime
• permission denied on key, yet the value is printed {#no-sudo} Without sudo, sysctl prints the line with the new value and exits with 0, but does not change the value. sysctl — kernel parameters at runtime
• A file in sysctl.d is not applied {#file-ignored} The name does not end in .conf — the file is skipped silently. sysctl — kernel parameters at runtime
• cannot stat /proc/sys/…: No such file or directory {#cannot-stat} The answer is sysctl: sysctl — kernel parameters at runtime
• invalid syntax, continuing {#invalid-syntax} On the command line name=value has no spaces. sysctl — kernel parameters at runtime
• Forwarding is on, but packets do not pass {#forwarding} Besides net.ipv4.ip_forward = 1 you need routes (ip) and allowing firewall rules, plus a masquerade rule for NAT. sysctl — kernel parameters at runtime
• The system doesn't finish booting In GRUB, pick a previous kernel (Advanced) — it boots → the new kernel is the problem. System boot and GRUB
• It hangs on boot because of fstab An error in /etc/fstab drops you into emergency mode. System boot and GRUB
• Slow boot systemd-analyze blame plus critical-chain show which units are slow (often a hung network wait or mount with a timeout, NetworkManager-wait-online). System boot and GRUB
• Forgot the root password GRUB → e → add init=/bin/bash to the kernel line → boot → mount -o remount,rw / → passwd → exec /sbin/init or reboot. System boot and GRUB
• initramfs prompt after a kernel update The root/a module wasn't found. System boot and GRUB
• Bootup is not yet finished {#not-finished} The boot is still going: systemd-analyze: what slows the boot and what is wrong in a unit
• Removed the first line of blame, and the boot is just as slow {#blame-misleads} blame shows how long each unit took to start, not who waited for whom. systemd-analyze: what slows the boot and what is wrong in a unit
• verify is silent, and the service does not start {#verify-silent} A misspelled key or a wrong value gives code 0 — read the output. systemd-analyze: what slows the boot and what is wrong in a unit
• Unit … not found {#unit-not-found} Requires= or Wants= names a unit that does not exist: systemd-analyze: what slows the boot and what is wrong in a unit
• Transaction order is cyclic {#ordering-cycle} The units wait for each other through After=. systemd-analyze: what slows the boot and what is wrong in a unit
• uname -r still shows the old version after apt upgrade Upgrading the kernel installs new files, but the running kernel is still the old one until a reboot happens. The kernel
• What's the difference between the kernel and the distribution? The kernel is one process and a set of drivers; a distribution is the kernel plus system utilities, a package manager, and configuration (see uname for the… The kernel
• A driver I need doesn't load That's almost always about a kernel module, not the kernel as a whole — see kernel-modules: The kernel
• I want to change kernel behavior without rebooting Some parameters change on the fly through sysctl (see sysctl), but not all of them: The kernel

Packages

• "Could not get lock /var/lib/dpkg/lock" apt/unattended-upgrades is already running. apt / dpkg — package management
• dpkg was interrupted, "broken packages" apt / dpkg — package management
• apt update: NO_PUBKEY / 404 / repository changed A broken or stale source in sources.list.d/. apt / dpkg — package management
• A specific version / rollback apt-cache policy PKG shows what's available; apt install pkg=version installs it; apt-mark hold pins it. apt / dpkg — package management
• Free up space used by the cache sudo apt clean (the whole cache) / autoclean (only stale files); autoremove --purge (see disk-usage). apt / dpkg — package management
• The package installs slowly / is huge Snap bundles its dependencies (heavier than apt) — this is normal. snap — Ubuntu packages
• Snap can't see files outside $HOME The sandbox (confinement) restricts access. snap — Ubuntu packages
• It updated and broke sudo snap revert package rolls back to the previous version (snap keeps it). snap — Ubuntu packages
• Is this apt or snap? which cmd, snap list | grep cmd, dpkg -S $(which cmd) (see which, apt). snap — Ubuntu packages
• Old revisions take up space snap keeps past revisions. snap — Ubuntu packages

Security

• "Permission denied" even though the permissions (rwx) look right AppArmor is blocking it. AppArmor
• The service works, but cannot access a new directory The profile does not allow that path. AppArmor
• The profile is not reread after an edit sudo apparmor_parser -r <profile> (or reload apparmor). AppArmor
• I want to temporarily disable it for one service sudo aa-complain <profile> (soft) or aa-disable. AppArmor
• A snap app and AppArmor Snap uses its own profiles (confinement); denials are also visible in journalctl -k (see snap snap connections). AppArmor
• Who changed an important file? Set a watch (auditctl -w file -p wa -k label), then ausearch -k label -i → uid/command/time. auditd — security auditing
• Too many events / noise / losses (lost) auditctl -s shows lost/backlog. auditd — security auditing
• Rules disappeared after a reboot auditctl -w/-a rules are temporary. auditd — security auditing
• I can't change or delete rules Immutable mode is on (-e 2) — rules are locked until reboot (by design, to protect against tampering). auditd — security auditing
• auditd vs journald journald handles operational service logs (journalctl); auditd handles who-did-what for security (an immutable trail). auditd — security auditing
• AppArmor denials are gone from journalctl -k {#apparmor-denials} While auditd runs, kernel audit messages, apparmor="DENIED" included, go to it and are written to /var/log/audit/audit.log (type=AVC), and journalctl -k gets… auditd — security auditing
• Permission denied on a port below 1024 {#bind-denied} PermissionError: Capabilities: root split into parts
• unable to set CAP_SETFCAP effective capability {#setfcap} setcap ran without sudo: Capabilities: root split into parts
• fatal error: Invalid argument {#invalid-argument} setcap did not understand the part name or the file. Capabilities: root split into parts
• The part is gone after a copy or an upgrade {#lost} getcap prints nothing and the program gets Permission denied again. Capabilities: root split into parts
• Root gets Operation not permitted {#root-denied} The process runs as UID 0 and the kernel refuses. Capabilities: root split into parts
• I banned myself From the console (hypervisor/serial) or from an allowed IP: fail2ban
• The jail isn't banning anyone Is enabled = true? fail2ban
• The address is not banned: Ignore by ip {#ignoreip} The address is in ignoreip: fail2ban
• It's banning, but the attack continues from other IPs That's a distributed brute force. fail2ban
• Not sure whether it's working at all fail2ban-client status sshd → Total/Currently failed and banned. fail2ban
• I locked myself out of SSH Get in through the hypervisor console/serial; ufw disable, or fix the rule set and reload it. Firewall — nftables / ufw
• A rule isn't taking effect Order matters: Firewall — nftables / ufw
• The port isn't reachable from outside, even though the service is listening sudo ufw status / nft list ruleset — is the port allowed; don't forget an external NAT/router. Firewall — nftables / ufw
• The port is open from the machine itself, though it was closed {#check-from-outside} curl http://192.0.2.1:8080/ on the machine itself answers 200 although the drop policy closes the port. Firewall — nftables / ufw
• gpg: decryption failed: No secret key There's no private key for this recipient (it was encrypted for a different key), or the key isn't imported. GPG — encryption and signatures
• Can't check signature: No public key The author's public key is missing. GPG — encryption and signatures
• Forgot the private key's passphrase It cannot be recovered. GPG — encryption and signatures
• Inappropriate ioctl for device (in scripts/CI) GPG can't prompt for a passphrase (no TTY). GPG — encryption and signatures
• WARNING: ... not certified with a trusted signature The key is imported but not trusted. GPG — encryption and signatures
• Password 123 is accepted {#weak-password-accepted} grep pwquality /etc/pam.d/common-password prints nothing — the module is not in the stack: PAM: login checks, password policy and lockout
• BAD PASSWORD on a password change {#bad-password} The reason is the text after the colon: PAM: login checks, password policy and lockout
• The account is locked due to N failed logins {#account-locked} pam_faillock counted deny failures within fail_interval. PAM: login checks, password policy and lockout
• Many failures, no lockout {#no-lockout} The pam_faillock lines are missing from common-auth or sit in the wrong place: PAM: login checks, password policy and lockout
• pam-auth-update: Local modifications {#local-modifications} The common-* files were edited by hand, and pam-auth-update refuses to rebuild them — the profile does not get enabled. PAM: login checks, password policy and lockout
• The service fails after ProtectSystem=strict {#read-only} The log shows Read-only file system or Permission denied on a write. Service hardening: User=, ProtectSystem, NoNewPrivileges
• status=217/USER {#217-user} The user from User= does not exist on the machine. Service hardening: User=, ProtectSystem, NoNewPrivileges
• Permission denied on a port below 1024 {#low-port} After User= the service lost the right to port 80 or 443. Service hardening: User=, ProtectSystem, NoNewPrivileges
• The rating dropped, the process did not change {#score-not-applied} systemd-analyze security shows 1.5 OK, while ps shows the service as root. Service hardening: User=, ProtectSystem, NoNewPrivileges
• sudo: The "no new privileges" flag is set {#nnp-sudo} The service's script calls sudo or another setuid program, and the unit has NoNewPrivileges=yes. Service hardening: User=, ProtectSystem, NoNewPrivileges
• Permission denied (publickey) Permissions on the server: sshd — the SSH server
• Connection refused The server isn't listening on the port: sshd — the SSH server
• Connection timed out A network/firewall-level issue (never reached sshd): sshd — the SSH server
• Changed the port — can't connect SELinux/the port isn't allowed, the firewall is still on the old port, or the client is connecting to 22. sshd — the SSH server
• Locked yourself out with a config change Get in through the hypervisor console/serial and roll back sshd_config (or sshd -t would have shown the error before reload — always check). sshd — the SSH server
• user is not in the sudoers file The user isn't in the sudo group. sudo — running commands as another user
• sudo: a terminal is required to read the password No TTY (over SSH without -t, or from a script). sudo — running commands as another user
• NOPASSWD doesn't work for switching users (root) NOPASSWD: sudo — running commands as another user
• The command works by hand but not through sudo (PATH/env) env_reset/secure_path clean the environment. sudo — running commands as another user
• Broke sudoers, lost access visudo won't let you save a file with an error. sudo — running commands as another user
• Locked yourself out of SSH The rule for 22/tcp must come before the DROP policy. The classic iptables firewall
• Rules disappeared after a reboot iptables isn't persistent. The classic iptables firewall
• Order matters Rules apply top to bottom, stopping at the first match. The classic iptables firewall
• Counters don't grow — the rule isn't firing -L -v shows per-rule packet counters. The classic iptables firewall
• Not sure whether it's really iptables or nftables On Ubuntu, iptables is often iptables-nft (the nftables backend). The classic iptables firewall
• certificate has expired {#expired} curl: TLS and certificates — openssl, the chain, verification
• unable to get local issuer certificate {#incomplete-chain} The client did not reach a trusted root. TLS and certificates — openssl, the chain, verification
• no alternative certificate subject name matches {#name-mismatch} curl: TLS and certificates — openssl, the chain, verification
• self-signed certificate curl: TLS and certificates — openssl, the chain, verification
• wrong version number curl: TLS and certificates — openssl, the chain, verification
• The server sends the old certificate The file is replaced but s_client shows the old notAfter: TLS and certificates — openssl, the chain, verification
• Updates are not installed {#not-running} 20auto-upgrades has "0" or does not exist. unattended-upgrades: security updates with no one at the keyboard
• apt waits for a lock: (unattended-upgr) {#lock} apt says Could not get lock /var/lib/dpkg/lock-frontend. unattended-upgrades: security updates with no one at the keyboard
• The update is installed, the vulnerability is still there {#still-vulnerable} The service runs an old copy of a file: unattended-upgrades: security updates with no one at the keyboard
• A service restarted by itself at night {#unexpected-restart} needrestart restarted it after a scheduled update. unattended-upgrades: security updates with no one at the keyboard
• visudo: unable to open /etc/sudoers: Permission denied visudo was run without root. visudo — edit sudoers safely
• syntax error and the What now? prompt The saved file failed the check: visudo — edit sudoers safely
• bad permissions, should be mode 0440 The file in /etc/sudoers.d was created with ordinary permissions. visudo — edit sudoers safely
• The rule exists, but sudo doesn't see it The file name contains a dot (deploy.conf) or ends with ~ — sudo silently skips such files, and visudo -c doesn't list them. visudo — edit sudoers safely
• The rule exists, but a password is still asked Of all matching rules, the last one wins. visudo — edit sudoers safely
• The tunnel is up, but there's no connectivity sudo wg show — is there a "latest handshake"? WireGuard — VPN on keys
• There's a handshake, but pings don't go through AllowedIPs doesn't include the needed addresses (for each peer, it's "which IPs are behind it"). WireGuard — VPN on keys
• Can't connect after a reboot systemctl enable wg-quick@wg0. WireGuard — VPN on keys
• Asymmetric / one side sees the other, but not vice versa AllowedIPs must be configured consistently on both sides. WireGuard — VPN on keys

Automation

• Could not match supplied host pattern {#no-hosts} The inventory was not read: ansible-playbook: running a playbook
• leaves us with no hosts to target {#limit-no-hosts} -l names a machine or group that is not in the inventory. ansible-playbook: running a playbook
• UNREACHABLE! {#unreachable} Ansible could not connect over SSH: ansible-playbook: running a playbook
• couldn't resolve module/action {#unknown-module} A typo in the module name, or the collection is missing. ansible-playbook: running a playbook
• The requested handler … was not found {#handler-not-found} notify names a handler that does not exist: ansible-playbook: running a playbook
• AnsibleUndefinedVariable {#undefined-variable} A template or task uses a variable that does not exist: ansible-playbook: running a playbook
• Syntax Error while loading YAML {#yaml-syntax} The indentation does not match: ansible-playbook: running a playbook
• Permission denied without become {#become} The task changes something owned by root, and the play runs as a user. ansible-playbook: running a playbook
• Every run is changed again {#always-changed} shell or command without creates:/changed_when:, or a template with a variable that changes every time (a date, a random number). ansible-playbook: running a playbook
• UNREACHABLE! {#unreachable} Ansible could not connect over SSH; the module never ran, the exit code is 4. Ansible: inventory, modules and one-off commands
• Destination /etc not writable {#not-writable} The module runs without root rights: Ansible: inventory, modules and one-off commands
• sudo: a password is required You need --become and working sudo. Ansible: inventory, modules and one-off commands
• A task is "changed" every time You're using command/shell instead of an idempotent module. Ansible: inventory, modules and one-off commands
• Vault: can't decrypt anything You need the vault password (--ask-vault-pass or --vault-password-file). Ansible: inventory, modules and one-off commands

Containers

• Unable to find image … locally {#not-found} No image with that name and tag is on the machine, so Docker went to the registry for it. Container images — layers, tags, registries and digests
• conflict: unable to delete … must be forced {#in-use} A container needs the image, even a stopped one. Container images — layers, tags, registries and digests
• Rebuilt the image, but the container runs the old one {#old-image} A container is created from the image that held the name at docker run time. Container images — layers, tags, registries and digests
• <none> instead of a name {#dangling} Dangling images remain after rebuilding under the same tag. Container images — layers, tags, registries and digests
• Permission denied on an archive from docker save {#save-permission} sudo docker save -o file.tar creates the file as root with mode 600, and student cannot read it. Container images — layers, tags, registries and digests
• no configuration file provided: not found {#no-file} The current directory has no compose.yaml. docker compose — a container stack in one file
• additional properties '…' not allowed {#unknown-key} A misspelled key or a key at the wrong indent level. docker compose — a container stack in one file
• yaml: line 3: found character that cannot start any token {#yaml-tab} Line 3 has a tab or another character YAML cannot start a value with. docker compose — a container stack in one file
• service "backend" refers to undefined volume {#undefined-volume} The service mounts a named volume missing from the top-level volumes:. docker compose — a container stack in one file
• external volume "…" not found {#external-volume} A volume with external: docker compose — a container stack in one file
• dependency failed to start: container … is unhealthy {#unhealthy} backend failed its healthcheck, and proxy with condition: docker compose — a container stack in one file
• The "X" variable is not set {#variable-not-set} The "MODE" variable is not set. docker compose — a container stack in one file
• the attribute version is obsolete {#version-obsolete} A version: docker compose — a container stack in one file
• A service in a restart loop {#restart-loop} docker compose ps shows Restarting: docker compose — a container stack in one file
• Pulling … failed to resolve reference {#pull-failed} The service image is not on the machine, so compose went to the registry for it. docker compose — a container stack in one file
• service "proxy" is not running {#not-running} exec works only in a running service. docker compose — a container stack in one file
• permission denied … docker.sock {#permission-denied} The user is not in the docker group. Docker — containers: running, ports, volumes, networks
• Cannot connect to the Docker daemon {#daemon-down} The service is not running: Docker — containers: running, ports, volumes, networks
• port is already allocated {#port-allocated} Bind for 0.0.0.0:8081 failed: Docker — containers: running, ports, volumes, networks
• The container name … is already in use {#name-in-use} A container with that name exists, maybe a stopped one (docker ps -a). Docker — containers: running, ports, volumes, networks
• The container exits at once with code 0 {#exited-zero} The main process finished without an error. Docker — containers: running, ports, volumes, networks
• docker stop takes 10 seconds and gives 137 {#stop-137} docker stop sends SIGTERM and waits 10 seconds. Docker — containers: running, ports, volumes, networks
• container … is not running {#not-running} docker exec works only in a running container. Docker — containers: running, ports, volumes, networks
• docker logs | grep finds nothing {#logs-stderr} docker logs returns the container output on two streams: Docker — containers: running, ports, volumes, networks
• Data gone after re-creating {#data-lost} The data was in the container layer. Docker — containers: running, ports, volumes, networks
• Containers cannot see each other by name {#no-dns} Both are on the default bridge network: Docker — containers: running, ports, volumes, networks
• volume is in use / has active endpoints {#in-use} A container needs the volume or network, maybe a stopped one. Docker — containers: running, ports, volumes, networks
• DEPRECATED: The legacy builder is deprecated {#legacy-builder} The old builder runs the build: Dockerfile — an image recipe and docker build
• failed to resolve reference at the FROM step {#from-not-found} The FROM image is not on the machine, so Docker went to the registry. Dockerfile — an image recipe and docker build
• COPY with a path above the context {#outside-context} The COPY path goes above the context: Dockerfile — an image recipe and docker build
• failed to compute cache key: not found {#file-not-found} The old builder says COPY failed: Dockerfile — an image recipe and docker build
• failed to read dockerfile {#no-dockerfile} failed to read dockerfile: Dockerfile — an image recipe and docker build
• executable file not found in $PATH {#exec-not-found} The image built and the container was created but did not start (status Created): Dockerfile — an image recipe and docker build
• A container from the new image exits at once with 0 {#exits-at-once} The start command finished or went to the background. Dockerfile — an image recipe and docker build
• The change did not reach the container {#change-not-applied} Check three things in order. Dockerfile — an image recipe and docker build