Inside the system: resources, kernel, processes / Stage 11 of 13

The system did not boot: emergency mode

Not running Starting usually ~20 s

Step 6 of 7 · lab · 20 min

The machine after a reboot

The power went off at night, and the reports machine did not come back into service — the console shows emergency mode. Find what stopped the boot, fix it and make the machine boot by itself with the reports in place.

Situation

The power in the server room went off at night. After it came back, the shop's reports machine did not return to service: its console shows emergency mode. The reports live on a separate disk that is mounted at /srv/reports.

The terminal is now this machine's console. If you see no prompt, press Enter.

Task

Bring the machine back into service so that it boots by itself:

After a reboot the machine boots into the normal mode, without emergency mode: systemctl is-system-running gives running.
The reports disk is sound, and the check at boot passes. Do not create the disk anew and do not change the /srv/reports line in /etc/fstab: the check at boot and the required mounting of the disk stay.
/srv/reports is mounted from the reports disk, the 2026 reports in /srv/reports/2026 are intact.

In emergency mode you are root, sudo is not needed. "Check" answers in emergency mode too, but it counts a boot that went without it: after the fix, reboot the machine — systemctl reboot.

There is no internet on the machine.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 3

After a reboot the machine boots into the normal mode, without emergency mode (systemctl is-system-running — running) — not done yet
The reports disk is sound and the same one (not created anew), the check at boot passes — not done yet
/srv/reports is mounted from the reports disk, the 2026 reports are intact — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 3

After a reboot the machine boots into the normal mode, without emergency mode (systemctl is-system-running — running) — not done yet
The reports disk is sound and the same one (not created anew), the check at boot passes — not done yet
/srv/reports is mounted from the reports disk, the 2026 reports are intact — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 3

After a reboot the machine boots into the normal mode, without emergency mode (systemctl is-system-running — running) — not done yet
The reports disk is sound and the same one (not created anew), the check at boot passes — not done yet
/srv/reports is mounted from the reports disk, the 2026 reports are intact — not done yet

Boot · picked for this lab

Boot targets: default, multi-user, rescue and emergency

Cheat sheet

$_ — insert into the terminal

systemctl get-default the default boot target
sudo systemctl set-default multi-user.target
systemctl list-units --type=target active targets
systemctl list-dependencies multi-user.target what it pulls in
systemctl list-dependencies --reverse network-online.target
systemctl is-system-running running, degraded, starting
sudo systemctl rescue switch to rescue now
sudo systemctl default back to the default target

Change the default target

text

student@ubuntu:~$ sudo systemctl set-default graphical.target
Removed "/etc/systemd/system/default.target".
Created symlink /etc/systemd/system/default.target
  → /usr/lib/systemd/system/graphical.target.

The target changes at the next boot. To switch to it now — sudo systemctl isolate <target>.

Who starts a unit at boot

bash

systemctl is-enabled nginx                  # enabled, disabled, static
systemctl show -p WantedBy,RequiredBy nginx  # which targets want it
systemctl list-dependencies --reverse nginx.service

Boot into rescue or emergency

In the GRUB menu select an entry and press e.
At the end of the line that starts with linux, add systemd.unit=rescue.target or systemd.unit=emergency.target.
Ctrl+X boots with this line, once.

On the lab machine the GRUB menu is hidden (GRUB_TIMEOUT=0) and goes to the VM screen (GRUB_TERMINAL=console), while the lesson terminal is the ttyS0 serial console: kernel and systemd messages go there, GRUB's do not. More about the boot loader — boot-grub.

Fix things from emergency and boot on

bash

journalctl -b -p err --no-pager  # errors of this boot
findmnt -no OPTIONS /            # ro? then:
mount -o remount,rw /            # root writable
nano /etc/fstab                  # fix the line
systemctl daemon-reload          # reread fstab
mount -a                         # mount everything from fstab
systemctl default                # continue the boot

journalctl -xb without --no-pager opens the journal in less — quit with q; how to find the lines you need — journalctl. mount -a without errors means fstab is fine, and systemctl default takes the boot to the default target; systemctl reboot checks the same with a reboot.

You are in emergency mode

The boot was stopped by a dependency of local-fs.target: most often an /etc/fstab line with a device that does not exist or a wrong UUID. The console shows before the prompt:

text

[ TIME ] Timed out waiting for device dev-d…-0c1d2e3f4a5b.
[DEPEND] Dependency failed for data.mount - /data.
[DEPEND] Dependency failed for local-fs.target - Local File Systems.

systemd waits 90 seconds for a missing device and only then goes to emergency; a mistake in an option or a file system type fails at once — on the lab machine emergency comes 10–15 seconds after the reboot. The same lines are in journalctl -xb. Fix the line using blkid; mount a disk the system can live without with nofail. More — mount.

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

The fstab line is right, but the disk failed its check at boot. For a line with 1 or 2 in the sixth field systemd runs systemd-fsck@… before mounting — fsck in automatic mode, which fixes only what is safe. An excerpt from journalctl -b:

text

reports: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
fsck failed with exit status 4.
Dependency failed for srv-reports.mount - /srv/reports.
Dependency failed for local-fs.target - Local File Systems.

In emergency that disk is not mounted, so it can be checked right away: fsck <device> without -a asks about every fix, fsck -y agrees to all of them. Then mount <directory> and systemctl reboot. nofail or 0 in the sixth field removes the stop, but the errors stay on the disk. More about the check — filesystems.

The service works, but is not running after a reboot

It was not enabled: systemctl is-enabled <service> prints disabled, and no target wants it. sudo systemctl enable <service> creates the link in multi-user.target.wants. static means the unit has no [Install] section — add one.

The service starts before the network and fails

The unit has no After=network-online.target, or only After=network.target. It needs the pair of lines from boot-targets; Wants= without After= sets no order.

Dependency failed for …

A unit with Requires= (or BindsTo=) did not start because its dependency failed to start. The cause is in the dependency's journal: journalctl -b -u <dependency>.

is-system-running: degraded

The boot finished, but some unit failed. systemctl --failed shows which; after the fix — sudo systemctl reset-failed.

rescue and isolate over SSH

systemctl rescue, systemctl emergency and isolate into a target without the network stop ssh.service: the SSH session drops, and you can come back only from the console. Switch targets when the console is at hand.

1 · Foundations

2 · The system

3 · Admin craft

4 · Server in production

5 · Operations

The system did not boot: emergency mode

The machine after a reboot

Situation

Task

Goals

Hints

Lab goals

Lab goals

Boot targets: default, multi-user, rescue and emergency

Cheat sheet

Change the default target

Who starts a unit at boot

Boot into rescue or emergency

Fix things from emergency and boot on

You are in emergency mode

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

The service works, but is not running after a reboot

The service starts before the network and fails

Dependency failed for …

is-system-running: degraded

rescue and isolate over SSH

The system did not boot: emergency mode

Situation

Task

Signing in is needed to start the machine

Change the default target

Who starts a unit at boot

Boot into rescue or emergency

Fix things from emergency and boot on

You are in emergency mode

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

The service works, but is not running after a reboot

The service starts before the network and fails

Dependency failed for …

is-system-running: degraded

rescue and isolate over SSH