Inside the system: resources, kernel, processes / Stage 11 of 13 · step 6 of 7

The machine after a reboot

~20 min

Not running Starting usually ~20 s

Situation

The power in the server room went off at night. After it came back, the shop's reports machine did not return to service: its console shows emergency mode. The reports live on a separate disk that is mounted at /srv/reports.

The terminal is now this machine's console. If you see no prompt, press Enter.

Task

Bring the machine back into service so that it boots by itself:

After a reboot the machine boots into the normal mode, without emergency mode: systemctl is-system-running gives running.
The reports disk is sound, and the check at boot passes. Do not create the disk anew and do not change the /srv/reports line in /etc/fstab: the check at boot and the required mounting of the disk stay.
/srv/reports is mounted from the reports disk, the 2026 reports in /srv/reports/2026 are intact.

In emergency mode you are root, sudo is not needed. "Check" answers in emergency mode too, but it counts a boot that went without it: after the fix, reboot the machine — systemctl reboot.

There is no internet on the machine.

Lab goals

0 of 3

After a reboot the machine boots into the normal mode, without emergency mode (systemctl is-system-running — running) — not checked yet
The reports disk is sound and the same one (not created anew), the check at boot passes — not checked yet
/srv/reports is mounted from the reports disk, the 2026 reports are intact — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Boot · picked for this lab

Boot targets: default, multi-user, rescue and emergency

Cheat sheet

$_ — insert into the terminal

systemctl get-default the default boot target
sudo systemctl set-default multi-user.target
systemctl list-units --type=target active targets
systemctl list-dependencies multi-user.target what it pulls in
systemctl list-dependencies --reverse network-online.target
systemctl is-system-running running, degraded, starting
sudo systemctl rescue switch to rescue now
sudo systemctl default back to the default target

Change the default target

text

student@ubuntu:~$ sudo systemctl set-default graphical.target
Removed "/etc/systemd/system/default.target".
Created symlink /etc/systemd/system/default.target
  → /usr/lib/systemd/system/graphical.target.

The target changes at the next boot. To switch to it now — sudo systemctl isolate <target>.

Who starts a unit at boot

bash

systemctl is-enabled nginx                  # enabled, disabled, static
systemctl show -p WantedBy,RequiredBy nginx  # which targets want it
systemctl list-dependencies --reverse nginx.service

Boot into rescue or emergency

In the GRUB menu select an entry and press e.
At the end of the line that starts with linux, add systemd.unit=rescue.target or systemd.unit=emergency.target.
Ctrl+X boots with this line, once.

On the lab machine the GRUB menu is hidden (GRUB_TIMEOUT=0) and goes to the VM screen (GRUB_TERMINAL=console), while the lesson terminal is the ttyS0 serial console: kernel and systemd messages go there, GRUB's do not. More about the boot loader — boot-grub.

Fix things from emergency and boot on

bash

journalctl -b -p err --no-pager  # errors of this boot
findmnt -no OPTIONS /            # ro? then:
mount -o remount,rw /            # root writable
nano /etc/fstab                  # fix the line
systemctl daemon-reload          # reread fstab
mount -a                         # mount everything from fstab
systemctl default                # continue the boot

journalctl -xb without --no-pager opens the journal in less — quit with q; how to find the lines you need — journalctl. mount -a without errors means fstab is fine, and systemctl default takes the boot to the default target; systemctl reboot checks the same with a reboot.

You are in emergency mode

The boot was stopped by a dependency of local-fs.target: most often an /etc/fstab line with a device that does not exist or a wrong UUID. The console shows before the prompt:

text

[ TIME ] Timed out waiting for device dev-d…-0c1d2e3f4a5b.
[DEPEND] Dependency failed for data.mount - /data.
[DEPEND] Dependency failed for local-fs.target - Local File Systems.

systemd waits 90 seconds for a missing device and only then goes to emergency; a mistake in an option or a file system type fails at once — on the lab machine emergency comes 10–15 seconds after the reboot. The same lines are in journalctl -xb. Fix the line using blkid; mount a disk the system can live without with nofail. More — mount.

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

The fstab line is right, but the disk failed its check at boot. For a line with 1 or 2 in the sixth field systemd runs systemd-fsck@… before mounting — fsck in automatic mode, which fixes only what is safe. An excerpt from journalctl -b:

text

reports: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
fsck failed with exit status 4.
Dependency failed for srv-reports.mount - /srv/reports.
Dependency failed for local-fs.target - Local File Systems.

In emergency that disk is not mounted, so it can be checked right away: fsck <device> without -a asks about every fix, fsck -y agrees to all of them. Then mount <directory> and systemctl reboot. nofail or 0 in the sixth field removes the stop, but the errors stay on the disk. More about the check — filesystems.

The service works, but is not running after a reboot

It was not enabled: systemctl is-enabled <service> prints disabled, and no target wants it. sudo systemctl enable <service> creates the link in multi-user.target.wants. static means the unit has no [Install] section — add one.

The service starts before the network and fails

The unit has no After=network-online.target, or only After=network.target. It needs the pair of lines from boot-targets; Wants= without After= sets no order.

Dependency failed for …

A unit with Requires= (or BindsTo=) did not start because its dependency failed to start. The cause is in the dependency's journal: journalctl -b -u <dependency>.

is-system-running: degraded

The boot finished, but some unit failed. systemctl --failed shows which; after the fix — sudo systemctl reset-failed.

rescue and isolate over SSH

systemctl rescue, systemctl emergency and isolate into a target without the network stop ssh.service: the SSH session drops, and you can come back only from the console. Switch targets when the console is at hand.

1 · Foundations

2 · The system

3 · Admin craft

4 · Server in production

5 · Operations

The machine after a reboot

Situation

Task

Lab goals

Hints

Boot targets: default, multi-user, rescue and emergency

Cheat sheet

Change the default target

Who starts a unit at boot

Boot into rescue or emergency

Fix things from emergency and boot on

You are in emergency mode

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

The service works, but is not running after a reboot

The service starts before the network and fails

Dependency failed for …

is-system-running: degraded

rescue and isolate over SSH

Leave the lab?

The machine after a reboot

Situation

Task

Signing in is needed to start the machine

Change the default target

Who starts a unit at boot

Boot into rescue or emergency

Fix things from emergency and boot on

You are in emergency mode

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

The service works, but is not running after a reboot

The service starts before the network and fails

Dependency failed for …

is-system-running: degraded

rescue and isolate over SSH