Packages, services, and logs / Stage 6 of 13 · step 6 of 7

Service doesn't come up after reboot

~10 min

Not running Starting usually ~20 s

Situation

The server was rebooted overnight as planned. In the morning the demo-web service was not running until the on-call engineer started it by hand. It is running now, but the next reboot may leave the server without it again.

Task

The demo-web service is running now.
At the next system boot demo-web starts by itself.

You do not need to reboot the machine to check: you can ask systemd about both states of the service.

The tools you need are already installed. There is no internet on this machine.

Lab goals

0 of 2

The demo-web service is active — not checked yet
The demo-web service is enabled at boot — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Services and systemd · picked for this lab

systemctl — managing systemd services

Cheat sheet

$_ — insert into the terminal

systemctl status nginx --no-pager state, no pager
sudo systemctl status nginx + the last journal lines
systemctl start nginx start it now
systemctl stop nginx stop it now
systemctl restart nginx restart it (drops connections)
systemctl enable --now nginx enable autostart and start it right away
systemctl cat nginx --no-pager the whole unit file
systemctl edit nginx open a drop-in override
sudo systemctl daemon-reload reread units after an edit
systemctl --failed list of failed units
sudo journalctl -xeu nginx why the unit failed

enable and start are independent operations: enable only registers the unit for autostart, it doesn't touch the current session's service, so after enable without --now the service stays inactive until reboot or an explicit start. The reverse is also true: start without enable won't survive a reboot.

Control

bash

sudo systemctl status nginx --no-pager   # state + journal tail
systemctl start nginx             # start it now
systemctl stop nginx              # stop it
systemctl restart nginx           # restart it (drops connections)
systemctl reload nginx            # reread the config without downtime (if the service supports it)
systemctl reload-or-restart nginx # reload, or restart if it can't
systemctl enable --now nginx      # enable autostart AND start it
systemctl disable --now nginx     # disable autostart AND stop it
systemctl mask nginx              # forbid starting it at all (even manually/as a dependency)
systemctl is-active --quiet nginx # a silent check for scripts (exit code)
systemctl is-enabled nginx

reload vs restart: reload asks the service to reread its config (no downtime), restart is a full restart.

Viewing and units

bash

systemctl list-units --type=service --state=running   # what's running
systemctl list-unit-files --state=enabled             # what's set to autostart
systemctl --failed                                    # what failed
systemctl cat nginx                                   # show the whole unit file
systemctl show nginx -p MainPID,ActiveState,ExecStart # specific properties
systemctl list-dependencies nginx                     # the dependency tree
systemctl edit nginx                                  # an override snippet (drop-in, safe)
systemctl edit --full nginx                           # edit the whole unit
sudo systemctl daemon-reload                          # after ANY unit edit

edit creates a drop-in at /etc/systemd/system/<unit>.d/override.conf — the packaged unit is untouched, updates won't overwrite it.

`is-active` lies after starting the binary manually

systemctl only sees what it started itself. Launch it through systemctl start, not by hand.

Changes to the unit aren't applied

After editing a .service file, you need sudo systemctl daemon-reload, then restart. If you edited a packaged unit by hand, use systemctl edit (a drop-in) instead.

The service is stuck "activating"

Check Type= in the unit and journalctl -xeu <unit>: often the wrong Type=notify without a readiness signal, or a hanging ExecStart.

The service "doesn't start and doesn't say why"

sudo systemctl status -l --no-pager <unit> plus sudo journalctl -u <unit> -b (for the current boot). systemctl reset-failed <unit> resets the restart counter (after fixing it).

Warning: some journal files were not opened due to insufficient permissions.

systemctl status showed the service state but no journal lines. Service logs are readable by root and the adm and systemd-journal groups; a regular user has none of these. Run it with rights: sudo systemctl status nginx --no-pager or sudo journalctl -u nginx -n 20. For good — group membership: sudo usermod -aG systemd-journal name (takes effect after a new login, see usermod).

The output opened in a pager, no prompt

Long output of status, cat, list-units doesn't fit the terminal window, so systemctl shows it through less: q quits. To print it straight to the terminal, add --no-pager.

enable doesn't work — "no installed section"

The unit has no [Install] section. Either it's a static unit, or enable it through a dependency/target.

stop and restart drop current connections to the service immediately; in production, for services with a graceful shutdown, check first whether reload is available. mask forbids the unit from starting at all, even as a dependency — the service won't come up together with others until you remove the mask (systemctl unmask), and this is easy to forget. disable --now on a system service that others depend on brings those down too: check systemctl list-dependencies --reverse <unit> beforehand.