Inside the system: resources, kernel, processes / Stage 10 of 12 · step 7 of 8

A service holds up the boot

~25 min

Not running Starting usually ~20 s

Situation

The warehouse server runs two services: the order desk lab-orders (http://127.0.0.1:8090) and the stock store lab-stock. At startup the order desk loads the stock levels from the store — without them it accepts no orders.

After every reboot the server takes half a minute longer to boot, and lab-orders ends up failed — the on-call admin brings the services up by hand later. A colleague insists the service dependencies are set up; the units are in /etc/systemd/system/.

The machine is now as it is after such a boot: lab-orders has just tried to start.

Task

lab-orders starts the lab-stock store itself and starts after it: the dependencies live in its unit (Wants= and After=), with no reverse ordering.
systemd-analyze verify lab-orders.service lab-stock.service prints no warnings.
lab-orders is running, enabled at boot and answering: curl http://127.0.0.1:8090/health.
After a reboot (sudo reboot) lab-orders comes up on its own at the first attempt and holds up the boot for no more than 10 seconds.

Constraints

Do not change the application code — /usr/local/bin/order-desk and /usr/local/bin/stock-store.
You may reboot the machine as often as you like: the terminal shows the boot and logs in as student again. "Check" looks at a boot that happened after the machine was prepared, so reboot it after your fix.

There is no internet on the machine; everything you need is installed. How the service starts is visible in systemctl status lab-orders and journalctl -u lab-orders; what holds up the boot — in systemd-analyze blame and systemd-analyze critical-chain.

Lab goals

0 of 4

lab-orders starts the lab-stock store itself and starts after it (Wants= and After=), with no reverse ordering — not checked yet
systemd-analyze verify lab-orders.service lab-stock.service prints no warnings — not checked yet
lab-orders is running, enabled at boot and answers on http://127.0.0.1:8090/health; the application code is unchanged — not checked yet
After a reboot (sudo reboot) lab-orders came up on its own at the first attempt and held up the boot for no more than 10 s — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Services and systemd · picked for this lab

systemd

Cheat sheet

$_ — insert into the terminal

systemctl status nginx state
systemctl start nginx start it now
systemctl stop nginx stop it now
systemctl enable nginx enable autostart
systemctl reload nginx reread the config without downtime
systemctl daemon-reload after editing unit files
journalctl -u nginx -f the service's logs in real time
systemd-analyze blame boot timings

Units are searched for in (in order of increasing priority):

/lib/systemd/system/ — packaged units (don't edit these);
/etc/systemd/system/ — local overrides (edit here).

.service structure:

ini

[Unit]
Description=My App
After=network-online.target

[Service]
ExecStart=/usr/bin/myapp
Restart=on-failure
User=myapp

[Install]
WantedBy=multi-user.target

Drop-in: extending a unit

A unit that comes from a package is changed with a drop-in: a package update overwrites the unit file itself, while the drop-in stays. A drop-in is a .conf file in /etc/systemd/system/<unit>.d/. systemd reads the unit, then its drop-ins in alphabetical order, and their lines add to or replace the unit's lines.

bash

sudo systemctl edit nginx       # opens an editor, saves override.conf
sudo mkdir -p /etc/systemd/system/demo.service.d       # or by hand:
printf '[Service]\nMemoryMax=256M\n' \
  | sudo tee /etc/systemd/system/demo.service.d/memory.conf
sudo systemctl daemon-reload    # reread units after a manual edit
systemctl cat demo              # the unit and all its drop-ins
systemctl show demo -p DropInPaths

systemctl edit runs daemon-reload itself. After a manual edit without daemon-reload, systemd keeps working with the old version of the unit. A key such as ExecStart= in a drop-in adds a second command; to replace the first one, put an empty ExecStart= line before the new one.

systemd skips a key in the wrong section: LimitNOFILE= under [Unit] has no effect, and the service gets the default limit. The warning shows in the journal after daemon-reload and in systemd-analyze verify — whose exit code is still 0:

bash

systemd-analyze verify /etc/systemd/system/demo.service

text

/etc/systemd/system/demo.service:3: Unknown key name 'LimitNOFILE' in
  section 'Unit', ignoring.

Socket activation

A .socket unit holds the port instead of the service: systemd listens itself and starts the service of the same name on the first connection. The lab pair is demo-sock.socket and demo-sock.service:

ini

# /etc/systemd/system/demo-sock.socket
[Socket]
ListenStream=8085

[Install]
WantedBy=sockets.target

# /etc/systemd/system/demo-sock.service
[Unit]
Requires=demo-sock.socket
After=demo-sock.socket

[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/demo-sock.py

ListenStream=8085 is a port with no address: the socket listens on all addresses, IPv4 and IPv6. Until the first request the port is held by the systemd process, and the service is inactive:

text

student@ubuntu:~$ sudo ss -tlnp | grep 8085
LISTEN 0  4096  *:8085  *:*  users:(("systemd",pid=1,fd=71))
student@ubuntu:~$ systemctl list-sockets | grep -E 'LISTEN|demo'
LISTEN     UNIT              ACTIVATES
[::]:8085  demo-sock.socket  demo-sock.service
student@ubuntu:~$ curl -s http://127.0.0.1:8085/
hello
student@ubuntu:~$ sudo ss -tlnp | grep 8085
LISTEN 0  4096  *:8085  *:*
  users:(("python3",pid=2769,fd=3),("systemd",pid=1,fd=71))

The address is changed with a drop-in. ListenStream= is a list: an empty value resets the unit's addresses; without it the new address is added to the old one.

bash

sudo mkdir -p /etc/systemd/system/demo-sock.socket.d
printf '[Socket]\nListenStream=\nListenStream=127.0.0.1:8085\n' \
  | sudo tee /etc/systemd/system/demo-sock.socket.d/local.conf
sudo systemctl daemon-reload
sudo systemctl restart demo-sock.socket
sudo ss -tlnp | grep 8085           # 127.0.0.1:8085

restart of the socket restarts the service with Requires= on it too: the service's MainPID changed (2769 → 2838). Stopping only the service is not enough — the next connection starts it again, and systemctl stop warns:

text

Stopping 'demo-sock.service', but its triggering units are still active:
demo-sock.socket

To close the port for good: sudo systemctl disable --now demo-sock.socket.

Targets

A target is a group of units systemd brings the boot up to: multi-user.target on a server, rescue.target and emergency.target for repairs. In detail — boot-targets.

bash

systemctl get-default              # the current default target
sudo systemctl set-default multi-user.target
systemctl isolate rescue.target    # switch right now

The service doesn't start

systemctl status <unit> — exit code and the last journal lines.
journalctl -xeu <unit> — details with explanations.
After manually editing a unit, always sudo systemctl daemon-reload.

"Unit not found"

Check the name and path: systemctl cat <unit>. If you edited the file — daemon-reload.

Slow boot

systemd-analyze critical-chain — who held the boot; the order of the investigation is in systemd-analyze.

A service holds up the boot

Situation

Task

Constraints

Signing in is needed to start the machine

Targets

The service doesn't start

"Unit not found"

Slow boot