Packages, services, and logs / Stage 8 of 13

Your own service unit and applying changes

Not running Starting usually ~20 s

Step 6 of 7 · lab · 30 min

Write a systemd unit for a service

A daemon is started by hand in a terminal. Turn it into a systemd service that survives a reboot and a crash.

Situation

A developer left a daemon on the server, /usr/local/bin/myapp: it runs in the foreground and listens on port 9000. So far it is started by hand in a terminal, so once the administrator logs out or the server reboots, the daemon is not running. A user myapp that cannot log in has already been created for the daemon.

Task

Turn the daemon into a systemd service myapp with the file /etc/systemd/system/myapp.service:

The service is running, and its process listens on port 9000.
The service's process runs as the user myapp.
The service starts at system boot and starts again by itself after a crash.

The daemon itself does not need changing.

The tools you need are already installed. There is no internet on this machine.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 6

The unit file /etc/systemd/system/myapp.service exists — not done yet
The myapp service is active — not done yet
Port 9000 is served by the myapp service's process — not done yet
The service's process runs as the user myapp — not done yet
The myapp service is enabled at boot — not done yet
myapp starts again after a crash — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 6

The unit file /etc/systemd/system/myapp.service exists — not done yet
The myapp service is active — not done yet
Port 9000 is served by the myapp service's process — not done yet
The service's process runs as the user myapp — not done yet
The myapp service is enabled at boot — not done yet
myapp starts again after a crash — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 6

The unit file /etc/systemd/system/myapp.service exists — not done yet
The myapp service is active — not done yet
Port 9000 is served by the myapp service's process — not done yet
The service's process runs as the user myapp — not done yet
The myapp service is enabled at boot — not done yet
myapp starts again after a crash — not done yet

Services and systemd · picked for this lab

systemd

Cheat sheet

$_ — insert into the terminal

systemctl status nginx state
systemctl start nginx start it now
systemctl stop nginx stop it now
systemctl enable nginx enable autostart
systemctl reload nginx reread the config without downtime
systemctl daemon-reload after editing unit files
journalctl -u nginx -f the service's logs in real time
systemd-analyze blame boot timings

Units are searched for in (in order of increasing priority):

/lib/systemd/system/ — packaged units (don't edit these);
/etc/systemd/system/ — local overrides (edit here).

.service structure:

ini

[Unit]
Description=My App
After=network-online.target

[Service]
ExecStart=/usr/bin/myapp
Restart=on-failure
User=myapp

[Install]
WantedBy=multi-user.target

Drop-in: extending a unit

A unit that comes from a package is changed with a drop-in: a package update overwrites the unit file itself, while the drop-in stays. A drop-in is a .conf file in /etc/systemd/system/<unit>.d/. systemd reads the unit, then its drop-ins in alphabetical order, and their lines add to or replace the unit's lines.

bash

sudo systemctl edit nginx       # opens an editor, saves override.conf
sudo mkdir -p /etc/systemd/system/demo.service.d       # or by hand:
printf '[Service]\nMemoryMax=256M\n' \
  | sudo tee /etc/systemd/system/demo.service.d/memory.conf
sudo systemctl daemon-reload    # reread units after a manual edit
systemctl cat demo              # the unit and all its drop-ins
systemctl show demo -p DropInPaths

systemctl edit runs daemon-reload itself. After a manual edit without daemon-reload, systemd keeps working with the old version of the unit. A key such as ExecStart= in a drop-in adds a second command; to replace the first one, put an empty ExecStart= line before the new one.

systemd skips a key in the wrong section: LimitNOFILE= under [Unit] has no effect, and the service gets the default limit. The warning shows in the journal after daemon-reload and in systemd-analyze verify — whose exit code is still 0:

bash

systemd-analyze verify /etc/systemd/system/demo.service

text

/etc/systemd/system/demo.service:3: Unknown key name 'LimitNOFILE' in
  section 'Unit', ignoring.

Socket activation

A .socket unit holds the port instead of the service: systemd listens itself and starts the service of the same name on the first connection. The lab pair is demo-sock.socket and demo-sock.service:

ini

# /etc/systemd/system/demo-sock.socket
[Socket]
ListenStream=8085

[Install]
WantedBy=sockets.target

# /etc/systemd/system/demo-sock.service
[Unit]
Requires=demo-sock.socket
After=demo-sock.socket

[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/demo-sock.py

ListenStream=8085 is a port with no address: the socket listens on all addresses, IPv4 and IPv6. Until the first request the port is held by the systemd process, and the service is inactive:

text

student@ubuntu:~$ sudo ss -tlnp | grep 8085
LISTEN 0  4096  *:8085  *:*  users:(("systemd",pid=1,fd=71))
student@ubuntu:~$ systemctl list-sockets | grep -E 'LISTEN|demo'
LISTEN     UNIT              ACTIVATES
[::]:8085  demo-sock.socket  demo-sock.service
student@ubuntu:~$ curl -s http://127.0.0.1:8085/
hello
student@ubuntu:~$ sudo ss -tlnp | grep 8085
LISTEN 0  4096  *:8085  *:*
  users:(("python3",pid=2769,fd=3),("systemd",pid=1,fd=71))

The address is changed with a drop-in. ListenStream= is a list: an empty value resets the unit's addresses; without it the new address is added to the old one.

bash

sudo mkdir -p /etc/systemd/system/demo-sock.socket.d
printf '[Socket]\nListenStream=\nListenStream=127.0.0.1:8085\n' \
  | sudo tee /etc/systemd/system/demo-sock.socket.d/local.conf
sudo systemctl daemon-reload
sudo systemctl restart demo-sock.socket
sudo ss -tlnp | grep 8085           # 127.0.0.1:8085

restart of the socket restarts the service with Requires= on it too: the service's MainPID changed (2769 → 2838). Stopping only the service is not enough — the next connection starts it again, and systemctl stop warns:

text

Stopping 'demo-sock.service', but its triggering units are still active:
demo-sock.socket

To close the port for good: sudo systemctl disable --now demo-sock.socket.

Targets

A target is a group of units systemd brings the boot up to: multi-user.target on a server, rescue.target and emergency.target for repairs. In detail — boot-targets.

bash

systemctl get-default              # the current default target
sudo systemctl set-default multi-user.target
systemctl isolate rescue.target    # switch right now

The service doesn't start

systemctl status <unit> — exit code and the last journal lines.
journalctl -xeu <unit> — details with explanations.
After manually editing a unit, always sudo systemctl daemon-reload.

"Unit not found"

Check the name and path: systemctl cat <unit>. If you edited the file — daemon-reload.

Slow boot

systemd-analyze critical-chain — who held the boot; the order of the investigation is in systemd-analyze.

Your own service unit and applying changes

Situation

Task

Signing in is needed to start the machine

Targets

The service doesn't start

"Unit not found"

Slow boot