systemd

systemd is the first process of the system (PID 1) and the service manager of most modern distributions. It starts and controls units, resolves dependencies between them, and parallelizes boot.

Types of units:

Management is through systemctl, journals through journalctl.

Dependencies and order

The [Unit] lines answer two questions: what to start together and in which order.

After= and Before= set only the order: if nobody starts x, After=x will not start it. That is why a dependency is written as a pair — Wants=x and After=x. Wants= without After= starts both units at once, and the service starts before what it needs is ready. Requires= without After= does not wait either: the service is already running when the dependency fails. A missing unit in Wants= is skipped silently; one in Requires= stops the service from starting. To check a unit after an edit, see systemd-analyze; the boot waiting chain is in systemd-analyze.

Units are searched for in (in order of increasing priority):

• /lib/systemd/system/ — packaged units (don't edit these);
• /etc/systemd/system/ — local overrides (edit here).

.service structure:

[Unit]
Description=My App
After=network-online.target

[Service]
ExecStart=/usr/bin/myapp
Restart=on-failure
User=myapp

[Install]
WantedBy=multi-user.target

Drop-in: extending a unit

A unit that comes from a package is changed with a drop-in: a package update overwrites the unit file itself, while the drop-in stays. A drop-in is a .conf file in /etc/systemd/system/<unit>.d/. systemd reads the unit, then its drop-ins in alphabetical order, and their lines add to or replace the unit's lines.

sudo systemctl edit nginx       # opens an editor, saves override.conf
sudo mkdir -p /etc/systemd/system/demo.service.d       # or by hand:
printf '[Service]\nMemoryMax=256M\n' \
  | sudo tee /etc/systemd/system/demo.service.d/memory.conf
sudo systemctl daemon-reload    # reread units after a manual edit
systemctl cat demo              # the unit and all its drop-ins
systemctl show demo -p DropInPaths

systemctl edit runs daemon-reload itself. After a manual edit without daemon-reload, systemd keeps working with the old version of the unit. A key such as ExecStart= in a drop-in adds a second command; to replace the first one, put an empty ExecStart= line before the new one.

systemd skips a key in the wrong section: LimitNOFILE= under [Unit] has no effect, and the service gets the default limit. The warning shows in the journal after daemon-reload and in systemd-analyze verify — whose exit code is still 0:

systemd-analyze verify /etc/systemd/system/demo.service

/etc/systemd/system/demo.service:3: Unknown key name 'LimitNOFILE' in
  section 'Unit', ignoring.

Socket activation

A .socket unit holds the port instead of the service: systemd listens itself and starts the service of the same name on the first connection. The lab pair is demo-sock.socket and demo-sock.service:

# /etc/systemd/system/demo-sock.socket
[Socket]
ListenStream=8085

[Install]
WantedBy=sockets.target

# /etc/systemd/system/demo-sock.service
[Unit]
Requires=demo-sock.socket
After=demo-sock.socket

[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/demo-sock.py

ListenStream=8085 is a port with no address: the socket listens on all addresses, IPv4 and IPv6. Until the first request the port is held by the systemd process, and the service is inactive:

student@ubuntu:~$ sudo ss -tlnp | grep 8085
LISTEN 0  4096  *:8085  *:*  users:(("systemd",pid=1,fd=71))
student@ubuntu:~$ systemctl list-sockets | grep -E 'LISTEN|demo'
LISTEN     UNIT              ACTIVATES
[::]:8085  demo-sock.socket  demo-sock.service
student@ubuntu:~$ curl -s http://127.0.0.1:8085/
hello
student@ubuntu:~$ sudo ss -tlnp | grep 8085
LISTEN 0  4096  *:8085  *:*
  users:(("python3",pid=2769,fd=3),("systemd",pid=1,fd=71))

The address is changed with a drop-in. ListenStream= is a list: an empty value resets the unit's addresses; without it the new address is added to the old one.

sudo mkdir -p /etc/systemd/system/demo-sock.socket.d
printf '[Socket]\nListenStream=\nListenStream=127.0.0.1:8085\n' \
  | sudo tee /etc/systemd/system/demo-sock.socket.d/local.conf
sudo systemctl daemon-reload
sudo systemctl restart demo-sock.socket
sudo ss -tlnp | grep 8085           # 127.0.0.1:8085

restart of the socket restarts the service with Requires= on it too: the service's MainPID changed (2769 → 2838). Stopping only the service is not enough — the next connection starts it again, and systemctl stop warns:

Stopping 'demo-sock.service', but its triggering units are still active:
demo-sock.socket

To close the port for good: sudo systemctl disable --now demo-sock.socket.

• systemctl — managing systemd services.
• systemd-analyze — boot time, the dependency chain, checking a unit.
• boot-targets — boot targets, rescue and emergency.
• journalctl — Reading the systemd journal (journald) — filters by service, time, priority.
• nginx — Web server and reverse proxy. The config is checked with nginx -t; reload has no downtime.
• Services under systemd — a Theory lesson on this topic