Server security / Stage 7 of 10 · step 5 of 6

The service can do anything

~25 min

Not running Starting usually ~20 s

Situation

The lab-tickets service — the tickets-api application — takes helpdesk tickets. Its unit was written in a hurry: the service runs as root, and nothing stops it from doing anything root can do. The security review is next week, and systemd-analyze security lab-tickets.service rates the service 9.6 UNSAFE right now.

The tickets live in /var/lib/tickets, the API key in /etc/tickets/api.key. The package created the tickets account for the service, but nobody ever put it into the unit. How to talk to the service is described in /usr/share/doc/tickets-api/README:

curl -s http://127.0.0.1:8097/health
key=$(sudo cat /etc/tickets/api.key)
curl -s -H "X-Api-Key: $key" http://127.0.0.1:8097/tickets

Task

lab-tickets is running and enabled: the earlier tickets are in place, and a new ticket (POST /tickets) is saved in /var/lib/tickets. The application code stays the same.
The service process does not run as root, has no privileges and runs with NoNewPrivileges=yes. The API key can be read by root and by the service itself — nobody else.
ProtectSystem=strict: the service cannot write to /etc, /usr, /var/lib, /var/log or /opt.
systemd-analyze security lab-tickets.service — 4.9 or lower, rated OK.

There is no internet on the machine; everything you need is already installed. Run administrative commands with sudo.

Lab goals

0 of 4

lab-tickets is running and enabled, the earlier tickets are in place, a new one is saved in /var/lib/tickets; the application code is unchanged — not checked yet
The service process is not root, has no privileges and runs with NoNewPrivileges; an outside user cannot read the API key — not checked yet
ProtectSystem=strict is in effect — the service cannot write to /etc, /usr, /var/lib, /var/log or /opt — not checked yet
systemd-analyze security lab-tickets.service — 4.9 or lower (rated OK) — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Services and systemd · picked for this lab

systemd

Cheat sheet

$_ — insert into the terminal

systemctl status nginx state
systemctl start nginx start it now
systemctl stop nginx stop it now
systemctl enable nginx enable autostart
systemctl reload nginx reread the config without downtime
systemctl daemon-reload after editing unit files
journalctl -u nginx -f the service's logs in real time
systemd-analyze blame boot timings

Units are searched for in (in order of increasing priority):

/lib/systemd/system/ — packaged units (don't edit these);
/etc/systemd/system/ — local overrides (edit here).

.service structure:

ini

[Unit]
Description=My App
After=network-online.target

[Service]
ExecStart=/usr/bin/myapp
Restart=on-failure
User=myapp

[Install]
WantedBy=multi-user.target

Drop-in: extending a unit

A unit that comes from a package is changed with a drop-in: a package update overwrites the unit file itself, while the drop-in stays. A drop-in is a .conf file in /etc/systemd/system/<unit>.d/. systemd reads the unit, then its drop-ins in alphabetical order, and their lines add to or replace the unit's lines.

bash

sudo systemctl edit nginx       # opens an editor, saves override.conf
sudo mkdir -p /etc/systemd/system/demo.service.d       # or by hand:
printf '[Service]\nMemoryMax=256M\n' \
  | sudo tee /etc/systemd/system/demo.service.d/memory.conf
sudo systemctl daemon-reload    # reread units after a manual edit
systemctl cat demo              # the unit and all its drop-ins
systemctl show demo -p DropInPaths

systemctl edit runs daemon-reload itself. After a manual edit without daemon-reload, systemd keeps working with the old version of the unit. A key such as ExecStart= in a drop-in adds a second command; to replace the first one, put an empty ExecStart= line before the new one.

systemd skips a key in the wrong section: LimitNOFILE= under [Unit] has no effect, and the service gets the default limit. The warning shows in the journal after daemon-reload and in systemd-analyze verify — whose exit code is still 0:

bash

systemd-analyze verify /etc/systemd/system/demo.service

text

/etc/systemd/system/demo.service:3: Unknown key name 'LimitNOFILE' in
  section 'Unit', ignoring.

Socket activation

A .socket unit holds the port instead of the service: systemd listens itself and starts the service of the same name on the first connection. The lab pair is demo-sock.socket and demo-sock.service:

ini

# /etc/systemd/system/demo-sock.socket
[Socket]
ListenStream=8085

[Install]
WantedBy=sockets.target

# /etc/systemd/system/demo-sock.service
[Unit]
Requires=demo-sock.socket
After=demo-sock.socket

[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/demo-sock.py

ListenStream=8085 is a port with no address: the socket listens on all addresses, IPv4 and IPv6. Until the first request the port is held by the systemd process, and the service is inactive:

text

student@ubuntu:~$ sudo ss -tlnp | grep 8085
LISTEN 0  4096  *:8085  *:*  users:(("systemd",pid=1,fd=71))
student@ubuntu:~$ systemctl list-sockets | grep -E 'LISTEN|demo'
LISTEN     UNIT              ACTIVATES
[::]:8085  demo-sock.socket  demo-sock.service
student@ubuntu:~$ curl -s http://127.0.0.1:8085/
hello
student@ubuntu:~$ sudo ss -tlnp | grep 8085
LISTEN 0  4096  *:8085  *:*
  users:(("python3",pid=2769,fd=3),("systemd",pid=1,fd=71))

The address is changed with a drop-in. ListenStream= is a list: an empty value resets the unit's addresses; without it the new address is added to the old one.

bash

sudo mkdir -p /etc/systemd/system/demo-sock.socket.d
printf '[Socket]\nListenStream=\nListenStream=127.0.0.1:8085\n' \
  | sudo tee /etc/systemd/system/demo-sock.socket.d/local.conf
sudo systemctl daemon-reload
sudo systemctl restart demo-sock.socket
sudo ss -tlnp | grep 8085           # 127.0.0.1:8085

restart of the socket restarts the service with Requires= on it too: the service's MainPID changed (2769 → 2838). Stopping only the service is not enough — the next connection starts it again, and systemctl stop warns:

text

Stopping 'demo-sock.service', but its triggering units are still active:
demo-sock.socket

To close the port for good: sudo systemctl disable --now demo-sock.socket.

Targets

A target is a group of units systemd brings the boot up to: multi-user.target on a server, rescue.target and emergency.target for repairs. In detail — boot-targets.

bash

systemctl get-default              # the current default target
sudo systemctl set-default multi-user.target
systemctl isolate rescue.target    # switch right now

The service doesn't start

systemctl status <unit> — exit code and the last journal lines.
journalctl -xeu <unit> — details with explanations.
After manually editing a unit, always sudo systemctl daemon-reload.

"Unit not found"

Check the name and path: systemctl cat <unit>. If you edited the file — daemon-reload.

Slow boot

systemd-analyze critical-chain — who held the boot; the order of the investigation is in systemd-analyze.

The service can do anything

Situation

Task

Signing in is needed to start the machine

Targets

The service doesn't start

"Unit not found"

Slow boot