Server security / Stage 2 of 10 · step 6 of 7

After the update

~25 min

Not running Starting usually ~20 s

Situation

The reports portal on this server is three services, each from its own package: lab-gateway (the API), lab-reports (the reports themselves) and lab-metrics (request statistics). A colleague installed security updates and went home. dpkg -l shows the new versions, yet the vulnerability scanner flagged the server again: the vulnerable code is still running.

Task

No portal service runs old code: there are no deleted files in /proc/<PID>/maps of their processes.
All three services are active and answer: curl http://127.0.0.1:8071/health, and the same for ports 8072 and 8073.
Restart only the services that have old code in memory: every restart drops the clients' connections and resets whatever the service has accumulated in memory. Do not reboot the machine.
The update stays in place: do not roll the package versions back, and do not edit the package files or the service units.

There is no internet on the machine; everything you need is already installed. Run administrative commands with sudo.

Lab goals

0 of 4

No portal service runs old code — there are no deleted files in /proc/PID/maps — not checked yet
lab-gateway, lab-reports and lab-metrics are active and answer /health — not checked yet
The service the update did not touch was not restarted, and the machine was not rebooted — not checked yet
The update stays in place — the package versions are new, the package files and service units are unchanged — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Processes · picked for this lab

lsof — open files and ports of processes

Cheat sheet

$_ — insert into the terminal

sudo lsof -i :5432 -nP who's on this port
sudo lsof +D /dir who holds a directory
sudo lsof +L1 deleted but open files
sudo lsof -p 1234 | wc -l how many fds a process has (a leak?)
sudo kill $(sudo lsof -t -i:8080) kill whoever holds a port

Option	Meaning
`-i [:port]`	network sockets (optionally by port/host)
`-nP`	skip DNS and port-name resolution (fast, numeric)
`-p PID`	files of a specific process
`-c name`	processes by command name
`-u user`	by user
`+D /dir`	recursively, everything under a directory (what's holding the fs)
`+L1`	files with fewer than 1 link (deleted but still open)
`-t`	just PIDs (for scripts: `kill $(lsof -t ...)`)

Common tasks

bash

sudo lsof -i :5432                # process on a port (like ss)
sudo lsof -i -nP                  # all network sockets (no resolution)
sudo lsof -iTCP -sTCP:LISTEN -nP  # only listening TCP sockets
sudo lsof +D /mnt/data            # who holds files in a dir
sudo lsof +L1                     # deleted but still open files
sudo lsof -p 1234                 # everything process 1234 opened
sudo lsof -c nginx                # everything nginx processes opened
sudo lsof -u alice -a -i          # user alice AND network (-a = AND)
sudo kill $(sudo lsof -t -i:8080) # kill whoever is on port 8080

The FD column and locks

text

student@ubuntu:~$ lsof /tmp/lk
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
flock   1094 student    3rW  REG  253,4        0 2703 /tmp/lk
sleep   1096 student    3r   REG  253,4        0 2703 /tmp/lk
flock   1112 student    3r   REG  253,4        0 2703 /tmp/lk

FD is the descriptor number, then the mode: r — read, w — write, u — both. A third letter is a file lock: W — a write lock on the whole file, R — a read lock on the whole file, w and r — on part of it. Instead of a number there can be cwd (current directory), txt (program file), mem (a memory-mapped file).

Here flock 1094 holds the lock (3rW), sleep 1096 is its child with the same open file, and flock 1112 has opened the file and waits: it has no lock letter. Waiters also show in /proc/locks (see proc-filesystem).

Deleted files: +L1 and DEL

A file that was deleted or replaced with a new one lives on while a process holds it — an open unlinked file. lsof +L1 shows open files that have no name left on disk. This is a service whose binary was replaced by an update:

text

student@ubuntu:~$ sudo lsof +L1
COMMAND    PID USER  FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
demo-slee 4214 root txt    REG  253,4    35336     0 4464
  /usr/local/bin/demo-sleep (deleted)

NLINK 0 means the file has no names; txt means it is the program file.

A deleted shared library in a process's memory does not show in +L1: such a file is not open but mapped into memory. lsof for the process marks it DEL in the FD column:

text

student@ubuntu:~$ sudo lsof -p 3830 | grep -E 'COMMAND|DEL'
COMMAND  PID USER   FD   TYPE  DEVICE SIZE/OFF  NODE NAME
sleep   3830 root  DEL    REG   253,4           2727
  /usr/local/lib/libdemo.so.1

The same files marked (deleted) are in /proc/<PID>/maps (see proc-filesystem). A replaced file is freed only when the process restarts.

df shows "full", du doesn't

A file that's deleted but still held open by a process doesn't free its space. Find it with sudo lsof +L1 → restart the process holding it (see disk-usage).

umount: target is busy

Something is holding the filesystem: sudo lsof +D /mnt/data (or fuser -vm /mnt/data). Close the processes (or cd out of the directory) and unmount (see mount).

A port is in use, but by what?

sudo lsof -i :PORT -nP or ss -tlnp.

A process is "leaking" file descriptors

sudo lsof -p PID | wc -l shows how many are open; compare against the ulimit -n limit. Steady growth means a leak in the application.

lsof is slow

It's resolving DNS/ports/mounts — add -nP; with a stuck NFS mount it can hang on the mount itself.

After the update

Situation

Task

Signing in is needed to start the machine

Common tasks

The FD column and locks

Deleted files: +L1 and DEL

df shows "full", du doesn't

umount: target is busy

A port is in use, but by what?

A process is "leaking" file descriptors

lsof is slow