Inside the system: resources, kernel, processes / Stage 4 of 12 · step 8 of 10

A service fails under load

~25 min

Not running Starting usually ~20 s

Situation

The lab-notify service — the notify-hub application — sends notifications for the intranet portal. Every open portal page subscribes to /events and keeps a connection to the service. While there were a few dozen subscribers, the service worked. Today there are about three hundred, and the service works one moment and fails the next: curl http://127.0.0.1:8090/health answers Connection refused every other time. Yesterday a colleague already changed the service settings — it did not help.

On this machine the subscribers are played by the lab-subscribers service: it keeps 300 subscriptions open and reconnects the dropped ones.

Task

lab-notify holds 1000 subscribers at once: it answers curl http://127.0.0.1:8090/health and does not restart. The application code stays the same.
Change the service settings in a drop-in. Do not edit the packaged unit /usr/lib/systemd/system/lab-notify.service or replace it with a copy: the next package update would overwrite it.
systemd-analyze verify lab-notify.service prints nothing.
The service holds the same load after a restart: sudo systemctl restart lab-notify.

There is no internet on the machine; everything you need is already installed. Run administrative commands with sudo.

Lab goals

0 of 4

lab-notify holds 1000 subscribers, answers /health and does not restart; the application code is unchanged — not checked yet
The setting lives in a drop-in of the service, the packaged unit is unchanged, and the process runs with it — not checked yet
systemd-analyze verify lab-notify.service prints nothing — not checked yet
The service holds the same load after a restart — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Processes · picked for this lab

ulimit and process limits: open files, processes, core

Cheat sheet

$_ — insert into the terminal

ulimit -a all limits of the shell
ulimit -Sn; ulimit -Hn soft and hard open files limit
ulimit -Sn 4096 raise only the soft one, up to hard
cat /proc/1234/limits limits of a running process
ls /proc/1234/fd | wc -l how many files it holds now
sudo prlimit --pid 1234 --nofile=4096:8192 change on the fly
systemctl show app -p LimitNOFILE -p LimitNOFILESoft

How many files are open and how many are allowed

bash

pid=$(systemctl show -p MainPID --value app)
sudo ls /proc/$pid/fd | wc -l
grep 'open files' /proc/$pid/limits

A descriptor count close to the soft limit means the service will soon get Too many open files. A count that grows and never drops means the service does not close its files (a leak); lsof -p <PID> shows what is open.

Raise a limit in your shell

bash

ulimit -Hn                # the ceiling
ulimit -Sn 65536          # up to the ceiling without root

This affects the shell and everything started from it.

Too many open files

The process hit its soft nofile limit (error code EMFILE, number 24):

text

student@ubuntu:~$ (ulimit -n 16; python3 -c \
  'f=[open("/etc/hostname") for i in range(20)]')
OSError: [Errno 24] Too many open files: '/etc/hostname'

Check Max open files in /proc/<PID>/limits and the number of files in /proc/<PID>/fd. For a service, raise LimitNOFILE with a drop-in and restart it. A descriptor count that grows without load means a leak in the application.

cannot modify limit: Operation not permitted

The soft limit is being raised above the hard one, or the hard one above its current value without root. root gets the same above fs.nr_open. Check ulimit -Hn; in your shell a hard limit lowered by ulimit -n cannot be restored — open a new shell.

Raised it in limits.conf, the service did not notice

limits.conf affects user logins through PAM. A service needs LimitNOFILE= in its unit or a drop-in (see ulimit).

systemctl show prints the new limit, the service fails as before

The service was not restarted after daemon-reload: a process gets its limits at start. Run sudo systemctl restart <service> and check /proc/<PID>/limits.

fork: retry: Resource temporarily unavailable

The user hit the nproc process limit (ulimit -u). Often it is a program that spawns processes in a loop. Find it (ps -u <user> | wc -l) and stop it; for a service use LimitNPROC= or TasksMax=.

ulimit -n in your working shell

ulimit -n 4096 lowers the hard limit too. Only root can raise it back — a regular user can only open a new shell. To experiment, run ulimit in a subshell: (ulimit -n 16; command).