Inside the system: resources, kernel, processes / Stage 4 of 12

ulimit and sysctl: system ceilings

Not running Starting usually ~20 s

Step 8 of 10 · lab · 25 min

A service fails under load

The notification service keeps one connection per subscriber. There are now three hundred subscribers, and it works one moment and fails the next, even though a colleague has already changed something. Find the cause and fix it so that the service holds a thousand subscribers, including after a restart.

Situation

The lab-notify service — the notify-hub application — sends notifications for the intranet portal. Every open portal page subscribes to /events and keeps a connection to the service. While there were a few dozen subscribers, the service worked. Today there are about three hundred, and the service works one moment and fails the next: curl http://127.0.0.1:8090/health answers Connection refused every other time. Yesterday a colleague already changed the service settings — it did not help.

On this machine the subscribers are played by the lab-subscribers service: it keeps 300 subscriptions open and reconnects the dropped ones.

Task

lab-notify holds 1000 subscribers at once: it answers curl http://127.0.0.1:8090/health and does not restart. The application code stays the same.
Change the service settings in a drop-in. Do not edit the packaged unit /usr/lib/systemd/system/lab-notify.service or replace it with a copy: the next package update would overwrite it.
systemd-analyze verify lab-notify.service prints nothing.
The service holds the same load after a restart: sudo systemctl restart lab-notify.

There is no internet on the machine; everything you need is already installed. Run administrative commands with sudo.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 4

lab-notify holds 1000 subscribers, answers /health and does not restart; the application code is unchanged — not done yet
The setting lives in a drop-in of the service, the packaged unit is unchanged, and the process runs with it — not done yet
systemd-analyze verify lab-notify.service prints nothing — not done yet
The service holds the same load after a restart — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 4

lab-notify holds 1000 subscribers, answers /health and does not restart; the application code is unchanged — not done yet
The setting lives in a drop-in of the service, the packaged unit is unchanged, and the process runs with it — not done yet
systemd-analyze verify lab-notify.service prints nothing — not done yet
The service holds the same load after a restart — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 4

lab-notify holds 1000 subscribers, answers /health and does not restart; the application code is unchanged — not done yet
The setting lives in a drop-in of the service, the packaged unit is unchanged, and the process runs with it — not done yet
systemd-analyze verify lab-notify.service prints nothing — not done yet
The service holds the same load after a restart — not done yet

Processes · picked for this lab

ulimit and process limits: open files, processes, core

Cheat sheet

$_ — insert into the terminal

ulimit -a all limits of the shell
ulimit -Sn; ulimit -Hn soft and hard open files limit
ulimit -Sn 4096 raise only the soft one, up to hard
cat /proc/1234/limits limits of a running process
ls /proc/1234/fd | wc -l how many files it holds now
sudo prlimit --pid 1234 --nofile=4096:8192 change on the fly
systemctl show app -p LimitNOFILE -p LimitNOFILESoft

How many files are open and how many are allowed

bash

pid=$(systemctl show -p MainPID --value app)
sudo ls /proc/$pid/fd | wc -l
grep 'open files' /proc/$pid/limits

A descriptor count close to the soft limit means the service will soon get Too many open files. A count that grows and never drops means the service does not close its files (a leak); lsof -p <PID> shows what is open.

Raise a limit in your shell

bash

ulimit -Hn                # the ceiling
ulimit -Sn 65536          # up to the ceiling without root

This affects the shell and everything started from it.

Too many open files

The process hit its soft nofile limit (error code EMFILE, number 24):

text

student@ubuntu:~$ (ulimit -n 16; python3 -c \
  'f=[open("/etc/hostname") for i in range(20)]')
OSError: [Errno 24] Too many open files: '/etc/hostname'

Check Max open files in /proc/<PID>/limits and the number of files in /proc/<PID>/fd. For a service, raise LimitNOFILE with a drop-in and restart it. A descriptor count that grows without load means a leak in the application.

cannot modify limit: Operation not permitted

The soft limit is being raised above the hard one, or the hard one above its current value without root. root gets the same above fs.nr_open. Check ulimit -Hn; in your shell a hard limit lowered by ulimit -n cannot be restored — open a new shell.

Raised it in limits.conf, the service did not notice

limits.conf affects user logins through PAM. A service needs LimitNOFILE= in its unit or a drop-in (see ulimit).

systemctl show prints the new limit, the service fails as before

The service was not restarted after daemon-reload: a process gets its limits at start. Run sudo systemctl restart <service> and check /proc/<PID>/limits.

fork: retry: Resource temporarily unavailable

The user hit the nproc process limit (ulimit -u). Often it is a program that spawns processes in a loop. Find it (ps -u <user> | wc -l) and stop it; for a service use LimitNPROC= or TasksMax=.

ulimit -n in your working shell

ulimit -n 4096 lowers the hard limit too. Only root can raise it back — a regular user can only open a new shell. To experiment, run ulimit in a subshell: (ulimit -n 16; command).

ulimit and sysctl: system ceilings

Situation

Task

Signing in is needed to start the machine

How many files are open and how many are allowed

Raise a limit in your shell

Too many open files

cannot modify limit: Operation not permitted

Raised it in limits.conf, the service did not notice

systemctl show prints the new limit, the service fails as before

fork: retry: Resource temporarily unavailable

ulimit -n in your working shell