Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

ulimit and process limits: open files, processes, core

8 min Ubuntu 24.04 · bash 5.2 · systemd 255

Process resource limits — open files, processes, core: ulimit, /proc//limits and a service LimitNOFILE.

Cheat sheet

7 commands Printable version
  • ulimit -a all limits of the shell
  • ulimit -Sn; ulimit -Hn soft and hard open files limit
  • ulimit -Sn 4096 raise only the soft one, up to hard
  • cat /proc/1234/limits limits of a running process
  • ls /proc/1234/fd | wc -l how many files it holds now
  • sudo prlimit --pid 1234 --nofile=4096:8192 change on the fly
  • systemctl show app -p LimitNOFILE -p LimitNOFILESoft

Overview

The kernel limits every process: how many files it keeps open, how many processes its user starts, how big a core file it writes on a crash. Limits are inherited: a child gets its parent's limits. The open files limit is the one people hit most. A connection socket is an open file too: a server with a thousand clients holds a thousand descriptors.

Soft and hard

Every limit has two values. The soft one is in effect now. The hard one is the ceiling up to which a process raises its soft limit by itself. Both can be lowered; only root raises the hard one, and not above fs.nr_open (1048576 on the training machine).

text 
student@ubuntu:~$ ulimit -Sn; ulimit -Hn
1024
1048576
student@ubuntu:~$ ulimit -n 4096; ulimit -Sn; ulimit -Hn
4096
4096
student@ubuntu:~$ ulimit -Hn 8192
-bash: ulimit: open files: cannot modify limit: Operation not permitted

ulimit -n without -S or -H in bash changes both values at once. The hard limit dropped to 4096, and student can no longer raise it in this shell. The soft one alone is ulimit -Sn 2048.

Kinds of limits

ulimit limits.conf systemd What On the machine
-n nofile LimitNOFILE open files and sockets 1024
-u nproc LimitNPROC processes of the user 1679
-c core LimitCORE core file size 0
-s stack LimitSTACK stack, KiB 8192
-l memlock LimitMEMLOCK locked memory, KiB 65536
-f fsize LimitFSIZE file size unlimited

The "On the machine" column holds student's soft values from ulimit -a.

Process limits: /proc/<PID>/limits

ulimit shows the limits of your shell. The limits of another process are in /proc/<PID>/limits, which any user can read. For the ssh service:

Limit Soft Limit Hard Limit Units
Max open files 1024 524288 files

prlimit changes the limits of a running process without a restart:

text 
student@ubuntu:~$ s=$(systemctl show -p MainPID --value ssh)
student@ubuntu:~$ sudo prlimit --pid $s --nofile
RESOURCE DESCRIPTION              SOFT   HARD UNITS
NOFILE   max number of open files 1024 524288 files
student@ubuntu:~$ sudo prlimit --pid $s --nofile=4096:8192

This lasts until the process restarts. A permanent value for a service goes into its unit.

A service limit: LimitNOFILE

systemd services get their limits from systemd: ulimit in a terminal and /etc/security/limits.conf do not affect them. The default is soft 1024, hard 524288:

bash 
systemctl show -p DefaultLimitNOFILE -p DefaultLimitNOFILESoft
text 
DefaultLimitNOFILE=524288
DefaultLimitNOFILESoft=1024

Your own limit goes into a drop-in:

bash 
sudo mkdir -p /etc/systemd/system/ssh.service.d
printf '[Service]\nLimitNOFILE=65536\n' \
  | sudo tee /etc/systemd/system/ssh.service.d/limits.conf
sudo systemctl daemon-reload
sudo systemctl restart ssh
grep 'open files' /proc/$(systemctl show -p MainPID --value ssh)/limits

LimitNOFILE=65536 sets both values; different ones are LimitNOFILE=4096:65536 (soft:hard). After daemon-reload, systemctl show already prints 65536, while the running process keeps the old limits until a restart:

Moment systemctl show /proc/<PID>/limits
after daemon-reload 65536 old values
after restart 65536 65536 and 65536

limits.conf: limits at login

/etc/security/limits.conf and the files /etc/security/limits.d/*.conf are read by a PAM module at login — ssh, login, su -l. That is how user sessions get their limits. On the training machine the file holds only comments.

text 
deploy  soft  nofile  65535
deploy  hard  nofile  65535

Recipes

2

How many files are open and how many are allowed

bash 
pid=$(systemctl show -p MainPID --value app)
sudo ls /proc/$pid/fd | wc -l
grep 'open files' /proc/$pid/limits

A descriptor count close to the soft limit means the service will soon get Too many open files. A count that grows and never drops means the service does not close its files (a leak); lsof -p <PID> shows what is open.

Raise a limit in your shell

bash 
ulimit -Hn                # the ceiling
ulimit -Sn 65536          # up to the ceiling without root

This affects the shell and everything started from it.

Troubleshooting

5

Too many open files

The process hit its soft nofile limit (error code EMFILE, number 24):

text 
student@ubuntu:~$ (ulimit -n 16; python3 -c \
  'f=[open("/etc/hostname") for i in range(20)]')
OSError: [Errno 24] Too many open files: '/etc/hostname'

Check Max open files in /proc/<PID>/limits and the number of files in /proc/<PID>/fd. For a service, raise LimitNOFILE with a drop-in and restart it. A descriptor count that grows without load means a leak in the application.

cannot modify limit: Operation not permitted

The soft limit is being raised above the hard one, or the hard one above its current value without root. root gets the same above fs.nr_open. Check ulimit -Hn; in your shell a hard limit lowered by ulimit -n cannot be restored — open a new shell.

Raised it in limits.conf, the service did not notice

limits.conf affects user logins through PAM. A service needs LimitNOFILE= in its unit or a drop-in (see ulimit).

systemctl show prints the new limit, the service fails as before

The service was not restarted after daemon-reload: a process gets its limits at start. Run sudo systemctl restart <service> and check /proc/<PID>/limits.

fork: retry: Resource temporarily unavailable

The user hit the nproc process limit (ulimit -u). Often it is a program that spawns processes in a loop. Find it (ps -u <user> | wc -l) and stop it; for a service use LimitNPROC= or TasksMax=.

Danger zone

1

ulimit -n in your working shell

ulimit -n 4096 lowers the hard limit too. Only root can raise it back — a regular user can only open a new shell. To experiment, run ulimit in a subshell: (ulimit -n 16; command).

See also

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Topics
Loading…