The kernel counts the open files of each process separately and gives each one its own ceiling.
A process sees an open file as a number — a file descriptor. Numbers 0, 1 and 2 are taken in every process by input, output and error output. That is why the shipper has 64 descriptors: 61 logs plus the three standard streams. The /proc/<PID>/fd directory lists these numbers, and you can read it only for your own processes; for anyone else's you need sudo.
Every process has a set of limits: how many files it keeps open, how many processes it starts, how large a file it writes. The kernel will not open a file beyond the limit: it returns the EMFILE error, and the program prints its text — Too many open files.
The limits of a live process are in /proc/<PID>/limits, and any user can read that file:
A process gets its limits from its parent at the moment it starts. Your shell got its own when you logged in, and everything you start from it begins with them. The shipper was started by systemd, so your 1024 has nothing to do with it.
You lowered the open-file limit in your shell to 100 (ulimit -n 100) and then started python3 from it. How many files will python3 be able to keep open?
-
64 belongs to the shipper process, which systemd started. python3 takes its limits from the shell it was started from.
-
A process has no default of its own: at start it copies its parent's limits, and the parent here is a shell with a limit of 100.
A process gets its parent's limits at start, so python3 has the same limit of 100. You can check it in a subshell without touching your own: (ulimit -n 100; grep 'open files' /proc/self/limits) — grep shows 100 in the Max open files line.
Key points
- To a process, an open file is a descriptor; 0, 1 and 2 are taken by the standard streams.
- The kernel opens nothing beyond the limit: the program gets EMFILE and writes Too many open files.
-
A process copies its parent's limits at start; the live values are in
/proc/<PID>/limits.