Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

/proc: processes and the kernel as files

11 min Ubuntu 24.04 · Linux 6.8

/proc is a filesystem where the kernel shows processes (/proc/) and its own state: memory, load, parameters.

Cheat sheet

8 commands Printable version
  • ls /proc/1234/ everything about process 1234
  • grep -E 'State|PPid|VmRSS' /proc/1234/status
  • ls -l /proc/1234/fd open files and sockets
  • cat /proc/1234/limits the process's limits
  • tr '\0' ' ' < /proc/1234/cmdline command line
  • sudo cat /proc/1234/environ | tr '\0' '\n' environment
  • ls -l /proc/1234/cwd /proc/1234/exe directory and program
  • cat /proc/1234/cgroup the process's service or session

Overview

/proc is a filesystem that does not exist on disk. The kernel builds the content of its files at the moment they are read. Every process has a /proc/<PID> directory, the other files describe the system as a whole, and /proc/sys holds kernel parameters. ps, top, free, uptime and pidstat take their data from here.

text 
student@ubuntu:~$ ls -l /proc/loadavg /proc/meminfo
-r--r--r-- 1 root root 0 Oct 21 00:17 /proc/loadavg
-r--r--r-- 1 root root 0 Oct 21 00:17 /proc/meminfo

The files have size 0: there is no data until someone reads them. Read them whole — with cat, grep, tr.

A process directory

File Contents Who can read
status name, state, PPid, UID, VmRSS memory, Threads everyone
cmdline command line, arguments separated by a zero byte everyone
limits resource limits: soft and hard everyone
cgroup the process's group: a service or a session everyone
oom_score, oom_score_adj OOM score and adjustment everyone
task/ the process's threads everyone
environ environment at start time owner and root
fd/ open descriptors: files, sockets, pipes owner and root
cwd, exe current directory and program file owner and root
io bytes read and written owner and root
maps memory areas and the files in them owner and root
stack the process's kernel stack root

Another user's process answers Permission denied without sudo, and wchan answers with zero. /proc/self is the directory of whatever process reads it: cat /proc/self/status shows cat.

status

text 
student@ubuntu:~$ sleep 300 &
[1] 971
student@ubuntu:~$ grep -E '^(Name|State|PPid|VmRSS)' /proc/971/status
Name:   sleep
State:  S (sleeping)
PPid:   956
VmRSS:      2028 kB
Field Meaning
State state: R, S, D, Z, T (see load-average)
PPid parent PID
Uid four UIDs: real, effective, saved, filesystem
VmRSS the process's memory in RAM (see memory-swap)
Threads number of threads
voluntary_ctxt_switches how many times the process gave up the CPU itself

fd: open files

text 
student@ubuntu:~$ ls -l /proc/$$/fd
total 0
lr-x------ 1 student student 64 Oct 21 00:17 0 -> /dev/null
…

Each link is an open descriptor: a number and what it points to. A socket looks like socket:[2076], a pipe like pipe:[…]. The number of descriptors is ls /proc/<PID>/fd | wc -l; compare it with Max open files in limits (see ulimit).

When a file was deleted but a process keeps it open, the link ends with (deleted), and the disk space is not freed (see deleted open file):

text 
student@ubuntu:~$ readlink /proc/956/fd/3
/tmp/app.log (deleted)

A systemd service has /dev/null as standard input, and its output goes to the journal socket: systemd-resolved has 0 -> /dev/null, 1 -> socket:[3608].

exe and maps: the program and its libraries

exe is a link to the program file; maps lists the process's memory areas and the files mapped into them: the program itself and every shared library. On the lab machine libc.so.6 is mapped into 38 processes at once.

An update replaces the file on disk with a new one, while the process keeps running the old one. The exe link and the maps lines then end in (deleted):

text 
student@ubuntu:~$ sudo readlink /proc/4214/exe
/usr/local/bin/demo-sleep (deleted)
student@ubuntu:~$ sudo grep deleted /proc/3830/maps | head -2
7ac7bbf1f000-7ac7bbf21000 r--p 00000000 fd:04 2727
  /usr/local/lib/libdemo.so.1 (deleted)
7ac7bbf21000-7ac7bbf33000 r-xp 00002000 fd:04 2727
  /usr/local/lib/libdemo.so.1 (deleted)

The maps columns are the area's addresses, permissions (r-xp is code), the offset in the file, the device, the inode and the path. A fix in the new version does not apply to such a process until it is restarted. To find all such processes:

bash 
sudo grep -l deleted /proc/[0-9]*/maps 2>/dev/null

Look at the path: /memfd:… (deleted) lines are memory with no file on disk and have nothing to do with updates. For services, needrestart prints the same with a ready restart command (see unattended-upgrades); lsof marks such files DEL (see lsof).

Locks: /proc/locks

text 
student@ubuntu:~$ cat /proc/locks | head -2
1: FLOCK  ADVISORY  WRITE 1094 fd:04:2703 0 EOF
1: -> FLOCK  ADVISORY  WRITE 1112 fd:04:2703 0 EOF

Each line is a file lock: the type (FLOCK from flock, POSIX from fcntl), WRITE or READ, the owner PID, the device and inode of the file (fd:04:2703 — inode 2703, as in the NODE column of lsof). A line with -> means process 1112 waits for this lock. Process 1094 holds it.

cmdline and environ

Arguments in cmdline and variables in environ are separated by a zero byte; tr turns them into spaces or lines:

text 
student@ubuntu:~$ tr '\0' ' ' < /proc/971/cmdline
sleep 300
student@ubuntu:~$ sudo cat /proc/1/environ | tr '\0' '\n' | head -2
HOME=/
init=/sbin/init

environ is the environment at start time: an export in a running shell does not get there; only processes started later see it. Kernel threads (kthreadd and others) have an empty cmdline.

limits and cgroup

/proc/<PID>/limits holds the limits the process runs with right now. They are changed by ulimit, a service's LimitNOFILE= and prlimit; a unit change without restarting the service does not reach this file (see ulimit).

text 
student@ubuntu:~$ cat /proc/1211/cgroup
0::/system.slice/ssh.service
student@ubuntu:~$ cat /proc/$$/cgroup
0::/user.slice/user-1000.slice/session-c1.scope

cgroup tells whose process it is: the ssh.service service or a user session. The short way is ps -o pid,unit,comm -p <PID>.

System files

File Contents
/proc/loadavg load average and task counts (see load-average)
/proc/meminfo memory: MemTotal, MemAvailable, Cached (see memory-swap)
/proc/pressure/ share of time waiting for CPU, memory, disk
/proc/cpuinfo CPUs and their model
/proc/uptime seconds since boot and total CPU idle time
/proc/cmdline parameters the kernel booted with
/proc/mounts mounted filesystems
/proc/swaps active swap
/proc/sys/ kernel parameters (see sysctl)

/proc/sys

Every kernel parameter is a file: vm.swappiness is /proc/sys/vm/swappiness, dots in the name stand for slashes. sysctl reads and writes the same files. Write with tee:

bash 
cat /proc/sys/vm/swappiness
echo 10 | sudo tee /proc/sys/vm/swappiness

Recipes

3

What is this process

bash 
grep -E 'Name|State|PPid|Uid' /proc/1211/status
tr '\0' ' ' < /proc/1211/cmdline; echo
sudo ls -l /proc/1211/cwd /proc/1211/exe
cat /proc/1211/cgroup

Name, parent, user, full command, working directory, program file and service — enough to tell where the process came from.

Find and recover a deleted but open file

bash 
sudo find /proc/[0-9]*/fd -lname '*(deleted)' 2>/dev/null
cp /proc/956/fd/3 /tmp/restored.log   # a copy of the content

find prints links like /proc/956/fd/3: the PID and the descriptor number. While the process lives, the file can be copied through this link. The space is freed when the process closes the file or exits.

A service's environment

bash 
pid=$(systemctl show -p MainPID --value ssh)
sudo cat /proc/$pid/environ | tr '\0' '\n'

Troubleshooting

4

Permission denied on another user's process

text 
student@ubuntu:~$ ls /proc/276/fd
ls: cannot open directory '/proc/276/fd': Permission denied

fd, environ, cwd, exe, io, maps of another user's process are readable by root only. Repeat with sudo.

sudo is there, yet Permission denied

text 
student@ubuntu:~$ sudo tr '\0' '\n' < /proc/1/environ
-bash: /proc/1/environ: Permission denied
student@ubuntu:~$ sudo echo 1 > /proc/sys/vm/drop_caches
-bash: /proc/sys/vm/drop_caches: Permission denied

The < and > redirections are done by your own shell without root rights; sudo applies to the command only. Read with sudo cat … | tr …, write with echo … | sudo tee ….

No such file or directory

The process has ended, and its directory went with it. A new process may have taken the PID — check cmdline.

environ lacks a variable that was set

The variable was set after the process started. environ will not show it — restart the process with the right environment.

Terms

1
shared library
A .so file with code that many programs use at once (libc.so.6, libssl.so.3): the kernel maps it into the memory of every process that uses it. A new version of the library reaches a process only after the process restarts. Also appears in: lsof, unattended-upgrades: security updates with no one at the keyboard

See also

  • processes — PIDs, states and signals.
  • ps — the process list built from /proc.
  • lsof — open files of all processes.
  • strace — a process's system calls.
  • ulimit — limits from /proc/<PID>/limits.
  • sysctl — kernel parameters in /proc/sys.
  • load-average — /proc/loadavg and state D.
  • memory-swap — /proc/meminfo and RSS.

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Topics
Loading…