OOM killer: when memory runs out

The OOM killer (out of memory) is the part of the kernel that frees memory when there is nowhere else to get it. First the kernel evicts the page cache and, if there is swap, rarely used program pages. When nothing is left to free, the OOM killer picks one process and ends it with SIGKILL. This signal cannot be caught: the program has no time to write anything to its log.

The machine or a cgroup

Memory runs out in two places:

A group limit is MemoryMax= in a systemd unit or --memory for a container (see cgroups-namespaces). It protects the rest of the machine: a service with MemoryMax=64M hits its own 64 MiB while neighbours keep working. That is why a cgroup OOM happens while free -h shows free memory.

A machine-wide report differs: the oom-kill: line carries constraint=CONSTRAINT_NONE and global_oom, and the result is Out of memory: Killed process 1500 (python3) … oom_score_adj:1000. That is how a python3 that took 400 MiB on a machine without swap ended.

How the victim is chosen

The kernel gives every process a score, oom_score: the more memory it holds (RSS, swap, page tables), the higher the score. The adjustment oom_score_adj, from -1000 to 1000, shifts it: -1000 means never touch, 1000 means kill first. The process with the highest score is killed.

student@ubuntu:~$ choom -p $$
pid 954's current OOM score: 673
pid 954's current OOM score adjust value: 0
student@ubuntu:~$ sudo choom -n 500 -p $$
pid 954's OOM score adjust value changed from 0 to 500
student@ubuntu:~$ choom -p $$
pid 954's current OOM score: 1006
pid 954's current OOM score adjust value: 500

systemd sets a service's adjustment from the unit's OOMScoreAdjust=: systemd-journald has -250. stress-ng gives its worker processes 1000 so that OOM hits them.

What the kernel writes

An OOM report in the kernel log runs to dozens of lines: a call stack, the group's memory, a task list. grep finds the main lines:

sudo journalctl -k -o cat | grep -E \
  'invoked oom|memory: usage|oom-kill:|Killed process'

The service hog with MemoryMax=64M, where tail piles its input up in memory, produced these lines:

Before the last line the kernel prints the group's tasks with their memory in 4 KiB pages (middle columns omitted here):

Tasks state (memory values in pages):
[  pid  ]   uid  tgid total_vm      rss … oom_score_adj name
[   1214]     0  1214      700      448 …             0 sh
[   1215]     0  1215     1531      448 …             0 head
[   1216]     0  1216    17702    16640 …             0 tail

The kernel log is readable by root and the groups adm and systemd-journal. Without sudo, journalctl -k prints -- No entries -- and dmesg prints Operation not permitted.

What systemd sees

systemd learns about an OOM in a service's group and stops the whole service (OOMPolicy=stop by default): its other processes get SIGTERM. The service result is oom-kill:

student@ubuntu:~$ sudo journalctl -u hog -o cat
Started hog.service - /usr/bin/sh -c "head -c 300M /dev/zero | tail".
hog.service: A process of this unit has been killed by the OOM killer.
hog.service: Failed with result 'oom-kill'.
student@ubuntu:~$ systemctl show hog -p Result -p NRestarts
Result=oom-kill
NRestarts=2

With Restart=on-failure the service comes back, hits the same limit, and NRestarts grows. The group's counters are in memory.events. The service vmhog is stress-ng under MemoryMax=96M with OOMPolicy=continue: it survives OOM, and stress-ng starts its worker again.

cat /sys/fs/cgroup/system.slice/vmhog.service/memory.events

low 0
high 0
max 15070
oom 102
oom_kill 102
oom_group_kill 0

max is how many times the group hit MemoryMax, oom_kill is how many of its processes the OOM killer ended. The file exists while the service runs.

Exit code 137 and Killed

A process killed with SIGKILL exits with code 137 = 128 + 9 (see exit-codes). A shell whose child died prints Killed. A Docker container after OOM shows Exited (137) and OOMKilled: true (see docker).

• memory-swap — free, available, page cache and swap.
• cgroups-namespaces — groups and their limits.
• systemd — units and drop-ins.
• journalctl — the kernel log -k and a service log -u.
• exit-codes — code 137 and other 128+N codes.
• docker — a container's memory limit and OOMKilled.
• kill — the signals SIGTERM and SIGKILL.