Back to the preview service: the machine has about 300 MiB available, yet the service process was killed for lack of memory. See how much memory the service itself is given.
Find the ceiling and the kernel line
Look at the memory ceiling of the lab-thumbs service and its restart count, then at the kernel log lines about running out of memory.
First, predict
The lab-thumbs unit has MemoryMax=64M. What will systemctl show print for MemoryMax?
Now start the exercise, run the command and compare your prediction with the output.
Every systemd service has its own cgroup — a group of processes for which the kernel counts and limits resources. The MemoryMax=64M line in the lab-thumbs unit is the memory ceiling of that group: all processes of the service and their file cache count towards it.
A preview of the panorama is a 48 MiB file plus its copy in memory. When the group hits the ceiling, the kernel first drops its cache, then calls the OOM killer inside the group. It picks the biggest process and ends it with SIGKILL. This signal cannot be caught, so thumbs had no time to write anything to its log.
The kernel records every such kill in its own log. Memory cgroup out of memory at the start of the line means the group ran out of memory. When the whole machine runs out, the line starts with Out of memory:. anon-rss is the process's own memory: about 64 MiB, right at the ceiling.
systemd sees that the process was killed for memory and records Result=oom-kill for the service. Restart=on-failure tells it to start the service again, NRestarts grows, the service takes on the panorama again — and hits the ceiling again.
journalctl -k has the line: Memory cgroup out of memory: Killed process 2210 (java) … anon-rss:524288kB. What does it say?
-
When the whole machine runs out, the line starts with
Out of memory:.Memory cgroupmeans a shortage inside a group. -
Killed processmeans the kernel ended the process with SIGKILL. java had no time to print an error of its own.
Memory cgroup out of memory means a process group ran out of memory, and Killed process means the kernel ended the process with SIGKILL. 524288 kB is 512 MiB: that much the process held when it was killed.