Linux Labs

Inside the system: resources, kernel, processes / Stage 8 of 12

dmesg and the kernel log

Not running Starting usually ~20 s
Leave the lesson

Step 6 of 8 · theory · 5 min

The kernel log in journald

The buffer lives until the machine is switched off. For kernel lines to survive a reboot, journald collects them.

journald reads the same buffer and keeps kernel lines in the journal along with everything else. sudo journalctl -k shows only them for the current boot, and -b -1 for the previous one, if the journal keeps it (the lesson machine has a single boot, and journalctl prints No journal boot entry found). The -p err option selects lines of level err and more important. student is not in the adm and systemd-journal groups, so it reads the system journal through sudo.

The full report of the OOM killer is there too — in the memory lesson you found it by a single line. First comes who asked for memory when none was left, and at the end — whom it killed:

python3 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, …
Memory cgroup stats for /system.slice/lab-hog.service:
…
Tasks state (memory values in pages):
[  pid  ]   uid  tgid total_vm      rss … oom_score_adj name
[   1088]     0  1088    25061     9760 …             0 python3
oom-kill:constraint=CONSTRAINT_MEMCG,…,task=python3,pid=1088,uid=0
Memory cgroup out of memory: Killed process 1088 (python3)
  total-vm:100244kB, anon-rss:32512kB, … oom_score_adj:0

Tasks state lists the processes of the group with their memory and the oom_score_adj adjustment: it shields a process from the OOM killer or, the other way round, offers it up. CONSTRAINT_MEMCG and the words Memory cgroup mean that the group's limit ran out; when the whole machine runs short, the result starts with Out of memory:.

Check yourself 1 question

The start and the end of an OOM report in sudo journalctl -k. What happened?

nginx invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
…
Memory cgroup out of memory: Killed process 2210 (java) total-vm:912344kB, anon-rss:401220kB, file-rss:6528kB, shmem-rss:0kB, UID:1001 pgtables:1120kB oom_score_adj:0
  • The kernel does the killing. invoked oom-killer names the process whose memory request ran into the shortage, and the kernel picks the victim itself.

  • There is one Killed process line, and it names java. nginx is named only as the process that asked for memory.

The first line names the process whose request ran out of memory, the last one names whom the kernel killed. The kernel picks the victim by the memory it holds, adjusted by oom_score_adj (java with 400 MB of anon-rss here), not by who asked last.

Key points

  • sudo journalctl -k — kernel lines from the journal: they survive a reboot, and -b -1 is the previous boot.
  • -p err in journalctl and -l in dmesg select lines by importance.
  • An OOM report starts with invoked oom-killer and ends with Killed process; in between are the group's processes and their memory.
Ubuntu 24.04
Loading…