Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

dmesg — kernel messages: OOM, disk and filesystem errors, segfaults

9 min util-linux 2.39 · Linux 6.8 · Ubuntu 24.04

dmesg prints the kernel ring buffer: messages about disks, filesystems, OOM, crashed programs and hardware.

Cheat sheet

8 commands Printable version
  • sudo dmesg -T | tail -20 latest messages, with time
  • sudo dmesg -l err,crit errors only
  • sudo dmesg -x | tail with the level of each line
  • sudo dmesg --since '-10min' the last 10 minutes
  • sudo dmesg -W wait for new messages
  • sudo journalctl -k -b -p err kernel errors of this boot
  • sudo journalctl -k -b -1 the kernel of the previous boot
  • sudo dmesg -T | grep -iE 'oom|error|segfault'

Overview

dmesg prints kernel messages: about detected hardware, disks and filesystems, processes the kernel ended, programs that crashed. Programs write to their own logs; the kernel writes here.

The ring buffer

The kernel puts messages into the ring buffer — a fixed-size piece of memory: when it is full, new lines push out the oldest ones. Every line starts with seconds since boot:

text 
student@ubuntu:~$ sudo dmesg | head -2
[    0.000000] Linux version 6.8.0-124-generic (buildd@lcy02-amd64-0…
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.8.0-124-generic …

-T turns the seconds into date and time. dmesg --help warns that after a suspend this time may be wrong; journalctl -k has the exact time.

Access: dmesg_restrict

Ubuntu sets the kernel parameter kernel.dmesg_restrict = 1, and only root can read the buffer:

text 
student@ubuntu:~$ dmesg
dmesg: read kernel buffer failed: Operation not permitted

Without sudo, journalctl -k prints -- No entries -- and a hint for the student: the system journal is readable by the groups adm and systemd-journal, and student is in neither.

Levels

Every message has a severity level. -x shows it, -l filters by it:

Level Example from the training machine
crit EXT4-fs error (device loop0): ext4_lookup:1866: … deleted inode referenced: 12
crit EXT4-fs (loop0): Remounting filesystem read-only
err Aborting journal on device loop0-8.
err Memory cgroup out of memory: Killed process 1635 (tail) …
info python3[1260]: segfault at 0 ip … error 4 in libc.so.6[…]
info EXT4-fs (loop0): mounted filesystem … r/w with ordered data mode.

-x puts the facility and level at the start of each line: kern :crit :, kern :err :. A segfault is info: the filter -l err,crit will not show it. Look for such lines with grep.

dmesg and journalctl -k

journald copies kernel messages into the journal. journalctl -k shows them with real time and keeps them longer than the buffer: on the training machine the journal is on disk (/var/log/journal), so previous boots are visible too.

Need Command
kernel of this boot sudo journalctl -k -b
kernel of the previous boot sudo journalctl -k -b -1
errors and worse only sudo journalctl -k -p err
exact ISO time sudo journalctl -k -o short-iso
message text only sudo journalctl -k -o cat

Options

10

The main options are in the table below; the full list is dmesg --help.

Option What it does Example
-T --ctime Message time as date and clock instead of seconds since boot; may be wrong after a suspend. sudo dmesg -T | tail
-l --level LIST Only messages of the given levels — emerg, alert, crit, err, warn, notice, info, debug; warn+ means this level and above. sudo dmesg -l err,crit,alert,emerg
-x --decode Print the facility and level of every line — kern :crit, kern :info. sudo dmesg -x | tail
-w --follow Print the buffer and wait for new messages, like tail -f. sudo dmesg -w
-W --follow-new Wait and print only new messages, without the old buffer. sudo dmesg -W
--since TIME Messages since a moment — '-5min', '2026-10-21 00:30'. sudo dmesg --since '-10min'
-H --human Human-friendly view — time deltas between lines, colours, a pager. sudo dmesg -H
-J --json JSON output — for scripts and jq. sudo dmesg -J | head
-C --clear Clear the kernel ring buffer; journalctl -k keeps the messages, dmesg does not. dangerous sudo dmesg -C
-c --read-clear Print the buffer and clear it. dangerous sudo dmesg -c

Nothing found. Try a shorter query.

Recipes

6

Kernel errors of this boot

bash 
sudo journalctl -k -b -p err -o short-iso
sudo dmesg -T -l err,crit,alert,emerg

Follow the messages

bash 
sudo dmesg -W                     # new lines only
echo '<3>lab: test error' | sudo tee /dev/kmsg   # a line of your own

-w first prints the whole buffer — hundreds of boot lines — and then waits. -W waits right away. A line written to /dev/kmsg lands in the buffer; <3> at the start means level err.

A crashed program: segfault

The program touched memory that is not its own. The kernel ends it with SIGSEGV (number 11), the exit code is 139 = 128 + 11 (see exit-codes), and an info line stays in the buffer:

text 
python3[1557]: segfault at 0 ip 00007de1a139429d sp 00007ffeed038a48
  error 4 in libc.so.6[7de1a1228000+188000] likely on CPU 0

python3[1557] is the name and PID, segfault at 0 is the address it touched (0 is a null pointer), in libc.so.6 is the library where it happened. The line is wrapped here for width. When programs crash often, the kernel skips some lines and writes show_signal_msg: 61 callbacks suppressed.

An ext4 filesystem error

text 
EXT4-fs error (device loop0): ext4_lookup:1866: inode #2: comm cat:
  deleted inode referenced: 12
Aborting journal on device loop0-8.
EXT4-fs (loop0): Remounting filesystem read-only

ext4 found damage: a directory points to a deleted inode. The program got Structure needs cleaning. What the kernel does next is set by the error behaviour in the filesystem superblock (errors=): with remount-ro the journal is aborted, and writes to the filesystem answer Input/output error. Details and e2fsck are in filesystems.

OOM: a process ended for lack of memory

bash 
sudo dmesg -T | grep -E 'invoked oom-killer|Killed process'

The lines Out of memory: Killed process and Memory cgroup out of memory: Killed process are the work of the oom-killer; that article shows how to read the whole report.

A task hung in D

text 
INFO: task touch:2147 blocked for more than 20 seconds.
      Not tainted 6.8.0-124-generic #124-Ubuntu
…
task:touch           state:D stack:0     pid:2147  tgid:2147  ppid:957

The process stayed in state D longer than kernel.hung_task_timeout_secs (120 seconds on Ubuntu; here the threshold was lowered to 20). A kernel stack follows — it shows what the process waits for (see load-average).

Troubleshooting

5

read kernel buffer failed: Operation not permitted

kernel.dmesg_restrict = 1: only root reads the buffer. Use sudo dmesg or sudo journalctl -k.

journalctl -k prints -- No entries --

Without sudo, a user outside the adm and systemd-journal groups sees only their own entries. Repeat with sudo.

The buffer does not start at boot, or is empty

The buffer overflowed and old lines were pushed out, or someone cleared it (dmesg -C). The full history is sudo journalctl -k -b.

No journal boot entry found

text 
student@ubuntu:~$ sudo journalctl -k -b -1
No journal boot entry found from the specified boot offset (-1).

The journal has no previous boot: the machine booted once, or the journal is kept in memory only (/run/log/journal). The list of boots is journalctl --list-boots.

Time in dmesg -T and the journal differ

-T counts time from boot and ignores suspends. The time in journalctl -k is recorded at the moment of the message.

Danger zone

1

dmesg -C and dmesg -c

Both clear the kernel buffer. After sudo dmesg -C on the training machine dmesg showed 0 lines while journalctl -k -b still had all 577. Do not clear the buffer on a server you are investigating: the next person will have only the journal.

Terms

2
kernel ring buffer
A fixed-size piece of memory where the kernel writes its messages; when it is full, new lines push out old ones. dmesg reads it (root only on Ubuntu), journalctl -k keeps a copy with exact times.
segfault (segmentation fault)
The program touched memory that is not its own; the kernel ends it with SIGSEGV (11), the exit code is 139, and the kernel log gets a line "name[PID]: segfault at address … in library".

See also

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Topics
Loading…