Packages, services, and logs / Stage 12 of 13

Log rotation and retention

Not running Starting usually ~20 s

Step 8 of 9 · lab · 25 min

Bound training journal retention

A separate systemd journal namespace has grown too large. Save the records you need, set a persistent limit and clean old archives of this namespace only.

Situation

The training service lab-retention-seed writes to a separate systemd journal namespace, lab-retention. The namespace has its own instance, systemd-journald@lab-retention.service, and its own configuration, /etc/systemd/journald@lab-retention.conf. Its archives already take more than 24 MiB. Among the old records are three needed for an incident review: their text starts with lab-evidence:. The main system journal is outside this task.

Task

The three lab-evidence: records of the lab-retention-seed service are saved to /home/student/journal-evidence.txt in short-iso format (with date and time): exactly those three lines, in a file you own.
/etc/systemd/journald@lab-retention.conf sets a persistent limit SystemMaxUse=16M; the other settings stay as they were, and there are no extra settings files.
The namespace's archived files take at most 16 MiB (the current active file doesn't count).
The namespace's journal instance runs with the new setting and accepts new records.
The main journal and its settings, the lab-retention-seed service and its program are untouched.

Constraints

Don't delete or truncate journal files by hand — use journalctl only.
Don't reconfigure or clean the main systemd-journald.
Don't disable logging or redirect it to /dev/null.

The tools you need are already installed. There's no internet on this machine.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 5

The three timestamped lab-evidence records are saved in journal-evidence.txt — not done yet
Only SystemMaxUse changed in the namespace configuration — to 16M — not done yet
Archives of the lab-retention namespace take at most 16 MiB — not done yet
The journal instance was restarted with the new setting and accepts new records — not done yet
The main journal, its settings and the task files are untouched — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 5

The three timestamped lab-evidence records are saved in journal-evidence.txt — not done yet
Only SystemMaxUse changed in the namespace configuration — to 16M — not done yet
Archives of the lab-retention namespace take at most 16 MiB — not done yet
The journal instance was restarted with the new setting and accepts new records — not done yet
The main journal, its settings and the task files are untouched — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 5

The three timestamped lab-evidence records are saved in journal-evidence.txt — not done yet
Only SystemMaxUse changed in the namespace configuration — to 16M — not done yet
Archives of the lab-retention namespace take at most 16 MiB — not done yet
The journal instance was restarted with the new setting and accepts new records — not done yet
The main journal, its settings and the task files are untouched — not done yet

Logs and audit · picked for this lab

journalctl — reading the systemd journal

Cheat sheet

$_ — insert into the terminal

journalctl -u nginx -f tail a service's logs
journalctl -xeu nginx investigate a failure
journalctl -p err -b errors from the current boot
journalctl -k kernel messages
journalctl _COMM=sshd -g fail by process, plus a text match
journalctl -b -1 the previous boot
journalctl --vacuum-time=7d clean up

Flag	Meaning
`-u <unit>`	only this service
`-f` / `-e`	follow / jump to the end
`-x`	add explanations to messages
`-p <prio>`	this priority or higher: emerg..debug (or err, warning)
`-b [N]`	the current (or Nth) boot
`-k`	kernel messages
`-g RE`	grep the text
`-o <fmt>`	output format: short, json, json-pretty, cat, verbose
`--since/--until`	a time range

Filtering by structured fields (without naming a service):

bash

journalctl _PID=1234              # by PID
journalctl _COMM=sshd             # by process name
journalctl _UID=1000 --since today
journalctl _SYSTEMD_UNIT=ssh.service -p warning
journalctl -o json-pretty -u nginx -n 1   # every field of one entry
journalctl --user -u myapp        # a user service

Commands

bash

journalctl -u nginx               # entries from the nginx service
journalctl -u nginx -f            # follow in real time (like tail -f)
journalctl -u nginx --since "10 min ago"
journalctl -xeu nginx             # tail plus explanations (x) when debugging
journalctl -p err -b              # only errors from the current boot
journalctl -k                     # kernel messages (like dmesg)
journalctl --since today --until "1 hour ago"
journalctl -b -1                  # the journal of the PREVIOUS boot (before the last reboot)
journalctl -g 'pattern'           # grep messages (regex)
journalctl --disk-usage           # how much space it takes
sudo journalctl --vacuum-time=7d  # drop entries older than 7 days

Your own lines in the journal: logger and -t

text

student@ubuntu:~$ logger -t cleanup "removed 3 files"
student@ubuntu:~$ journalctl -t cleanup --no-pager -n 2
Oct 20 10:50:48 ubuntu cleanup[1143]: removed 3 files

logger -t tag writes a line to the journal under that tag, and journalctl -t tag shows only those lines. Output of a script run by a service is tagged with the script name: journalctl -t clean.sh. Outside the adm and systemd-journal groups, student sees only its own entries — read other users' and system entries with sudo.

The journal is empty after a reboot

journald isn't persistent. Create /var/log/journal (see "Overview"). journalctl -b -1 will then show the previous boot.

The journal is filling up the disk

journalctl --disk-usage, then --vacuum-time=7d or --vacuum-size=500M. A permanent cap goes in /etc/systemd/journald.conf (SystemMaxUse=500M).

You need logs for a specific failure

journalctl -xeu <unit> --since "5 min ago" is the most common diagnostic query (see systemctl).

The time in the journal looks "wrong"

journald stores entries in UTC internally and shows the local timezone. Check timedatectl; use journalctl --utc for UTC.

A service's logs show up in /var/log too (duplicates)

journald and rsyslog can both log the same thing. See rsyslog for text files vs journalctl for structured, per--u queries.

Log rotation and retention

Situation

Task

Constraints

Signing in is needed to start the machine

Commands

Your own lines in the journal: logger and -t

The journal is empty after a reboot

The journal is filling up the disk

You need logs for a specific failure

The time in the journal looks "wrong"

A service's logs show up in /var/log too (duplicates)