Who changed the file: auditd

Not running Starting usually ~20 s

Step 5 of 6 · lab · 20 min

The config changes by itself

The portal settings keep reverting to someone else's, even though they were just fixed. Put an audit rule on the file, find from the events who rewrites it and with what, and stop it without touching anyone else.

Situation

The company portal reads its settings from /etc/portal/portal.conf. This morning it turned out to have debug = true, open sign-up and the address of someone else's database. You put the working settings back — a minute later the file was the same as before:

cat /etc/portal/portal.conf
ls -l /etc/portal/portal.conf

The portal group can write the config — the developers and the deployment account. Your crontab -l is empty, and /etc/cron.d holds only the settings backup.

Task

Put an audit rule on writes to /etc/portal/portal.conf. The rule is in effect now and lives in /etc/audit/rules.d — it survives a reboot.
Use the audit events to find who rewrites the config. Write to ~/culprit.txt, on one line, the user name and the full path of the program that writes to the file.
Take that job off the schedule.
Leave everything else as it was: cron runs, the other users' jobs and the backup are in place, and the portal group keeps write access to the config.

There is no internet on the machine; everything you need is already installed. Run administrative commands with sudo.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 4

An audit rule watches writes to /etc/portal/portal.conf — it is in effect now and lives in /etc/audit/rules.d — not done yet
~/culprit.txt names the user and the full path of the program that writes to the config — not done yet
The job that changes the config is no longer scheduled — not done yet
Everything else is as before — cron runs, the other jobs and the backup are in place, the portal group can write the config — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 4

An audit rule watches writes to /etc/portal/portal.conf — it is in effect now and lives in /etc/audit/rules.d — not done yet
~/culprit.txt names the user and the full path of the program that writes to the config — not done yet
The job that changes the config is no longer scheduled — not done yet
Everything else is as before — cron runs, the other jobs and the backup are in place, the portal group can write the config — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 4

An audit rule watches writes to /etc/portal/portal.conf — it is in effect now and lives in /etc/audit/rules.d — not done yet
~/culprit.txt names the user and the full path of the program that writes to the config — not done yet
The job that changes the config is no longer scheduled — not done yet
Everything else is as before — cron runs, the other jobs and the backup are in place, the portal group can write the config — not done yet

Security · picked for this lab

auditd — security auditing

Cheat sheet

$_ — insert into the terminal

sudo auditctl -w /etc/passwd -p wa -k passwd_changes watch a file
sudo auditctl -a always,exit -F arch=b64 -S execve -F euid=0 -k root_cmd
sudo auditctl -l current rules
sudo auditctl -s subsystem status
sudo ausearch -k passwd_changes -i by key, readable
sudo aureport --summary
permanent: /etc/audit/rules.d/*.rules → augenrules --load

The daemon itself is configured in /etc/audit/auditd.conf — not the same thing as audit rules (those live in rules.d, see "Recipes").

log_file = /var/log/audit/audit.log
max_log_file = 8                  # MB, rotation threshold
max_log_file_action = ROTATE     # ROTATE · KEEP_LOGS · SUSPEND · SYSLOG
num_logs = 5
space_left_action = SYSLOG        # what to do when disk space runs low
admin_space_left_action = SUSPEND

Apply changes:

bash

sudo systemctl restart auditd     # reread auditd.conf
sudo auditctl -s                  # subsystem healthy again, no losses

Important. max_log_file_action = SUSPEND and admin_space_left_action = SUSPEND stop recording events once the disk fills up — a compliance system would rather go silent than lose events unnoticed. For looser requirements use ROTATE/SYSLOG; the choice trades audit reliability against availability.

Rules: watch and syscall

bash

# watch a file or dir: -p = r read, w write, x exec, a attr
sudo auditctl -w /etc/passwd -p wa -k passwd_changes
sudo auditctl -w /etc/sudoers -p wa -k sudoers
sudo auditctl -w /etc/ssh/sshd_config -p wa -k sshd_conf

# syscall rules (-a always,exit): catch actions
sudo auditctl -a always,exit -F arch=b64 -S execve -F euid=0 \
  -k root_cmd                     # root commands
sudo auditctl -a always,exit -F arch=b64 -S unlink,unlinkat \
  -F dir=/etc -k etc_delete

sudo auditctl -l                  # current rules
sudo auditctl -D                  # delete all (temporarily)
# permanent — /etc/audit/rules.d/*.rules + sudo augenrules --load
# lock rules against change until reboot: last rule -e 2 (immutable)

Searching events

bash

sudo ausearch -k passwd_changes           # by key (-k)
sudo ausearch -k passwd_changes -i        # -i: names, not numbers
sudo ausearch -f /etc/passwd -ts today    # by file, for today
sudo ausearch -ui 1000 -ts recent         # UID, last 10 minutes
sudo ausearch -m USER_LOGIN --start today # by event type (logins)
sudo aureport --summary                   # summary report
sudo aureport -au                         # authentication report
sudo aureport -f -i                       # file-event report

Reading an event

On the lab machine the auditd service is off: sudo systemctl enable --now auditd. The rule -w /etc/demo.conf -p wa -k democonf caught a write to the file from a cron job of the user deploy. One event is several records with a common number; the main ones are PROCTITLE and SYSCALL:

text

student@ubuntu:~$ sudo ausearch -k democonf -i \
> | grep -E '^type=(SYSCALL|PROCTITLE)'
type=PROCTITLE msg=audit(10/21/26 05:49:01.453:288) :
  proctitle=/bin/sh /usr/local/bin/tweak.sh
type=SYSCALL msg=audit(10/21/26 05:49:01.453:288) : arch=x86_64
  syscall=openat success=yes exit=3 … auid=deploy uid=deploy gid=deploy
  euid=deploy … tty=(none) ses=4 comm=tweak.sh exe=/usr/bin/dash
  subj=unconfined key=democonf

Field	Meaning
`auid`	who logged in — it does not change after `sudo` or `su`
`uid`, `euid`	which user the process ran as at that moment
`comm`	the process name (15 characters)
`exe`	the executable; for a script it is the interpreter, `dash` here
`proctitle`	the whole command line — for a script it holds the script's path
`tty`, `ses`	the terminal and login session number; cron has no terminal
`key`	the rule key from `-k`

The auid of a cron job is the crontab owner: cron opens a session for them through PAM (pam_loginuid). A write through sudo sh -c 'echo … >> /etc/demo.conf' gives uid=root, while auid is the user who logged in; processes without a login (services, commands through the guest agent) have auid=unset. The per-file summary is aureport -f -i:

text

# date time file syscall success exe auid event
1. 10/21/26 05:48:29 /etc/ openat yes /usr/bin/dash unset 279
2. 10/21/26 05:49:01 /etc/ openat yes /usr/bin/dash deploy 288

Permanent rules: rules.d

bash

echo '-w /etc/demo.conf -p wa -k democonf' \
  | sudo tee /etc/audit/rules.d/demo.rules
sudo augenrules --load
sudo auditctl -l

augenrules builds /etc/audit/audit.rules from all the rules.d/*.rules files and loads it into the kernel; auditd does the same at start. A rule written straight into audit.rules is gone after the next augenrules --load: the file is rebuilt, and auditctl -l does not list the rule. Only root can read audit.rules. augenrules --load prints the subsystem state (enabled 1, backlog …) — that is not an error.

Who changed an important file?

Set a watch (auditctl -w file -p wa -k label), then ausearch -k label -i → uid/command/time. For something that already happened (no watch was set), there are no events — auditing is not retroactive. That's why key watches are set in advance (in rules.d).

Too many events / noise / losses (lost)

auditctl -s shows lost/backlog. Write narrow rules (specific files/syscalls/keys), raise -b (backlog), and watch the size of /var/log/audit/ (its own rotation is in auditd.conf: max_log_file, num_logs).

Rules disappeared after a reboot

auditctl -w/-a rules are temporary. Permanent ones go in /etc/audit/rules.d/*.rules, applied with augenrules --load (or when auditd starts).

I can't change or delete rules

Immutable mode is on (-e 2) — rules are locked until reboot (by design, to protect against tampering).

auditd vs journald

journald handles operational service logs (journalctl); auditd handles who-did-what for security (an immutable trail). Different subsystems.

AppArmor denials are gone from journalctl -k

While auditd runs, kernel audit messages, apparmor="DENIED" included, go to it and are written to /var/log/audit/audit.log (type=AVC), and journalctl -k gets no new lines. Find them with sudo ausearch -m AVC -i (see apparmor).

Who changed the file: auditd

Situation

Task

Signing in is needed to start the machine

Rules: watch and syscall

Searching events

Reading an event

Permanent rules: rules.d

Who changed an important file?

Too many events / noise / losses (lost)

Rules disappeared after a reboot

I can't change or delete rules

auditd vs journald

AppArmor denials are gone from journalctl -k