Server security / Stage 9 of 10 · step 5 of 6

The config changes by itself

~20 min

Not running Starting usually ~20 s

Situation

The company portal reads its settings from /etc/portal/portal.conf. This morning it turned out to have debug = true, open sign-up and the address of someone else's database. You put the working settings back — a minute later the file was the same as before:

cat /etc/portal/portal.conf
ls -l /etc/portal/portal.conf

The portal group can write the config — the developers and the deployment account. Your crontab -l is empty, and /etc/cron.d holds only the settings backup.

Task

Put an audit rule on writes to /etc/portal/portal.conf. The rule is in effect now and lives in /etc/audit/rules.d — it survives a reboot.
Use the audit events to find who rewrites the config. Write to ~/culprit.txt, on one line, the user name and the full path of the program that writes to the file.
Take that job off the schedule.
Leave everything else as it was: cron runs, the other users' jobs and the backup are in place, and the portal group keeps write access to the config.

There is no internet on the machine; everything you need is already installed. Run administrative commands with sudo.

Lab goals

0 of 4

An audit rule watches writes to /etc/portal/portal.conf — it is in effect now and lives in /etc/audit/rules.d — not checked yet
~/culprit.txt names the user and the full path of the program that writes to the config — not checked yet
The job that changes the config is no longer scheduled — not checked yet
Everything else is as before — cron runs, the other jobs and the backup are in place, the portal group can write the config — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Security · picked for this lab

auditd — security auditing

Cheat sheet

$_ — insert into the terminal

sudo auditctl -w /etc/passwd -p wa -k passwd_changes watch a file
sudo auditctl -a always,exit -F arch=b64 -S execve -F euid=0 -k root_cmd
sudo auditctl -l current rules
sudo auditctl -s subsystem status
sudo ausearch -k passwd_changes -i by key, readable
sudo aureport --summary
permanent: /etc/audit/rules.d/*.rules → augenrules --load

The daemon itself is configured in /etc/audit/auditd.conf — not the same thing as audit rules (those live in rules.d, see "Recipes").

log_file = /var/log/audit/audit.log
max_log_file = 8                  # MB, rotation threshold
max_log_file_action = ROTATE     # ROTATE · KEEP_LOGS · SUSPEND · SYSLOG
num_logs = 5
space_left_action = SYSLOG        # what to do when disk space runs low
admin_space_left_action = SUSPEND

Apply changes:

bash

sudo systemctl restart auditd     # reread auditd.conf
sudo auditctl -s                  # subsystem healthy again, no losses

Important. max_log_file_action = SUSPEND and admin_space_left_action = SUSPEND stop recording events once the disk fills up — a compliance system would rather go silent than lose events unnoticed. For looser requirements use ROTATE/SYSLOG; the choice trades audit reliability against availability.

Rules: watch and syscall

bash

# watch a file or dir: -p = r read, w write, x exec, a attr
sudo auditctl -w /etc/passwd -p wa -k passwd_changes
sudo auditctl -w /etc/sudoers -p wa -k sudoers
sudo auditctl -w /etc/ssh/sshd_config -p wa -k sshd_conf

# syscall rules (-a always,exit): catch actions
sudo auditctl -a always,exit -F arch=b64 -S execve -F euid=0 \
  -k root_cmd                     # root commands
sudo auditctl -a always,exit -F arch=b64 -S unlink,unlinkat \
  -F dir=/etc -k etc_delete

sudo auditctl -l                  # current rules
sudo auditctl -D                  # delete all (temporarily)
# permanent — /etc/audit/rules.d/*.rules + sudo augenrules --load
# lock rules against change until reboot: last rule -e 2 (immutable)

Searching events

bash

sudo ausearch -k passwd_changes           # by key (-k)
sudo ausearch -k passwd_changes -i        # -i: names, not numbers
sudo ausearch -f /etc/passwd -ts today    # by file, for today
sudo ausearch -ui 1000 -ts recent         # UID, last 10 minutes
sudo ausearch -m USER_LOGIN --start today # by event type (logins)
sudo aureport --summary                   # summary report
sudo aureport -au                         # authentication report
sudo aureport -f -i                       # file-event report

Reading an event

On the lab machine the auditd service is off: sudo systemctl enable --now auditd. The rule -w /etc/demo.conf -p wa -k democonf caught a write to the file from a cron job of the user deploy. One event is several records with a common number; the main ones are PROCTITLE and SYSCALL:

text

student@ubuntu:~$ sudo ausearch -k democonf -i \
> | grep -E '^type=(SYSCALL|PROCTITLE)'
type=PROCTITLE msg=audit(10/21/26 05:49:01.453:288) :
  proctitle=/bin/sh /usr/local/bin/tweak.sh
type=SYSCALL msg=audit(10/21/26 05:49:01.453:288) : arch=x86_64
  syscall=openat success=yes exit=3 … auid=deploy uid=deploy gid=deploy
  euid=deploy … tty=(none) ses=4 comm=tweak.sh exe=/usr/bin/dash
  subj=unconfined key=democonf

Field	Meaning
`auid`	who logged in — it does not change after `sudo` or `su`
`uid`, `euid`	which user the process ran as at that moment
`comm`	the process name (15 characters)
`exe`	the executable; for a script it is the interpreter, `dash` here
`proctitle`	the whole command line — for a script it holds the script's path
`tty`, `ses`	the terminal and login session number; cron has no terminal
`key`	the rule key from `-k`

The auid of a cron job is the crontab owner: cron opens a session for them through PAM (pam_loginuid). A write through sudo sh -c 'echo … >> /etc/demo.conf' gives uid=root, while auid is the user who logged in; processes without a login (services, commands through the guest agent) have auid=unset. The per-file summary is aureport -f -i:

text

# date time file syscall success exe auid event
1. 10/21/26 05:48:29 /etc/ openat yes /usr/bin/dash unset 279
2. 10/21/26 05:49:01 /etc/ openat yes /usr/bin/dash deploy 288

Permanent rules: rules.d

bash

echo '-w /etc/demo.conf -p wa -k democonf' \
  | sudo tee /etc/audit/rules.d/demo.rules
sudo augenrules --load
sudo auditctl -l

augenrules builds /etc/audit/audit.rules from all the rules.d/*.rules files and loads it into the kernel; auditd does the same at start. A rule written straight into audit.rules is gone after the next augenrules --load: the file is rebuilt, and auditctl -l does not list the rule. Only root can read audit.rules. augenrules --load prints the subsystem state (enabled 1, backlog …) — that is not an error.

Who changed an important file?

Set a watch (auditctl -w file -p wa -k label), then ausearch -k label -i → uid/command/time. For something that already happened (no watch was set), there are no events — auditing is not retroactive. That's why key watches are set in advance (in rules.d).

Too many events / noise / losses (lost)

auditctl -s shows lost/backlog. Write narrow rules (specific files/syscalls/keys), raise -b (backlog), and watch the size of /var/log/audit/ (its own rotation is in auditd.conf: max_log_file, num_logs).

Rules disappeared after a reboot

auditctl -w/-a rules are temporary. Permanent ones go in /etc/audit/rules.d/*.rules, applied with augenrules --load (or when auditd starts).

I can't change or delete rules

Immutable mode is on (-e 2) — rules are locked until reboot (by design, to protect against tampering).

auditd vs journald

journald handles operational service logs (journalctl); auditd handles who-did-what for security (an immutable trail). Different subsystems.

AppArmor denials are gone from journalctl -k

While auditd runs, kernel audit messages, apparmor="DENIED" included, go to it and are written to /var/log/audit/audit.log (type=AVC), and journalctl -k gets no new lines. Find them with sudo ausearch -m AVC -i (see apparmor).

The config changes by itself

Situation

Task

Signing in is needed to start the machine

Rules: watch and syscall

Searching events

Reading an event

Permanent rules: rules.d

Who changed an important file?

Too many events / noise / losses (lost)

Rules disappeared after a reboot

I can't change or delete rules

auditd vs journald

AppArmor denials are gone from journalctl -k