Bash: text processing and scripts / Final lab

Cleaning up exports on a schedule

~40 min

Not running Starting usually ~20 s

Situation

The reporting service puts client exports into /srv/exports — one directory per client, *.csv files. An export is kept for 14 days. For that the previous admin wrote the /usr/local/bin/clean-exports script and runs it from cron every night at 03:30.

Complaints over the month:

the client acme cannot download yesterday's report — it is already gone;
the client blue river still has exports older than two months;
the nightly backup of the exports (the backup-exports service, from 03:00) writes tar: exports/…: File removed before we read it;
nobody knows when the cleanup ran or what it removed.

Your manager asks you to move the cleanup to a systemd timer: there every run shows up in systemctl and in the journal.

Task

Fix the cleanup and move it from cron to a systemd timer:

The clean-exports.timer timer is enabled and running, the schedule is up to you; cron no longer starts the cleanup.
The clean-exports.service service, started by hand, finishes successfully.
A run removes *.csv exports older than 14 days from /srv/exports — in any subdirectory and with any name.
Exports younger than 14 days and files outside /srv/exports stay in place.
Every run writes to the service journal how many exports it removed.
The cleanup does not overlap: while another process holds the backup lock — the backup itself or a second cleanup run — it removes nothing.

The check starts the service itself on its own exports in /srv/exports, reads the journal of that run and tries a run while the lock is busy. It removes its own files afterwards.

The machine has no Internet; everything you need is already installed.

Lab goals

0 of 6

The clean-exports.timer timer is enabled and running, cron does not start the cleanup — not checked yet
A manual start of clean-exports.service finishes successfully — not checked yet
A run removes *.csv exports older than 14 days in any subdirectory of /srv/exports — not checked yet
Exports younger than 14 days and files outside /srv/exports stay in place — not checked yet
A run writes to the service journal how many exports it removed — not checked yet
While another process holds the backup lock, the cleanup removes nothing — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Services and systemd · picked for this lab

Cron — the task scheduler

Cheat sheet

$_ — insert into the terminal

crontab -e edit your own crontab
crontab -l show the current crontab
*/15 * * * * /usr/local/bin/backup.sh every 15 minutes
systemctl list-timers list systemd timers
journalctl -u backup.service log for a specific job
grep CRON /var/log/syslog what cron ran

cron

A crontab line has five time fields plus a command:

┌── minute (0-59)
│ ┌── hour (0-23)
│ │ ┌── day of month (1-31)
│ │ │ ┌── month (1-12)
│ │ │ │ ┌── day of week (0-7, 0 and 7 = Sun)
│ │ │ │ │
* * * * *  command

bash

crontab -e                        # edit the user's crontab
crontab -l                        # show it
*/15 * * * * /usr/local/bin/backup.sh   # every 15 minutes
0 3 * * *   /usr/local/bin/nightly.sh    # daily at 03:00

System cron lives in /etc/cron.d/, /etc/crontab, /etc/cron.{daily,weekly}.

Other users' crontabs

Personal crontabs live in /var/spool/cron/crontabs/, one file per user. The directory is closed: only root sees the list, and only the owner reads a file.

text

student@ubuntu:~$ ls -ld /var/spool/cron/crontabs
drwx-wx--T 2 root crontab 4096 Oct 21 05:48 /var/spool/cron/crontabs
student@ubuntu:~$ sudo ls -l /var/spool/cron/crontabs
-rw------- 1 deploy crontab 209 Oct 21 05:48 deploy
student@ubuntu:~$ crontab -l -u deploy
must be privileged to use -u
student@ubuntu:~$ sudo crontab -l -u deploy
* * * * * /usr/local/bin/tweak.sh

Every cron run writes to the log: CRON[2086]: (deploy) CMD (/usr/local/bin/tweak.sh) — that is how you find whose job ran.

bash

sudo crontab -e -u deploy                 # fix or delete a line
sudo crontab -l -u deploy | grep -v tweak \
  | sudo crontab -u deploy -              # drop one line
sudo crontab -r -u deploy                 # delete ALL of deploy's jobs

Do not edit the file in /var/spool/cron/crontabs/ by hand: crontab checks the syntax and gives the file the right mode and owner. After crontab -r you get no crontab for deploy: every line is gone, not one.

Schedule and time zone

cron reads the schedule in the machine's time zone: 0 3 * * * means 03:00 by the time date shows. The machine's zone is in timedatectl (timedatectl); on the lab machine it is Etc/UTC.

cron picks up a zone change without a restart. The job 19 21 * * * fired at 21:19 Tokyo time after sudo timedatectl set-timezone Asia/Tokyo:

text

student@node1:~$ cat /tmp/cron-tz.log
Wed Oct 21 21:19:01 JST 2026

A TZ=Asia/Tokyo line in a crontab changes only the job's environment: date inside it prints Tokyo time, but the job still fires by the machine's zone. A server is usually kept in UTC — then the schedule is written in UTC. date shows what a time in another zone is in UTC:

text

student@ubuntu:~$ date -u -d 'TZ="Europe/Moscow" 03:00'
Wed Oct 21 00:00:00 UTC 2026

03:00 in Moscow is 00:00 UTC: the schedule line is 0 0 * * *.

systemd timers

A pair of files: .service (what to do) + .timer (when).

ini

# backup.timer
[Unit]
Description=Nightly backup
[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
[Install]
WantedBy=timers.target

bash

sudo systemctl enable --now backup.timer
systemctl list-timers             # timers and next runs
journalctl -u backup.service      # log of a job's execution

The OnCalendar schedule, the service environment and timer troubleshooting are covered in systemd-timers.

A cron job does not run

Cron has a minimal $PATH and no interactive-session environment (aliases, variables from ~/.bashrc) — use absolute paths to commands and files, do not rely on what an ordinary terminal shows.
Logs: journalctl -u cron (Ubuntu 24.04) or grep CRON /var/log/syslog (if cron logs to syslog rather than only to journald).
Every line must end with a newline; a % character in a command means a newline to cron, so it needs escaping (\%).

A timer did not fire while the machine was off

Add Persistent=true — it runs the missed job after the machine comes back up.

I don't know when the next run is

systemctl list-timers --all shows every timer, including inactive ones, with the last and next run times.

A user's crontab job is not visible in the system crontab

crontab -e edits the current user's crontab (stored under /var/spool/cron/crontabs/) — separate from the system files /etc/crontab and /etc/cron.d/*, where an extra field names the user to run the command as. Mixing up the formats is a common reason a job "works for me in crontab -e" but not as a system job.

Cleaning up exports on a schedule

Situation

Task

Signing in is needed to start the machine

cron

Other users' crontabs

Schedule and time zone

systemd timers

A cron job does not run

A timer did not fire while the machine was off

I don't know when the next run is

A user's crontab job is not visible in the system crontab