The timer will not begin a backup while the previous one is running: a second systemctl start waits for the first run to end. By hand, though, the script can be started at any moment.
A real backup of a big directory takes a long time — in backup.sh, sleep 3 stands in for it. Say a colleague started a backup by hand overnight while the service was running. Repeat that on one line: & runs the first command in the background, the second starts at once, and wait waits for the background one.
Run two backups at once
In ~/sched-lab, run backup.sh twice on one line — the first in the background — then look at what is in archive.
First, predict
How many archives will be in archive after the two runs?
Now start the exercise, run the command and compare your prediction with the output.
Both runs packed data into the same temporary file, archive/partial.tar.gz. The first renamed it into an archive, and the second could no longer find the file — hence mv: cannot stat. Had the packing taken longer, the second run would also have overwritten the first one's file halfway.
A lock keeps two copies apart. flock holds a lock file while a command runs: flock -n /run/lock/backup.lock ./backup.sh runs the script if the lock is free and exits at once with status 1 if it is taken. The -n option means do not wait; without it, the second copy waits for the first and then runs as well. Any user can write to the /run/lock directory, which is there for locks.
For the script itself to take the lock — both in the service and when run by hand — it goes at the top of the script:
exec 9>file opens the file as stream 9 for the whole run of the script — the same way streams 1 and 2 are open. flock -n 9 takes the lock on that stream. There is no need to release the lock: it is freed by itself when the script ends.