Each log holds its own part of the truth about an outage.
A timeline is put together from the records the machine kept by itself. Here there are three kinds of them, and each answers its own question:
| Source | What it holds | Command |
|---|---|---|
| the sudo journal | what people did | sudo journalctl -t sudo |
| the service journal | what the program did | sudo journalctl -u lab-app |
| the site check | what users saw | sudo journalctl -t lab-sitecheck |
The start and the end of an outage come from the third source — from what users saw; for a site behind nginx that also includes the answer codes in its access.log. A change can lie for seconds or hours before something breaks: the new app.py sat on disk for six seconds while the old process served requests. So the time of the change is a separate timeline line. From the length of the outage you later work out the impact: how many requests got no answer and what share of the error budget that ate.
The machine's journal for 15 minutes is over a thousand lines: the boot, the console login, services. -t tags keep people and the site check from them in one list by time, and the service journal is read with a separate -u query, as in the lesson on the runbook. It is the service journal that knows why the site did not come back by itself: lab-app's unit has Restart=on-failure.
Look into the service's journal
lab-app's unit has Restart=on-failure — systemd restarts the failed service by itself. Look at how the service is set up and at what its journal for 15 minutes says about errors and restarts.
First, predict
What did systemd do with the new version that crashed at start?
Now start the exercise, run the command and compare your prediction with the output.