A runbook turns the first minutes of an outage from a blind search into a list of steps.
An outage that hits users is called an incident: the site does not answer, pages take ten seconds to load, orders are not saved. While it lasts, every request left without an answer spends the error budget. That is why the first minutes are planned in advance — in a runbook.
A runbook is written for every alert: what it means, how to confirm it, where to look, how to bring the service back and whom to call if that fails. Its steps are ready commands: at three in the morning you run them without recalling how the system is built. The “site 502” runbook looks like this:
| Step | What to do |
|---|---|
| 1. Confirm | curl -s -o /dev/null -w '%{http_code}\n' http://localhost/ |
| 2. Start the timeline | write the time of the alert to ~/incident/timeline.txt |
| 3. What changed last | sudo journalctl -t sudo --since -1h, ls -lt of the config directories |
| 4. Bring the site back | roll back the last change, keeping a copy of the changed file |
| 5. Check | the same curl; the time goes into the timeline |
| 6. Report | in the team chat; not done in 15 minutes — call the senior on-call |
The third step comes this early because outages most often start with a change: a release, a config edit, a package update. A change minutes before the alert is the first suspect. The -t sudo tag in journalctl belongs to sudo records: every command run through it leaves a line with the user, the directory and the command itself after COMMAND=.
In the last hour nobody but you has run sudo. What will sudo journalctl -t sudo --since -1h print?
-
The command itself runs through sudo, and sudo logged it before journalctl began to read the journal.
-
The command already runs as root through sudo, so it reads the whole journal.
-
root's commands without sudo do not reach the journal: only sudo sets the sudo tag — on its own runs.
sudo logs each of its runs — including the one you opened the journal with: the output has a line with your name and COMMAND=/usr/bin/journalctl -t sudo --since -1h. That is why the last lines of the sudo journal are usually your own.
Key points
- An incident is an outage that hits users; its first minutes follow the runbook of that alert.
- A runbook: confirm, start the timeline, find the last change, bring the service back, check, report.
-
A change minutes before the alert is the first suspect; commands run with sudo show in
journalctl -t sudo.