A postmortem is worth its action items: what to change so that the outage does not repeat or ends sooner.
Each action item closes one “what let it happen”: a check that was missing, an alert that stayed silent, a runbook step. A good action item is concrete and checkable — a week later it is clear whether it is done:
| Will not do | Will do |
|---|---|
| be more careful with releases | after the restart the release waits for a 200 from the site, otherwise brings back the previous version |
| keep an eye on nginx | the release runs nginx -t before the reload and stops on an error |
| improve monitoring | an alert on Start request repeated too quickly for the site's services |
Which action closes the gap “the release did not check that lab-app came up”?
-
A reminder cannot be checked a week later, and at three in the morning it will not work. An action changes the release, not people's memory.
-
A senior engineer will not see the crash either if the release does not check for it. The gap is closed by a check in the release itself.
-
Release 1.5.0 crashes at every start, and the hundredth restart will crash too. There will be more tries, and the site will not come back.
The action is concrete and checkable: a week later you can see whether the release waits for a 200. It also shortens the outage — the rollback happens without a person.
Sometimes an outage is made of two events far apart in time. A latent failure lies unnoticed — for example, a config with an error that the service has not reread yet. A trigger is an ordinary action that exposes it: a restart, a reboot, a load spike. In the timeline of such an outage the change comes before the start, the root cause names both parts, and each of them needs an action item.
A postmortem is written as a file: the sections “Timeline” (“time event” lines, as in timeline.txt), “Root cause” and “Action items”, optionally “Impact”. Headings are markdown lines like ## Timeline.
Write the postmortem
Just now the site stopped answering altogether and a few seconds later worked again. Work out from the machine's logs what happened and write a postmortem to ~/postmortem.md with the sections “Timeline”, “Root cause” and “Action items”. The timeline holds the time the site stopped answering.
Goals · 0 of 1
- Write the postmortem to ~/postmortem.md not done yet