Postmortem — a blameless outage review

A postmortem (outage review) is a document written after an incident (incident-response), once the service works again. It has four parts: what happened minute by minute, what users felt, why it happened and what the team will change. The goal is that the same outage does not repeat and that everyone who was not on shift gets its lessons.

# Review: the site was down on 22.10

### Timeline
12:00:40 a release changed the site config; nginx reload failed
12:00:52 scheduled nginx restart — it did not start:
         unknown directive "proxy_pas"
12:00:53 lab-sitecheck: the site does not answer — outage starts
12:01:21 lab-sitecheck: the site answers 200 — outage ends

### Impact
For 28 s the site answered nobody: 0.2 % of the 30-day error
budget at SLO 99.5 %.

### Root cause
A typo in the site config. The release did not run `nginx -t`,
nobody noticed the failed reload, and the restart stopped the site.

### Action items
- The release runs `nginx -t` and stops on an error.
- An alert on a failed nginx reload.

In Russian the sections are called Хронология, Влияние, Причина and Действия.

Timeline

The timeline is the outage events in time order, a line per event: UTC time first, then a fact. The format is the same as the timeline kept during the incident (incident-response):

HH:MM:SS event

If nobody kept it, the timeline is assembled from the logs. Each source knows its own part:

The start and end of the outage are taken from what users saw: the first error in access.log or in the site check, and the first good response after them. The time of the edit is a separate line. Between an edit and the outage start there may be seconds or days: a typo waits in the file until the service is restarted. Journal lines go into the timeline as they are (-o short-iso), only trimmed to the fact.

Blameless

A blameless review starts from the fact that people always make mistakes, and the system has to catch a mistake before users do. The review asks what let the mistake reach them: which check was missing, what was not visible, why the rollback took ten minutes.

So the review uses roles instead of names: "the release", "the on-call", deploy. A person named as the culprit will keep quiet about their next mistake, and the next outage will take longer to find.

Root cause

An outage rarely has a single cause. Usually it is a pair:

• a latent defect — something broken in advance and waiting: a typo in a config, a disk 95 % full, an expired certificate;
• a trigger — the event that made the defect visible: a restart, a load spike, the nightly rotation.

A chain of "why" leads to them: the site was down — nginx did not start — the config had a typo — the release did not check the config — the release has no such check. Stop the chain at something the team can change. The root cause section names the broken components: the nginx site config, pg_hba.conf, the alerting rule.

Impact

How long the outage lasted (as users saw it), how many requests failed, what share of the error budget it ate (sli-slo). The numbers come from access.log for the outage window — see the recipe below.

Action items

Action items are what the team will change so the outage does not repeat or is found sooner. Each one is concrete and checkable, with an owner and a deadline:

"Be more careful" cannot be checked, and in a month the outage repeats.

• incident-response — the first minutes, runbook, live timeline.
• sli-slo — the error budget for the impact section.
• journalctl — time filters, -t, -u, -o short-iso.
• nginx — access.log: codes and response times.
• date — unix time to clock time and back.
• sed-awk — fields and counting in awk.