Linux Labs

SRE practices: SLOs, incidents, redundancy / Stage 5 of 8

A blameless postmortem

Not running Starting usually ~20 s
Leave the lesson

Step 1 of 6 · on a machine · 5 min

No timeline

What you will learn

  • Put an outage timeline together from the logs — what people did, what the program did and what users saw.
  • Write a blameless postmortem — what broke and what let it happen.
  • Word the action items after an outage — concrete and checkable.

The site works again: after a release of a new lab-app version the on-call engineer brought back the previous one. Reviewing the outage takes a timeline, and nobody kept one as it went. Rebuild it from the machine's journal.

Exercise · do it

Put together the outage timeline

Just now the site answered 502 for a few seconds: after a release of a new lab-app version the on-call engineer brought back the previous one. Nobody kept a timeline. Look at the machine's journal for 15 minutes and write to ~/timeline.txt, as “time event” lines, when the site stopped answering and when it worked again.

Sign in and start The machine is available after signing in

Goals · 0 of 1

  • Write the start and the end of the outage to ~/timeline.txt not done yet
Check yourself 1 question

When did the site begin to answer 502, according to the journal?

  • That is the end of the outage: right after the on-call engineer's commands the journal has RESOLVED.

  • After the copy the old process went on serving requests for six more seconds: FIRING comes after the restart, not after cp.

The site began to answer 502 right after the lab-app restart; after the file was copied it still worked.

Step goals

0 of 1
  • Write the start and the end of the outage to ~/timeline.txt — not done yet
Ubuntu 24.04

Step goals

0 of 1
  • Write the start and the end of the outage to ~/timeline.txt — not done yet
Loading…