A runbook and the first ten minutes

The site has to work again before you understand why it broke.

Why port 8001 breaks the site can be found out, but while you are finding out, the site is down. So an incident holds two jobs, and they go in turn. First restoration: give the service back to users in the fastest way that is easy to undo. Then the cause: what happened and how to keep it from happening again — that is the job of the review after the outage, the lesson on a blameless postmortem.

If the outage began with a change, the fastest way is a rollback: put everything back as it was before the change. A rollback needs no understanding of the failure: before the edit the site worked, after it the site does not, so you put things back as they were. Other ways to restore are restarting a failed service or moving traffic to a standby machine.

Before rolling back, keep the change: copy the file to the incident directory. The rollback wipes the edit, and the review needs exactly that edit — the copy lets you calmly look for what was wrong with it later. The application reads its settings file at start, so after the rollback it is restarted.