SRE practices: SLOs, incidents, redundancy / Stage 4 of 8

A runbook and the first ten minutes

Not running Starting usually ~20 s

Step 6 of 7 · lab · 20 min

The site broke after a release

After the latest release the bookshop answers error 500 at every address. Put the site back on the previous working release without the visitors noticing the switch, and keep the failed release for the review.

Situation

An alert came in: the books.lab.test bookshop answers with error 500. The home page, the catalogue pages, even the style files — everything returns 500. Visitors are on the site right now.

The site is static content on nginx. CI deploys it: every release is unpacked into its own directory /srv/books/releases/<version>, and nginx serves the files from /srv/books/current.

Task

Bring the site back first; the cause will be reviewed later.

The site runs again on the release that served the pages before the failure: /srv/books/current is a link to that release.
Every address of http://books.lab.test/ answers 200.
The visitors do not lose a single answer: nginx is not restarted or stopped.
The failed release stays in releases/ untouched — the failure will be reviewed from it.

Constraints

Don't change the nginx configuration: the site is still served from /srv/books/current.
Don't edit the release files.

The nginx logs are read with sudo. There is no internet on the machine.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 4

/srv/books/current is a link to the release that was live before the failed rollout — not done yet
A series of requests to http://books.lab.test/ during the check — every answer is 200, the pages come from that release — not done yet
nginx was not restarted or stopped — only reloaded; the site configuration is unchanged — not done yet
The failed release stays in releases/ untouched — for the review — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 4

/srv/books/current is a link to the release that was live before the failed rollout — not done yet
A series of requests to http://books.lab.test/ during the check — every answer is 200, the pages come from that release — not done yet
nginx was not restarted or stopped — only reloaded; the site configuration is unchanged — not done yet
The failed release stays in releases/ untouched — for the review — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 4

/srv/books/current is a link to the release that was live before the failed rollout — not done yet
A series of requests to http://books.lab.test/ during the check — every answer is 200, the pages come from that release — not done yet
nginx was not restarted or stopped — only reloaded; the site configuration is unchanged — not done yet
The failed release stays in releases/ untouched — for the review — not done yet

SRE · picked for this lab

Incident response — the first minutes, runbook and timeline

Cheat sheet

$_ — insert into the terminal

1. the symptom as the user sees it
curl -s -o /dev/null -w '%{http_code}\n' http://localhost/
2. timeline: one line per action, UTC time
mkdir -p ~/incident; cd ~/incident
echo "$(date -u +%T) alert 'site 502', confirmed" >> timeline.txt
3. what changed last
sudo journalctl -t sudo --since -1h | grep COMMAND who ran what
sudo find /etc /opt /srv -newermt '-1 hour' -type f fresh files
ls -lt /etc/nginx/sites-enabled/ | head
4. a copy before the rollback — evidence for the review
sudo cp -a /etc/default/lab-app ~/incident/
5. check with the same request, write the time down

Confirm the symptom with a series of requests

bash

for i in $(seq 20); do
  curl -s -o /dev/null -w '%{http_code}\n' http://localhost/
done | sort | uniq -c

A single curl may hit a lucky moment: with one dead backend of two, only part of the requests fails. A series shows the error share.

Recent changes in one list

bash

sudo journalctl -t sudo --since -1h -o short-iso | grep COMMAND
sudo find /etc /opt /srv -newermt '-1 hour' -type f \
  -printf '%TT %p\n' | sort

The first command lists what people ran with sudo, the second lists files with their modification time (with fractions of a second) in order. -newermt also takes an exact time: -newermt '2026-10-22 12:00'.

Roll back a config edit

bash

sudo cp -a /etc/nginx/sites-available/site ~/incident/site.bad
sudo cp -a /root/site.bak /etc/nginx/sites-available/site
sudo nginx -t && sudo systemctl reload nginx
echo "$(date -u +%T) config rolled back" >> ~/incident/timeline.txt

The bad version is kept for the review, the checked config is applied with reload without dropping connections (nginx). Configs kept in git are rolled back with git checkout <previous commit> -- <file>. Rolling back a release by switching a symlink — ln.

Rolled back, did not help

The wrong change was rolled back: compare the outage start in the request log and the site check with the times of changes.
The service did not reread the config: it needs reload or restart.
There were several changes: revert them one at a time, checking after each.
The outage began before the change — look for another cause: disk, memory, network, a dependency (database, a neighbouring service).

The restart helped, the outage came back

The restart removed the symptom and the cause stayed: a memory leak, a growing log, exhausted connections. Save systemctl status, the service journal and free -m before the next restart — they show what piles up.

Nothing in the sudo journal

The change did not go through sudo: a root login, a release running as the service user, automation (cron, ansible, package upgrades). Look at file modification times (stat, find -newermt) and at the whole journal around the outage start: journalctl --since '12:00' --until '12:05'.

Times in the logs do not match

journalctl prints time in the machine's time zone, date -u in UTC. On the lab machine they are the same. On a machine with another zone, compare in one: journalctl --utc, TZ=UTC date.

Fixing without a copy

Editing in place, deleting the "broken" file, restarting without saving the journal erases the evidence. The review is then built on guesses, and the same outage repeats. A copy in ~/incident/ takes seconds.

Several edits at once

Three changes in a row with no check in between: the service is back, but which one helped is unclear, and the other two may have added a new breakage. One change — one check — one line in the timeline.

A runbook and the first ten minutes

Situation

Task

Constraints

Signing in is needed to start the machine

Confirm the symptom with a series of requests

Recent changes in one list

Roll back a config edit

Rolled back, did not help

The restart helped, the outage came back

Nothing in the sudo journal

Times in the logs do not match

Fixing without a copy

Several edits at once