SRE practices: SLOs, incidents, redundancy / Stage 4 of 8 · step 6 of 7

The site broke after a release

~20 min

Not running Starting usually ~20 s

Situation

An alert came in: the books.lab.test bookshop answers with error 500. The home page, the catalogue pages, even the style files — everything returns 500. Visitors are on the site right now.

The site is static content on nginx. CI deploys it: every release is unpacked into its own directory /srv/books/releases/<version>, and nginx serves the files from /srv/books/current.

Task

Bring the site back first; the cause will be reviewed later.

The site runs again on the release that served the pages before the failure: /srv/books/current is a link to that release.
Every address of http://books.lab.test/ answers 200.
The visitors do not lose a single answer: nginx is not restarted or stopped.
The failed release stays in releases/ untouched — the failure will be reviewed from it.

Constraints

Don't change the nginx configuration: the site is still served from /srv/books/current.
Don't edit the release files.

The nginx logs are read with sudo. There is no internet on the machine.

Lab goals

0 of 4

/srv/books/current is a link to the release that was live before the failed rollout — not checked yet
A series of requests to http://books.lab.test/ during the check — every answer is 200, the pages come from that release — not checked yet
nginx was not restarted or stopped — only reloaded; the site configuration is unchanged — not checked yet
The failed release stays in releases/ untouched — for the review — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

SRE · picked for this lab

Incident response — the first minutes, runbook and timeline

Cheat sheet

$_ — insert into the terminal

1. the symptom as the user sees it
curl -s -o /dev/null -w '%{http_code}\n' http://localhost/
2. timeline: one line per action, UTC time
mkdir -p ~/incident; cd ~/incident
echo "$(date -u +%T) alert 'site 502', confirmed" >> timeline.txt
3. what changed last
sudo journalctl -t sudo --since -1h | grep COMMAND who ran what
sudo find /etc /opt /srv -newermt '-1 hour' -type f fresh files
ls -lt /etc/nginx/sites-enabled/ | head
4. a copy before the rollback — evidence for the review
sudo cp -a /etc/default/lab-app ~/incident/
5. check with the same request, write the time down

Confirm the symptom with a series of requests

bash

for i in $(seq 20); do
  curl -s -o /dev/null -w '%{http_code}\n' http://localhost/
done | sort | uniq -c

A single curl may hit a lucky moment: with one dead backend of two, only part of the requests fails. A series shows the error share.

Recent changes in one list

bash

sudo journalctl -t sudo --since -1h -o short-iso | grep COMMAND
sudo find /etc /opt /srv -newermt '-1 hour' -type f \
  -printf '%TT %p\n' | sort

The first command lists what people ran with sudo, the second lists files with their modification time (with fractions of a second) in order. -newermt also takes an exact time: -newermt '2026-10-22 12:00'.

Roll back a config edit

bash

sudo cp -a /etc/nginx/sites-available/site ~/incident/site.bad
sudo cp -a /root/site.bak /etc/nginx/sites-available/site
sudo nginx -t && sudo systemctl reload nginx
echo "$(date -u +%T) config rolled back" >> ~/incident/timeline.txt

The bad version is kept for the review, the checked config is applied with reload without dropping connections (nginx). Configs kept in git are rolled back with git checkout <previous commit> -- <file>. Rolling back a release by switching a symlink — ln.

Rolled back, did not help

The wrong change was rolled back: compare the outage start in the request log and the site check with the times of changes.
The service did not reread the config: it needs reload or restart.
There were several changes: revert them one at a time, checking after each.
The outage began before the change — look for another cause: disk, memory, network, a dependency (database, a neighbouring service).

The restart helped, the outage came back

The restart removed the symptom and the cause stayed: a memory leak, a growing log, exhausted connections. Save systemctl status, the service journal and free -m before the next restart — they show what piles up.

Nothing in the sudo journal

The change did not go through sudo: a root login, a release running as the service user, automation (cron, ansible, package upgrades). Look at file modification times (stat, find -newermt) and at the whole journal around the outage start: journalctl --since '12:00' --until '12:05'.

Times in the logs do not match

journalctl prints time in the machine's time zone, date -u in UTC. On the lab machine they are the same. On a machine with another zone, compare in one: journalctl --utc, TZ=UTC date.

Fixing without a copy

Editing in place, deleting the "broken" file, restarting without saving the journal erases the evidence. The review is then built on guesses, and the same outage repeats. A copy in ~/incident/ takes seconds.

Several edits at once

Three changes in a row with no check in between: the service is back, but which one helped is unclear, and the other two may have added a new breakage. One change — one check — one line in the timeline.

The site broke after a release

Situation

Task

Constraints

Signing in is needed to start the machine

Confirm the symptom with a series of requests

Recent changes in one list

Roll back a config edit

Rolled back, did not help

The restart helped, the outage came back

Nothing in the sudo journal

Times in the logs do not match

Fixing without a copy

Several edits at once