SRE practices: SLOs, incidents, redundancy / Stage 5 of 8

A blameless postmortem

Not running Starting usually ~20 s

Step 5 of 6 · lab · 30 min

Event intake has stalled and restarts are off-limits

Under load the event intake service can no longer save events to its volume and keeps them in memory, and a restart would lose them. Get the space back without a restart and make sure the service log never eats the volume again.

Situation

The lab-ingest service takes events from the shop front-ends — five per second, non-stop. It keeps accepted events in memory and every two seconds saves them as a batch to its volume /srv/ingest (the spool/ directory), and it writes every event to the log /srv/ingest/log/ingest.log.

An alert: curl localhost:8090/healthz answers 503, events no longer reach the disk and pile up in memory. The on-call engineer before you already "cleaned up" the volume, but no space showed up.

Don't restart lab-ingest: the events waiting in memory would be lost on a restart.

Task

The /srv/ingest volume has room again: less than half is used. It is the same volume — don't grow it or swap it.
The service runs as the same process as during the failure.
Event intake is back: new events are accepted (202), the in-memory queue is written to disk.
The service writes its log to /srv/ingest/log/ingest.log, and no deleted but open files remain on the volume.
The log will not eat the volume again: a logrotate rule rotates ingest.log as soon as it is larger than 10 MiB, keeps three archives at most, and after rotation the service writes to the new file without a restart. logrotate runs every hour on this machine.

Constraints

Don't restart or stop lab-ingest and lab-clients, and don't change their programs or units.

Some facts are visible only with sudo. There is no internet on the machine.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 5

The /srv/ingest volume has room again — less than half is used, the volume is the same — not done yet
The lab-ingest service was not restarted — the same process as during the failure keeps running — not done yet
Event intake is back — the check's events are accepted and the in-memory queue is written to disk — not done yet
The service writes its log to /srv/ingest/log/ingest.log, no deleted open files remain on the volume — not done yet
A logrotate rule rotates ingest.log by size (10 MiB at most, 3 archives at most), and after rotation the service writes to the new file without a restart — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 5

The /srv/ingest volume has room again — less than half is used, the volume is the same — not done yet
The lab-ingest service was not restarted — the same process as during the failure keeps running — not done yet
Event intake is back — the check's events are accepted and the in-memory queue is written to disk — not done yet
The service writes its log to /srv/ingest/log/ingest.log, no deleted open files remain on the volume — not done yet
A logrotate rule rotates ingest.log by size (10 MiB at most, 3 archives at most), and after rotation the service writes to the new file without a restart — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 5

The /srv/ingest volume has room again — less than half is used, the volume is the same — not done yet
The lab-ingest service was not restarted — the same process as during the failure keeps running — not done yet
Event intake is back — the check's events are accepted and the in-memory queue is written to disk — not done yet
The service writes its log to /srv/ingest/log/ingest.log, no deleted open files remain on the volume — not done yet
A logrotate rule rotates ingest.log by size (10 MiB at most, 3 archives at most), and after rotation the service writes to the new file without a restart — not done yet

SRE · picked for this lab

Incident response — the first minutes, runbook and timeline

Cheat sheet

$_ — insert into the terminal

1. the symptom as the user sees it
curl -s -o /dev/null -w '%{http_code}\n' http://localhost/
2. timeline: one line per action, UTC time
mkdir -p ~/incident; cd ~/incident
echo "$(date -u +%T) alert 'site 502', confirmed" >> timeline.txt
3. what changed last
sudo journalctl -t sudo --since -1h | grep COMMAND who ran what
sudo find /etc /opt /srv -newermt '-1 hour' -type f fresh files
ls -lt /etc/nginx/sites-enabled/ | head
4. a copy before the rollback — evidence for the review
sudo cp -a /etc/default/lab-app ~/incident/
5. check with the same request, write the time down

Confirm the symptom with a series of requests

bash

for i in $(seq 20); do
  curl -s -o /dev/null -w '%{http_code}\n' http://localhost/
done | sort | uniq -c

A single curl may hit a lucky moment: with one dead backend of two, only part of the requests fails. A series shows the error share.

Recent changes in one list

bash

sudo journalctl -t sudo --since -1h -o short-iso | grep COMMAND
sudo find /etc /opt /srv -newermt '-1 hour' -type f \
  -printf '%TT %p\n' | sort

The first command lists what people ran with sudo, the second lists files with their modification time (with fractions of a second) in order. -newermt also takes an exact time: -newermt '2026-10-22 12:00'.

Roll back a config edit

bash

sudo cp -a /etc/nginx/sites-available/site ~/incident/site.bad
sudo cp -a /root/site.bak /etc/nginx/sites-available/site
sudo nginx -t && sudo systemctl reload nginx
echo "$(date -u +%T) config rolled back" >> ~/incident/timeline.txt

The bad version is kept for the review, the checked config is applied with reload without dropping connections (nginx). Configs kept in git are rolled back with git checkout <previous commit> -- <file>. Rolling back a release by switching a symlink — ln.

Rolled back, did not help

The wrong change was rolled back: compare the outage start in the request log and the site check with the times of changes.
The service did not reread the config: it needs reload or restart.
There were several changes: revert them one at a time, checking after each.
The outage began before the change — look for another cause: disk, memory, network, a dependency (database, a neighbouring service).

The restart helped, the outage came back

The restart removed the symptom and the cause stayed: a memory leak, a growing log, exhausted connections. Save systemctl status, the service journal and free -m before the next restart — they show what piles up.

Nothing in the sudo journal

The change did not go through sudo: a root login, a release running as the service user, automation (cron, ansible, package upgrades). Look at file modification times (stat, find -newermt) and at the whole journal around the outage start: journalctl --since '12:00' --until '12:05'.

Times in the logs do not match

journalctl prints time in the machine's time zone, date -u in UTC. On the lab machine they are the same. On a machine with another zone, compare in one: journalctl --utc, TZ=UTC date.

Fixing without a copy

Editing in place, deleting the "broken" file, restarting without saving the journal erases the evidence. The review is then built on guesses, and the same outage repeats. A copy in ~/incident/ takes seconds.

Several edits at once

Three changes in a row with no check in between: the service is back, but which one helped is unclear, and the other two may have added a new breakage. One change — one check — one line in the timeline.

A blameless postmortem

Situation

Task

Constraints

Signing in is needed to start the machine

Confirm the symptom with a series of requests

Recent changes in one list

Roll back a config edit

Rolled back, did not help

The restart helped, the outage came back

Nothing in the sudo journal

Times in the logs do not match

Fixing without a copy

Several edits at once