Linux Labs

SRE practices: SLOs, incidents, redundancy / Stage 4 of 8

A runbook and the first ten minutes

Not running Starting usually ~20 s
Leave the lesson

Step 1 of 7 · on a machine · 4 min

The site answers 502

What you will learn

  • Take the first steps of a runbook — confirm the symptom and find what changed on the machine last.
  • Bring a service back by rolling back the last change, keeping the changed file for the review.
  • Keep an incident timeline as you go — “time event” lines by the machine clock.

You are on call. An alert comes in: the site answers 502. The team has a runbook for this alert — a short instruction with commands. Its first steps: make sure the site really does not work, and look at what changed on the machine in the last hour.

Exercise · look

Take the runbook's first steps

An alert came in: the site answers 502. By the runbook, first make sure the site really answers 502, then look at the journal of sudo commands and site checks for the last hour.

First, predict

The alert came just now. What will the journal show before the FIRING line?

Now start the exercise, run the command and compare your prediction with the output.

Sign in and start The machine is available after signing in
Check yourself 1 question

lab-app is active, and the site answers 502. What in the journal is connected with the outage?

  • lab-sitecheck started before the change and stayed silent until then: the site answered 200. FIRING appeared only after lab-app was restarted.

  • active means the lab-app process runs, but says nothing about where it waits for requests. A second before FIRING deploy changed the port and restarted the service.

  • Your command is logged after FIRING: the outage began before you opened the journal.

The site began to answer 502 a second after the port change and the lab-app restart.

Ubuntu 24.04
Loading…