Linux Labs

SRE practices: SLOs, incidents, redundancy / Stage 4 of 8

A runbook and the first ten minutes

Not running Starting usually ~20 s
Leave the lesson

Step 5 of 7 · on a machine · 6 min

Where else changes hide

Not every change goes through sudo — then you follow its traces.

The sudo journal only sees people's commands. A release is often made by a service or a timer running as root, and the sudo journal does not have it. Then three sources help:

  • file modification times: ls -lt puts the freshest file of a directory first — look at the service's config directories: /etc/default, /etc/nginx/sites-available. Wider — sudo find /etc -newermt '-1 hour' -type f (find); on a fresh machine this list also holds files the system writes at boot — the ssh keys, machine-id;
  • the machine's journal for the same minutes — sudo journalctl --since -15min: which services started and what they wrote;
  • the release log and the config history: if the configs live in git, the repository history shows the edit (the lesson on git on a server).

Tags of one field add up in the journal: -t sudo -t lab-sitecheck gives the records of both. But -t together with -u means “both at once”, and such a query returns -- No entries --.

Exercise · do it

Another 502

The site answers 502 again, but this time the sudo journal is empty. Find what changed last, save the changed file as a copy in ~/incident, roll the change back and keep a timeline as you go in ~/incident/timeline.txt — with the time the site began to answer 502.

Sign in and start The machine is available after signing in

Goals · 0 of 3

  • Save the changed file in ~/incident not done yet
  • Roll the change back and bring the site back not done yet
  • Write the timeline in ~/incident/timeline.txt not done yet

Step goals

0 of 3
  • Save the changed file in ~/incident — not done yet
  • Roll the change back and bring the site back — not done yet
  • Write the timeline in ~/incident/timeline.txt — not done yet
Ubuntu 24.04

Step goals

0 of 3
  • Save the changed file in ~/incident — not done yet
  • Roll the change back and bring the site back — not done yet
  • Write the timeline in ~/incident/timeline.txt — not done yet
Loading…