Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

Incident response — the first minutes, runbook and timeline

11 min Ubuntu 24.04 · systemd 255 · sudo 1.9

An incident is an outage users can see: restore the service by the runbook first, look for the cause later.

Cheat sheet

7 commands Printable version
  • 1. the symptom as the user sees it
  • curl -s -o /dev/null -w '%{http_code}\n' http://localhost/
  • 2. timeline: one line per action, UTC time
  • mkdir -p ~/incident; cd ~/incident
  • echo "$(date -u +%T) alert 'site 502', confirmed" >> timeline.txt
  • 3. what changed last
  • sudo journalctl -t sudo --since -1h | grep COMMAND who ran what
  • sudo find /etc /opt /srv -newermt '-1 hour' -type f fresh files
  • ls -lt /etc/nginx/sites-enabled/ | head
  • 4. a copy before the rollback — evidence for the review
  • sudo cp -a /etc/default/lab-app ~/incident/
  • 5. check with the same request, write the time down

Overview

An incident is when a service does not do for users what it should: the site answers 502, responses take seconds, the error share eats the budget faster than normal (sli-slo). An alert reports it (alertmanager), or a user does.

Incident response is an order of actions that shortens the outage: confirm the symptom → restore the service → understand the cause → review the outage (postmortem). The first two steps are the on-call's — whoever answers alerts during this shift.

Restore first

Every minute of an outage means errors for users and budget spent. So the first goal is to bring the service back by the fastest reversible means. The cause is looked for once the service works: calmly, from the timeline and the saved copies.

Means When it fits
roll back the last change the outage began right after an edit or a release
restart the service the service crashed or hung, the config did not change
switch to a standby machine one machine out of several is broken
take a server out of the pool one backend of several answers with errors

Reversible means the step can be undone and leaves evidence behind. Before a rollback, copy the changed file to ~/incident/ (cp -a keeps the time and permissions). Before a restart, save systemctl status and the last lines of the service journal: a restart wipes the process state.

What changed last

Most outages start after a change: a config edit, a release, a package upgrade. Where to look:

  • sudo journalctl -t sudo --since -1h — commands run with sudo: who ran what and when;
  • sudo find /etc /opt /srv -newermt '-1 hour' -type f and ls -lt in the config directory — which files changed (find);
  • the release log and the history of the config repository (git);
  • journalctl -u <service> --since -1h — when the service restarted;
  • /var/log/apt/history.log — which packages were upgraded.

A sudo line in the journal (wrapped for width):

text 
Oct 22 12:00:41 node1 sudo[1234]:   deploy : TTY=pts/0 ;
  PWD=/home/deploy ; USER=root ;
  COMMAND=/usr/bin/sed -i s/8000/8001/ /etc/default/lab-app

User deploy changed the port in the service file at 12:00:41. If the site began answering 502 at 12:00:43, this is suspect number one. Compare the time of a change with the start of the symptom: a change seconds or minutes before it is a rollback candidate.

Runbook

A runbook is an instruction for one alert: what it means, which commands to run first, how to restore the service and whom to call. It is written in advance, without haste; during an outage the on-call follows the steps instead of inventing them.

markdown 
# Runbook: <alert> — <what the user sees>
What it means: <which component reported it, what is broken>.
### The first ten minutes
1. Confirm the symptom: <one command>.
2. Start the timeline, a line per action:
   echo "$(date -u +%T) …" >> ~/incident/timeline.txt
3. What changed in the last hour: journalctl -t sudo,
   find -newermt, the release log.
4. Restore reversibly: a copy in ~/incident/, a rollback,
   a restart or a switchover.
5. Check with the same request as in step 1; time into the timeline.
6. Tell <where>; not restored in 15 minutes — call <whom>.
### Later, when the service works
The cause — from the timeline and copies; the review — postmortem.md.

Keep the runbook where it is found during an outage: next to the service (/srv/runbooks/<alert>.md) and linked from the alert text.

Timeline as you go

The timeline is a list of outage events by time, a line per event:

text 
12:00:41 deploy: sed in /etc/default/lab-app — port 8000 → 8001
12:00:43 lab-sitecheck: the site answers 502 — outage starts
12:03:10 on-call put port 8000 back, restarted lab-app
12:03:11 lab-sitecheck: the site answers 200 — outage ends

Time first and in UTC (date -u +%T), then a fact: what was seen or done and by whom, as a role. No judgements in the line. Write it as you go, appending with >>: under stress the times and the order of steps are forgotten, and the review needs both. Whoever joins later reads the timeline and sees what has been done. How to assemble a timeline from the logs after the outage — postmortem.

Roles

When an incident lasts longer than a few minutes or involves more than one person, the roles are split:

Role What it does
incident commander decides what to do, hands out tasks, does not fix things
communications tells users and management what is going on
responders fix things per the commander's tasks and write the timeline

In a small team one on-call does it all, but the decision "tell people" is made apart from "fix it": users and colleagues need to know about the outage while it is being fixed.

When the incident is over

The service answers the same request that confirmed the symptom, and the time is in the timeline. If the fix is temporary (a rollback, a manual restart), a task for the real fix is opened. Next comes the outage review (postmortem).

Recipes

3

Confirm the symptom with a series of requests

bash 
for i in $(seq 20); do
  curl -s -o /dev/null -w '%{http_code}\n' http://localhost/
done | sort | uniq -c

A single curl may hit a lucky moment: with one dead backend of two, only part of the requests fails. A series shows the error share.

Recent changes in one list

bash 
sudo journalctl -t sudo --since -1h -o short-iso | grep COMMAND
sudo find /etc /opt /srv -newermt '-1 hour' -type f \
  -printf '%TT %p\n' | sort

The first command lists what people ran with sudo, the second lists files with their modification time (with fractions of a second) in order. -newermt also takes an exact time: -newermt '2026-10-22 12:00'.

Roll back a config edit

bash 
sudo cp -a /etc/nginx/sites-available/site ~/incident/site.bad
sudo cp -a /root/site.bak /etc/nginx/sites-available/site
sudo nginx -t && sudo systemctl reload nginx
echo "$(date -u +%T) config rolled back" >> ~/incident/timeline.txt

The bad version is kept for the review, the checked config is applied with reload without dropping connections (nginx). Configs kept in git are rolled back with git checkout <previous commit> -- <file>. Rolling back a release by switching a symlink — ln.

Troubleshooting

4

Rolled back, did not help

  • The wrong change was rolled back: compare the outage start in the request log and the site check with the times of changes.
  • The service did not reread the config: it needs reload or restart.
  • There were several changes: revert them one at a time, checking after each.
  • The outage began before the change — look for another cause: disk, memory, network, a dependency (database, a neighbouring service).

The restart helped, the outage came back

The restart removed the symptom and the cause stayed: a memory leak, a growing log, exhausted connections. Save systemctl status, the service journal and free -m before the next restart — they show what piles up.

Nothing in the sudo journal

The change did not go through sudo: a root login, a release running as the service user, automation (cron, ansible, package upgrades). Look at file modification times (stat, find -newermt) and at the whole journal around the outage start: journalctl --since '12:00' --until '12:05'.

Times in the logs do not match

journalctl prints time in the machine's time zone, date -u in UTC. On the lab machine they are the same. On a machine with another zone, compare in one: journalctl --utc, TZ=UTC date.

Danger zone

2

Fixing without a copy

Editing in place, deleting the "broken" file, restarting without saving the journal erases the evidence. The review is then built on guesses, and the same outage repeats. A copy in ~/incident/ takes seconds.

Several edits at once

Three changes in a row with no check in between: the service is back, but which one helped is unclear, and the other two may have added a new breakage. One change — one check — one line in the timeline.

Terms

3
incident
An outage users can see: the site answers with errors, responses take seconds, the error budget melts faster than normal. Restore the service first, look for the cause later. Also appears in: Postmortem, journalctl, Load balancing
runbook
An instruction for one alert: what it means, which commands to run first, how to restore the service and whom to call. Written in advance, followed during an outage. Also appears in: Postmortem
on-call
Whoever answers alerts during this shift: confirms the outage, restores the service by the runbook and keeps the timeline. Also appears in: Alertmanager: who gets alerts and how often, Postmortem, SLI and SLO

See also

  • postmortem — the blameless outage review.
  • sli-slo — the error budget and its burn rate.
  • journalctl — --since, -t, -u.
  • alertmanager — where an alert comes from.
  • find — searching by modification time.
  • git — config history.
  • ln — rolling back a release with a symlink.
  • systemctl — status, reload, restart.

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Tools on this topic

Topics
Loading…