SRE practices: SLOs, incidents, redundancy

Availability and latency from a log, SLOs and error budgets; the first minutes of an incident by a runbook, rolling back a release and a blameless postmortem; a load balancer with health checks and a keepalived floating address — up to an on-call shift on two machines.

8 stages5 labs~5 hours medium

Progress Stages: 0 of 8

The track has not been started. The first stage is «SLI: the share of good requests», ~30 min.

Start stage 1

Module 1

SLIs, SLOs and error budgets

Availability and p95 latency from a log over a time window; an SLO, its error budget and the burn rate.

3 stages · ~2 hours

Stage 1 of 8
1 · SLI: the share of good requestsSLI: the share of good requests
6 steps · ~30 min 6 steps · ~30 min
1. All green from the outside exercises 0 of 1 on a machine 4 min
2. 2 Good and bad answers on a machine 6 min
3. 3 A time window on a machine 7 min
4. 4 Answer time: p95 on a machine 7 min
5. 5 Availability for those who complain on a machine 4 min
6. 6 Summary summary 2 min
Start stage 1
2
Stage 2 of 8
2 · SLOs and error budgetsSLOs and error budgets
6 steps · 1 lab · ~1 hour 6 steps · 1 lab · ~1 hour
1. How much more can go wrong exercises 0 of 1 on a machine 6 min
2. 2 SLOs and error budgets on a machine 8 min
3. 3 Burn rate on a machine 8 min
4. 4 A stricter target on a machine 5 min
5. 5 Availability and p95 from the log lab 30 min
6. 6 Summary summary 3 min
Start stage 2
3 · Module test 10 questions · pass mark 70%

Module 2

Incident response

Restore the service first, look for the cause later: a runbook, what changed last, a timeline; a blameless postmortem and action items.

3 stages · ~2 hours

4
Stage 4 of 8
4 · A runbook and the first ten minutesA runbook and the first ten minutes
7 steps · 1 lab · ~50 min 7 steps · 1 lab · ~50 min
1. The site answers 502 exercises 0 of 1 on a machine 4 min
2. 2 The runbook theory 7 min
3. 3 Bring the site back first on a machine 6 min
4. 4 A timeline as you go theory 5 min
5. 5 Where else changes hide on a machine 6 min
6. 6 The site broke after a release lab 20 min
7. 7 Summary summary 2 min
Start stage 4
5
Stage 5 of 8
5 · A blameless postmortemA blameless postmortem
6 steps · 1 lab · ~1 hour 6 steps · 1 lab · ~1 hour
1. No timeline exercises 0 of 1 on a machine 5 min
2. 2 Three sources on a machine 8 min
3. 3 Without looking for someone to blame theory 6 min
4. 4 Action items on a machine 9 min
5. 5 Event intake has stalled and restarts are off-limits lab 30 min
6. 6 Summary summary 2 min
Start stage 5
6 · Module test 10 questions · pass mark 70%

Module 3

Redundancy

Several backends behind an nginx upstream, passive health checks, retrying on another backend and timeouts; a keepalived floating address that moves to the standby machine.

2 stages · ~2 hours

7
Stage 7 of 8
7 · A load balancer and health checksA load balancer and health checks
10 steps · 2 labs · ~2 hours 10 steps · 2 labs · ~2 hours
1. Every other answer is a 500 exercises 0 of 1 on a machine 4 min
2. 2 Copies behind one address on a machine 3 min
3. 3 The copy is stopped on a machine 3 min
4. 4 Passive health checks theory 6 min
5. 5 A slow copy on a machine 5 min
6. 6 A floating address on a machine 4 min
7. 7 VRRP and keepalived theory 4 min
8. 8 Every other request hangs lab 30 min
9. 9 The site address does not move to the standby machine lab 30 min
10. 10 Summary summary 1 min
Start stage 7
8 · Module test 9 questions · pass mark 70%

Final lab
On-call shift ~50 min
Final test 15 questions · pass mark 80%

SLIs, SLOs and error budgets

1 · SLI: the share of good requestsSLI: the share of good requests

2 · SLOs and error budgetsSLOs and error budgets

Incident response

4 · A runbook and the first ten minutesA runbook and the first ten minutes

5 · A blameless postmortemA blameless postmortem

Redundancy

7 · A load balancer and health checksA load balancer and health checks

Already know this?