Linux Labs

SRE practices: SLOs, incidents, redundancy / Stage 2 of 8

SLOs and error budgets

Not running Starting usually ~20 s
Leave the lesson

Step 3 of 6 · on a machine · 8 min

Burn rate

What is left tells you how much budget there is, and the burn rate tells you how fast it melts.

A budget is handy to measure in time too. 0.5 % of 30 days is 3.6 hours: that is how long the shop can be fully down in a month and still meet the SLO. Yesterday's two and a half hours, when nine answers out of ten were bad, ate more than half of that reserve.

The burn rate compares the share of bad answers over a short window — an hour, a day — with the allowed share:

text
burn rate = share of bad answers in the window / (1 − SLO)

At a rate of 1 the budget runs out exactly at the end of the 30 days. At 2 — in 15 days, at 10 — in 3 days. During yesterday's outage 90 % of answers were bad, and the rate was 0.9 / 0.005 = 180: at that rate a month's budget burns in 4 hours.

That is why an SLO alert is set on the burn rate: a rule "rate above 14.4 over the last hour" (an hour takes 2 % of the month's budget) fires in the first minutes of an outage, while plenty of budget is still left. How such rules are written, the lesson on alerts covered.

Exercise · do it

Count yesterday's burn rate

Count the burn rate of the 99.5 % SLO's budget for yesterday and put one number in ~/slo/burn.txt.

Sign in and start The machine is available after signing in

Goals · 0 of 1

  • Put yesterday's burn rate in ~/slo/burn.txt not done yet
Check yourself 1 question

All month the burn rate of the 99.5 % SLO's budget stays at exactly 1. What happens by the end of the 30 days?

  • A month's budget burns in a day at a rate of about 30. At 1 it lasts exactly 30 days.

  • The budget runs out in 15 days at a rate of 2. At 1 it is spent evenly and runs out at the end of the window.

  • The rate is a ratio of two shares: 1 means exactly 0.5 % of answers are bad, however many there are.

At a rate of 1 there are exactly as many bad answers as the SLO allows: the budget runs out at the end of the window, and availability lands exactly on the target.

Step goals

0 of 1
  • Put yesterday's burn rate in ~/slo/burn.txt — not done yet
Ubuntu 24.04

Step goals

0 of 1
  • Put yesterday's burn rate in ~/slo/burn.txt — not done yet
Loading…