SLOs and error budgets

The team's target is an SLO, and the reserve between the target and "not a single error" is the error budget.

The availability from the lesson on the share of good requests is an SLI, an indicator. An SLO (Service Level Objective) says what it should be: the SLI, the target and the window together. "Availability of at least 99.5 % over 30 days" is the shop's SLO. The window is rolling — always the last 30 days from the current moment, so yesterday's outage will stay in the count for 29 more days. A contract with a customer where missing the target costs money is called an SLA; its target is usually softer than the internal SLO, so the team notices a problem before the customer does.

With a 99.5 % target, up to 0.5 % of answers may be bad — and the SLO is still met. Those 0.5 % are the error budget:

budget = (1 − 0.995) × all requests in the window
left   = budget − bad answers in the window

For example, with 400,000 requests over 30 days the budget is 2000 bad answers. If 1400 have already happened, 600 are left. While more than zero is left, the SLO holds, and the budget can be spent on risk: rolling out new versions, changing the configuration. Once it is gone, by the team's agreement rollouts are frozen and the time goes into reliability.

The logs for 30 days are kept in several files, the way logrotate usually lays them out: today's access.log, yesterday's access.log.1 and compressed older ones. zcat -f prints compressed and plain files one after another, so one pipeline counts over all of them at once.