SLI: the measured share of good requests over a window. SLO: its target. The gap to 100 % is the error budget.
awk -v f="$(date -d '1 hour ago' +%s)" '$NF >= f' access.log > w.log
awk '$9 >= 500 {b++} END {printf "%.2f\n", 100*(NR-b)/NR}' w.log
awk '{print $(NF-1)}' w.log | sort -n > /tmp/t
k=$(( ($(wc -l < /tmp/t) * 95 + 99) / 100 )); sed -n "${k}p" /tmp/t
awk 'BEGIN {print 400000 * (1 - 0.995)}'
2000 bad allowed
awk 'BEGIN {print 0.07 / (1 - 0.995)}'
14
ping works, curl / returns 200, and users still see errors: some
requests fail or take seconds. An "is the server alive" check misses
this. The request log shows it once you count how many requests went
well and how fast.
An SLI (service level indicator) is a measured share of good events: good responses out of all, fast responses out of all. An SLO (service level objective) says which SLI counts as normal over a time window: "99.5 % of responses without a 5xx over 30 days". The gap between the target and 100 % is the error budget: how many bad requests the target still allows.
SLI = good ÷ all. What counts as good is decided in advance from what the user sees, and written down next to the SLO:
| Response | For an availability SLI |
|---|---|
| 2xx, 3xx | good |
| 4xx | usually good: the client erred, the service answered correctly |
| 5xx | bad: the server failed |
| 499 in nginx | the client closed the connection before the response |
Only nginx writes 499. If clients leave because the server is slow, 499 is a bad response too; if it is network drops on the client side, it is left out of the count.
Two common SLIs:
Count them in the log of whatever stands first on the request path: nginx or the load balancer in front of the application. It also sees the requests that never reached the application: 502 and 504 are nginx's own responses and are missing from the application log (nginx).
An SLI is always counted over a period: an hour, a day, 30 days. One hundred errors are 7 % of an hour with 1400 requests and 0.01 % of a month with the same traffic. A window is set by two timestamps: the start is included, the end is not. That way adjacent windows never count the same line twice.
Time in the combined format is $time_local, like
[22/Oct/2026:10:03:58 +0000]: the month is a word, so these stamps
cannot be compared as text. For windows, $msec is appended to the
log line — unix time in seconds with milliseconds, a number:
Here 0.004 is $request_time, the response time in seconds, and
1792677901.123 is $msec. awk field numbers count from the end of
the line, $(NF-1) and $NF: a user agent with spaces shifts the
numbers from the start. How to set up such a format — nginx.
A window is a numeric comparison: $NF >= start && $NF < end, with
the start and end from date -d '…' +%s.
The mean response time hides the tail. 95 responses of 0.05 s and 5 of 10 s average 0.55 s, while every twentieth user waited 10 seconds.
The percentile p95 is the time 95 % of responses fit into. p50 is the median, p99 marks the worst one percent. To compute it, sort the times ascending and take the value at position ⌈0.95 × N⌉, rounded up (the nearest-rank method). Out of 100 values it is the 95th, out of 1400 the 1330th, out of 7 the 7th.
Other tools interpolate between neighbouring values: histogram_quantile
in promql does. On a small sample the results differ, so the method
is written down together with the SLO.
An SLO is a target for an SLI over a window: "99.5 % of responses without a 5xx over 30 days", "95 % of responses faster than 1 s over a day". The target is set below 100 %: every release, restart and network glitch spends part of it.
An SLA (service level agreement) is a contract with a customer with consequences for a breach: refunds, penalties. The SLA number is looser than the team's own SLO, so the team learns about a problem before the contract is breached.
How much downtime a target allows:
| SLO | Per 30 days | Per day |
|---|---|---|
| 99 % | 7 h 12 min | 14 min 24 s |
| 99.5 % | 3 h 36 min | 7 min 12 s |
| 99.9 % | 43 min 12 s | 1 min 26 s |
| 99.95 % | 21 min 36 s | 43 s |
| 99.99 % | 4 min 19 s | 9 s |
Downtime minutes are a simplification: they hold if an outage fails every request and the load is even. From a log, the budget is counted in requests.
With an SLO of 99.5 % and 400,000 requests over 30 days, 2000 bad
ones are allowed. Yesterday's outage produced 1400 5xx responses: 70 %
is spent, 600 requests are left, 30 % of the budget. How many requests
will arrive before the window ends is unknown yet: use the expected
number from past windows, or compare shares — the share of bad
requests in the window against 1 − SLO.
The budget is a rule for decisions. Budget left — releases and risky work can go ahead. Budget gone — risky changes wait and the team fixes whatever burns it. The rule is written down in advance (an error budget policy) and is not debated during an outage.
The burn rate is how many times the share of bad requests exceeds
the allowed one: the share of bad requests over the latest window ÷
(1 − SLO).
0.07 ÷ 0.005): the
30-day budget is gone in two days, one hour of the outage eats 2 %.Alerts are set on the burn rate. A fast one — above 14.4 over the last hour: 2 % of the budget went in an hour, page the on-call. A slow one — above 6 over 6 hours: 5 % went, a task for the working day. How to write the rule — prometheus, the 5xx share in PromQL — promql.
A log with $msec at the end of the line:
Rotated parts are read together with the current file: zcat -f
access.log.2.gz access.log.1 access.log | awk … (-f passes
uncompressed files through as is). Give the window in the same time
zone the log uses; on the lab machine that is UTC (+0000).
Without $msec, in plain combined, mktime turns the time into a
number (on Ubuntu 24.04 awk is GNU awk):
Prints the number of requests in the window, the number of 5xx
responses and availability as a percentage with two decimals. In
combined the status code is field 9. Check the field with
awk '{print $9}' w.log | sort | uniq -c: it should list only codes.
sort -n is required: without it 10.002 sorts before 2.001
(sort). An empty window gives n=0 and an empty answer. The
share of fast responses uses the same field:
awk '$(NF-1) <= 0.3 {f++} END {print f/NR}' w.log.
bad is the 5xx count for the window from the availability recipe,
total is all requests in the window or their expected number.
curl from another machine.access.log.1 after the nightly rotation (logrotate).-n — string order instead of numeric order;$upstream_response_time writes -
for responses nginx served itself and 0.000, 0.002 with a comma
when nginx retried the request on another server (nginx).head -n 1 and tail -n 1 of the log show which period it covers.
Common causes: the window is in a different time zone from the log;
combined time compared as text while the month is a word ($4
breaks where months meet); the log is already rotated and the lines
are in access.log.1 or .2.gz.
awk '{print $9}' | sort | uniq -c shows "GET, sizes or words —
the request line has spaces or scanner garbage in it ("-", binary
bytes). Taking the code by quotes is more robust: the first word after
the request line.
A 100 % target leaves no budget. Any restart, release or network glitch breaks it, and the budget rule then forbids changes forever. A target higher than that of what the service depends on (network, hosting, database) cannot be met either.
log_format and $request_time.-n.Found an inaccuracy? Report an error in the article