SRE practices: SLOs, incidents, redundancy / Final lab

On-call shift

~50 min

Not running Starting usually ~20 s

Situation

You are taking over the on-call shift. The order site shop.lab.test runs on two machines. Users open it by the name shop.lab.test — a floating address (VIP) kept by keepalived: node1 is the primary machine, node2 is the standby. node1 runs the nginx balancer and Prometheus, which scrapes both machines. node2 runs two application backends (the lab-shop@8081 and lab-shop@8082 services), PostgreSQL with the shop database and a separate /var/log/shop volume for the application logs.

Last night the previous shift changed things on both machines. A few minutes ago the complaints started: the site shows no orders. The previous on-call engineer managed to pass on two more things: node2 is running out of space, and the alert rule that was supposed to warn about it never showed up in Prometheus. The site runbook is /srv/runbooks/shop-down.md on node1.

node1 is your terminal. Run commands on node2 over SSH: ssh node2 'command', or open a shell there — ssh node2, and exit to come back. sudo works without a password on both machines. There is no browser to the machines: look at the site and at Prometheus from the terminal — curl, jq, promtool.

Task

Bring the site back to the users first, then deal with the rest.

http://shop.lab.test/ on the floating address returns the order list from the database — for every request. As intended, node1 holds the address, and keepalived runs on both machines.
Only the application (the shop role) connects to the shop database. That requirement was introduced yesterday and it stays: the report role is not let into the shop database.
The /var/log/shop volume on node2 is less than 80 % full, and the application log grows the same way as before last night's changes.
The LogVolumeAlmostFull rule from /etc/prometheus/rules/ is loaded into Prometheus.
~/postmortem.md on node1 holds the incident review with three sections: - ## Timeline — at least three events, one line each: HH:MM:SS what happened (machine time, UTC, as in the logs). One of them is the moment the site stopped returning the orders; - ## Root cause — what exactly broke: every fault you found; - ## Action items — what to do so that it does not happen again, at least one item.

"Check" itself sends 20 requests to the site on the floating address and 20 to each backend on node2 (to see how much the application log grows). The check takes about 10 seconds.

Constraints

Everything must survive a reboot of both machines: services are enabled, settings live in files, and the running services have re-read those files after your edits.
Do not move the floating address to node2 and do not turn keepalived off: the standby is there for a real failure.
/var/log/shop stays a separate volume from /etc/fstab.

There is no internet in the lab — only the network between node1 and node2, everything you need is installed. The machines' logs use UTC.

Lab goals

0 of 5

http://shop.lab.test/ on the floating address returns the orders from the database for each of 20 requests; node1 holds the address, keepalived runs on both machines — not checked yet
The application connects to the shop database while the report role is refused — the database is the application's only; pg_hba.conf has no errors and PostgreSQL has reloaded it — not checked yet
The /var/log/shop volume on node2 is less than 80 % full, the application log grows as it did before the changes, the backends run with the settings from /etc/default/shop — not checked yet
The LogVolumeAlmostFull rule is loaded into Prometheus from its file, and the last configuration reload succeeded — not checked yet
~/postmortem.md on node1 has a Timeline with the outage start time from the logs, a Root cause naming every fault you found, and Action items — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Services and systemd · picked for this lab

nginx — web server and reverse proxy

Cheat sheet

$_ — insert into the terminal

sudo nginx -t check the config (sudo only)
sudo systemctl reload nginx apply, keep connections
sudo nginx -T | grep -n server_name which sites nginx sees
ls -l /etc/nginx/sites-enabled/ enabled sites
curl -i -H 'Host: docs.lab.test' http://127.0.0.1/ who answers
sudo tail -n 5 /var/log/nginx/error.log why 404, 403, 502
sudo tail -n 5 /var/log/nginx/access.log what was asked, the code
sudo ss -tlnp | grep nginx which ports it listens on

The Debian and Ubuntu layout:

/etc/nginx/nginx.conf — common settings, with an include of sites at the bottom;
/etc/nginx/sites-available/ — site files;
/etc/nginx/sites-enabled/ — links to enabled sites; nginx reads only these;
/var/log/nginx/access.log and error.log — the logs.

Enable a site

bash

sudo ln -s /etc/nginx/sites-available/docs /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

A file in sites-available without a link in sites-enabled is invisible to nginx. To disable a site, remove the link; the file stays.

nginx -t checks the whole config and names the file and line of an error. reload re-reads the config without dropping connections; with an error in the config nginx keeps the old one. systemctl reload returns at once, while the new worker processes take connections a moment later: a curl on the same line may still see the old state.

Logs

access.log has a line for every request, error.log says why a request failed and what broke at start. The default access.log format is combined:

text

127.0.0.1 - - [20/Oct/2026:10:03:58 +0000] "POST / HTTP/1.1" 405 166 "-"

Field by field: $remote_addr (client address), -, $remote_user, [$time_local], "$request" (method, path, version), $status (code), $body_bytes_sent (body size), "$http_referer", "$http_user_agent" — the user agent "curl/8.5.0" did not fit in the example. log_format defines your own format — in a site file outside the server block:

nginx

log_format timed '$remote_addr [$time_local] "$request" '
                 '$status $body_bytes_sent $request_time';
access_log /var/log/nginx/app.log timed;

nginx passes on the code that the application behind the proxy returned: a 500 from the application is in access.log, while error.log has not a line about it, since nginx did its part. Look for the cause of such an error in the application's own journal (journalctl). error.log gets what nginx itself failed at: 502 and 504 below.

$request_time is how many seconds nginx spent on the request. Files with .1 and .2.gz next to a log are older ones that logrotate moves aside and compresses on schedule (/etc/logrotate.d/nginx).

log_format works only in the http context: in a site file outside the server block or in /etc/nginx/conf.d/<name>.conf. A format name is declared once for the whole config — nginx -t rejects a second log_format with the same name. For an SLI (sli-slo) the response time and the request time in unix time ($msec) are appended to the line:

nginx

log_format sre '$remote_addr - $remote_user [$time_local] "$request" '
               '$status $body_bytes_sent "$http_referer" '
               '"$http_user_agent" $request_time $msec';

Behind a balancer (nginx) the backend fields help:

Variable	What it writes
`$upstream_addr`	the address of the copy that answered
`$upstream_status`	the copy's response code; a refused connection is `502`
`$upstream_response_time`	how many seconds the copy took

If nginx retried a request on another copy, these fields are lists separated by a comma and a space: 127.0.0.1:8082, 127.0.0.1:8081 and 502, 200. For responses nginx served itself, without a backend, the fields hold -; when no copy is alive, the upstream group name. Put such fields in quotes in the format, or awk field numbers shift.

Proxy: path and headers

Whether there is a URI after the address in proxy_pass decides which path the application receives:

`location /api/` and `proxy_pass`	Request `/api/x` goes out as
`http://127.0.0.1:3003`	`/api/x` — path unchanged
`http://127.0.0.1:3003/`	`/x` — `/api/` replaced with `/`

By default the application sees Host: 127.0.0.1:3003 — the address from proxy_pass. The site name and the client address are passed in headers:

nginx

proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

Balancing: upstream

Several copies of an application behind one address are an upstream block in the http context (in a site file outside server), and proxy_pass points at its name. Why and how it works — load-balancing.

nginx

upstream app {
    server 127.0.0.1:8081 max_fails=2 fail_timeout=10s;
    server 127.0.0.1:8082 max_fails=2 fail_timeout=10s;
}
server {
    listen 80;
    location / {
        proxy_pass http://app;
        proxy_next_upstream error timeout http_500 http_502 http_503;
        proxy_connect_timeout 2s;
        proxy_read_timeout 3s;
    }
}

Parameters of a server line in upstream:

Parameter	Default	What it does
`max_fails=N`	1	failures within `fail_timeout` that switch the copy off; 0 — do not count
`fail_timeout=T`	10s	the failure counting window and how long the copy stays off
`weight=N`	1	the copy's share of requests
`backup`	—	the copy gets requests only when the others are off
`down`	—	the copy is switched off by hand

Directives in location next to proxy_pass:

Directive	Default	What it does
`proxy_next_upstream`	`error timeout`	what counts as a failure and is retried on another copy; `http_500` and others add codes
`proxy_next_upstream_tries`	0 (no limit)	how many copies to try per request
`proxy_connect_timeout`	60s	how long to wait for a connection to the copy
`proxy_read_timeout`	60s	how long to wait for the copy's answer

What the machine shows (nginx 1.24, one copy of two not listening): the client gets 200 — nginx retries the request on the live copy; error.log gets one connect() failed (111: Connection refused) line per fail_timeout while the copy is off. A copy that answers 500 itself gives the client 500 every other time until proxy_next_upstream has http_500. On a one-core machine nginx runs one worker process, and the round robin is strict: 8081, 8082, 8081…

HTTPS and the redirect from HTTP

nginx

server {
    listen 443 ssl;
    server_name shop.lab.test;
    ssl_certificate     /etc/nginx/tls/fullchain.pem;
    ssl_certificate_key /etc/nginx/tls/shop.key;
    root /var/www/shop;
}
server {
    listen 80;
    server_name shop.lab.test;
    return 301 https://$host$request_uri;
}

ssl_certificate is the chain file: the site certificate first, the intermediate after it. $request_uri is the path with the query after ?: http://shop.lab.test/x?y=1 redirects to https://shop.lab.test/x?y=1. Certificates and how they are checked — openssl-tls.

Who will answer a name

bash

curl -i -H 'Host: docs.lab.test' http://127.0.0.1/
sudo nginx -T | grep -n 'server_name\|listen'

The first command shows the answer as a client sees it, the second which listen and server_name nginx actually read, in file order.

Find where the proxy breaks

bash

curl -s -o /dev/null -w '%{http_code}\n' http://app.lab.test/
sudo tail -n 1 /var/log/nginx/error.log
curl -i http://127.0.0.1:3000/        # the application directly

The error.log line names the upstream — where nginx went. If the application answers directly, the problem is in proxy_pass or permissions; if not, it is in the application.

Someone else's site answers

The request's Host matched no server_name, so the default server answered. Check the name in server_name and that the site's link exists in sites-enabled: sudo nginx -T | grep server_name. The name is right but default answers — compare listen: a block bound to the machine's address does not see requests to another address, such as the VIP (nginx).

404 Not Found

error.log has the path nginx built: open() "/var/www/portal/nope.html" failed (2: No such file or directory). Compare it with the real path of the file: usually root or alias is wrong.

403 Forbidden

(13: Permission denied) — www-data cannot read the file or enter a directory on the path; namei -l shows permissions along the whole path. directory index of "…" is forbidden — the directory has no file from index.

502 Bad Gateway

nginx did not open a connection to the backend. The error.log line names the reason:

In error.log	Meaning
`(111: Connection refused)`	nothing listens on the address or socket
`(13: Permission denied)`	`www-data` has no right to the socket
`(2: No such file or directory)`	no socket file at that path
`upstream prematurely closed connection`	there was a connection, but the application closed it before finishing the answer: it crashed on this request

In full: connect() failed (111: Connection refused) while connecting to upstream, … upstream: "http://127.0.0.1:3001/". For a socket — connect() to unix:/run/app/app.sock failed (13: Permission denied). nginx logs errors 13 and 2 at level [crit] and 111 at [error]: a search for [error] alone misses them. A socket file left behind after the application stopped also gives 111. Socket permissions and paths — unix-sockets.

504 Gateway Time-out

There is a connection but no answer: upstream timed out (110: Connection timed out) while reading response header from upstream. The application hangs or answers slower than proxy_read_timeout (60 s by default).

no live upstreams

no live upstreams while connecting to upstream, … upstream: "http://app/" and 502 for the client: every copy of the upstream group is switched off after failures (max_fails), there is nobody to try. $upstream_addr holds the group name instead of an address. Fix at least one copy; after fail_timeout nginx tries them again (load-balancing).

duplicate "log_format" name

[emerg] duplicate "log_format" name "sre" in … — a format with this name is already declared in another file. Find both: sudo nginx -T | grep -n 'log_format'. Keep one; sites write with it via access_log … sre;.

Cannot assign requested address

bind() to 10.0.0.100:80 failed (99: Cannot assign requested address) — listen has an address the machine does not have right now: the floating address lives on the other machine (keepalived). Listen on listen 80 and tell sites apart by server_name, or allow binding to a foreign address: sysctl net.ipv4.ip_nonlocal_bind=1.

nginx -t: test failed

unexpected "}" in …:5 — the line before } has no ;;
a duplicate default server for 0.0.0.0:80 — default_server is set in two blocks on one port;
open() "/run/nginx.pid" failed (13: Permission denied) — nginx -t was run without sudo.

conflicting server name

[warn] conflicting server name "docs.lab.test" on 0.0.0.0:80, ignored — one name sits in two blocks on one port. The test still passes, but requests for that name go to the block loaded first, and the second never gets them. Find both:

bash

grep -n 'docs.lab.test' /etc/nginx/sites-enabled/*

grep -r on the directory does not work here: it skips the links in sites-enabled.

key values mismatch

SSL_CTX_use_PrivateKey(…) failed (… key values mismatch) — the first certificate in the ssl_certificate file does not match the key. Either the key is wrong, or the chain file has the intermediate first. The site certificate goes first.

The new certificate is not served

nginx reads certificates at start and on reload. After replacing the file — sudo nginx -t && sudo systemctl reload nginx, then check a second later.

The service does not start: port in use

sudo ss -tlnp | grep ':80 ' shows who holds the port. Stop it or change listen.

A new listen did not apply

Changing the address on the same port (listen 80 → listen 127.0.0.1:80) does not work through reload: systemctl reload returns 0, nginx stays on the old sockets, and error.log says bind() to 127.0.0.1:80 failed (98: Address already in use). Only sudo systemctl restart nginx applies such a change; sudo ss -tlnp | grep nginx shows what really listens.

restart with an error in the config

On reload with an error the old config keeps working; restart stops nginx and cannot start it again: every site on the machine is down. Before applying anything — sudo nginx -t.

On-call shift

Situation

Task

Constraints

Signing in is needed to start the machine

Who will answer a name

Find where the proxy breaks

Someone else's site answers

404 Not Found

403 Forbidden

502 Bad Gateway

504 Gateway Time-out

no live upstreams

duplicate "log_format" name

Cannot assign requested address

nginx -t: test failed

conflicting server name

key values mismatch

The new certificate is not served

The service does not start: port in use

A new listen did not apply

restart with an error in the config