SRE practices: SLOs, incidents, redundancy / Stage 7 of 8 · step 9 of 10

The site address does not move to the standby machine

~30 min

Not running Starting usually ~20 s

Situation

The shop site shop.lab.test lives on a floating address — a VIP: the name points to the .100 address in the machines' network (a line in /etc/hosts on both machines, with the short name vip there too). keepalived holds the address: normally it is on node1, and if node1 goes down, node2 should pick it up. Both machines run nginx with the same page, only each one is signed with its own name: shop: served by node1 or shop: served by node2.

During a drill keepalived was stopped on node1 — this is how a machine going down looks to its neighbour. The address left node1, and the site vanished: nothing moved to node2.

node1 is your terminal. Run commands on node2 over SSH: ssh node2 'command', or log in with ssh node2 and come back with exit. sudo works without a password on both machines. Where the address is now and who answers on it:

ip -br a; ssh node2 ip -br a
curl -s http://shop.lab.test/

Task

Normally the floating address is on node1, and only there; the site http://shop.lab.test/ answers with node1's page.
When keepalived stops on node1, within 10 seconds the address is on node2, and the site answers on it with node2's page.
When keepalived is running on node1 again, within 10 seconds the address comes back to node1.

"Check" stops keepalived on node1 on its own, looks at where the address went and who answers on it, then starts keepalived again and waits for the address to come back. The check takes up to 40 seconds.

Constraints

All of it must survive a reboot of both machines: the services are enabled, the settings are in files, and the running services have read those files after the change.
Only keepalived manages the floating address: don't assign it by hand (ip addr add) and don't put it into the network settings.
node1 stays the main machine: don't change its keepalived settings and don't hand the address to node2 for good.

There is no internet in the lab — only the network between node1 and node2, everything you need is installed. Run the drill yourself: sudo systemctl stop keepalived on node1, look at where the address is and what the site answers, then sudo systemctl start keepalived.

Lab goals

0 of 4

Normally the floating address is on node1 and only there, and http://shop.lab.test/ answers with node1's page — not checked yet
When keepalived stops on node1, within 10 s the address is on node2 and the site answers on it with node2's page — not checked yet
When keepalived is running on node1 again, within 10 s the address comes back to node1 — not checked yet
keepalived and nginx on both machines are enabled and run with the settings from their files — all of it survives a reboot — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Network · picked for this lab

Load balancing — copies behind one address and health checks

Cheat sheet

$_ — insert into the terminal

sudo nginx -T | grep -A4 '^upstream' copies and their parameters
tail -n 5 /var/log/nginx/access.log $upstream_addr: who answered
sudo tail -n 3 /var/log/nginx/error.log why a copy dropped out
curl -s -o /dev/null -w '%{http_code}\n' 127.0.0.1:8082 copy alone
curl -s -o /dev/null -w '%{http_code} %header{x-upstream}\n' localhost
sudo nginx -t && sudo systemctl reload nginx apply without downtime
ip -br a VIP is the second address

Take a copy out of rotation for maintenance

nginx

server 127.0.0.1:8082 down;

sudo nginx -t && sudo systemctl reload nginx — the copy gets no new requests, the current ones finish. To return it, remove down and reload again. reload keeps connections, restart drops them (nginx).

How many requests each copy got

bash

grep -o 'upstream=[^ ]*' /var/log/nginx/lb-access.log | sort | uniq -c

The upstream= field is the label of $upstream_addr in the log format; in your own format look for the same field. A skew means weight, ip_hash, a copy switched off after failures, or one marked backup.

Find the sick copy

bash

for p in 8081 8082; do
  curl -s -o /dev/null -w "$p %{http_code} %{time_total}\n" \
    http://127.0.0.1:$p/
done

A request to each copy directly, bypassing the balancer: the code and the response time. For a copy on another machine, use its address instead of 127.0.0.1.

Every other request fails

One copy of two answers 500 or 503 by itself. For nginx such an answer is a success: the copy stays in rotation, and every other time the user gets an error. Fix the copy, take it out (down) or add the codes to proxy_next_upstream (load-balancing). Which copy answers with errors — $upstream_addr on the log lines with 500.

A request hangs for a minute, then answers

A copy is unreachable over the network or hung, and nginx waits proxy_connect_timeout or proxy_read_timeout — 60 s each. In error.log — upstream timed out (110: Connection timed out). Set 2–3 seconds (load-balancing).

no live upstreams

text

[error] … no live upstreams while connecting to upstream,
  … upstream: "http://app/"

Every copy of the group is switched off after failures — the client gets 502, and $upstream_addr holds the group name (app) instead of an address. Fix at least one copy: after fail_timeout nginx tries them again.

The copy is fixed but gets no requests

fail_timeout has not passed since the last failure;
the line still has down or backup;
the copy is given by name and the address changed: nginx resolves names in upstream when loading the config — reload is needed.

No line about the copy being disabled in error.log

nginx writes upstream server temporarily disabled at the warn level, while error_log on Ubuntu keeps error and above. A disabled copy shows as Connection refused and upstream timed out in error.log and as lists in $upstream_addr.

max_fails=0

The failure count is off: a dead copy stays in rotation forever. With refused connections this is an extra retry on every request, with a hung copy — a timeout on every request sent to it.

Retrying requests that change data

non_idempotent in proxy_next_upstream allows retrying POST on another copy. If the first copy did the work and failed while answering, the order or payment goes through twice.