A load balancer and health checks

The shop site answers every other time — every other request hangs until the client gives up. One of the two backends is down, and backends will keep going down. Make the load balancer on node1 withstand it.

Situation

The shop runs on two machines. On node2 the application runs as two instances — the backends lab-backend@8081 and lab-backend@8082 listen on ports 8081 and 8082. On node1 nginx takes customers' requests on port 80 and spreads them between the backends: the lb site, the file /etc/nginx/sites-available/lb.

Since this morning customers have been complaining: the page either opens at once or hangs until the browser gives up. The shop's mobile app waits no longer than 5 seconds for an answer.

node1 is your terminal. Run commands on node2 over SSH: ssh node2 'command', or log in with ssh node2 and come back with exit. sudo works without a password on both machines. nginx records which backend answered in the X-Upstream header and in the upstream= field of the log /var/log/nginx/lb-access.log. A request "like the mobile app":

curl -so /dev/null -m 5 \
  -w '%{http_code} %{time_total}s %header{x-upstream}\n' localhost/

Task

Backend 8082 is down: not even connection attempts reach its port — this is how a machine that switched off looks from outside. It will be brought back, but backends will keep going down. Make sure customers don't notice:

1. While one backend is down, every request to the site gets 200 within 5 seconds.
2. nginx takes the down backend out: of 20 requests in a row, no more than three go to it.
3. Once the backend is reachable again, it gets requests again within 20 seconds.

In the lab the outage is played by the lab-outage command on node2: ssh node2 sudo lab-outage off "brings" backend 8082 back, on "crashes" it again, status tells whether it is down and how many connection attempts never reached it. "Check" crashes 8082 on its own and sends 20 requests in a row, then brings it back and waits for requests to reach it again. At the end 8082 returns to the state it was in before the check. The check takes up to a minute.

Constraints

• nginx is serving customers: don't stop or restart it. Test a change with sudo nginx -t and apply it with a reload — sudo systemctl reload nginx.
• Both backends stay in service: don't take either of them out of the balancing or make it a backup.
• Don't change the application on node2.

There is no internet in the lab — only the network between node1 and node2, everything you need is installed. Look at the result the way the check does: a series of requests — for i in $(seq 10); do curl …; done with the command above, and what nginx did with each request — tail /var/log/nginx/lb-access.log and sudo tail /var/log/nginx/error.log.