Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

Load balancing — copies behind one address and health checks

11 min Ubuntu 24.04 · nginx 1.24 · keepalived 2.2

A load balancer takes requests on one address, spreads them across copies of a service and drops sick ones.

Cheat sheet

7 commands Printable version
  • sudo nginx -T | grep -A4 '^upstream' copies and their parameters
  • tail -n 5 /var/log/nginx/access.log $upstream_addr: who answered
  • sudo tail -n 3 /var/log/nginx/error.log why a copy dropped out
  • curl -s -o /dev/null -w '%{http_code}\n' 127.0.0.1:8082 copy alone
  • curl -s -o /dev/null -w '%{http_code} %header{x-upstream}\n' localhost
  • sudo nginx -t && sudo systemctl reload nginx apply without downtime
  • ip -br a VIP is the second address

Overview

A single server is a single point of failure: it goes down and the site is down. A load balancer takes requests on one address and spreads them across several identical copies of a service — backends (upstream). One copy fails, requests go to the others, and users do not notice. On Linux the balancer is usually nginx or haproxy.

Copies and round robin

nginx 
upstream app {
    server 127.0.0.1:8081;
    server 127.0.0.1:8082;
}
server {
    listen 80;
    location / {
        proxy_pass http://app;
        add_header X-Upstream $upstream_addr always;
    }
}

upstream is a group of copies under one name; proxy_pass sends requests to the group. By default nginx goes round the copies in turn (round robin): 8081, 8082, 8081… Other methods and the parameters of the server line are in nginx:

Parameter What it does
least_conn; a request goes to the copy with fewer connections
ip_hash; one client — always the same copy
weight=3 the copy gets three times more requests
backup the copy gets requests only when the others are unavailable
down the copy is switched off by hand

Who answered shows in the log as $upstream_addr and $upstream_status, and in the X-Upstream header. If nginx retried a request on another copy, the fields become comma-separated lists:

text 
… 200 upstream=127.0.0.1:8082, 127.0.0.1:8081 ustatus=502, 200

The first copy did not answer (502 — connection refused), the second answered 200. The user got 200.

Passive health check

The balancer has to know which copy is sick. nginx judges by real requests — a passive health check. A failed request to a copy adds one to a counter. max_fails failures within fail_timeout — the copy is switched off for fail_timeout. Then nginx tries it again with one request.

nginx 
server 127.0.0.1:8082 max_fails=2 fail_timeout=10s;

The defaults are max_fails=1, fail_timeout=10s: after the first failure the copy rests for 10 seconds. max_fails=0 turns the count off — the copy is always considered alive. A copy fixed within those 10 seconds gets requests only after them. What counts as a failure is set by proxy_next_upstream (below).

A passive check has two downsides: a user's request pays for it, and it notices a sick copy only through traffic.

Retry on another copy

proxy_next_upstream says which outcomes count as a failure and are retried on the next copy. The default is error timeout: the copy did not accept the connection or did not answer in time.

The copy What the user sees by default
does not listen (Connection refused) 200: nginx retried on another copy
answers 500 500 every other time: the copy's answer is no failure
hangs without answering an answer after 60 s: it waits proxy_read_timeout

Error codes from the copy are added to the list explicitly:

nginx 
proxy_next_upstream error timeout http_500 http_502 http_503 http_504;

A request that may change something (POST) is not resent once nginx has passed it to a copy: charging money twice is worse than one error. How many retries and how many seconds for them — proxy_next_upstream_tries and proxy_next_upstream_timeout.

Timeouts

Directive Waits for Default
proxy_connect_timeout the connection to the copy 60 s
proxy_read_timeout the copy's answer between two reads 60 s
proxy_send_timeout the copy to accept the request 60 s

A copy unreachable over the network (packets vanish) and a hung copy cost every request a minute. Short timeouts (2–3 s) together with timeout in proxy_next_upstream turn the minute into a couple of seconds and a retry on a live copy. A proxy_read_timeout that is too short cuts slow but healthy answers: heavy reports, exports.

Active health check

With an active health check the balancer polls each copy itself every few seconds — for example with GET /health. A copy is switched off after several failures in a row and back on after several successes. A sick copy drops out before a user reaches it. That is how haproxy checks (the check parameter of a server line); free nginx has no active checks, only passive ones.

Floating address

The balancer is a single point of failure too. It is built from two machines with a floating address (VIP, virtual IP): clients go to the VIP, which lives on one machine — the MASTER. keepalived on both machines exchanges VRRP advertisements. When the MASTER's advertisements stop, the standby (BACKUP) takes the address, in about 3 seconds with advert_int 1. On systemctl stop keepalived the MASTER announces its departure itself, and the address moves at once.

text 
$ ip -br a show ens2
ens2   UP   10.55.90.11/24 10.55.90.100/24 fe80::5054:ff:fe8c:5836/64

The second address on the interface is the VIP: this machine is the MASTER now. If the machines do not hear each other's advertisements, the address ends up on both — split brain, see keepalived.

Recipes

3

Take a copy out of rotation for maintenance

nginx 
server 127.0.0.1:8082 down;

sudo nginx -t && sudo systemctl reload nginx — the copy gets no new requests, the current ones finish. To return it, remove down and reload again. reload keeps connections, restart drops them (nginx).

How many requests each copy got

bash 
grep -o 'upstream=[^ ]*' /var/log/nginx/lb-access.log | sort | uniq -c

The upstream= field is the label of $upstream_addr in the log format; in your own format look for the same field. A skew means weight, ip_hash, a copy switched off after failures, or one marked backup.

Find the sick copy

bash 
for p in 8081 8082; do
  curl -s -o /dev/null -w "$p %{http_code} %{time_total}\n" \
    http://127.0.0.1:$p/
done

A request to each copy directly, bypassing the balancer: the code and the response time. For a copy on another machine, use its address instead of 127.0.0.1.

Troubleshooting

5

Every other request fails

One copy of two answers 500 or 503 by itself. For nginx such an answer is a success: the copy stays in rotation, and every other time the user gets an error. Fix the copy, take it out (down) or add the codes to proxy_next_upstream (load-balancing). Which copy answers with errors — $upstream_addr on the log lines with 500.

A request hangs for a minute, then answers

A copy is unreachable over the network or hung, and nginx waits proxy_connect_timeout or proxy_read_timeout — 60 s each. In error.log — upstream timed out (110: Connection timed out). Set 2–3 seconds (load-balancing).

no live upstreams

text 
[error] … no live upstreams while connecting to upstream,
  … upstream: "http://app/"

Every copy of the group is switched off after failures — the client gets 502, and $upstream_addr holds the group name (app) instead of an address. Fix at least one copy: after fail_timeout nginx tries them again.

The copy is fixed but gets no requests

  • fail_timeout has not passed since the last failure;
  • the line still has down or backup;
  • the copy is given by name and the address changed: nginx resolves names in upstream when loading the config — reload is needed.

No line about the copy being disabled in error.log

nginx writes upstream server temporarily disabled at the warn level, while error_log on Ubuntu keeps error and above. A disabled copy shows as Connection refused and upstream timed out in error.log and as lists in $upstream_addr.

Danger zone

2

max_fails=0

The failure count is off: a dead copy stays in rotation forever. With refused connections this is an extra retry on every request, with a hung copy — a timeout on every request sent to it.

Retrying requests that change data

non_idempotent in proxy_next_upstream allows retrying POST on another copy. If the first copy did the work and failed while answering, the order or payment goes through twice.

Terms

5
load balancer
Takes requests on one address, spreads them across copies of a service and drops sick copies from rotation. On Linux it is usually nginx with an upstream block or HAProxy. Also appears in: SLI and SLO
round robin
Handing out requests in turn: the first copy, the second, the first again. The default order of an nginx upstream. Also appears in: DNS and name resolution, nginx
passive health check
The balancer judges a copy by real requests: max_fails failures within fail_timeout switch the copy off for fail_timeout. In nginx the default is 1 failure and 10 s.
active health check
The balancer polls every copy itself every few seconds and switches it off after several failures in a row, before users reach it. HAProxy checks this way (check). Also appears in: HAProxy
single point of failure
A part of a system without which nothing works: one server, one balancer, one disk. It is removed with copies and failover.

See also

  • nginx — upstream, server, retries, timeouts.
  • haproxy — a balancer with active checks of copies.
  • keepalived — a floating address over VRRP.
  • http — 5xx response codes.
  • curl — a request to a copy directly, -w with code and time.
  • sli-slo — how many errors the target allows.
  • incident-response — taking a server out of the pool to restore service.

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Topics
Loading…