This step explains why the stopped copy spoiled no answers while the sick one spoiled every other.
When a copy refuses the connection, nginx counts the attempt as failed and sends the same request to the next copy. The client gets the answer from 8081 and never learns about the refusal: it stays as a line in error.log. Which answers count as failures is set by the proxy_next_upstream directive. By default it is error timeout: the copy did not connect, dropped the connection or did not answer in time.
nginx also counts failures. Once a copy has max_fails failures within fail_timeout, nginx takes it out for fail_timeout, then tries it again with a single request. This is a passive health check: there are no separate probe requests, nginx judges a copy by its answers to real ones.
| Directive | Default | What it sets |
|---|---|---|
max_fails |
1 | how many failures take a copy out |
fail_timeout |
10s | the period for counting failures and for how long the copy is out |
proxy_next_upstream |
error timeout |
which failures send the request to the next copy |
max_fails and fail_timeout go on the copy's line: server 127.0.0.1:8082 max_fails=2 fail_timeout=20s;. proxy_next_upstream goes into the location.
A 500 answer is not on that list: to nginx it is an ordinary answer from the application, and it goes to the client. To make a 500 count as a failure, add it: proxy_next_upstream error timeout http_500;. Then nginx retries the request on another copy and takes the sick one out for fail_timeout.
nginx cannot poll the copies on its own — that is an active health check, which haproxy has. But it checks only what it asks: copy 8082 answers 200 on /healthz.
The upstream has the line server 127.0.0.1:8082 max_fails=3 fail_timeout=30s;, copy 8082 is stopped, 8081 is running. How many Connection refused lines will six requests in a row leave in error.log?
-
The client will get
200, but every failed attempt to connect to 8082 writes a line to error.log. -
In the exercise
max_failswas 1. Now three failures take the copy out: 8082 gets the second, fourth and sixth requests. -
Every other request goes to 8082, and 8081 answers the rest with no errors. There are three attempts to connect to 8082.
Going round, 8082 gets the second, fourth and sixth requests; each refusal is a line in error.log and a retry on 8081. The third failure takes the copy out for 30 s, while the client gets 200 all six times.
Key points
-
A failure is whatever
proxy_next_upstreamlists: on such a failure nginx retries the request on another copy. -
max_failsfailures withinfail_timeouttake a copy out forfail_timeout— which is why six requests left a single refusal line. -
By default a
500answer goes to the client;http_500turns on retries and taking the copy out for it.