Web stack / Stage 2 of 9 · step 7 of 8

Fix 502 Bad Gateway

~25 min

Not running Starting usually ~20 s

Situation

The server runs an internal notes service: the notes-app application, the lab-notes service. nginx makes it available to users under the name notes.lab.test. After the service was updated, the site answers 502 Bad Gateway, even though systemctl status lab-notes shows the service running.

The name notes.lab.test is in /etc/hosts and points to this same machine, so you can run curl http://notes.lab.test/ right here.

Task

The lab-notes service runs as the notes user, and the application code stays the same.
http://notes.lab.test/ answers 200 with a response from the application.
The application socket is not writable by everyone, and nginx runs as www-data, as before.
The site still answers after the application restarts: sudo systemctl restart lab-notes.

There is no internet on the machine; everything you need is already installed. Run administrative commands with sudo.

Lab goals

0 of 4

The lab-notes service runs as notes, and the application code is unchanged — not checked yet
http://notes.lab.test/ answers 200 with a response from the notes application — not checked yet
The application socket is not writable by everyone, and nginx runs as www-data — not checked yet
The site still answers after the application restarts — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Services and systemd · picked for this lab

nginx — web server and reverse proxy

Cheat sheet

$_ — insert into the terminal

sudo nginx -t check the config (sudo only)
sudo systemctl reload nginx apply, keep connections
sudo nginx -T | grep -n server_name which sites nginx sees
ls -l /etc/nginx/sites-enabled/ enabled sites
curl -i -H 'Host: docs.lab.test' http://127.0.0.1/ who answers
sudo tail -n 5 /var/log/nginx/error.log why 404, 403, 502
sudo tail -n 5 /var/log/nginx/access.log what was asked, the code
sudo ss -tlnp | grep nginx which ports it listens on

The Debian and Ubuntu layout:

/etc/nginx/nginx.conf — common settings, with an include of sites at the bottom;
/etc/nginx/sites-available/ — site files;
/etc/nginx/sites-enabled/ — links to enabled sites; nginx reads only these;
/var/log/nginx/access.log and error.log — the logs.

Enable a site

bash

sudo ln -s /etc/nginx/sites-available/docs /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

A file in sites-available without a link in sites-enabled is invisible to nginx. To disable a site, remove the link; the file stays.

nginx -t checks the whole config and names the file and line of an error. reload re-reads the config without dropping connections; with an error in the config nginx keeps the old one. systemctl reload returns at once, while the new worker processes take connections a moment later: a curl on the same line may still see the old state.

Logs

access.log has a line for every request, error.log says why a request failed and what broke at start. The default access.log format is combined:

text

127.0.0.1 - - [20/Oct/2026:10:03:58 +0000] "POST / HTTP/1.1" 405 166 "-"

Field by field: $remote_addr (client address), -, $remote_user, [$time_local], "$request" (method, path, version), $status (code), $body_bytes_sent (body size), "$http_referer", "$http_user_agent" — the user agent "curl/8.5.0" did not fit in the example. log_format defines your own format — in a site file outside the server block:

nginx

log_format timed '$remote_addr [$time_local] "$request" '
                 '$status $body_bytes_sent $request_time';
access_log /var/log/nginx/app.log timed;

$request_time is how many seconds nginx spent on the request. Files with .1 and .2.gz next to a log are older ones that logrotate moves aside and compresses on schedule (/etc/logrotate.d/nginx).

Proxy: path and headers

Whether there is a URI after the address in proxy_pass decides which path the application receives:

`location /api/` and `proxy_pass`	Request `/api/x` goes out as
`http://127.0.0.1:3003`	`/api/x` — path unchanged
`http://127.0.0.1:3003/`	`/x` — `/api/` replaced with `/`

By default the application sees Host: 127.0.0.1:3003 — the address from proxy_pass. The site name and the client address are passed in headers:

nginx

proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

HTTPS and the redirect from HTTP

nginx

server {
    listen 443 ssl;
    server_name shop.lab.test;
    ssl_certificate     /etc/nginx/tls/fullchain.pem;
    ssl_certificate_key /etc/nginx/tls/shop.key;
    root /var/www/shop;
}
server {
    listen 80;
    server_name shop.lab.test;
    return 301 https://$host$request_uri;
}

ssl_certificate is the chain file: the site certificate first, the intermediate after it. $request_uri is the path with the query after ?: http://shop.lab.test/x?y=1 redirects to https://shop.lab.test/x?y=1. Certificates and how they are checked — openssl-tls.

Who will answer a name

bash

curl -i -H 'Host: docs.lab.test' http://127.0.0.1/
sudo nginx -T | grep -n 'server_name\|listen'

The first command shows the answer as a client sees it, the second which listen and server_name nginx actually read, in file order.

Find where the proxy breaks

bash

curl -s -o /dev/null -w '%{http_code}\n' http://app.lab.test/
sudo tail -n 1 /var/log/nginx/error.log
curl -i http://127.0.0.1:3000/        # the application directly

The error.log line names the upstream — where nginx went. If the application answers directly, the problem is in proxy_pass or permissions; if not, it is in the application.

Someone else's site answers

The request's Host matched no server_name, so the default server answered. Check the name in server_name and that the site's link exists in sites-enabled: sudo nginx -T | grep server_name.

404 Not Found

error.log has the path nginx built: open() "/var/www/portal/nope.html" failed (2: No such file or directory). Compare it with the real path of the file: usually root or alias is wrong.

403 Forbidden

(13: Permission denied) — www-data cannot read the file or enter a directory on the path; namei -l shows permissions along the whole path. directory index of "…" is forbidden — the directory has no file from index.

502 Bad Gateway

nginx did not open a connection to the backend. The error.log line names the reason:

In error.log	Meaning
`(111: Connection refused)`	nothing listens on the address or socket
`(13: Permission denied)`	`www-data` has no right to the socket
`(2: No such file or directory)`	no socket file at that path

In full: connect() failed (111: Connection refused) while connecting to upstream, … upstream: "http://127.0.0.1:3001/". For a socket — connect() to unix:/run/app/app.sock failed (13: Permission denied). nginx logs errors 13 and 2 at level [crit] and 111 at [error]: a search for [error] alone misses them. A socket file left behind after the application stopped also gives 111. Socket permissions and paths — unix-sockets.

504 Gateway Time-out

There is a connection but no answer: upstream timed out (110: Connection timed out) while reading response header from upstream. The application hangs or answers slower than proxy_read_timeout (60 s by default).

nginx -t: test failed

unexpected "}" in …:5 — the line before } has no ;;
a duplicate default server for 0.0.0.0:80 — default_server is set in two blocks on one port;
open() "/run/nginx.pid" failed (13: Permission denied) — nginx -t was run without sudo.

conflicting server name

[warn] conflicting server name "docs.lab.test" on 0.0.0.0:80, ignored — one name sits in two blocks on one port. The test still passes, but requests for that name go to the block loaded first, and the second never gets them. Find both:

bash

grep -n 'docs.lab.test' /etc/nginx/sites-enabled/*

grep -r on the directory does not work here: it skips the links in sites-enabled.

key values mismatch

SSL_CTX_use_PrivateKey(…) failed (… key values mismatch) — the first certificate in the ssl_certificate file does not match the key. Either the key is wrong, or the chain file has the intermediate first. The site certificate goes first.

The new certificate is not served

nginx reads certificates at start and on reload. After replacing the file — sudo nginx -t && sudo systemctl reload nginx, then check a second later.

The service does not start: port in use

sudo ss -tlnp | grep ':80 ' shows who holds the port. Stop it or change listen.

A new listen did not apply

Changing the address on the same port (listen 80 → listen 127.0.0.1:80) does not work through reload: systemctl reload returns 0, nginx stays on the old sockets, and error.log says bind() to 127.0.0.1:80 failed (98: Address already in use). Only sudo systemctl restart nginx applies such a change; sudo ss -tlnp | grep nginx shows what really listens.

restart with an error in the config

On reload with an error the old config keeps working; restart stops nginx and cannot start it again: every site on the machine is down. Before applying anything — sudo nginx -t.

Fix 502 Bad Gateway

Situation

Task

Signing in is needed to start the machine

Who will answer a name

Find where the proxy breaks

Someone else's site answers

404 Not Found

403 Forbidden

502 Bad Gateway

504 Gateway Time-out

nginx -t: test failed

conflicting server name

key values mismatch

The new certificate is not served

The service does not start: port in use

A new listen did not apply

restart with an error in the config