A site and its database on two machines

~50 min

Not running Starting usually ~20 s

Situation

The orders site app.lab.test and its PostgreSQL database used to live on one machine, node2. The site has moved to node1: nginx with HTTPS and the application, the lab-shop service, now run there. The shop database stayed on node2. Since the move, curl https://app.lab.test/ on node1 answers with an error instead of the order list.

node1 is your terminal. Run commands on node2 over SSH: ssh node2 'command', or enter its shell with ssh node2 and come back with exit. sudo works without a password on both machines. There is no other way into node2, so don't lock yourself out of SSH.

Task

curl https://app.lab.test/ on node1 gets HTTP 200 and the order list from the database on node2.
The shop role logs in to the database from node1 with its password, and with a wrong password neither shop nor postgres can log in from node1.
Port 5432 on node2 is open to node1 and closed to other addresses on the network, and stays so after node2 reboots.

The lab has no internet access, only the network between node1 and node2. Everything you need is installed on both machines.

Lab goals

0 of 3

curl https://app.lab.test/ on node1 gets 200 and the order list from the database on node2 — not checked yet
From node1 the shop role logs in with its password, and with a wrong password neither shop nor postgres gets in — not checked yet
Port 5432 on node2 is open to node1 and closed to other addresses on the network — after a node2 reboot too — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Services and systemd · picked for this lab

nginx — web server and reverse proxy

Cheat sheet

$_ — insert into the terminal

sudo nginx -t check the config (sudo only)
sudo systemctl reload nginx apply, keep connections
sudo nginx -T | grep -n server_name which sites nginx sees
ls -l /etc/nginx/sites-enabled/ enabled sites
curl -i -H 'Host: docs.lab.test' http://127.0.0.1/ who answers
sudo tail -n 5 /var/log/nginx/error.log why 404, 403, 502
sudo tail -n 5 /var/log/nginx/access.log what was asked, the code
sudo ss -tlnp | grep nginx which ports it listens on

The Debian and Ubuntu layout:

/etc/nginx/nginx.conf — common settings, with an include of sites at the bottom;
/etc/nginx/sites-available/ — site files;
/etc/nginx/sites-enabled/ — links to enabled sites; nginx reads only these;
/var/log/nginx/access.log and error.log — the logs.

Enable a site

bash

sudo ln -s /etc/nginx/sites-available/docs /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

A file in sites-available without a link in sites-enabled is invisible to nginx. To disable a site, remove the link; the file stays.

nginx -t checks the whole config and names the file and line of an error. reload re-reads the config without dropping connections; with an error in the config nginx keeps the old one. systemctl reload returns at once, while the new worker processes take connections a moment later: a curl on the same line may still see the old state.

Logs

access.log has a line for every request, error.log says why a request failed and what broke at start. The default access.log format is combined:

text

127.0.0.1 - - [20/Oct/2026:10:03:58 +0000] "POST / HTTP/1.1" 405 166 "-"

Field by field: $remote_addr (client address), -, $remote_user, [$time_local], "$request" (method, path, version), $status (code), $body_bytes_sent (body size), "$http_referer", "$http_user_agent" — the user agent "curl/8.5.0" did not fit in the example. log_format defines your own format — in a site file outside the server block:

nginx

log_format timed '$remote_addr [$time_local] "$request" '
                 '$status $body_bytes_sent $request_time';
access_log /var/log/nginx/app.log timed;

$request_time is how many seconds nginx spent on the request. Files with .1 and .2.gz next to a log are older ones that logrotate moves aside and compresses on schedule (/etc/logrotate.d/nginx).

Proxy: path and headers

Whether there is a URI after the address in proxy_pass decides which path the application receives:

`location /api/` and `proxy_pass`	Request `/api/x` goes out as
`http://127.0.0.1:3003`	`/api/x` — path unchanged
`http://127.0.0.1:3003/`	`/x` — `/api/` replaced with `/`

By default the application sees Host: 127.0.0.1:3003 — the address from proxy_pass. The site name and the client address are passed in headers:

nginx

proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

HTTPS and the redirect from HTTP

nginx

server {
    listen 443 ssl;
    server_name shop.lab.test;
    ssl_certificate     /etc/nginx/tls/fullchain.pem;
    ssl_certificate_key /etc/nginx/tls/shop.key;
    root /var/www/shop;
}
server {
    listen 80;
    server_name shop.lab.test;
    return 301 https://$host$request_uri;
}

ssl_certificate is the chain file: the site certificate first, the intermediate after it. $request_uri is the path with the query after ?: http://shop.lab.test/x?y=1 redirects to https://shop.lab.test/x?y=1. Certificates and how they are checked — openssl-tls.

Who will answer a name

bash

curl -i -H 'Host: docs.lab.test' http://127.0.0.1/
sudo nginx -T | grep -n 'server_name\|listen'

The first command shows the answer as a client sees it, the second which listen and server_name nginx actually read, in file order.

Find where the proxy breaks

bash

curl -s -o /dev/null -w '%{http_code}\n' http://app.lab.test/
sudo tail -n 1 /var/log/nginx/error.log
curl -i http://127.0.0.1:3000/        # the application directly

The error.log line names the upstream — where nginx went. If the application answers directly, the problem is in proxy_pass or permissions; if not, it is in the application.

Someone else's site answers

The request's Host matched no server_name, so the default server answered. Check the name in server_name and that the site's link exists in sites-enabled: sudo nginx -T | grep server_name.

404 Not Found

error.log has the path nginx built: open() "/var/www/portal/nope.html" failed (2: No such file or directory). Compare it with the real path of the file: usually root or alias is wrong.

403 Forbidden

(13: Permission denied) — www-data cannot read the file or enter a directory on the path; namei -l shows permissions along the whole path. directory index of "…" is forbidden — the directory has no file from index.

502 Bad Gateway

nginx did not open a connection to the backend. The error.log line names the reason:

In error.log	Meaning
`(111: Connection refused)`	nothing listens on the address or socket
`(13: Permission denied)`	`www-data` has no right to the socket
`(2: No such file or directory)`	no socket file at that path

In full: connect() failed (111: Connection refused) while connecting to upstream, … upstream: "http://127.0.0.1:3001/". For a socket — connect() to unix:/run/app/app.sock failed (13: Permission denied). nginx logs errors 13 and 2 at level [crit] and 111 at [error]: a search for [error] alone misses them. A socket file left behind after the application stopped also gives 111. Socket permissions and paths — unix-sockets.

504 Gateway Time-out

There is a connection but no answer: upstream timed out (110: Connection timed out) while reading response header from upstream. The application hangs or answers slower than proxy_read_timeout (60 s by default).

nginx -t: test failed

unexpected "}" in …:5 — the line before } has no ;;
a duplicate default server for 0.0.0.0:80 — default_server is set in two blocks on one port;
open() "/run/nginx.pid" failed (13: Permission denied) — nginx -t was run without sudo.

conflicting server name

[warn] conflicting server name "docs.lab.test" on 0.0.0.0:80, ignored — one name sits in two blocks on one port. The test still passes, but requests for that name go to the block loaded first, and the second never gets them. Find both:

bash

grep -n 'docs.lab.test' /etc/nginx/sites-enabled/*

grep -r on the directory does not work here: it skips the links in sites-enabled.

key values mismatch

SSL_CTX_use_PrivateKey(…) failed (… key values mismatch) — the first certificate in the ssl_certificate file does not match the key. Either the key is wrong, or the chain file has the intermediate first. The site certificate goes first.

The new certificate is not served

nginx reads certificates at start and on reload. After replacing the file — sudo nginx -t && sudo systemctl reload nginx, then check a second later.

The service does not start: port in use

sudo ss -tlnp | grep ':80 ' shows who holds the port. Stop it or change listen.

A new listen did not apply

Changing the address on the same port (listen 80 → listen 127.0.0.1:80) does not work through reload: systemctl reload returns 0, nginx stays on the old sockets, and error.log says bind() to 127.0.0.1:80 failed (98: Address already in use). Only sudo systemctl restart nginx applies such a change; sudo ss -tlnp | grep nginx shows what really listens.

restart with an error in the config

On reload with an error the old config keeps working; restart stops nginx and cannot start it again: every site on the machine is down. Before applying anything — sudo nginx -t.

A site and its database on two machines

Situation

Task

Signing in is needed to start the machine

Who will answer a name

Find where the proxy breaks

Someone else's site answers

404 Not Found

403 Forbidden

502 Bad Gateway

504 Gateway Time-out

nginx -t: test failed

conflicting server name

key values mismatch

The new certificate is not served

The service does not start: port in use

A new listen did not apply

restart with an error in the config