Packages, services, and logs / Stage 7 of 13

Diagnosing a failed service

Not running Starting usually ~20 s

Step 7 of 8 · lab · 15 min

Fix nginx

The nginx site stopped opening after yesterday's work. Find the cause and bring the site back.

Situation

The server ran a website on nginx. After yesterday's work on the server the site stopped opening: curl http://localhost/ says Couldn't connect to server. Nobody remembers what exactly was changed.

Task

The default site answers at http://localhost/ with status 200 again.
nginx runs as a systemd service, and its configuration passes the nginx -t check.
The default site — the file /etc/nginx/sites-enabled/default and its listen lines — stays in place: fix the error rather than work around it.

The tools you need (nginx, curl) are already installed. There is no internet on this machine.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 4

The nginx configuration passes nginx -t — not done yet
The nginx service is active — not done yet
http://localhost/ returns HTTP 200 — not done yet
The default site is still in place — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 4

The nginx configuration passes nginx -t — not done yet
The nginx service is active — not done yet
http://localhost/ returns HTTP 200 — not done yet
The default site is still in place — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 4

The nginx configuration passes nginx -t — not done yet
The nginx service is active — not done yet
http://localhost/ returns HTTP 200 — not done yet
The default site is still in place — not done yet

Services and systemd · picked for this lab

nginx — web server and reverse proxy

Cheat sheet

$_ — insert into the terminal

sudo nginx -t check the config (sudo only)
sudo systemctl reload nginx apply, keep connections
sudo nginx -T | grep -n server_name which sites nginx sees
ls -l /etc/nginx/sites-enabled/ enabled sites
curl -i -H 'Host: docs.lab.test' http://127.0.0.1/ who answers
sudo tail -n 5 /var/log/nginx/error.log why 404, 403, 502
sudo tail -n 5 /var/log/nginx/access.log what was asked, the code
sudo ss -tlnp | grep nginx which ports it listens on

The Debian and Ubuntu layout:

/etc/nginx/nginx.conf — common settings, with an include of sites at the bottom;
/etc/nginx/sites-available/ — site files;
/etc/nginx/sites-enabled/ — links to enabled sites; nginx reads only these;
/var/log/nginx/access.log and error.log — the logs.

Enable a site

bash

sudo ln -s /etc/nginx/sites-available/docs /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

A file in sites-available without a link in sites-enabled is invisible to nginx. To disable a site, remove the link; the file stays.

nginx -t checks the whole config and names the file and line of an error. reload re-reads the config without dropping connections; with an error in the config nginx keeps the old one. systemctl reload returns at once, while the new worker processes take connections a moment later: a curl on the same line may still see the old state.

Logs

access.log has a line for every request, error.log says why a request failed and what broke at start. The default access.log format is combined:

text

127.0.0.1 - - [20/Oct/2026:10:03:58 +0000] "POST / HTTP/1.1" 405 166 "-"

Field by field: $remote_addr (client address), -, $remote_user, [$time_local], "$request" (method, path, version), $status (code), $body_bytes_sent (body size), "$http_referer", "$http_user_agent" — the user agent "curl/8.5.0" did not fit in the example. log_format defines your own format — in a site file outside the server block:

nginx

log_format timed '$remote_addr [$time_local] "$request" '
                 '$status $body_bytes_sent $request_time';
access_log /var/log/nginx/app.log timed;

$request_time is how many seconds nginx spent on the request. Files with .1 and .2.gz next to a log are older ones that logrotate moves aside and compresses on schedule (/etc/logrotate.d/nginx).

Proxy: path and headers

Whether there is a URI after the address in proxy_pass decides which path the application receives:

`location /api/` and `proxy_pass`	Request `/api/x` goes out as
`http://127.0.0.1:3003`	`/api/x` — path unchanged
`http://127.0.0.1:3003/`	`/x` — `/api/` replaced with `/`

By default the application sees Host: 127.0.0.1:3003 — the address from proxy_pass. The site name and the client address are passed in headers:

nginx

proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

HTTPS and the redirect from HTTP

nginx

server {
    listen 443 ssl;
    server_name shop.lab.test;
    ssl_certificate     /etc/nginx/tls/fullchain.pem;
    ssl_certificate_key /etc/nginx/tls/shop.key;
    root /var/www/shop;
}
server {
    listen 80;
    server_name shop.lab.test;
    return 301 https://$host$request_uri;
}

ssl_certificate is the chain file: the site certificate first, the intermediate after it. $request_uri is the path with the query after ?: http://shop.lab.test/x?y=1 redirects to https://shop.lab.test/x?y=1. Certificates and how they are checked — openssl-tls.

Who will answer a name

bash

curl -i -H 'Host: docs.lab.test' http://127.0.0.1/
sudo nginx -T | grep -n 'server_name\|listen'

The first command shows the answer as a client sees it, the second which listen and server_name nginx actually read, in file order.

Find where the proxy breaks

bash

curl -s -o /dev/null -w '%{http_code}\n' http://app.lab.test/
sudo tail -n 1 /var/log/nginx/error.log
curl -i http://127.0.0.1:3000/        # the application directly

The error.log line names the upstream — where nginx went. If the application answers directly, the problem is in proxy_pass or permissions; if not, it is in the application.

Someone else's site answers

The request's Host matched no server_name, so the default server answered. Check the name in server_name and that the site's link exists in sites-enabled: sudo nginx -T | grep server_name.

404 Not Found

error.log has the path nginx built: open() "/var/www/portal/nope.html" failed (2: No such file or directory). Compare it with the real path of the file: usually root or alias is wrong.

403 Forbidden

(13: Permission denied) — www-data cannot read the file or enter a directory on the path; namei -l shows permissions along the whole path. directory index of "…" is forbidden — the directory has no file from index.

502 Bad Gateway

nginx did not open a connection to the backend. The error.log line names the reason:

In error.log	Meaning
`(111: Connection refused)`	nothing listens on the address or socket
`(13: Permission denied)`	`www-data` has no right to the socket
`(2: No such file or directory)`	no socket file at that path

In full: connect() failed (111: Connection refused) while connecting to upstream, … upstream: "http://127.0.0.1:3001/". For a socket — connect() to unix:/run/app/app.sock failed (13: Permission denied). nginx logs errors 13 and 2 at level [crit] and 111 at [error]: a search for [error] alone misses them. A socket file left behind after the application stopped also gives 111. Socket permissions and paths — unix-sockets.

504 Gateway Time-out

There is a connection but no answer: upstream timed out (110: Connection timed out) while reading response header from upstream. The application hangs or answers slower than proxy_read_timeout (60 s by default).

nginx -t: test failed

unexpected "}" in …:5 — the line before } has no ;;
a duplicate default server for 0.0.0.0:80 — default_server is set in two blocks on one port;
open() "/run/nginx.pid" failed (13: Permission denied) — nginx -t was run without sudo.

conflicting server name

[warn] conflicting server name "docs.lab.test" on 0.0.0.0:80, ignored — one name sits in two blocks on one port. The test still passes, but requests for that name go to the block loaded first, and the second never gets them. Find both:

bash

grep -n 'docs.lab.test' /etc/nginx/sites-enabled/*

grep -r on the directory does not work here: it skips the links in sites-enabled.

key values mismatch

SSL_CTX_use_PrivateKey(…) failed (… key values mismatch) — the first certificate in the ssl_certificate file does not match the key. Either the key is wrong, or the chain file has the intermediate first. The site certificate goes first.

The new certificate is not served

nginx reads certificates at start and on reload. After replacing the file — sudo nginx -t && sudo systemctl reload nginx, then check a second later.

The service does not start: port in use

sudo ss -tlnp | grep ':80 ' shows who holds the port. Stop it or change listen.

A new listen did not apply

Changing the address on the same port (listen 80 → listen 127.0.0.1:80) does not work through reload: systemctl reload returns 0, nginx stays on the old sockets, and error.log says bind() to 127.0.0.1:80 failed (98: Address already in use). Only sudo systemctl restart nginx applies such a change; sudo ss -tlnp | grep nginx shows what really listens.

restart with an error in the config

On reload with an error the old config keeps working; restart stops nginx and cannot start it again: every site on the machine is down. Before applying anything — sudo nginx -t.

Diagnosing a failed service

Situation

Task

Signing in is needed to start the machine

Who will answer a name

Find where the proxy breaks

Someone else's site answers

404 Not Found

403 Forbidden

502 Bad Gateway

504 Gateway Time-out

nginx -t: test failed

conflicting server name

key values mismatch

The new certificate is not served

The service does not start: port in use

A new listen did not apply

restart with an error in the config