Linux Labs

Observability: metrics, alerts, logs / Stage 8 of 9

Investigating with logs

Not running Starting usually ~20 s
Leave the lesson

Step 2 of 6 · on a machine · 7 min

Who writes what

The cause of a 500 answer is known to whoever gave it out — and nginx only passed it on.

nginx keeps two logs. access.log has a line for every request: client address, time, request and answer code. error.log holds nginx's own failures: it could not connect to the application, did not get an answer in time, did not find a file. The application behind nginx is called the upstream. If it answered with code 500, to nginx that is an ordinary answer: it writes it to access.log, passes it to the client and does not count it as an error. That is why error.log is empty. How to split access.log into fields and count answer codes, you saw in the labs “A report from the nginx log” and “Count the errors in the logs”.

The application knows the cause. lab-shop is a systemd service, and its output goes to the service's journal, which you read with journalctl -u in the lesson on the system journal. Time is the common thread between the logs. In access.log it is written as [22/Oct/2026:04:51:09 +0000], in the journal as Oct 22 04:51:09; +0000 means UTC, like the machine's clock. The second of the first 500 from access.log goes into --since (in the Wiki):

bash
sudo journalctl -u lab-shop --since "2026-10-22 04:51:09"
Exercise · look

Find the cause in the application's journal

Find the first 500 answer in access.log and read the lab-shop service's journal.

First, predict

In the second of the first 500 nginx wrote nothing to error.log. What is in the lab-shop journal in that second?

Now start the exercise, run the command and compare your prediction with the output.

Sign in and start The machine is available after signing in
Ubuntu 24.04
Loading…