Linux Labs

Observability: metrics, alerts, logs / Stage 8 of 9

Investigating with logs

Not running Starting usually ~20 s
Leave the lesson

Step 6 of 6 · summary · 2 min

Summary

  • nginx's access.log records every answer with its code, error.log only nginx's own failures; nginx passes a 500 answer from the application on and does not count it as an error.
  • The cause of an answer is looked for in the application's journal by the second from access.log: journalctl -u <service> --since.
  • journalctl -o json shows all of a record's fields — _SYSTEMD_UNIT, _PID, PRIORITY; jq picks them, and sort | uniq -c counts.
  • Kernel messages (journalctl -k) are on the same timeline: the OOM killer and 502s with Connection refused match to the second.
  • error.log at the start of the lesson was empty because the application gave out the 500, and the cause — a PermissionError on the orders file — is written in its journal.
Ubuntu 24.04
Loading…