Linux Labs

Inside the system: resources, kernel, processes / Final lab

A slow server: triaging three complaints

hard ~45 min
Not running Starting usually ~20 s
Report a problem

Situation

This server runs the catalog site of an online shop, the lab-catalog service on port 8080. Since the morning the server has received three complaints, each with its own key:

  • slow — the home page http://localhost:8080/ takes half a second to open, while it usually needs hundredths of a second. Response time: curl -s -o /dev/null -w '%{time_total}\n' http://localhost:8080/.
  • memory — monitoring reports that the server has less available memory with every minute, although there are no more visitors than usual.
  • errors — when there are many visitors, requests to the product cards /api/item/<number> end with error 500. A rush of 200 simultaneous requests:

bash curl -s -Z --parallel-max 200 --parallel-immediate -o /dev/null -w '%{http_code}\n' 'http://localhost:8080/api/item/[1-200]' | sort | uniq -c

The developers will deal with their code later. Your job is to find on the machine what causes each complaint and remove the cause through system and service settings. Do not change the programs in /usr/local/bin. sudo works without a password.

Task

  1. slow: the home page answers in under 0.25 s, and still does after the server reboots.
  2. memory: the process that eats the memory keeps running, but cannot take more than 128 MiB, and when it reaches the limit it comes back by itself — after a reboot too.
  3. errors: all 200 requests from the command above get code 200, after a service restart and a reboot too.
  4. A report ~/triage.txt with one line per complaint: the key, the PID of the culprit process and the cause in your own words.

slow: <PID> <cause> memory: <PID> <cause> errors: <PID> <cause>

The PID is the one you found during the triage. After a restart the process gets a new PID; the old one is accepted too.

Visible only to you. Saved as you type.

All notes

Lab goals

0 of 4
  • slow: the home page http://localhost:8080/ answers in under 0.25 s — after a reboot too — not checked yet
  • memory: the process that ate the memory runs within 128 MiB and comes back by itself — after a reboot too — not checked yet
  • errors: all 200 simultaneous requests to /api/item/… get code 200 — after a service restart and a reboot too — not checked yet
  • ~/triage.txt has the PID of the culprit process and the cause for each complaint — not checked yet

Hints

0 of 3 open · they open one at a time
Hint 1
Show the solution after 3 failed checks

Leave the lab?

The machine keeps running for 10 more minutes, then it is removed. Files are lost; completed goals stay.

Loading…