Inside the system: resources, kernel, processes / Stage 2 of 12 · step 7 of 9

A service restarts in a loop

~25 min

Not running Starting usually ~20 s

Situation

The feedd service builds the product feed for the marketplaces: every half second it rebuilds the /var/lib/feedd/feed.xml file, and the marketplaces fetch it. Last night feedd was updated to version 2.4. Since then the marketplaces have been getting a stale feed, and the lab-feedd service is up one moment and restarting the next.

The service unit came with the package: /usr/lib/systemd/system/lab-feedd.service. The application settings are in /etc/feedd/feedd.conf.

Task

The service memory limit is 128 MiB: that is how much it is given on this machine. The limit survives a reboot, and the package unit file stays as it is — do not edit it or replace it with a copy: a package update will overwrite it.
feedd runs without restarting: the check will wait until the service process has run for 30 seconds in a row.
The feed is refreshed every half second as before, and the process memory does not grow from job to job. Do not change the feedd code in /usr/lib/feedd.

The machine has no internet access. Manage the service and edit files in /etc with sudo.

Lab goals

0 of 3

The lab-feedd memory limit is 128 MiB, set by a drop-in for the unit, and survives a reboot — not checked yet
feedd runs for 30 seconds in a row without restarting — not checked yet
The feed is refreshed every half second, and the process memory does not grow from job to job; the feedd code is unchanged — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Processes · picked for this lab

Memory and swap: free, available and the page cache

Cheat sheet

$_ — insert into the terminal

free -h memory and swap; the reserve is available
grep -E '^(Mem|Cached)' /proc/meminfo same from the kernel, KiB
ps -eo pid,rss,comm --sort=-rss | head -5 who holds the most
swapon --show active swap; empty means no swap
vmstat 1 5 si and so — swap reads and writes
sysctl vm.swappiness kernel's eagerness to swap, 60 by default
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches drop the cache

Add a swap file

bash

sudo fallocate -l 512M /swapfile    # a file of the right size
sudo chmod 600 /swapfile            # readable by root only
sudo mkswap /swapfile               # format it as swap
sudo swapon /swapfile               # turn it on
swapon --show

text

student@ubuntu:~$ swapon --show
NAME      TYPE SIZE USED PRIO
/swapfile file 512M   0B   -2

To turn swap on at boot, add a line to /etc/fstab:

bash

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo swapoff /swapfile; sudo swapon -a; swapon --show   # check

swapon -a turns on everything listed in fstab: if the file is back in the list, the line has no typos. Instead of fallocate, sudo dd if=/dev/zero of=/swapfile bs=1M count=512 works too — slower, but on any filesystem.

Three details people get wrong:

Size. 512M means 512 MiB (536870912 bytes), 512MB means 512,000,000 bytes. The "Data size converter" tool turns one into the other.
Permissions. chmod 600 goes before swapon: otherwise any user could read program memory from the file.
Path in fstab. A swap file goes into fstab by its path, /swapfile. A UUID=… none swap sw 0 0 line for a file passes the swapon -a check, but at boot systemd waits for the device /dev/disk/by-uuid/…, and such links exist only for disks and partitions. After 90 seconds the journal says Timed out waiting for device, and swap stays off.

Turn swap off

bash

sudo swapoff /swapfile   # pages from swap return to memory

Then remove the line from /etc/fstab and delete the file.

Drop the cache before a measurement

bash

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

This is only for experiments: to measure reading from disk rather than from the cache. It does not help normal work; the cache fills up again. sudo echo 3 > /proc/sys/vm/drop_caches will not work: your own shell does the redirection without root rights and answers Permission denied.

free is almost zero

Normal: the page cache took the free memory. Look at available. Worry when available is small and si/so grow in vmstat.

The server is slow, si and so grow

Memory is short, and the kernel shuffles pages between memory and swap. Find who holds the most (ps -eo pid,rss,comm --sort=-rss | head) and stop or limit it (a service's MemoryMax, see oom-killer). Adding swap only buys time; it adds no memory.

A process ended, the log says Killed process

Memory ran out and the kernel ended the process — see oom-killer.

mkswap or swapon: insecure permissions 0644

mkswap prints insecure permissions 0644, fix with: chmod 0600 /swapfile, swapon prints insecure permissions 0644, 0600 suggested. Swap turns on anyway, but any user could read program memory from the file. sudo chmod 600 /swapfile.

swapon failed: Device or resource busy

The file is already on. Check swapon --show.

Swap is gone after a reboot

swapon lasts until a reboot. Add a line to /etc/fstab (recipe above). The line is there, but the boot log says Timed out waiting for device /dev/disk/by-uuid/… and Dependency failed for …swap — the file is listed by UUID=; replace it with the path.

swap.img.swap in the failed list

text

student@ubuntu:~$ systemctl --failed --no-legend
● swap.img.swap loaded failed failed /swap.img
student@ubuntu:~$ sudo swapon -a
swapon: cannot open /swap.img: No such file or directory

systemd makes a <path>.swap unit for every swap line in fstab. The file was deleted but the line stayed — the unit fails on every boot. Remove the line from /etc/fstab, then run sudo systemctl daemon-reload and sudo systemctl reset-failed.

vm.swappiness is back to 60

The value was set with sysctl vm.swappiness=… — it lives until a reboot. Put the line into a file under /etc/sysctl.d/. The kernel rejects values above 200: sysctl: setting key "vm.swappiness": Invalid argument.

swapoff when memory is short

swapoff brings every page from swap back into memory. If there is no room, the command thrashes the disk for a long time, and then the oom-killer comes. Before swapoff, compare used in the Swap row with available.

A service restarts in a loop

Situation

Task

Signing in is needed to start the machine

Add a swap file

Turn swap off

Drop the cache before a measurement

free is almost zero

The server is slow, si and so grow

A process ended, the log says Killed process

mkswap or swapon: insecure permissions 0644

swapon failed: Device or resource busy

Swap is gone after a reboot

swap.img.swap in the failed list

vm.swappiness is back to 60

swapoff when memory is short