Inside the system: resources, kernel, processes / Stage 2 of 12

Memory, swap and the OOM killer

Not running Starting usually ~20 s

Step 7 of 9 · lab · 25 min

A service restarts in a loop

Since an update, the marketplace feed service restarts every few seconds, and the feed is not refreshed. Find what stops it and fix it so that it runs within the memory it is given.

Situation

The feedd service builds the product feed for the marketplaces: every half second it rebuilds the /var/lib/feedd/feed.xml file, and the marketplaces fetch it. Last night feedd was updated to version 2.4. Since then the marketplaces have been getting a stale feed, and the lab-feedd service is up one moment and restarting the next.

The service unit came with the package: /usr/lib/systemd/system/lab-feedd.service. The application settings are in /etc/feedd/feedd.conf.

Task

The service memory limit is 128 MiB: that is how much it is given on this machine. The limit survives a reboot, and the package unit file stays as it is — do not edit it or replace it with a copy: a package update will overwrite it.
feedd runs without restarting: the check will wait until the service process has run for 30 seconds in a row.
The feed is refreshed every half second as before, and the process memory does not grow from job to job. Do not change the feedd code in /usr/lib/feedd.

The machine has no internet access. Manage the service and edit files in /etc with sudo.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 3

The lab-feedd memory limit is 128 MiB, set by a drop-in for the unit, and survives a reboot — not done yet
feedd runs for 30 seconds in a row without restarting — not done yet
The feed is refreshed every half second, and the process memory does not grow from job to job; the feedd code is unchanged — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 3

The lab-feedd memory limit is 128 MiB, set by a drop-in for the unit, and survives a reboot — not done yet
feedd runs for 30 seconds in a row without restarting — not done yet
The feed is refreshed every half second, and the process memory does not grow from job to job; the feedd code is unchanged — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 3

The lab-feedd memory limit is 128 MiB, set by a drop-in for the unit, and survives a reboot — not done yet
feedd runs for 30 seconds in a row without restarting — not done yet
The feed is refreshed every half second, and the process memory does not grow from job to job; the feedd code is unchanged — not done yet

Processes · picked for this lab

Memory and swap: free, available and the page cache

Cheat sheet

$_ — insert into the terminal

free -h memory and swap; the reserve is available
grep -E '^(Mem|Cached)' /proc/meminfo same from the kernel, KiB
ps -eo pid,rss,comm --sort=-rss | head -5 who holds the most
swapon --show active swap; empty means no swap
vmstat 1 5 si and so — swap reads and writes
sysctl vm.swappiness kernel's eagerness to swap, 60 by default
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches drop the cache

Add a swap file

bash

sudo fallocate -l 512M /swapfile    # a file of the right size
sudo chmod 600 /swapfile            # readable by root only
sudo mkswap /swapfile               # format it as swap
sudo swapon /swapfile               # turn it on
swapon --show

text

student@ubuntu:~$ swapon --show
NAME      TYPE SIZE USED PRIO
/swapfile file 512M   0B   -2

To turn swap on at boot, add a line to /etc/fstab:

bash

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo swapoff /swapfile; sudo swapon -a; swapon --show   # check

swapon -a turns on everything listed in fstab: if the file is back in the list, the line has no typos. Instead of fallocate, sudo dd if=/dev/zero of=/swapfile bs=1M count=512 works too — slower, but on any filesystem.

Three details people get wrong:

Size. 512M means 512 MiB (536870912 bytes), 512MB means 512,000,000 bytes. The "Data size converter" tool turns one into the other.
Permissions. chmod 600 goes before swapon: otherwise any user could read program memory from the file.
Path in fstab. A swap file goes into fstab by its path, /swapfile. A UUID=… none swap sw 0 0 line for a file passes the swapon -a check, but at boot systemd waits for the device /dev/disk/by-uuid/…, and such links exist only for disks and partitions. After 90 seconds the journal says Timed out waiting for device, and swap stays off.

Turn swap off

bash

sudo swapoff /swapfile   # pages from swap return to memory

Then remove the line from /etc/fstab and delete the file.

Drop the cache before a measurement

bash

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

This is only for experiments: to measure reading from disk rather than from the cache. It does not help normal work; the cache fills up again. sudo echo 3 > /proc/sys/vm/drop_caches will not work: your own shell does the redirection without root rights and answers Permission denied.

free is almost zero

Normal: the page cache took the free memory. Look at available. Worry when available is small and si/so grow in vmstat.

The server is slow, si and so grow

Memory is short, and the kernel shuffles pages between memory and swap. Find who holds the most (ps -eo pid,rss,comm --sort=-rss | head) and stop or limit it (a service's MemoryMax, see oom-killer). Adding swap only buys time; it adds no memory.

A process ended, the log says Killed process

Memory ran out and the kernel ended the process — see oom-killer.

mkswap or swapon: insecure permissions 0644

mkswap prints insecure permissions 0644, fix with: chmod 0600 /swapfile, swapon prints insecure permissions 0644, 0600 suggested. Swap turns on anyway, but any user could read program memory from the file. sudo chmod 600 /swapfile.

swapon failed: Device or resource busy

The file is already on. Check swapon --show.

Swap is gone after a reboot

swapon lasts until a reboot. Add a line to /etc/fstab (recipe above). The line is there, but the boot log says Timed out waiting for device /dev/disk/by-uuid/… and Dependency failed for …swap — the file is listed by UUID=; replace it with the path.

swap.img.swap in the failed list

text

student@ubuntu:~$ systemctl --failed --no-legend
● swap.img.swap loaded failed failed /swap.img
student@ubuntu:~$ sudo swapon -a
swapon: cannot open /swap.img: No such file or directory

systemd makes a <path>.swap unit for every swap line in fstab. The file was deleted but the line stayed — the unit fails on every boot. Remove the line from /etc/fstab, then run sudo systemctl daemon-reload and sudo systemctl reset-failed.

vm.swappiness is back to 60

The value was set with sysctl vm.swappiness=… — it lives until a reboot. Put the line into a file under /etc/sysctl.d/. The kernel rejects values above 200: sysctl: setting key "vm.swappiness": Invalid argument.

swapoff when memory is short

swapoff brings every page from swap back into memory. If there is no room, the command thrashes the disk for a long time, and then the oom-killer comes. Before swapoff, compare used in the Swap row with available.

Memory, swap and the OOM killer

Situation

Task

Signing in is needed to start the machine

Add a swap file

Turn swap off

Drop the cache before a measurement

free is almost zero

The server is slow, si and so grow

A process ended, the log says Killed process

mkswap or swapon: insecure permissions 0644

swapon failed: Device or resource busy

Swap is gone after a reboot

swap.img.swap in the failed list

vm.swappiness is back to 60

swapoff when memory is short