Inside the system: resources, kernel, processes / Stage 2 of 12 · step 8 of 9

A build runs out of memory

~20 min

Not running Starting usually ~20 s

Situation

Every night the lab-build.timer timer starts the search index build of the docs site — the lab-build service. The build is given 192 MiB of RAM: the site and the database need the rest. There are more pages now, and the index has not been updated for several nights in a row. The build can be started in the daytime too: sudo systemctl start lab-build.

The build cannot be given more RAM. The decision is to give the machine swap: whatever does not fit into 192 MiB goes to the disk — the build runs slower, but it finishes.

Task

The swap is a file of exactly 512 MiB, and it is on. Process memory ends up in swap, so only root may read and write the file (mode 600).
After a reboot the swap comes on by itself, and the swap lines in /etc/fstab cause no errors at boot.
The system is reluctant to send memory to swap: vm.swappiness = 10 now and after a reboot, and the setting is written as a file in /etc/sysctl.d.
The sudo systemctl start lab-build build passes — the check will start it itself. Do not change the build unit or its memory limit.

The machine has no internet access; the programs you need are already installed.

Lab goals

0 of 4

A swap file of exactly 512 MiB is on, owned by root with mode 600 — not checked yet
The swap file is listed in /etc/fstab, and there are no swap lines for files that do not exist — not checked yet
vm.swappiness = 10 now and after a reboot, from a file in /etc/sysctl.d — not checked yet
The nightly lab-build build passes within the same 192 MiB memory limit — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Processes · picked for this lab

Memory and swap: free, available and the page cache

Cheat sheet

$_ — insert into the terminal

free -h memory and swap; the reserve is available
grep -E '^(Mem|Cached)' /proc/meminfo same from the kernel, KiB
ps -eo pid,rss,comm --sort=-rss | head -5 who holds the most
swapon --show active swap; empty means no swap
vmstat 1 5 si and so — swap reads and writes
sysctl vm.swappiness kernel's eagerness to swap, 60 by default
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches drop the cache

Add a swap file

bash

sudo fallocate -l 512M /swapfile    # a file of the right size
sudo chmod 600 /swapfile            # readable by root only
sudo mkswap /swapfile               # format it as swap
sudo swapon /swapfile               # turn it on
swapon --show

text

student@ubuntu:~$ swapon --show
NAME      TYPE SIZE USED PRIO
/swapfile file 512M   0B   -2

To turn swap on at boot, add a line to /etc/fstab:

bash

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo swapoff /swapfile; sudo swapon -a; swapon --show   # check

swapon -a turns on everything listed in fstab: if the file is back in the list, the line has no typos. Instead of fallocate, sudo dd if=/dev/zero of=/swapfile bs=1M count=512 works too — slower, but on any filesystem.

Three details people get wrong:

Size. 512M means 512 MiB (536870912 bytes), 512MB means 512,000,000 bytes. The "Data size converter" tool turns one into the other.
Permissions. chmod 600 goes before swapon: otherwise any user could read program memory from the file.
Path in fstab. A swap file goes into fstab by its path, /swapfile. A UUID=… none swap sw 0 0 line for a file passes the swapon -a check, but at boot systemd waits for the device /dev/disk/by-uuid/…, and such links exist only for disks and partitions. After 90 seconds the journal says Timed out waiting for device, and swap stays off.

Turn swap off

bash

sudo swapoff /swapfile   # pages from swap return to memory

Then remove the line from /etc/fstab and delete the file.

Drop the cache before a measurement

bash

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

This is only for experiments: to measure reading from disk rather than from the cache. It does not help normal work; the cache fills up again. sudo echo 3 > /proc/sys/vm/drop_caches will not work: your own shell does the redirection without root rights and answers Permission denied.

free is almost zero

Normal: the page cache took the free memory. Look at available. Worry when available is small and si/so grow in vmstat.

The server is slow, si and so grow

Memory is short, and the kernel shuffles pages between memory and swap. Find who holds the most (ps -eo pid,rss,comm --sort=-rss | head) and stop or limit it (a service's MemoryMax, see oom-killer). Adding swap only buys time; it adds no memory.

A process ended, the log says Killed process

Memory ran out and the kernel ended the process — see oom-killer.

mkswap or swapon: insecure permissions 0644

mkswap prints insecure permissions 0644, fix with: chmod 0600 /swapfile, swapon prints insecure permissions 0644, 0600 suggested. Swap turns on anyway, but any user could read program memory from the file. sudo chmod 600 /swapfile.

swapon failed: Device or resource busy

The file is already on. Check swapon --show.

Swap is gone after a reboot

swapon lasts until a reboot. Add a line to /etc/fstab (recipe above). The line is there, but the boot log says Timed out waiting for device /dev/disk/by-uuid/… and Dependency failed for …swap — the file is listed by UUID=; replace it with the path.

swap.img.swap in the failed list

text

student@ubuntu:~$ systemctl --failed --no-legend
● swap.img.swap loaded failed failed /swap.img
student@ubuntu:~$ sudo swapon -a
swapon: cannot open /swap.img: No such file or directory

systemd makes a <path>.swap unit for every swap line in fstab. The file was deleted but the line stayed — the unit fails on every boot. Remove the line from /etc/fstab, then run sudo systemctl daemon-reload and sudo systemctl reset-failed.

vm.swappiness is back to 60

The value was set with sysctl vm.swappiness=… — it lives until a reboot. Put the line into a file under /etc/sysctl.d/. The kernel rejects values above 200: sysctl: setting key "vm.swappiness": Invalid argument.

swapoff when memory is short

swapoff brings every page from swap back into memory. If there is no room, the command thrashes the disk for a long time, and then the oom-killer comes. Before swapoff, compare used in the Swap row with available.

A build runs out of memory

Situation

Task

Signing in is needed to start the machine

Add a swap file

Turn swap off

Drop the cache before a measurement

free is almost zero

The server is slow, si and so grow

A process ended, the log says Killed process

mkswap or swapon: insecure permissions 0644

swapon failed: Device or resource busy

Swap is gone after a reboot

swap.img.swap in the failed list

vm.swappiness is back to 60

swapoff when memory is short