Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

Memory and swap: free, available and the page cache

11 min Ubuntu 24.04 · Linux 6.8 · procps 4.0

How to read free: used and free memory, the page cache and available; swap, a swap file and vm.swappiness.

Cheat sheet

7 commands Printable version
  • free -h memory and swap; the reserve is available
  • grep -E '^(Mem|Cached)' /proc/meminfo same from the kernel, KiB
  • ps -eo pid,rss,comm --sort=-rss | head -5 who holds the most
  • swapon --show active swap; empty means no swap
  • vmstat 1 5 si and so — swap reads and writes
  • sysctl vm.swappiness kernel's eagerness to swap, 60 by default
  • sync; echo 3 | sudo tee /proc/sys/vm/drop_caches drop the cache

Overview

A machine's memory is shared by programs, the kernel and the file cache. free -h shows how much goes where. On the 512 MiB training machine:

total used free shared buff/cache available
Mem 458Mi 153Mi 5.4Mi 964Ki 320Mi 304Mi
Swap 0B 0B 0B

Columns of free

Column Meaning
total memory the kernel manages
used taken by programs and the kernel
free not used by anything
shared shared process memory and tmpfs
buff/cache page cache and buffers — copies of disk data in memory
available how much can go to new programs without swap

total is below 512 MiB: the kernel took part of the memory at boot for its code and internal tables (sudo dmesg | grep Memory:).

Page cache and available

The kernel keeps copies of files it has read and written in memory — the page cache. Reading again comes from memory, without the disk. Almost all free memory goes to the cache, so free on a running machine is close to zero. When a program needs memory, the kernel evicts cache pages: their copy is on disk anyway.

available is free memory plus the part of the cache the kernel can release. That is the real reserve. An experiment on the training machine: read many files, then drop the cache.

Moment free buff/cache available
after cat /usr/bin/* > /dev/null 5.4Mi 320Mi 304Mi
after dropping the cache 266Mi 59Mi 304Mi

free grew fifty times, while available did not change: the cache was available to programs before the drop too. The same numbers in KiB are in /proc/meminfo:

text 
student@ubuntu:~$ grep -E '^(Mem|Cached)' /proc/meminfo
MemTotal:         469236 kB
MemFree:            5540 kB
MemAvailable:     311944 kB
Cached:           301900 kB

Process memory: RSS

RSS is how much of a process's memory is in RAM right now, in KiB. VSZ is how many addresses it reserved; most of them may be unused. Process memory is judged by RSS: the RSS column in ps, RES in top, VmRSS in /proc/<pid>/status.

text 
student@ubuntu:~$ ps -o pid,rss,vsz,pmem,comm -C systemd-journald,sshd
    PID   RSS    VSZ %MEM COMMAND
    275 16792  42256  3.5 systemd-journal
    773  8244  12028  1.7 sshd

The sum of all RSS can exceed used: shared libraries sit in memory once, but count in full in the RSS of each process.

Swap

Swap is disk space where the kernel moves rarely needed program pages to free memory. The disk is many times slower than memory: while swap helps, programs keep working, only slower. Without swap the kernel calls the oom-killer as soon as memory runs out.

The training machine has no swap: swapon --show prints nothing, and free shows Swap: 0B 0B 0B. Swap can be a disk partition or a file; a file is easier to add and remove.

Swap activity shows in vmstat 1: si is how many KiB per second are read back from swap into memory, so is how many are written to swap. A 380 MiB process under memory shortage produced these lines (some columns omitted):

r b swpd free si so wa
0 1 131688 5920 152012 150756 56
0 1 130904 5096 135084 130312 53

si and so at about 150 MiB per second, wa above 50: the machine waits for the disk more than it works. Who holds how much in swap is VmSwap in /proc/<pid>/status.

vm.swappiness

The kernel parameter vm.swappiness (0 to 200, 60 on Ubuntu) sets how eagerly the kernel, short of memory, moves program pages to swap compared with cache pages. A lower value keeps programs in memory whole for longer. Servers often use 10.

bash 
sysctl vm.swappiness                     # current value
sudo sysctl vm.swappiness=10             # until reboot
echo 'vm.swappiness = 10' \
  | sudo tee /etc/sysctl.d/90-swappiness.conf
sudo sysctl --system                     # apply sysctl.d files

A value from a file in /etc/sysctl.d/ survives a reboot. Kernel parameters in detail — sysctl.

Recipes

3

Add a swap file

bash 
sudo fallocate -l 512M /swapfile    # a file of the right size
sudo chmod 600 /swapfile            # readable by root only
sudo mkswap /swapfile               # format it as swap
sudo swapon /swapfile               # turn it on
swapon --show
text 
student@ubuntu:~$ swapon --show
NAME      TYPE SIZE USED PRIO
/swapfile file 512M   0B   -2

To turn swap on at boot, add a line to /etc/fstab:

bash 
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo swapoff /swapfile; sudo swapon -a; swapon --show   # check

swapon -a turns on everything listed in fstab: if the file is back in the list, the line has no typos. Instead of fallocate, sudo dd if=/dev/zero of=/swapfile bs=1M count=512 works too — slower, but on any filesystem.

Three details people get wrong:

  • Size. 512M means 512 MiB (536870912 bytes), 512MB means 512,000,000 bytes. The "Data size converter" tool turns one into the other.
  • Permissions. chmod 600 goes before swapon: otherwise any user could read program memory from the file.
  • Path in fstab. A swap file goes into fstab by its path, /swapfile. A UUID=… none swap sw 0 0 line for a file passes the swapon -a check, but at boot systemd waits for the device /dev/disk/by-uuid/…, and such links exist only for disks and partitions. After 90 seconds the journal says Timed out waiting for device, and swap stays off.

Turn swap off

bash 
sudo swapoff /swapfile   # pages from swap return to memory

Then remove the line from /etc/fstab and delete the file.

Drop the cache before a measurement

bash 
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

This is only for experiments: to measure reading from disk rather than from the cache. It does not help normal work; the cache fills up again. sudo echo 3 > /proc/sys/vm/drop_caches will not work: your own shell does the redirection without root rights and answers Permission denied.

Troubleshooting

8

free is almost zero

Normal: the page cache took the free memory. Look at available. Worry when available is small and si/so grow in vmstat.

The server is slow, si and so grow

Memory is short, and the kernel shuffles pages between memory and swap. Find who holds the most (ps -eo pid,rss,comm --sort=-rss | head) and stop or limit it (a service's MemoryMax, see oom-killer). Adding swap only buys time; it adds no memory.

A process ended, the log says Killed process

Memory ran out and the kernel ended the process — see oom-killer.

mkswap or swapon: insecure permissions 0644

mkswap prints insecure permissions 0644, fix with: chmod 0600 /swapfile, swapon prints insecure permissions 0644, 0600 suggested. Swap turns on anyway, but any user could read program memory from the file. sudo chmod 600 /swapfile.

swapon failed: Device or resource busy

The file is already on. Check swapon --show.

Swap is gone after a reboot

swapon lasts until a reboot. Add a line to /etc/fstab (recipe above). The line is there, but the boot log says Timed out waiting for device /dev/disk/by-uuid/… and Dependency failed for …swap — the file is listed by UUID=; replace it with the path.

swap.img.swap in the failed list

text 
student@ubuntu:~$ systemctl --failed --no-legend
● swap.img.swap loaded failed failed /swap.img
student@ubuntu:~$ sudo swapon -a
swapon: cannot open /swap.img: No such file or directory

systemd makes a <path>.swap unit for every swap line in fstab. The file was deleted but the line stayed — the unit fails on every boot. Remove the line from /etc/fstab, then run sudo systemctl daemon-reload and sudo systemctl reset-failed.

vm.swappiness is back to 60

The value was set with sysctl vm.swappiness=… — it lives until a reboot. Put the line into a file under /etc/sysctl.d/. The kernel rejects values above 200: sysctl: setting key "vm.swappiness": Invalid argument.

Danger zone

1

swapoff when memory is short

swapoff brings every page from swap back into memory. If there is no room, the command thrashes the disk for a long time, and then the oom-killer comes. Before swapoff, compare used in the Swap row with available.

Terms

3
page cache
Copies of files that were read and written, kept in memory so that reading again skips the disk. In free it is buff/cache; the kernel frees it when programs need memory, so the reserve is available. Also appears in: OOM killer: when memory runs out, System monitoring: where to start when the server is slow, iostat and pidstat
swap
A disk partition or file where the kernel moves rarely needed program pages to free memory. It helps when memory is short, but the disk is slower than memory: busy swap means slowness. Also appears in: System monitoring: where to start when the server is slow, OOM killer: when memory runs out, /proc: processes and the kernel as files, top and 10 more
RSS (resident set size)
How much of a process's memory is in RAM right now, in KiB: the RSS column in ps, RES in top, VmRSS in /proc/<pid>/status. Shared libraries count in the RSS of every process. Also appears in: OOM killer: when memory runs out, /proc: processes and the kernel as files, iostat and pidstat, ps and 1 more

See also

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Tools on this topic

Topics
Loading…