Storage / Stage 1 of 11

Out of space — which resource?

Not running Starting usually ~20 s

Step 6 of 8 · lab · 15 min

Root partition is full

A service stopped saving data: the disk is full. Find what took the space and free it without touching the service's data.

Situation

An application service keeps its data in /var/lib/app and writes its log to /var/log/app. Since this morning it cannot save a single record and reports No space left on device.

Task

Give the service room to write:

the filesystem holding /var/lib/app must have at least 1 GB free;
the service data in /var/lib/app is needed: do not delete or change it.

The tools you need are already installed. There's no internet on this machine.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 2

At least 1 GB is free on the filesystem of /var/lib/app — not done yet
The service data in /var/lib/app is in place and unchanged — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 2

At least 1 GB is free on the filesystem of /var/lib/app — not done yet
The service data in /var/lib/app is in place and unchanged — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 2

At least 1 GB is free on the filesystem of /var/lib/app — not done yet
The service data in /var/lib/app is in place and unchanged — not done yet

Storage and file systems · picked for this lab

Disk space: df, du, inode

Cheat sheet

$_ — insert into the terminal

df -h free space per partition
df -i free inodes per partition
du -xh --max-depth=1 / | sort -h top directories (within one filesystem)
ncdu / interactive overview
sudo lsof +L1 deleted-but-open files
sudo journalctl --vacuum-time=7d trim the system journal
sudo apt clean clear the package cache

df / du / ncdu

bash

df -h                             # free space by partition (human-readable)
df -i                             # free inodes by partition
df -h /var                        # a specific path (which partition it's on)
df -hT                            # + filesystem type

du -sh /var/*                     # size of each subdirectory of /var
du -h --max-depth=1 / | sort -h   # top "heavy" top-level directories
du -xh --max-depth=1 /            # -x: stay within one filesystem (skip /proc, mounts)
du -sh /var/log                   # how much space logs take up
ncdu /                            # interactive overview (the most convenient; apt install ncdu)

ls -lhS | head                    # the largest files in a directory
find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null   # files over 500 MB

-x/-xdev is the key to a clean result: count only the current filesystem, without descending into mounted and virtual ones (/proc, /sys).

What to clean up

bash

sudo journalctl --vacuum-time=7d  # trim the system journal to 7 days
sudo journalctl --disk-usage      # how much space the journal takes up
sudo apt clean                    # cache of downloaded packages
sudo apt autoremove --purge       # old kernels and dependencies
docker system df ; docker system prune -af   # docker images/layers (often huge)
du -sh ~/.cache/*                 # per-user caches

df says "full", but du can't find that much

A file was deleted but is still open by a process — the space isn't freed until it closes:

bash

sudo lsof +L1                     # files with 0 links but still open (see lsof)

Restart the process holding it (often a log that got rotated incorrectly).

No space left on device, even though `df -h` shows free space

You're out of inodes: df -i. Find the directory full of tiny files and clean it up:

bash

for d in /*; do printf '%8s %s\n' "$(find "$d" -xdev 2>/dev/null | wc -l)" "$d"; done | sort -rn

Logs filled up the disk

du -sh /var/log/*; set up rotation (logrotate) and the journal (journalctl --vacuum-time=, see journalctl).

Deleted files, but the space didn't come back

Either a deleted-but-open file (see lsof +L1 above), or it's a different partition — check df -h path.

Out of space — which resource?

Situation

Task

Signing in is needed to start the machine

df / du / ncdu

What to clean up

df says "full", but du can't find that much

No space left on device, even though df -h shows free space

Logs filled up the disk

Deleted files, but the space didn't come back

No space left on device, even though `df -h` shows free space