Storage / Stage 2 of 11 · step 6 of 7

/data is empty after a reboot

~20 min

Not running Starting usually ~20 s

Situation

A service keeps its data on a separate disk that is mounted at /data. After a server reboot the service cannot find its files: the /data directory is empty.

Task

Make sure that:

sudo mount -a completes without errors;
the service's data disk is mounted at /data, with its files in place;
the disk mounts at /data from its /etc/fstab entry, after a reboot too.

Constraints

The data on the disk is needed: do not format it or recreate the service's files.

The tools you need are already installed. There's no internet on this machine.

Lab goals

0 of 4

mount -a completes without errors — not checked yet
The service's data disk is mounted at /data — not checked yet
The disk mounts at /data from its /etc/fstab entry and survives a reboot — not checked yet
The service's files on the disk are intact — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Storage and file systems · picked for this lab

Mounting and fstab

Cheat sheet

$_ — insert into the terminal

lsblk -f devices and filesystems
findmnt the mount tree
blkid /dev/sdX1 UUID for fstab
sudo mount -a apply fstab without a reboot (a check!)
sudo mount -o remount,rw / remount root rw (recovery)
sudo mount --bind /srv/data /var/www/data a bind mount
fuser -vm /mnt who's holding the filesystem (for umount)

/etc/fstab mounts filesystems at boot. Fields: what · where · type · options · dump · pass.

UUID=1234-ABCD  /data  ext4  defaults,noatime  0  2
tmpfs           /scratch tmpfs size=2G,noexec   0  0

pass (the 6th field): 1 for root, 2 for others, 0 — skip fsck.
After editing, always check that you haven't broken booting:

bash

sudo mount -a                     # mount everything from fstab WITHOUT a reboot (you'll see the error right away)
sudo systemctl daemon-reload      # reread the generated mount units
findmnt --verify                  # check fstab for errors

Mounting

bash

sudo mount /dev/sdb1 /mnt/data    # mount manually (temporary, until reboot)
sudo mount UUID=1234-ABCD /mnt    # by UUID
sudo umount /mnt/data             # unmount
sudo umount -l /mnt/data          # lazy: detach now, free it once released (be careful)
blkid /dev/sdb1                   # UUID and fs type (for fstab)
sudo mount -o remount,rw /        # remount root rw (recovery)
sudo mount -o remount,ro /data    # switch to ro on the fly

Common -o options: ro/rw, noexec, nosuid, nodev (security), noatime (speed), _netdev (network filesystems).

Bind mounts and tmpfs

bash

# bind mount: show a directory at another spot in the tree
sudo mount --bind /srv/data /var/www/data
#   in fstab:  /srv/data  /var/www/data  none  bind  0 0

# tmpfs: a filesystem in RAM (fast scratch space, cleared on reboot)
sudo mount -t tmpfs -o size=1G tmpfs /mnt/ram

A new disk in fstab: UUID and nofail

bash

sudo blkid -s UUID -o value /dev/vdb1     # the new filesystem's UUID
sudo mkdir -p /srv/data
echo 'UUID=<uuid> /srv/data ext4 defaults,nofail 0 2' |
  sudo tee -a /etc/fstab
sudo systemctl daemon-reload              # systemd re-reads fstab
sudo findmnt --verify                     # the table has no errors
sudo mount -a && findmnt /srv/data        # mounted from the table

nofail is for data disks. Without it every fstab line is required for boot: if the disk is missing, systemd waits for it and then stops the boot in emergency mode — and someone has to get to the console. With nofail the system boots without the disk, and the directory stays empty. Root / and /boot are never marked this way.

No reboot is needed to test it: sudo umount /srv/data, then sudo mount -a — if the partition is back in the directory, the line is right.

wrong fs type, bad option, bad superblock

mount answers wrong fs type, bad option, bad superblock on /dev/vdb, missing codepage or helper program, or other error and suggests looking at dmesg. Most often the device has no filesystem: a whole new disk, or a partition before mkfs. Check lsblk -f — the FSTYPE column is empty. A new disk is partitioned first (fdisk-parted), then a filesystem is created and the partition is mounted (/dev/vdb1, not /dev/vdb). Less often the cause is a wrong -t type or an option the filesystem does not know; sudo dmesg | tail prints the details.

your fstab has been modified, but systemd still uses the old version

mount prints this hint after fstab is edited: systemd builds its own mount units from the table and has not re-read it yet. Run sudo systemctl daemon-reload.

"target is busy" on umount

Something is using the filesystem. Find what: lsof +D /mnt/data or fuser -vm /mnt/data. Close the processes, leave the directory (cd /), or umount -l (lazy).

An error in fstab → the system didn't boot

A bad line can drop the boot into emergency mode. Always run sudo mount -a/findmnt --verify after editing. In recovery: mount -o remount,rw / and fix fstab.

You mounted it, but the space isn't where you expected

If the mount point already had files in it, they're "hidden" under the mounted filesystem (not deleted). umount brings them back.

A network filesystem hangs the boot

Without _netdev, the system waits for the filesystem before networking is up. Add _netdev (and nofail for non-critical mounts) to the fstab options (see nfs).

The wrong filesystem/device

/dev/sdX names can shift between reboots — mount by UUID (blkid), not by /dev/sdX.

/data is empty after a reboot

Situation

Task

Constraints

Signing in is needed to start the machine

Mounting

Bind mounts and tmpfs

A new disk in fstab: UUID and nofail

wrong fs type, bad option, bad superblock

your fstab has been modified, but systemd still uses the old version

"target is busy" on umount

An error in fstab → the system didn't boot

You mounted it, but the space isn't where you expected

A network filesystem hangs the boot

The wrong filesystem/device