Storage / Stage 5 of 12

RAID 1: a disk has failed

Not running Starting usually ~20 s

Step 7 of 8 · lab · 25 min

A mirror on one disk

Monitoring reported that the mirror with the accounting documents runs without one disk. The failed disk has been pulled out and a new one attached. Give the array its second disk back so that it also assembles with both after a reboot.

Situation

The accounting documents live in /srv/docs — on the RAID 1 mirror /dev/md0 of two disks. At night monitoring sent DegradedArray: the array runs without one disk. The failed disk has already been pulled out and a new, empty one attached instead. The machine also has a disk with last year's archive attached — it is not mounted, and it must not be touched.

Task

Give the array its second disk back:

The array is clean, both disks are active (2 of 2), the rebuild is over.
The second disk of the array is the new empty disk, the archive on another disk is intact.
/srv/docs is mounted from the array, the documents are intact.
The array line in /etc/mdadm/mdadm.conf matches the current array.
After a reboot the array assembles by itself with both disks.

Once the array has been rebuilt, reboot the machine (sudo reboot) and press "Check": the check looks at how the array was assembled at boot. "Check" waits for the rebuild by itself.

The array state — cat /proc/mdstat, the details — sudo mdadm --detail /dev/md0. There is no internet on the machine.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 5

The documents array is clean, both disks are active (2 of 2), the rebuild is over — not done yet
The second disk of the array is the new empty disk; the archive on another disk is intact — not done yet
/srv/docs is mounted from the array, the documents are intact — not done yet
The array line in /etc/mdadm/mdadm.conf matches the current array — not done yet
After a reboot (sudo reboot) the array assembles by itself with both disks — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 5

The documents array is clean, both disks are active (2 of 2), the rebuild is over — not done yet
The second disk of the array is the new empty disk; the archive on another disk is intact — not done yet
/srv/docs is mounted from the array, the documents are intact — not done yet
The array line in /etc/mdadm/mdadm.conf matches the current array — not done yet
After a reboot (sudo reboot) the array assembles by itself with both disks — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 5

The documents array is clean, both disks are active (2 of 2), the rebuild is over — not done yet
The second disk of the array is the new empty disk; the archive on another disk is intact — not done yet
/srv/docs is mounted from the array, the documents are intact — not done yet
The array line in /etc/mdadm/mdadm.conf matches the current array — not done yet
After a reboot (sudo reboot) the array assembles by itself with both disks — not done yet

Storage and file systems · picked for this lab

mdadm — software RAID: state, disk replacement, mdadm.conf

Cheat sheet

$_ — insert into the terminal

cat /proc/mdstat every array: [UU] or [U_]
sudo mdadm --detail /dev/md0 state and disk roles
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 \
--run /dev/vdb /dev/vdc a mirror of two disks
sudo mdadm /dev/md0 --fail /dev/vdc mark a disk as failed
sudo mdadm /dev/md0 --remove /dev/vdc take the failed one out
sudo mdadm /dev/md0 --add /dev/vdd add a new one, rebuild
sudo mdadm --wait /dev/md0 wait for the rebuild to end
sudo mdadm --detail --scan the ARRAY line for mdadm.conf
sudo update-initramfs -u mdadm.conf into initramfs

The first argument picks a mode: --create, --assemble, --detail, --examine, --monitor, --grow. A command like mdadm /dev/md0 --add /dev/vdd is manage mode: the array first, then actions on disks, several in a row if needed (--fail X --remove X). The full list is in man mdadm.

Create a mirror

text

$ sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 \
    /dev/loop11 /dev/loop12
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array?

The note only matters for an array holding /boot: answer y or add --run, and mdadm does not ask. Then comes mdadm: array /dev/md0 started. and a check of the copies (resync) — the array can already be formatted:

bash

sudo mkfs.ext4 -L data /dev/md0
sudo blkid /dev/md0                 # the file system UUID — for fstab

In /etc/fstab the array is mounted by its file system UUID, with nofail if the system can boot without it — mount. To have the array assembled at boot, record it in mdadm.conf — mdadm.

Replace a failed disk

The order: mark the disk failed, remove it, add a working one, wait for the rebuild.

text

$ sudo mdadm /dev/md0 --fail /dev/loop12
mdadm: set /dev/loop12 faulty in /dev/md0
$ sudo mdadm /dev/md0 --remove /dev/loop12
mdadm: hot removed /dev/loop12 from /dev/md0
$ sudo mdadm /dev/md0 --add /dev/loop13
mdadm: added /dev/loop13
$ cat /proc/mdstat
md0 : active raid1 loop13[2] loop11[0]
      261120 blocks super 1.2 [2/1] [U_]
      [======>..............]  recovery = 31.1% (81664/261120)
        finish=0.1min speed=20416K/sec
$ sudo mdadm --wait /dev/md0
$ cat /proc/mdstat
md0 : active raid1 loop13[2] loop11[0]
      261120 blocks super 1.2 [2/2] [UU]

A disk the kernel already declared failed is marked (F) — it needs no --fail, go straight to --remove.
--remove without --fail is refused for a working disk: hot remove failed … Device or resource busy.
Files on the array stay available the whole time: the rebuild runs under the mounted file system.
--wait returns when the rebuild is over; with 256 MiB images that takes seconds.
The array UUID does not change when a disk is replaced, so the ARRAY line in mdadm.conf stays valid.

Record the array in mdadm.conf

mdadm --detail --scan prints an ARRAY line for every assembled array:

text

$ sudo mdadm --detail --scan
ARRAY /dev/md0 metadata=1.2 UUID=ee095d69:d8d31363:75f3592e:351da00a

bash

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
grep ^ARRAY /etc/mdadm/mdadm.conf   # one line per array, no repeats
sudo update-initramfs -u            # copy mdadm.conf to initramfs

The array is found by its UUID: disk names are not needed in the line. Do not write the --detail --scan --verbose output with devices=/dev/loop11,… into the file — it goes stale after a disk is replaced. A spare in the array adds spares=1 to the line: mdmonitor compares the number of spares against it. Running the tee -a command twice leaves two identical lines, and --assemble --scan refuses to assemble the array — mdadm.

Why update-initramfs -u. Arrays are assembled by the early boot stage — initramfs, and mdadm there reads its own copy of mdadm.conf made when the image was last built. If that copy does not know the array, it is assembled under a number from the top end — /dev/md127 (mdadm) — or not at all. On the lab machine update-initramfs -u takes about 5 seconds and prints update-initramfs: Generating /boot/initrd.img-6.8.0-124-generic; to see that the file made it into the image:

text

$ lsinitramfs /boot/initrd.img-$(uname -r) | grep mdadm.conf
etc/mdadm/mdadm.conf
etc/modprobe.d/mdadm.conf

An array of loop images stored on the root file system does not need the initramfs: it is assembled after the root, by a service — mdadm.

Assemble and stop an array

text

$ sudo umount /mnt/raid
$ sudo mdadm --stop /dev/md0
mdadm: stopped /dev/md0
$ sudo mdadm --assemble --scan
mdadm: /dev/md/0 has been started with 2 drives and 1 spare.

--assemble --scan looks for superblocks on every disk and assembles the arrays from mdadm.conf, or every array it finds if there are no entries. /dev/md/0 is a link to /dev/md0 by the array's name. A mounted array cannot be stopped — umount first (mdadm).

Hot spare

A disk added to a healthy array becomes a spare:

text

$ sudo mdadm /dev/md0 --add /dev/loop12
mdadm: added /dev/loop12
$ cat /proc/mdstat
md0 : active raid1 loop12[3](S) loop13[2] loop11[0]
      261120 blocks super 1.2 [2/2] [UU]

When a working disk fails, the kernel starts a rebuild onto the spare by itself. To have a spare from the start, create the array with --spare-devices=1.

Retire a disk from an array

A disk removed from an array keeps its superblock: mdadm --examine shows the old array UUID and role. So that the disk is no longer recognised as an array member:

text

$ sudo mdadm --zero-superblock /dev/loop12
$ sudo mdadm --examine /dev/loop12
mdadm: No md superblock detected on /dev/loop12.

Grow an array

bash

sudo mdadm /dev/md0 --add /dev/vde           # joins as a spare
sudo mdadm --grow /dev/md0 --raid-devices=3  # and gets a slot

In RAID 1 a third disk is one more copy, the size stays the same. In RAID 5 the size grows: after the array is reshaped, grow the file system (resize2fs) or the LVM volume.

Failure alerts

mdmonitor.service (mdadm --monitor --scan) watches the arrays and mails the MAILADDR address from mdadm.conf. On the lab machine the service is static: nobody enables it, it starts by itself when an array appears (systemctl is-active mdmonitor → active). The mdcheck_start.timer timer checks the copies of every array on the first Sunday of the month at 01:00. The mail only leaves if mail is set up: the lab machine has none, and the journal keeps a DegradedArray event and sendmail: not found. So /proc/mdstat is also checked by monitoring. student can read /proc/mdstat; mdadm --detail needs sudo.

hot remove failed … Device or resource busy

text

mdadm: hot remove failed for /dev/loop12: Device or resource busy

The disk is still working: the kernel does not give it up while it is in service. --fail first, then --remove, or both at once: mdadm /dev/md0 --fail /dev/loop12 --remove /dev/loop12.

Cannot open … Device or resource busy

text

mdadm: Cannot open /dev/loop11: Device or resource busy

--add or --create with a disk that is already in use: it belongs to this or another array, is mounted or open. Check lsblk and cat /proc/mdstat — most likely the disk is the wrong one.

create aborted

Nobody answered y to Continue creating array? (for example, the command ran from a script without a terminal). Add --run.

Cannot get exclusive access

text

mdadm: Cannot get exclusive access to /dev/md0:Perhaps a running
  process, mounted filesystem or active volume group?

--stop on an array that is in use: a file system is mounted on it, a file is open or an LVM volume group lives on it. sudo umount first (or vgchange -an for a volume group), then --stop.

Duplicate MD device names in conf file

text

mdadm: Devices UUID-1ba9062a:… and UUID-1ba9062a:… have the same
  name: /dev/md0
mdadm: Duplicate MD device names in conf file were found.

/etc/mdadm/mdadm.conf has two ARRAY lines with one name — usually mdadm --detail --scan | tee -a ran twice. --assemble --scan assembles nothing. Keep one line per array (grep ^ARRAY /etc/mdadm/mdadm.conf) and update the boot image: sudo update-initramfs -u.

Cannot remove …, array will be failed

text

mdadm: Cannot remove /dev/loop11 from /dev/md127, array will be failed.

--fail on the last working disk: the kernel keeps it, the array stays degraded and keeps working. Check which disk actually dropped out — cat /proc/mdstat.

/dev/md127 instead of /dev/md0 after a reboot

The array is assembled, but under a number from the top end. The superblock holds an array name like ubuntu:0 — machine and number. When mdadm cannot find the array in mdadm.conf and the machine is not the one in the name (the mdadm.conf copy in the initramfs does not know the array, the disks came from another server), it takes a free number from the top and a link by the full name:

text

$ sudo mdadm --assemble --scan --homehost=otherhost
mdadm: /dev/md/ubuntu:0 has been started with 2 drives.
$ ls -l /dev/md/
lrwxrwxrwx 1 root root 8 Oct 27 15:03 ubuntu:0 -> ../md127

The data is intact. Record the array and update the boot image — mdadm. Mount file systems on the array by UUID: then the md number does not matter in /etc/fstab.

The array was not assembled at boot

There is no md0 line in /proc/mdstat, and mdadm --detail /dev/md0 prints mdadm: cannot open /dev/md0: No such file or directory. Causes in order: the disks are not there yet when assembly runs, there is no ARRAY line or it has a wrong UUID (compare with mdadm --examine --scan), the boot image was not updated. Assemble by hand — sudo mdadm --assemble --scan. If the array was in /etc/fstab without nofail, the boot stops in emergency mode — boot-targets.

An array of loop devices does not assemble by itself at boot — neither with an ARRAY line nor after update-initramfs: the udev rule assembles an array when a disk appears, and a loop device gets its contents later, at losetup. The same service that attaches the images assembles it: after losetup — udevadm settle and mdadm --assemble --scan --run. --run starts a degraded array too; without it an array missing a disk does not start at boot.

The old disk came back as a spare

A disk that was removed and added again becomes (S) in a healthy array. That is expected: its slot is already taken by the disk that replaced it.

--create on disks with data

mdadm --create on disks that already hold data or another array overwrites the superblock and can destroy the data. An existing array is assembled with --assemble; --create is only for disks you are sure about.

--zero-superblock

Erases the array record from a disk for good. Use it only on a disk already removed from the array and no longer needed in it.

mkfs on /dev/md0

mkfs on top of the array wipes its file system on every disk at once: RAID only protects against a disk failure — raid.

1 · Foundations

2 · The system

3 · Admin craft

4 · Server in production

5 · Operations

RAID 1: a disk has failed

A mirror on one disk

Situation

Task

Goals

Hints

Lab goals

Lab goals

mdadm — software RAID: state, disk replacement, mdadm.conf

Cheat sheet

Create a mirror

Replace a failed disk

Record the array in mdadm.conf

Assemble and stop an array

Hot spare

Retire a disk from an array

Grow an array

Failure alerts

hot remove failed … Device or resource busy

Cannot open … Device or resource busy

create aborted

Cannot get exclusive access

Duplicate MD device names in conf file

Cannot remove …, array will be failed

/dev/md127 instead of /dev/md0 after a reboot

The array was not assembled at boot

The old disk came back as a spare

--create on disks with data

--zero-superblock

mkfs on /dev/md0

RAID 1: a disk has failed

Situation

Task

Signing in is needed to start the machine

Create a mirror

Replace a failed disk

Record the array in mdadm.conf

Assemble and stop an array

Hot spare

Retire a disk from an array

Grow an array

Failure alerts

hot remove failed … Device or resource busy

Cannot open … Device or resource busy

create aborted

Cannot get exclusive access

Duplicate MD device names in conf file

Cannot remove …, array will be failed

/dev/md127 instead of /dev/md0 after a reboot

The array was not assembled at boot

The old disk came back as a spare

--create on disks with data

--zero-superblock

mkfs on /dev/md0