Linux Labs
Linux Labs
EN

Storage / Stage 5 of 12

RAID 1: a disk has failed

Not running Starting usually ~20 s
Leave the lesson

Step 4 of 8 · on a machine · 6 min

Replacing a disk

Here: how to give the mirror its second disk back — remove the failed one, add a new one and wait for the rebuild.

The failed disk is taken out of the array with mdadm --remove. Only a disk marked (F) or a spare (S) can be removed: mdadm refuses a working disk (hot remove failed … Device or resource busy), so a working but suspicious disk is first marked by hand with mdadm --fail. On a server the disk is then swapped physically, and the new one is added with mdadm --add (in the Wiki).

bash 
sudo mdadm /dev/md0 --fail /dev/loop1    # mark the disk as failed
sudo mdadm /dev/md0 --remove /dev/loop1  # take it out of the array
sudo mdadm /dev/md0 --add /dev/loop2     # add the new disk

The new disk joins as a spare, (S), and the array starts a rebuild (recovery) at once: the kernel copies the whole content of the working disk onto it. You can follow it in /proc/mdstat — the line recovery = 37.5% with an estimate finish=. The 128 MiB training disks rebuild in seconds, disks of several terabytes take hours. While the rebuild runs, the array stays degraded and the filesystem works as usual.

Exercise · do it

Replace the failed disk

Remove the failed disk /dev/loop1 from the /dev/md0 array, add the new disk /dev/loop2 in its place and look at /proc/mdstat to see how the rebuild went.

Sign in and start The machine is available after signing in

Goals · 0 of 2

  • Remove the failed disk from the array
  • Give the mirror its second disk back
Check yourself 1 question

The mirror's rebuild onto the new disk has reached 40% when the other disk, the one it copies from, fails. What happens to the array's data?

  • A rebuild copies the disk's blocks in order, not files. The new disk holds the first 40% of the blocks, and no filesystem can be put together from them.

  • There is nothing to switch to: until the rebuild ends the new disk is not a working one — mdadm --detail calls it spare rebuilding, and /proc/mdstat keeps showing [2/1] [U_].

  • The new disk was the target of the rebuild: it holds no full copy yet, and there is nothing to continue from.

Until the rebuild is over, the only full copy sits on one disk, and losing it costs all the data.

Found an inaccuracy? Report an error in the lesson

Step goals

0 of 2
  • Remove the failed disk from the array
  • Give the mirror its second disk back
Ubuntu 24.04

Step goals

0 of 2
  • Remove the failed disk from the array
  • Give the mirror its second disk back
Loading…