Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

RAID: mirrors, parity and degraded arrays

9 min Linux · md (mdadm 4.3)

RAID combines disks into one device that keeps working after one of them fails (except RAID 0).

Cheat sheet

4 commands Printable version
  • cat /proc/mdstat every array: [UU] or [U_]
  • sudo mdadm --detail /dev/md0 State: clean, degraded, recovering
  • lsblk disks and the array on top (raid1)
  • sudo dmesg | grep md/raid when and which disk dropped out

Overview

RAID (redundant array of independent disks) combines several disks into one block device. Data is spread over the disks so that one failure does not stop the work: the system reads and writes through the remaining ones. The exception is RAID 0: it keeps no spare copies.

On Linux the kernel builds the array (the md driver) and mdadm manages it. The array shows up as /dev/md0; you create a file system (filesystems) or an LVM volume on it, just like on a plain disk. A hardware RAID controller does the same internally and shows the system one ready-made disk; then /proc/mdstat is empty and you check the state with the controller's own tool.

Levels

N is the number of disks in the array, all of the same size.

Level Minimum disks Survives the failure of Usable space How it works
RAID 0 2 no disk N disks stripe: blocks go to each disk in turn
RAID 1 2 N − 1 disks 1 disk mirror: every disk holds a full copy
RAID 5 3 1 disk N − 1 disks stripe plus parity that rebuilds any one disk
RAID 6 4 2 disks N − 2 disks stripe plus two parities
RAID 10 4 1 disk in each pair N / 2 disks stripe over mirrored pairs

RAID 10 on four disks survives two failures if they are in different pairs and loses data if both disks of one pair fail. RAID 5 writes slower than a mirror: every write recalculates parity.

Mirror

RAID 1 writes every block to all disks of the array and reads from any of them. Two 256 MiB disks give a 255 MiB array: part of the space holds the superblock — mdadm's record of the array members at the start of each disk.

text 
$ lsblk /dev/loop11 /dev/loop12
NAME   MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
loop11   7:11   0  256M  0 loop
└─md0    9:0    0  255M  0 raid1
loop12   7:12   0  256M  0 loop
└─md0    9:0    0  255M  0 raid1

Degraded

A degraded array works but has lost part of its redundancy: one or more disks dropped out. The data is intact and available, writes go to the remaining disks. One more failure beyond what the level survives loses data: for a two-disk RAID 1 that is the failure of the last disk.

text 
$ cat /proc/mdstat
md0 : active raid1 loop12[1](F) loop11[0]
      261120 blocks super 1.2 [2/1] [U_]

[2/1] — the array should have two disks, one is working. [U_] is the same by slot: the first slot is filled (U, up), the second is empty. (F) marks a failed (faulty) disk. The kernel log at that moment:

text 
md/raid1:md0: Disk failure on loop12, disabling device.
md/raid1:md0: Operation continuing on 1 devices.

mdadm --detail prints State : clean, degraded or active, degraded (active means the array was written to recently). How to read every field — mdadm.

Rebuild

After a disk is replaced the kernel copies onto the new disk everything that belongs there — a rebuild. In /proc/mdstat it shows as recovery = N% with an estimate finish= and a rate speed=. Checking the copies right after an array is created is called resync and looks the same; the array already works meanwhile.

A rebuild reads every block of the surviving disks. On the lab machine an array of 256 MiB images rebuilds in seconds, a multi-terabyte disk takes hours, and the array stays degraded all that time. The rate is bounded by /proc/sys/dev/raid/speed_limit_min and speed_limit_max (KiB/s, 1000 and 200000 on the lab machine).

Hot spare

A hot spare is a disk in the array without a slot of its own: /proc/mdstat marks it (S), mdadm --detail says spare. When a working disk fails, the kernel takes the spare and starts a rebuild without waiting for a person. A disk added to a healthy array becomes a spare.

RAID is not a backup

RAID protects against one thing: a disk failure. A deleted file, mkfs on the wrong device, a database corrupted by an application, ransomware — the array writes all of it to every copy at once. Only a copy kept elsewhere helps there: restic, rsync, tar.

Recipes

2

Check that an array is healthy

bash 
cat /proc/mdstat                         # [UU] — all in place
sudo mdadm --detail /dev/md0 | grep -E 'State :|Devices :'

A healthy array shows State : clean and Failed Devices : 0. Any degraded, (F) or _ in the square brackets means a disk must be replaced: mdadm.

Practise without disks

An array can be built from any block devices, including loop devices — files attached as disks:

bash 
truncate -s 256M r1.img r2.img
sudo losetup /dev/loop11 r1.img; sudo losetup /dev/loop12 r2.img
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 \
     --run /dev/loop11 /dev/loop12
sudo mdadm /dev/md0 --fail /dev/loop12   # "break" a disk
cat /proc/mdstat                         # [2/1] [U_]

Loop devices are not attached again after a reboot, and an array of them does not assemble by itself: the service that attaches the images does it, or you do it by hand — mdadm.

Troubleshooting

3

[U_] in /proc/mdstat

The array is degraded, the data is still intact. Find the disk that dropped out ((F) in /proc/mdstat, faulty or removed in mdadm --detail), remove it from the array and add a working one — mdadm. Until the rebuild finishes the array has no protection.

The rebuild takes hours

That is expected on large disks: the kernel reads the surviving disks in full. The estimate is finish= in /proc/mdstat. Under load the rebuild slows down, but not below speed_limit_min. Do not power the machine off while it runs.

A disk failed and no mail came

The failure is reported by mdmonitor — the mdadm service that mails the address in MAILADDR of /etc/mdadm/mdadm.conf. Without a mail service the message never leaves: on the lab machine the journal keeps DegradedArray and sendmail: not found. So a check of /proc/mdstat goes into monitoring. More — mdadm.

Danger zone

2

A second failure during a rebuild

While the array is degraded or rebuilding there is no redundancy. Disks of the same batch and age often fail one after another, and a rebuild loads the surviving disk harder than usual. Replace the disk right away and keep a fresh copy of the data outside the array.

RAID 0

RAID 0 keeps no copies: the failure of any disk loses the whole array, and the more disks, the more likely that is. Use it only for data you can afford to lose.

Terms

3
degraded array
A RAID array that lost a disk: the data is intact and available, but no margin is left — the next failure loses data. In /proc/mdstat — [2/1] [U_]. Also appears in: mdadm
array rebuild
Copying data onto a new RAID disk after the failed one is replaced; /proc/mdstat shows recovery = N%. While it runs, the array stays degraded. Also appears in: mdadm, PAM: login checks, password policy and lockout, Dockerfile, Capabilities: root split into parts and 4 more
hot spare
A disk in a RAID array without a slot of its own, marked (S) in /proc/mdstat: when a working disk fails, the kernel starts a rebuild onto it by itself. Also appears in: mdadm

See also

  • mdadm — creating an array, replacing a disk, mdadm.conf.
  • lvm — volumes on top of an array.
  • loop-device — a file instead of a disk for experiments.
  • filesystems — a file system on /dev/md0.
  • lsblk — the tree of disks and arrays.
  • restic — backups that RAID does not replace.

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Topics
Loading…