As a starting point, since someone is sure to mention it, I KNOW THAT RAID IS NOT A BACKUP!!. On the other hand I’ve yet to meet someone who can tell me an affordable way to back up a 12TB array short of buying enough disks to clone it.
I’ve had a home-build linux md server for quite some time now. Brought the array online ~2011 and have grown it multiple times since then. It was at 7 x 2TB (apparently a bad idea all by itself, not that the software would warn you when you add the disk to the pool) disks and over the years had survived disk failures, power cuts while being grown, and 2 house moves.
Basic Setup
- Linux MD disk array
- 7 x 2TB 3.5″ SATA Disks
- Whole disk (or array in this case) encryption (cryptsetup)
- XFS Filesystem on top
Where it started to go wrong
It started, as these things usually do with an e-mail from my server, cheerfully informing me that one of the disks had died and that the raid array was degraded. Not really a problem as this had happened before without major drama.
- Order a new disk
- Figure out which one is /dev/sdf
hwinfo --disks helps a lot for this since it can get serial numbers
- Shut the server down
- Swap in the new disk
- Add it to the array, and take things easy while it rebuilds (approx time required ~10 hours when disks are not being used)
And that was when I hit the dreaded Unrecoverable Read Error 🙁 Much has been written about this, but it was my first experience of it (and won’t forget it in a hurry).
[ 772.229943] md: bind<sdh>
[ 772.793789] RAID conf printout:
[ 772.793794] --- level:5 rd:7 wd:6
[ 772.793798] disk 0, o:1, dev:sdg
[ 772.793802] disk 1, o:1, dev:sdh
[ 772.793805] disk 2, o:1, dev:sdi
[ 772.793809] disk 3, o:1, dev:sdf
[ 772.793812] disk 4, o:1, dev:sdd
[ 772.793816] disk 5, o:1, dev:sdb
[ 772.793819] disk 6, o:1, dev:sda
[ 772.794310] md: recovery of RAID array md0
[ 772.794316] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 772.794368] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 772.794378] md: using 128k window, over a total of 1953514496k.
...
[35295.808096] ata9.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x6 frozen
[35295.808105] ata9.00: failed command: READ FPDMA QUEUED
[35295.808116] ata9.00: cmd 60/08:00:20:01:ce/00:00:ab:00:00/40 tag 0 ncq 4096 in
[35295.808118] res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[35295.808124] ata9.00: status: { DRDY }
[35397.139031] sd 8:0:0:0: [sdf] Add. Sense: Unrecovered read error - auto reallocate failed
[35397.139041] sd 8:0:0:0: [sdf] CDB: Read(10): 28 00 ab cd f8 58 00 03 08 00
[35397.139057] end_request: I/O error, dev sdf, sector 2882403160
[35397.139066] md/raid:md0: read error not correctable (sector 2882403160 on sdf).
[35397.139073] md/raid:md0: Disk failure on sdf, disabling device.
[35397.139075] md/raid:md0: Operation continuing on 5 devices.
[35397.139116] ata9: EH complete
[35397.708320] md: md0: recovery done.
[35397.822385] RAID conf printout:
[35397.822391] --- level:5 rd:7 wd:5
[35397.822397] disk 0, o:1, dev:sdg
[35397.822401] disk 1, o:1, dev:sdh
[35397.822405] disk 2, o:1, dev:sdi
[35397.822409] disk 3, o:0, dev:sdf
[35397.822412] disk 4, o:1, dev:sdd
[35397.822416] disk 5, o:1, dev:sdb
[35397.822419] disk 6, o:1, dev:sda
[35397.824183] RAID conf printout:
[35397.824188] --- level:5 rd:7 wd:5
[35397.824191] disk 0, o:1, dev:sdg
[35397.824195] disk 2, o:1, dev:sdi
[35397.824198] disk 3, o:0, dev:sdf
[35397.824201] disk 4, o:1, dev:sdd
[35397.824205] disk 5, o:1, dev:sdb
[35397.824208] disk 6, o:1, dev:sda
[35397.824218] RAID conf printout:
[35397.824221] --- level:5 rd:7 wd:5
[35397.824224] disk 0, o:1, dev:sdg
[35397.824228] disk 2, o:1, dev:sdi
[35397.824231] disk 3, o:0, dev:sdf
[35397.824234] disk 4, o:1, dev:sdd
[35397.824237] disk 5, o:1, dev:sdb
[35397.824240] disk 6, o:1, dev:sda
[35397.824380] RAID conf printout:
[35397.824386] --- level:5 rd:7 wd:5
[35397.824390] disk 0, o:1, dev:sdg
[35397.824394] disk 2, o:1, dev:sdi
[35397.824398] disk 4, o:1, dev:sdd
[35397.824401] disk 5, o:1, dev:sdb
[35397.824404] disk 6, o:1, dev:sda
At which point the array is basically dead. Which seems a bit insane – read error in one sector makes 12TB of data unavailable but there you go. We apparently can’t mount, rebuild or do anything – bear in mind that at this point all our data is still intact, except for that one dodgy sector. Looking at the raid array in more detail (it is a mess):
[~]=> sudo mdadm --detail /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Mon May 16 22:34:59 2011
Raid Level : raid5
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Raid Devices : 7
Total Devices : 5
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Apr 29 12:28:08 2015
State : active, FAILED, Not Started
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
UUID : 4a3bba9b:8299bd9b:3cb2199d:f2060f75
Events : 0.1434672
Number Major Minor RaidDevice State
0 8 96 0 active sync /dev/sdg
1 0 0 1 removed
2 8 128 2 active sync /dev/sdi
3 0 0 3 removed
4 8 48 4 active sync /dev/sdd
5 8 16 5 active sync /dev/sdb
6 0 0 6 removed
7 8 112 - spare /dev/sdh
Pay close attention to the metadata version – this is where I went horribly wrong (more on that later). From the list of disks we have /dev/sdf removed automatically due to the read error and /dev/sdh in as a spare since it was in the process of being synced when everything went pear shaped.
Steps to recovery
At this point I was quite annoyed and a bit desperate so I concocted a plan that would have worked, but for one mistake. My plan was simple
- Remove /dev/sdh from the md0 set – it’s basically useless now anyway
- Get /dev/sdf back in by recreating the array
- Run XFS checks on the file system to be sure that is ok
- Use gddrescue to clone the slightly dodgy /dev/sdf to /dev/sdh – with multiple retries to try and get the problem sector
- Reassemble the array using /dev/sdh rather then /dev/sdf
- Buy another new disk and slot it in and hope the rebuild works this time
- Low level format /dev/sdf to get the sector marked as bad. If that fails, bin it.
- Convert to RAID-6 so this can’t happen again
Step 1 went as expected:
[~]=> sudo mdadm --remove /dev/md0 /dev/sdh
mdadm: hot removed /dev/sdh from /dev/md0
[~]=> sudo mdadm --detail /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Mon May 16 22:34:59 2011
Raid Level : raid5
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Raid Devices : 7
Total Devices : 5
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Wed Apr 29 12:41:58 2015
State : active, FAILED, Not Started
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : 4a3bba9b:8299bd9b:3cb2199d:f2060f75
Events : 0.1434673
Number Major Minor RaidDevice State
0 8 96 0 active sync /dev/sdg
1 0 0 1 removed
2 8 128 2 active sync /dev/sdi
3 8 80 3 removed
4 8 48 4 active sync /dev/sdd
5 8 16 5 active sync /dev/sdb
6 8 0 6 active sync /dev/sda
So I decided to re-create the array rather then re-assemble (with assume-clean), and here I went very wrong. I didn’t specify the metadata version.
MD RAID Metadata Versions
It used to be the default format of raid superblock during array creation on most distributions until 2009, the older version-0.90 superblock format had several limitations that limited its applicability for use on large arrays or arrays with many component devices.
The version-0.90 superblock limits the number of component devices within an array to 28, and limits each component device to a maximum size of 2TB on kernel version <3.1 and 4TB on kernel version >=3.1.
The superblock is 4K long and is written into a 64K aligned block that starts at least 64K and less than 128K from the end of the device (i.e. to get the address of the superblock round the size of the device down to a multiple of 64K and then subtract 64K). The available size of each device is the amount of space before the super block, so between 64K and 128K is lost when a device in incorporated into an MD array.
Also, and this is important – it is stored at the end of the device. If you want mdadm to use this you have to specify –metadata=0.9
The shiny new (and now the default) is version 1.2 which grabs 4K from the beginning of the device, and while doing so torches the XFS superblock.
XFS Gone
When I realised my mistake, got the drives into a sort of working order, and mounted, I was presented with
[~]=> sudo mount /dev/mapper/storage /mnt/storage
mount: /dev/mapper/storage: can't read superblock
From our message log
[ 1337.665651] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
[ 1337.666154] SGI XFS Quota Management subsystem
[ 1337.666931] XFS (dm-3): Mounting Filesystem
[ 1339.452417] XFS (dm-3): Log inconsistent (didn't find previous header)
[ 1339.452441] XFS (dm-3): failed to find log head
[ 1339.452447] XFS (dm-3): log mount/recovery failed: error 5
[ 1339.452524] XFS (dm-3): log mount failed
xfs_repair, after throwing around many dots seemed to be completely unhappy with any of the secondary superblocks, and UFS Recover could only re-assemble some bits and pieces.
And that was the end of that.
Lessons learned
- Never more the 4 devices in RAID-5
- Always make a backup
- Be very very careful when trying to repair
- For tools that have been in service for longer then a few months, be very very careful. Some idiot developer has probably changed the defaults in a way that will screw you
