It appears that the 27-component-device limit is specific to
non-metadata arrays ("mdadm --build"). More research:
When the RAID assembly fails --
# mdadm --build /dev/md/md-test --level=linear --raid-devices=28
/dev/loop{0..27}
mdadm: ADD_NEW_DISK failed for /dev/loop27: Device or resource busy
-- this shows up in the kernel log:
kernel: [126679.205703] md: nonpersistent superblock ...
kernel: [126679.205718] md: md125: array is limited to 27 devices
kernel: [126679.205722] md: export_rdev(loop27)
kernel: [126679.221644] md: md125 stopped.
The problem seems to be some kind of kernel limit. A Google search finds
this book:
https://books.google.ca/books?id=EbWbAgAAQBAJ&pg=PT75&lpg=PT75
quote: "MD_SB_DISKS indicates that each software array is limited to 27
member disks."
This appears to be a constant relating to the MD-RAID v0.90 superblock
format:
http://lxr.free-electrons.com/ident?i=MD_SB_DISKS
some discussion:
* The version-0.90 Superblock Format *
Though it used to be the default format of raid superblock during
array creation on most distributions until 2009, the older
version-0.90 superblock format has several limitations that limit
its applicability for use on large arrays or arrays with many
component devices.
The version-0.90 superblock limits the number of component devices
within an array to 28, and limits each component device to a maximum
size of 2TB on kernel version <3.1 and 4TB on kernel version >=3.1.
...
* The version-1 Superblock Format *
The newer and well-supported version-1 superblock format is
more-expansion friendly than the previous format. It is the default
as of v3.1.1. More specifically, --metadata=1.2 is used as of
v3.1.2.
The version-1 superblock is capable of supporting arrays with 384+
component devices, and supports arrays with 64-bit sector lengths.
https://raid.wiki.kernel.org/index.php/RAID_superblock_formats
When the RAID assembly succeeds --
# mdadm --build /dev/md/md-test --level=linear --raid-devices=27
/dev/loop{0..26}
mdadm: array /dev/md/md-test built and started.
#
# mdadm --detail /dev/md/md-test
/dev/md/md-test:
Version :
Creation Time : Thu Feb 23 02:50:20 2017
Raid Level : linear
Array Size : 1769472 (1728.00 MiB 1811.94 MB)
Raid Devices : 27
Total Devices : 27
State : clean
Active Devices : 27
Working Devices : 27
Failed Devices : 0
Spare Devices : 0
Rounding : 64K
Number Major Minor RaidDevice State
0 7 0 0 active sync /dev/loop0
1 7 1 1 active sync /dev/loop1
2 7 2 2 active sync /dev/loop2
3 7 3 3 active sync /dev/loop3
...
-- notice that "Version:", the superblock type, is left unspecified.
This was a non-metadata array ("mdadm --build"). An internal-metadata
array ("mdadm --create") allows the use of more than 27 component block
devices, with a v1.2 superblock:
# mdadm --create /dev/md/md-test --level=linear --raid-devices=32
/dev/loop{0..31}
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/md-test started.
#
# mdadm --detail /dev/md/md-test
/dev/md/md-test:
Version : 1.2
Creation Time : Thu Feb 23 03:12:21 2017
Raid Level : linear
Array Size : 2094592 (2045.50 MiB 2144.86 MB)
Raid Devices : 32
Total Devices : 32
Persistence : Superblock is persistent
Update Time : Thu Feb 23 03:12:21 2017
State : clean
Active Devices : 32
Working Devices : 32
Failed Devices : 0
Spare Devices : 0
Rounding : 0K
Name : quadlie:md-test (local to host quadlie)
UUID : 79a9278b:125af417:91b0c7ab:75267ede
Events : 0
Number Major Minor RaidDevice State
0 7 0 0 active sync /dev/loop0
1 7 1 1 active sync /dev/loop1
2 7 2 2 active sync /dev/loop2
3 7 3 3 active sync /dev/loop3
...
mdadm refuses to attempt creating the same array with a v0.90
superblock:
# mdadm --create /dev/md/md-test --level=linear --metadata=0 --raid-devices=32
/dev/loop{0..31}
mdadm: 0.90 metadata supports at most 27 devices per array
This error message is, however, a lot more informative than
"/dev/loop27: Device or resource busy", the initial one with a
non-metadata array.
mdadm won't allow the v1.2 superblock type with a non-metadata array:
# mdadm --build /dev/md/md-test --level=linear --metadata=1 --raid-devices=32
/dev/loop{0..31}
mdadm: :option --metadata not valid in build mode
-- so apparently, we're out of luck; it can't be done.
We may infer that when assembling a non-metadata array ("mdadm
--build"), the in-kernel superblock defaults to the v0.90 type, which
imposes the 27-component-device limit. Unfortunately, mdadm doesn't
allow you to override this default, and tell the kernel to use the v1.2
superblock. From the mdadm manpage:
-e, --metadata=
Declare the style of RAID metadata (superblock) to be used. The
default is 1.2 for --create, and to guess for other operations.
Choosing the v0.90 superblock for "mdadm --build" is wrong. There
doesn't seem to be any reason why the in-kernel superblock in this case
shouldn't be the v1.2 type, which would eliminate this problem, and
other limitations of the v0.90 superblock.
This should be fixed. However, it seems to be an upstream (possibly
kernel), rather than specifically Debian, problem.
-- Ian Bruce
