RAID 10 on RHEL 5

August 31, 2007 5:26 PM

How did *you* achieve a software RAID 10 (or 1+0) setup on your RHEL5 (or 4) box?

I've spent the last three days running down rabbit hole after rabbit hole trying to build a Centos 5 system with its root filesystem (/) on a nested software RAID 1+0 device.

Now I've given up and adopted a compromise, with / on a RAID 5 and other frequently used directories on a RAID 10 (not true nested raid, mind you, but the single-level version), but I know I'll get myself into this situation again, and I want to hear if anyone else has found a better way out.

Some of the snags I encountered:
* Disk Druid doesn't support raid "10" or nested levels of software raid (1+0, 0+1), and it won't recognize such filesystems if I configure them through the terminal.
* While I could configure nested levels of raid post-install with mdadm, I couldn't get any to be recognized on reboot.

My setup:
* Two 64-bit Intel processors
* Four 15K 74G scsi disks
* 4G RAM

posted by quasistoic to computers & internet (12 comments total)

Can you instead use 2 mirrors and use the volume manager to span across them? I vaguely recall reading that you shouldn't layer software raids, but volumes on top of raids was OK.
posted by Danaid at 5:36 PM on August 31, 2007

@Danaid: I'm guessing the reason he's going for RAID 10 is to try and avoid the performance hit of RAID5, when it comes to disk writes.

@ quasistoic: While it has been some time since I messed with software RAID (and thus read the following with a heaping helping of Morton's), from my own experience with software RAID I'm guessing you're not going to have any luck whatsoever getting it to work on the root filesystem in this configuration; that is, if your raid controller doesn't support hardware RAID 1+0 (which I assume it does not, else why would you be resorting to software RAID).

Of course, if you are going for RAID 1+0 for performance reasons, isn't implementing it in software going to counterbalance the increase over doing a hardware RAID 5 ?
posted by namewithoutwords at 6:33 PM on August 31, 2007

What's wrong with the single level version? That looks like RAID 10 to me. Who cares if it's nested or not, as long as the actual logic is right?
posted by Malor at 7:37 PM on August 31, 2007

No idea what magic RHEL wants, but on Debian/Ubuntu/every other Linux it goes something like:

mdadm --create /dev/md1 --level=10 --raid-devices=4 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2

Then mkfs your favourite filesystem on /dev/md1 and add to fstab. Don't forget to set the partition type to Linux md RAID so it's auto-detected.

Not sure how you're booting off RAID5.. was support for that added to lilo/grub at some point, or is that hardware RAID? Normally you'd use RAID-1 for that.

Performance wise, md RAID-10's pretty good. We make quite heavy use of it in our database servers:

md1 : active raid10 sdaa[23] sdz[22] sdy[21] sdx[20] sdw[19] sdv[18] sdu[17] sdt[16] sds[15] sdr[14] sdq[13] sdp[12] sdn[11] sdm[10] sdl[9] sdk[8] sdj[7] sdi[6] sdh[5] sdg[4] sdf[3] sde[2] sdd[1] sdc[0]
426796800 blocks 64K chunks 2 near-copies [24/24] [UUUUUUUUUUUUUUUUUUUUUUUU]
posted by Freaky at 7:47 PM on August 31, 2007

We use hardware RAID controllers, which I assume you would benefit from given that they offload a lot of the performance hit you'll see doing it in software...

I'm sure that software RAID 10 is faster than software RAID 5, but I wouldn't be at all surprised if hardware RAID 5 wasn't faster than software RAID 10.

Hopefully that makes sense...
posted by togdon at 8:56 PM on August 31, 2007

You've already encountered the real reason not to do this - which is that you want to keep your boot environment as simple as possible, so that any recovery you need to do later is as simple as possible and can be done using standard tools with very little custom configuration to get you back up and running as quickly as possible when something goes wonky. Use RAID1 for /, and you can boot directly off of either of the partitions in the mirror from a disk without any RAID drivers. Is whatever extra fault tolerance you'll get out of RAID10 going to be worth the extra hassle? If you're writing to / frequently enough that you care what the performance is like, your partitioning scheme needs some work anyway.
posted by Caviar at 7:32 AM on September 1, 2007

togdon: The performance advantages of hardware RAID are, in my experience, extremely overstated. What you might gain in battery backed up cache and XOR offload (rejoice, you're saving 2% of one of your precious CPUs in IO bound tasks) and "clever" IO reordering, you tend to lose several times over in firmware, driver and management interface bugs and incompatibilities, hardware limitations, and good old stupid design decisions.

Much as I would like the $40,000 or so we've spent on hardware RAID to have been worth every penny, there's a reason we're using GEOM/md/ZFS RAID and our collection of hardware RAID cards and disk shelves are mostly gathering dust.

(Anyone want to buy some Sun 3320 RAID units?)
posted by Freaky at 9:51 AM on September 1, 2007

We do hardware RAID using the controllers in mid-range HP and occasionally Dell boxes (think HP DL360). The advantage, beyond the speed bump, is that with a hardware controller the drives are usually hot swappable. With the latest generation boxes you usually get eight slots for SAS drives, we typically configure the OS to be on two in a RAID 1, and a second drive using the rest with either RAID 5 or RAID 10, depending on the speeds needed.

I agree on the ultra high end RAID boxes of yesteryear being a terrible deal in the modern era (Sun A1000's etc.), but a hardware RAID card in a server works well and avoids a lot of the problems that the poster seems to be experiencing.
posted by togdon at 7:39 PM on September 1, 2007

Most SCA/SAS/SATA controllers do hot-swap just fine, with the added advantage that you can actually *see* the disks, poke at SMART, and have sensible management tools which integrate into the OS instead of coming in years-old barely-supported binary blobs.

Hardware RAID can simplify some use-cases, especially if you happen to luck upon a vendor/card/firmware/OS/driver combination which work well over an extended period of time. In my experience such combinations are way more painful to find (and often use) than software RAID and a well known generic disk controller.
posted by Freaky at 10:44 PM on September 2, 2007

Well, given that the poster is using CentOS I can tell them that modern HPs, the DL360g5 for instance, work very well out of the box (really anything using the Compaq Smart Array controllers, and not the ancient or cheaper Silicon Image controllers works well). The cciss driver has been included in the kernel that ships with RHEL and its derivatives since at least 3.x, if not before then. I'm fairly certain that it was first included in the 2.2.x kernel tree, if not earlier.

We've also had good luck with the Dell controllers, although I'm less familiar with specific models there.
posted by togdon at 8:30 PM on September 3, 2007

@Danaid - Like namewithoutwords implied, that won't improve the performance any over RAID 1, which is significantly slower than RAID 5.

@Malor- The single level version is what I ended up going with, but I was hoping to avoid it in my production system because of its newness. I've found very few discussions of its performance and stability in practice, and the single discussion I found that was most relevant to what I was trying to accomplish had one participant (reliability unverified) describing this form of raid10 as "still in the eats its children stage". That's discouraging, to say the least.

@Freaky - We're not booting off a RAID 5: /boot is on its own RAID 1 partition. I was hoping to put everything else on raid 10 (or 1+0), but couldn't figure out how. It's reassuring to hear at least one person say that linux software raid 10 is working for them in a production environment. Can I ask how long you've been using it?
posted by quasistoic at 11:27 AM on September 4, 2007

quasistoic: Ah, ok. raid10 / should be fine then. Just make sure the raid10 module is actually being loaded properly and the partition type's correct... I don't know if Red Hat's hacked-about kernels change the situation at all though.

We've been running md raid10 in production for a bit over 2 years, running fairly heavy MySQL servers (>900m rows, 500 qps) with 4, 8, 12 and 24 drives. No real problems with it so far, just some odd but seemingly harmless read patterns (preferring one side of an array more than might be expected). A more statistically significant sample might show more serious issues, though; it is still marked EXPERIMENTAL in the kernel config after all.
posted by Freaky at 12:44 PM on September 4, 2007