Confused about that LSI card? Join the crowd ...

A recent post prompted me to think, there's a lot of confusion regarding LSI HBA/RAID cards.  If you're confused, it is probably understandable...  it can be a bit complicated!

ZFS users on FreeNAS should avoid using the hardware RAID features of a RAID card.  ZFS and FreeNAS work best when the drive is managed directly by FreeNAS, including having SMART data available from the drive.  Any use of "virtual drives" or one-drive RAID0 JBOD modes likely involves the controller writing its own proprietary configuration to the drive and then that means that if you ever need to switch controllers, you're going to have extra special trouble because you need to copy the data off the old drive, not just move the old drive to a new SATA controller port.

LSI makes a lot of hardware.  They make HBA ("Host Bus Adapter") and RAID cards.  In general, a HBA is probably a better choice than a RAID card.  The HBA cards I've seen are most often driven by the LSI Logic Fusion-MPT2 SAS driver in FreeBSD ("mps") but older ones may be driven by the LSI Logic Fusion-MPT SCSI driver ("mpt").  Most of the current RAID cards seem to be driven by the LSI Logic MegaRAID SAS driver ("mfi") which comes with some significant caveats, including being unable to see your drives via the FreeBSD CAM subsystem or access diagnostics via SMART, and all maintenance and management must be done via the "mfiutil" tool or via the BIOS.  If you have an older controller of some sort, one that doesn't do SATA-III, you may need some other drivers and you will also be limited to no more than 2TB drives.  If you are considering buying one, don't, unless you're getting it for $10 or free or something like that...  such controllers include the IBM ServeRAID BR10i and Intel RAID Controller SASUC8I (LSI SAS3082E-R/LSI SAS1068E chipset) and are driven in FreeBSD by the LSI Logic Fusion-MPT SCSI driver ("mpt").  It is possible to crossflash these cards to be a generic LSI SAS1068E card, but the silicon is still capped at 2TB drives.

[HR][/HR]
A favored card in the FreeNAS community is the IBM ServeRAID M1015, a budget RAID card that can be found inexpensively (~$75) on eBay.  This card comes by default with IBM's version of the LSI RAID firmware on it.  It's basically an LSI 9240-8i.  You don't want to run that, you instead want to crossflash it to be an LSI 9211-8i (also known as LSI SAS2008) in IT mode, making it a basic SAS/SATA HBA card.  Two SFF8087 connectors provide up to 8 SAS/SATA channels directly via breakout cables, or, if a compatible SAS expander is used, possibly many more SAS/SATA channels.  The card consumes around 10 watts and should have at least some airflow in order to maintain proper cooling. It is important to crossflash this card!  If you do, it works under the "mps" driver, and it becomes a plain HBA.  If you don't, then it is under the "mfi" driver and it is a RAID card. If you get an M1015 and crossflash it to IT mode, you end up with one of the best HBA controllers available for FreeNAS, in my opinion.  These cards often come without brackets or with a low-profile bracket to fit into 2U rackmount servers.  Getting a proper bracket is recommended.

[HR][/HR]
Configuration of LSI 6Gb/s HBA products

This is the BIOS probe of an M1015 in IT mode with 8 drives attached via a single SFF8087 and an LSI SAS expander (Supermicro 24-drive SAS backplane).  Directly attached drives look just about the same, just fewer of them:



You can hit control-C during that to get to the card configuration utility.



Shows you the adapter type and firmware revision. Note firmware version 15 and IT mode. You can look under SAS Topology at your attached disks, but really there's not a whole lot to be seen here in IT mode.



Within FreeBSD, that'll probe as a controller serviced by the mps driver, and the drives will appear as normal "daX" devices and appear in "camcontrol devlist", so pretty straightforward pleasure there. A "dmesg | grep mps" would show something like:



And so far, things like hot adding a drive seem to work fine.  It has been a very pleasant and flexible controller to work with, as we use them with ESXi in -IR mode as well.  I expect that other LSI SAS 6Gb/s HBA's would closely resemble the above; if you run into something different, please let me know.

[HR][/HR]On to LSI 6Gb/s RAID:

The higher end LSI RAID products are a bit different.  Aimed at high performance and high availability applications, they tend to have more ports, cache, perhaps battery for the cache, and options for managing and configuring the array.  These devices are intended to abstract out the physical storage and present the system with virtual devices, which means that they've got an incredibly flexible and rich featureset and the ability to attach lots of drives both directly and indirectly, in many RAID levels, etc., which is great for Windows but not so great for FreeNAS and ZFS, where the operating system is quite capable and competent at dealing with the raw disk devices.

In some cases, users have come to the forum inquiring about a many-port controller where some combination would do.  It is worth noting:  With ZFS, you do want a decent SATA connection for your drives.  Motherboard ports are generally fine.  The $1 PCIe add-on card in the clearance bin at the local computer store is probably dodgy.  A quality HBA from LSI will be reliable.  If you need 10 SATA-III ports, you do NOT NEED A 16-PORT LSI RAID CONTROLLER FOR $800.  You don't even need a 16-port LSI HBA for $500.  You can get away with an M1015 and two motherboard SATA ports.  Really!  If you need 20 ports, consider two M1015's and four motherboard SATA ports.  Even a SATA-II port can go much faster than a contemporary spinny hard drive can.  Yes!  It works just fine.  If you need 24 ports, you definitely DO NOT NEED A 24-PORT LSI MegaRAID 9280-24i4e LSI RAID CONTROLLER FOR $1500.  Get the point?

If you haven't figured it out, the point is to steer away from port-dense and extremely pricey RAID controllers.  Don't let me stop you from spending $1500, but if you're going to spend $1500, buy a nice Supermicro chassis with a built-in SAS expander backplane like the SC846BE26-R920B along with an M1015 and cables, it is a better choice for FreeNAS.

But if you're stuck with an LSI RAID controller, here's some hopefully helpful information because you're probably sitting there wondering "what now."

We have a Supermicro LSI 2208 built into the mainboard on this server.  Planning to use it for ESXi so yes we needed a RAID controller, but right now it is running FreeNAS for fun testing.  The "Driver for new LSI card" thread came across so I went looking to see what the deal was.

This is how you can identify the LSI BIOS that uses MFI-based drivers.  When you're booting, it'll say "SAS-MFI BIOS" and talk about "WebBIOS".  Our Supermicro LSI MegaRAID 2208:



If you hit control-H to go into the WebBIOS (yay for consistency), it'll give you tools to configure your controller and the attached drives. They try to make it look Windows-y and while it's possible to control from the keyboard (use ALT plus the highlighted characters, and ENTER to select), it helps to have a mouse. The controller selection screen there will tell you the make and firmware of your controller.



The LSI intention is for individual drives to be aggregated into virtual drives, such as that virtual drive 0 which is a RAID1 of two SSD's, which is a common config here for our ESXi hosts. But that's not really what you want for FreeNAS. If you just plug drives into the LSI, they'll show up as "Unconfigured Good" drives:



so at this point you know the controller and the drives are talking. So the question becomes, what then. Apparently it is possible to configure the LSI controllers to pass through unconfigured-good drives to the underlying OS but they don't do it by default, and I don't see an immediately obvious way to set that. And it is more difficult because the disks don't show up in "camcontrol devlist". A "dmesg | grep mfi" from the FreeNAS CLI yields



which shows the RAID1 virtual device happily showing up as mfid0, but no mfid1. You can do a "mfiutil show drives" to see attached drives and "mfiutil show volumes" to show the available volumes. So what you probably want to do is do "mfiutil show drives" and then take note of the number (first column) of each "UNCONFIGURED GOOD" drive, then run "mfiutil create jbod NUMBER" for each of those numbers. Be warned that creating JBOD is almost certainly guaranteed to be overwriting what is on the disk. I don't think the MegaRAID Firmware Interface system has a way to cope with acting as a dumb SATA/SAS controller, so you will not be able to migrate disks back and forth from an LSI MFI controller to other random types of controller.  So you're locked in to LSI if you have to use MFI, at least as far as I can tell.  You also lose the ability to talk to the drive via SMART to look at its diagnostics directly.

Once you have used mfiutil (or the BIOS) to create your virtual devices on the RAID controller, they will be visible to FreeNAS and should be available in the volume manager.  FreeNAS will not see MFI-attached devices that are "UNCONFIGURED GOOD", you must configure them and make them into virtual devices for the controller, at which point they'll be presented to FreeNAS.  However, even once that happens, they will not show up as FreeBSD CAM devices, so if you're used to being able to do "camcontrol devlist" or other camcontrol ops, they won't be there for management in that manner.

The takeaway from all of this?  Try to avoid MFI based controllers if you plan to use ZFS.  They're not hopeless, but the inexpensive HBA models are more appropriate and better suited to the task.

 

Good point, I had forgotten about that.  There are so many vendors with so many stupid flash update requirements that we normally don't bother trying to upgrade card firmware in-place unless it is something like the HP gear which just gives you a CD image to update everything in the system automatically.

It's totally unrealistic to require Windows in order to update firmware on a network or storage controller, because you're not going to install Windows on your massive UNIX server just to update some firmware.  It is a great reason to have a spare PC laying around running something like XP or Vista, with a floppy and a CDROM drive.

 

You would think this to be the case, but some of LSI's firmware update tools don't work on some mainboards, annoyingly including some of Supermicro's.  I can't remember the exact message or I'd Google it and show you the entertaining discussions of the issue, which basically have Supermicro support blaming LSI, and LSI doing the same.  It was *more* entertaining because the mitigation suggested was to use the EFI based tools, which were a complete and total disaster.  *My* mitigation of sticking it in an HP desktop and flashing it... once I discovered that the problem was the combination of mainboard and LSI...  five minutes and worked perfectly.  I long ago forgot the specifics and so when we get a batch of M1015 in, I just go dredge out the bench PC and flash away.  One of the things that led to my frustrations in this thread.

It does mean I've got to actually get off my butt and lay hands on hardware, which is kind of offensive in this age of IP-KVM and IPMI/BMC.

Oops.  I'm an idiot.  Look right above at 9C1 Newbee's post.  There's the error and a synopsis of the cause.

 

It's may be supported. If you have one already I'd try it and post back. If you don't have one I wouldn't spend money in it. Sure, it's "newer" but it won't affect the performance of ZFS....

Edit:  Remember, the LSI 2008 (IBM 1015) is just under $100 on ebay, and I'm betting you won't pay that little for the 2308.

 

because the M1015 is usually about 1/4th the price on ebay ...?


Sent from my iPhone using Tapatalk 2

 

So yes the going rate for the M1015 is about $90-$100, which is about $15 more than it used to go for.  The eBay price on the 9211-8i appears to have dropped substantially, I don't really know what the reason for that might be.  The retail price is still around $250, so the idea that you should acquire your SAS2008 HBA of whatever branding off of eBay is still valid.

 

I was gonna say I was just involved with the purchase of 2 M1015s and they were about 1/2 the price of the 9211-8i.

 

Why is it that whenever I go to buy something on eBay, I can see that some of them sold two months ago for half what people are asking for it today.  Heh.

 

I dunno.  Paid $95 each for my cards.

 

Fakes are becoming a bigger and bigger problem.  I always stick with buyers that have good return policies in case they are fakes.  Generally with ebay I stick to purchases from inside the USA if its an expensive purchase.  You're more likely to get a refund from a scammer if they are in the USA than overseas from my experience.

 

ick.