Confused about that LSI card? Join the crowd ...

Discussion in 'Hardware' started by jgreco, Mar 19, 2013.

  1. jgreco Resident Grinch

    Member Since:
    May 29, 2011
    Message Count:
    3,418
    Likes Received:
    303
    Trophy Points:
    83
    Location:
    WHO(1)ville, Unixland
    jgreco, Mar 19, 2013

    A recent post prompted me to think, there's a lot of confusion regarding LSI HBA/RAID cards. If you're confused, it is probably understandable... it can be a bit complicated!

    ZFS users on FreeNAS should avoid using the hardware RAID features of a RAID card. ZFS and FreeNAS work best when the drive is managed directly by FreeNAS, including having SMART data available from the drive. Any use of "virtual drives" or one-drive RAID0 JBOD modes likely involves the controller writing its own proprietary configuration to the drive and then that means that if you ever need to switch controllers, you're going to have extra special trouble because you need to copy the data off the old drive, not just move the old drive to a new SATA controller port.

    LSI makes a lot of hardware. They make HBA ("Host Bus Adapter") and RAID cards. In general, a HBA is probably a better choice than a RAID card. The HBA cards I've seen are most often driven by the LSI Logic Fusion-MPT2 SAS driver in FreeBSD ("mps") but older ones may be driven by the LSI Logic Fusion-MPT SCSI driver ("mpt"). Most of the current RAID cards seem to be driven by the LSI Logic MegaRAID SAS driver ("mfi") which comes with some significant caveats, including being unable to see your drives via the FreeBSD CAM subsystem or access diagnostics via SMART, and all maintenance and management must be done via the "mfiutil" tool or via the BIOS. If you have an older controller of some sort, one that doesn't do SATA-III, you may need some other drivers and you will also be limited to no more than 2TB drives. If you are considering buying one, don't, unless you're getting it for $10 or free or something like that... such controllers include the IBM ServeRAID BR10i and Intel RAID Controller SASUC8I (LSI SAS3082E-R/LSI SAS1068E chipset) and are driven in FreeBSD by the LSI Logic Fusion-MPT SCSI driver ("mpt"). It is possible to crossflash these cards to be a generic LSI SAS1068E card, but the silicon is still capped at 2TB drives.

    [HR][/HR]
    A favored card in the FreeNAS community is the IBM ServeRAID M1015, a budget RAID card that can be found inexpensively (~$75) on eBay. This card comes by default with IBM's version of the LSI RAID firmware on it. You don't want to run that, you instead want to crossflash it to be an LSI 9211-8i (also known as LSI SAS2008) in IT mode, making it a basic SAS/SATA HBA card. Two SFF8087 connectors provide up to 8 SAS/SATA channels directly via breakout cables, or, if a compatible SAS expander is used, possibly many more SAS/SATA channels. The card consumes around 10 watts and should have at least some airflow in order to maintain proper cooling. It is important to crossflash this card! If you do, it works under the "mps" driver, and it becomes a plain HBA. If you don't, then it is under the "mfi" driver and it is a RAID card. If you get an M1015 and crossflash it to IT mode, you end up with one of the best HBA controllers available for FreeNAS, in my opinion. These cards often come without brackets or with a low-profile bracket to fit into 2U rackmount servers. Getting a proper bracket is recommended.

    [HR][/HR]
    Configuration of LSI 6Gb/s HBA products

    This is the BIOS probe of an M1015 in IT mode with 8 drives attached via a single SFF8087 and an LSI SAS expander (Supermicro 24-drive SAS backplane). Directly attached drives look just about the same, just fewer of them:

    screen6.PNG

    You can hit control-C during that to get to the card configuration utility.

    screen7.PNG

    Shows you the adapter type and firmware revision. Note firmware version 15 and IT mode. You can look under SAS Topology at your attached disks, but really there's not a whole lot to be seen here in IT mode.

    screen8.PNG

    Within FreeBSD, that'll probe as a controller serviced by the mps driver, and the drives will appear as normal "daX" devices and appear in "camcontrol devlist", so pretty straightforward pleasure there. A "dmesg | grep mps" would show something like:

    screen9.PNG

    And so far, things like hot adding a drive seem to work fine. It has been a very pleasant and flexible controller to work with, as we use them with ESXi in -IR mode as well. I expect that other LSI SAS 6Gb/s HBA's would closely resemble the above; if you run into something different, please let me know.

    [HR][/HR]On to LSI 6Gb/s RAID:

    The higher end LSI RAID products are a bit different. Aimed at high performance and high availability applications, they tend to have more ports, cache, perhaps battery for the cache, and options for managing and configuring the array. These devices are intended to abstract out the physical storage and present the system with virtual devices, which means that they've got an incredibly flexible and rich featureset and the ability to attach lots of drives both directly and indirectly, in many RAID levels, etc., which is great for Windows but not so great for FreeNAS and ZFS, where the operating system is quite capable and competent at dealing with the raw disk devices.

    In some cases, users have come to the forum inquiring about a many-port controller where some combination would do. It is worth noting: With ZFS, you do want a decent SATA connection for your drives. Motherboard ports are generally fine. The $1 PCIe add-on card in the clearance bin at the local computer store is probably dodgy. A quality HBA from LSI will be reliable. If you need 10 SATA-III ports, you do NOT NEED A 16-PORT LSI RAID CONTROLLER FOR $800. You don't even need a 16-port LSI HBA for $500. You can get away with an M1015 and two motherboard SATA ports. Really! If you need 20 ports, consider two M1015's and four motherboard SATA ports. Even a SATA-II port can go much faster than a contemporary spinny hard drive can. Yes! It works just fine. If you need 24 ports, you definitely DO NOT NEED A 24-PORT LSI MegaRAID 9280-24i4e LSI RAID CONTROLLER FOR $1500. Get the point?

    If you haven't figured it out, the point is to steer away from port-dense and extremely pricey RAID controllers. Don't let me stop you from spending $1500, but if you're going to spend $1500, buy a nice Supermicro chassis with a built-in SAS expander backplane like the SC846BE26-R920B along with an M1015 and cables, it is a better choice for FreeNAS.

    But if you're stuck with an LSI RAID controller, here's some hopefully helpful information because you're probably sitting there wondering "what now."

    We have a Supermicro LSI 2208 built into the mainboard on this server. Planning to use it for ESXi so yes we needed a RAID controller, but right now it is running FreeNAS for fun testing. The "Driver for new LSI card" thread came across so I went looking to see what the deal was.

    This is how you can identify the LSI BIOS that uses MFI-based drivers. When you're booting, it'll say "SAS-MFI BIOS" and talk about "WebBIOS". Our Supermicro LSI MegaRAID 2208:

    screen1.PNG

    If you hit control-H to go into the WebBIOS (yay for consistency), it'll give you tools to configure your controller and the attached drives. They try to make it look Windows-y and while it's possible to control from the keyboard (use ALT plus the highlighted characters, and ENTER to select), it helps to have a mouse. The controller selection screen there will tell you the make and firmware of your controller.

    screen2.PNG

    The LSI intention is for individual drives to be aggregated into virtual drives, such as that virtual drive 0 which is a RAID1 of two SSD's, which is a common config here for our ESXi hosts. But that's not really what you want for FreeNAS. If you just plug drives into the LSI, they'll show up as "Unconfigured Good" drives:

    screen3.PNG

    so at this point you know the controller and the drives are talking. So the question becomes, what then. Apparently it is possible to configure the LSI controllers to pass through unconfigured-good drives to the underlying OS but they don't do it by default, and I don't see an immediately obvious way to set that. And it is more difficult because the disks don't show up in "camcontrol devlist". A "dmesg | grep mfi" from the FreeNAS CLI yields

    screen5.PNG

    which shows the RAID1 virtual device happily showing up as mfid0, but no mfid1. You can do a "mfiutil show drives" to see attached drives and "mfiutil show volumes" to show the available volumes. So what you probably want to do is do "mfiutil show drives" and then take note of the number (first column) of each "UNCONFIGURED GOOD" drive, then run "mfiutil create jbod NUMBER" for each of those numbers. Be warned that creating JBOD is almost certainly guaranteed to be overwriting what is on the disk. I don't think the MegaRAID Firmware Interface system has a way to cope with acting as a dumb SATA/SAS controller, so you will not be able to migrate disks back and forth from an LSI MFI controller to other random types of controller. So you're locked in to LSI if you have to use MFI, at least as far as I can tell. You also lose the ability to talk to the drive via SMART to look at its diagnostics directly.

    Once you have used mfiutil (or the BIOS) to create your virtual devices on the RAID controller, they will be visible to FreeNAS and should be available in the volume manager. FreeNAS will not see MFI-attached devices that are "UNCONFIGURED GOOD", you must configure them and make them into virtual devices for the controller, at which point they'll be presented to FreeNAS. However, even once that happens, they will not show up as FreeBSD CAM devices, so if you're used to being able to do "camcontrol devlist" or other camcontrol ops, they won't be there for management in that manner.

    The takeaway from all of this? Try to avoid MFI based controllers if you plan to use ZFS. They're not hopeless, but the inexpensive HBA models are more appropriate and better suited to the task.
    jmp, KTrain, cyberjock and 3 others like this.
  2. 9C1 Newbee Member

    Member Since:
    Oct 9, 2012
    Message Count:
    109
    Likes Received:
    3
    Trophy Points:
    18
    Location:
    Fresno California
    9C1 Newbee, Mar 27, 2013

    Thank You for the education. M1015 on the way.
  3. mjt5282 New Member

    Member Since:
    Mar 19, 2013
    Message Count:
    1
    Likes Received:
    0
    Trophy Points:
    1
    mjt5282, Mar 30, 2013

    this tutorial was very helpful, thank you.
  4. 9C1 Newbee Member

    Member Since:
    Oct 9, 2012
    Message Count:
    109
    Likes Received:
    3
    Trophy Points:
    18
    Location:
    Fresno California
    9C1 Newbee, Apr 13, 2013

    PROBLEM: While I was flashing the M1015 to IT mode, during the process I got an error message "Failed to initialize PAL".

    SOLUTION: I used a different computer to perform the flash without issue. Once the M1015 was flashed to IT mode, I was able to place the card back into the intended machine and use it with out issue.

    A quick search of the web turned up the same results for others during the flash process. The error seems to manifest on certain Intel chipsets. That made sense due to the fact I was attempting to flash the controller with Supermicro X9SCM-F with an Intel chipset, which resulted in the error. I ended up successfully flashing the M1015 in a Dell inspiron. I am surprised it worked because getting %#ck^d over by Dell machines seems to be the status quo for me lately.
  5. jgreco Resident Grinch

    Member Since:
    May 29, 2011
    Message Count:
    3,418
    Likes Received:
    303
    Trophy Points:
    83
    Location:
    WHO(1)ville, Unixland
    jgreco, Apr 14, 2013

    Good point, I had forgotten about that. There are so many vendors with so many stupid flash update requirements that we normally don't bother trying to upgrade card firmware in-place unless it is something like the HP gear which just gives you a CD image to update everything in the system automatically.

    It's totally unrealistic to require Windows in order to update firmware on a network or storage controller, because you're not going to install Windows on your massive UNIX server just to update some firmware. It is a great reason to have a spare PC laying around running something like XP or Vista, with a floppy and a CDROM drive.
  6. pbucher Member

    Member Since:
    Oct 15, 2012
    Message Count:
    155
    Likes Received:
    12
    Trophy Points:
    18
    pbucher, Apr 25, 2013

    Actually for LSI cards you can make a DOS bootable using some old USB thumb drive with the firmware files on it and slip that into your uber Unix box and boot it up and do the firmware work without messing up your nice unix box.

    As per the readme make sure you use firmware version 13. There is a very tight relationship between the FreeBSD driver and the firmware of the card. While using a newer version of the firmware can be tempting I've found that LSI tends not to test newer firmware with FreeBSD unless they also update the FreeBSD driver also. And once they do update the FreeBSD driver you need to either compile it yourself or wait for FreeNAS to include it(at which time I'm sure they'll update the readme to list the required firmware version).
  7. jgreco Resident Grinch

    Member Since:
    May 29, 2011
    Message Count:
    3,418
    Likes Received:
    303
    Trophy Points:
    83
    Location:
    WHO(1)ville, Unixland
    jgreco, Apr 25, 2013

    You would think this to be the case, but some of LSI's firmware update tools don't work on some mainboards, annoyingly including some of Supermicro's. I can't remember the exact message or I'd Google it and show you the entertaining discussions of the issue, which basically have Supermicro support blaming LSI, and LSI doing the same. It was *more* entertaining because the mitigation suggested was to use the EFI based tools, which were a complete and total disaster. *My* mitigation of sticking it in an HP desktop and flashing it... once I discovered that the problem was the combination of mainboard and LSI... five minutes and worked perfectly. I long ago forgot the specifics and so when we get a batch of M1015 in, I just go dredge out the bench PC and flash away. One of the things that led to my frustrations in this thread.

    It does mean I've got to actually get off my butt and lay hands on hardware, which is kind of offensive in this age of IP-KVM and IPMI/BMC.

    Oops. I'm an idiot. Look right above at 9C1 Newbee's post. There's the error and a synopsis of the cause.
  8. pbucher Member

    Member Since:
    Oct 15, 2012
    Message Count:
    155
    Likes Received:
    12
    Trophy Points:
    18
    pbucher, Apr 25, 2013

    I know what you mean, you need LSI's EFI tools to flash the card and some kind of EFI boot tool. In other words save yourself a ton of trouble and pull the card and put it in a junk box and flash it.

    Now I've been able to do standard firmware upgrades from inside FreeNAS once, but the switch from IR to IT mode requires a DOS or EFI environment for some reason :(
  9. KMR Member

    Member Since:
    Dec 3, 2012
    Message Count:
    176
    Likes Received:
    5
    Trophy Points:
    18
    Occupation:
    IT
    Location:
    Great White North
    KMR, Jul 8, 2013

    I had the "Failed to initialize PAL" error with my M1015 card. It is caused by UEFI and there is a work around. You need to boot into UEFI shell and use the sas2flash.efi utility that is included with the download pack from servethehome.com (the link is above). This article explains the process: http://brycv.com/blog/2012/flashing-it-firmware-to-lsi-sas9211-8i/

    I performed the flash on my Supermicro board and it worked fine after I figured out what was going on.
  10. scotch_tape Member

    Member Since:
    Mar 11, 2013
    Message Count:
    69
    Likes Received:
    0
    Trophy Points:
    6
    Occupation:
    http://superfix.me/
    Location:
    Texas
    Home page:
    scotch_tape, Jul 11, 2013

    Any opinion about the LSI SAS 2308 controller? I dont see it on the FreeBSD Hardware list.....
  11. cyberjock Forum Guard Dog/Admin

    Member Since:
    Mar 25, 2012
    Message Count:
    10,480
    Likes Received:
    462
    Trophy Points:
    83
    cyberjock, Jul 11, 2013

    It's may be supported. If you have one already I'd try it and post back. If you don't have one I wouldn't spend money in it. Sure, it's "newer" but it won't affect the performance of ZFS....

    Edit: Remember, the LSI 2008 (IBM 1015) is just under $100 on ebay, and I'm betting you won't pay that little for the 2308.
  12. scotch_tape Member

    Member Since:
    Mar 11, 2013
    Message Count:
    69
    Likes Received:
    0
    Trophy Points:
    6
    Occupation:
    http://superfix.me/
    Location:
    Texas
    Home page:
    scotch_tape, Jul 11, 2013

    The new Haswell Supermicro boards have an LSI SAS 2308 on them for only $50 more. I'm seriously thinking about it.....
  13. Z300M Member

    Member Since:
    Sep 9, 2011
    Message Count:
    291
    Likes Received:
    3
    Trophy Points:
    18
    Location:
    Michigan
    Z300M, Jul 23, 2013

    Any reason not to just buy an LSI9211-8i in the first place (at the right price) rather than cross-flashing an M1015?
  14. jgreco Resident Grinch

    Member Since:
    May 29, 2011
    Message Count:
    3,418
    Likes Received:
    303
    Trophy Points:
    83
    Location:
    WHO(1)ville, Unixland
    jgreco, Jul 23, 2013

    because the M1015 is usually about 1/4th the price on ebay ...?


    Sent from my iPhone using Tapatalk 2
  15. Z300M Member

    Member Since:
    Sep 9, 2011
    Message Count:
    291
    Likes Received:
    3
    Trophy Points:
    18
    Location:
    Michigan
    Z300M, Jul 23, 2013

    M1015 cards on eBay these days are considerably more expensive than I see mentioned in older messages in these forums, and I see LSI 9211-8i cards having sold recently for little more, and in some cases even less, than the M1015.
  16. jgreco Resident Grinch

    Member Since:
    May 29, 2011
    Message Count:
    3,418
    Likes Received:
    303
    Trophy Points:
    83
    Location:
    WHO(1)ville, Unixland
    jgreco, Jul 23, 2013

    So yes the going rate for the M1015 is about $90-$100, which is about $15 more than it used to go for. The eBay price on the 9211-8i appears to have dropped substantially, I don't really know what the reason for that might be. The retail price is still around $250, so the idea that you should acquire your SAS2008 HBA of whatever branding off of eBay is still valid.
  17. cyberjock Forum Guard Dog/Admin

    Member Since:
    Mar 25, 2012
    Message Count:
    10,480
    Likes Received:
    462
    Trophy Points:
    83
    cyberjock, Jul 23, 2013

    I was gonna say I was just involved with the purchase of 2 M1015s and they were about 1/2 the price of the 9211-8i.
  18. jgreco Resident Grinch

    Member Since:
    May 29, 2011
    Message Count:
    3,418
    Likes Received:
    303
    Trophy Points:
    83
    Location:
    WHO(1)ville, Unixland
    jgreco, Jul 23, 2013

    Why is it that whenever I go to buy something on eBay, I can see that some of them sold two months ago for half what people are asking for it today. Heh.
  19. cyberjock Forum Guard Dog/Admin

    Member Since:
    Mar 25, 2012
    Message Count:
    10,480
    Likes Received:
    462
    Trophy Points:
    83
    cyberjock, Jul 23, 2013

    I dunno. Paid $95 each for my cards. :p
  20. Z300M Member

    Member Since:
    Sep 9, 2011
    Message Count:
    291
    Likes Received:
    3
    Trophy Points:
    18
    Location:
    Michigan
    Z300M, Jul 26, 2013

    On further consideration and after further reading, I see claims that some of the "LSI"-branded boards from China are in fact fakes and are perhaps even using sub-par chips. Even some of the "Made in China" boards being sold by US-based vendors on eBay may be of that kind. So it may be better to buy an IBM-branded board and flash it to IT mode after all.

Share This Page