ZFS vs hardware raid question

Discussion in 'Hardware' started by Milkwerm, Feb 20, 2013.

Thread Status:
Not open for further replies.
  1. Milkwerm Member

    Member Since:
    Jun 26, 2011
    Message Count:
    33
    Likes Received:
    0
    Trophy Points:
    6
    Occupation:
    Sys Admin.
    Location:
    Wellington, New Zealand
    Milkwerm, Feb 20, 2013

    Hi
    With hardware raid, in order to maximise IO, "WriteBack" (with a BBU) is often used (and prefered for hosting Virtual machines)
    My question is, Does a ZFS Raidz set get the same advantage from any particular setting/configuration in order to maximise IO throughput?
  2. cyberjock Forum Guard Dog/Admin

    Member Since:
    Mar 25, 2012
    Message Count:
    10,376
    Likes Received:
    454
    Trophy Points:
    83
    cyberjock, Feb 21, 2013

    From my experimenting with ZFS on my raid controller(with 4GB of on-card RAM) your best bet is to disable the write cache and leave the read cache enabled with "normal" or "conservative". This provides the best performance and reliability.

    Using the write cache on the RAID controller can cause corruption because when ZFS commits writes to the hard drive they are silently written to the raid cache and not to the hard drive. This can really mess up a zpool and your data on a loss of power or system crash. Not to mention using the write cache can cause conflicts between reads and other writes because the hard drive will be busy when ZFS thinks the drives are idle.
  3. Milkwerm Member

    Member Since:
    Jun 26, 2011
    Message Count:
    33
    Likes Received:
    0
    Trophy Points:
    6
    Occupation:
    Sys Admin.
    Location:
    Wellington, New Zealand
    Milkwerm, Feb 21, 2013

    Thanks Cyberjock.
    By read cache "on" and write cache "off" your talking about a "writethrough" setup which is usually the fall back option for Writeback if/when a BBU battery fails?. Very interesting information you have posted that I hadn't thought about, but not why I was asking . My fault, I'll try to re-explain the question.
    I disable the raid card altogether when setting up RaidZ disks (raidz1), taking hardware caching out of the equation. This is on an old S5000PAL intel board and I remove the Raid key from the daughter board. So in a completely ZFS software controlled raid are there options like writethrough/writeback that can speed up the I/O or do we have to look to SSD drives for arc cache etc as eqivilents?
  4. cyberjock Forum Guard Dog/Admin

    Member Since:
    Mar 25, 2012
    Message Count:
    10,376
    Likes Received:
    454
    Trophy Points:
    83
    cyberjock, Feb 21, 2013

    No. On my card (Areca 1280ML) I have settings for the read cache and write cache.

    Write cache options: Disabled, Write-Through, Write-Back
    Read ahead-cache options: Disabled, Conservative, Normal, Aggressive.

    So all write cache options should be off/disabled. This is as recommended by ZFS for reliability and happens to provide the best performance anyway. So turning it on is pretty much a bad idea for both reasons. I have done one-on-ones with people and it seems that regardless of the controller used, write cache off seems to always work best even if you ignore the ZFS reasons. Even if it did provide a performance boost, unless performance was terrible and the absolute only way to fix it was enabling it I would strongly recommend against using it. You can basically unplug your BBU because it will serve no purpose at all.

    For read ahead I set mine to off at first, then changed it to normal. Normal seems to result in a significant(more than 150% increase in performance!) of scrubs. Also the system is much more responsive and generally seems to just run smoother.

    If you have one of those $1000 RAID controllers with 2GB+ of RAM on card and a BBU and you were looking to save some money I'd ebay the card(or return it to the manufacturer) and get an IBM 1015 RAID card. The 1015s are typically less than $100 so you can pocket quite a bit of money by changing cards. The only reason I kept mine is because I'm too lazy to sell mine. But I know the day is coming when I'll finally get smart and spend some time to ebay my card.

    Edit: Note that if you make these changes using a controller CLI utility you will most likely have to reboot for the changes to take effect.
  5. ioudas New Member

    Member Since:
    Nov 15, 2013
    Message Count:
    4
    Likes Received:
    1
    Trophy Points:
    3
    ioudas, Nov 15, 2013

    I signed up to say... this advice is about the worst advice ive ever seen on the internet anywhere. Ever. Disabling will just give you more problems.

    Food for thought.
  6. cyberjock Forum Guard Dog/Admin

    Member Since:
    Mar 25, 2012
    Message Count:
    10,376
    Likes Received:
    454
    Trophy Points:
    83
    cyberjock, Nov 15, 2013

    Oh, thank goodness we have people with 1 post like yourself to come in here and educate us in what we get wrong. What would we do without you!?
  7. ioudas New Member

    Member Since:
    Nov 15, 2013
    Message Count:
    4
    Likes Received:
    1
    Trophy Points:
    3
    ioudas, Nov 15, 2013

    Give out crappy advice to the masses? Thats what it appears you would do.
    jyavenard likes this.
  8. warri Active Member

    Member Since:
    Jun 6, 2011
    Message Count:
    888
    Likes Received:
    29
    Trophy Points:
    28
    Location:
    Germany
    warri, Nov 15, 2013

    ioudas, do you have any backing on why it is a "crappy" advice? Some explanation would be helpful at least.

    As far as I now, cyberjock is somehow right here. The general recommendation for ZFS seems to be to disable write-back mode if you have no battery backed cache on the RAID controller. In this case I guess you could still use "write-through" - since that is supposed to write the changes to both the disk and cache.
    On the other hand, if you have a controller with a BBU it should be fine to use "write-back" - if you get your system up and running in a reasonable time frame after a power failure.

    Disclaimer: Solely based only on google and some reading, have never used such a setup myself ;)

    Some sources:
    http://forums.freebsd.org/showthread.php?t=39040
    http://forums.freebsd.org/showthread.php?t=29233
    http://serverfault.com/questions/114547/what-is-a-raid-controller-bbu-for
  9. ioudas New Member

    Member Since:
    Nov 15, 2013
    Message Count:
    4
    Likes Received:
    1
    Trophy Points:
    3
    ioudas, Nov 15, 2013

    Failure to understand the concepts of hw raid mostly. From cyber jocks post he mentions nothing about BBU raid controllers. Your comments here actually reflect reality.

    Edit also: you will notice performance differences with disabled vs write back.. Disabled not being the best.
  10. cyberjock Forum Guard Dog/Admin

    Member Since:
    Mar 25, 2012
    Message Count:
    10,376
    Likes Received:
    454
    Trophy Points:
    83
    cyberjock, Nov 15, 2013

    If you read around here at all you'd know I had a BBU, and deliberately removed it because I had to disable the write cache to get reasonable performance. All of this is discussed above as well as plenty of other posts explaining in detail why having a write cache enabled doesn't garner you extra performance.

    And I have discussed BBU raid controllers in this thread as well as plenty of other posts. So before you jump in here and accuse me of stuff, feel free to actually do more than make an account and talk trash without some iota of information to support your comment. ;)
  11. cyberjock Forum Guard Dog/Admin

    Member Since:
    Mar 25, 2012
    Message Count:
    10,376
    Likes Received:
    454
    Trophy Points:
    83
    cyberjock, Nov 15, 2013

    And to be honest, this thread was last posted to in February. As such, I'm going to close this since we shouldn't be necro'ing threads just to argue something that has long since passed. If you want to discuss it, feel free to make your own thread to discuss the topic.
Thread Status:
Not open for further replies.

Share This Page