Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!

[cyberjock]

I've put together a Powerpoint presentation(and PDF) that gives some useful info for newbies to FreeNAS. I decided to create this slideshow because in the last 5 months I've been on this forum I've seen a lot of people confused about vdevs, zpools, zils, l2arcs, etc. Hopefully we can put to rest alot of the confusion once and for all.

We get a large amount of duplicate threads with the same questions being asked every other day. Personally, we get them so often I decided to stop answering them and decided a better use of my time would be to create this presentation. I literally read every thread and every post that goes on the forum. So if I don't answer either the answer is in the thread or the answer is found in this presentation or FreeNAS manual. Answering every 3rd thread with "Consult the manual" gets a little old after a while and I have better uses for my time.

This presentation also contains a lot of information that is explained in a little more detail for new users. It includes many common errors newbies make and can save you some heartache. If you are brand new to FreeBSD, I recommend reading the manual cover to cover. There are a lot of recommendations throughout the manual, and they are typically there because they are an error trap for many people.

I've saved this in Powerpoint because I have some animations in the slideshow. I'm not sure what other formats would work. If you would like this in another format that supports animations please let me know and I'll see what I can do. Currently I provide this in a powerpoint presentation and PDF. The PDF has no animations therefore the Powerpoint is preferred.

I'll try to keep it updated as necessary.

Updated as of: 3/29/2013 FreeNAS 8.3.1

Powerpoint: https://dl.dropbox.com/u/57989017/Fr...uide%20PPT.zip (This is the preferred format because it includes animations)

PDF via Google Docs: https://docs.google.com/open?id=0BzH...Ws5MFJGRnl5Rjg
PDF: https://dl.dropbox.com/u/57989017/Fr...uide%20PDF.zip

[admin]

Is it possible to generate a PDF for the non-animated portions for those of us who don't use Windows?

[cyberjock]

Is it possible to generate a PDF for the non-animated portions for those of us who don't use Windows?

Link has been added for PDF.

[bfishernc]

nicely done - confirmed everything I've learned so far (as a noob)

[paleoN]

First, let me say good job. I would also like to officially request a web-based version of this. You can even use animated gifs and no external program needed to view it.

Now, on to the critiques.

• 
  VDevs are allocated in RAID formats such as RAID0(striped), RAIDZ and RAIDZ2
  
  

You left out mirrors.

• 
  You cannot add more hard drives to a VDev once it is created.
  

Keep it. It's a very important point. Also, it's inaccurate in the one exception that you can turn a stripe into a mirror. Perhaps asterisk it?

• 
  Failure of a ZIL will result in a loss of data.
  

I'd change this to a loss of the entire pool. Which is what will happen with ZFS v15, what we use now.

I'd add to the L2ARC section that using an L2ARC will decrease the amount of RAM available to the ARC cache. This is particularly noticeable with a large L2ARC.

Perhaps add the caveats that most people do not need a ZIL, L2ARC, or see a benefit from them and the single best increase in performance will come from maxing out their RAM. If they even need to do that.

[cyberjock]

I didn't leave out mirrors, at least not intentionally. I was simply providing a few examples.

I didn't know that you can turn a stripe into a mirror. I didn't know you could turn a VDev stripe into a mirror. I'll have to look into that. Can I find more information in the manual?

As for the failure of a ZIL, will it's failure cause the pool to go offline and be unrecoverable? Or is the data that is in the ZIL and not committed to the pool simply lost? I've never played with ZILs myself because I don't see a need for them in any of my circumstances.

Can you explain the cause/effect of the L2ARC changing the amount of RAM allocated to the ARC? I believe you, I've just never heard of this before and I'm very curious to know more about this phenomenon and the though process behind it.

I am trying to shy away from comments about what to recommend people. Each setup can potentially have different situations. I'd prefer to show what things you shouldn't do for data security than things you shouldn't do for performance reasons. Of course, I'm arguing this idea with myself because far too many people have first posts with comments like "I have 2GB of RAM and 30TB of storage and I get kernel panics all the time". My concern is that the presentation could suddenly include situations such as "recommend using an Intel NIC because they are so cheap, they work, and they have excellent performance". Maybe I should add a slide of other random tidbits of information for noobs at the end with comments like these.

[paleoN]

I didn't leave out mirrors, at least not intentionally. I was simply providing a few examples.

Just all the others.

I didn't know that you can turn a stripe into a mirror. I didn't know you could turn a VDev stripe into a mirror. I'll have to look into that. Can I find more information in the manual?

From the zpool man page:

zpool attach [-f] pool device new_device

Attaches new_device to an existing zpool device. The existing
device cannot be part of a raidz configuration. If device is not
currently part of a mirrored configuration, device automatically
transforms into a two-way mirror of device and new_device. If
device is part of a two-way mirror, attaching new_device creates a
three-way mirror, and so on. In either case, new_device begins to
resilver immediately

And from the ZFS Administration Guide see Example 4–6.

As for the failure of a ZIL, will it's failure cause the pool to go offline and be unrecoverable?

The entire pool unrecoverable:

• There is a bug/feature with this version of ZFS that will cause you to lose your pool if your ZIL goes offline or gets detached.

Or is the data that is in the ZIL and not committed to the pool simply lost? I've never played with ZILs myself because I don't see a need for them in any of my circumstances.

This will be the expected behavior with ZFS v28 which will be in FreeNAS 8.3 as it's in FreeBSD 8.3.

Also, the ZIL is for sync writes only.

Can you explain the cause/effect of the L2ARC changing the amount of RAM allocated to the ARC? I believe you, I've just never heard of this before and I'm very curious to know more about this phenomenon and the though process behind it.

ZFS has to keep track of what's in the L2ARC tables which of necessity requires RAM. I believe it's all or almost all metadata which also means you may hit the metadata cache limit for ZFS depending on how much RAM & how large the L2ARC. I've read that it uses 25MB of RAM per 1GB of L2ARC, but I remember thinking that was high & the source wasn't the best. I will have to see if I can track down some decent references, as I'm no expert on the L2ARC, and maybe even bookmark them this time.

Also, the L2ARC is for random reads mostly static data, i.e. databases. You will see little to no benefit for streaming workloads.

I am trying to shy away from comments about what to recommend people. Each setup can potentially have different situations. I'd prefer to show what things you shouldn't do for data security than things you shouldn't do for performance reasons.

Understandable and you're right each setup is different. It's just the ZIL and L2ARC are specifically for performance. Most noobs seem to think, "I have a SSD. I'll just add it to the pool and it will be faster." Where in truth it depends on their workload and even then it's usually much cheaper to max the RAM on their motherboard than get an SSD. Which will also get them superior performance.

[paleoN]

Found a good source.

> Is there a guideline in which relation L2ARC size should be to RAM?

Approximately 200 bytes per record. I use the following example:
Suppose we use a Seagate LP 2 TByte disk for the L2ARC
+ Disk has 3,907,029,168 512 byte sectors, guaranteed
+ Workload uses 8 kByte fixed record size
RAM needed for arc_buf_hdr entries
+ Need = ~(3,907,029,168 - 9,232) * 200 / 16 = ~48 GBytes

Don't underestimate the RAM needed for large L2ARCs
-- richard

Now it's not nearly as bad as that. Most of us won't be using 2TB L2ARC and have only zvols or tuned the zfs record size to 8K.

A much more typical scenario using a 300 GB L2ARC:

So, depending on the data, you need somewhere between 440 MBytes and 111 GBytes to hold the L2ARC headers.

Once again you would never actually see the 111GB. For an 8K record size like above the L2ARC would use about 6.8GB of RAM. A 32K record size would drop it down to 1.7GB.

It's been my understanding/assumption that ZFS auto-tunes the L2ARC. Which would mean if you had a SSD that was "too big" or you had some crazy record sizes you would just never fully use it. The impression I have now is the "auto-tuning" is done when arc_meta_limit is hit. By default arc_meta_limit is 25% of arc_max. Using a L2ARC would fill this. What's more I believe there will be memory contention between ARC & L2ARC metadata. While the ARC would win most of it, the L2ARC would still use some.

[calgarychris]

Thanks for putting together the presentation, very helpful from this noobie's prespective!

[cyberjock]

Ok, I've added all of your recommendations Paleon and the files have been updated accordingly .

Edit: I actually added lots of small things. I also made the comment that emulation using VMWare works great for experimenting if you don't want to dedicate hardware. Anyone know if Openbox works with FreeNAS for experimenting?