ESXI + FreeNAS 8

[Miniwehats]

I have also posted this on Vmware forums:

Hello I am fairly new to setting up Esxi but I have a little test server running in my home and have settup a few VMs on it and played around a bit with it. I am building a new Esxi server in the next few weeks and am having some issues figuring out how exactly I am going to do some things. Please forgive my ignorance I have had a difficult time finding answeres to my questions. And I am also having an issue articulating this into a understandable question.

What I want to do : Building a new ESXI Server - AMD quad 3.6 FX 16gb RAM booting ESXI off USB.
-FreeNAS VM
-Server 2008 VM
-Random other VMs

What I want to do is build all this in one single box but my issues is how FreeNas will load and where since all my VMs will be build from ISCSI volumes from FreeNAS. FreeNAS runs off a USB drive as well.

Loading FreeNAS as a VM off a usb flash drive, where will the data store for the VM go? Can I build it onto the flash drive? Will I need to just have a single local disk on the ESXI server to have this one single data store? All my orther VMs will run from the FreeNAS volumes i create for them. So if my FreeNAS volume goes down ALL my VMs will go down. Granted I can backup the config and reload FreeNAS and it should all come back up. Should I create a Hardware Mirror of 2 drives locally on the ESXI server to just host the data store of FreeNAS so incase one goes down it can use the mirror?

After reviewing this it seems best practice would be to build my FreeNAS box seperate from the ESXI server but I would really like to build it all in one box to save money.

So would you recommend doing a Mirror on the esxi just for the FreeNAS data store? Or building a second box?

Sorry for the confusion here I am just trying to get this all figured out. And thank you for any help you can provide!

[louisk]

I'm confused about how you will boot the FreeNAS VM from the storage off the FreeNAS VM. That sounds like a catch 22.

I would suggest you make FreeNAS a physical box, and do NFS to ESXi.

The other alternative would be to have all the disk in the ESXi machine, and have FreeNAS share some of that to other machines.

[Miniwehats]

Sorry for the confusion, all the disks are in the same box. I just decided to settup a Hardware Mirror Raid of 2x SSDs to hold the esxi data store and the FreeNAS data store. All the other drives for the raid will be assigned to the FreeNAS VM. This should work out fine right?

[survive]

Hi Miniwehats,

I'm with louisk on this.....

Build 2 boxes, one for ESXi , the other for FreeNAS. If that's not an option build one box running ESXi (you'll need a proper RAID card) and use FreeNAS in a VM to share out the storage.

-Will

[Miniwehats]

I would like to build a seperate box but the cost just goes way up. I will do a hardware raid on the box with two SSD's in a mirror to hold the Data store for the esxi and the FreeNAS. All the other disks will be assigned to the FreeNAS vm. I have the raid controller built into the motherboard and also a PCI sata 3 card.

[hoppel118]

Hallo,

what about this? I want to do something like this. Did you get it to work?

Here is my project in german: http://forums.freenas.org/showthread...5-oder-Raid-Z1)

with kind regards

Hoppel

[Bever]

Hi Miniwehats,

I'm with louisk on this.....

Build 2 boxes, one for ESXi , the other for FreeNAS. If that's not an option build one box running ESXi (you'll need a proper RAID card) and use FreeNAS in a VM to share out the storage.

-Will

Actually, you don't need a proper RAID card for ESXi. Well, not really anyway. I went down the same route. I virtualized FreeNAS on ESXi5. I'm using a €100 Asus mobo that supports RAID, but it is what is called fake RAID. A software version onboard, not supported by ESX.

So instead of buying an expensive supported SATA RAID card, I searched the internet for a way to use my local disks as RDM's in ESXi. You can then just use those disks within the FreeNAS virtual, and let FreeNAS build a RAIDZ volume with them. Until now I had no issues with that. Of course, the actual FreeNAS OS is installed on a different local disk (the disk on which ESXi is installed).

[Letni]

I believe this is possible.. Here is the idea

1. ESXi would boot the physical hardware from a USB stick
- In order to have a "holder" ESX datastore to hold the configuration for the FreeNAS VM, two options are available.
A. Instead of booting from USB, use a very small IDE or SATA drive (even IDE Flash - Cardflash) that will boot ESX AND have space to hold a basic VMFS 3.x datastore
B. Have a secondary USB drive that could be used as the datastore if booting ESX from a first USB flash drive

2. A VM would be configured for FreeNAS that would have the following configuration (VM definition would have to sit on the small datastore on the first USB stick)
- Second Physical USB stick with FreeNAS loaded - configured to pass through to the VM - defined as the boot device in the bios of the VM
- Physical disks all configured as RDM devices to the VM (this is where your ZPOOL would be created)
- This VM would want at minimum 8 GB RAM to fullfill ZFS best practices, so your ESX box probably wants 24-32 GB RAM (gives enough to run other VMs)
- iSCSI target would be defined on FreeNAS to talk to defined ZFS Volumes (created in your ZFS pool sitting on physical disks)

3. ESX would be pointed back to the iSCSI target (internal networking to the box) for actual data stores for OTHER VMs and be formatted as VMFS

Unsure:
1. What type of performance to expect for NAS to FreeNAS to your ZFS datasets
2. What type of performance to expect for iSCSI back from a VM (accessing physical disks) to the physical ESXi
3. Can a ESX datastore reside on a USB flash disk also booting it..

[ZFSuser]

Hi,
I have a machine booting ESXi 5, then running Freenas 8.04, then ESXi running guests on iscsi shares from the Freenas vm.
ESXi boots from USB. ESXi 5 boots Freenas from an SSD. The SSD is required for the rdm disk creation. You cannot create rdm devices on USB devices.
FreeNAS runs ZFS on 3 x 2TB (rdm) drives. The spare space on the SSD is used by FreeNAS as a ZFS cache.

This is has been working fine, for 2 months. iSCSI, SMB, NFS all work. Various write speeds 30-100MB/s (using dd random).
*I do seem to get a hang occassionally...which started further investigations and testing. I think it is a ZFS ram usage issue.
*This is a home environment for study. I agree with the other posts, if you want pure performance your NAS should be dedicated h/ware.

Through testing (on my second box with the exact same mobo, ram, cpu) I have found a problem. And would interested in your test results.
Please refer too thread below, which has hardware details and specifics on problem.
FreeNAS-8.0.4-RELEASE-p3-x64 (11703) virtual machine on ESXI 5 with rdm disk problem

[jgreco]

I believe this is possible.. Here is the idea

1. ESXi would boot the physical hardware from a USB stick
- In order to have a "holder" ESX datastore to hold the configuration for the FreeNAS VM, two options are available.
A. Instead of booting from USB, use a very small IDE or SATA drive (even IDE Flash - Cardflash) that will boot ESX AND have space to hold a basic VMFS 3.x datastore
B. Have a secondary USB drive that could be used as the datastore if booting ESX from a first USB flash drive

ESXi cannot use a USB flash device for datastore, to the best of my knowledge.

2. A VM would be configured for FreeNAS that would have the following configuration (VM definition would have to sit on the small datastore on the first USB stick)
- Second Physical USB stick with FreeNAS loaded - configured to pass through to the VM - defined as the boot device in the bios of the VM
- Physical disks all configured as RDM devices to the VM (this is where your ZPOOL would be created)
- This VM would want at minimum 8 GB RAM to fullfill ZFS best practices, so your ESX box probably wants 24-32 GB RAM (gives enough to run other VMs)
- iSCSI target would be defined on FreeNAS to talk to defined ZFS Volumes (created in your ZFS pool sitting on physical disks)

3. ESX would be pointed back to the iSCSI target (internal networking to the box) for actual data stores for OTHER VMs and be formatted as VMFS

You'll have a bootstrap problem: the ESXi box wants to mount its iSCSI (or NFS) devices during bootup, before it starts virtual machines. Your FreeNAS instance won't be running, and so when the ESXi box finally gets around to giving up and booting anyways, it'll not start any of your other VM's because the datastores are inaccessible. At least, that's how it works on 4.1.