My new NAS box, running on 8.0.2

[TECK]

Update: I upgraded certain components of my NAS (December 1, 2011), in order to improve the overall performance.

Original Build Parts
1 x Fractal Design Array R2 300W SFX PSU Case
1 x Supermicro X7SPA-HF-D525-O Intel Atom D525 1.8GHz Motherboard
6 x Western Digital WD20EARS-00MVWB0 3-Platter 2TB Drive
1 x Patriot Xporter XT Boost PEF4GUSB 4GB USB 2.0 Flash Drive (FreeNAS)

Upgrade Parts
2 x Corsair CMSO4GX3M1A1333C9 4GB DDR3-1333 CL9-9-9-24 204PIN SODIMM Memory
1 x HighPoint Rocket 620 PCI-Express 2.0 x1 SATA III (6.0Gb/s) Controller Card
1 x Crucial M4 CT064M4SSD2 2.5" 64GB SATA III SSD
6 x Xion 18" SATA II with Metal Latch Cable
1 x Rosewill 10" SATA III with Metal Latch Cable

Hardware Details
I picked Supermicro because their well-known product reliability, as well because they are the only ones who offer dual Intel NIC's on an Atom based motherboard.
I decided to use the HighPoint Rocket 620 x1 controller because it has a Marvell 88SE9128 chip, supported out of the box by FreeBSD. Be aware that this is not a RAID controller. If you plan to use 2 disks, the controller will generate HUGE bottlenecks. However, when the controller is connected to a single drive there are no issues. In my tests, it allowed me to read data locally at 400MB/sec. I could not believe it.

Even if we are dealing with a x1 controller, FreeNAS detects the SSD disk as SATA III:

ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)

HighPoint added something in their new firmware, compared to previous version tested by FreeBSD developers. From their instructions manual (see page 5), they display instructions for 1.0.0.1003 BIOS version which matches their description how the controller works (2 AHCI channels). However, when I hit CTRL+M in my box the BIOS version shows as being 1.0.0.1012 and the firmware as 2.1.0.1404, which might explain the new "feature":

pass1 at ahcich7 bus 0 scbus7 target 0 lun 0
pass1: <Marvell 91xx Config 1.01> Removable Processor SCSI-0 device 
pass1: 150.000MB/s transfers (SATA 1.x, UDMA4, ATAPI 12bytes, PIO 8192bytes)

If you have any insides related to this matter, please let us know. I attempted to contact Highpoint support without success, they never replied to my emails neither responded to my technical support ticket.

I recently found out that the X7SPA-HF-D525 supports up to 8GB of RAM, even if Supermicro advertise it at 4GB. The manual also says that 1666 MHz and 1333 MHz memory may be used, but will operate at 800 MHz. With the memory upgraded, FreeNAS is reporting 8177MB of RAM:



I also think there is a myth related to the fact that Atom processors cannot handle more than 4GB of RAM. I'm not sure if all Atom processors support the larger memory setup, or just the Atom D525 installed on my Supermicro motherboard. My system uses on a regular basis 6GB of wired RAM, which is technically impossible based on Intel specs:



Hardware Setup
The OKGear 20cm SATA 6Gbps cable I initially used to connect the SSD disk was not "OK" at all. The sleeve snapped right away when I tried to bend a little the cable and hook it into Rocket controller. I ended up using a Rosewill 10" SATA III cable instead.

The disks cage, ready for RaidZ2:



The "skinned down" USB stick that holds the OS:



I had a hell of a time to get the cables organized inside, in order to achieve a clean wiring:



The Rocket controller that adds a SATA3 port for Crucial SSD:



Disk Specs
The dmesg info related to disks (da0 USB disk, ada0 SSD disk, ada1 regular disk):

da0: < Patriot Memory PMAP> Removable Direct Access SCSI-0 device 
da0: 40.000MB/s transfers
da0: 3824MB (7831552 512 byte sectors: 255H 63S/T 487C)

ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich8 bus 0 scbus8 target 0 lun 0
ada1: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich9 bus 0 scbus9 target 0 lun 0
ada2: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich10 bus 0 scbus10 target 0 lun 0
ada3: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ahcich11 bus 0 scbus11 target 0 lun 0
ada4: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ahcich12 bus 0 scbus12 target 0 lun 0
ada5: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada6 at ahcich13 bus 0 scbus13 target 0 lun 0
ada6: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

Disk Read/Write Stats
My setup has all FreeNAS settings set to default, no special tweaks or adjustments.

Hyper-threading enabled:

$ dd if=/dev/zero of=/mnt/nas/media/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 602.091800 secs (178335235 bytes/sec)

$ dd if=/mnt/nas/media/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 363.791025 secs (295153467 bytes/sec)

Hyper-threading disabled:

$ dd if=/dev/zero of=/mnt/nas/media/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 685.600273 secs (156613389 bytes/sec)

$ dd if=/mnt/nas/media/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 401.614066 secs (267356628 bytes/sec)

I get better results with hyper-threading enabled, anything else related to BIOS configuration has stock settings.

Data Transfer Speed
Based on my comparison tests done before and after upgrade, the extra memory and Rocket + Crucial combo set as cache drive boosted the data speed by over 500Mbit/sec. Previously, I was reading data at 400Mbit/sec and writing it at 300Mbit/sec. With the SSD cache enabled and the memory upgrade, I can read data at 900Mbit/sec and write it at 600Mbit/sec.

CIFS reads from Network Drive:



CIFS writes to Network Drive:

[ProtoSD]

Looks sweet, I like the drive cage design with the drives verticle... unlike my Lian case where the drives are stacked horizontally so the heat accumulates on up the stack.

I don't even see where your USB stick is... Anyway, nice clean build. I might only recommend SATA cables with clips so they don't wiggle loose somehow, or get accidently yanked while your digging around for some reason.

[TECK]

Thanks. I looked at your case also but decided to go with Fractal Design because it was smaller and did not have a cut for DVD drive. The case is ultra quiet with all 6 drives humming inside, very happy with the end results. I plan to use this box for storing all my development work (Linux) and media files (Windows).



Wiring "fun", playing with the Xion SATA cables:




Part of my working desk:

[ProtoSD]

Now I think I see your USB stick, no wonder I didn't recognize it, it's all naked! You shouldn't need to worry about it getting too hot, they're very low power devices for one and you've got everything so clean that there shouldn't be any heat build up significant enough to cause problems.

I like my case even though the drive cage is poorly designed because with my PicoPS I have a whole lot more room to add more drives if I install another controller. I also plan to mod my drive cage so the drives are vertical like yours at some point.

To be honest, I never had any problems with permissions on 8.0 because I pretty much opened them wide. My network is pretty secure, so I'm not worried. Supposedly the permission problems were fixed, but I'm waiting to hear from someone that could be still having problems, we'll see.

I use CIFS and NFS because I have a MythTV system and mount my NAS to it. I never tested NFS with Windows, so you could be right.

[Bohs Hansen]

Impressed with how good you got your cables aligned. Didn't manage that myself with the same case Well I didn't really try to hard either

[Darkaine]

I... Just might have to get that case. Very clean and efficient. Nice work.

[TECK]

Thanks guys for the nice comments.

I use CIFS and NFS because I have a MythTV system and mount my NAS to it. I never tested NFS with Windows, so you could be right.

I see. Well, apparently in 8.0.1 you only need to enable the NFS shares and that will do it in both Linux and Windows. I find it redundant to use CIFS/SMB for Windows and NFS for Linux. When I mean "permissions", I refer to the current 8.0 issue where in Windows you can create directories but not files on a NFS share. I opened tot eh max the perms, I could not do anything about it. I think is best to open a separate thread for that.

P.S. I'm glad you used a Supermicro Atom mobo, they rock the dual NIC's.

[TECK]

Upgraded my box, see the OP for hardware changes.

Based on my comparison tests done before and after upgrade, the extra memory and Rocket + Crucial combo set as cache drive boosted the data speed by over 500Mbit/sec. Previously, I was reading data at 400Mbit/sec and writing it at 300Mbit/sec. With the SSD cache enabled and the memory upgrade, I can read data at 900Mbit/sec and write it at 600Mbit/sec.

8GB of memory on the Supermicro X7SPA-HF-D525-O Intel Atom D525 1.8GHz mobo:



Network Disk to Local Disk (almost tapping the max network speed):



Local Disk to Network Disk:

[Letni]

Very Nice Rig!

I'm still a n00b here trying to discover the flaws of this product before I trust my data on it (I currently build CentOS boxes to run as custom NAS servers, so I'm not a n00b in that sense)... But couple of questions.

1. Do you have spindown successfully functional with your rig on FreeNAS 8.X?
2. Do you Bond the dual NIC on the supermicro board ? If so, do you notice any performance benefit
3. Do you know if the SSD drive gave you more benefit or the additional 4 GB Ram?
4. How did you enable the "console" output within the web GUI at the bottom? (its in your screenshot above)


Thanks

LetnI

[TECK]

1. I don't want my disks to spindown, the NAS is used mostly for development files shared across several CentOS boxes. However, the motherboard has ACPI/APM power management but the disks show no APM support. Here it is the info you need:

# ataidle ada1
Model:                  WDC WD20EARS-00MVWB0                    
Serial:                 WD-WBDXU6435219
Firmware Rev:           51.0AB51
ATA revision:           ATA-8
LBA 48:                 yes
Geometry:               16383 cyls, 16 heads, 63 spt
Capacity:               1863GB
SMART Supported:        yes
SMART Enabled:          yes
Write Cache Supported:  yes
Write Cache Enabled:    yes
APM Supported:          no
AAM Supported:          no

2. I use a Cisco SA520W-K9 mixed with a SG 300-10 switch, so I can use LACP without issues. Reading data at 115MB/sec is more than enough for my needs, so I use one NIC now. However, I know for a fact that bonding the 2 NIC's will increase the speed to over 150-160MB/sec. The problem is none of current drives I use in other boxes are worth the trouble, they would not handle properly the amount of data thrown at.

3. The added RAM is mostly the big gainer. The SSD cache helps if your working dataset is larger than system RAM, but small enough that a significant percentage of it will fit on the SSD. In my case, I have a dataset with several MySQL databases that is 30GB in size. So that set will get cached all the time on a fast drive, serving data at same speed but avoiding the ZFS hits. Again, this is is really beneficial if you run your connections through LACP... Since I like technology, I wanted to take advantage of all features FreeNAS offers related to performance.

4. There is an option in Settings > Advanced > Show console messages in the footer