Part 1 here: https://www.mmit.me/blog/2016/10/28/i-need-nas-the-10gbe-home-nas-guide/
We have a number of candidate operating systems and disk formats which we could run on this system. The requirement is Windows or CIFS shares. So we have plain old Ubuntu Server with Samba, on ext4 or ZFS disks, Windows Server 2016, FreeNAS, NAS4Free, and the Omnios/Napp-it combination if we’re feeling really amititious. We started off by ordering a bunch of these:
https://www.amazon.co.uk/gp/product/B00LLER2CS/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1
and a box of these to tag them with (thanks Paul at TinkerTry for the tip-off about the USB sticks and tags):
https://www.amazon.com/gp/product/B001E682C2/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1
Inevitably we tried the easiest first. FreeNAS 9.10 was first up. The ISO went onto one of the USB sticks and a fresh stick went into the machine as an install target. We had to use Linux and dd to write the ISO to the USB stick; Rufus wouldn’t have it. When you boot the Supermicro (F11 for boot menu), what’s nice is that even though the installer USB and the target USB were identical, it didn’t show the installer USB as a possible target, so there was no confusion about where to install it. Installation went smoothly. Once FreeNAS was up, it’s just a matter of pointing a web browser at it. The only things to do were to create a volume using the SM951 and a shared folder under it, create a user, then ensure the user owns the share.
To connect the two machines we used just a straight 10m Cat 6 patch cable and static IP addresses using a different subnet to the rest of the network. 10Gbe has auto-crossover built in so you don’t have to worry about that any more. (During install we had the cable plugged into the main 1Gbe switch using DHCP so we had internet access in case it was required for the install.)
We pointed Windows Explorer at the share, entered the username and password, and bingo. We have a NAS. We found a 5GB DVD rip to test with and dragged it to the share. We saw about 150MB/s. Disappointing. Then we dragged it the other way, back off the share, and saw about 300-400MB/s (bouncing around quite a bit). Better but still disappointing. Then we remembered about jumbo frames. We set up the client NIC (the X550-T2 if you remember) to 9014 frame size in Windows 10, and the server mtu to 9014 using ifconfig in the FreeNAS shell at the console. Upload increased to about 200MB/s – still disappointing, and download increased to… 1GB/s! Yay! That’s what I’m talking about!
This is good. It means several things. Firstly that our trusty old 10m Cat 6 cable is capable of negotiating a 10G link. Secondly it means the X557 network drivers in BSD are up and running. Thirdly, and most importantly, it means the CPU on the server is capable of saturating a 10G link on an SMB share. This is big news. We’re essentially home and dry. We spent a lot of money to get to this point with no guarantee of success, but now we can go ahead and spend some more on the rest of the system (like a case!).
FreeNAS uses ZFS of course, and the way ZFS writes files is quite complicated. Firstly it’s a copy-on-write filesystem, which means that data is never overwritten. If you write the same file twice, the second copy is written to an unused area of disk, and only when it is verified as having been written correctly (and its checksums calculated and compared) is the old copy released. Furthermore, writes are cached. If you only have one disk, they are cached to that disk, then read off again and re-written to a permanent location on the same disk. This is quite arduous, even for an SM951. It’s not surprising that writes are significantly slower. There’s also the issue of synchronous and asynchronous writes. The default setting of synchronous adds even more delay. Your humble correspondent is not going to write a diatribe on ZFS inner workings and tuning here; the internet is already awash with such things, written by people much better qualified. But we have an understanding of why our writes are not as blistering fast as our reads. In fact, searching the web for reasons for slow writes to ZFS we find many stories of people complaining of 10MB/s. We consider ourselves fortunate.
We suspect ZFS will never give us writes anywhere near 1GB/s, at least not without considerable research, experimentation, tweaking and tuning. Please feel free to chip in in the comments if you have any experience of this. Ideally we’d like it not to try to cache at all, and just write to the M.2 drive straight off. Anyone know how to do this? Also we’re not averse to a bit of asynchronous writing (although on paper it shouldn’t be necessary – the drive is faster than the link). Fiddling about with aio_fork on the SMB share didn’t seem to do much.
So armed with our newly and expensively acquired knowledge, we move on to Ubuntu Server 16.10 and ext4 in the hope of finding the goodness of 1GB/s writes. On this occasion Rufus does the business with the ISO quite happily (select dd format when given the choice), but the issue of having identical installer and target USB sticks leads us to come unstuck. The Ubuntu installer is not clever enough to remove the device that it is running on from the list of possible install targets. If you are installing to a USB stick, make sure you have two different ones that will identify differently (different capacities is ideal – we had a 16GB and a 32GB, both of which are more than big enough). Choose to install the Samba server when given the opportunity during the install so you don’t have to install it manually later. The SM951 was still formatted as ZFS from the FreeNAS install so it didn’t show up, although it was present as /dev/sda. A quick parted sorted that out and mkfs then turned it into ext4. It’s then a simple matter to mount it under your home directory. Don’t forget to change the ownership from root otherwise Windows won’t have a hope of talking to it. Then it’s a little bit of jiggery pokery in /etc/samba/smb.conf, set up jumbo frames again, and you’re off.
So, back to Windows 10 and mount the share. Drag the DVD rip onto the share and… 900+MB/s! Not disappointed anymore. Drag it back – 1GB/s. Nice. Just for fun we took an 80GB VM file off the Qnap and plonked it on the Win10 machine. That took about 12 minutes (showing 112MB/s – pretty good for 1Gbe). Drag it to the I Need NAS share and it wrote in about 1’30”. Drag it back and it read in about 1’15”. That pretty much bears out what we saw. Reads about 10% faster than writes. I’m happy with that.
More to come…
Part 3 here: https://www.mmit.me/blog/2016/11/23/i-need-nas-part-3-final-assembly/