We really wanted to run ZFS on our NAS for best data integrity. Our previous experiment with FreeNAS showed disappointing performance with ZFS. Our final hope is to go to the home of ZFS, Solaris. This is where we’ll find the best performance.
Solaris as such isn’t really a thing any more, however, there are various open source forks which are still being maintained. We chose OmniOS, a complete Solaris-like distribution based on the Illumos kernel, and currently at r151020. The other contender was OpenIndiana, but that didn’t have driver support for our X557 10Gbe ports.
As with FreeNAS, we had to use dd on a Linux box to create USB installation media, and we used a different, identifiable, USB stick as our installation target. Installation went smoothly and all our storage and network ports showed up. It’s been nearly 20 years since we did any work on a Solaris box, so we had to resort to much googling to set up the network ports and storage the way we wanted. By this time we had some additional storage too. We had installed two Samsung 1TB 850 Pros, in addition to the SM951, with the intention of striping them to get the performance we need to match the 10Gbe network.
After much mucking around with ZFS basics, we had a mirrored boot (USB) drive (taking up both the USB 3.0 ports on the rear I/O panel), a ZFS vdev consisting of the two 850s striped for performance, and another vdev which was the SM951. We set up some SMB shares and connected our X99 Windows 10 client with its X550-T2 network card.
Interestingly, the performance of the 850 pair and the 951 over the network shares was comparable. Reads were around 900MB/s and writes were around 300MB/s.
We also tried a single 850 Pro and got reads of 500MB/s and writes of 300MB/s.
So it looks as if we can’t do anything to improve write speeds. Striping doesn’t seem to make any difference, but our reads are just what we hoped for.
This server is now in use as VM storage. VMs boot up and shut down nice and fast, and we’re happy. The box is effectively silent and idle power consumption is around 35w (with 4 SSDs and an M.2 installed) which is higher than we’d hoped, but in the scheme of things it’s pretty good, especially considering the performance we’re getting.