When he just had to go faster, Brian O’Connor knew what he needed.
Here at MMIT, we need to go faster too. We’ve had a QNAP TS-453S Pro NAS unit for a couple of years. We chose it because its use of 2.5 inch drives makes it much more compact than a NAS that uses 3.5 inch drives, and also because we were intending to use SSDs in it. The main reasons for this were low noise and low power consumption, but also so we could guarantee to max out its two 1Gbe network connections for two users simultaneously.
But now our use case has changed and we are doing quite a lot of virtualization. VM files can be 100GB and larger, and waiting for them to copy, back up, or even load from a share was getting tedious, so we starting thinking about 10Gbe. Qnap’s 10Gbe units are large, expensive and loud, so we started to think about designing our own.
We wanted it to be silent (or as close as we could get), low power consumption (so it could be left on day and night for remote access), as small as possible, as cheap as possible, to have the ability to saturate a 10Gbe link, to house several terabytes of storage (although not dozens – we’re going to have to go with flash after all, to get the performance and the quietness), and, ideally run ZFS, for best data security. That’s quite a set of requirements to juggle.
“I need one of these, one of the big ones, no actually, let’s make it two”
The Intel Xeon D was the obvious place to start, as this SoC is very power efficient and integrates Intel X557 10Gbe right on the chip. Supermicro makes a wide range of mini-ITX motherboards with these chips on, with 10Gbe RJ45 ports. (Yes, SFP+ is more power efficient, but we went with RJ45 for backward compatibility with 1Gbe and existing (short) Cat 6 cabling, and ease of use. Port count is going to be low.) The lowest TDP available for one of these CPUs is 25 watts, so we went with that (silent, low power consumption remember). The motherboard we chose was the X10SDV-2C-TLN2F. This has a 2-core, 4-thread Pentium D1508 (a Xeon in all but name, still supports ECC RAM and has the all-important onboard X557). We don’t need a particularly beefy CPU for this server. It might do some transcoding but it’s unlikely to be more than one stream at a time. The main concern was that Samba shares are single-threaded and need decent GHz to max out a 10Gbe link. This one is 2.2GHz with boost to 2.6. That’s as high as we could get in a 25w TDP. We didn’t know if it would be enough, but it was all we had.
https://www.lambda-tek.com/-~sh/B3136222
To that we added a 32GB stick of Hynix ECC RAM from Supermicro’s compatibility list. The motherboard has four DDR4 DIMM slots so this lets us retain the option of maxing out the RAM at 128GB in the future if desired. FreeNAS will use everything you can throw at it, right? And yes, we’ll only get single-channel performance for now but…
https://www.lambda-tek.com/-~sh/B2377762
The motherboard has a 1U high passive heatsink over the CPU, designed for a 1U chassis with front-to-back airflow provided by noisy datacentre-style fans. We can’t be doing with that. This will not be rack-mounted anyway. We got a Noctua NF-A6x25 fan for the CPU, and we’ll probably have to hot-glue the rubber mounting pins to the heatsink. At least the mounting holes line up with the heatsink bolts. Props to smithydll on YouTube for his tip about this fan. https://www.youtube.com/watch?v=mgswfpkkoMI
We went for the FLX version, that comes with the low noise adapter and the ultra low noise adapter rather than the PWM version which adapts its own speed, because we wanted to have absolute control over noise. We would monitor temperatures carefully.
https://quietpc.co.uk/nf-a6x25
We couldn’t see this system drawing any more than 100w under any circumstances and probably no more than 50w under any realistic load, so we chose a Streacom Nano 120w power supply. This has an external power brick meaning less heat in our case, and is fanless and therefore totally noiseless.
We were taking a risk here. I remember servethehome reporting that a X11SBA-LN4F wouldn’t post with a Mini-Box PicoPSU, which is a similar design. We really didn’t want to go even down the SFX route (let alone plonk for an ATX) due to case size constraints, efficiency concerns, heat and noise, so we took the plunge and ordered.
https://quietpc.co.uk/st-nano-psus
We’ll come on to storage later. For initial testing, we had a 512GB Samsung SM951 lying around, which should be fine for network performance testing with the various candidate operating systems that we have. It should easily allow us to max out 10Gbe in both directions, and it fits neatly into the M.2 slot on the motherboard. (Currently out of stock at the link below where I got it, but don’t worry, 960 Pros are out now!)
https://www.flexxmemory.co.uk/categories/storage.html?_bc_fsnf=1&Form+factor=M.2&Size=512GB
Everything went together pretty well. We tried the CPU fan without any speed reduction at first. We plugged the IPMI port into our main 1Gbe switch, touched the two power pins together with a screwdriver tip and held our breath. The fan spun. The green LED glowed. Phew. The PSU is compatible. That was a big relief! After this stage you can just switch it on and off from the IPMI. The only other thing we did at this stage was flash the BIOS to 1.1c which we found on the Supermicro website. We were hoping to find updated IPMI firmware too, but no dice yet. The current firmware uses a clunky Java remote console which only works with Internet Explorer (what’s that?). Some Supermicro Xeon D boards now have a nice new HTML5 version but I guess we’ll have to wait for that. The fan ran at 2800 rpm and was a bit noisier than we’d hoped but we had plenty of scope for making it quieter later. Temperatures settled down to 28 C at the CPU and 42 C at the 10G chip at idle.
Just for fun we took the fan off the CPU to see what would happen. The CPU temperature rose steadily and reached 65 C after about 10 minutes. The 10G chip was pushing 70 C. Both were still rising at this point and we chickened out and put the fan back on. If it’ll do that at idle with no case we certainly won’t be able run it fanless. Ah well.
As this is our first 10Gbe system, we need a client system to test it with! So finally we bought an Intel X550-T2 for one of our X99 Windows 10 client systems, and plugged it in.
https://www.uk.insight.com/en-gb/productinfo/network-adapters/0005970053-00000001
We still lack a case and some proper storage, but at least we can now do some testing.
Bring on the software, charge up your NOS tanks and let the street-racing begin!
Part 2 here: https://www.mmit.me/blog/2016/10/28/i-need-nas-part-2-performance-testing/
Thanks to Paul Braren at TinkerTry for his ceaseless research on Supermicro Xeon D boards. Without you this would not have been possible. https://tinkertry.com
