In this article I list the specifications of the crunchers I am currently running on World Community Grid. It follows a discussion, why I chose the components I did.My main aim was to build machines that are as cost-efficient as possible, meaning maximum output with minimal cost. To achieve that the following requirements must be met:
- high energy efficiency, as electricity is expensive here
- low component costs, as long as this does not hurt performance too much. For high energy efficiency however, it is worth spending a bit more
First, a picture of them all together. They are in a metal cabinet, which has some holes at the bottom to make cabling easy. Some case fans are blowing fresh air over them.
Here is a list of the components and some specifications:
|CPU||Ryzen 1700||Ryzen 1700||Ryzen 1700||Ryzen 2700|
|Clock rate||3025 MHz||3025 MHz||3025 MHz||3225 MHz|
|Vcore||900 mV||994 mV||975 mV||963 mV|
|CPU Cooler||Wraith Spire||Wraith Spire||Wraith Spire||Enermax ETS-N31-02|
|Mainboard||ASRock AB350M||ASUS Prime B350M-K||ASUS Prime B350M-K||ASUS Prime B350M-K|
|RAM||1× 4 GB DDR4-2400||1× 4 GB DDR4-2400||1× 4 GB DDR4-2400||1× 4 GB DDR4-2400|
|Drive||32 GB M.2 SATA SSD||16 GB M.2 SATA SSD||500 GB 2.5″ SATA III HDD||250 GB 2.5″ SATA III HDD|
|Power supply||Dell D265EM, 265 W, 80+ Gold||Dell D265EM, 265 W, 80+ Gold||Dell D265EM, 265 W, 80+ Gold, with power splitting cable feeding 2 mainboards|
|Operating System||Ubuntu Server 18.04||Ubuntu Server 18.04||Ubuntu Server 18.04||Ubuntu Server 18.04|
|Power consumption at full CPU load
|~ 76 W||~ 76 W||~ 75 W||~ 75 W|
Here are some more pictures of them:
This is Ryzen 1700_1. The GPU in it is only there temporarily for troubleshooting, a Radeon HD5450.
This is Ryzen 1700_2:
This is Ryzen 1700_3. You can see the ATX power splitting cable and the PSU feeding this and Ryzen 2700_1 at the same time. You also see the two HDDs of Ryzen 1700_3 and Ryzen 2700_1 behind.
This is Ryzen 2700_1:
Now, here are some comments on why I chose the components I did.
The Ryzen 1700 has 8 cores/16 threads and a TDP of 65 W. It is more or less identical to the Ryzen 1700X, which only runs at higher clock rates – but consumes also more energy. Not a good choice for a cruncher, like I described in an article about cruncher efficiency. The Ryzen 2700 is the successor, with almost identical specifications but manufactured with the 12 nm process instead of 14 nm. It can therefore run at higher clock rates. Base clock is 3000 MHz for the Ryzen 1700, 3200 MHz for the Ryzen 2700, at which they need approximately the same power. Although the Ryzen 2700 is a bit more efficient, the 1700 is still a good choice. It can be bought for much less (either used or new). If the higher price for the Ryzen 2700 is made up by its slightly higher efficiency is not clear and depends on how high electricity rates are. At my high rates the picture is not clear, so I would decide between 1700 and 2700 depending on which I get a good deal on.
What would be alternatives? I think the Threadripper 2990WX might also be a good choice. It has 32 cores – 4 times of what the Ryzen 2700 has! The cores have the same architecture, so the CPU speed per core should be the same at the same clock rate. It is also much more expensive – you definitely pay more per core. However, it could be slightly more energy efficient as the Ryzen 2700, as you maybe have not 4 times the overhead of the other components. However, I do not have precise real-life data on that, so take that as speculation. Unfortunately, it runs not on the AM4 platform as the Ryzen 1700 and 2700, but on TR4. Mainboards for that are much more expensive. As the Threadripper 2990WX has a masive TDP of 250 W, cooling is also not as easy.
What about Intel? As of today, I know no Intel CPU that comes close to the Ryzens when it comes to cheap, efficient multi-threaded performance (when you need high performance per thread, like for gaming, that is a completely different matter).
What could be worth waiting for is the third generation Ryzen. They will likely have more than 8 cores, possibly up to 16. They are expected to be introduced in the second half of 2019.
Clock rates and Core Voltage
I run all CPUs at low clock rates for high energy efficiency. I also undervolted the CPU to further decrease power consumption. Strangely, both mainboard models I used ignore all voltage settings when I did not overclock. So I “overclocked” them to 25 MHz above stock.
The core voltage to which a CPU is able to be undervolted while running stable is a bit different for every CPU and must be tested. I did this by lowering the core voltage step by step, until the machine crashed as soon as BOINC started. From there I increased voltage by 37.5 mV again. This seems a bit crude, but I tested for several months which value works. At 25 mV above “crash voltage” for example, the machines ran also fine, but work units aborted every few days due to computation errors.
The voltage of Ryzen 1700_1 seems to be significantly lower than that of the other machines. However, the voltage in the table is the setting in the BIOS. The real voltage at the CPU might be different. I tried running 1700_1 on one of the other boards and had to enter a higher value there, too.
Three of my Ryzens run with Wraith Spire, that comes with the boxed version of the Ryzen 1700. These are very decent coolers. Even with mild overclocking they run fine. Only if you want to try a more extreme overclock or want an extremely silent cooler, I would recommend something else. As I do not overclock, they cool the CPU very well. CPU temperatures can be as low as 45 °C (113 °F) in the winter and in the summer they always stayed below 70 °C (158 °F).
My Ryzen 2700 uses another cooler, as I got it though a good deal without the boxed cooler. Then I decided to get a cheap tower cooler (a design with heat pipes, the Wraith Spire, a so-called top blow cooler uses no heat pipes) and see how it compares to the Wraith Spire. The Enermax ETS-N31-02 was the cheapest tower cooler I could find (about 20 US$). CPU temperatures are around 3-4 °C lower as with the Wraith Spire. So no noticable improvement, but for my purpose I see no reason to spend more for a cooler – with overclocking the situation would be different.
First thing to note is, I used very basic mainboards, which I figured are enough for my purpose. I do not need any fancy features, many PCIe slots or anything. Of course, the VRM’s of cheaper boards are less powerful, but with low clock rate and undervolting this is also nothing I need.
What I definitely needed was the ability to undervolt, and that is typically coupled with the ability to overclock. There are three lines of chipsets for the AM4 socket: Ax20 (A320), Bx50 (B350, B450) and Xx70 (X370, X470). The A models allow no overclocking and although they are really cheap they where no option for me therefore. The B models are still relatively cheap and allow overclocking, so the B350 chipset was the optimal choice (the B450 only adds support of the 2. generation Ryzen’s Precision Boost Overdrive, which I do not need anyway). The X370 adds more connectivity options as well as SLI support – things I don’t need.
Regarding form factor: the smaller the better. Mini-ITX would be the optimal choice, but the boards are much more expensive. And as I am not too much space restrained, I dropped this option and chose a mATX board.
The boards I chose – the ASRock AB350M and the ASUS Prime B350M-K work well, but on the other hand I have no idea how other similarly priced boards would perform. I would probably prefer the ASRock boards over the ASUS next time, as this has a VRM heat spreader, which the ASUS has not. But for now this was no problem. Even in the summer the VRM temps were still fine – with overclocking, again, this might be different.
What about longevity? Are these relatively cheap boards really up to the task of running 24/7? Very hard to say from my small sample size, but for now everything is fine. The ASRock board runs since 14 months now, the ASUS boards since 7, 7 and 5 months. (update more than a year later, April 2020: all four boards still running without problems 24/7)
4 GB of RAM for a 16 thread machine seems low. However, for my purpose it is enough and my aim was to spend as little money as possible for a maximum output. The operating system itself needs only several 100 MB of RAM and many of the current World Community Grid projects also do not need much RAM. I have to be cautious with Microbiome Immunity Project (~300 MB RAM per unit currently) and Fight AIDS@home 2 (~500 MB currently) and limit the number of these run simultaniously with app_config.xml.
The clock rate of the RAM from what some team mates tested does not play any significant role for output of current WCG projects, so I settled with reasonably priced 2400 MHz modules.
With one module, dual channel of course does not work and the RAM runs in single-channel-mode only. Also not of great significance at WCG projects currently.
First I installed Linux on 16 GB USB sticks – absolutely fine, as WCG does not write much data to the disk. However, after some months several of them failed. Buying higher quality USB sticks might have solved this – I know others who run their crunchers on USB sticks without problems. However, I found the price difference of these compared to small SSDs too low. So I bought two M.2 SSDs. I like the form factor, as you simply plug them into the mainboard and need no cables. Small SSDs are really cheap now, and as BOINC does not need much space, even the 16 GB SSD in one of them is sufficient.
For the other two I used old 2.5″ drives from my laptops, which I exchanged for SSDs years ago. 2.5″ drives should consume only slightly more than SSDs and were an option therefore, as I had them anyway. 3.5″ HDDs are no option because they consume more electricity, which adds up over time
To prevent problems with the limited life of SSDs when too much data is written to them, I allowed to “write to disk at most every 1200 seconds” in the device profiles.
Not much to write here: I used a GPU only to set up the system (a cheap Radeon HD5450). Now they run without a GPU and I use SSH to connect to them. I measured the difference in power consumption: it is about 10 W for the whole system – a significant reduction in electricity consumption. And the HD5450 is an efficient GPU at idle.
The efficiency of the power supply is essential for overall electricity consumption, so I wanted an 80+ Gold rated one (80+ Platinum and Titanium are even better, but too expensive for the benefit I think).
As power supplies should be run at 50% load for maximum efficiency and my crunchers use very low power, it was not easy to find something suitable. I found some 80+ Gold rated used PSUs from a Dell Optiplex on Ebay very cheap and used them. They are only 265 W PSUs, but still underutilized. So I bought a power splitting cable for 24-pin ATX and the 12V-4pin-CPU connector and two of the crunchers run on one PSU now. The power consumption of the two machines combined decreased slightly, but not by very much. Unfortunately this did not save much money – the power splitting cables, although shipped from China, cost almost as much as another PSU would have cost (I got them for 15 Euros apiece).
I first used Lubuntu as I was not too comfortable with Linux command line in the beginning. But when I decided to remove the GPU I switched to Ubuntu Server.
The power consumption of the crunchers is approximately 76 W for the whole system at full CPU load. People are often amazed when they hear this number. But it is really nothing special when some essential rules are taken into account: low clock speed, undervolting and efficient PSU being the most important ones.
The values are only approximate, as they fluctuate a bit. Also with some projects (MIP especially) they fall even a bit lower. The machines sharing a PSU use even a bit less, as the PSU efficiency is higher when it is better untilized – the HDDs instead of SSDs offsetting part of it.