Friday 20 January 2012

Some thoughts on a Hyper-V Mini-Cluster

While working on a FreeBSD grid system many years ago we came across the following that we used as a model for our own designs:



Mr. Gardner built a beautiful machine from the ground up.

The project we were working on at the time was scrapped due to a number of external factors.

However, the knowledge we gained in trying to create a grid based on FreeBSD helped to gain a grasp on how we were to go about building a Hyper-V cluster a number of years back when documentation was sadly lacking.

And that brings us to the point of this blog post:


Intel’s new Intel Server Board S1200KP in Mini-ITX form factor.

And, while searching for a quality image of the board we hit this gem:


That is a really neat looking box and fits right in with some of the directions we were looking at. One catch though is that the box is made in the Ukraine! :(

It looks as though they really ran the gamut as far as the setup for a Mini-ITX based solution. Management and redundancy is built right in. Kewl!

In the end we may look at a low power cluster configuration for folks wanting some redundancy at an ultra-low entry price point. How that materializes is yet to be seen.

Born to tinker. ;)

Philip Elder
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.


Andy said...

See your instrested that Might be instrested in case we used to build some 1u daul mini itx systems icore system just beg to to test hyper v on rack full them

Philip Elder Cluster MVP said...


The one drawback that we have found with the commodity hardware is in the components themselves.

With the Intel Modular Server or Intel Server Systems that do similar things to the cases you mention we get full integration and high grade components plus management and monitoring.

For 1U we will stick with the full Intel Server Systems configuration. We will probably be seeing similar dual Mini-ITX or half Micro-ATX (dual Xeon boards are already in this configuration) in the coming generations of Intel server products.

We are not at the point where the S1200KP would be considered for anything more than a very small failover cluster setup. We still need to come up with a storage destination (Maybe Q-Nap).

Thanks for the pointers though. We may look into them a bit more when the time comes.


Anonymous said...

I just now ran across this article.

There is a lot more to say about cost vs benefits in small low power clusters.

I built the mini-itx cluster shown on in late 2003 and later posted it to early in 2004.

The reasoning and rationale for building it, and building it thw way I did was driven by several factors:

I wanted to build a Beowulf cluster with sufficient scale to have the same characteristics as larger clusters.
This meant it needed more than 2-3 nodes.

Another issue was cost. I wanted to build it cheaply. At the time, the mini-itx boards cost about $75 each , so they were picked in the early trials.

The chassis was made as the cheapest possible chassis that I could make using simple hand tools. It was just enough to keep it from falling apart.
Total cost for the tow 6-node chassis, including spare parts came to about $300

The next consideration was power. It needed to use very little electrical power so that my utility bills were not insanely high, and that generated heat would not force the need for additional air conditioning. Using IBM microdrives reduced power use a lot, reduced heating, and saved quite a lot of space in the chassis.

Heat was a concern. Nice in the winter, but a big problem in the summer. The mini-itx boards used very little power, which made a sizable home cluster a lot easier , and cheaper. Using IBM microdrives reduced power use a lot.

Another consideration was noise. Big clusters are noisy, and have a lot of fans. The low power use of the mini-itx cluster meant that it needed few fans, and was very quiet.

Lastly, size was important. I initially put it in a bookcase because it looked nice. At that time it was the highest cpu density which you could build... until Blue Gene came along...

The total cost for the mini-itx cluster was about $3000

In 2010, I repopulated the chassis with new hardware. it was faster, more cpu cores, and served as the processing backend for a high end 24 core opteron machine with a lot of ram and many big hard drives. The mini-cluster was where I could offload jobs. It was a NUMA workstation cluster with a Beowulf backend. Also, the workstation and cluster mini-itx boards had GPUs. In all, the workstation, and cluster, with GPS coudrack up andestimated2TFLP.

The total cost of that was about $11000

If you want to see what I am doing now, go to:

At about the same time, both CRAY and SGI offered similar workstation clusters that featured a big workstation backed by a cluster of mini-itx or micro-atx boards. These ran windows, not linux! These were capable clusters in their own right. They used a similar vertical chassis design , with the motherboards on hot-swap trays (a big improvement).

Their price tag was a whopping $25K!

Now those have gone away, and we have a smallish cluster in a chassis with wheels from SGI called the "OCTANE III". it is the same idea, with motherboards still on a slide out tray for easy servicing. It is the same general idea, but they seem to have gone back to the idea that using a lot of power and being large is not a bad thing.

I am not so sure that buying a ready to go solution is always the answer. It is true that it will be more developed, but you must also pay quite a lot more. If you know how to do this kind of thing and don't mind developing your own solutions, building from cots hardware is likely the way to go.

Ultimately, the companies who are offering this stuff are using the same hardware that you can buy, but they have developed it into a package and worked through most if the major issues for you.... so that is a plus... but it comes at a premium that can have a dollar price on the hardware that is elevated by three to five times what you can expect to pay for similar performance and reliability.

Glen Gardner