Storage Configuration: Know Your Workloads for IOPS or Throughput

Here we have a practical example of how devastating a poorly configured disk subsystem can be.

The above was one of the first Iometer test runs we did on our Storage Spaces setup. The above 45K IOPS was running on 17, yes seventeen, 100GB SSD400S.a HGST SAS SSDs.
Obviously the configuration was just whacked. :(
Imagine the surprise and disappointment one would have supplying a $100K SAN and ending up with the above results after the unit was put into production and the client was complaining that things were not happening anywhere near as fast as expected.
What we are discovering is that tuning a storage subsystem is an art.
There are so many factors that one needs to keep in mind as far as the types of workloads that will be running on the disk subsystem right through to the hardware driving it all.
After running a large number of tests using Iometer, and with some significant input from fellow MVP Tim Barrett, we are beginning to gain some insight into how to configure things for the given workload.
This is a snip taken of a Simple Storage Space utilizing _just two_ 100GB HGST SSD400S.a SAS SSDs (same disks as above):

Note how we are now running at 56K IOPS. :)
Microsoft has an awesome, in-depth, document on setting things up for Storage Spaces performance here:

• TechNet: Storage Spaces - Designing for Performance

We suggest firing the above article into OneNote for later reference as it will prove invaluable in figuring out the basics for configuring a Storage Spaces disk subsystem. It can actually provide a good frame of reference for storage performance in general.
Our goal for our Proof-of-Concept testing that we are doing was around 1M IOPS.
Given what we are seeing so far we will hopefully end up running at about 650K to 750K IOPS! That's not too shabby for our "commodity hardware" setup. :)
Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Cluster Starter Labs: Hyper-V, Storage Spaces, and Scale-Out File Server

The following are a few ways to go about setting up a lab environment to test out various Hyper-V and Scale-Out File Server Clusters that utilize Storage Spaces to tie in the storage.

Asymmetric Hyper-V Cluster

• (2) Hyper-V Nodes with single SAS HBA
• (1) Dual Port SAS JBOD (must support SES-3)

In the above configuration we set up the node OS Roles and then enable Cluster. Once cluster is enabled we can import our not initialized shared storage into Cluster Disks and them move them over to Cluster Shared Volumes.
In this scenario one should split the storage up three ways.

1. 1GB-2GB for Witness Disk
2. 49.9% CSV 0
3. 49.9% CSV 1

Once the virtual disks have been set up in Storage Spaces we run the quorum configuration wizard to set the witness disk up.
We use two CSVs in this setup so as to assign 50% of the available storage to each node. This shares the I/O load. Keep this in mind when looking to deploy this type of cluster into a client setting as well as the need to make sure all paths between the nodes and the disks are redundant (dual SAS HBAs and a dual expander/controller JBOD).

Symmetric Hyper-V Cluster with Scale-Out File Services

• (2) Scale-Out File Server Nodes with single SAS HBA
• (1) Dual Port SAS JBOD
• (2) Hyper-V Nodes

For this particular set up we configure our two storage nodes in a SOFS cluster and utilize Storage Spaces to deliver our shares for Hyper-V to access. We will have a witness share for the Hyper-V cluster and then at least one file share for our VHDX files depending on how our storage is set up.

Lab Hardware

The HP MicroServer would be one option for server nodes. Dell C1100 1U off-lease servers can be found on eBay for a song. Intel RS25GB008 or LSI 6Gb SAS Host Bus Adapters (HBAs) are also easily found.
For the JBOD one needs to make sure the unit supports the full compliment of SAS commands being passed through to the disks. To run with cluster two SAS ports that access all of the storage installed in the drive bays is mandatory.
The Intel JBOD2224S2DP (WSC SS Site) is an excellent unit to work with that compares feature wise with DataON, Quanta, and the Dell JBODs now on the Windows Server Catalogue Storage Spaces List.
Some HGST UltraStar 100GB and 200GB SAS SSDs (SSD400 A and B Series) can be had via eBay every once in a while for SSD Tier and SSD Cache testing in Storage Spaces. We are running with the HGST product because it is a collaborative effort between Intel and HGST.

Storage Testing

For storage in the lab it is preferred to have at least 6 of the drives one would be using in production. With six drives we can run the following tests:

• Single Drive IOPS and Throughput tests
• Storage Spaces Simple
• Dual Drive IOPS and Throughput tests
• Storage Spaces Simple and Two-Way Mirror
• Three Drive IOPS and Throughput tests
• Storage Spaces Simple, Two-Way Mirror, and Three-Way Mirror
• ETC to 6 drives+

There are a number of factors involved in storage testing. The main thing is to establish a baseline performance metric based on a single drive of each type.
A really good, and in-depth, read on Storage Spaces performance:

• TechNet: Storage Spaces - Designing for Performance

And, the Microsoft Word document outlining the setup and the Iometer settings Microsoft used to achieve their impressive 1M IOPS Storage Spaces performance:

• Microsoft Downloads: Achieving Over 1-Million IOPS from Hyper-V VMs in a Scale-Out File Server Cluster Using Windows Server 2012 R2

Our previous blog post on a lab setup with a few suggested hardware pieces:

• A Storage Spaces and Hyper-V Cluster Lab Guide

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Hyper-V 2 Node JBOD DAS Cluster IOPS Run

This is just the beginning of our testing with Iometer:


The setup is the following:

• Hyper-V Nodes
• Intel Server Systems R2208GZ4GC, dual Intel Xeon E5-2680v2, 32GB ECC, dual Intel RAID Controller RS25GB008 SAS HBAs
• Intel Storage Systems JBOD2224S2DP JBOD
• (17) 100GB HGST SSD400 SAS SSDs
• Windows Server 2012 R2 U1
• Storage Spaces with Simple across all SSDs
• Small Witness Disk
• (2) CSVs at ~760GB each

We had no idea as far as what to expect with this setup since each CSV is being managed by one node.
As we go along learning how Iometer works and how the disks react to various workloads we will publish additional results.
Then we will go on to run the same tests again only with the above setup configured in a Scale-Out File Server Cluster with a 10GbE backend facilitated by a pair of Intel Server Adapter X540T2 NICs in each node, NETGEAR 10GbE XS712T switches, and a pair of Hyper-V Nodes.
Hopefully with the Active/Active setup we get with SOFS and SMB Multi-Channel our performance will come out a bit better!
This test configuration was Option 1 in the Three Intel Server Systems based Hyper-V and Scale-Out File Server Clusters Post.
Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business