Hyper-V 2 Node JBOD DAS Cluster IOPS Run

This is just the beginning of our testing with Iometer:


The setup is the following:

• Hyper-V Nodes
• Intel Server Systems R2208GZ4GC, dual Intel Xeon E5-2680v2, 32GB ECC, dual Intel RAID Controller RS25GB008 SAS HBAs
• Intel Storage Systems JBOD2224S2DP JBOD
• (17) 100GB HGST SSD400 SAS SSDs
• Windows Server 2012 R2 U1
• Storage Spaces with Simple across all SSDs
• Small Witness Disk
• (2) CSVs at ~760GB each

We had no idea as far as what to expect with this setup since each CSV is being managed by one node.
As we go along learning how Iometer works and how the disks react to various workloads we will publish additional results.
Then we will go on to run the same tests again only with the above setup configured in a Scale-Out File Server Cluster with a 10GbE backend facilitated by a pair of Intel Server Adapter X540T2 NICs in each node, NETGEAR 10GbE XS712T switches, and a pair of Hyper-V Nodes.
Hopefully with the Active/Active setup we get with SOFS and SMB Multi-Channel our performance will come out a bit better!
This test configuration was Option 1 in the Three Intel Server Systems based Hyper-V and Scale-Out File Server Clusters Post.
Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Three Intel Server Systems based Hyper-V and Scale-Out File Server Clusters

Here are three base Intel Server Systems configurations we are working on for our Intel Modular Server replacement in a Data Centre or client setting.

Unfortunately, the Intel JBOD does not self-power at this time. So, for SMB/SME solutions we will be supplying a DataON DNS-1640 2U JBOD as it will automatically power-up after a full power outage.

All solution sets are based on Windows Server 2012 R2 as a starting point for Hyper-V, Storage Spaces, and SOFS.

• Option 1: Asymmetric Hyper-V Cluster via Storage Spaces CSV
  • Intel Server System R2208GZ4GC, Dual E5-2640, 128GB ECC or 256GB ECC, 120GB SSD RAID 1, dual SAS HBAs, add-in Intel i350T4 PCIe
  • Intel JBOD2224S2DP
• Option 2: Hyper-V Cluster via SMBv3 Scale-Out File Server cluster and Storage Spaces
  • Intel Server System R1208JP4OC, E5-2640, 128GB ECC, 120GB SSD RAID 1, dual SAS HBAs, Intel X540T2 I/O Module, Intel X540T2 PCIe
  • Intel JBOD2224S2DP
  • Intel Server System R1208JP4OC, E5-2640, 128GB ECC, 120GB SSD RAID 1, Intel i350T4 PCIe, Intel X540T2 I/O Module, Intel X540T2 PCIe
  • NETGEAR XS712T 10GbE Switches
• Option 3: Hyper-V Cluster via SMBv3 Scale-Out File Server cluster and Storage Spaces with enclosure resilience
  • (3) Intel Server System R2208GZ4GC, Dual E5-2640, 128GB ECC, 120GB SSD RAID 1, SIX SAS HBAs, Intel X540T2 I/O Module, Intel X540T2 PCIe
  • (3) Intel JBOD2224S2DP
  • (2) Intel Server System R2208GZ4GC, Dual E5-2640, 128GB ECC, 120GB SSD RAID 1, Intel i350T4 PCIe, Intel X540T2 I/O Module, Intel X540T2 PCIe
  • (2) NETGEAR 24-Port 10GbE Switches
• Storage Networking Option
  • Option 2 and Option 3 can be facilitated by InfiniBand NICs and Switches
    • Enables RDMA and 56Gbps per connection
    • Microsoft's 1.4M IOPS demo based on InfiniBand backend
    • Intel Server Systems have an InfiniBand I/O Module with the second being a Mellanox PCIe

The first setup is relatively simple while the second two require some structuring around how the networking is configured to allow for SMB Multi-Channel on the storage network side.

At this point the above setups utilizing Intel Server Systems provide us with an amazing value for our IT budgets.

5 year warranties and next business day on-site support options can be had too.

We purchase our Intel Channel product primarily through ASI Canada. Ingram Micro, Synnex Canada, and Tech Data Canada are also Intel Authorized Distributors.

As an FYI we continue to build our own server systems because the experience proves to be invaluable when it comes to troubleshooting problems especially when software vendors are pointing fingers.

Building our own systems also gives us a very strong foundation for creating server configurations that will work with a client workload set.

And finally, it allows us to be very particular with Tier 1 vendors when it comes to creating a server configuration using their hardware.

EDIT: Note that we _always_ install a physical DC on our cluster networks. For option 1 it would probably be an HP MicroServer while the others would be a 1U single socket with some storage for ISOs.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Microsoft Fails Long Term Open Licensing Customers with Office for iPad O365 Requirement

We received an inquiry from one of our clients about obtaining Office for their iPad. Obviously the paid version is required in order to edit Office documents.

We fired an e-mail off to our licensing contact at Synnex Canada and received the following:

The clear-cut answer is that Microsoft has seemingly set up a requirement for Open License customers to double pay.

How is that?

Well, in the case of our clients they have been on Open Value Agreements for quite a few cycles in some cases. When OVL hits the third renewal (years 7-9) the SA Only costs bring the overall cost of that license and SA close to the OEM costs.

For clients that have regular refresh rates of about 36 months for their equipment this makes sense on so many levels.

So, our clients have already purchased their Office Professional Plus or Office Standard licenses and are on SA Only for them.

To get Office for iPad for their devices there is no SKU to add to their existing agreement. They would have to purchase an Office 365 subscription either personally or for the organization in an O365 MidSize Business plan which on Open Value is $200/Year/User.

Ouch.

It looks like we've just been double billed.

IMNSHO this decision of Microsoft's to force the Office 365 subscription for existing Open License clients is very poorly thought through.

This situation really begs the question as to whether Open License is a valued proposition going forward as Microsoft further segments products into the subscription side to force businesses that have already paid for their licenses to pay yet again ... and again ... and again.

Hopefully Microsoft will come up with something to remove the double billing.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Some Mandatory Reading for Sys Admins

There have been some critical human attributable error related news stories that we have seen lately.

Many of us, if not all of us, have had situations that have had negative impacts on our client's networks or our own networks due to something we did or did not do.

The following four links come via the Patch Management list serve (Ben S.):

• The Unofficial Unix Administration Horror Story Summary Part 1
• The Unofficial Unix Administration Horror Story Summary Part 2
• The Unofficial Unix Administration Horror Story Summary Part 3
• The Unofficial Unix Administration Horror Story Summary Part 4

As with anything out there on the Interwebs take a PDF of the four parts to keep them safe. :)

There are many pearls of wisdom contained in the above pages as well as some humorous stories that we _all_ can relate to.

• Lesson #1: Learn from our mistakes
• Lesson #2: Keep a detailed history of our mistakes
• Lesson #3: Learn from our mistakes

When we make a mistake that directly impacts our clients we make sure to apologize and make some sort of restitution offer to them.

Contrary to some folks that claim that admitting mistakes can kill a business we have found over the years that being honest with our clients actually builds up their trust in us.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

SMB Kitchen ASP Project: Content Posted

I've posted a number of items to the ASP Project Knowledgebase.

Client Quote Template

This four sheet quote template allows us to create a quote for a server, desktop, storage, laptop, or other system based on expected pricing, current pricing, and then actual pricing to give us full control over our margins and expected sell prices.

The first sheet is where we work. The others are used to present the prospect or client a fully detailed list with price or just a fully detailed list for pasting into a proposal. The Build Sheet is part numbers, distribution source, pricing, and Order/Receive status.

All calculations are handled by this spreadsheet including leasing using the numbers our brokers supply us.

Client Audit Notes Template

In the Documentation link you will find our audit notes template:

In it you will find what we have built up for the last decade or more to manage our client's very diverse network infrastructures.

Client E-mail Templates

Finally there are three information templates that we use to e-mail our clients once in a while depending on the circumstances.

If you have not already done so please sign up. There is a lot more to come in the way of content produced by Amy, Susan, and myself for the SMB Kitchen ASP Project.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Hyper-V Standalone: Configure the Host as Authoritative Time Server and Guests to Use It

We have a number of single server virtualization solutions where the VMs are running at quite a high load point.

This tends to create timing issues with the VMs not being able to keep up with keeping time. In some cases we have SQL VMs that can skew time visibly.

So, we need a reliable time source. We are not able to set pool.ntp.org as the time source for the primary DC in this situation as it would not be keeping time very well either thus requiring a more frequent polling interval. Do that with ntp.org and the server will receive a Kiss-of-Death packet in short order.

So, we configure our Hyper-V host to be an authoritative time server polling ntp.org's servers every 30 minutes.

To do so we need to set things up on the host.

1. Allow UDP Port 123 Inbound on the firewall
2. Configure the host as a time server: KB816042 How to configure an authoritative time server in Windows Server
3. We run the Fix It Myself steps
4. Verify Type is set to NTP
• 
5. We specify the regional pool.ntp.org servers
6. Set AnnounceFlags to 5 in
• HKLM\SYSTEM\CurrentControlSet\Services\W32Time\Config\AnnounceFlags
7. Enable the NTPServer service by setting Enabled to 1
• 
8. Default Poll Interval
• 
9. Set the SpecialPollInterval to 15 minutes
• 
10. Set the phase correction settings to 30 minutes
• 
11. net stop w32time && net start w32time

We use this free tool to test our setup from a client system:

• NTP Server Tool by Galleon Systems

Drop in the Hyper-V host's name and we should see the following:

Once we have a successful time poll we run the following command set on the PDCe VM as per our previous blog post on time skew:

1. w32tm /config /syncfromflags:manual "/manualpeerlist:Hyper-V.DOMAIN.LOCAL,0x1" /reliable:yes
2. w32tm /config /update
3. net stop w32time && net start w32time
4. w32tm /resync /force
5. w32tm /query /source

The final result should be the Hyper-V server as source time.

We then create a batch file with the step 4 command in it and schedule that batch file to run as frequently as every minute on the most demanding VMs.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business