Hyper-V 2 Node JBOD DAS Cluster IOPS Run

This is just the beginning of our testing with Iometer:


The setup is the following:

• Hyper-V Nodes
• Intel Server Systems R2208GZ4GC, dual Intel Xeon E5-2680v2, 32GB ECC, dual Intel RAID Controller RS25GB008 SAS HBAs
• Intel Storage Systems JBOD2224S2DP JBOD
• (17) 100GB HGST SSD400 SAS SSDs
• Windows Server 2012 R2 U1
• Storage Spaces with Simple across all SSDs
• Small Witness Disk
• (2) CSVs at ~760GB each

We had no idea as far as what to expect with this setup since each CSV is being managed by one node.
As we go along learning how Iometer works and how the disks react to various workloads we will publish additional results.
Then we will go on to run the same tests again only with the above setup configured in a Scale-Out File Server Cluster with a 10GbE backend facilitated by a pair of Intel Server Adapter X540T2 NICs in each node, NETGEAR 10GbE XS712T switches, and a pair of Hyper-V Nodes.
Hopefully with the Active/Active setup we get with SOFS and SMB Multi-Channel our performance will come out a bit better!
This test configuration was Option 1 in the Three Intel Server Systems based Hyper-V and Scale-Out File Server Clusters Post.
Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Three Intel Server Systems based Hyper-V and Scale-Out File Server Clusters

Here are three base Intel Server Systems configurations we are working on for our Intel Modular Server replacement in a Data Centre or client setting.

Unfortunately, the Intel JBOD does not self-power at this time. So, for SMB/SME solutions we will be supplying a DataON DNS-1640 2U JBOD as it will automatically power-up after a full power outage.

All solution sets are based on Windows Server 2012 R2 as a starting point for Hyper-V, Storage Spaces, and SOFS.

• Option 1: Asymmetric Hyper-V Cluster via Storage Spaces CSV
  • Intel Server System R2208GZ4GC, Dual E5-2640, 128GB ECC or 256GB ECC, 120GB SSD RAID 1, dual SAS HBAs, add-in Intel i350T4 PCIe
  • Intel JBOD2224S2DP
• Option 2: Hyper-V Cluster via SMBv3 Scale-Out File Server cluster and Storage Spaces
  • Intel Server System R1208JP4OC, E5-2640, 128GB ECC, 120GB SSD RAID 1, dual SAS HBAs, Intel X540T2 I/O Module, Intel X540T2 PCIe
  • Intel JBOD2224S2DP
  • Intel Server System R1208JP4OC, E5-2640, 128GB ECC, 120GB SSD RAID 1, Intel i350T4 PCIe, Intel X540T2 I/O Module, Intel X540T2 PCIe
  • NETGEAR XS712T 10GbE Switches
• Option 3: Hyper-V Cluster via SMBv3 Scale-Out File Server cluster and Storage Spaces with enclosure resilience
  • (3) Intel Server System R2208GZ4GC, Dual E5-2640, 128GB ECC, 120GB SSD RAID 1, SIX SAS HBAs, Intel X540T2 I/O Module, Intel X540T2 PCIe
  • (3) Intel JBOD2224S2DP
  • (2) Intel Server System R2208GZ4GC, Dual E5-2640, 128GB ECC, 120GB SSD RAID 1, Intel i350T4 PCIe, Intel X540T2 I/O Module, Intel X540T2 PCIe
  • (2) NETGEAR 24-Port 10GbE Switches
• Storage Networking Option
  • Option 2 and Option 3 can be facilitated by InfiniBand NICs and Switches
    • Enables RDMA and 56Gbps per connection
    • Microsoft's 1.4M IOPS demo based on InfiniBand backend
    • Intel Server Systems have an InfiniBand I/O Module with the second being a Mellanox PCIe

The first setup is relatively simple while the second two require some structuring around how the networking is configured to allow for SMB Multi-Channel on the storage network side.

At this point the above setups utilizing Intel Server Systems provide us with an amazing value for our IT budgets.

5 year warranties and next business day on-site support options can be had too.

We purchase our Intel Channel product primarily through ASI Canada. Ingram Micro, Synnex Canada, and Tech Data Canada are also Intel Authorized Distributors.

As an FYI we continue to build our own server systems because the experience proves to be invaluable when it comes to troubleshooting problems especially when software vendors are pointing fingers.

Building our own systems also gives us a very strong foundation for creating server configurations that will work with a client workload set.

And finally, it allows us to be very particular with Tier 1 vendors when it comes to creating a server configuration using their hardware.

EDIT: Note that we _always_ install a physical DC on our cluster networks. For option 1 it would probably be an HP MicroServer while the others would be a 1U single socket with some storage for ISOs.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Microsoft Fails Long Term Open Licensing Customers with Office for iPad O365 Requirement

We received an inquiry from one of our clients about obtaining Office for their iPad. Obviously the paid version is required in order to edit Office documents.

We fired an e-mail off to our licensing contact at Synnex Canada and received the following:

The clear-cut answer is that Microsoft has seemingly set up a requirement for Open License customers to double pay.

How is that?

Well, in the case of our clients they have been on Open Value Agreements for quite a few cycles in some cases. When OVL hits the third renewal (years 7-9) the SA Only costs bring the overall cost of that license and SA close to the OEM costs.

For clients that have regular refresh rates of about 36 months for their equipment this makes sense on so many levels.

So, our clients have already purchased their Office Professional Plus or Office Standard licenses and are on SA Only for them.

To get Office for iPad for their devices there is no SKU to add to their existing agreement. They would have to purchase an Office 365 subscription either personally or for the organization in an O365 MidSize Business plan which on Open Value is $200/Year/User.

Ouch.

It looks like we've just been double billed.

IMNSHO this decision of Microsoft's to force the Office 365 subscription for existing Open License clients is very poorly thought through.

This situation really begs the question as to whether Open License is a valued proposition going forward as Microsoft further segments products into the subscription side to force businesses that have already paid for their licenses to pay yet again ... and again ... and again.

Hopefully Microsoft will come up with something to remove the double billing.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Some Mandatory Reading for Sys Admins

There have been some critical human attributable error related news stories that we have seen lately.

Many of us, if not all of us, have had situations that have had negative impacts on our client's networks or our own networks due to something we did or did not do.

The following four links come via the Patch Management list serve (Ben S.):

• The Unofficial Unix Administration Horror Story Summary Part 1
• The Unofficial Unix Administration Horror Story Summary Part 2
• The Unofficial Unix Administration Horror Story Summary Part 3
• The Unofficial Unix Administration Horror Story Summary Part 4

As with anything out there on the Interwebs take a PDF of the four parts to keep them safe. :)

There are many pearls of wisdom contained in the above pages as well as some humorous stories that we _all_ can relate to.

• Lesson #1: Learn from our mistakes
• Lesson #2: Keep a detailed history of our mistakes
• Lesson #3: Learn from our mistakes

When we make a mistake that directly impacts our clients we make sure to apologize and make some sort of restitution offer to them.

Contrary to some folks that claim that admitting mistakes can kill a business we have found over the years that being honest with our clients actually builds up their trust in us.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

SMB Kitchen ASP Project: Content Posted

I've posted a number of items to the ASP Project Knowledgebase.

Client Quote Template

This four sheet quote template allows us to create a quote for a server, desktop, storage, laptop, or other system based on expected pricing, current pricing, and then actual pricing to give us full control over our margins and expected sell prices.

The first sheet is where we work. The others are used to present the prospect or client a fully detailed list with price or just a fully detailed list for pasting into a proposal. The Build Sheet is part numbers, distribution source, pricing, and Order/Receive status.

All calculations are handled by this spreadsheet including leasing using the numbers our brokers supply us.

Client Audit Notes Template

In the Documentation link you will find our audit notes template:

In it you will find what we have built up for the last decade or more to manage our client's very diverse network infrastructures.

Client E-mail Templates

Finally there are three information templates that we use to e-mail our clients once in a while depending on the circumstances.

If you have not already done so please sign up. There is a lot more to come in the way of content produced by Amy, Susan, and myself for the SMB Kitchen ASP Project.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Hyper-V Standalone: Configure the Host as Authoritative Time Server and Guests to Use It

We have a number of single server virtualization solutions where the VMs are running at quite a high load point.

This tends to create timing issues with the VMs not being able to keep up with keeping time. In some cases we have SQL VMs that can skew time visibly.

So, we need a reliable time source. We are not able to set pool.ntp.org as the time source for the primary DC in this situation as it would not be keeping time very well either thus requiring a more frequent polling interval. Do that with ntp.org and the server will receive a Kiss-of-Death packet in short order.

So, we configure our Hyper-V host to be an authoritative time server polling ntp.org's servers every 30 minutes.

To do so we need to set things up on the host.

1. Allow UDP Port 123 Inbound on the firewall
2. Configure the host as a time server: KB816042 How to configure an authoritative time server in Windows Server
3. We run the Fix It Myself steps
4. Verify Type is set to NTP
• 
5. We specify the regional pool.ntp.org servers
6. Set AnnounceFlags to 5 in
• HKLM\SYSTEM\CurrentControlSet\Services\W32Time\Config\AnnounceFlags
7. Enable the NTPServer service by setting Enabled to 1
• 
8. Default Poll Interval
• 
9. Set the SpecialPollInterval to 15 minutes
• 
10. Set the phase correction settings to 30 minutes
• 
11. net stop w32time && net start w32time

We use this free tool to test our setup from a client system:

• NTP Server Tool by Galleon Systems

Drop in the Hyper-V host's name and we should see the following:

Once we have a successful time poll we run the following command set on the PDCe VM as per our previous blog post on time skew:

1. w32tm /config /syncfromflags:manual "/manualpeerlist:Hyper-V.DOMAIN.LOCAL,0x1" /reliable:yes
2. w32tm /config /update
3. net stop w32time && net start w32time
4. w32tm /resync /force
5. w32tm /query /source

The final result should be the Hyper-V server as source time.

We then create a batch file with the step 4 command in it and schedule that batch file to run as frequently as every minute on the most demanding VMs.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

QuickBooks: Save Open Windows For Next Session

Search Foo was turning up literally nothing on how to set a window, in this case Create Invoices, in QuickBooks to open by default the next time the company file was opened.

Why?

Because we invoice on the fly. When QuickBooks gets opened via RemoteApp it could be a bad thing to have things like vendor windows open, product properties with pricing opened, and so much more while the client is standing right there.

An ask via Intuit's Twitter account turned up the following resolution:

• Edit --> Preferences --> Desktop View --> Desktop: Save when closing company

When the above window was opened the setting was "Don't save the desktop" which was a bit frustrating.

Prior to our upgrade to version 2013 the only window that would open was Create Invoices. After our upgrade that somehow was lost with any searching, when the thought arouse to look, turning up nothing due to improper search terms.

We can now safely open QuickBooks anywhere we are working to fire off an invoice via e-mail right there and then.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

A Microsoft Cluster Troubleshooting Guide

Here are some of the tools we can use when troubleshooting a cluster, Scale-Out File Server, Hyper-V, and other cluster issues:

Failover Cluster Manager 

The FCM gives us the ability to dig into the various Windows Logs and delimit them by time, node, and log type.

• FCM --> Cluster Name --> Cluster Events --> Query

We set up a few different queries out of the box. One with everything Cluster, Failover Clustering, and Hyper-V related. We then create a subset of queries. The various queries get saved to a local folder on the management DC/RSAT system.

Get-ClusterLog

The Get-ClusterLog PowerShell commandlet allows us to pull the full log set from one or all nodes. Note that the default output folder is \\NODE\C$\Windows\Cluster\Cluster.LOG (C:\Windows\Cluster\Cluster.LOG) unless specified in the command.

This log can be very busy and a bit of a challenge to work through. If one has a good idea of what to look for then the log can be quite informative.

• Get-ClusterLog -Destination .
  • Places the log in the local directory (we create C:\Temp on all nodes for this kind of thing)
• (get-cluster).ClusterLogLevel=5
  • There are five levels with 5 being the most verbose. Default level is 3 and best left there unless absolutely needed. Level 5 file can be large.

EDIT: The Default cluster log location is C:\Windows\Cluster\Reports\Cluster.log

Microsoft Message Analyzer

This is an in-depth tool. There is no way around it. Thus, a learning curve is required.

• TechNet Blogs: MessageAnalyzer
• Download the Microsoft Message Analyzer

However, there is an amazing amount of information that we can then have at our fingertips and not only that colour coded!

One can use a series of filters under the log file settings to delimit by time period among others.

We can set up our columns:

Once we have our Cluster column, for example when looking for a problematic cluster component, we can set up a filter:

And that is just the tip of the iceberg. One will need to spend some time with this tool to really get into its abilities such as colour coding source node, information levels, and so much more!

Please check the Message Analyzer Blog for more information.

Note that an absence of System Centre and its components is deliberate. We find, at least at this time, that Failover Cluster Manager provides a far superior cluster management experience.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

A very cool blog milestone

We are not much on beating our own drums and blowing our own horns but this is really cool:

We are about to hit 3,000,000 page views.

Why that number over any other? Because it is the first time in a long while we've checked the stats. ;)

And, it just happens to tickle the kewl factor neat feeling area.

Thank you to all of our readers that have stuck it out over the years. We know there are some of you on our e-mail list that have been with us from pretty much the beginning of this blogging journey.

Thanks to all of you that have taken the time to comment, ping us with a question, and even reach out for assistance on various projects. It is very muchly appreciated.

Have a great day!

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

SMB Kitchen ASP Project Chat at 1700 MST - Join Us!

We are holding our last freebie author's chat this afternoon/early evening depending on where you are in North America or later for those of you across the Pond. :)

• SMB Kitchen ASP Project Webinar Lync Link

For Windows 8 and up please install Lync from the Store. Once installed and logged into the meeting hit the Charms --> Settings to verify both microphone and speaker settings.

We will cover some of the content we have published to date, some of the ongoing concerns for IT Professionals today, and what we are planning on publishing in the coming weeks.

If we have time we can have a Virtualization and Hyper-V Q&A session too!

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Windows 8.1 and Windows Server 2012 R2 Update 1 Caveats

Whenever we hear about updates for any device firmware, application software, or operating system software it is _always_ a good idea to read the Release Notes (we probably all know the acronym that comes to mind here ;) ), README.TXT, _and_ do a search for the update to see if anyone is complaining about it. Unfortunately, this major update for Windows 8.1 and Windows Server 2012 R2 has its issues some of which are full-stop problems. The first place to start for this update is here:

• Gladiator@MSFT BLog: Information Regarding the Latest Update for Windows 8.1

There is a lot of information there.

1. Update 1 is the new baseline for all updates going forward.
• Meaning, no more updates to that OS if the bits are earlier than 8.1 U1 or 2012 R2 U1.
2. Update 1 breaks SSL communications between endpoints and WSUS

This last one is a deal breaker for many enterprises, medium enterprises, and especially in our own SMB/SME environments where WSUS is virtually everywhere for patch management.
We just stood up a new cluster on 2012 R2. After our Cluster-Aware Update run:

Our cluster nodes now have the update. Since this cluster setup is Greenfield with WSUS ultimately ending up _on_ the cluster the nodes were updated via Microsoft Update.

• WSUS Product Team Blog: Windows 8.1 Update (KB 2919355) prevents interaction with WSUS 3.2 over SSL

The workaround for this situation is to enable TLS 1.2 as instructed in the above blog post. Since we are deploying Windows Server 2012 R2 into client sites we will have no choice but to make this change.
Then, when Microsoft releases an update to the update to hopefully fix the problem we will need to test that update extensively _especially_ in a cluster setting!
Yo Microsoft! There is a huge pool of folks willing to test and break this stuff for you! Please get us involved in the early bits for operating systems, applications, and updates again. This ongoing situation of releasing patches and updates to the public without testing them on disparate systems is a _bad_ thing. :(
EDIT: Updated the Gladiator link since between Live Writer and Blogger it got mangled.
Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

A Small Chunk of Bandwidth for Windows Server 2012 R2 and Windows 8.1 Pro/Enterprise with Update 1

Okay, so here we go updating our ISO library:

And:

And finally:

Note that all Windows 8.1/Server 2012 R2 operating systems in the field _must_ be updated to Update 1 or they will not receive any further starting with Patch Tuesday this coming May!

That does not give us a lot of time to test the patch setup. But test we must! :(

By the way, is there any need to download the 32-bit version of a desktop OS anymore? We've not had to use one in ages!

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Intel NUC: Wall Mount Monitor And NUC

We initially had a bit of a puzzle as to how to mount both a monitor and the Intel NUC on the same wall mount monitor arm.

In the end the solution proved to be quite simple.

As seen above we took the stock Intel NUC mounting plate and attached it on top of the Ergotron's mounting plate using the Erogotron supplied mounting screws. The Intel supplied screws were a bit too short to pass through both mounting plates.

The finished product looks like this:

It will be used as a training station for our client. The keyboard and mouse are wireless so they can sit or stand anywhere in the area to work on the machine.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Cluster-Aware Update Runs: How Long?

When one needs to provision a new Windows Server 2012 RTM/R2 cluster one of the time factors to keep in mind is the Cluster-Aware Update runs that will happen once the cluster has been brought up.

The above is a new Windows Server 2012 R2 four node cluster that we finished configuring last night.

The first CAU run started around 2300Hrs. As can be seen each node took about an hour to run through the process.

Even though we are not too far into the product release cycle for R2 one needs to keep in mind the update maintenance process if one is not using an up-to-date image for deployment. This would be especially true if deploying 2012 RTM or 2008 R2 clusters.

Cluster node configuration:

• Intel Server System SR1695GPRX2AC with RMM
• Intel Xeon Processor X3470, 32GB ECC, 120GB Intel SSD RAID 1, Dual Intel SAS HBAs

Some important Cluster update tracking links:

• TechNet: List of Failover Cluster Hotfixes for Windows Server 2012
• Microsoft Support: KB2784261 Recommended hotfixes and updates for Windows Server 2012-based failover clusters
• Microsoft Support: KB2920151 Recommended hotfixes and updates for Windows Server 2012 R2-based failover clusters

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

A couple of Published Articles and an ASP Document Published

While we have been quite busy I have managed to put some aside to do some writing.

•   Windows Server News: Restoring a Windows Phone
• I walk through how to get a Windows Phone right back to where it was before the previous one pancaked on the floor!
• The Microsoft ID associated with the phone is priceless.
• Experts-Exchange: Some Hyper-V Hardware and Software Best Practices
• An aggregate of may posts on this blog, Yahoo Groups, and other locations.

I have also posted the first of many documents to the SMB Kitchen ASP Project that we use for our ongoing day-to-day business here:

This document is the foundation on which we manage all of our clients. We track _everything_ in this document. It gets posted to a SharePoint library with versioning enabled for major and minor versions as well as mandatory check-out and check-in.

Soon to come will be some of the quote templates we use to cost out or IT Solution Sets for our clients.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business