Monday, 6 July 2015

Happy Monday Because …

Kittens! :)


Momma’s Momma was a Blue Point Siamese with an unknown Papa. In this case the kitten’s Papa is our male black and white cat “Two Face”.


The black and grey tabby and the calico are females while the fully black and white one, like the Papa, and the cream coloured one are males.

There are many things in life that can bring a smile to our face. This is most certainly one of them.

Happy Monday everyone and thank you for reading! :D

Philip Elder
Microsoft Cluster MVP
Co-Author: SBS 2008 Blueprint Book

Tuesday, 30 June 2015

Hyper-V Virtualization 101: Hardware Considerations

When we deploy a Hyper-V virtualization solution we look at:

  1. VM IOPS Requirements
  2. VM vRAM Requirements
  3. VM vCPU Requirements

In that order.

Disk Subsystem

The disk subsystem tends to be the first bottleneck.

For a solution with dual E5-2600 series CPUs and 128GB of RAM requiring 16 VMs or thereabouts we'd be at 16 to 24 10K SAS drives at the minimum for this setup with a 1GB hardware RAID controller (non-volatile or battery backed cache).

RAID 6 is our go-to for array configuration.

Depending on workloads one can look at Intel's DC S3500 series SSDs or the higher endurance DC S3700 series models to get more IOPS out of the disk subsystem.


Keep in mind that the physical RAM is split between the two processors so one needs to be mindful of how the vRAM is divvied up between the VMs.

Too much vRAM on one or two VMs can cause the physical RAM to be juggled between the two physical CPUs (NUMA).

Note that each VM’s vRAM gets a file written to disk. So, if we are allocating 125GB of vRAM to the VMs there will be 125GB of files on disk.


And finally, each vCPU within a VM represents a thread to the physical CPU. For VMs with multiple vCPUs every thread (vCPU) for that VM needs to be processed by the CPU's pipeline in parallel. So, the more vCPUs we assign to a VM the more the CPU's logic needs to juggle the threads to have them processed.

The end result? More vCPUs is not always better.

I have an Experts Exchange article on Some Hyper-V Hardware and Software Best Practices that should be of some assistance too. In it I speak about the need to tweak the BIOS settings on the server, hardware configurations to eliminate single point of failures (SPFs), and more.


In the end, it is up to us to make sure we test out our configurations before we deploy them. Having a high five figure SAN installed to solve certain performance “issues” only to find out they exist _after_ the fact can be a very bad place to be in.

We test all aspects of a standalone and clustered system to discover its strengths and weaknesses. While this can be a very expensive policy, to date we’ve not had one performance issue with our deployments.

Our testing can also be quite beneficial to present an IOPS and throughput reports based on sixteen different allocation sizes (hardware and software) to our client _and_ the vendor complaining about our system. ;)

Philip Elder
Microsoft Cluster MVP
Co-Author: SBS 2008 Blueprint Book

Wednesday, 17 June 2015

What's up, what's been happing, and what will be happening.

Wow, it's been a while hasn't it? :)

We've been _very_ busy with our business as well as a Cloud services start-up and Third Tier is keeping me hopping too.

I have a regular monthly Webinar via Third Tier where we've been spending time on the Third Tier product called "Be the Cloud". It is a solution set developed to provide a highly available backend for client facing services based on our SBS (Small Business Solution).

We, that is my family, took a much needed break in May for a couple of weeks of downtime as we'd not had any pause for a good 18 months prior. We were ready for that.

So, why the blogging pause?

There are a number of reasons.

One is that I've been so busy researching and working on new things that there hasn't been a lot of time left over for writing them all out. Ongoing client needs are obviously a part of that too.

Another had to do with waiting until we were okay to publish information on the upcoming Windows Server release. We Cluster MVPs, and others, were privileged to be very deeply involved with the early stages of the new product. But, we were required to remain mum. So, instead of risking anything I decided to hold off on publishing anything Server vNext related.

Plus, we really didn't have a lot of new content to post since we've about covered the gamut in Windows Server 2012 RTM/R2 and Windows Desktop. Things have been stable on that front other than a few patch related bumps in the road. So, nothing new there meant nothing new to write about. ;)

And finally, the old grey matter just needed a break. After all, I've been writing on this blog since the beginning of 2007! :)

So, what does this mean going forward?

It means that we will begin publishing content on a regular basis again once we've began serious work with Windows Server vNext.

We have a whole host of lab hardware on the way that has a lot to do with what's happening in the new version of Windows Server that ties into our v2 for Be the Cloud and our own Cloud services backend.

We're also establishing some new key vendor relationships that will broaden our solution matrix with some really neat new features. As always, we build our solution sets and test them vigorously before considering a sale to a client.

And finally, we're reworking our PowerShell library into a nice and tidy OneNote notebook set to help us keep consistent across the board. This is quite time consuming as it becomes readily apparent that many steps are in the grey matter but not in Notepad or OneNote.

Things we're really excited about:
  • Storage Spaces Direct (S2D)
  • Storage Replication
  • Getting the Start Menu back for RDSH Deployments
    • Our first deployment on TP2 is going to happen soon so hopefully we do indeed have control over that feature again!
  • Deploying our first TP2 RDS Farm
  • Intel Server Systems are on the S2D approval list!
    • The Intel Server System R2224WTTYS is an excellent platform
  • Promise Storage J5000 series JBODs just got on the Storage Spaces approved list.
    • We've had a long history with Promise and are looking forward to re-establishing that relationship.
  • We've started working with Mellanox for assistance with RDMA Direct and RoCE.
  • 12Gb SAS in HBAs and JBODs rocks for storage
    • 2 Node SOFS Cluster with JBOD is 96Gbps of aggregate ultra-low latency SAS bandwidth per node!
  • NVMe based storage (PCIe direct)
The list could go on and on as they come to mind. :)

Thank you all for your patience with the lack of posting lately. And, thank you all for your feedback and support over the years. It has been a privilege to get to know some of you and work with some of you as well.

We are most certainly looking forward to the many things we have coming down the pipe. 2015 is shaping up to be our best year ever with 2016 looking to build on that!

Philip Elder
Microsoft Cluster MVP
Co-Author: SBS 2008 Blueprint Book

Tuesday, 14 April 2015

RDS Error: RemoteApp - The digital signature of this RDP File cannot be verified.

The following error was received on a client’s system this morning:

image RemoteApp

The digital signature of this RDP File cannot be verified. The remote connection cannot be started.

In this case the RDSH is using self-issued certificates for both Broker services. They had expired.

  1. Server Manager –> Remote Desktop Services –> Collections –> Tasks –> Edit Deployment Properties
  2. Click Certificates
  3. Click on the first Broker service and then the Create new certificate button
    • image
  4. Set a password and save to C:\Temp\2015-04-14-SelfIssuedSSL.pfx
  5. Click on the second Broker service and Select an Existing Certificate
  6. Choose the above newly created certificate

In the case where our client’s domains are .LOCAL or .CORP or some other non-Internet facing TLD we leave those two self-issued.

If we have an Internet facing domain then we use a third party trusted certificate as can be seen in the snip above.

Because we are deploying a lot of Remote Desktop Services solutions we always use an Internet TLD for the internal domain after making sure the client owns that domain and its registered for a decade.

Philip Elder
Microsoft Cluster MVP
Co-Author: SBS 2008 Blueprint Book

Tuesday, 31 March 2015

Our Default OU and Group Policy Structure

Over the years between our experiences with the Small Business Server Organizational Unit (OU) and Group Policy Object (GPO) structures plus wearing out a few copies of Jeremy Moskowitz’s books we’ve come to hone our Group Policy configurations down to an _almost_ science. ;)

Today, with our own Small Business Solution (SBS) in production we use the following OU and GPO structure as a starting point:

imageWe tailor all GPO settings around the intended recipient of those settings.

We use the WMI filters to delineate desktop OS versus Server and DC based operating systems. Note that the GPOs for those two sets of systems are not present in the above snip.

They would be:

  • Default Update Services Client Computers Policy
  • Default Update Services Server Computers Policy

Both enable Client-Side Targeting for WSUS managed updating.

NOTE: We _never_ edit the Default Domain Policy or the Default Domain Controllers Policy. EVER!

When we need something we create the GPO and link it to the OU containing the intended recipient objects or we scope the GPO via Security Group membership.

Some GP Pearls

All GPOs scoped to computers have the User Configuration settings disabled while GPOs scoped to users have the Computer Configuration settings disabled.


We don’t use Group Policy Loopback Processing. There’s just too much room for unintended consequences. Our structure above gives us the flexibility we need to hone our GPO settings down to a user or computer if need be.

Filters are OU membership, Security Group membership, or WMI Filtering.

GPO settings are like Cascading Style Sheets. Settings cascade from the domain level down through the OU structure to the recipient object. The closer the GPO to that object the more weight that GPO’s settings have.

We do not duplicate settings or put opposite settings in GPOs further down the OU structure. We create and link our GPOs accordingly.

We always enable the Group Policy Central Store (blog post) on our DCs. This makes things so much easier in the long run!

We always enable the AD Recycle Bin and if at all possible have the Forest and Domain at the latest available OS version level.

We test any intended changes we intend to make in a lab setting on a restored version of our client’s networks first! We test _any_ and _all_ intended settings changes/additions in a lab first!

Philip Elder
Microsoft Cluster MVP
Co-Author: SBS 2008 Blueprint Book

Thursday, 12 March 2015

Hyper-V: Broadcom Gigabit NICs and Virtual Machine Queues (VMQ)

Here is an explanation posted to the Expert’s Exchange forum that we believe needs a broader audience.


VMQ is a virtual networking structure allowing virtual Switch (vSwitch) networking to be processed by the various cores in a CPU. Without VMQ only one core would be processing those packets.

In a Gigabit setting the point is moot since the maximum of 100MB/Second or thereabouts per physical port is not going to tax any modern CPU core.

In a 10GbE setting where we have an abundance of throughput available to the virtual switch things change very quickly. We can then see a single core processing the entire virtual switch being a bottleneck.

In that setting, and beyond, VMQ starts tossing vSwitch processes out across the CPU's cores to distribute the load. Thus, we essentially eliminate the CPU core as a bottleneck source.

For whatever reason, Broadcom did not disable this setting in their 1Gb NIC drivers. As we understand things the specification for VMQ requires it to be disabled on 1GbE ports.

VMQ enabled on Broadcom NICs has caused no end of grief over the last number of years for countless Hyper-V admins. With Broadcom NICs one needs to disable Virtual Machine Queues (VMQ) on _all_ Broadcom Gigabit physical ports in a system to avoid what becomes a vSwitch traffic.


The above is a summary of conversations had with networking specialists. If there are any corrections or specifics that all y’all have about the VMQ structures please feel free to comment! :)

Philip Elder
Microsoft Cluster MVP
Co-Author: SBS 2008 Blueprint Book

Tuesday, 10 March 2015

Cluster: Asymmetric or iSCSI SAN Storage Configuration and Performance Considerations

We When we set up a new asymmetric cluster, or if one is using an iSCSI SAN for central storage, the following is a guideline to how we would configure our storage.

Our configuration would be as follows:

  • JBOD or SAN Storage
    • 6TB of available storage
  • (2) Hyper-V Nodes
    • 256GB ECC RAM Each
    • 120GB DC S3500 Series Intel SSD RAID 1 for OS
    • Dual 6Gbsp SAS HBAs (JBOD) or Dual Intel X540T2 10GbE (iSCSI)

There are three key storage components we need to configure.

  1. Cluster Witness (non-CSV)
    • 1.5GB Storage
  2. Common Files (CSV 1)
    • Hyper-V Settings Files
    • VM Memory Files
    • 650GB Storage
  3. Our VHDX CSVs (balance of 5,492.5GB split 50/50)
    • CSV 2 at 2,746.25GB
    • CSV 3 at 2,746.25GB

Given that our two nodes have a sum total 512GB of RAM available to the VMs, though we’d be provisioning a maximum of 254GB of vRAM at best, we would set up our Common Files CSV with 650GB of available storage.


We split up our storage for VHDX files into at least two Storage Spaces/LUNs. Each node would own one of the resulting CSVs.

We do this to split up the I/O between the two nodes. If we had just one 5.5TB CSV then all I/O for that CSV would be processed by just the owner node.

It becomes pretty obvious that having all I/O managed by just one of the nodes may present a bottleneck to overall storage performance. At the least, it leaves one node not carrying a share of the load.

Performance Considerations

Okay, we have our storage configured as above.

Now it’s time to set up our workloads.

  • VM 0: DC
  • VM 2: Exchange 2013
  • VM 3-6: RDHS Farm (Remote Desktop Services)
  • VM 7: SQL
  • VM 8: LoBs Line-of-Business apps), WSUS, File, and Print

Our highest IOPS load would be SQL followed by our two RDSH VMs and then our LoB VM. Exchange likes a lot more RAM than I/O.

When provisioning our VHDX files we would be careful to make sure our high IOPS VMs are distributed between the two CSVs as evenly as possible. This way we avoid sending most of our I/O through one node.

Why 650GB for Common Files?

Even though our VM memory files would take up about 254GB of that available storage one also needs space for the configuration files themselves, though they are quite small in size, and also additional space for those just-in-case moments.

One such moment is when an admin pulls the trigger on a snapshot/checkpoint. By default the differencing disk would be dropped into the Common Files storage location.

One would hope that monitoring software would throw up an alarm letting folks know that their cluster is going to go full-stop when that location runs out of space! But, sometimes that is _not_ the case so we need room to run our needed merge process to get things going again.

How do I know?

Okay, all of the above is just fine and dandy and begs the following question: How do I really know how the cluster will perform?

No one client’s environment is like another. So, we need to make sure we take performance baselines across their various workloads and make sure to talk to LoB vendors about their products and what they need to perform.

We have a standing policy to build out a proof-of-concept system prior to reselling that solution to our clients. As a result of both running baselines with various apps and building out our clusters ahead of time we now have a pretty good idea of what needs to be built into a cluster solution to meet our client’s needs.

That being said, we need to test our configurations thoroughly. Nothing could be worse than setting up a $95K cluster configuration that was promised to outperform the previous solution only to have that solution fall flat on its face. :(

Test. Test. Test. And, test again!

NOTE: We do _not_ deploy iSCSI solutions anywhere in our solution’s matrix. We are a direct attached storage (SAS based DAS) house. However, the configuration principles mentioned above apply for those deploying Hyper-V clusters on iSCSI based storage.

EDIT 2015-03-26: Okay, so fingers were engaged prior to brain on that first word! ;)

Philip Elder
Microsoft Cluster MVP
Co-Author: SBS 2008 Blueprint Book