Thursday, 26 May 2016

Hyper-V Virtualization 101: Hardware and Performance

This is a post made to the SBS2K Yahoo List.


VMQ on Broadcom Gigabit NICs needs to be disabled at the NIC Port driver level. Not in the OS. Broadcom has not respected the spec for Gigabit NICs at all. I’m not so sure they have started to do so yet either. :S

In the BIOS:

  • ALL C-States: DISABLED
  • Power Profile: MAX
  • Intel Virtualization Features: ENABLED
  • Intel Virtualization for I/O: ENABLED

For the RAID setup we’d max out the available drive bays on the server. Go smaller volume and more spindles to achieve the required volume. This gains us more IOPS which are critical in smaller virtualization settings.

Go GHz over Cores. In our experience we are running mostly 2vCPU and 3vCPU VMs so ramming through the CPU pipeline quicker gets things done faster than having more threads in parallel at slower speeds.

Single RAM sticks per channel preferred with all being identical. Cost of 32GB DIMMs has come down. Check them out for your application. Intel’s CPUs are set up in three tiers. Purchase the RAM speed that matches the CPU tier. Don’t purchase faster RAM as that’s more expensive and thus money wasted.

Be aware of NUMA boundaries for the VMs. That means that each CPU may have one or more memory controller each. Each controller manages a chunk of RAM attached to that CPU. When a VM is set up with more vRAM than what is available on one memory controller that memory gets split up. That costs in performance.

Bottlenecks not necessarily in order:

  • Disk subsystem is vastly underperforming (in-guest latency and in-guest/host Disk Queue Length are key measures)
    • Latency: Triple digits = BAD
    • Disk Queue Length: > # Disks / 2 in RAID 6 = BAD (8 disks in RAID 6 then DQL of 4-5 is okay)
  • vCPUs assigned is greater than the number of physical cores – 1 on one CPU (CPU pipeline has to juggle those vCPU threads in parallel)
  • vRAM assigned spans NUMA nodes or takes up too much volume on one NUMA node
  • Broadcom Gigabit VMQ at the port level

The key in all of this though and it’s absolutely CRITICAL is this: Know your workloads!

All of the hardware and software performance knowledge in the world won’t help if we don’t know what our workloads are going to be doing.

An unhappy situation is spec’ing out a six to seven figure hyper-converged solution and having the client come back and say, “Take it away I’m fed up with the poor performance”. In this case the vendor over-promised and under-delivered.

Some further reading:

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Thursday, 12 May 2016

RDMA via RoCE 101 for Storage Spaces Direct (S2D)

We’ve decided to run with RoCE (RDMA over Converged Ethernet) for our Storage Spaces Direct (S2D) proof of concept (PoC).


  • (4) Intel Server Systems R2224WTTYS
    • Dual Intel Xeon Processors, 256GB ECC, Dual Mellanox ConnectX-3, and x540-T2
    • Storage is a mix of 10K SAS and Intel SATA SSDs to start
  • (2) Mellanox MSX1012 56Gbps Switches
  • (2) NETGEAR XS712T 10GbE Switches
  • (2) Cisco SG500x-48 Gigabit Switches
  • APC 1200mm 42U Enclosure
  • APC 6KV 220v UPS with extended runtime batteries

The following is a list of resources we’ve gathered together as of this writing:

This is, by far, not the most comprehensive of lists. The best place to start in our opinion is with Didier’s video and the PDF of the slides in that video. Then move on to the Mellanox resources.

We’ll update this blog post as we come across more materials and eventually get a process guide in place.

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Tuesday, 26 April 2016

Remote Desktop Services 301: Some Advanced RDS Setup Guidelines

Here are some of the key tenants we’ve picked up deploying standalone and farm based Remote Desktop Services (RDS) for our Small Business Solution (SBS) on-premises and in our cloud.

Virtual Desktop Infrastructure, or VDI for short, covers both Remote Desktop Services standalone, farm, and desktop operating system deployments.

This list, while not totally comprehensive, covers a lot of ground.

  • Hardware, Storage, and Networking
    • GHz and Cores are king
      • Balance core count and GHz versus cost
      • NOTE: Server 2016 licensing will be, and is as of this writing, based on core count!
        • Today is 2 sockets tomorrow is a server total of 16 cores
        • Additional Cores purchased in pairs
        • Example 1: Dual Socket 8 Core pair = 16 Cores total so OK in current and 2016 licensing
        • Example 2: Dual Socket 12 Core pair = 24 Cores total so base license of 16 Cores plus a purchase of 4 licenses (2 cores per license) would be required
        • NOTE: Examples may not line up with actual license terms! Please verify when 2016 goes live.
    • RAM is cheap so load up now versus later
    • Balanced RAM setup is better than Unbalanced
      • Balanced: 4x 16GB in primary memory channel slot
        • Best performance
      • Unbalanced: 4x 16GB in primary and 4x 8GB in secondary
        • Performance hit
    • ~500MB of RAM per user per session to start
    • 25-50 IOPS per user depending on workloads/workflows RDS/VDI
      • Average 2.5” 10K SAS is ~250 to 400 IOPS depending on stack format (stripe/block sizes)
    • Latency kills
      • Direct Attached SAS or Hyper-Converged is best
      • Lots of small reads/writes
    • Average 16bpp RDS session single monitor 23” or 24” wide: ~95KB/Second
      • Average dual monitor ~150KB/Second
      • Bandwidth use is reduced substantially with a newer OS serving and connecting remotely (RDP version)
  • LoBs (Line of Business applications)
    • QuickBooks, Chrome, Firefox, and Sage are huge performance hogs in RDSH
      • Be mindful of LoB requirements and provision wisely
    • Keep the LoB database/server backend somewhere else
    • In larger settings dedicate an RDSH and RemoteApp to the resource hog LoB(s)
  • User Profile Disks
    • Office 2013 and Exchange 2013 are a wash in this setting
      • Search is difficult if not broken
    • Search Index database will bloat and fill the system disk! (Blog post with “fix”)
    • Office 2016, though still a bit buggy as of this writing, and Exchange 2016 address UPDs and search
    • Be mindful of network fabrics between UPDs and RDSH(s)
    • Set the UPD initial size a lot larger than needed as it can’t be changed later without a series of manual steps
      • UPDs are dynamic
      • Keep storage limits in mind because of this
  • Printing
    • Printers with PCL 5/6 engines built-in are preferred
      • Host-based printers are a no-go for us
    • HP Professional series LaserJet printers are our go-to
    • HP MFPs are preferred over Copiers
      • Copier engines tend to be hacked into Windows while the HP MFP is built as a printer out of the box

Previous post on the matter: Some Remote Desktop Session Host Guidelines.

Here’s a snippet of Intel’s current Intel Xeon Processor E5-2600v4 line sorted by base frequency in GHz:


Depending on the deployment type we’re deploying either E5-2643v4 or E5-2667v4 processors for higher density setups at this time. We are keeping at or under eight cores per socket unless we absolutely require more due to the upcoming sockets to cores changes in Windows Server licensing.

If you’d like a copy of that spreadsheet ping us or visit the Intel Ark site.

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book
Our Cloud

Friday, 26 February 2016

Security: A Sample E-mail How-To Guide For End Users

With the plethora of e-mail born Office documents with active macros in them to pull down malware/ransomware we sent out the following e-mail to all of our clients for distribution internally.


Good day everyone,

It’s gotten to the point now where we are considering a universal restriction on incoming Office Documents. By that we mean plucking them right out of the e-mail via ExchangeDefender by default.

We have somehow travelled back to the 1990s where the bad guys are setting up Office documents with a Macro, an automatic script that runs when the document gets opened, that goes on to pull down their nefarious malware or ransomware.

Here are some steps to help protect us:

  1. Microsoft Office has a Save As PDF feature built-in. Please have all outside folks send a PDF instead of an Office document
    1. This is especially critical for Resumes. All job postings _must_ request PDF and note that Office documents would be deleted on the spot!
    2. If collaboration is required for Office documents use ShareFile
    3. Preferred over Dropbox since security is questionable with the Dropbox service
  2. Most Office documents that have Macros built-in have an “m” in the extension
    1. clip_image001
    2. Save the Office document to Downloads and verify!
    3. If extensions are not shown then right click the file and left click on Properties
    4. clip_image002
  3. Users _should_ be prompted:
    1. clip_image003
  4. Obviously, the answer should be to NOT click that button
  5. If they do, there is one last cause for pause
    1. clip_image004
  6. This is what happens if I try and click on something that is Macro driven _before_ clicking Enable Content
    1. clip_image005

Along with the need to be mindful of any Microsoft Office attachments in our e-mail we should also remember the following:

  1. Never click on a link in an e-mail without at the least verifying its destination:
    1. clip_image006
    2. Hover the mouse cursor over the link to verify
    3. As a rule: Never, ever, click on a link in an e-mail. Go to the web site after opening a new browser window (IE, Firefox, Chrome, Safari)
  2. It may _look_ like it came from someone you know but never trust that. Call and ask!
    1. There are a few exceptions to this rule thus make sure to hover your mouse over the link before clicking!
    2. Advanced users can check the headers
      1. image
      2. image
      3. Follow the flow from origin server to destination server
  3. Don’t save important site’s information in the browser
    1. Banking IDs and passwords
    2. CRA and critical site’s IDs and passwords
    3. Do not disable the secondary question for any computer
      1. Banking sites use this feature to help protect the account as one example
      2. Answer the question, it only takes a couple seconds and could save your savings!
  4. Never call the 800 number that comes up in a Search for Support!
    1. Go to the manufacturer’s web site and click on the Support link to find the correct phone number
  5. Never believe a pop-up message that says your computer is infected with something!
    1. And never, EVER, call the 800 number on that pop-up!
    2. Don’t click anywhere, close and save your work if needed then, reboot!
    3. Do NOT click anywhere in the pop-up window. Looks are deceiving as all areas of that pop-up = YES/ACCEPT/CONTINUE
  6. Never volunteer a credit card number or banking information to anyone
    1. Social Security/Social Insurance Numbers too!
    2. Folks can garner a lot about us online. Never volunteer any information when asked via any incoming call/e-mail/forum
    3. Always call them back!
  7. Caller says they are from the bank, CRA, or other seemingly critical business?
    1. Ask for their badge number, an 800 number to call, and an extension
    2. Open a browser and verify the 800 number belongs to the bank/CRA/CritBiz.
    3. Then call them back after hanging up if the number proves true!

While the above list is far from complete, by following these guidelines we can greatly reduce the chances of a malware or ransomware infection.

And, as always, e-mail or call if you are not sure about something!


Please feel free to use this as a template for training users!

Have a great weekend everyone. It’s +10C here and much like an awesome Spring day!

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book

Thursday, 25 February 2016

Some Thoughts On Security Layering for SMB and SME

We are by no means masters of security for our SMB and SME clients. Since we have to wear many hats we sometimes need to bring folks in that can help us to fine tune the security layers in our client’s networks.

Here are some of the Pearls (blog category) that we have garnered over the years. This was posted originally to the SBS2K Yahoo List and has been modified for this post.


Layering is important.

Some examples follow.

Windows Firewall

  • Windows Firewall is managed by Group Policy
    • All Profiles: ON
    • All Profiles: Block ON
    • All Profiles: Logging ON
    • All Profiles: Pop-Up for new services ON
    • DOMAIN Profile: Custom Inbound rule sets for required services beyond the default.
    • Private and Public Profile: INBOUND BLOCK ALL
      • If data sharing is required then a small and inexpensive NAS should be set up

Mail Sanitation and Continuity

ExchangeDefender (xD), for us, is one of the principle ways we keep bad stuff outside of the network.
Why allow it to hit the edge in the first place? Plus, it eliminates SMTP Auth attacks as the WAN IP is not published via MX among other attacks. Interested? Ping us and we’ll set you up.

Edge (Router)

A solid edge device, we use SonicWALL, with a BLOCK ALWAYS rule for ALL outbound traffic is a key element. Rule sets for outbound traffic are very specific and tailored to a client’s needs.

  • Examples:
    • DNS queries to non on-premises DNS servers are blocked. All DNS queries must go through the on-premises DCs.
    • On-Premises edge only or DCs can have the DNS Forwarders set to DNS filtering services.
    • SMTP traffic outbound only from the on-premises Exchange server. Or, local copiers/MFPs to ISP SMTP server IP only
    • Inbound is HTTPS via ANY
    • SMTP via xD subnets only.
    • RDP on ANY port should NEVER be published to the Internet.
      • RD Gateway with Network Level Authentication is a must today.
      • Any exceptions require a static IP on the source end to allow inbound rule filtering based on IP.
      • Look up TSGrinder if not sure why…

Ransomware Protection

Third Tier’s Ransomware Protection Kit is another layer of protection. Everything is contained in this kit to deploy a very tight layer of protection against today’s Ransomware.

Microsoft Office Group Policy Security

Office Group Policy structures with Macros disabled by default, non-local sources blocked, and other security settings for Office files provide another layer.

  • This one gives users grief because they need a few extra steps to get to the documents.
  • We’ve started requesting that clients have a PDF only policy on their Jobs listing pages and such.

A User Focused Effort

IMNSHO, A/V at the endpoint has become virtually useless today. Things seem to be a lot more targeted on the virus side with ransomware taking over as the big cash cow. We still install A/V on all endpoints. :)

What we are saying, is that the principle portion of the risk of infection comes via the user.

A well trained user means the risk of infection drops substantially.

A user’s browsing habits and link clicking are the two key areas of training we focus on. Sites visited are another.

We suggest to clients that a company policy of allowing browsing for business related tasks only while connected to the company’s network resources. This policy can further reduce exposure.

Part of our training regimen is a somewhat regular e-mail from an outside account to users to test them and challenge them is a good idea every once in a while.

  • Link hovering to discover the true destination
  • Attached Word doc with *BUZZ WRONG* when opened
  • Just because it SAYS it’s “FROM” someone we know doesn’t mean it is!

Backup Protection

Oh, and protect the backup loop (blog post on closing the backup loop)!

BTW, we just heard about another NAS based backup that was ransomware encrypted as a result of the destination folder being open to users.

Anyone, and I mean ANYONE, that has a backup structure, whether NAS or HDD based, that allows users and admins access outside of the backup software username and password setup needs to close that loop NOW. Not on the To Do List, not for tomorrow, not next week, but NOW.

Just in case: Close that Backup Loop Now.

Hyper-V Standalone Setups

One more point of order: In standalone Hyper-V settings leave the host in workgroup mode.

No one on the network should have the admin username and password to that host. No. One. It should be documented somewhere but not public knowledge.

Please feel free to add the layers you use to this post via comments.

Thanks for reading!

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book

Tuesday, 16 February 2016

Cluster 101: Some Hyper-V and SOFS Cluster Basics

Our focus here at MPECS Inc. has grown into providing cluster-based solutions to clients near and far over the last eight years or so as well as cluster infrastructure solutions for small to medium I.T. shops.

There were so many misconceptions when we started the process to build out our first Hyper-V cluster in 2008.

The call in to us was for a large food manufacturing company that had a very specific requirement for their SQL, ColdFusion, and mail workloads to be available. The platform of choice was the Intel Modular Server with an attached Promise VTrak E310sD for extra storage.

So, off we went.

We procured all of the hardware through the Intel and Promise demo program. There was _no_ way we were going to purchase close to $100K of hardware on our own!

Back then, there was a dearth of documentation … though that hasn’t changed all that much! ;)

It took six months of trial and error plus working with the Intel, Promise, LSI, and finally contacts at Microsoft to figure out the right recipe for standing up a Hyper-V cluster.

Once we had everything down we deployed the Intel Modular Server with three nodes and the Promise VTrak E310sD for extra storage.

Node Failure

One of the first discoveries: A cluster setup does not mean the workload stays up if the node it’s on spontaneously combusts!

What does that mean? It means that when a node suddenly goes offline because of a hardware failure the guest virtual machines get moved over to an available node in a powered off state.

To the guest OS it is as if someone hit the reset button on the front of a physical server. And, as anyone that has experienced a failed node knows the first prompt when logging in to the VM is the “What caused the spontaneous restart” prompt.

Shared Storage

Every node in a Hyper-V cluster needs identical access to the storage the VHD(x) files are going to reside on.

In the early days, there really was not a lot of information indicating exactly what this meant. Especially since we decided right from day one to avoid any possible solution set based on iSCSI. Direct Attached Storage (DAS) via SAS was the way we were going to run with. The bandwidth was vastly superior with virtually no latency. No other shared storage in a cluster setting could match the numbers. And, to this day the other options still can’t match DAS based SAS solutions.

It took some time to figure out, but in the end we needed a Shared Storage License (SharedLUNKey) for the Intel Modular Server setup and a storage shelf with the needed LUN Sharing and/or LUN Masking plus LUN Sharing depending on our needs.

We had our first Hyper-V cluster!

Storage Spaces

When Storage Spaces came along in 2012 RTM we decided to venture into Clustered Storage Spaces via 2 nodes and a shared JBOD. That process took about two to three months to figure out.

Our least expensive cluster option based on this setup (blog post) is deployed at a 15 seat accounting firm. The cost versus the highly available workloads benefit ratio is really attractive. :)

We have also ventured into providing backend storage via Scale-Out File Server clusters for Hyper-V cluster frontends. Fabric between the two starts with 10GbE and SMB Multichannel.


All Broadcom and vendor rebranded Broadcom NICs require VMQ disabled for each network port!

A best practice for setting up each node is to have a minimum of four ports available. Two for the management network and Live Migration network and two for the virtual switch team. Our preference is for a pair of Intel Server Adapter i350-T4s set up as follows:

  • Port 0: Management Team (both NICs)
  • Port 1 and 2: vSwitch (no host OS access both NICs)
  • Port 3: Live Migration networks (LM0 and LM1)

For higher end setups, we install at least one Intel Server Adapter X540-T2 to bind our Live Migration network to each port. In a two node clustered Storage Spaces setting the 10GbE ports are direct connected.

Enabling Jumbo Frames is mandatory for any network switch and NIC carrying storage I/O or Live Migration.


In our experience GHz is king over cores.

The maximum amount of memory per socket/NUMA node that can be afforded should be installed.

All components that can be should be run in pairs to eliminate as many single points of failure (SPFs) as is possible.

  • Two NICs for the networking setup
  • Two 10GbE NICs at the minimum for storage access (Hyper-V <—> SOFS),
  • Two SAS HBAs per SOFS node
  • Two power supplies per node

On the Scale-Out File Server cluster and Clustered Storage Spaces side of things one could scale up the number of JBODs to provide enclosure resilience thus protecting against a failed JBOD.

The new DataON DNS-2670 70-bay JBOD supports eight SAS ports per controller for a total of 16 SAS ports. This would allow us to scale out to eight SOFS nodes and eight JBODs using two pairs of LSI 9300-16e (PCIe 8x)  or the higher performance LSI 9302-16e (PCIe 16x) SAS HBAs per node! Would we do it? Probably not. Three or four SOFS nodes would be more than enough to serve the eight direct attached JBODs. ;)

Know Your Workloads

And finally, _know your workloads_!

Never, ever, rely on a vendor for performance data on their LoB or database backend. Always make a point of watching, scanning, and testing an already in-place solution set for performance metrics or the lack thereof. And, once baselines have been established in testing the results remain private to us.

The two key ingredients in any standalone or cluster virtualization setting are:

  1. IOPS
  2. Threads
  3. Memory

A balance must be struck between those three relative to the budget involved. It is our job to make sure our solution meets the workload requirements that have been placed before us.


We’ve seen a progression in the technologies we are using to deploy highly available virtualization and storage solutions.

While the technology does indeed change over time the above guidelines have stuck with us since the beginning.

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book

Thursday, 11 February 2016

Philip’s Ultra-Healthy and Quick Technician’s Breakfast!

We’re all super busy. Eating breakfast is an important part of it with a home grown meal being way better than most anything a fast food place can serve. It’s a lot less expensive in the long run plus the time savings is huge!

This breakfast meal assumes the person is in some sort of regular excercise routine which is also an important part of keeping ourselves healthy. Right? ;)

  • Breakfast Sloppy Toast
    • (3) Large Eggs
    • ~125ml to ~200ml of Half & Half Cream
    • (2) Whole Grain, 12 Grain, or other such solid bread
    • 1/8” slices of Cheddar, Marble, Havarti, Mozza, or other favourite cheese
    • A good chunk of baby spinach
    • A proper Pyrex microwave dish and cover
      • Plastic containers melt into the food :P
      • Ours is just larger WxL wise than the bread slizes and tall enough to host the lot

With the above:

  1. Break the 3 eggs into the Pyrex dish
  2. Start whisking
  3. Add cream until well frothed
  4. Place first slize of bread in the mix
  5. Cover the bread with the cheese slices
  6. Drop spinach in and evenly distribute
  7. Place second slice of bread in
  8. Use a spat to flip the stack over
  9. Press in to allow mix to soak into the new slice
  10. Cover and microwave for 5:15 at 60%
    1. Let sit for about a minute after the cycle completes
  11. Microwave for about 1:45 to 2:45 at 60% depending on microwave power

Once it’s done let it cool off for a good five minutes.

Total time put in to the above: Less than 4 minutes.

Time savings over the week?

Assuming a minimum 15-20 minute wait at Timmys (Tim Hortons) that’s easily 15 minutes per day or more.

Cost savings?

Two breakfast egg sandwiches with cheese and bacon on an English muffin is $6.20. The savings can be quite substantial.

I usually have a couple of Vietnamese bird peppers to chow on while eating the above to accentuate the flavour. ;)

While this breakfast is not for everyone, it follows a 50/25/25 rule for protien/fat/carbs. Tie that in to a good regular cardio workout we’re good to go!

Thanks for reading. :)

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book