Tuesday, 20 September 2016

Windows Server 2016: Storage Spaces Direct (S2D) and VMware Virtual SAN IOPS?

We’re not VMware experts here. Let’s make that very clear. 

However, we do have the ear of many a VMware expert so we are able to ask: Is the Windows Server 2016 hyper-converged Storage Spaces Direct (S2D) setup similar to the VMware Virtual SAN platform? Apples to Apples if you will?

The resounding answer has been, “Yes!”

That leads us to the following articles that are about Intel NVMe all-flash hyper-converged setups:

  1. Record Performance of All Flash NVMe Configuration – Windows Server 2016 and Storage Spaces Direct - IT Peer Network
    • An Intel based four node S2D setup
  2. VMware Virtual SAN 6.2 Sets New Record - IT Peer Network
    • A VMware based 8 node Virtual SAN 6.2 setup

What’s interesting to note is the following:

  1. Intel/S2D setup at 4 nodes
    • 3.2M IOPS at 4K Read
      • 930K IOPS at 70/30 Read/Write
  2. VMware setup at 8 nodes
    1. 1.2M IOPS @ Capacity Mode
      • 500K IOPS at 70/30 Read/Write
    2. ~830K IOPS @ Balance Mode
      • 800K IOPS at 70/30 Read/Write

What is starting to become apparent, at least to us, is that the folks over at the newly minted Dell EMC _really_ need to pay attention to Microsoft’s feature depth in Windows Server 2016. In fact, that goes without saying for all vendors into virtualization, high performance computing (HPC), storage, and data centre products.

We’re pretty excited about the hyper-converged S2D feature set in Windows Server 2016. So much so, we have invested quite heavily in our Proof-of-Concept (PoC).

image

The bill of materials (BoM) for the setup so far:

  • S2D Nodes (4)
    • Intel Server Systems R2224WTTYS with dual E5-2640, 256GB ECC, Intel JBOD HBA, Intel X540-T2 10GbE, Mellanox 56GbE NICs, Intel PCIe 750 NVMe, Intel SATA SSDs, and Seagate 10K SAS spindles
  • Hyper-V Nodes (2)
    • Intel Server Systems R2208GZ4GC with dual E5-2650, 128GB ECC, Intel X540-T2 10GbE, and Mellanox 56GbE NICs
  • Lab DC
    • An Intel Server setup including S1200BT series board and Intel Xeon E3 processor.

Our testing includes using the S2D setup as a hyper-converged platform but also as a Scale-Out File Server (SOFS) cluster destination for a Hyper-V cluster on the two Hyper-V nodes. Then, some testing of various configurations beyond.

We believe Windows Server 2016 is looking to be one of the best server operating systems Microsoft has ever released. Hopefully we won’t be seeing any major bugs in the production version!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Our SMB/SME HA Solution
Our Cloud Service

Tuesday, 26 July 2016

Some Disaster Recovery Planning On-Premises, Hybrid, and Cloud Thoughts

This was a post to the SBS2K Yahoo list in response to a comment about the risks of encrypting all of our domain controllers (which we have been moving towards for a year or two now). It’s been tweaked for this blog post.

***

We’ve been moving to 100% encryption in all of our standalone and cluster settings.

Encrypting a setup does not change _anything_ as far as Disaster Recovery Plans go. Nothing. Period.

The “something can go wrong there” attitude should apply to everything from on-premises storage (we’ve been working with a firm that had Gigabytes/Terabytes of data lost due to the previous MSP’s failures) and services to Cloud resident data and services.

No stone should be left unturned when it comes to backing up data and Disaster Recovery Planning. None. Nada. Zippo. Zilch.

The new paradigm from Microsoft and others has migrated to “Hybrid” … for the moment. Do we have a backup of the cloud data and services? Is that backup air-gapped?

Google lost over 150K mailboxes a number of years back, we worked with one panicked call who lost everything, with no return. What happens then?

Recently, a UK VPS provider had a serious crash and, as it turns out lost _a lot_ of data. Where are their clients now? Where’s their client’s business after such a catastrophic loss?

Some on-premises versus cloud based backup experiences:

  • Veeam/ShadowProtect On-Premises: Air-gapped (no user access to avoid *Locker problems), encrypted, off-site rotated, and high performance recovery = Great.
  • Full recovery from the Cloud = Dismal.
  • Partial recovery of large files/numerous files/folders from the Cloud = Dismal.
  • Garbage In = Garbage Out = Cloud backup gets the botched bits in a *Locker event.
  • Cloud provider’s DC goes down = What then?
  • Cloud provider’s Services hit a wall and failover fails = What then (this was a part of Google’s earlier mentioned problem me thinks)?
    • ***Remember, we’re talking Data Centers on a grand scale where failover testing has been done?!?***
  • At Scale:
    • Cloud/Mail/Services providers rely on a myriad of systems to provide resilience
      • Most Cloud providers rely on those systems to keep things going
    • Backups?
      • Static, air-gapped backups?
      • “Off-Site” backups?
        • These do not, IMO, exist at scale
  • The BIG question: Does the Cloud service provider have a built-in backup facility?
    • Back up the data to local drive or NAS either manually or via schedule
    • Offer a virtual machine backup off their cloud service

There is an assumption, and we all know what that means right?, that seems to be prevalent among top tier cloud providers that their resiliency systems will be enough to protect them from that next big bang. But, has it? We seem to already have examples of the “not”.

In conclusion to this rather long winded post I can say this: It is up to us, our client’s trusted advisors, to make bl**dy well sure our client’s data and services are properly protected and that a down-to-earth backup exists of their cloud services/data.

We really don’t enjoy being on the other end of a phone call “OMG, my data’s gone, the service is offline, and I can’t get anywhere without it!” :(

Oh, and BTW, our SBS 2003/2008/2011 Standard/Premium sites all had 100% Uptime across YEARS of service. :P

We did have one exception in there due to an inability to cool the server closet as the A/C panel was full. Plus, the building’s HVAC had a bunch of open primary push ports (hot in winter cold in summer) above the ceiling tiles which is where the return air is supposed to happen. In the winter the server closet would hit +40C for long periods of time as the heat would settle into that area. ShadowProtect played a huge role in keeping this firm going plus technology changes over server refreshes helped (cooler running processors and our move to SAS drives).

*** 

Some further thoughts and references in addition to the above forum post.

The moral of this story is quite simple. Make sure _all_ data is backed up and air-gapped. Period.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Thursday, 14 July 2016

Hyper-V and Cluster Important: A Proper Time Setup

We’ve been deploying DCs into virtual settings since 2008 RTM. There was not a lot of information on virtualizing anything back then. :S

In a domain setting having time properly synchronized is critical. In the physical world the OS has the on board CMOS timer to keep itself in check along with the occasional poll of ntp.org servers (we use specific sets for Canada, US, EU/UK, and other areas).

In a virtual setting one needs to make sure that time sync between host and guest PDCe/DC time authority is disabled. The PDCe needs to get its time from an outside, and accurate, source. The caveat with a virtual DC is that it no longer has a physical connection with the local CMOS clock.

What does this mean? We’ve seen high load standalone and clustered guests have their time skew before our eyes. It’s not as much of a problem in 2012 R2 as it was in 2008 RTM/R2 but time related problems still happen.

This is an older post but outlines our dilemma: MPECS Inc. Blog: Hyper-V: Preparing A High Load VM For Time Skew.

This is the method we use to set up our PDCe as authority and all other DCs as slaves: Hyper-V VM: Set Up PDCe NTP Time Server plus other DC’s time service.

In a cluster setting we _always_ deploy a physical DC as PDCe: MPECS Inc. Blog: Cluster: Why We Always Deploy a Physical DC in a Cluster Setting. The extra cost is 1U and a very minimal server to keep the time and have a starting place if something does go awry.

In higher load settings where time gets skewed scripting the time sync with a time server within the guest DC to happen more frequently means the time server will probably send a Kiss-O-Death packet (blog post). When that happens the PDCe will move on through its list of time servers until there are no more. Then things start breaking and clusters in the Windows world start stalling or failing.

As an FYI: A number of years ago we had a client call us to ask why things were wonky with the time and some services seemed to be offline. To the VMs everything seemed to be in order but their time was whacked as was the cluster node’s time.

After digging in and bringing things back online by correcting the time on the physical DC, the cluster nodes, and the VMs everything was okay.

When the owner asked why things went wonky the only explanation I had was that something in the time source system must have gone bad which subsequently threw everything out on the domain.

They indicated that a number of users had complained about their phone’s time being whacked that morning too. Putting two and two together there must have been a glitch in the time system providing time to our client site and the phone provider’s systems. At least, that’s the closest we could come to a reason for the time mysteriously going out on two disparate systems.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Tuesday, 5 July 2016

Outlook Crashes - Exchange 2013 or Exchange 2016 Backend

We’ve been deploying Office/Outlook 2016 and Exchange 2016 CU1 in our Small Business Solution (SBS) both on-premises and in our Cloud.

Since we’re deploying Exchange with a single common name certificate we are using the _autodiscover._tcp.domain.com method for AutoDiscover.

A lot of our searching turned up “problems” around AutoDiscover but pretty much all of them were red herrings.

It turns out the Microsoft has deprecated the RPC over HTTPS setup in Outlook 2016. What does this mean?

MAPI over HTTPS is the go-to for Outlook communication with Exchange going forward.

Well, guess what?

MAPI over HTTPS is _disabled_ out of the box!

In Exchange PowerShell check on the service’s status:

Get-OrganizationConfig | fl *mapi*

To enable:

Set-OrganizationConfig -MapiHttpEnabled $true

Then, we need to set the virtual directory configuration:

Get-MapiVirtualDirectory -Server EXCHANGESERVER | Set-MapiVirtualDirectory -InternalUrl  https://mail.DOMAIN.net/MAPI -ExternalUrl https://mail.DOMAIN.net/MAPI

Verify the settings took:

Get-MapiVirtualDirectory -Server EXCHANGESERVER | FL InternalUrl,ExternalUrl

And finally, test the setup:

Test-OutlookConnectivity -RunFromServerId EXCHANGESERVER -ProbeIdentity OutlookMapiHttpSelfTestProbe

EXCHANGESERVER needs to be changed to the Exchange server name.

Hat Tip: Mark Gossa: Exchange 2013 and Exchange 2016 MAPI over HTTP

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Thursday, 26 May 2016

Hyper-V Virtualization 101: Hardware and Performance

This is a post made to the SBS2K Yahoo List.

***

VMQ on Broadcom Gigabit NICs needs to be disabled at the NIC Port driver level. Not in the OS. Broadcom has not respected the spec for Gigabit NICs at all. I’m not so sure they have started to do so yet either. :S

In the BIOS:

  • ALL C-States: DISABLED
  • Power Profile: MAX
  • Intel Virtualization Features: ENABLED
  • Intel Virtualization for I/O: ENABLED

For the RAID setup we’d max out the available drive bays on the server. Go smaller volume and more spindles to achieve the required volume. This gains us more IOPS which are critical in smaller virtualization settings.

Go GHz over Cores. In our experience we are running mostly 2vCPU and 3vCPU VMs so ramming through the CPU pipeline quicker gets things done faster than having more threads in parallel at slower speeds.

Single RAM sticks per channel preferred with all being identical. Cost of 32GB DIMMs has come down. Check them out for your application. Intel’s CPUs are set up in three tiers. Purchase the RAM speed that matches the CPU tier. Don’t purchase faster RAM as that’s more expensive and thus money wasted.

Be aware of NUMA boundaries for the VMs. That means that each CPU may have one or more memory controller each. Each controller manages a chunk of RAM attached to that CPU. When a VM is set up with more vRAM than what is available on one memory controller that memory gets split up. That costs in performance.

Bottlenecks not necessarily in order:

  • Disk subsystem is vastly underperforming (in-guest latency and in-guest/host Disk Queue Length are key measures)
    • Latency: Triple digits = BAD
    • Disk Queue Length: > # Disks / 2 in RAID 6 = BAD (8 disks in RAID 6 then DQL of 4-5 is okay)
  • vCPUs assigned is greater than the number of physical cores – 1 on one CPU (CPU pipeline has to juggle those vCPU threads in parallel)
  • vRAM assigned spans NUMA nodes or takes up too much volume on one NUMA node
  • Broadcom Gigabit VMQ at the port level

The key in all of this though and it’s absolutely CRITICAL is this: Know your workloads!

All of the hardware and software performance knowledge in the world won’t help if we don’t know what our workloads are going to be doing.

An unhappy situation is spec’ing out a six to seven figure hyper-converged solution and having the client come back and say, “Take it away I’m fed up with the poor performance”. In this case the vendor over-promised and under-delivered.

Some further reading:

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Thursday, 12 May 2016

RDMA via RoCE 101 for Storage Spaces Direct (S2D)

We’ve decided to run with RoCE (RDMA over Converged Ethernet) for our Storage Spaces Direct (S2D) proof of concept (PoC).

image

  • (4) Intel Server Systems R2224WTTYS
    • Dual Intel Xeon Processors, 256GB ECC, Dual Mellanox ConnectX-3, and x540-T2
    • Storage is a mix of 10K SAS and Intel SATA SSDs to start
  • (2) Mellanox MSX1012 56Gbps Switches
  • (2) NETGEAR XS712T 10GbE Switches
  • (2) Cisco SG500x-48 Gigabit Switches
  • APC 1200mm 42U Enclosure
  • APC 6KV 220v UPS with extended runtime batteries

The following is a list of resources we’ve gathered together as of this writing:

This is, by far, not the most comprehensive of lists. The best place to start in our opinion is with Didier’s video and the PDF of the slides in that video. Then move on to the Mellanox resources.

We’ll update this blog post as we come across more materials and eventually get a process guide in place.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Tuesday, 26 April 2016

Remote Desktop Services 301: Some Advanced RDS Setup Guidelines

Here are some of the key tenants we’ve picked up deploying standalone and farm based Remote Desktop Services (RDS) for our Small Business Solution (SBS) on-premises and in our cloud.

Virtual Desktop Infrastructure, or VDI for short, covers both Remote Desktop Services standalone, farm, and desktop operating system deployments.

This list, while not totally comprehensive, covers a lot of ground.

  • Hardware, Storage, and Networking
    • GHz and Cores are king
      • Balance core count and GHz versus cost
      • NOTE: Server 2016 licensing will be, and is as of this writing, based on core count!
        • Today is 2 sockets tomorrow is a server total of 16 cores
        • Additional Cores purchased in pairs
        • Example 1: Dual Socket 8 Core pair = 16 Cores total so OK in current and 2016 licensing
        • Example 2: Dual Socket 12 Core pair = 24 Cores total so base license of 16 Cores plus a purchase of 4 licenses (2 cores per license) would be required
        • NOTE: Examples may not line up with actual license terms! Please verify when 2016 goes live.
    • RAM is cheap so load up now versus later
    • Balanced RAM setup is better than Unbalanced
      • Balanced: 4x 16GB in primary memory channel slot
        • Best performance
      • Unbalanced: 4x 16GB in primary and 4x 8GB in secondary
        • Performance hit
    • ~500MB of RAM per user per session to start
    • 25-50 IOPS per user depending on workloads/workflows RDS/VDI
      • Average 2.5” 10K SAS is ~250 to 400 IOPS depending on stack format (stripe/block sizes)
    • Latency kills
      • Direct Attached SAS or Hyper-Converged is best
      • Lots of small reads/writes
    • Average 16bpp RDS session single monitor 23” or 24” wide: ~95KB/Second
      • Average dual monitor ~150KB/Second
      • Bandwidth use is reduced substantially with a newer OS serving and connecting remotely (RDP version)
  • LoBs (Line of Business applications)
    • QuickBooks, Chrome, Firefox, and Sage are huge performance hogs in RDSH
      • Be mindful of LoB requirements and provision wisely
    • Keep the LoB database/server backend somewhere else
    • In larger settings dedicate an RDSH and RemoteApp to the resource hog LoB(s)
  • User Profile Disks
    • Office 2013 and Exchange 2013 are a wash in this setting
      • Search is difficult if not broken
    • Search Index database will bloat and fill the system disk! (Blog post with “fix”)
    • Office 2016, though still a bit buggy as of this writing, and Exchange 2016 address UPDs and search
    • Be mindful of network fabrics between UPDs and RDSH(s)
    • Set the UPD initial size a lot larger than needed as it can’t be changed later without a series of manual steps
      • UPDs are dynamic
      • Keep storage limits in mind because of this
  • Printing
    • Printers with PCL 5/6 engines built-in are preferred
      • Host-based printers are a no-go for us
    • HP Professional series LaserJet printers are our go-to
    • HP MFPs are preferred over Copiers
      • Copier engines tend to be hacked into Windows while the HP MFP is built as a printer out of the box

Previous post on the matter: Some Remote Desktop Session Host Guidelines.

Here’s a snippet of Intel’s current Intel Xeon Processor E5-2600v4 line sorted by base frequency in GHz:

image

Depending on the deployment type we’re deploying either E5-2643v4 or E5-2667v4 processors for higher density setups at this time. We are keeping at or under eight cores per socket unless we absolutely require more due to the upcoming sockets to cores changes in Windows Server licensing.

If you’d like a copy of that spreadsheet ping us or visit the Intel Ark site.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud