Thursday, 11 February 2016

Philip’s Ultra-Healthy and Quick Technician’s Breakfast!

We’re all super busy. Eating breakfast is an important part of it with a home grown meal being way better than most anything a fast food place can serve. It’s a lot less expensive in the long run plus the time savings is huge!

This breakfast meal assumes the person is in some sort of regular excercise routine which is also an important part of keeping ourselves healthy. Right? ;)

  • Breakfast Sloppy Toast
    • (3) Large Eggs
    • ~125ml to ~200ml of Half & Half Cream
    • (2) Whole Grain, 12 Grain, or other such solid bread
    • 1/8” slices of Cheddar, Marble, Havarti, Mozza, or other favourite cheese
    • A good chunk of baby spinach
    • A proper Pyrex microwave dish and cover
      • Plastic containers melt into the food :P
      • Ours is just larger WxL wise than the bread slizes and tall enough to host the lot

With the above:

  1. Break the 3 eggs into the Pyrex dish
  2. Start whisking
  3. Add cream until well frothed
  4. Place first slize of bread in the mix
  5. Cover the bread with the cheese slices
  6. Drop spinach in and evenly distribute
  7. Place second slice of bread in
  8. Use a spat to flip the stack over
  9. Press in to allow mix to soak into the new slice
  10. Cover and microwave for 5:15 at 60%
    1. Let sit for about a minute after the cycle completes
  11. Microwave for about 1:45 to 2:45 at 60% depending on microwave power

Once it’s done let it cool off for a good five minutes.

Total time put in to the above: Less than 4 minutes.

Time savings over the week?

Assuming a minimum 15-20 minute wait at Timmys (Tim Hortons) that’s easily 15 minutes per day or more.

Cost savings?

Two breakfast egg sandwiches with cheese and bacon on an English muffin is $6.20. The savings can be quite substantial.

I usually have a couple of Vietnamese bird peppers to chow on while eating the above to accentuate the flavour. ;)

While this breakfast is not for everyone, it follows a 50/25/25 rule for protien/fat/carbs. Tie that in to a good regular cardio workout we’re good to go!

Thanks for reading. :)

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book

Tuesday, 9 February 2016

Hyper-V 101: What Windows Server Media Should I Use?

This may seem like a bit of a silly N00b style post but there’s a good reason for it.

How many of us are using Windows Server Media to install hosts via USB Flash then guests via ISO?

I venture to guess almost all of us.

Okay, POLL Time: What is the _date stamp_ on the Setup.EXE located on that flash/ISO?

As of today, if it’s a date earlier than November 22, 2014 then it’s _too old_ to be used in production systems:


Please log on to the Microsoft Volume Licensing Service Centre, MSDN, or TechNet to download a newer ISO.

Then update the flash drives used to install Hyper-V hosts and nodes.

It should be Standard Order of Procedure (SOP) to keep operating system load souces up to date.

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book

Monday, 8 February 2016

Remote Desktop Session Host: The System Partition is Getting Full?

In our on-premises and Cloud Desktop RDSenvironments we’ve discovered a number of different things that cause problems for users in Windows Server 2012 R2, Exchange 2013, and Office 2013.

All of our Remote Desktop Session Hosts (RDSHs) are set up with two VHDX files. One for the operating system and one for the data and User Profile Disks (UPDs).

Unfortunately, while UPDs give us a great flexbility option that allow us to have then on the network thus avoid local profile pains in a RDS Farm setting they have a number of different negative impacts on user experience and RDSH health.

One that impacts both is the mysterious filling up of the system partition.

As it turns out, Outlook 2013 and Exchange 2013 plus UPDs means Outlook search is almost completely broken.

But, that doesn’t stop the Windows Server Search Service from doing its best to catalog everything anyway!

What does that mean?

Well, eventually we have a search database that can grow to epic proportions.

Since all of our OS partitions are rather small we end up with session hosts getting their system partition filled rather quickly on a busy RDSH. This is especially true in a Farm setting.

So, what are our options?

Well, we could disable the Windows Search Service. This would be a bad idea since users wouldn’t be able to find _anything_ anymore. We’d go from the occasional complaint to constant complaints. So, not good.

The alternative is to reset the Windows Search index.

  1. Start –> Indexing Options
  2. Advanced button (UAC)
  3. Click the Rebuild button

And, voila! In some cases we get 45GB to 60GB of space back in short order!

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book

Saturday, 6 February 2016

Hyper-V Virtualization 101: Some Basics Around Hyper-V Setups

Here are some pearls learned over our years working with Hyper-V:
  • Host/Node Setup
    • Make sure the host and all nodes in a cluster have the BIOS Settings identical (All settings)
    • Leave ~1GB to 1.5GB physical RAM to the host
    • We leave 1.5GB of space on a dedicated to VM LUN
    • We leave ~50GB to ~100GB of free space on a shared LUN/Partition
    • We set the MiniDump option and lock the swap file down to 840MB
      • wmic RECOVEROS set DebugInfoType = 3
    • Always set up a standalone host with two partitions: OS and Data
  • Hyper-V
    • Hyper-V lays out a file equivalent in size to the vRAM assigned to VMs. We must have space for them.
    • Snapshots/CheckPoints create differencing disks. These _grow_ over time.
    • Deleting Snapshots/CheckPoints requires enough free space to create an entirely new Parent VHDX.
    • vRAM assigned to the VM should not traverse NUMA nodes (performance) (more on hardware).
    • vCPUs = Threads in CPU and must be processed in parallel thus # physical cores - 1 is best.
    • GHz is preferred over CPU Core counts for most workloads.
  • Storage
    • Be aware of the IOPS required to run _all_ workloads on the host/nodes.
    • More smaller sized spindles is better than less larger size spindles = More IOPS.
    • 10GbE should be the minimum bandwidth considered for any iSCSI deployments.
    • At least _two_ 10GbE switches are mandatory for the storage path
  • Networking
    • Broadcom physical NIC ports must always have VMQ turned off (blog post)
    • We prefer to use Intel Gigabit and 10Gb Ethernet Server Adapters
    • We start with a minumum of 4 physical ports in our hosts/nodes
  • UPS Systems
    • UPS Systems should have at least 1-1.5 Hours of runtime.
    • Host/Nodes and storage should be tested for shutdown durations.

There are quite a lot of these types of posts on our blog. Please click through the category tags to find more!

Thanks for reading.

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book

Friday, 5 February 2016

Some Remote Desktop Session Host Guidelines

We’ve put about four years and two versions into our Small Business Solution (SBS). We have it running on-premises on standalone Hyper-V servers as well as on Hyper-V clusters (Clustered Storage Spaces and Hyper-V cluster we just deployed for a 15 seat accounting firm).

It is the foundation for the Cloud Office services we’ve been offering for the last year or so.

Since our Cloud Office solution runs in Remote Desktop Services we figured we’d share some pearls around delivering Remote Desktop Session Host based environments to clients:

  • ~512MB/User is cutting it tight
  • ~20 to 25 users in a 12GB to 16GB vRAM Hyper-V VM works okay with 2-3 vCPUs
  • RDP via 8.1 RDP clients saturates a 1Mb DSL uplink at ~13-15 users depending on workload
  • ALL browsers can bring the RDSHs to their knees
  • Printing can be a bear to manage (Use Universal Print Drivers and Isolation where possible)
  • Group Policy configuration and lockdown is mandatory
  • Two partitions with User Profile Disks (UPDs), if used, on the second partition
  • NOTE: UPDs + Office 2013 and earlier + Exchange 2013 and earlier = Broken Search!!!
  • NOTE: RDSH Search Indexes for Outlook OSTs in UPDs can fill up the C: partition!
    • Office 2016 and Exchange 2016 together are supposed to address the broken search situation in RDSH setups were UPDs are used. We have yet to begin testing the two together.

Our Cloud Office (SBS) is running on clusters we’ve designed based on Scale-Out File Server and Hyper-V.

Need a clustered solution for your SMB/SME clients? Drop us a line. They are _very_ affordable. ;)

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book

Thursday, 4 February 2016

Protecting a Backup Repository from Malware and Ransomware

With the abundance of malware and ransomware it’s absolutely necessary that we take the time to examine our backup structures.

  1. Volume Shadow Copies
    • Obviously not a “backup”
    • Most ransomware today kills these
  2. Backup to Disk/NAS
    • Rotated or streamed off-site
  3. Cloud Backup
    • Streamed off-site
  4. Backup Tiers
    1. Current, off-site 1, off-site 2, 6 Month, 12 Month, ETC…

With our last mile issue up here we are very careful about anything Cloud since most upload speeds are not capable enough nor are the download speeds capable of a decent recovery time.

Now, what is _the most important_ aspect to our backup setup?


It must be a closed loop!

What does that mean?

That means that at no point in the backup structure can anyone have access to the backups via the network or console.

Now, since almost all of our backups are streamed across the wire it takes a bit of a process to make sure our loop is closed.

  • NAS
    • ShadowProtect user with unique pass phrase (SPUP) and MOD on the repository root folder
      • Other than the NAS Admin account no other user account is set up with access
      • Turn on the NAS Recycle Bin!
        • Most ransomware creates a new file then deletes the old one
        • Create a separate username and folder structure for user facing resources!
  • ShadowProtect
    • Network destination set up with SPUP
  • ShadowProtect Backups
    • Encrypted AES 256-bit with a long pass phrase
  • ImageManager
    • All managed backups are set up to be accessed via SPUP only
      • No repository, whether NAS or USB HDD is left with Users MOD
      • No repository is left without a restricted username and password protecting it!

Recently, we know of a domain joined standalone Hyper-V server get hit by ransomware. As a rule we don’t join a standalone Hyper-V to the guest domain. This is just one more reason for us not to do so.

And finally, some of the more obvious aspects around backups and domain operation in general:

  • Users are Standard Users on the domain
    • If they absolutely need local admin because they are still running QuickBooks 2009 then make that choice
    • Standard User accounts have _NO_ access to any aspect of the backup loop
      • None, Nada, Zippo, Zilch! ;)
    • Domain Admin accounts should have no access to any aspect of the backup loop
      • Many client sites have one or two users (hopefully not more?!?!?) that know these credentials
    • Access via UNC will pop up an authentication dialogue box.
      • Use the SPUP and _do not save_ the credentials!
  • Backups are managed by us, spot recovered by us, and quarterly bare metal/hypervisor restored by us
    • No client intervention other than perhaps the off-site rotation (we do this too)
  • If some user or users insist running as DOMAIN ADMINs then REMOVE Admin’s MOD from USB HDD/NAS NTFS/File System
    • Leave only the SPUP with MOD

So, what spawned this blog post?

Hearing of a ShadowProtect destination NAS getting wiped out by ransomware. This should not be possible on our managed networks ever!

What spawned our lockdown of the backup structures?

Many years back we had a user that neglected to rotate the tape libraries and a faulty BackupExec that reported all being rosy until their server went full-stop and we had to recover (one aspect of the recovery in an SBS environment).

When we arrived, the person rotating the magazines turned sheet white when we asked for the off-site magazines. Oops. :(

We dropped BackupExec as their support failed to help us after three days of wrangling (Thursday afternoon until we cut the cord at 1730Hrs Saturday evening). We did end up recovering the full 650GB of data short of 24 files belonging to one of the firm’s partners across four to five days.

After that we went to all of our clients and proposed a managed backup strategy where we took care of all aspects of the backup. They all approved the changes after hearing what happened at the one firm. ;)

So, we tested and switched all of our clients to ShadowProtect 3.x and set up all backups so that no user could access them.

In our not so humble opinion, backups are not, and should never be, a user’s responsibility.

Thus, they should never have access to them even if they rotate them!

TIP: Need to do a side-by-side recovery or migration? ForensiT’s User Profile Wizard

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book

Thursday, 28 January 2016

Cluster: A Simple Cluster Storage Setup Guide

In a cluster setting we have a set way to configure our shared storage whether it resides on a SOFS (Scale-Out File Server) cluster or some sort of network based storage.

First, the process to set up the storage itself:

  1. Configure the LUN
    • LUN ID must be identical for all Hyper-V nodes for SAN/NAS
  2. Connect all nodes to the storage
    • iSCSI Target for SAN/NAS
  3. Format NTFS and set OFFLINE on Node01
  4. Node2 and up ignore Initialize in Disk Management and set OFFLINE
    • This step is optional depending on the setup

When it comes to the storage we configure the following LUNs for all of our cluster setups;

  1. 1.5GB LUN
    • Set up for the Witness Disk
    • Add to Cluster Storage but NOT CSV
  2. ???GB LUN
    • Sum of all physical RAM on the nodes plus 150GB
    • Add to Cluster Shared Volumes
    • All Hyper-V nodes set to deliver VM settings files to this location
    • Don’t forget that Hyper-V writes a file that is equivalent in size for _all_ VMs running on the cluster or standalone host!
  3. Minimum 50% Storage LUN x2
    • Divide the remaining storage into two or more LUNs depending on workload and storage requirements
    • A minimum of 2 LUNs allows for storage load to be shared across the SAN’s two storage controllers, the two iSCSI networks, and the two or more Hyper-V nodes

In a SOFS setting we set up a File Share Witness for our Hyper-V compute clusters and deliver the HA shares via SMB Multichannel and a minimum of 10GbE for the VHDX files.


The PowerShell steps for any of the above are here to avoid copy and paste issues.

Set Default Paths:

Set-VMHost -VirtualHardDiskPath “C:\ClusterStorage” –VirtualMachinePath “C:\ClusterStorage\Volume1”

We point the VHDX setting to the CSV root just in case. Our PowerShell scripts for setting up VMs put the VHDX files into the right storage location.

Set Quorum Up:

Set-ClusterQuorum -NodeAndDiskMajority "Cluster Virtual Disk (Witness Disk)"

Philip Elder
Microsoft High Availability MVP
Co-Author: SBS 2008 Blueprint Book