Cluster: A Simple Cluster Storage Setup Guide

In a cluster setting we have a set way to configure our shared storage whether it resides on a SOFS (Scale-Out File Server) cluster or some sort of network based storage.

First, the process to set up the storage itself:

1. Configure the LUN
• LUN ID must be identical for all Hyper-V nodes for SAN/NAS
2. Connect all nodes to the storage
• iSCSI Target for SAN/NAS
3. Format NTFS and set OFFLINE on Node01
4. Node2 and up ignore Initialize in Disk Management and set OFFLINE
• This step is optional depending on the setup

When it comes to the storage we configure the following LUNs for all of our cluster setups;

1. 1.5GB LUN
• Set up for the Witness Disk
• Add to Cluster Storage but NOT CSV
2. ???GB LUN
• Sum of all physical RAM on the nodes plus 150GB
• Add to Cluster Shared Volumes
• All Hyper-V nodes set to deliver VM settings files to this location
• Don’t forget that Hyper-V writes a file that is equivalent in size for _all_ VMs running on the cluster or standalone host!
3. Minimum 50% Storage LUN x2
• Divide the remaining storage into two or more LUNs depending on workload and storage requirements
• A minimum of 2 LUNs allows for storage load to be shared across the SAN’s two storage controllers, the two iSCSI networks, and the two or more Hyper-V nodes

In a SOFS setting we set up a File Share Witness for our Hyper-V compute clusters and deliver the HA shares via SMB Multichannel and a minimum of 10GbE for the VHDX files.

PowerShell

The PowerShell steps for any of the above are here to avoid copy and paste issues.

Set Default Paths:

Set-VMHost -VirtualHardDiskPath “C:\ClusterStorage” –VirtualMachinePath “C:\ClusterStorage\Volume1”

We point the VHDX setting to the CSV root just in case. Our PowerShell scripts for setting up VMs put the VHDX files into the right storage location.

Set Quorum Up:

Set-ClusterQuorum -NodeAndDiskMajority "Cluster Virtual Disk (Witness Disk)"

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Opinion: Yammer Needs To Go

With all due respect, Microsoft really needs to cut and run from Yammer.

We bill into the hundreds of dollars per hour.

Yammer is such a bleed on time that it is costing us substantially to try and get any kind of use out of it.

We’ve all but abandoned the platform for all but necessary tasks.

We already wasted a couple of hours this week trying to get to our content downloads for our business needs.

We can’t imagine the impact to Microsoft employee productivity that Yammer has had. But, given our experience it has to be substantial.

Yammer needs to go. In our not so humble opinion it’s worse than a 1988 Yugo and it’s 3-6 month air cooled engine life.

Image Credit: Bing Search

Microsoft trying to fix it is no different than trying to fix a Yugo. In our opinion it’s good money after bad.

[/RANT]

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Windows 7, 10, and Skylake?

There’s been a lot of furor and FUD around Windows 7 on the upcoming Skylake platform being blocked or not?

Has anyone seen a revision to the x86 instruction set? We certainly haven’t. We’d most certainly have seen something somewhere about it.

There most certainly has been changes to the processor’s periphery in additional instruction sets for various tasks the CPU would perform in a given era. The Intel/AMD platforms are littered with their various efforts over the last two decades.

The only way we see something like this happening, where an OS won’t run on a certain platform, is by hard coding that into the OS. In the case of Windows 7, that would mean backporting a patch of some sort. Folks would figure out RPQ what patch caused the hard-code and that would be the end of that for many.

We realize Microsoft wants folks to move to Windows 10. There’s a huge revenue opportunity to be had there.

However, surreptitiously going about it with sneaky ads, involuntary downloads to users computers, delivery of the OS to ADDS based environments unannounced (coming to a domain near you), and any other clandestine methodology will, and is, do a lot of harm.

Pushing, pulling, and dragging folks who are kicking and screaming to be left where they are is not a good way to go about it.

[/.25c]

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Script: Clear DNS Cache and Restart the DNS Cache and DNS Services

Here’s a simple CMD (Command Line) script for clearing the DNS Cache and restarting both the service and the DNS service:

DnsCMD SERVERNAME /ClearCache
Net Stop DNSCache && Net Start DNSCache
Net Stop DNS && Net Start DNS
DnsCMD SERVERNAME /ClearCache
Pause

The above script was set up to be run both on a scheduled task and manually on an older SBS 2011 Essentials VM as it seems to have DNS lookup problems.

This script solves the problem.

This server is slated to be refreshed soon.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

APC Network Management Error: Somone is currently logged into the ACP Management Web Server

We received an alarm from one of our network managed APC UPS systems. When we went to log on we received the following notice:

Notice

Someone is currently logged into the APC Management Web Server.

Please try again later.

To “fix” this “problem” we need to log on via Telnet. If the Telnet Client is not installed then go ahead and install it since it is also a good tool for testing connections and SMTP issues.

1. Install Windows Features
   • 
2. Once complete close the window.
   • 
3. From there Start –> Telnet [Enter]
4. Open 192.168.99.99
5. Log on using the admin username and password
6. Exit
   • 

Refresh the browser window and attempt to log on again. It should be successful.

If not, a paperclip reset or serial connection to the network management would be the next step.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Microsoft MVP Renewed and what’s on for 2016.

Happy New Year everyone!

I have received a confirmation that I am a Microsoft MVP for yet another year!

With the restructuring of the MVP program my Cluster, now High Availability, focus has been melded in to the Cloud and Datacenter Management umbrella category.

What this means for the MVP program is, at this time, beyond me.

For us here at MPECS Inc. we're in the process of working on a number of exciting things.

A 4 node Storage Spaces Direct cluster in TP4+

• Claus Joergensen: Microsoft and Intel showcase Storage Spaces Direct with NVM Express at IDF '15
• Claus Joergensen: Hardware options for evaluating Storage Spaces Direct in Technical Preview 4
• Claus Joergensen: Storage Spaces Direct in Technical Preview 4
1. Scale-Out File Server and Hyper-V Clusters
• We're testing in TP4
• We're testing 12Gbps SAS HBAs and JBODs with some solution sets already in production
2. Hyper-V Cluster with Clustered Storage Spaces
• 2 Nodes and 1 JBOD
• We have 12Gbps HBAs and JBODs tested and passed in this configuration
3. Mellanox InfiniBand and Ethernet fabric
• We're working with a set of Mellanox demo Ethernet switches and NICs to test our setups with RDMA via RoCE

This is going to be a busy year for us on both the SBS (Small Business Solution), Cluster, and the Cloud front via a new business venture we started mid-2015.

We hope the New Year brings many blessings for our families and our businesses!

Thanks for reading everyone! :)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Some I.T. Professional Business Pearls

Here are some thoughts on the many aspects of running an I.T. business in today’s world that have been garnered over my 13 years in business so far.

• Never reveal business operations
• Financial
• Client’s names (don’t ever reveal clients supported to others)
• Projects, Purchases, or other such products and services WE provide
• Never volunteer _any_ information
• Always keep scheduling information time related
• I’m running behind, I’ll call in 15
• I’m working on something that went sideways, let’s delay to tomorrow please
• Always call, text, reach out in-person versus e-mail
• If calling follow-up with an e-mail “thanks for allowing us to delay by a day” or some such note
• We keep a priority band that our clients are aware of
• Priority 1: Business Critical Outage
• Priority 2: User down or problematic app
• Priority 3: App updates, changes and such
• Priority 4: All of the other stuff
• Never, ever, give a customer poop for leaving
• I learned this one the hard way
• Business is business no matter how the termination was handled by the now former client
• Never, ever, take things personally
• Respect their decision and acquiesce with grace and integrity
• Be silent
• Resist the urge to be defensive - shut this one right down
• Cooperate with the next I.T. company if need be
• Give over the keys to the kingdom with ease
• Always, and everywhere, do everything in writing
• “Yes, Ms. Customer, I’d be more than happy to help you do X and it’ll be done on Y” via e-mail after a conversation
• Always confirm project add-ons and scope creep with an e-mail indicating back charges and extras as they fall out of scope
• Keep an extensive set of audit notes for each client
• Keep an extensive change log for all clients
• Snip everything, name them accordingly, and keep them forever
• Be disciplined and document everything
• Use a time keeper
• Outlook Tasks or CRM with due dates and reminders
• OneNote notebook with Surface 4/Pro and pen close at hand
• Note keeper pocket notebook and mechanical pencils
• Write all requests down and transfer what needs to be to Outlook or CRM

The above is the culmination of my 13 years running our I.T. company. I hope it helps! :)

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Exchange Stall: Purging Exchange Logs

We’ve got our SBS (Small Business Solution) set up for client’s all-in-the-Cloud experience.

What we’re finding is that some vendor’s systems don’t trigger VSS within a VM running Exchange thus leaving Exchange thinking it is not being backed up. This means that the Exchange VM eventually stalls due to no space on the partition hosting the logs.

To remedy that run the following in a script say once a week, or more depending on the volume of mail, on the Exchange VM:

1. Elevated CMD
2. DiskShadow
3. Add Volume C:
4. Begin Backup
5. Create
6. End Backup 

Once the snapshot completes Exchange will think it’s been backed up and consolidate all of the logs.

NOTE: Make sure the VM/Server is backed up!

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Folder Permissions: How To Properly Disinherit Permissions

We run into a lot of ACL corruption issues and access issues when a folder has not been disinherited properly.

The following is the best method for disinheriting permissions a folder receives from its parent:

1. Right click on the folder and click Properties
2. Click the Advanced button
3. Click the Change Permissions button if required
4. Click the Disable inheritance button
5. Click the Convert inherited permissions into explicit permissions on this object.
• 
• DO NOT CLICK REMOVE
6. Click on DOMAIN\Domain Users or MACHINE\Users and then the Remove button
• This removes access to that folder to all domain users
7. Add the necessary security groups and give them MOD
8. OPTION: On existing folder sets one can click Replace all child object permission entries with inheritable permission entries from this object
• Does one want to click this? If there are customized permissions _below_ the folder being disinherited those permissions would be lost.
9. Click Apply and OK.

From there our folder would now have the necessary permissions for users in the specific security group(s) to make changes.

We enable Access-based Enumeration on _all_ shares we deploy by default. This means that users that are not in the above assigned security group(s) will not see the folder in their File Explorer.

One of the warning signs that the above process was not followed will be for domain admin or local admin accounts to get a UAC prompt when navigating the physical folder set.

As a rule we follow a trunk –> branch –> leaf structure for our folders. All users have a single point of entry with some subfolders having their inheritance blocked.

From there we prefer to _not_ disinherit any further down-level folders unless absolutely necessary because that inevitably leads to access issues and/or permissions corruption.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

E-mail NDR: #5.1.1 SMTP; 550 Rejecting for Sender Policy Framework (SPF too many lookups)

One of our clients was having issues sending an e-mail to one of our regional ISP’s e-mail servers.

• Remote Server returned ‘<inbound.server.com #5.1.1 SMTP; 550 Rejecting for Sender Policy Framework>’
• SPF too many lookups

There was no further information. The specifics came from a very helpful mail technician at the ISP.

So, we started to dig around and came up with the following:

• Sender Policy Framework Site: OpenSPF.org
• Kitterman Technical Services’ excellent SPF Tool

A ticket went through Third Tier’s Help Desk this week that was based on this problem with Dave Shakelford pointing to the following blog post:

• JangoMail Help Center: SPF ‘too many lookups” Error

So, what does all of this mean?

It means that we need to make sure all of our clients that send mail via an ISP SMTP server, third party sanitation and continuity service, or mail hosting service need to have a correct SPF record in place.

As the JangoMail blog post makes clear, we may have to jump through a few hoops to get it right, but get it right we must as our client’s mail is critical to their business.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

A 2 Node Hyper-V Cluster with Clustered Storage Spaces

We are in the process of finishing up a client’s migration from clustered SBS 2011 Standard to our SBS (Small Business Solution) stack solution and the following cluster configuration: The above setup is as follows in order of appearance top to bottom:

• 1U Intel Xeon E3 series server running as PDCe, ISO storage, and other non-essential roles
• 1U single socket Intel Xeon E5 series Hyper-V node
• On-Board i350T4 plus add-in i350T4
• Dual 6Gbps Intel/LSI SAS HBAs
• 1U single socket Intel Xeon E5 series Hyper-V node
• On-Board i350T4 plus add-in i350T4
• Dual 6Gbps Intel /LSI SAS HBAs
• 2U DataON DNS-1640d JBOD
• Connected via dual 6Gbps SAS cables per node

Operating systems across the board for all physical and virtual servers is Windows Server 2012 R2.
Storage sharing and arbitration is handled by clustered Storage Spaces. The above setup has 1.2TB 10K HGST SAS drives (DataON HCL approved) set up in a Storage Spaces 3-way mirror with the standard Space having 2 columns.
The client we deployed this cluster into had a cluster already in place based on Windows Server 2008 R2. They are all of 15-18 seats and value the uptime insurance a cluster gives them as downtime for them is expensive.
Note that the cost of this particular setup based on Intel Server Systems and the DataON JBOD is very reasonable.
Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Warning: Sage 50 2016 Server Manager will Reboot Without Prompting

We were installing the database manager on the backend server for a client.

It started the .NET 4.5.2 FULL installer then suddenly disappeared after a few minutes.

So, keep in mind that this one will need to be done when no one is accessing that backend server! Or for that matter that the user that we are working on setting up or updating will have their machine reboot spontaneously.

Also, Sage released this version as an “update”. We received quite a few calls from our accounting firms when they could no longer connect to their Sage/Simply data on the server.

In the end, it turned out that no prompt was given to the user that the “update” was actually an “upgrade”. 

Yo, Sage! A little warning would be appropriate please.

EDIT: Updated a bit for specifics.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

User Profile Tip: Windows Explorer Favorites

Some of us are big on redirecting most folders to the server.

Some of us have learned the hard way to leave AppData and its contents alone on the local desktop. ;)

By default we redirect Desktop, My Documents/Documents, and Favorites.

We recently did a profile refresh using our Event ID 1511, 1515 Profile loss to TEMP method for one of our clients. Their profile had become hopelessly corrupted.

Now, to date we’ve not encountered too many folks that avidly use the Windows Explorer Favorites (pinning):

The above is a snip of my own Windows Explorer pins.

Okay, so we don’t redirect that folder and we’ve not really had to migrate those links before.

That begged the question: Where the chicken are they?!?

Our search foo both on the local machine, via AppData where we thought they should be, and via Internet turned up nothing but one clue: %UserProfile%\Links.

Bingo!

We copied the files from the UserProfile–OLD folder into their new UserProfile\Links folder and they were happy to have them back.

We’ve since added this step to our checklist and will pay a bit more attention to our client’s environments to see if we need to redirect %UserProfile%\Links to save on some time later on.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

A Brief on Storage Spaces

This is a repost of a comment made on an Experts Exchange question.

***

Storage Spaces (SS) is pretty unique. It's driven by Microsoft's need to run data centres full of storage but not foot the bill for huge SAN arrays.
There is a Windows Server Catalogue page of approved hardware for a SS solution.
Our preference is for Quanta and DataON for non-Tier 1 solutions. We've done a lot of testing, and have deployed to on-premises and data centre based clients, prior to being confident in our solution set. That get's expensive _fast_ as we do not deploy anything we've not tested first.
As can be seen, Dell has their products on the SS list and Microsoft chose them for their Cloud Platform System.
A lot of planning and foresight has gone into SS especially with the upcoming Windows Server 2016 feature set (should they all make it into RTM). There are a lot of big storage vendor aimed features that will allow us to deploy SS solutions sets at a fraction of the $/GB cost of the big box vendors.
As an FYI we have a solution set for IaaS vendors that has been in production for close to two years now that works flawlessly. Backend is 10GbE to start with 40Gb SMB Direct (RDMA) and 56Gb SMB Direct (RDMA) over InfiniBand as options.
The solution can scale from 60 drives (4TB, 6TB, or 8GB NearLine SAS) in one 60-bay JBOD to three JBODs, four, or more. With three or more we get enclosure resilience. A full enclosure of drives can fail and SS keeps moving along until that enclosure is brought back up or replaced.
Storage Spaces' cost per IOP, cost per GB/Second (throughput), and cost per GB can't be compared.
Check out Storage Spaces Direct (S2D). Our v2 data centre product will be based on S2D with an all flash option providing _millions_ of IOPS to tenants. Storage fabric via RoCE while storage to compute would be RDMA via InfiniBand.

***

Further Reading:

• TechNet: Storage Spaces FAQ
• TechNet: Storage Spaces: Designing for Performance
• MPECS Inc. Blog Category: Scale-Out File Services (SOFS)
• MPECS Inc. Blog Category: Storage Spaces

There is a very strong economic motivation to get Storage Spaces right for Microsoft and for us. We’ve staked our company direction on Microsoft’s direction with storage while Microsoft’s driver is reducing the overall cost of storage in their Azure data centres.
We believe our Cloud Services Provider data centre backend products are some of the best available and Storage Spaces is a critical piece of the puzzle.
Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Third Tier: Be the Cloud – Client Facing Environment Demo & Chat

Tomorrow at 1700MST is my regular monthly Third Tier chat.

Since we’ve spent a fair amount of time on our Be the Cloud product (IaaS to your clients) backend setup, now it’s time to have a look at what the BtC clients would be working with.

The client environment is based on our SBS (Small Business Solution) that provides a Small Business Server like IT experience but greatly improved.

• Remote Web Portal access
• Remote desktop access
• Exchange services (OWA, EAS, OA, Public Folders, ETC)
• RemoteApp based application access
• SharePoint CMS for Check Out/In and Versioning in Documentation

Our goal in designing our SBS was to give our clients as close to a Small Business Server experience as possible.

I believe we’ve done that in spades and after our chat tomorrow I believe you’ll agree. :)

When: 1700MST

Where: Third Tier Join.me

A recording of the chat, barring any technical difficulties, would be posted on the Third Tier blog at a later date.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book