Exchange 2013+: Set Up a Receive Connector for MFP/Copier/Device Relay

The following are the two steps required to enable an internal anonymous relay in Exchange 2013/2016/20*.

Step 1: Create the Receive Connector

New-ReceiveConnector –Name MFP-APP-AnonRelay –Usage Custom –Bindings 0.0.0.0:25 –RemoteIPRanges 192.168.25.1-192.168.25.10,192.168.25.225-192.168.25.254 –Comment “Allows anonymous relay” –TransportRole FrontEndTransport –AuthMechanism None –PermissionGroups AnonymousUsers

Variables:

• -Name: Change this if needed but must match for both steps
• -RemoteIPRanges: Only put trusted device IP addresses in this section

Once the receive connector is set up it can be managed via EAC.

Step 2: Allow Anonymous Rights

Get-ReceiveConnector “MFP-APP-AnonRelay” | Add-ADPermission -User “NT AUTHORITY\ANONYMOUS LOGON” -ExtendedRights “Ms-Exch-SMTP-Accept-Any-Recipient”

Variable:

• The Receive Connector name must match the one set in Step 1

Conclusion

Once the above steps are set up there is no need to set a username and password on any device that has an allowed IP.

For obvious reasons one should never put an Internet IP address in this rule! But, that being said, one always denies all SMTP 25/587 inbound traffic to a third party sanitation provider's subnets right (we use ExchangeDefender for our own and our client's needs)?

Also, this setup is for on-premises Exchange.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Web Site
Our Cloud Service

Exchange: ERROR: The internal transport certificate cannot be removed... FIX

We recently renewed an Exchange server's trusted certificate.

When we went to remove the old certificate in EAC we received the following error:

error
A special Rpc error occurs on server SERVERNAME: The internal transport certificate cannot be removed because that would cause the Microsoft Exchange Transport service to stop.
To replace the internal transport certificate, create a new certificate. The new certificate will automatically become the internal transport certificate. you can then remove the existing certificate.

Searching turned up a lot of suggestions to just delete the old certificate in the Personal certificates store. Somehow, that did not strike as being the correct methodology since the error makes it clear that the old certificate is still in use.

The proper methodology is to run the following PowerShell in the Exchange Shell to create and bind a new self-issued certificate. Since the certificate is bound to internal services there are no trust issues as indicated by the error message.

New-ExchangeCertificate -IncludeServerFQDN -IncludeServerNetBIOSName -Confirm:$False

The result would be something like this:

Once the command has completed we were able to delete the expired third party certificate in EAC.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Some Disaster Recovery Planning On-Premises, Hybrid, and Cloud Thoughts

This was a post to the SBS2K Yahoo list in response to a comment about the risks of encrypting all of our domain controllers (which we have been moving towards for a year or two now). It’s been tweaked for this blog post.

***

We’ve been moving to 100% encryption in all of our standalone and cluster settings.

Encrypting a setup does not change _anything_ as far as Disaster Recovery Plans go. Nothing. Period.

The “something can go wrong there” attitude should apply to everything from on-premises storage (we’ve been working with a firm that had Gigabytes/Terabytes of data lost due to the previous MSP’s failures) and services to Cloud resident data and services.

No stone should be left unturned when it comes to backing up data and Disaster Recovery Planning. None. Nada. Zippo. Zilch.

The new paradigm from Microsoft and others has migrated to “Hybrid” … for the moment. Do we have a backup of the cloud data and services? Is that backup air-gapped?

Google lost over 150K mailboxes a number of years back, we worked with one panicked call who lost everything, with no return. What happens then?

Recently, a UK VPS provider had a serious crash and, as it turns out lost _a lot_ of data. Where are their clients now? Where’s their client’s business after such a catastrophic loss?

Some on-premises versus cloud based backup experiences:

• Veeam/ShadowProtect On-Premises: Air-gapped (no user access to avoid *Locker problems), encrypted, off-site rotated, and high performance recovery = Great.
• Full recovery from the Cloud = Dismal.
• Partial recovery of large files/numerous files/folders from the Cloud = Dismal.
• Garbage In = Garbage Out = Cloud backup gets the botched bits in a *Locker event.
  • Image based backups are especially vulnerable if bit rot, unrecognized sector failures, and such happen
  • The backup needs to be protected from user access (blog post)
• Cloud provider’s DC goes down = What then?
• Cloud provider’s Services hit a wall and failover fails = What then (this was a part of Google’s earlier mentioned problem me thinks)?
  • ***Remember, we’re talking Data Centers on a grand scale where failover testing has been done?!?***
• At Scale:
• Cloud/Mail/Services providers rely on a myriad of systems to provide resilience
  • Most Cloud providers rely on those systems to keep things going
• Backups?
• Static, air-gapped backups?
• “Off-Site” backups?
• These do not, IMO, exist at scale
• The BIG question: Does the Cloud service provider have a built-in backup facility?
• Back up the data to local drive or NAS either manually or via schedule
• Offer a virtual machine backup off their cloud service

There is an assumption, and we all know what that means right?, that seems to be prevalent among top tier cloud providers that their resiliency systems will be enough to protect them from that next big bang. But, has it? We seem to already have examples of the “not”.

In conclusion to this rather long winded post I can say this: It is up to us, our client’s trusted advisors, to make bl**dy well sure our client’s data and services are properly protected and that a down-to-earth backup exists of their cloud services/data.

We really don’t enjoy being on the other end of a phone call “OMG, my data’s gone, the service is offline, and I can’t get anywhere without it!” :(

Oh, and BTW, our SBS 2003/2008/2011 Standard/Premium sites all had 100% Uptime across YEARS of service. :P

We did have one exception in there due to an inability to cool the server closet as the A/C panel was full. Plus, the building’s HVAC had a bunch of open primary push ports (hot in winter cold in summer) above the ceiling tiles which is where the return air is supposed to happen. In the winter the server closet would hit +40C for long periods of time as the heat would settle into that area. ShadowProtect played a huge role in keeping this firm going plus technology changes over server refreshes helped (cooler running processors and our move to SAS drives).

*** 

Some further thoughts and references in addition to the above forum post.

• May 13, 2016: The Register: Salesforce.com crash caused DATA LOSS
• April 18, 2016: BBC: Web host 123-reg deletes sites in clean-up error
• The definition of irony:
•  
• September 20, 2015: The Register: AWS outage knocks Amazon, Netflix, Tinder and IMDb in MEGA data collapse
• November 2014: Microsoft Azure had a series of outages some lasting days. Fortunately, no data loss happened that we know of though VMs were offline
• Bing Search: Azure Outage
• Note results for earlier problems in April 2014
• August 6, 2012 WIRED: HOW APLE AND AMAZON SECURITY FLAWS LED TO MY EPIC HACKING
• Mat Hanon’s horrific story of complete data loss
• Please read this and then secure a password vault and use it to manage _different_ passwords for _ALL_ sites and client admins!
• We have 2FA enabled on every service that has it available
• We use KeePass for our vault
• February 27, 2011: endgadget: Gmail Accidentally resetting accounts, years of correspondence vanish into the cloud?
• As mentioned, we had frontline experience with this Google data loss situation. :(
• Note that threads around this issue seemingly stopped being updated with no further status reports by Google or others

The moral of this story is quite simple. Make sure _all_ data is backed up and air-gapped. Period.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Hyper-V and Cluster Important: A Proper Time Setup

We’ve been deploying DCs into virtual settings since 2008 RTM. There was not a lot of information on virtualizing anything back then. :S

In a domain setting having time properly synchronized is critical. In the physical world the OS has the on board CMOS timer to keep itself in check along with the occasional poll of ntp.org servers (we use specific sets for Canada, US, EU/UK, and other areas).

In a virtual setting one needs to make sure that time sync between host and guest PDCe/DC time authority is disabled. The PDCe needs to get its time from an outside, and accurate, source. The caveat with a virtual DC is that it no longer has a physical connection with the local CMOS clock.

What does this mean? We’ve seen high load standalone and clustered guests have their time skew before our eyes. It’s not as much of a problem in 2012 R2 as it was in 2008 RTM/R2 but time related problems still happen.

This is an older post but outlines our dilemma: MPECS Inc. Blog: Hyper-V: Preparing A High Load VM For Time Skew.

This is the method we use to set up our PDCe as authority and all other DCs as slaves: Hyper-V VM: Set Up PDCe NTP Time Server plus other DC’s time service.

In a cluster setting we _always_ deploy a physical DC as PDCe: MPECS Inc. Blog: Cluster: Why We Always Deploy a Physical DC in a Cluster Setting. The extra cost is 1U and a very minimal server to keep the time and have a starting place if something does go awry.

In higher load settings where time gets skewed scripting the time sync with a time server within the guest DC to happen more frequently means the time server will probably send a Kiss-O-Death packet (blog post). When that happens the PDCe will move on through its list of time servers until there are no more. Then things start breaking and clusters in the Windows world start stalling or failing.

As an FYI: A number of years ago we had a client call us to ask why things were wonky with the time and some services seemed to be offline. To the VMs everything seemed to be in order but their time was whacked as was the cluster node’s time.

After digging in and bringing things back online by correcting the time on the physical DC, the cluster nodes, and the VMs everything was okay.

When the owner asked why things went wonky the only explanation I had was that something in the time source system must have gone bad which subsequently threw everything out on the domain.

They indicated that a number of users had complained about their phone’s time being whacked that morning too. Putting two and two together there must have been a glitch in the time system providing time to our client site and the phone provider’s systems. At least, that’s the closest we could come to a reason for the time mysteriously going out on two disparate systems.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Outlook Crashes - Exchange 2013 or Exchange 2016 Backend

We’ve been deploying Office/Outlook 2016 and Exchange 2016 CU1 in our Small Business Solution (SBS) both on-premises and in our Cloud.

Since we’re deploying Exchange with a single common name certificate we are using the _autodiscover._tcp.domain.com method for AutoDiscover.

A lot of our searching turned up “problems” around AutoDiscover but pretty much all of them were red herrings.

It turns out the Microsoft has deprecated the RPC over HTTPS setup in Outlook 2016. What does this mean?

MAPI over HTTPS is the go-to for Outlook communication with Exchange going forward.

Well, guess what?

MAPI over HTTPS is _disabled_ out of the box!

In Exchange PowerShell check on the service’s status:

Get-OrganizationConfig | fl *mapi*

To enable:

Set-OrganizationConfig -MapiHttpEnabled $true

Then, we need to set the virtual directory configuration:

Get-MapiVirtualDirectory -Server EXCHANGESERVER | Set-MapiVirtualDirectory -InternalUrl  https://mail.DOMAIN.net/MAPI -ExternalUrl https://mail.DOMAIN.net/MAPI

Verify the settings took:

Get-MapiVirtualDirectory -Server EXCHANGESERVER | FL InternalUrl,ExternalUrl

And finally, test the setup:

Test-OutlookConnectivity -RunFromServerId EXCHANGESERVER -ProbeIdentity OutlookMapiHttpSelfTestProbe

EXCHANGESERVER needs to be changed to the Exchange server name.

Hat Tip: Mark Gossa: Exchange 2013 and Exchange 2016 MAPI over HTTP

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Remote Desktop Services 301: Some Advanced RDS Setup Guidelines

Here are some of the key tenants we’ve picked up deploying standalone and farm based Remote Desktop Services (RDS) for our Small Business Solution (SBS) on-premises and in our cloud.

Virtual Desktop Infrastructure, or VDI for short, covers both Remote Desktop Services standalone, farm, and desktop operating system deployments.

This list, while not totally comprehensive, covers a lot of ground.

• Hardware, Storage, and Networking
• GHz and Cores are king
• Balance core count and GHz versus cost
• NOTE: Server 2016 licensing will be, and is as of this writing, based on core count!
• Today is 2 sockets tomorrow is a server total of 16 cores
• Additional Cores purchased in pairs
• Example 1: Dual Socket 8 Core pair = 16 Cores total so OK in current and 2016 licensing
• Example 2: Dual Socket 12 Core pair = 24 Cores total so base license of 16 Cores plus a purchase of 4 licenses (2 cores per license) would be required
• NOTE: Examples may not line up with actual license terms! Please verify when 2016 goes live.
• RAM is cheap so load up now versus later
• Balanced RAM setup is better than Unbalanced
• Balanced: 4x 16GB in primary memory channel slot
• Best performance
• Unbalanced: 4x 16GB in primary and 4x 8GB in secondary
• Performance hit
• ~500MB of RAM per user per session to start
• 25-50 IOPS per user depending on workloads/workflows RDS/VDI
• Average 2.5” 10K SAS is ~250 to 400 IOPS depending on stack format (stripe/block sizes)
• Latency kills
• Direct Attached SAS or Hyper-Converged is best
• Lots of small reads/writes
• Average 16bpp RDS session single monitor 23” or 24” wide: ~95KB/Second
• Average dual monitor ~150KB/Second
• Bandwidth use is reduced substantially with a newer OS serving and connecting remotely (RDP version)
• LoBs (Line of Business applications)
• QuickBooks, Chrome, Firefox, and Sage are huge performance hogs in RDSH
• Be mindful of LoB requirements and provision wisely
• Keep the LoB database/server backend somewhere else
• In larger settings dedicate an RDSH and RemoteApp to the resource hog LoB(s)
• User Profile Disks
• Office 2013 and Exchange 2013 are a wash in this setting
• Search is difficult if not broken
• Search Index database will bloat and fill the system disk! (Blog post with “fix”)
• Office 2016, though still a bit buggy as of this writing, and Exchange 2016 address UPDs and search
• Be mindful of network fabrics between UPDs and RDSH(s)
• Set the UPD initial size a lot larger than needed as it can’t be changed later without a series of manual steps
• UPDs are dynamic
• Keep storage limits in mind because of this
• Printing
• Printers with PCL 5/6 engines built-in are preferred
• Host-based printers are a no-go for us
• HP Professional series LaserJet printers are our go-to
• HP MFPs are preferred over Copiers
• Copier engines tend to be hacked into Windows while the HP MFP is built as a printer out of the box

Previous post on the matter: Some Remote Desktop Session Host Guidelines.

Here’s a snippet of Intel’s current Intel Xeon Processor E5-2600v4 line sorted by base frequency in GHz:

Depending on the deployment type we’re deploying either E5-2643v4 or E5-2667v4 processors for higher density setups at this time. We are keeping at or under eight cores per socket unless we absolutely require more due to the upcoming sockets to cores changes in Windows Server licensing.

If you’d like a copy of that spreadsheet ping us or visit the Intel Ark site.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud

Some Thoughts On Security Layering for SMB and SME

We are by no means masters of security for our SMB and SME clients. Since we have to wear many hats we sometimes need to bring folks in that can help us to fine tune the security layers in our client’s networks.

Here are some of the Pearls (blog category) that we have garnered over the years. This was posted originally to the SBS2K Yahoo List and has been modified for this post.

***

Layering is important.

Some examples follow.

Windows Firewall

• Windows Firewall is managed by Group Policy
• All Profiles: ON
• All Profiles: Block ON
• All Profiles: Logging ON
• All Profiles: Pop-Up for new services ON
• DOMAIN Profile: Custom Inbound rule sets for required services beyond the default.
• Private and Public Profile: INBOUND BLOCK ALL
• If data sharing is required then a small and inexpensive NAS should be set up

Mail Sanitation and Continuity

ExchangeDefender (xD), for us, is one of the principle ways we keep bad stuff outside of the network.
Why allow it to hit the edge in the first place? Plus, it eliminates SMTP Auth attacks as the WAN IP is not published via MX among other attacks. Interested? Ping us and we’ll set you up.

Edge (Router)

A solid edge device, we use SonicWALL, with a BLOCK ALWAYS rule for ALL outbound traffic is a key element. Rule sets for outbound traffic are very specific and tailored to a client’s needs.

• Examples:
• DNS queries to non on-premises DNS servers are blocked. All DNS queries must go through the on-premises DCs.
• On-Premises edge only or DCs can have the DNS Forwarders set to DNS filtering services.
• SMTP traffic outbound only from the on-premises Exchange server. Or, local copiers/MFPs to ISP SMTP server IP only
• Inbound is HTTPS via ANY
• SMTP via xD subnets only.
• RDP on ANY port should NEVER be published to the Internet.
• RD Gateway with Network Level Authentication is a must today.
• Any exceptions require a static IP on the source end to allow inbound rule filtering based on IP.
• Look up TSGrinder if not sure why…

Ransomware Protection

Third Tier’s Ransomware Protection Kit is another layer of protection. Everything is contained in this kit to deploy a very tight layer of protection against today’s Ransomware.

Microsoft Office Group Policy Security

Office Group Policy structures with Macros disabled by default, non-local sources blocked, and other security settings for Office files provide another layer.

• This one gives users grief because they need a few extra steps to get to the documents.
• We’ve started requesting that clients have a PDF only policy on their Jobs listing pages and such.

A User Focused Effort

IMNSHO, A/V at the endpoint has become virtually useless today. Things seem to be a lot more targeted on the virus side with ransomware taking over as the big cash cow. We still install A/V on all endpoints. :)

What we are saying, is that the principle portion of the risk of infection comes via the user.

A well trained user means the risk of infection drops substantially.

A user’s browsing habits and link clicking are the two key areas of training we focus on. Sites visited are another.

We suggest to clients that a company policy of allowing browsing for business related tasks only while connected to the company’s network resources. This policy can further reduce exposure.

Part of our training regimen is a somewhat regular e-mail from an outside account to users to test them and challenge them is a good idea every once in a while.

• Link hovering to discover the true destination
• Attached Word doc with *BUZZ WRONG* when opened
• Just because it SAYS it’s “FROM” someone we know doesn’t mean it is!

Backup Protection

Oh, and protect the backup loop (blog post on closing the backup loop)!

BTW, we just heard about another NAS based backup that was ransomware encrypted as a result of the destination folder being open to users.

Anyone, and I mean ANYONE, that has a backup structure, whether NAS or HDD based, that allows users and admins access outside of the backup software username and password setup needs to close that loop NOW. Not on the To Do List, not for tomorrow, not next week, but NOW.

Just in case: Close that Backup Loop Now.

Hyper-V Standalone Setups

One more point of order: In standalone Hyper-V settings leave the host in workgroup mode.

No one on the network should have the admin username and password to that host. No. One. It should be documented somewhere but not public knowledge.

Please feel free to add the layers you use to this post via comments.

Thanks for reading!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Remote Desktop Session Host: The System Partition is Getting Full?

In our on-premises and Cloud Desktop RDSenvironments we’ve discovered a number of different things that cause problems for users in Windows Server 2012 R2, Exchange 2013, and Office 2013.

All of our Remote Desktop Session Hosts (RDSHs) are set up with two VHDX files. One for the operating system and one for the data and User Profile Disks (UPDs).

Unfortunately, while UPDs give us a great flexbility option that allow us to have then on the network thus avoid local profile pains in a RDS Farm setting they have a number of different negative impacts on user experience and RDSH health.

One that impacts both is the mysterious filling up of the system partition.

As it turns out, Outlook 2013 and Exchange 2013 plus UPDs means Outlook search is almost completely broken.

But, that doesn’t stop the Windows Server Search Service from doing its best to catalog everything anyway!

What does that mean?

Well, eventually we have a search database that can grow to epic proportions.

Since all of our OS partitions are rather small we end up with session hosts getting their system partition filled rather quickly on a busy RDSH. This is especially true in a Farm setting.

So, what are our options?

Well, we could disable the Windows Search Service. This would be a bad idea since users wouldn’t be able to find _anything_ anymore. We’d go from the occasional complaint to constant complaints. So, not good.

The alternative is to reset the Windows Search index.

1. Start –> Indexing Options
2. Advanced button (UAC)
3. Click the Rebuild button

And, voila! In some cases we get 45GB to 60GB of space back in short order!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Some Remote Desktop Session Host Guidelines

We’ve put about four years and two versions into our Small Business Solution (SBS). We have it running on-premises on standalone Hyper-V servers as well as on Hyper-V clusters (Clustered Storage Spaces and Hyper-V cluster we just deployed for a 15 seat accounting firm).

It is the foundation for the Cloud Office services we’ve been offering for the last year or so.

Since our Cloud Office solution runs in Remote Desktop Services we figured we’d share some pearls around delivering Remote Desktop Session Host based environments to clients:

• ~512MB/User is cutting it tight
• ~20 to 25 users in a 12GB to 16GB vRAM Hyper-V VM works okay with 2-3 vCPUs
• RDP via 8.1 RDP clients saturates a 1Mb DSL uplink at ~13-15 users depending on workload
• ALL browsers can bring the RDSHs to their knees
• Printing can be a bear to manage (Use Universal Print Drivers and Isolation where possible)
• Group Policy configuration and lockdown is mandatory
• Two partitions with User Profile Disks (UPDs), if used, on the second partition
• NOTE: UPDs + Office 2013 and earlier + Exchange 2013 and earlier = Broken Search!!!
• NOTE: RDSH Search Indexes for Outlook OSTs in UPDs can fill up the C: partition!
• Office 2016 and Exchange 2016 together are supposed to address the broken search situation in RDSH setups were UPDs are used. We have yet to begin testing the two together.

Our Cloud Office (SBS) is running on clusters we’ve designed based on Scale-Out File Server and Hyper-V.

Need a clustered solution for your SMB/SME clients? Drop us a line. They are _very_ affordable. ;)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Script: Clear DNS Cache and Restart the DNS Cache and DNS Services

Here’s a simple CMD (Command Line) script for clearing the DNS Cache and restarting both the service and the DNS service:

DnsCMD SERVERNAME /ClearCache
Net Stop DNSCache && Net Start DNSCache
Net Stop DNS && Net Start DNS
DnsCMD SERVERNAME /ClearCache
Pause

The above script was set up to be run both on a scheduled task and manually on an older SBS 2011 Essentials VM as it seems to have DNS lookup problems.

This script solves the problem.

This server is slated to be refreshed soon.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Exchange Stall: Purging Exchange Logs

We’ve got our SBS (Small Business Solution) set up for client’s all-in-the-Cloud experience.

What we’re finding is that some vendor’s systems don’t trigger VSS within a VM running Exchange thus leaving Exchange thinking it is not being backed up. This means that the Exchange VM eventually stalls due to no space on the partition hosting the logs.

To remedy that run the following in a script say once a week, or more depending on the volume of mail, on the Exchange VM:

1. Elevated CMD
2. DiskShadow
3. Add Volume C:
4. Begin Backup
5. Create
6. End Backup 

Once the snapshot completes Exchange will think it’s been backed up and consolidate all of the logs.

NOTE: Make sure the VM/Server is backed up!

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Folder Permissions: How To Properly Disinherit Permissions

We run into a lot of ACL corruption issues and access issues when a folder has not been disinherited properly.

The following is the best method for disinheriting permissions a folder receives from its parent:

1. Right click on the folder and click Properties
2. Click the Advanced button
3. Click the Change Permissions button if required
4. Click the Disable inheritance button
5. Click the Convert inherited permissions into explicit permissions on this object.
• 
• DO NOT CLICK REMOVE
6. Click on DOMAIN\Domain Users or MACHINE\Users and then the Remove button
• This removes access to that folder to all domain users
7. Add the necessary security groups and give them MOD
8. OPTION: On existing folder sets one can click Replace all child object permission entries with inheritable permission entries from this object
• Does one want to click this? If there are customized permissions _below_ the folder being disinherited those permissions would be lost.
9. Click Apply and OK.

From there our folder would now have the necessary permissions for users in the specific security group(s) to make changes.

We enable Access-based Enumeration on _all_ shares we deploy by default. This means that users that are not in the above assigned security group(s) will not see the folder in their File Explorer.

One of the warning signs that the above process was not followed will be for domain admin or local admin accounts to get a UAC prompt when navigating the physical folder set.

As a rule we follow a trunk –> branch –> leaf structure for our folders. All users have a single point of entry with some subfolders having their inheritance blocked.

From there we prefer to _not_ disinherit any further down-level folders unless absolutely necessary because that inevitably leads to access issues and/or permissions corruption.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

E-mail NDR: #5.1.1 SMTP; 550 Rejecting for Sender Policy Framework (SPF too many lookups)

One of our clients was having issues sending an e-mail to one of our regional ISP’s e-mail servers.

• Remote Server returned ‘<inbound.server.com #5.1.1 SMTP; 550 Rejecting for Sender Policy Framework>’
• SPF too many lookups

There was no further information. The specifics came from a very helpful mail technician at the ISP.

So, we started to dig around and came up with the following:

• Sender Policy Framework Site: OpenSPF.org
• Kitterman Technical Services’ excellent SPF Tool

A ticket went through Third Tier’s Help Desk this week that was based on this problem with Dave Shakelford pointing to the following blog post:

• JangoMail Help Center: SPF ‘too many lookups” Error

So, what does all of this mean?

It means that we need to make sure all of our clients that send mail via an ISP SMTP server, third party sanitation and continuity service, or mail hosting service need to have a correct SPF record in place.

As the JangoMail blog post makes clear, we may have to jump through a few hoops to get it right, but get it right we must as our client’s mail is critical to their business.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

User Profile Tip: Windows Explorer Favorites

Some of us are big on redirecting most folders to the server.

Some of us have learned the hard way to leave AppData and its contents alone on the local desktop. ;)

By default we redirect Desktop, My Documents/Documents, and Favorites.

We recently did a profile refresh using our Event ID 1511, 1515 Profile loss to TEMP method for one of our clients. Their profile had become hopelessly corrupted.

Now, to date we’ve not encountered too many folks that avidly use the Windows Explorer Favorites (pinning):

The above is a snip of my own Windows Explorer pins.

Okay, so we don’t redirect that folder and we’ve not really had to migrate those links before.

That begged the question: Where the chicken are they?!?

Our search foo both on the local machine, via AppData where we thought they should be, and via Internet turned up nothing but one clue: %UserProfile%\Links.

Bingo!

We copied the files from the UserProfile–OLD folder into their new UserProfile\Links folder and they were happy to have them back.

We’ve since added this step to our checklist and will pay a bit more attention to our client’s environments to see if we need to redirect %UserProfile%\Links to save on some time later on.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

SMB Kitchen ASP Author’s Chat: Wednesday the 17th Open To All!

You are invited to a preview during our SMBKitchen Chat on December 17th 7pm eastern. Download to your calendar now!

Our Chat will introduce the following new items at Third Tier. In addition we'll cover our usual content about security and patching. Join us for this special sneak preview of...

Tech Your Books: You know your books are supposed to do more for you than just keep you out of trouble with the IRS right? They are supposed to be a powerful business tool that helps you be more profitable and provide information so you can make evidence backed decisions about your business. It’s time to Tech Your Books!

Look, A Whale! What do you do when you’ve got a Whale? Be it a crisis or a project, your company could benefit greatly if you choose to have a third party representing your organization.

Our First Product: SBS IaaS Ever since Microsoft ceased production of small business server saying that small businesses would prefer to be "in the cloud" IT providers have wondered where that leaves them. Well we have a solution for you - Be The Cloud! 

Super Secret News: ASP members have been enjoying reading classified documents from US intelligence and security agencies giving them the early information on the latest hacks, industry targets and break-out infections. Learn about our after ASP plans to bring this information to you.

In addition to these great new Third Tier services, we are continuing to offer advanced helpdesk and microstaffing.If you would like to read more about each of these, we have some information available on our website already with more to come soon. http://www.thirdtier.net

And ItsA Gonna Be A Gooder! ;)

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

ASP Webinar Today at 1700 MST: Setting up ARR and URL ReWrite for One WAN IP

We've had a number of requests to do a Webinar specifically on how we get things set up for our Small Business Solution (SBS) whereby we only require one WAN IP address for all Internet facing services being published.

At 1700 MST please join me as I walk through a step-by-step of installing and configuring ARR and URL ReWrite.

• Install ARR and URL ReWrite
• Configure ARR settings in IIS
• Configure ARR settings for EAS
• IIS Sites and settings
• SSL configuration
• URL ReWrite rule configuration
• ReWrite rule testing

By the time we are finished we will have a fully functional setup.

For this Webinar we will be using our three VM Azure based lab running a DC, Exchange 2013, and RDS.

Please do join us if you can and if not the event is being recorded and will be available to subscribers in the ASP Knowledgebase.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Microsoft: Excitement Rekindled

Back in the day, heh ;) , I was privileged to work with the SBS team both prior and post MVP award.

Many of us in SBS Land can remember just how active the team was around SBS 2003 RTM/R2, SBS 2008 Standard/Premium, and finally SBS 2011 Standard.

By the latter product release much of the old guard had moved on but the new team and us SBS MVPs had a strong business relationship.

The key here is in the relationships and communication levels we were experiencing with the teams.

For must of us both within the MVP realms but also in the public that came to a full-stop when development of the then Windows Server 2012 pre-release cycle became better known.

We were all held in the dark. This situation continued with the 2012 R2 development cycle.

Silence. Dead Silence *insert sound of crickets here*

For those of us both within the MVP community and outside the MVP community that are truly passionate about the clients we serve, the products we work with, and just all-around geeking out about tech the silence was deafening and deadening.

Tie that into the seemingly endless “Death to the IT Pro” messages coming out of Microsoft over the last number of years and there were a lot of flames snuffed out these last two product release cycles.

Change for the Good

One thing is for certain: Something has changed at Microsoft.

While I most certainly cannot provide “insight” into Microsoft’s motivations for the recent change I can most certainly say that the change is welcome. Big Time!

We now have a public release of Windows vNext bits. We have forums to discuss these bits.

For many of us MVPs we have, and I can’t say “unprecedented” given previous experience, an amazing amount of information flowing from our product teams.

The conversations we’ve been having with the team(s) lately have been refreshing. Yes, they’ve been listening to us.

And, I’ll make no bones about it, I feel very excited about this upcoming product release like I had been with the then new versions of SBS Standard!

So, hat’s off to you in Microsoft whoever you are that allowed the floodgates of communication, shared tech passion, and business relationships to happen again!

And thanks to the Microsoft Teams that have been sharing your product excitement. It is refreshing and invigorating for us to experience and work with!

We both stand to benefit greatly from this “new” openness from Microsoft.

Oh, and Cluster is definitely key to our company’s future. ;)

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Our SBS (Small Business Solution) Options with Standalone and Cluster Hardware Considerations

We’ve received a number of questions about the “How” we present our SBS to prospective and existing clients.

Our primary focus is on what we have provided with Small Business Server starting with SBS 2003 Standard.

• Active Directory permissions based security
• Remote Web Access (RWA/RWW) Portal
• Remote Desktop access via RD Gateway (since SBS 2008 Standard)
• RemoteApp access via RD Gateway for LoBs (since SBS 2008 Standard)
• E-mail services access via Outlook, Outlook Anywhere, Exchange ActiveSync, and Outlook Web Access
• Remote Folders and Files access
• SharePoint based document management system
• SQL backend for LoB, SharePoint, and other needs

We focus on the services the prospect would require while our existing clients are already used to them.

Once we have an understanding of the prospect’s needs, since we already know our client’s business really well, we move forward with a proposal that would be geared towards their business size and sensitivity to downtime.

On the services front where we are installing into a standalone host we would have two options:

1. Base
1. Requires two Windows Server OS Licenses
2. DC, Exchange, RDS, and LoB (WSUS and LoBs)
2. Premium Add-On
1. Requires one Windows Server OS License
2. SQL and SharePoint

Obviously the server and CALs would also be needed for the various components that will be installed into the guest OS.

If we are setting up a cluster then one needs to consider the number of VMs running on one or more of the nodes in the event of a node failure.

On the hardware side we would have a number of options:

1. Entry-Level Single
1. E3-1270v3, 32GB ECC, Hardware RAID, 8x 2.5” 10K SAS
2. Mid-Level Single
1. Single Socket 1U R1208JP4OC, E5-2600 series, 128GB ECC, Hardware RAID, 8x 2.5” 10K SAS
3. High-Level Single
1. Dual Socket 2U R2208GZ4GC, E5-2600 pair, 128GB-256GB ECC, Hardware RAID, 8x 2.5” 10K SAS or 16x 2.5” 10K SAS
4. Entry-Level Asymmetric Cluster
1. Pair of 1U R1208JP4OC or 2U R2208GZ4GC and an Intel JBOD2224S2DP
5. Mid-Level Cluster
1. Four 2U R2208GZ4GC and an Intel JBOD2224S2DP
• Two Scale-Out File Server cluster nodes
• Two Hyper-V cluster nodes
6. High-End Cluster
1. Six 2U R2208GZ4GC and three Intel JBOD2224S2DP units
• Three Scale-Out File server cluster nodes
• Three Intel JBODS with Two-Way or Three-Way Mirror and Enclosure Resilience
• Three Hyper-V server cluster nodes

Within the above hardware configurations we would have a lot of flexibility that allows us to customize to the specific needs of the prospective client or our clients.

We work with a number of different firms that are prime candidates for at least an asymmetric cluster setup to minimize the possibility of downtime. The cost associated with these entry-level clusters versus a single larger server for the host platform makes them very attractive.

The basic VM configuration would involve fixed VHDX files unless the files are installed on dedicated partitions/LUNs. Note that we would use a shared set of partitions/LUNs if there are around 10 or more VMs as things get to be a bit of a bear to manage otherwise.

Our base VM configurations would be as follows:

• DC: 4GB, 95GB OS VHDX, and 1TB Data VHDX
• Exchange: 8GB, 95GB OS VHDX, and 250GB + 20GB/Mailbox Data VHDX
• RDS: 4GB+, 95GB OS VHDX, and 100GB + 20GB/User Profile Disk
• LoB: 8GB, 95GB OS VHDX, and 1TB Data VHDX Minimum
• SQL: 16GB, 95GB OS VHDX, and 250GB+ Data VHDX
• SharePoint: 16GB, 95GB OS VHDX, 200GB Data VHDX

We have a set of PowerShell steps and scripts that we use to configure these environments. PowerShell helps to greatly reduce the amount of time required to set things up. It also gives us consistency across all of our client deployments which is vital to troubleshooting if the need arises.

Shameless Plug: We’ve spent some time on the above in our SMBKitchen ASP Author Chats. If you are looking for more information the Author Chat is one of the best ways to do so.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Third Tier Brain Explosion: Our New SBS!

I will be presenting along with our Third Tier team as a Pre-Day Brain Explosion (BE) for the GFIMax conference in Orlando, Florida on the afternoon of September 7th and the morning of the 8th.

We are splitting things into two days to give folks a chance to both digest all of the content but also to allow BE attendees the chance to meet and chat with us.

Cost for the Pre-Day BE is a very reasonable $99.

Registration for the event can be done on the Third Tier Portal under the Purchase Support option (windowed scroll bar in there that's a bit difficult to see) or on GFI's site (Pre-Day Landing Page).

The content I will be working with will be the technical aspects of delivering our new SBS.

What is that you might ask?

S = Small

B = Business

S = Solution

It is our very complete solution set that we have been deploying to our client sites for about six months to a year now.

• RWW/RWA Replacement
• RDWeb and RDGateway
• Exchange/OWA/EAS
• Exchange 2013 CU4 (SP1)
• SharePoint
• SharePoint Foundation 2013
• Remote Desktop Services
• Remote Desktop Session Host and RemoteApps
• RD Endpoint access

Our last ASP SMB Kitchen subscriber chat was spent walking through SBS and how it is essentially seamless to our clients. One WAN IP is all that is required just as it was with Small Business Server.

The beauty of the solution is the simplicity with which things change for our end users: Virtually not at all. :)

We supply an on-premises solution that gives everything Small Business Server has given them. Plus, we clearly demonstrate that our product is on par with or better than any Cloud based solution out there just as SBS has been to small businesses for the last ten or more years.

Looking for that on-prem replacement? Please subscribe to our ASP Project as our monthly chats and content supplied by our team will facilitate that search.

Want the technical pearls to deploying this kind of solution for your clients? Then please do register for the Brain Explosion and wear a brain bucket! :)

Thanks for reading.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

One Post SBS Configuration

We are doing the following and are quite successful with the setup:

• 2x Windows Server STD
• Windows CALs
• Exchange STD
• Exchange CALs
• RDS CALs

With that we set up one host with Hyper-V (2012 R2 preferred).

• VM 1: DC
• VM 2: Exchange 2013 CU3
• VM 3: RDS
• VM 4: LoB, WSUS

We just finished migrating our last SBS 2003 out to this setup (though with two servers and a few extra licenses).

For larger firms we can set up two identical servers and have licensing in place to allow for the following:

• Server 1 & 2: DC VM with DHCP Failover enabled (new 2012 R2 feature)
• Server 1: LoB VM with Replica to Server 2
• Server 1: Exchange
• Server 1: RDS VM with Replica to Server 2

Because Exchange and SQL have their own built-in redundancy features we have the option to configure in-guest clustering to build out the required redundancy for them.

Or, we can go with two servers with dual SAS HBAs and a dual controller SAS direct attached storage (MD3220, VTrak E610sD, DS3524) and set up an actual Hyper-V Failover Cluster. This option works very well for the very downtime conscious client.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
Third Tier: Enterprise Solutions for Small Business