Security: A Sample E-mail How-To Guide For End Users

With the plethora of e-mail born Office documents with active macros in them to pull down malware/ransomware we sent out the following e-mail to all of our clients for distribution internally.

***

Good day everyone,

It’s gotten to the point now where we are considering a universal restriction on incoming Office Documents. By that we mean plucking them right out of the e-mail via ExchangeDefender by default.

We have somehow travelled back to the 1990s where the bad guys are setting up Office documents with a Macro, an automatic script that runs when the document gets opened, that goes on to pull down their nefarious malware or ransomware.

Here are some steps to help protect us:

1. Microsoft Office has a Save As PDF feature built-in. Please have all outside folks send a PDF instead of an Office document
1. This is especially critical for Resumes. All job postings _must_ request PDF and note that Office documents would be deleted on the spot!
2. If collaboration is required for Office documents use ShareFile
3. Preferred over Dropbox since security is questionable with the Dropbox service
2. Most Office documents that have Macros built-in have an “m” in the extension
1. 
2. Save the Office document to Downloads and verify!
3. If extensions are not shown then right click the file and left click on Properties
4. 
3. Users _should_ be prompted:
1. 
4. Obviously, the answer should be to NOT click that button
5. If they do, there is one last cause for pause
1. 
6. This is what happens if I try and click on something that is Macro driven _before_ clicking Enable Content
1. 

Along with the need to be mindful of any Microsoft Office attachments in our e-mail we should also remember the following:

1. Never click on a link in an e-mail without at the least verifying its destination:
1. 
2. Hover the mouse cursor over the link to verify
3. As a rule: Never, ever, click on a link in an e-mail. Go to the web site after opening a new browser window (IE, Firefox, Chrome, Safari)
2. It may _look_ like it came from someone you know but never trust that. Call and ask!
1. There are a few exceptions to this rule thus make sure to hover your mouse over the link before clicking!
2. Advanced users can check the headers
1. 
2. 
3. Follow the flow from origin server to destination server
3. Don’t save important site’s information in the browser
1. Banking IDs and passwords
2. CRA and critical site’s IDs and passwords
3. Do not disable the secondary question for any computer
1. Banking sites use this feature to help protect the account as one example
2. Answer the question, it only takes a couple seconds and could save your savings!
4. Never call the 800 number that comes up in a Search for Support!
1. Go to the manufacturer’s web site and click on the Support link to find the correct phone number
5. Never believe a pop-up message that says your computer is infected with something!
1. And never, EVER, call the 800 number on that pop-up!
2. Don’t click anywhere, close and save your work if needed then, reboot!
3. Do NOT click anywhere in the pop-up window. Looks are deceiving as all areas of that pop-up = YES/ACCEPT/CONTINUE
6. Never volunteer a credit card number or banking information to anyone
1. Social Security/Social Insurance Numbers too!
2. Folks can garner a lot about us online. Never volunteer any information when asked via any incoming call/e-mail/forum
3. Always call them back!
7. Caller says they are from the bank, CRA, or other seemingly critical business?
1. Ask for their badge number, an 800 number to call, and an extension
2. Open a browser and verify the 800 number belongs to the bank/CRA/CritBiz.
3. Then call them back after hanging up if the number proves true!

While the above list is far from complete, by following these guidelines we can greatly reduce the chances of a malware or ransomware infection.

And, as always, e-mail or call if you are not sure about something!

***

Please feel free to use this as a template for training users!

Have a great weekend everyone. It’s +10C here and much like an awesome Spring day!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Some Thoughts On Security Layering for SMB and SME

We are by no means masters of security for our SMB and SME clients. Since we have to wear many hats we sometimes need to bring folks in that can help us to fine tune the security layers in our client’s networks.

Here are some of the Pearls (blog category) that we have garnered over the years. This was posted originally to the SBS2K Yahoo List and has been modified for this post.

***

Layering is important.

Some examples follow.

Windows Firewall

• Windows Firewall is managed by Group Policy
• All Profiles: ON
• All Profiles: Block ON
• All Profiles: Logging ON
• All Profiles: Pop-Up for new services ON
• DOMAIN Profile: Custom Inbound rule sets for required services beyond the default.
• Private and Public Profile: INBOUND BLOCK ALL
• If data sharing is required then a small and inexpensive NAS should be set up

Mail Sanitation and Continuity

ExchangeDefender (xD), for us, is one of the principle ways we keep bad stuff outside of the network.
Why allow it to hit the edge in the first place? Plus, it eliminates SMTP Auth attacks as the WAN IP is not published via MX among other attacks. Interested? Ping us and we’ll set you up.

Edge (Router)

A solid edge device, we use SonicWALL, with a BLOCK ALWAYS rule for ALL outbound traffic is a key element. Rule sets for outbound traffic are very specific and tailored to a client’s needs.

• Examples:
• DNS queries to non on-premises DNS servers are blocked. All DNS queries must go through the on-premises DCs.
• On-Premises edge only or DCs can have the DNS Forwarders set to DNS filtering services.
• SMTP traffic outbound only from the on-premises Exchange server. Or, local copiers/MFPs to ISP SMTP server IP only
• Inbound is HTTPS via ANY
• SMTP via xD subnets only.
• RDP on ANY port should NEVER be published to the Internet.
• RD Gateway with Network Level Authentication is a must today.
• Any exceptions require a static IP on the source end to allow inbound rule filtering based on IP.
• Look up TSGrinder if not sure why…

Ransomware Protection

Third Tier’s Ransomware Protection Kit is another layer of protection. Everything is contained in this kit to deploy a very tight layer of protection against today’s Ransomware.

Microsoft Office Group Policy Security

Office Group Policy structures with Macros disabled by default, non-local sources blocked, and other security settings for Office files provide another layer.

• This one gives users grief because they need a few extra steps to get to the documents.
• We’ve started requesting that clients have a PDF only policy on their Jobs listing pages and such.

A User Focused Effort

IMNSHO, A/V at the endpoint has become virtually useless today. Things seem to be a lot more targeted on the virus side with ransomware taking over as the big cash cow. We still install A/V on all endpoints. :)

What we are saying, is that the principle portion of the risk of infection comes via the user.

A well trained user means the risk of infection drops substantially.

A user’s browsing habits and link clicking are the two key areas of training we focus on. Sites visited are another.

We suggest to clients that a company policy of allowing browsing for business related tasks only while connected to the company’s network resources. This policy can further reduce exposure.

Part of our training regimen is a somewhat regular e-mail from an outside account to users to test them and challenge them is a good idea every once in a while.

• Link hovering to discover the true destination
• Attached Word doc with *BUZZ WRONG* when opened
• Just because it SAYS it’s “FROM” someone we know doesn’t mean it is!

Backup Protection

Oh, and protect the backup loop (blog post on closing the backup loop)!

BTW, we just heard about another NAS based backup that was ransomware encrypted as a result of the destination folder being open to users.

Anyone, and I mean ANYONE, that has a backup structure, whether NAS or HDD based, that allows users and admins access outside of the backup software username and password setup needs to close that loop NOW. Not on the To Do List, not for tomorrow, not next week, but NOW.

Just in case: Close that Backup Loop Now.

Hyper-V Standalone Setups

One more point of order: In standalone Hyper-V settings leave the host in workgroup mode.

No one on the network should have the admin username and password to that host. No. One. It should be documented somewhere but not public knowledge.

Please feel free to add the layers you use to this post via comments.

Thanks for reading!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Cluster 101: Some Hyper-V and SOFS Cluster Basics

Our focus here at MPECS Inc. has grown into providing cluster-based solutions to clients near and far over the last eight years or so as well as cluster infrastructure solutions for small to medium I.T. shops.

There were so many misconceptions when we started the process to build out our first Hyper-V cluster in 2008.

The call in to us was for a large food manufacturing company that had a very specific requirement for their SQL, ColdFusion, and mail workloads to be available. The platform of choice was the Intel Modular Server with an attached Promise VTrak E310sD for extra storage.

So, off we went.

We procured all of the hardware through the Intel and Promise demo program. There was _no_ way we were going to purchase close to $100K of hardware on our own!

Back then, there was a dearth of documentation … though that hasn’t changed all that much! ;)

It took six months of trial and error plus working with the Intel, Promise, LSI, and finally contacts at Microsoft to figure out the right recipe for standing up a Hyper-V cluster.

Once we had everything down we deployed the Intel Modular Server with three nodes and the Promise VTrak E310sD for extra storage.

Node Failure

One of the first discoveries: A cluster setup does not mean the workload stays up if the node it’s on spontaneously combusts!

What does that mean? It means that when a node suddenly goes offline because of a hardware failure the guest virtual machines get moved over to an available node in a powered off state.

To the guest OS it is as if someone hit the reset button on the front of a physical server. And, as anyone that has experienced a failed node knows the first prompt when logging in to the VM is the “What caused the spontaneous restart” prompt.

Shared Storage

Every node in a Hyper-V cluster needs identical access to the storage the VHD(x) files are going to reside on.

In the early days, there really was not a lot of information indicating exactly what this meant. Especially since we decided right from day one to avoid any possible solution set based on iSCSI. Direct Attached Storage (DAS) via SAS was the way we were going to run with. The bandwidth was vastly superior with virtually no latency. No other shared storage in a cluster setting could match the numbers. And, to this day the other options still can’t match DAS based SAS solutions.

It took some time to figure out, but in the end we needed a Shared Storage License (SharedLUNKey) for the Intel Modular Server setup and a storage shelf with the needed LUN Sharing and/or LUN Masking plus LUN Sharing depending on our needs.

We had our first Hyper-V cluster!

Storage Spaces

When Storage Spaces came along in 2012 RTM we decided to venture into Clustered Storage Spaces via 2 nodes and a shared JBOD. That process took about two to three months to figure out.

Our least expensive cluster option based on this setup (blog post) is deployed at a 15 seat accounting firm. The cost versus the highly available workloads benefit ratio is really attractive. :)

We have also ventured into providing backend storage via Scale-Out File Server clusters for Hyper-V cluster frontends. Fabric between the two starts with 10GbE and SMB Multichannel.

Networking

All Broadcom and vendor rebranded Broadcom NICs require VMQ disabled for each network port!

A best practice for setting up each node is to have a minimum of four ports available. Two for the management network and Live Migration network and two for the virtual switch team. Our preference is for a pair of Intel Server Adapter i350-T4s set up as follows:

• Port 0: Management Team (both NICs)
• Port 1 and 2: vSwitch (no host OS access both NICs)
• Port 3: Live Migration networks (LM0 and LM1)

For higher end setups, we install at least one Intel Server Adapter X540-T2 to bind our Live Migration network to each port. In a two node clustered Storage Spaces setting the 10GbE ports are direct connected.

Enabling Jumbo Frames is mandatory for any network switch and NIC carrying storage I/O or Live Migration.

Hardware

In our experience GHz is king over cores.

The maximum amount of memory per socket/NUMA node that can be afforded should be installed.

All components that can be should be run in pairs to eliminate as many single points of failure (SPFs) as is possible.

• Two NICs for the networking setup
• Two 10GbE NICs at the minimum for storage access (Hyper-V <—> SOFS),
• Two SAS HBAs per SOFS node
• Two power supplies per node

On the Scale-Out File Server cluster and Clustered Storage Spaces side of things one could scale up the number of JBODs to provide enclosure resilience thus protecting against a failed JBOD.

The new DataON DNS-2670 70-bay JBOD supports eight SAS ports per controller for a total of 16 SAS ports. This would allow us to scale out to eight SOFS nodes and eight JBODs using two pairs of LSI 9300-16e (PCIe 8x)  or the higher performance LSI 9302-16e (PCIe 16x) SAS HBAs per node! Would we do it? Probably not. Three or four SOFS nodes would be more than enough to serve the eight direct attached JBODs. ;)

Know Your Workloads

And finally, _know your workloads_!

Never, ever, rely on a vendor for performance data on their LoB or database backend. Always make a point of watching, scanning, and testing an already in-place solution set for performance metrics or the lack thereof. And, once baselines have been established in testing the results remain private to us.

The two key ingredients in any standalone or cluster virtualization setting are:

1. IOPS
2. Threads
3. Memory

A balance must be struck between those three relative to the budget involved. It is our job to make sure our solution meets the workload requirements that have been placed before us.

Conclusion

We’ve seen a progression in the technologies we are using to deploy highly available virtualization and storage solutions.

While the technology does indeed change over time the above guidelines have stuck with us since the beginning.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Philip’s Ultra-Healthy and Quick Technician’s Breakfast!

We’re all super busy. Eating breakfast is an important part of it with a home grown meal being way better than most anything a fast food place can serve. It’s a lot less expensive in the long run plus the time savings is huge!

This breakfast meal assumes the person is in some sort of regular excercise routine which is also an important part of keeping ourselves healthy. Right? ;)

• Breakfast Sloppy Toast
• (3) Large Eggs
• ~125ml to ~200ml of Half & Half Cream
• (2) Whole Grain, 12 Grain, or other such solid bread
• 1/8” slices of Cheddar, Marble, Havarti, Mozza, or other favourite cheese
• A good chunk of baby spinach
• A proper Pyrex microwave dish and cover
• Plastic containers melt into the food :P
• Ours is just larger WxL wise than the bread slizes and tall enough to host the lot

With the above:

1. Break the 3 eggs into the Pyrex dish
2. Start whisking
3. Add cream until well frothed
4. Place first slize of bread in the mix
5. Cover the bread with the cheese slices
6. Drop spinach in and evenly distribute
7. Place second slice of bread in
8. Use a spat to flip the stack over
9. Press in to allow mix to soak into the new slice
10. Cover and microwave for 5:15 at 60%
1. Let sit for about a minute after the cycle completes
11. Microwave for about 1:45 to 2:45 at 60% depending on microwave power

Once it’s done let it cool off for a good five minutes.

Total time put in to the above: Less than 4 minutes.

Time savings over the week?

Assuming a minimum 15-20 minute wait at Timmys (Tim Hortons) that’s easily 15 minutes per day or more.

Cost savings?

Two breakfast egg sandwiches with cheese and bacon on an English muffin is $6.20. The savings can be quite substantial.

I usually have a couple of Vietnamese bird peppers to chow on while eating the above to accentuate the flavour. ;)

While this breakfast is not for everyone, it follows a 50/25/25 rule for protien/fat/carbs. Tie that in to a good regular cardio workout we’re good to go!

Thanks for reading. :)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Hyper-V 101: What Windows Server Media Should I Use?

This may seem like a bit of a silly N00b style post but there’s a good reason for it.

How many of us are using Windows Server Media to install hosts via USB Flash then guests via ISO?

I venture to guess almost all of us.

Okay, POLL Time: What is the _date stamp_ on the Setup.EXE located on that flash/ISO?

As of today, if it’s a date earlier than November 22, 2014 then it’s _too old_ to be used in production systems:

Please log on to the Microsoft Volume Licensing Service Centre, MSDN, or TechNet to download a newer ISO.

Then update the flash drives used to install Hyper-V hosts and nodes.

It should be Standard Order of Procedure (SOP) to keep operating system load souces up to date.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Remote Desktop Session Host: The System Partition is Getting Full?

In our on-premises and Cloud Desktop RDSenvironments we’ve discovered a number of different things that cause problems for users in Windows Server 2012 R2, Exchange 2013, and Office 2013.

All of our Remote Desktop Session Hosts (RDSHs) are set up with two VHDX files. One for the operating system and one for the data and User Profile Disks (UPDs).

Unfortunately, while UPDs give us a great flexbility option that allow us to have then on the network thus avoid local profile pains in a RDS Farm setting they have a number of different negative impacts on user experience and RDSH health.

One that impacts both is the mysterious filling up of the system partition.

As it turns out, Outlook 2013 and Exchange 2013 plus UPDs means Outlook search is almost completely broken.

But, that doesn’t stop the Windows Server Search Service from doing its best to catalog everything anyway!

What does that mean?

Well, eventually we have a search database that can grow to epic proportions.

Since all of our OS partitions are rather small we end up with session hosts getting their system partition filled rather quickly on a busy RDSH. This is especially true in a Farm setting.

So, what are our options?

Well, we could disable the Windows Search Service. This would be a bad idea since users wouldn’t be able to find _anything_ anymore. We’d go from the occasional complaint to constant complaints. So, not good.

The alternative is to reset the Windows Search index.

1. Start –> Indexing Options
2. Advanced button (UAC)
3. Click the Rebuild button

And, voila! In some cases we get 45GB to 60GB of space back in short order!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Hyper-V Virtualization 101: Some Basics Around Hyper-V Setups

Here are some pearls learned over our years working with Hyper-V:

• Host/Node Setup
• Make sure the host and all nodes in a cluster have the BIOS Settings identical (All settings)
• Leave ~1GB to 1.5GB physical RAM to the host
• We leave 1.5GB of space on a dedicated to VM LUN
• We leave ~50GB to ~100GB of free space on a shared LUN/Partition
• We set the MiniDump option and lock the swap file down to 840MB
• wmic RECOVEROS set DebugInfoType = 3
• Always set up a standalone host with two partitions: OS and Data
• Hyper-V
• Hyper-V lays out a file equivalent in size to the vRAM assigned to VMs. We must have space for them.
• Snapshots/CheckPoints create differencing disks. These _grow_ over time.
• Deleting Snapshots/CheckPoints requires enough free space to create an entirely new Parent VHDX.
• vRAM assigned to the VM should not traverse NUMA nodes (performance) (more on hardware).
• vCPUs = Threads in CPU and must be processed in parallel thus # physical cores - 1 is best.
• GHz is preferred over CPU Core counts for most workloads.
• Storage
• Be aware of the IOPS required to run _all_ workloads on the host/nodes.
• More smaller sized spindles is better than less larger size spindles = More IOPS.
• 10GbE should be the minimum bandwidth considered for any iSCSI deployments.
• At least _two_ 10GbE switches are mandatory for the storage path
• Networking
• Broadcom physical NIC ports must always have VMQ turned off (blog post)
• We prefer to use Intel Gigabit and 10Gb Ethernet Server Adapters
• We start with a minumum of 4 physical ports in our hosts/nodes
• UPS Systems
• UPS Systems should have at least 1-1.5 Hours of runtime.
• Host/Nodes and storage should be tested for shutdown durations.

There are quite a lot of these types of posts on our blog. Please click through the category tags to find more!

Thanks for reading.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Some Remote Desktop Session Host Guidelines

We’ve put about four years and two versions into our Small Business Solution (SBS). We have it running on-premises on standalone Hyper-V servers as well as on Hyper-V clusters (Clustered Storage Spaces and Hyper-V cluster we just deployed for a 15 seat accounting firm).

It is the foundation for the Cloud Office services we’ve been offering for the last year or so.

Since our Cloud Office solution runs in Remote Desktop Services we figured we’d share some pearls around delivering Remote Desktop Session Host based environments to clients:

• ~512MB/User is cutting it tight
• ~20 to 25 users in a 12GB to 16GB vRAM Hyper-V VM works okay with 2-3 vCPUs
• RDP via 8.1 RDP clients saturates a 1Mb DSL uplink at ~13-15 users depending on workload
• ALL browsers can bring the RDSHs to their knees
• Printing can be a bear to manage (Use Universal Print Drivers and Isolation where possible)
• Group Policy configuration and lockdown is mandatory
• Two partitions with User Profile Disks (UPDs), if used, on the second partition
• NOTE: UPDs + Office 2013 and earlier + Exchange 2013 and earlier = Broken Search!!!
• NOTE: RDSH Search Indexes for Outlook OSTs in UPDs can fill up the C: partition!
• Office 2016 and Exchange 2016 together are supposed to address the broken search situation in RDSH setups were UPDs are used. We have yet to begin testing the two together.

Our Cloud Office (SBS) is running on clusters we’ve designed based on Scale-Out File Server and Hyper-V.

Need a clustered solution for your SMB/SME clients? Drop us a line. They are _very_ affordable. ;)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Protecting a Backup Repository from Malware and Ransomware

With the abundance of malware and ransomware it’s absolutely necessary that we take the time to examine our backup structures.

1. Volume Shadow Copies
• Obviously not a “backup”
• Most ransomware today kills these
2. Backup to Disk/NAS
• Rotated or streamed off-site
3. Cloud Backup
• Streamed off-site
4. Backup Tiers
1. Current, off-site 1, off-site 2, 6 Month, 12 Month, ETC…

With our last mile issue up here we are very careful about anything Cloud since most upload speeds are not capable enough nor are the download speeds capable of a decent recovery time.

Now, what is _the most important_ aspect to our backup setup?

Anyone?

It must be a closed loop!

What does that mean?

That means that at no point in the backup structure can anyone have access to the backups via the network or console.

Now, since almost all of our backups are streamed across the wire it takes a bit of a process to make sure our loop is closed.

• NAS
• ShadowProtect user with unique pass phrase (SPUP) and MOD on the repository root folder
• Other than the NAS Admin account no other user account is set up with access
• Turn on the NAS Recycle Bin!
• Most ransomware creates a new file then deletes the old one
• Create a separate username and folder structure for user facing resources!
• ShadowProtect
• Network destination set up with SPUP
• ShadowProtect Backups
• Encrypted AES 256-bit with a long pass phrase
• USB HDD
• Root folder share given MOD to SPUP and Users REMOVED
• Whether domain user on a cluster or local user on standalone Hyper-V
• Blog post: Folder Permissions: How To Properly Disinherit Permissions
• ImageManager
• All managed backups are set up to be accessed via SPUP only
• No repository, whether NAS or USB HDD is left with Users MOD
• No repository is left without a restricted username and password protecting it!

Recently, we know of a domain joined standalone Hyper-V server get hit by ransomware. As a rule we don’t join a standalone Hyper-V to the guest domain. This is just one more reason for us not to do so.

And finally, some of the more obvious aspects around backups and domain operation in general:

• Users are Standard Users on the domain
• If they absolutely need local admin because they are still running QuickBooks 2009 then make that choice
• Standard User accounts have _NO_ access to any aspect of the backup loop
• None, Nada, Zippo, Zilch! ;)
• Domain Admin accounts should have no access to any aspect of the backup loop
• Many client sites have one or two users (hopefully not more?!?!?) that know these credentials
• Access via UNC will pop up an authentication dialogue box.
• Use the SPUP and _do not save_ the credentials!
• Backups are managed by us, spot recovered by us, and quarterly bare metal/hypervisor restored by us
• No client intervention other than perhaps the off-site rotation (we do this too)
• If some user or users insist running as DOMAIN ADMINs then REMOVE Admin’s MOD from USB HDD/NAS NTFS/File System
• Leave only the SPUP with MOD

So, what spawned this blog post?

Hearing of a ShadowProtect destination NAS getting wiped out by ransomware. This should not be possible on our managed networks ever!

What spawned our lockdown of the backup structures?

Many years back we had a user that neglected to rotate the tape libraries and a faulty BackupExec that reported all being rosy until their server went full-stop and we had to recover (one aspect of the recovery in an SBS environment).

When we arrived, the person rotating the magazines turned sheet white when we asked for the off-site magazines. Oops. :(

We dropped BackupExec as their support failed to help us after three days of wrangling (Thursday afternoon until we cut the cord at 1730Hrs Saturday evening). We did end up recovering the full 650GB of data short of 24 files belonging to one of the firm’s partners across four to five days.

After that we went to all of our clients and proposed a managed backup strategy where we took care of all aspects of the backup. They all approved the changes after hearing what happened at the one firm. ;)

So, we tested and switched all of our clients to ShadowProtect 3.x and set up all backups so that no user could access them.

In our not so humble opinion, backups are not, and should never be, a user’s responsibility.

Thus, they should never have access to them even if they rotate them!

TIP: Need to do a side-by-side recovery or migration? ForensiT’s User Profile Wizard

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book