Some Disaster Recovery Planning On-Premises, Hybrid, and Cloud Thoughts

This was a post to the SBS2K Yahoo list in response to a comment about the risks of encrypting all of our domain controllers (which we have been moving towards for a year or two now). It’s been tweaked for this blog post.

***

We’ve been moving to 100% encryption in all of our standalone and cluster settings.

Encrypting a setup does not change _anything_ as far as Disaster Recovery Plans go. Nothing. Period.

The “something can go wrong there” attitude should apply to everything from on-premises storage (we’ve been working with a firm that had Gigabytes/Terabytes of data lost due to the previous MSP’s failures) and services to Cloud resident data and services.

No stone should be left unturned when it comes to backing up data and Disaster Recovery Planning. None. Nada. Zippo. Zilch.

The new paradigm from Microsoft and others has migrated to “Hybrid” … for the moment. Do we have a backup of the cloud data and services? Is that backup air-gapped?

Google lost over 150K mailboxes a number of years back, we worked with one panicked call who lost everything, with no return. What happens then?

Recently, a UK VPS provider had a serious crash and, as it turns out lost _a lot_ of data. Where are their clients now? Where’s their client’s business after such a catastrophic loss?

Some on-premises versus cloud based backup experiences:

• Veeam/ShadowProtect On-Premises: Air-gapped (no user access to avoid *Locker problems), encrypted, off-site rotated, and high performance recovery = Great.
• Full recovery from the Cloud = Dismal.
• Partial recovery of large files/numerous files/folders from the Cloud = Dismal.
• Garbage In = Garbage Out = Cloud backup gets the botched bits in a *Locker event.
  • Image based backups are especially vulnerable if bit rot, unrecognized sector failures, and such happen
  • The backup needs to be protected from user access (blog post)
• Cloud provider’s DC goes down = What then?
• Cloud provider’s Services hit a wall and failover fails = What then (this was a part of Google’s earlier mentioned problem me thinks)?
  • ***Remember, we’re talking Data Centers on a grand scale where failover testing has been done?!?***
• At Scale:
• Cloud/Mail/Services providers rely on a myriad of systems to provide resilience
  • Most Cloud providers rely on those systems to keep things going
• Backups?
• Static, air-gapped backups?
• “Off-Site” backups?
• These do not, IMO, exist at scale
• The BIG question: Does the Cloud service provider have a built-in backup facility?
• Back up the data to local drive or NAS either manually or via schedule
• Offer a virtual machine backup off their cloud service

There is an assumption, and we all know what that means right?, that seems to be prevalent among top tier cloud providers that their resiliency systems will be enough to protect them from that next big bang. But, has it? We seem to already have examples of the “not”.

In conclusion to this rather long winded post I can say this: It is up to us, our client’s trusted advisors, to make bl**dy well sure our client’s data and services are properly protected and that a down-to-earth backup exists of their cloud services/data.

We really don’t enjoy being on the other end of a phone call “OMG, my data’s gone, the service is offline, and I can’t get anywhere without it!” :(

Oh, and BTW, our SBS 2003/2008/2011 Standard/Premium sites all had 100% Uptime across YEARS of service. :P

We did have one exception in there due to an inability to cool the server closet as the A/C panel was full. Plus, the building’s HVAC had a bunch of open primary push ports (hot in winter cold in summer) above the ceiling tiles which is where the return air is supposed to happen. In the winter the server closet would hit +40C for long periods of time as the heat would settle into that area. ShadowProtect played a huge role in keeping this firm going plus technology changes over server refreshes helped (cooler running processors and our move to SAS drives).

*** 

Some further thoughts and references in addition to the above forum post.

• May 13, 2016: The Register: Salesforce.com crash caused DATA LOSS
• April 18, 2016: BBC: Web host 123-reg deletes sites in clean-up error
• The definition of irony:
•  
• September 20, 2015: The Register: AWS outage knocks Amazon, Netflix, Tinder and IMDb in MEGA data collapse
• November 2014: Microsoft Azure had a series of outages some lasting days. Fortunately, no data loss happened that we know of though VMs were offline
• Bing Search: Azure Outage
• Note results for earlier problems in April 2014
• August 6, 2012 WIRED: HOW APLE AND AMAZON SECURITY FLAWS LED TO MY EPIC HACKING
• Mat Hanon’s horrific story of complete data loss
• Please read this and then secure a password vault and use it to manage _different_ passwords for _ALL_ sites and client admins!
• We have 2FA enabled on every service that has it available
• We use KeePass for our vault
• February 27, 2011: endgadget: Gmail Accidentally resetting accounts, years of correspondence vanish into the cloud?
• As mentioned, we had frontline experience with this Google data loss situation. :(
• Note that threads around this issue seemingly stopped being updated with no further status reports by Google or others

The moral of this story is quite simple. Make sure _all_ data is backed up and air-gapped. Period.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Protecting a Backup Repository from Malware and Ransomware

With the abundance of malware and ransomware it’s absolutely necessary that we take the time to examine our backup structures.

1. Volume Shadow Copies
• Obviously not a “backup”
• Most ransomware today kills these
2. Backup to Disk/NAS
• Rotated or streamed off-site
3. Cloud Backup
• Streamed off-site
4. Backup Tiers
1. Current, off-site 1, off-site 2, 6 Month, 12 Month, ETC…

With our last mile issue up here we are very careful about anything Cloud since most upload speeds are not capable enough nor are the download speeds capable of a decent recovery time.

Now, what is _the most important_ aspect to our backup setup?

Anyone?

It must be a closed loop!

What does that mean?

That means that at no point in the backup structure can anyone have access to the backups via the network or console.

Now, since almost all of our backups are streamed across the wire it takes a bit of a process to make sure our loop is closed.

• NAS
• ShadowProtect user with unique pass phrase (SPUP) and MOD on the repository root folder
• Other than the NAS Admin account no other user account is set up with access
• Turn on the NAS Recycle Bin!
• Most ransomware creates a new file then deletes the old one
• Create a separate username and folder structure for user facing resources!
• ShadowProtect
• Network destination set up with SPUP
• ShadowProtect Backups
• Encrypted AES 256-bit with a long pass phrase
• USB HDD
• Root folder share given MOD to SPUP and Users REMOVED
• Whether domain user on a cluster or local user on standalone Hyper-V
• Blog post: Folder Permissions: How To Properly Disinherit Permissions
• ImageManager
• All managed backups are set up to be accessed via SPUP only
• No repository, whether NAS or USB HDD is left with Users MOD
• No repository is left without a restricted username and password protecting it!

Recently, we know of a domain joined standalone Hyper-V server get hit by ransomware. As a rule we don’t join a standalone Hyper-V to the guest domain. This is just one more reason for us not to do so.

And finally, some of the more obvious aspects around backups and domain operation in general:

• Users are Standard Users on the domain
• If they absolutely need local admin because they are still running QuickBooks 2009 then make that choice
• Standard User accounts have _NO_ access to any aspect of the backup loop
• None, Nada, Zippo, Zilch! ;)
• Domain Admin accounts should have no access to any aspect of the backup loop
• Many client sites have one or two users (hopefully not more?!?!?) that know these credentials
• Access via UNC will pop up an authentication dialogue box.
• Use the SPUP and _do not save_ the credentials!
• Backups are managed by us, spot recovered by us, and quarterly bare metal/hypervisor restored by us
• No client intervention other than perhaps the off-site rotation (we do this too)
• If some user or users insist running as DOMAIN ADMINs then REMOVE Admin’s MOD from USB HDD/NAS NTFS/File System
• Leave only the SPUP with MOD

So, what spawned this blog post?

Hearing of a ShadowProtect destination NAS getting wiped out by ransomware. This should not be possible on our managed networks ever!

What spawned our lockdown of the backup structures?

Many years back we had a user that neglected to rotate the tape libraries and a faulty BackupExec that reported all being rosy until their server went full-stop and we had to recover (one aspect of the recovery in an SBS environment).

When we arrived, the person rotating the magazines turned sheet white when we asked for the off-site magazines. Oops. :(

We dropped BackupExec as their support failed to help us after three days of wrangling (Thursday afternoon until we cut the cord at 1730Hrs Saturday evening). We did end up recovering the full 650GB of data short of 24 files belonging to one of the firm’s partners across four to five days.

After that we went to all of our clients and proposed a managed backup strategy where we took care of all aspects of the backup. They all approved the changes after hearing what happened at the one firm. ;)

So, we tested and switched all of our clients to ShadowProtect 3.x and set up all backups so that no user could access them.

In our not so humble opinion, backups are not, and should never be, a user’s responsibility.

Thus, they should never have access to them even if they rotate them!

TIP: Need to do a side-by-side recovery or migration? ForensiT’s User Profile Wizard

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Windows Server 2012 R2 and Two Smaller Servers Over One Big One

Having some thoughts on designing client's IT solutions to provide a relatively simple setup that allows for business to continue on in the event of a hardware failure.

Windows Server 2012 R2 gives us a few more options to facilitate business continuity.

Two smaller servers running their workloads allows for a number of different scenarios for recoverability:

• Hyper-V Replica
• For obvious reasons
• DHCP Failover (built-in, run the wizard after installing the DHCP Role on two systems)
• Very easy to do and gives clients full DHCP if one box goes down (no need to flip a switch somewhere else to enable DHCP)
• Shares all Scope Options and Reservations between the two

Some of the benefits of this setup are:

• AD is covered in the event of a full-stop
• Hiccups can be taken care of by Burflags and/or AD Recycle Bin
• AD continues despite one server going full-stop
• File services and LoBs come back online when replica failover kicks in
• A good backup regimen with restore tests allow flexibility (ShadowProtect)

Our preference has grown into having two key resources duplicated:
•    AD/DNS/DHCP across two separate VMs (2x servers)
•    Hyper-V Replica for VM hosting files and key LoBs

That folks is a poor man’s/woman’s "Cluster" setup.

Yes, there is a bit of extra cost involved for the licensing side of things. And, there may be a price difference on the hardware side of things.

But, when we look at the lifetime of the solution and take that extra cost we can then draw up a dollar amount per user per month using a 36 or 48 month amortization table (or even 60 month if five year warranty) and justify it as the cost of insurance relative to business stoppage costs. This works for us pretty much every time! :)

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
Third Tier: Enterprise Solutions for Small Business

SBS 2003 R2 Premium Migration to 2012 R2 Domain and Exchange 2013

We are beginning our adventure migrating our last SBS 2003 R2 Premium server setup to a completely new setup.

We used the ShadowProtect backup image to restore to our Hyper-V server and utilized the Hardware Independent Restore process to inject the Hyper-V 2012 R2 VM drivers so we did not get any blue screens on the restored VM OS.

Our goal will be to end up with an RWA setup in 2012 Essentials R2 or we will be pitching the new Scorpion Software AuthAnvil Portal setup as an RWA replacement to this firm (and eventually all firms we manage).

Given that most accounting firms need to log into many different sites for their day-to-day routines we believe that new portal service will meet that need along with the partners that would prefer a short PIN to log on. :)

Plus it will give them a huge step up in security.

For now, we have their server up and running on one of our Server 2012 R2 Hyper-V lab setups as we will be running through the migration process a few times to make sure we have everything down.

We set up a Windows 7 Professional SP1 VM to verify that the SBS 2003 was happy:

The SBS Connect Computer wizard was run to successfully connect the Win7 VM to the SBS domain. From there we installed Office 2010 SP1 and reset a couple key user's passwords to hook into their profiles.

We are now ready to begin the migration process in our lab.

1. Install: Windows Server 2012 R2 DC VM
2. Install: Windows Server 2008 R2 OS Temp VM
1. Exchange 2010 with current SP installed
3. Migrate Mailboxes and Public Folders
4. Install: Windows Server 2012 RTM VM
1. Install Exchange 2013 and CU3
5. Migrate Mailboxes and Public Folders
6. Install: Windows Server 2012 R2 VM(s)
1. LoB Migration

Once we have run through the above process we will then move on to migrating their actual production network.

Philip Elder
Microsoft MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business

Troubleshooting ShadowProtect Backup Failure 503 Fatal I/O Error

We have one SBS 2008 riding on a cluster that has started to fail its full backups but only at certain times.

•   StorageCraft KB: 503 Fatal I/O error (-31 A device attached to the system is not functioning).

The KB indicates that the problem is resident on the source if the error falls on a read or on the destination if on a write.

In this case our failure was on a write so we started to focus in on the destination.

For this cluster setup we have the backups stream across the wire to the standalone DC on an HP MicroServer that was also protected by ShadowProtect.

We looked into network connectivity as well as for disk I/O errors in the Event Logs with no results.

The last place to look was in the ShadowProtect setup on the DC itself.

Sure enough, the DC was set to run an incremental close to the same time the one backup on the SBS VM was failing.

We changed the standalone DC backup schedule to run one incremental at night to avoid any further conflicts with the VM backups that were streaming to it.

We now had a successful backup set on the SBS VM.

Philip Elder
Microsoft MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Cause For Pause: Accounting Firm Possibly Done In Due to Technician Error and Cryptolocker (reddit)

This article came across one of the lists I am a part of and really brought home our own experiences back when Backup Exec and Symantec spent three days working with us to recover a backup that in the end proved to be unrecoverable.

In the above case we were fortunate to have other methods in place to protect the data but we did end up losing the domain and 24 of a partner's files out of 650GB of data (the failure was progressive - garbage in garbage out).

• reddit: TL;DR - Accounting firm gets Cryptolocker Virus. Tech wipes the server to clean it because he has Carbonite backups. He can't remember password to the privately managed encryption key file and can't download the firms backup. Everything lost. : talesfromtechsupport

The BUE fail taught us to advocate strongly for us to be the ones to rotate the backups (the person responsible in the above case failed to rotate the two magazines) and to do a quarterly _full_ bare metal or hypervisor restore of the backup.

It also drove us to find a different backup and restore method that gave us portability for the backed up server along with good recoverability. We came across and have been running with StorageCraft's ShadowProtect product ever since. Since then we have had some spectacular recoveries completed as a result of ShadowProtect and the skills learned via Jeff Middleton's SwingIT migration methods.

One of the other lessons we learned early in our IT careers and is exemplified in the above article is the thoroughness with which we keep our client's audit notes. We document absolutely _everything_ about their network setups. They get any updated versions after they have been updated. One can never be too sure!

A full bare metal/hypervisor restored backup is the ONLY known good backup. Period. Full Stop.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Windows Live Writer

CryptoLocker Word Of Caution

One of the things we have done from the get-go when it comes to setting up ShadowProtect to stream backups to either a drive set connected to a standalone Hyper-V host or to the standalone DC in a Hyper-V cluster setting is to set the shares to allow the Domain Admin MOD.

Inheritance on the folder’s NTFS permission set is removed/copied out then Domain Users/Machine Users group will get removed altogether.

We do this for a number of reasons

• Users cannot connect to the ShadowProtect images
• They are password protected and are using at least AES128bit
• Users cannot delete the images

While we are into our client’s servers on a regular basis sometimes the occasional domain admin account password will expire in the interim.

ShadowProtect will start failing to back up to the shared folder as a result of not being able to log on so a small bonus in the mix.

We are seeing CryptoLocker problems abound lately where someone clicks on a link in an e-mail or is drawn to a compromised site. What that means is that _any_ file/folder set the user has permissions to access and modify may end up encrypted by the malware.

The _only_ way to “recover” from this situation is via Shadow Copies or backup.

If the backup drive and/or backup folder destinations for those ShadowProtect backup files, or any other product that lays down files for backup, is open for users to access then we all know what can happen.

Point of order: Any backup product that uses the volume snapshot service should have its backup times staggered over the Volume Shadow Copy snapshots as having two snapshots running simultaneously could end up with data toast on both sides.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Windows Live Writer

Creating a Fixed VHD in Windows 7 and a ShadowProtect P2V

We have a VHD creation process running in the Windows 7 Disk Management console:

The above VHD is being created on a network share hosted by one of our Hyper-V servers (2008 R2). It will be a 160GB fixed VHD that will host the Windows 7 machine’s OS once ShadowProtect is finished.

While that process is running ShadowProtect is taking an image of the Windows 7 machine as it will be P2Vd onto the above Hyper-V host.

This image is running to a 2008 R2 file server.

The machine is a Core i7-875K with a pair of 80GB Intel X25-M SSDs running in RAID 0 and is soon to be retired.

We will use our P2V Hyper-V Integration Services (previous blog post) step to get rid of the RAID signature and enable IS in the OS. Then, we will use our Hardware Independent Restore (previous blog post) steps to clean out any left-over devices from the physical machine.

We find out of all products available to us, including Microsoft’s own Disk2VHD too, that ShadowProtect gives us the most reliable method of moving a physical machine into a virtual setting or taking a VM and restoring it to hardware.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Windows Live Writer

Our Largest Disaster Recovery Navigated Successfully To Date

This was posted to the Spiceworks forum here: Announcing the Master of Data Disaster Contest!

One of the largest catastrophic failures that we successfully navigated took out a client's SBS 2003 R2 Premium server. There were a number of factors involved in this failure including heat (server closet did not have active cooling and the landlord did not have the HVAC set up correctly so heat was pumped in from above the false ceiling during the winter months), a SATA RAID array having failing members, and eventually the RAID Controller and disks doing a full-stop.

It took a week using various SwingIT methods (Jeff Middleton's SBS Migration methodologies) to recover their SBS to a lab server in our shop, SwingIT recover the AD/Content to a new server, forklift the Exchange databases, and drop the new server back into their office. ShadowProtect was an important part of this recovery.

After that week the _only_ user related problem we had after dropping the new server in and bringing it online (it used the previous server's name) was two users that called because they could not log on. While connected to the backup DC they had changed their password. Since the recovered server was using an AD that was a week old they had to use their previous password to log on. As noted in one of the linked posts we did end up needing to rebuild the forklifted Exchange 2003 databases.

The first heart stopping, cold hands inducing, and sinking stomach moment:

• http://bit.ly/KQPOAj (SBS, ShadowProtect, and an Event ID 55 NTFS Error ...)

The second heart stopping, cold hands inducing, and sinking stomach moment where server death was now known to be immanent:

• http://bit.ly/XQMXAx (Images No Good ... Catastrophic SBS Failure ... Down What?!?)

Now, becoming less and less relevant but having a second DC on an SBS STD network can be a killer when recovering:

• http://bit.ly/KJ8lOE

SBS - Exchange Information Store is Corrupt? Recreating the Store

• http://bit.ly/14padWb
• Once we were down the line from point of recovery we still had issues with the Exchange databases we forklifted into place. This post outlines all of the steps to get content out of Exchange, create a new database set, and merge it back in. Exchange 2003.

SBS & ShadowProtect - Some Hardware Independent Restore considerations

• http://bit.ly/Xo0akx
• This post is relevant for any full bare metal or hypervisor restore of a server.

SBS Disaster Recovery - Finished

• http://bit.ly/10a5Kd9

As noted in the Event ID 55 NTFS post, previous to this recovery BackupExec had totally failed us in a failure at this client site (heat and IDE/SATA are a killer combination). After three days on the phone with Symantec support we cut loose and rebuilt the domain and data from scratch managing to pull their entire data set (short 24 files belonging to one partner) back together. That's when we went looking for something better and came to ShadowProtect by StorageCraft.

Since this last major recovery we have been fortunate that we have not had any major failures to deal with.

And, just this week our client with the many major failures has moved to a new business condo they purchased. We will now have a proper climate controlled server room for their IT solution to live in. No more heat problems! W00t!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Windows Live Writer

Western Digital USB Drives Use A Proprietary Drive PCB for USB

This comes via the SBS2K Yahoo Group.

• Spiceworks Community: WD 500gb my passport Help!!

Apparently, Western Digital in their great wisdom has decided to mass produce a hard drive controller with only one interface on it: USB.

And:

As a rule we have been using the StarTech 4 bay SuperSpeed drive dock with bare drives for our backup rotations that we run.

For clients that have been running their own backup rotations we have been recommending the WD non-Green USB drives as we have had pretty good success with them. The same goes for the Seagate USB drives that are not “Green” in nature.

The one thing we have noticed with the manufacturer built USB drives is that there is a tendency to use 5400 RPM drives. And now, perhaps we are limited in how we can access a drive if something goes wrong with the controller.

Our preference for bare backup drives are:

1. Seagate Enterprise Storage (ES or Constellation) series SATA
2. Western Digital Black Label (non-Advanced Format for VHD/Backup VHD)

Both drives are 7200 RPM with better firmware than the retail/consumer drives. So, they will perform better and last longer due to ongoing handling during rotations.

We may request that in the future clients only purchase a certain drive and USB enclosure for their backups so that we have some control over what drives are used for backups.

As usual, it is Buyer Beware.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Windows Live Writer

StorageCraft IT Edition: Network Backup at AES128bit Performance

This is _neat_ to see:

This is a Windows 7 Enterprise x64 machine being backed up across the wire (Gigabit) to a Windows Server 2008 R2 file server using ShadowProtect IT Edition (v4.x.x).

We used AES 128bit encryption with a pass phrase for this particular backup.

• Intel Core i7-875K
• Intel DP55KG (BIOS 3878)
• 4GB Kingston ValueRAM

When at all possible it is our preference to run our backup and restore to a network location as USB 2 is just _too_ slow.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Windows Live Writer

The ONLY Place To Snapshot A DC VM is in the Lab Right?

And that is only to work through all of the KBs that follow in this blog post to gain AD recovery skills right?

On one of the lists we are a part of there is an active discussion going on about having a second DC on a smaller network for "redundancy" purposes.

When it comes to an SBS Standard based network there are some caveats for that second DC whether it is virtual or physical:

• MPECS Inc. Blog: SBS Disaster Recovery with a Second DC Caveats

As we have learned in our past recovery situations that second DC can actually be a hindrance instead of a help when there is a need to restore Small Business Server Standard.

Virtual DCs

Now, when everything is virtualized one may be tempted to snapshot a DC prior to making any changes to provide a "fall-back" if things go sideways.

• Microsoft KB: 888794: Things to consider when you host Active Directory domain controllers in virtual hosting environments

Some things to consider via the mentioned KB:

• DC should remain running continuously.
• Do not pause the DC VM for long periods of time.
• Problems may happen.
• System State backups are critical but have a shelf life.
• In multi-DC environments daily DC System State backups of at least two (2) DCs should be the norm.

When a DC is recovered back from a snapshot the following KB may be applicable:

• Microsoft KB: 875495: How to detect and recover from a USN rollback in Windows Server 2003, Windows Server 2008, and Windows Server 2008 R2
• Implications of rolling back a DC and the processes to possibly recover.

Now, take all of the above and read the following:

• Ask the Directory Services Team: DC's and VM's - Avoiding the Do-Over
• Mark Ramey goes over the possibilities when it comes to recovering DCs with some excellent visuals.

The point we are making?

It's okay to have a DC or three in a virtual lab that are used to break and tear apart then step back using a snapshot to then run through the above processes to figure out the recovery path of a restored-from-snapshot DC VM.

However, in a production environment, whether it be our own or our client's location, DC VM snapshots should _never_ be used. Period.

• Microsoft TechNet: Running Domain Controllers in Hyper-V (Deployment Considerations for Virtualized Domain Controllers)

A good backup, that is one that has been fully recovered to bare metal and/or hypervisor, along with a System State backup, are the only way to go. Then, being familiar with the above processes and caveats to having multiple DCs in a production environment is a must.

WS Backup & StorageCraft ShadowProtect

All of our current, as of Windows Server 2008 R2, smaller client networks with the exception of those running on Hyper-V failover clusters (Win2K8 R2) are running a single DC.

In most cases that DC is Small Business Server 2008/2011 Standard.

Why?

Because we test our client's backups on a quarterly basis as part of our ongoing services we provide them.

Test restoring our client's systems on a regular basis gives us full confidence in our ability to restore their single SBS/DC using ShadowProtect and in some cases the native Windows Server Backup.

Introducing a second DC into the mix, in the case of SBS networks, brings about caveats that we need not deal with (see first blog post link) especially when times may be stressful already.

The key to being confident in a single DC environment is in the backup solution set.

To repeat: Confidence in our backup solution is the key to our deploying a single DC solution.

If we are not versed in restoring the backups we deploy at our client sites, at that on a regular basis, then how can we have the confidence to recommend a single DC solution to our clients? If we don't restore our client's backups how will we be aware of what is needed if things really go sideways and a restore is required?

We _are_ confident in our backup solutions built upon Windows Server Backup and now especially on StorageCraft's ShadowProtect Version 4. SP v4 has proven that once again we will be deploying ShadowProtect at all of our client sites as the Hyper-V restore throughput problems we saw in the past are no more.

ShadowProtect's Hardware Independent Restore feature is also a must for P2V and V2V restore situations even between Hypervisor versions.

• MPECS Inc. Blog: ShadowProtect Restore to Hyper-V VM Tips

In the end, it is our preference to keep a single DC in our small to medium solution sets. KISS is our preference. And, a single DC with no snapshots taken follows that line of simplicity. Plus, recovery becomes that much simpler.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Windows Server 2012 Hyper-V How To: Convert VHD to VHDX

We stood up a VM on a Windows Server 2008 R2 Hyper-V server and are looking to test it on a newly installed Windows Server 2012 Beta Hyper-V.

First test, convert the VHD to VHDX:

1. In Hyper-V Manager on 2012 click Edit Disk
• 
2. Read then click Next on the Before You Begin window.
3. Click the Browse button and locate the VHD to be converted
• We put the VHD into the default storage location we set up on this server.
• 
• Note the caveat:
• 
4. In the Choose Action window click the Convert radio button and click Next.
5. In the Convert Virtual Hard Disk window choose the format which is VHDX in this case.
• 
6. Choose between Fixed or Dynamically expanding.
• 
7. Select the name and location.
• 
8. If Next is clicked then a summary screen will be presented along with the Finish button.
9. Click Finish.
10. The Editing process will start.
1. 
11. Check the folder the original VHD and the exported VHDX are in and:
• 
• We can see the progress being made. . .
• 
• On this particular system it looks as though we are seeing about 20GB of conversion over about 20 minutes.
• Intel Xeon X3350, 8GB ECC, Intel RAID, 3x Seagate ES in RAID 5.
• NOTE: There must be enough storage available for the exported VHDX!
• If storage is shared then it is doubly important that the full capacity of the dynamically expanding VHDX be taken into consideration.

We had some trouble running the ShadowProtect restore in Windows Server 2012 Hyper-V. The best recovery rate we could get was about 1.5MB/Second.

But, on Server 2008 R2 SP1 with all of the recent hotfixes we were able to get 21MB/Second across a Gigabit LAN using a single vCPU and 1GB of fixed RAM for the being restored VM. ShadowProtect version was 4.2.5 or thereabouts (most recent edition).

Check out the following blog for tips on avoiding BSODs in a P2V situation:

• MPECS Inc. Blog: Hyper-V Restore Tips using Hardware Independent Restore Blog Post

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

SBS 2003 To SBS 2011 Backup – Run A System State Backup

Before starting any changes on the source SBS 2003 server it is _absolutely critical_ to take a full backup (we use ShadowProtect) and a System State backup:

Note the time it took to run the System State backup shown above: 2 min, 3 sec. With a couple of minutes to get the backup configuration steps done this will be one of the best 5 minutes spent in the entire migration process _if things fail catastrophically_ and there is a need to step back.

That check mark should never be checked if a proper full server backup and a System State backup has not been taken.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

P2V Using ShadowProtect V4.x – Quicker But with BSOD 0x0000007B for Windows Server 2003 R2 in Hyper-V

When we were running P2V processes using v3 of ShadowProtect we were waiting quite a while as the ShadowProtect Recovery Environment did not work too well on the Hyper-V side.

ShadowProtect version 4 uses the Win7/Win2K8 R2 codebase for its Recovery Environment so we get full keyboard and mouse functionality while working via RDP plus a pleasant 20MB/Second as shown above.

The P2V we were running was for a legacy Line of Business application running on Windows 2003 Standard R2 that cannot run on a current OS. Not only that, the LoB does an awesome job for what is does and there is no current application out there that comes even close.

On first boot after doing a straight restore we hit a BSOD: STOP: 0x0000007B

A problem has been detected and Windows has been shut down to prevent damage to your computer.

Nothing in our searches turned up anything really useful other than a similar error indicating an inaccessible boot device.

• Microsoft KB 324103: Advanced troubleshooting for "Stop 0x0000007B" errors in Windows XP

Since we did not do an HIR for this P2V we went back and ran the restore process again after extracting the Hyper-V Integration Services as explained here:

• winDeploy Blog: Extracting Hyper-V R2 Drivers

The commands are as follows:

1. msiexec /a d:\support\amd64\Windows5.x-HyperVIntegrationServices-x64.msi TARGETDIR=C:\AIPx64
2. msiexec /a d:\support\x86\Windows5.x-HyperVIntegrationServices-x86.msi TARGETDIR=C:\AIPx86

Once we had our x86 version drivers we were able to load all of the Integration Services drivers using the Hardware Independent Restore feature of ShadowProtect. About 20-30 minutes later we had our server back online.

We then ran through the Integration Services install via the Action drop down menu to load the ISO in the VM.

All was good to go from there.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

ShadowProtect Backup Via USB 3.0

This is pretty neat:

A standard 500GB Seagate 7200 SATA drive is plugged into one of the NexStar USB 3.0 docks to run a backup of the drive.

It is averaging about 90MB/Second for the job as of now. The job finished in a little over three minutes averaging 87.5MB/Second!

Compare that to the average 20MB/Second that same drive would have run on a USB 2.0 connection and we can see the time savings benefits right away for the refresh we did on this Data Mule system (previous blog post).

NOTE: We could not get the Vantec NexStar USB 3.0 dock (NST-D300S3) to work when it was plugged directly into the USB 3.0 port on the back of the Intel Desktop Board DQ67SW. We needed to plug a Vantec USB 3.0 Hub (UGT-MH430U3) into the motherboard’s port and then the NexStar into that before any drive would be picked up.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Making a change? Back up! Touching that server? Back up!

We are in the process of migrating one of our long time non-profit clients from SBS 2003 R2 Premium to SBS 2011 Standard.

Their old server is _very_ tired and needs to be replaced.

We ran the needed updates and configuration tests before going forward with the changes for the migration itself. Prior to doing this we backed the server up.

We used NTBackup to create our System State backup shown at the bottom of the above screenshot. We then ran an incremental backup in ShadowProtect 3.x for SBS.

We were then confident to install and run the Source Prep Tool from the SBS 2011 TOOLS folder and move on to installing SBS 2011 in Migration Mode.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

ShadowProtect 4.0.1–Windows 7 Ultimate x64 Restore Good To Go

We supplied a number of new machines into a new client when we did a new SBS 2008 install not too long ago.

One of the machine’s hard disks was causing some flaky behaviour and throwing Disk errors in the Event Logs.

We installed ShadowProtect 4.0.1 Desktop edition on the machine yesterday and set up a scheduled backup to run every hour to a network share.

When we arrived this morning we ran a final Incremental backup prior to shutting down the system. We swapped out the bad drive for a new one and then booted off of our ShadowProtect IT Edition’s USB flash drive.

Once we created the necessary disk signature, rebooted, we were able to restore the 100MB partition that Windows 7 creates and then the actual system partition.

There was some initial concern because that 100MB partition normally does not have a drive letter associated with it plus the restore process flagged a bad BCD Database restore in red.

Once we ran through the actual system drive’s restore process there was a note in the Details pane that showed that the BCD Database was edited and successfully corrected.

We then closed ShadowProtect IT Edition and the system rebooted. It complained about not being shut down properly, but Windows 7 Ultimate did indeed boot up without a hiccup.

A quick check in Disk Manager showed that 100MB partition as not having a drive letter and everything was good to go!

Total time to swap out a bad drive and restore the system on an Intel Core 2 Duo E8500 on an Intel DQ45EK Mini-ITX motherboard with a 320GB 7200 RPM Momentus was about an hour and 10 minutes. A full OS install, driver install, updates, and application install would have been about 3-4 hours.

Note that ShadowProtect must be installed and configured on a Backup Schedule in order to have access to creating Incremental backups. A manual backup (Backup Once) will not allow for any incremental backups.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

ShadowProtect – Garbage In = Garbage Out

The situation we are in is not the fault of ShadowProtect specifically, but one that we have encountered before where the system’s disks were starting to fail and the backup was image based. Thus, any image backup software can be found to have bad data within the image.

We are in the process of finalizing an SBS 2008 server setup for a new client on-site. Their “server” system was experiencing spontaneous reboots with Check Disk running on each start up. This morning the system refused to come back up.

We had set up a ShadowProtect backup on that machine to help protect the data that was on it.

However, as the above screenshot of a BeyondCompare data copy out of the mounted ShadowProtect image shows us, if the data going in is bad the data coming out will be bad.

We experienced a situation like this at an unprecedented level a few years ago where a client’s very large RAID 5 array decided to start dropping sectors on the drives. We ended up recovering that SBS domain using the Swing method and the data through a combination of backup recoveries across the two servers at that site.

In this case we have no other fallback methods other than some of the users may have a copy of the corrupted files on the own systems since the central “server” has been behaving quite flaky for a while now.

So, we will be mounting the ShadowProtect backups that have been running on the flaky system for a couple of weeks now to see if we can find any of the now corrupted files that still may be in good shape.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

ShadowProtect Restore Of Windows 7 To Hyper-V Tips And More

We jumped the gun on the last post.

Using the Hardware Independent Restore feature of ShadowProtect we were able to get some Windows 7 OSs into Hyper-V via P2V along with some troublesome Windows Server 2003 OSs.

We needed to extract the .MSI contents of the VMGuest.iso into individual folders and ISO them. We inserted those drivers using the HIR feature of ShadowProtect and were greeted with the Windows 7 logon screen not too long afterwards.

The post is here:

• ShadowProtect Restore to Hyper-V VM Tips

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer