Tuesday 5 March 2013

Our Largest Disaster Recovery Navigated Successfully To Date

This was posted to the Spiceworks forum here: Announcing the Master of Data Disaster Contest!

One of the largest catastrophic failures that we successfully navigated took out a client's SBS 2003 R2 Premium server. There were a number of factors involved in this failure including heat (server closet did not have active cooling and the landlord did not have the HVAC set up correctly so heat was pumped in from above the false ceiling during the winter months), a SATA RAID array having failing members, and eventually the RAID Controller and disks doing a full-stop.

It took a week using various SwingIT methods (Jeff Middleton's SBS Migration methodologies) to recover their SBS to a lab server in our shop, SwingIT recover the AD/Content to a new server, forklift the Exchange databases, and drop the new server back into their office. ShadowProtect was an important part of this recovery.

After that week the _only_ user related problem we had after dropping the new server in and bringing it online (it used the previous server's name) was two users that called because they could not log on. While connected to the backup DC they had changed their password. Since the recovered server was using an AD that was a week old they had to use their previous password to log on. As noted in one of the linked posts we did end up needing to rebuild the forklifted Exchange 2003 databases.

The first heart stopping, cold hands inducing, and sinking stomach moment:

The second heart stopping, cold hands inducing, and sinking stomach moment where server death was now known to be immanent:

Now, becoming less and less relevant but having a second DC on an SBS STD network can be a killer when recovering:

SBS - Exchange Information Store is Corrupt? Recreating the Store

  • http://bit.ly/14padWb
  • Once we were down the line from point of recovery we still had issues with the Exchange databases we forklifted into place. This post outlines all of the steps to get content out of Exchange, create a new database set, and merge it back in. Exchange 2003.

SBS & ShadowProtect - Some Hardware Independent Restore considerations

  • http://bit.ly/Xo0akx
  • This post is relevant for any full bare metal or hypervisor restore of a server.

SBS Disaster Recovery - Finished

As noted in the Event ID 55 NTFS post, previous to this recovery BackupExec had totally failed us in a failure at this client site (heat and IDE/SATA are a killer combination). After three days on the phone with Symantec support we cut loose and rebuilt the domain and data from scratch managing to pull their entire data set (short 24 files belonging to one partner) back together. That's when we went looking for something better and came to ShadowProtect by StorageCraft.

Since this last major recovery we have been fortunate that we have not had any major failures to deal with.

And, just this week our client with the many major failures has moved to a new business condo they purchased. We will now have a proper climate controlled server room for their IT solution to live in. No more heat problems! W00t!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Windows Live Writer

No comments: