Wednesday 4 November 2009

We Know We Are In Trouble When? An Image Based Backup Flaw

Have a look at this and ask yourself, “What is missing here?”

image

The bottom two items in the above screenshot were just added the previous evening when this post was written.

The next screenshot of the ShadowProtect console tells us that we have a real problem:

image

Okay, so, now what is going on with our backups?

We replaced the troublesome Thermaltake USB hard drive enclosure with a new NexStar3 by Vantec since a hard drive size increase was needed anyway.

The BIOS in this particular server was quite out of date so we did the necessary updates to eliminate the warm boot hang when the NexStar3 was plugged into the server and turned on.

The BMC refused to take a proper update. We needed to force recover the BMC firmware though the BMC firmware still choked when we attempted an update to the backplane.

Once we powered up the server after the last BMC recovery attempt one of the drives in the RAID 1 array decided it was no longer a member. We verified in the RAID controller’s BIOS (SRCSASRB) that the hot spare had jumped into the array and began to be rebuilt which it had.

With that confirmed, we booted into the OS. Now, we had also updated the ShadowProtect version to 3.5 before running the BIOS/firmware updates too.

We logged into the RAID Web Console to have a look at what was up there because a pop-up came up indicating a fatal error:

image

So, where does that leave us?

In a pretty bad position. We shut down the Information Store and copied out the databases and took a System State. The second server on this network is not a DC since that is just too much of a hassle to work with in recovery situations.

The second server does have the Hyper-V role installed and is serving virtual desktops for RWW access.

We have the SBS 2003 R2 ISO as well as the Windows Server 2003 SP1 ISO being downloaded from Microsoft’s site at this moment. Tomorrow morning we will do a test install of one of them and recover the System State backup to make sure that things will recover enough to Swing back to the original server if things go completely sideways.

Our StorageCraft forum post on the ShadowProtect error:

So, we have inadvertently discovered another ShadowProtect/image based backup flaw. If the underlying hardware hiccups as is the case here, the backup will not run successfully until that hardware hiccup is fixed or things really go south on us.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

No comments: