Cause For Pause: Accounting Firm Possibly Done In Due to Technician Error and Cryptolocker (reddit)

This article came across one of the lists I am a part of and really brought home our own experiences back when Backup Exec and Symantec spent three days working with us to recover a backup that in the end proved to be unrecoverable.

In the above case we were fortunate to have other methods in place to protect the data but we did end up losing the domain and 24 of a partner's files out of 650GB of data (the failure was progressive - garbage in garbage out).

• reddit: TL;DR - Accounting firm gets Cryptolocker Virus. Tech wipes the server to clean it because he has Carbonite backups. He can't remember password to the privately managed encryption key file and can't download the firms backup. Everything lost. : talesfromtechsupport

The BUE fail taught us to advocate strongly for us to be the ones to rotate the backups (the person responsible in the above case failed to rotate the two magazines) and to do a quarterly _full_ bare metal or hypervisor restore of the backup.

It also drove us to find a different backup and restore method that gave us portability for the backed up server along with good recoverability. We came across and have been running with StorageCraft's ShadowProtect product ever since. Since then we have had some spectacular recoveries completed as a result of ShadowProtect and the skills learned via Jeff Middleton's SwingIT migration methods.

One of the other lessons we learned early in our IT careers and is exemplified in the above article is the thoroughness with which we keep our client's audit notes. We document absolutely _everything_ about their network setups. They get any updated versions after they have been updated. One can never be too sure!

A full bare metal/hypervisor restored backup is the ONLY known good backup. Period. Full Stop.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Windows Live Writer

Windows Server 2012 and Essentials: Error 80073718 on Update(s)

We may be seeing some issues in Windows Server 2012 and W2012E:

• TechNet Forum: Problems Installing WSE2012 Updates, Including Update Rollup 1
• TechNet Forum: Windows Server 2012 - Windows Update Error 80073718 on 5 updates
• TechNet Forum: server 2012 windowsupdate (sic) error 80073718

In the end the problem points to a "servicing stack corruption" with the OP needing to re-install the operating system from scratch.

While this may be a possible suggestion for a new server setup, those that have had their systems in production for any amount of time are left in a bit of a conundrum.

At this time there is no "fix" for the "problem".

So given that Microsoft would not recommend a third party action, our suggestion for those that have a full server setup in production is to look into running a Swing Migration. Fellow MVP Jeff Middleton's methodologies will preserve Active Directory, Group Policy, and more.

If the setup is a VM then there are a number of options one can pursue to Swing the AD off and back on again.

This method would save a good chunk of time as one would not need to migrate the local profiles over to the rebuilt domain nor have to deal with Group Policy Tattoos if one rebuilds with the same domain name. Plus, if there are on-premises services running in Exchange, SQL, and other Line of Business applications the time savings would be _huge_.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Windows Live Writer

Our Largest Disaster Recovery Navigated Successfully To Date

This was posted to the Spiceworks forum here: Announcing the Master of Data Disaster Contest!

One of the largest catastrophic failures that we successfully navigated took out a client's SBS 2003 R2 Premium server. There were a number of factors involved in this failure including heat (server closet did not have active cooling and the landlord did not have the HVAC set up correctly so heat was pumped in from above the false ceiling during the winter months), a SATA RAID array having failing members, and eventually the RAID Controller and disks doing a full-stop.

It took a week using various SwingIT methods (Jeff Middleton's SBS Migration methodologies) to recover their SBS to a lab server in our shop, SwingIT recover the AD/Content to a new server, forklift the Exchange databases, and drop the new server back into their office. ShadowProtect was an important part of this recovery.

After that week the _only_ user related problem we had after dropping the new server in and bringing it online (it used the previous server's name) was two users that called because they could not log on. While connected to the backup DC they had changed their password. Since the recovered server was using an AD that was a week old they had to use their previous password to log on. As noted in one of the linked posts we did end up needing to rebuild the forklifted Exchange 2003 databases.

The first heart stopping, cold hands inducing, and sinking stomach moment:

• http://bit.ly/KQPOAj (SBS, ShadowProtect, and an Event ID 55 NTFS Error ...)

The second heart stopping, cold hands inducing, and sinking stomach moment where server death was now known to be immanent:

• http://bit.ly/XQMXAx (Images No Good ... Catastrophic SBS Failure ... Down What?!?)

Now, becoming less and less relevant but having a second DC on an SBS STD network can be a killer when recovering:

• http://bit.ly/KJ8lOE

SBS - Exchange Information Store is Corrupt? Recreating the Store

• http://bit.ly/14padWb
• Once we were down the line from point of recovery we still had issues with the Exchange databases we forklifted into place. This post outlines all of the steps to get content out of Exchange, create a new database set, and merge it back in. Exchange 2003.

SBS & ShadowProtect - Some Hardware Independent Restore considerations

• http://bit.ly/Xo0akx
• This post is relevant for any full bare metal or hypervisor restore of a server.

SBS Disaster Recovery - Finished

• http://bit.ly/10a5Kd9

As noted in the Event ID 55 NTFS post, previous to this recovery BackupExec had totally failed us in a failure at this client site (heat and IDE/SATA are a killer combination). After three days on the phone with Symantec support we cut loose and rebuilt the domain and data from scratch managing to pull their entire data set (short 24 files belonging to one partner) back together. That's when we went looking for something better and came to ShadowProtect by StorageCraft.

Since this last major recovery we have been fortunate that we have not had any major failures to deal with.

And, just this week our client with the many major failures has moved to a new business condo they purchased. We will now have a proper climate controlled server room for their IT solution to live in. No more heat problems! W00t!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Windows Live Writer

SBS 2008 and 2011 Active Directory Password Complexity

To manage the password setup in SBS 2008 and SBS 2011 Active Directory we use the SBS Console:

1. Open the SBS Console.
2. Click on the Users and Groups tab.
3. If not already selected, click on the Users tab.
4. Under Tasks click on Change password policies.
• 
5. Set the password policy settings accordingly.
• 
6. Click Apply and OK.
1. NOTE: All users will be required to change their passwords on their next log on if any changes are made to these settings!

This wizard is normally used during the set up process only.

If this wizard is used somewhere in the middle of an SBS Migration or SBS Swing Migration the consequences may be dire. It is best left as the final step _after_ at least one week of the post-migrated server being in production.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

SBS Migration – SBS 2003 to SBS 2011 Swing Migration Officially Available

We have made no secret about our support of Jeff Middleton’s Swing methodology here.

There are many times where the skills learned by following the Swing Migration documentation to Swing SBS from one box to another have come through to save a client’s SBS environment after a catastrophic failure.

Since many SBS environments are single server or single DC, the Swing methodology teaches us a lot about how to either move AD and all of the bits with it or to recover AD and all of the bits with it.

We also learn that introducing another DC into the mix brings about a unique environment for recovery that is unique to SBS.

Many of the skill sets learned are out there somewhere, however one of the principle benefits of the SwingIT Kit is Jeff’s ability to pull all of those disparate resources into one organized place for us to work through.

Jeff also includes many tools and scripts to facilitate a clean migration from start to finish.

And one final point, the Swing method provides a fall back for the migration process where the Microsoft method does not once mailboxes have begun to move.

• SBS Migration by Jeff Middleton

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

MVP Jeff Middleton Swing Migration Deep Dive Canada

Fellow MVP Jeff Middleton will be touring across Canada in the coming weeks. If you have not seen Jeff speak in person, then this is the opportunity to do so.

He is a very dynamic speaker. He keeps us completely engaged with his style and the content is _always_ relevant to our SBS IT Business practices.

Having been to a few of Jeff’s presentations, I do highly recommend checking this out.

A video of Jeff can be found here:

• Jeff Middleton: Getting up to Speed with SBSmigration.com

The Small Business Specialist Web page for the events:

• SBSC Canada Blog: Join the 2010 SBS Migration Tour – Montreal, Mississauga, Richmond, Calgary

A quick link to the various registration pages:

• Where & When:
• Montreal:
  April 21
• Mississauga:
  April 22
• Richmond:
  May 4
• Calgary:
  May 5

There is a minimal cost associated with the event of $99 Canadian.

Here is the description from the Partner Program page for the event:

I am registered for the Calgary event. Registration will fill up fast, so it would be a good idea to jump in early.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

SBS 2003 to SBS 2008 Swing Migration

Up to now we have been using the Microsoft method for migrating existing SBS 2003 to SBS 2008. For all intents and purposes, this method has worked quite well for us.

We are now in a position with one of our clients, yesterday’s recovery (previous blog post), where we need to expedite their migration, but need to do so with minimal interruption to their operations and our own.

So, we are looking to run a Swing Migration for this particular case. We are doing so because the Swing method allows us to run most of the migration here in the shop while our client remains totally productive.

This in turn allows us to work on the processes as time permits and then insert the newly migrated SBS 2008 into their production network with minimal impact on their productivity and minimal need for on-site time.

Click on the Swing Migration category (blog category link) to read about some of our experiences with the method.

From migrating SBS 2003 onto new hardware or off existing hardware and back on again, the Swing method is an awesome way to bring about change on a production network with minimal impact on our clients.

The Swing method has also been instrumental in providing the resources and abilities to bring back SBS catastrophic failures seemingly from the dead and most importantly providing us with the _confidence_ to bring those networks back.

We strongly recommend that any IT technician working with SBS have an SBSMigration.com subscription to augment the needed skills to migrate and recover SBS networks.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Preparing for the Distinct Possibility of a Server Disaster

Besides being an awesome method for migrating SBS to SBS on the same or different hardware, Jeff Middleton’s Swing method brings about a whole new way of viewing disasters and disaster recovery situations.

Because of our previous experiences using the Swing method in past recoveries, we are in the process of preparing our OS backup-less server for the possibility of an all-out failure.

As mentioned in our earlier post this morning on the missing backup and the previous post to that we had a hardware hiccup on one of our client’s SBS servers. The hiccup is enough to cause ShadowProtect to not be able to take an image of the OS partition.

So far, despite the RAID controller throwing the fatal error, no NTFS 55 errors have shown up in SBS’s Event Logs. Hopefully this means that the bad section on the drive is in an empty area. We will be flipping out the array members this afternoon and rechecking the consistency to see if the error has disappeared.

Array member number one’s status:

Array member number two’s status:

Note the distinct lack of error counts for both drive’s media as well as a zero for the Predictive Fail Count on both drives. This is indeed a real puzzle.

So, here are a few of the things we are doing to prepare for the possibility of the main SBS crashing:

• We have moved the live Exchange private and public stores to the data partition that is being backed up.
• We have installed Win2k3 using the SBS media into Hyper-V on the second server and using the Swing method have introduced it as a DC on the network.
• We have the previous backup drive with ShadowProtect Images that can be used to restore SBS into Hyper-V thus providing us with a frame of reference for the setup.
• It is a little too far out of date to use as a recovery source though . . . maybe.

The second bullet will thus eliminate a step if we need to do a fresh build of the SBS box and re-introduce it back into the network using the Swing method.

We will be that much further ahead for recovery if this happens between the temporary DC and having the existing server recovered on our own lab Hyper-V box for reference.

In this particular case, our client’s second server also has a mirror of their own client data on it. We will be able to mount the ShadowProtect image from the good SBS backup and copy the changed files over using BeyondCompare.

While we are not in an ideal state with this particular server situation, we are in a position to expedite a recovery in short order thus leaving our client with very little down time.

ExchangeDefender will be very helpful in this situation too. If things do go sideways and the stores take a hit, we will have access to our client’s LiveArchive so that they can keep active with their e-mail.

More to come . . . hopefully a happy resolution with the new replacement drives cleaning things up!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

We Know We Are In Trouble When? An Image Based Backup Flaw

Have a look at this and ask yourself, “What is missing here?”

The bottom two items in the above screenshot were just added the previous evening when this post was written.

The next screenshot of the ShadowProtect console tells us that we have a real problem:

Okay, so, now what is going on with our backups?

We replaced the troublesome Thermaltake USB hard drive enclosure with a new NexStar3 by Vantec since a hard drive size increase was needed anyway.

The BIOS in this particular server was quite out of date so we did the necessary updates to eliminate the warm boot hang when the NexStar3 was plugged into the server and turned on.

The BMC refused to take a proper update. We needed to force recover the BMC firmware though the BMC firmware still choked when we attempted an update to the backplane.

Once we powered up the server after the last BMC recovery attempt one of the drives in the RAID 1 array decided it was no longer a member. We verified in the RAID controller’s BIOS (SRCSASRB) that the hot spare had jumped into the array and began to be rebuilt which it had.

With that confirmed, we booted into the OS. Now, we had also updated the ShadowProtect version to 3.5 before running the BIOS/firmware updates too.

We logged into the RAID Web Console to have a look at what was up there because a pop-up came up indicating a fatal error:

So, where does that leave us?

In a pretty bad position. We shut down the Information Store and copied out the databases and took a System State. The second server on this network is not a DC since that is just too much of a hassle to work with in recovery situations.

The second server does have the Hyper-V role installed and is serving virtual desktops for RWW access.

We have the SBS 2003 R2 ISO as well as the Windows Server 2003 SP1 ISO being downloaded from Microsoft’s site at this moment. Tomorrow morning we will do a test install of one of them and recover the System State backup to make sure that things will recover enough to Swing back to the original server if things go completely sideways.

Our StorageCraft forum post on the ShadowProtect error:

• All Partitions But OS Backup Successfully?

So, we have inadvertently discovered another ShadowProtect/image based backup flaw. If the underlying hardware hiccups as is the case here, the backup will not run successfully until that hardware hiccup is fixed or things really go south on us.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

The Catch-22 – Almost

We have already mentioned that we are working on a pretty messed up SBS install.

• One DIY and one "Professional"
• ShadowProtect Image Recovery into a Hyper-V VM

The deeper into the test Swing we go, the clearer it becomes that we need to Swing the actual production environment.

There are so many things going on in the background, from the Terminal Server that runs their proprietary accounting LoB, to the Linux box that they are going to deprecate at some point that has their old accounting system on it.

Going with a completely fresh SBS install is looking more and more prohibitive due to all of the “customizations” that we are seeing as we go through the Swing steps.

While we will be able to tidy up Group Policy, DNS, and have functional SBS Wizards and features like the Companyweb and Remote Web Workplace, we will not be able to tighten up the security structures as we had hoped, and we will not be able to restructure much in the SBS domain.

While there are still some unknowns that may rear their heads and try to bite us, the Swing Migration is giving us the confidence we need to come through this coming weekend and the actual Swing Migration with a fairly stable SBS and users that find everything working as expected … we hope! ;)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*All Mac on SBS posts will not be written on a Mac until we replace our now missing iMac! (previous blog post)

Windows Live Writer

ShadowProtect Image Recovery into a Hyper-V VM

We need to get to images restored into VMs running on Hyper-V for the test Swing Migration we are doing for the messed up “Professional” SBS in this post.

The deeper we are digging into this the more and more it may be a good idea to start fresh and work the client workstations from there.

But, we have committed to the business owners to run through the Swing test anyway, so we are well into it now.

The existing SBS was imaged using ShadowProtect and has been restored to one of our lab servers using the Hardware Independent Restore feature. We cleaned up the IBM XServe hardware pieces in the Device Manager and made sure that the lab server hardware drivers were installed and stable.

• Cleanup methodology blog post: SBS & ShadowProtect - Some Hardware Independent Restore considerations.

We are running a portion of the Swing virtually with the NewSBS ending up on the same lab server we have the restored SBS on now which is a Swing Off and Swing On again.

On this particular SBS network there is a Terminal Server. So, we took an image of it as well as one critical workstation. We are restoring both images to VMs running on Hyper-V Server 2008 (yes, we got it up and running though with no add-in RAID controller).

When we went to boot from the ShadowProtect I.T. Edition ISO in the VM, the WinRE would choke on the networking portion of the ShadowProtect startup routine. We ended up needing to uninstall the standard Hyper-V NIC and use a Legacy NIC to get things rolling. We were using the Windows Vista WinRE version too.

We placed the two ShadowProtect images on a network share to run the restore process in the Hyper-V based VMs using drive mapping feature in the ShadowProtect recovery environment.

When it comes to restoring an encrypted ShadowProtect image that is password protected using AES 128bit security, CPU crunching capabilities and GHz speed sure make a huge difference on the restore times.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*All Mac on SBS posts will not be written on a Mac until we replace our now missing iMac! (previous blog post)

Windows Live Writer

Thoughts on Supporting SBS and SBS 2008 Migrations

Up to now, a large part of the product focus for us has been learning the SBS 2008 product and its various components and abilities.

We are now in a position where we need to accomplish some SBS 2008 migrations from existing SBS 2003 domains for us and for some of our clients.

One of the most important aspects of gearing up for a migration is for us to run through the migration virtually first.

The general sentiment in the SBS community when it came to installing SBS 2003 and now SBS 2008 for anyone contemplating going on to support SBS was to install SBS at least three (3) times before embarking on that endeavour.

Why?

The simple answer is, how can we hope to support a product we know little or nothing about?

SBS is a pretty complicated product that Microsoft has taken great pains to bring all of the component parts together and make them play nicely in the same sandbox.

And they do play very nicely in the same sandbox! :)

Now, once we have run the practice installs a number of times, the next step in the process is to get familiar with how to bring SBS back from the dead.

Disaster Recovery skills are a prerequisite in our industry. We must take the time to get to know the backup and recovery processes that are built into the SBS products, but also get familiar with some third party products that can help us along the way too.

It is very disappointing to encounter a dead SBS situation where the individual, technician, or whomever has been supporting that small business’ network has not performed any kind of test recovery on the client’s backup solution previously.

OUCH!

There is simply no excuse for a “professional” I.T. services provider single man shop or organization with a number of technicians to have not tested their client’s backups.

The new SBS 2008 product comes with quite extensive migration documentation that Microsoft has put together for us. The documentation has room to grow yet, but it is still very good at providing the foundation we need to go forward and migrate an existing SBS 2003 network to a new SBS 2008 box.

Now that we have passed about 4 months since the SBS 2008 product’s release, there is a good foundation of knowledge out there on SBS 2008 along with some additional directions to take with regards to SBS 2008 migrations.

Ultimately, experience with both SBS 2008  and SBS 2003 products will be the single most important factor for performing a successful SBS 2003 to SBS 2008 migration.

We cannot emphasize enough that any SBS migration project should be tested virtually first.

Here are some suggestions based on how we are proceeding with our SBS 2003 to SBS 2008 migrations:

• Snapshot the SBS 2003 box before touching it for the actual migration. Put that snapshot away. Have a fallback in place in case things go awry.
• There are awesome imaging products available to get a snapshot of that existing client SBS server that can be used to swing into a VM.
• On the SBS VM, reset the user account’s passwords and make sure the POP3Connector service is disabled if the VM will be given Internet access.
• Test the user accounts for access to their mailboxes.
• Optional step: Snapshot one of the local workstations and restore it to a VM on the same virtual network as the now virtualized SBS 2003 VM.
• Once both VMs are confirmed to be communicating with each other properly and the user accounts are fully functional, shut them down and copy those VHDs to have a backup of the configuration. A snapshot will do, but having those VHDs is good insurance.
• Install SBS 2008 in Migration Mode and run through the migration steps.
• Install SBS 2008 in Migration Mode and run through the migration steps.
• Install SBS 2008 in Migration Mode and run through the migration steps.

No, the last two steps are not a typo.

Run through the migration process at least three times for the first time through a migration attempt! It is very important to get at least remotely comfortable with the process as it is very complicated.

Having some familiarity with how the process works with the specific client servers and a workstation will improve the chances of a successful migration immensely.

After running through the process three times, and subsequently the actual migration, then the rule of thumb should be to virtualize a client’s SBS 2003 and at least one workstation to do a practice run through before running the actual process on the client’s production environment.

Perhaps after running through a dozen or two SBS 2003 to SBS 2008 migrations, the technician may be comfortable enough to run through the process without the trial run … maybe.

As soon as we introduce a Line of Business application or applications into the mix, we have a need for that practice run. Toss into the mix any number of third party AntiVirus, database, accounting, and other applications and we will soon discover that nothing will beat the knowledge gained in a run through.

The need for the practice run becomes all the more acute the longer the SBS 2003 box has been around, or if it was an in-place upgrade from SBS 2000 that was an in-place upgrade from SBS 4.x!

Don’t know the history of the SBS 2003 box? Client is not too sure either and the audit notes are few and far between if they exist at all? Then run through the process virtually first! Get to know that box’s behaviour during the migration process because there very well could be some problem what will not show itself until well into the process.

We used this rule of thumb when we were learning the Swing Migration developed by Jeff Middleton too. We ran through the process a number of times prior to running our first Swing to make sure we had a good grasp on the process.

Ultimately, there is no replacement for experience. None.

Further reading:

• TechNet Library: Migrate to Windows Small Business Server 2008 from Windows Small Business Server 2003.
• Microsoft Downloads: Windows Small Business Server 2008 Migration Help (CHM file – place on desktop and remove Block by right click and Properties to get the file to work).
• TechNet Video: Chris Almida - SBS 2008 Migration demo and interview.
• Official SBS Blog: SBS Migration Tagged Posts.
• TechNet SBS Site.
• Susan Bradley: SBS 2008 Migration Tagged Posts.
• Jeff Middleton: SBSMigration.com (Yes, a Swing is in the works for SBS 2003 to SBS 2008! :) )

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*All Mac on SBS posts will not be written on a Mac until we replace our now missing iMac!

Windows Live Writer

Webinar: Jeff Middleton - Migration Concepts with SBS 2008

Jeff Middleton, an SBS MVP, is hosting a Webinar on SBS 2008 migration concepts.

Microsoft Partner Program: 5W/50 Series - Migration Concepts with SBS 2008 Friday January 9 at 09:00AM Pacific Time (US & Canada).

We have been SwingIT subscription holders for about two to three years now. Jeff's migration techniques, called a Swing Migration, to Swing SBS 2003 onto new hardware, or even Swing it off the old hardware and back on again are an absolute must learn for anyone supporting SBS.

While the Swing Migration method is great for migrating the existing OS, the techniques are an absolutely phenomenal foundation to work from when it comes to recovering SBS from a catastrophic failure. Once comfortable with them, it is a lot easier to stay calm in the midst of establishing a recovery plan for a failed SBS server. The techniques help establish the big picture, directions to take, and ultimately the best road to follow through to a successful SBS recovery.

Come have a listen. It will be worth your while!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

DCPromo Error - Unable to convert computer account NewDC$

We are facilitating a Swing Migration in the Eastern US for a new client by providing phone support and feedback to the local I.T. professional who is doing the actual Swing.

When promoting a domain controller into the existing SBS domain they received the following error:

Operation Failed

Active Directory Wizard was unable to convert the computer account NewDC$ to a domain controller account. Access denied.

We have not encountered this error on any of the Swings we have done to date.

A quick search turned up the problem:

• Microsoft KB 232070: When you run Dcpromo.exe to create a replica domain controller, you receive the "Failed to modify the necessary properties for the machine account. Access is denied." error message.

Working with the I.T. professional we discovered that the policy setting for Enable computer and user accounts to be trusted for delegation was blank.

This is what the policy setting in the Default Domain Controllers GPO should be out of the box on one of our SBS domains:

Enable computer and user accounts to be trusted for delegation

Now, something to keep in mind when making any changes to these types of policy settings: Do not click the Add User or Group button and type the name of a user or group then click the OK button.

Take an extra step or two to make sure that the proper Active Directory object is selected:

1. Click the Add User or Group button.
2. Click the Browse button.
3. Type the name of the user or group. In this case we will use administrators.
4. Click the Check Names button.
5. A successful query will underline your user or group thus confirming the correct object is selected. Any other possibilities and you will get a "Which one do you want" type prompt.
6. Click OK.
7. Click OK.
8. Click Apply and OK.
9. Click Start --> Run --> GPUpdate /Force [Enter]
10. Check the SBS App Log for the SceCli information Event ID 1704 indicating a successful replication.

You can then rerun DCPromo on the problematic server. Make sure that Windows Firewall service is disabled on the soon to be DC!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

SBS Disaster Recovery - Finished

Wow ...

After one full week of working on the failed SBS from last week, many attempts to Swing the old SBS onto new hardware, and recover data, we are finally finished.

The new SBS box was installed very early yesterday morning. We brought it online, shifted the client desktops off of the backup server, and verified their data.

Outlook, their shares, printers, and passwords were as they were with the exception of a couple of users that needed to change their passwords while on the backup DC.

A number of years ago, Jeff Middleton, the author of the SBS Swing Migration was up here in Edmonton sharing some of his Katrina Experiences. It was an amazing and very inspiring time for myself personally.

One of the questions he was asked was about introducing a second DC into the SBS network to provide some redundancy for Active Directory. His response as I recall it was something along the lines of, "That introduces a whole new can of worms".

He wasn't kidding. We ran into all sorts of hiccups, hurdles, and road blocks recovering and subsequently installing the new SBS because of the backup DC.

Since there are a number of things sitting here needing attention, some chronicles on the SBS disaster recovery (DR) will be forth coming.

This DR was probably one of the most challenging we have faced to date.

But, with the time, tools, and talent that we have we triumphed! And, most importantly: Our client is very pleased with the outcome.

We were able to salvage Active Directory - meaning no desktop profile impact at all, all of their data, and bring them back over to the new SBS server with a reboot and a couple of extra startup scripts.

More to come ... and a nod to Jeff Middleton: Thanks! Those Swing skills came in very handy with this recovery. :)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

SBS Swing Migrations - Using W2K3 VMs during the process

We have mentioned in the past how one can utilize Virtual PC 2007 based Server 2003 VMs to facilitate a smooth SBS Swing Migration.

Using a Windows Server 2003 R2 VM setup can streamline the steps. One can have a W2K3 SP2/R2 base VM install that can be used to copy from and start a new VM instance for the Swing Migration. The SP2 install can be used in the case where the SBS 2K3 Active Directory has not been extended for the new Server 2003 R2 components.

One of the significant benefits to using a Server 2003 VM in the Swing process is the ability to both enable Undo Disks and also to Save State and subsequently copy the VM VHDs after each phase of the Swing.

This is a great way for someone who is new to the whole process to get comfortable with running through the Swing steps and not have to worry about starting from Ground Zero if they make a mistake.

Having the saved VMs also gives us the ability to go back to them and run through the various server components to see the how and where things were modified and/or deleted.

The same is true for those who are Swinging SBS for a client's production domain environment. Having that Server 2003 VM can be a huge time saver if something hiccups or dies during the Swing process or during the final phase of the SBS setup post-Swing.

Note that when we are working with client's SBS production domains, we always create a ShadowProtect image of any of the servers needed for the Swing before we start. This gives us a quick fall back if things don't work out as we expect.

We also tend to take a quick ShadowProtect image of the new SBS install at the various Swing stages to make sure that we do have a fallback in case things hiccup or blow up.

We use one of our Vista lab systems running VPC 2007. Besides the DQ35JOE's on board Gigabit adapter, the system has a low profile Intel PRO/1000 PT Dual Port (EXPI9402PT) server adapter installed as our lab systems are setup in the Antec Minuet chassis.

The JOE's on board NIC is connected to our internal network, and one of the Intel PT's ports is connected to a stand alone dedicated Gigabit switch that the old SBS and new SBS servers will also connect to depending on which stage we are at in the Swing Migration.

The Windows Server 2003 Virtual Machine setup has helped us out during the Swing process on countless occasions.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Images No Good ... Catastrophic SBS Failure ... Now What?!?

In what turned out to be an SBS catastrophic failure yesterday, we were purposed with installing some fresh hard drives and restoring the OS and data partitions via a ShadowProtect (SP) backup image set.

Well, things did not go even close to plan. We had hoped that we would be in and out in under 4 hours under optimal conditions.

It did not matter how many times, which image version we used, or combination of array sizes in the SRCS16's BIOS settings, we could not get a successful recovery. After one successful restore we went into the Recovery Console to run CHKDSK against the troubled partitions. After that, the OS choked on a missing sys file. :(

Of the times that we did manage to get the SBS booted up, we found a plethora of Event ID 55s in the logs.

On many of the OS boot attempts we were greeted with:

Checking file system on H:

So, it began to look like the corruption ran pretty far back into our backup image sets.

We all know that hindsight is 20/20! ;)

So, in hindsight, the most expedient method of recovering this server would have been to SwingIT off the original hardware and SwingIt back onto a fresh install of SBS on that original hardware. Given that we did not know we would end up being on-site for 12 hours making recovery attempts and eventually rolling out the backup DC setup to provide authentication and shares it was not a viable option until well into the wee hours of the morning.

We now have the go ahead to Swing onto a new server instead of back onto the existing one. Since they are up and running, we took out the old SBS box that has a somewhat stable recovery on it ... though not stable enough for production ... to use in our Swing Migration.

For now, they are running via the backup DC and data mirror, along with the backup DC providing Internet access via RRAS and a second NIC. It is not ideal as there are a number of network dependent applications that required some fiddling to get working, but at least they are not twiddling their thumbs and loosing money hand over fist.

This is one scenario where having our client's email setup as follows pays off:

• MX 100 ispmailserver1.myisp.com
• MX 50 ispmailserver2.myisp.com
• MX 25 mysbsmail.mysbsdomain.com

The ISP email is pulled down to the SBS box via the POP3Connector which is set to 1 hour intervals.

At least for now they still have access to the outside world via Webmail and the server will get all of their incoming mail when it comes back online. Any critical emails can be BCCd back to themselves for later download.

While we have tried to keep the impact on our client's business down to a minimum, there have been a number of hiccups before things started to settle down. So, to provide our client some restitution for the lost time, we will provide some of our billable time to them at no cost.

Since they are our longest running client at close to 10 years now, it only makes sense to have a little give-and-take in the business relationship.

There are a couple of important lessons here:

• Test those SP backup images by restoring them
• Test their durability by restoring them to different hardware
• Having a second DC can provide an Active Directory source for a Swing Migration in the event of a total SBS failure.

In our case, we were guilty of not having enough time to run through their more recent image to do some restore tests. This too is another motivation to give our client a break on the otherwise very expensive I.T. week they are having.

Given the volume of work with this situation, and others, there may be a smattering of blog posts for a while ...

Thanks for reading and supporting us! :)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

SBS, ShadowProtect, and an Event ID 55 NTFS Error ...

We went in to a client site yesterday evening to finish a warranty swap of some flaky memory sticks.

Up to the time that the server was shut down last night, there were no indications of any problems with the two RAID arrays, a 500GB RAID 1 and 1TB RAID 5, on the box. They were partitioned as follows:

• RAID 1
• 50GB OS partition
• 430GB ServerData
• 10GB SwapISACache
• RAID 5
• 1TB NetworkData

The unit is a 3 year old SR2400 with dual 3.0GHz Xeons and 4GB of RAM. We had changed out the hard drives to give the server more storage about two years ago and did a warranty swap out of the Intel SE7520JR2 motherboard (previous blog post) about a month ago twice. The first swap was a struggle to get the replacement SE7520JR2 board to recognize the 4 sticks of Kingston RAM. The second swap was because the first warranty replacement board lost 2GB of RAM on a patch reboot. :(

Once we had the new Kingston RAM sticks installed, we fired up the box we went straight into the BIOS to confirm that all 4 1GB sticks were there which they were.

Subsequently, when booting into the OS, the initial Windows Server 2003 scroller kept going and going. While not necessarily a bad thing, after three years, we know how long this particular box takes to boot.

The heart started to sink at that point.

Then the kicker:

Once of your disks needs to be checked for consistency

Um, this NetworkData partition is a 1TB RAID 5 array! The previous ServerData partition took only a few minutes. For this partition, it looks like we are going to be here for a while ...

I: NetworkData is 95 percent completed.

Well, okay ... maybe not ... but then ...

Inserting an index entry into index $0 of file 25.

We have all had that one minute seems like an eternity experience when something stressful like this was going on. The amount of time that the above message scrolled on the screen seemed like one even though it may have lasted only 3 or 4 minutes.

The only thing that kept us from hitting that reset button was the fact that the 4 drives in the RAID 5 array where this was occurring were pinned. Meaning, their lights were on constantly due to disk activity.

Then a little light in what was seemingly turning into a catastrophic failure:

Correcting error in index $I30 for file 9377.

This went on for over 20 minutes.

Then we faced something one hopes to never face that late at night expecting to pop in and pop out for a quick task ... the proverbial nail in the coffin - a catastrophic failure:

An unspecified error occurred.
.

It is at this point that it has become pretty clear that we were in for the duration.

But then ...

Windows is starting up.

Again ... must resist pushing buttons (best Captain Kirk voice) ... keeping those fingers tied up and away from the power and/or reset on the front of the server. Just in case, we left it alone. And, thankfully, the above screen is what we were greeted with.

Soon we saw:

The Active Directory is rebuilding indices. Please wait

This stage took a couple of minutes. The Initializing Network Interfaces stage took another 10-15 minutes.

We were eventually greeted with a Services Failed to Start error and subsequently the logon screen.

*Phew*

It looks as though the OS partition has made it through this relatively unscathed. The service chokes were for SQL, WSUS, WSS, and a LoB application that had their databases stored on one of the soon to be discovered absent partitions. Exchange had also choked.

One lesson in all of this: A server may stay up and running almost indefinitely when experiencing a sector breakdown on a disk member or members of the array. To some degree the RAID controller will compensate. However, in our experience, as soon as the server is downed, or rebooted, those sector gremlins can jump out and make their presence known as was the case here.

Another lesson from this: We keep the Exchange databases on the OS partition for this very reason. If the Exchange databases were on a different partition and/or array and it fails, we loose Exchange and email communication. If we have an SBS OS that boots with a relatively happy Exchange ... the databases intact too, then at least our client will not loose their ability to communicate with the outside world while we would be working on the data recovery side of things.

Back to this SBS box: Once into the OS, we were able to eventually get into the Event Logs and sure enough, out of the four partitions, the three besides the OS partition were toast.

Amazing ... simply amazing.

From the server Event Log:

Event Type: Error
Event Source: Ntfs
Event Category: Disk
Event ID: 55
Date: 4/15/2008
Time: 7:01:32 AM
User: N/A
Computer: MY-SBS01
Description:
The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume .

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 0d 00 04 00 02 00 52 00 ......R.
0008: 02 00 00 00 37 00 04 c0 ....7..À
0010: 00 00 00 00 02 01 00 c0 .......À
0018: 00 00 00 00 00 00 00 00 ........
0020: 00 00 00 00 00 00 00 00 ........
0028: 81 01 22 00 .".

The above Event Log messages were numerous.

Just in case, we initiated the ChkDsk utility from within the GUI. It too crashed on the two critically needed partitions.

We made sure that the relevant services that had folders on the ServerData partition were shutdown, and we fired up ShadowProtect to bring that partition back. We were fortunate that this particular partition recovery cooperated and we were able to fire up the relevant services and their LoB app that had the database server logs on that partition too.

The 1TB RAID 5 array did not cooperate at all. Even after a 10 hour plus SBS OS initiated ShadowProtect restore attempt that we began very early this morning. It failed and caused the SBS server to spontaneously reboot around 10:30AM this morning. This also means no LoB application for now. We are fortunate that it was not critical to the daily functioning of our client's business.

So, where does that leave us?

In this client's case, we have a backup DC that also has a live data mirror on it! So, with SBS at least functional, we were able to email users a simple batch file to disconnect the original Company share and connect them to the backup server's Company share.

On the SBS box, we made sure to restart and verify the services running on the now restored ServerDATA partition, and have left the RAID 5 array partition alone for now.

The extra expense of having that backup DC/Data Mirror box sitting there has just paid for itself in spades. For this client, we are talking a hit against the firm in the magnitude of $1K/hour for down time. The share switch took a relatively small period of time. If SBS went down totally, the backup server is setup to bring DHCP, DNS, and a secondary ISP gateway online within very short order to at least keep the firm functional.

If things happened to end up with a nonfunctional SBS OS, we would have also had the option of bringing down one of our Quad Core Xeon 3000 series servers that sits on our bench just for this task: A ShadowProtect Hardware Independent Restore of a client's entire SBS setup. We would bring them back online fully functional on newer, albeit temporary, hardware setup until such time as a new permanent server could be installed.

Having the ShadowProtect backup setup in place gives us a good number of very flexible options to make sure that there is very little or no impact on our client's daily business operations in the event of a server failure.

Given the age of this particular 2U system, we are now talking to the partners about a replacement SBS 1U to be Swung in by the end of this week.

ShadowProtect

There is definitely one thing that has been made especially clear in the midst of all of this: The last time we experienced a catastrophic failure of this magnitude, we had BackupExec and two 72GB x6 HP Tape Libraries to fall back on. The recovery took a whole long weekend because of the sheer volume of data and struggles with BUE.

This time around, while the stress levels were and are still high, they were no where near the levels they were at when the last SBS catastrophic failure happened.

Even if the partners decide on a set of replacement hard drives as a temporary measure until their high season dies down near the end of the summer, we will have them back online with solid storage tonight. Can't say the same for tape and BUE ... especially with the volumes of data we are talking about.

ShadowProtect gives us the options ... with the hard drive replacement, between StorageCraft's ShadowProtect, and the disaster recovery training (one can connect the dots) one receives doing Swing Migrations, we will be able to do the following:

• Restore a clean version of SBS from the previous evening's ShadowProtect image
• Recover the Exchange databases from the ShadowProtect incremental we will do tonight on the SBS partition
• Forklift those databases into the recovered SBS Exchange
• Restore the ServerData and NetworkData partition data from last night's clean ShadowProtect image
• Copy the backup server's live data mirror changes that were made by the client's users back to the SBS box.

As it turns out, we just received a call back from our partner contact at the firm. They prefer to go with the hard drive replacement until their business slows down this summer to keep things as close to status quo as possible for now.

We can do that! We have the technology and the skills! ;)

A big thanks to both StorageCraft for a great product and Jeff Middleton of SBSMigration for the awesome skill set we have gained via the SwingIt! Kit. Without this product and those skills, we would be in a very bad situation getting worse by the minute ... and very likely out of a really good client ... or ...

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

SBS Migration - What it can do for you

Folks, we can't say this enough: Please, Please, Please stop by Jeff's Web site: SBSMigration.com and pick up a subscription!

Why?

Because, when it comes to professional skills development, you are going to receive a return on your investment that cannot be measured in dollars and cents.

Investment ... we hear about that and see the spam in our inboxes on a regular basis. However, an SBS Migration subscription is just that: An investment in your company and employee's futures.

Here are some of the many ways that investment will pay dividends:

• SBS 2008 (Cougar) that will not have an in-place upgrade
• Performing Swing Migrations will prepare you for what is to come
• Bringing an existing SBS domain over to new hardware with little or no AD domain impact
• Ability to take an existing SBS domain off the existing hardware and Swing it back onto that existing hardware
• Ability to tailor out Active Directory messes during the Swing process
• That day when Exchange dies a death unknown to any before ...
• SBS integration into an existing Server 2003 AD domain with Exchange
• Users not knowing their server changed last night or over the weekend! ;)
• Generating Disaster Recovery scenarios and how to actually recover from them

There are so many ways that the Swing method has helped us out. It has been one of the best investments we have made in SBS related product tools.

Once the subscription is in hand, use ShadowProtect to transfer your production SBS domain onto some lab hardware via Hardware Independent Restore and start Swinging It! Using ShadowProtect means no impact whatsoever on the production SBS AD domain and gives us the opportunity to practice our Hardware Independent Restore skills. ;)

We suggest working with physical hardware, especially when Swinging between hardware configurations, or Swinging off a piece of hardware and back on again, to give one a better perspective on how the Swing will behave across the various server hardware configurations we work with daily.

This situation also presents an opportunity to utilize the various new and old hardware systems around the shop. When we receive new server configurations that are not slated for delivery until a week or more down the road, we can "burn in" that box on a couple of Swing Migrations. This gives us an excellent view on how SBS will behave on that new configuration.

While working with virtual SBS environments can do the job, virtual machines are not necessarily the best path to take here. System drivers and utilities, among other factors, also present variables that can impact the success or failure of a Swing Migration.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

SBS Migration - ShadowProtect speeds up those final steps

When we are working our way through an SBS Migration we have now taken to making sure we have created a full ShadowProtect (SP) backup the night before we plan on the final swap out of the old server with the new.

When we arrive on-site with the new box, we immediately login to the old server and run an incremental SP image. One of the main reasons for this is the amount of time it takes to create the incremental image: On an old P4 based server box we Swung out this last weekend we had 4 partitions of various data volumes incrementally backed up in about 20 minutes ... with no worries about missing anything post Swing.

As soon as the SP image begins to initialize, we kill the Internet connection to the old SBS box. We are not worried about loosing anything with regards to email since our clients are setup with two tiers of MX records:

• The ISP that is hosting the @MyInternetDomain.com that is gathered by POP3Connector once an hour
• Direct SMTP via direct Internet connection and MX pointer.

While this type of email MX setup is not exactly the best possible method for running email redundancy, it does work relatively well. This is especially true for our smaller or non-profit clients that may not be in a financial position to afford services like ExchangeDefender, or Reputation Services (Trend).

As soon as the backup is finished, we down the old server and bring up the new one. We run the CEICW right away on the new SBS box to pickup the correct ISP IP(s) and ISP DNS settings. We initially CEICW'd the new SBS box using our in-house Workbench Internet access for the ongoing configuration needs.

From there it is a matter of:

• Mounting the fresh SP image in Read Only mode
• Copying the Exchange databases
• ForkLifting them to bring the existing Exchange mailboxes online
• A quick SyncToy run via the SP image client data against the data we already have in place from a previous SP image we used earlier in the Swing.

The above steps greatly simplify and speed up many elements in the Swing Migration.

Having that final image also saves us from having to fire up the old box if there is any "just in case" situations with the client's data, email data, and general setup during those first few vulnerable days.

One of the last things we do before turning out the lights: Setup SP and run a full backup of the server.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.