SBS - Exchange Information Store is Corrupt? Recreating the Store

A while back we dealt with a catastrophic failure where our backup images were also toast. Links to the relevant posts are at the end of this one.

The Exchange databases managed to be recovered, and they actually mounted on the new SBS box we used the SwingIT method to replace the failed box with.

The databases would mount initially, but later on, they would not mount without a manual start of the Information Store service.

It was clear that we managed to get things back, but not back to 100% on the Exchange side of things.

Given that all of these events occured during the client's peak season, we left well enough alone until the time was right for us to start working on clearing up the issue.

That time was yesterday. Total time into this project based on 15 user mailboxes at about 2.5GB was 12-14 hours. Note that a good portion of this time was in testing things before going ahead with the proceedure. A good ShadowProtect backup was created just before beginning the process.

So, we attempt an integrity check on the offline store:

ESEUtil /G - Database is CORRUPTED!

Then, we try to run ESEUtil in defragment mode:

ESEUtil Error -613

Things are not looking too good by this point.

One pair of last ditch efforts:

• C:\Program Files\Exchsrvr\bin>isinteg -fix -t "c:\program files\exchsrvr\mdbdata\priv1.edb" -test alltests
• Once this utility finished, no errors were found! Huh?!?
• eseutil /mh "c:\program files\exchsrvr\mdbdata\priv1.edb"
• Results with no errors, but current databases are in Clean Shutdown state.

Where to next? We need to get rid of the bad databases while preserving all of the client Exchange data.

Enter Exmerge:

• Microsoft Exchange Server Mailbox Merge Wizard (ExMerge) download page.
• Read the ExMerge Manual that accompanies this download before anything else! There are a lot of important little details therein that need to be followed to make this process work!
• KB174197: Microsoft Exchange Mailbox Merge program (Exmerge.exe) information.
• KB273642: ExMerge Does Not Work Unless You Have Receive As and Send As Permissions on the Store (method did not work for us).
• TechNet: How to Grant Your Administrative Logon Account Temporary Rights to Read All Mailboxes in an Exchange Database (did work for us).
• Note, this step should be accomplished before anything else as it takes time for the permissions to set.
• TechNet: ExMerge Strategies and Best Practices.

The key instruction set that is particularly applicable to SBS: KB313184: How to recover the information store on Exchange 2000 Server or Exchange Server 2003 in a single site. After reading the ExMerge Manual, the other resources above, and following this Knowledgebase article, we were good to go through the process successfully.

We made sure to shutdown both the POP3Connector and the SMTP service before going any further. We did not unplug the DSL modem as Internet access may have been needed at some point through the process.

Note that when it comes time to begin the import process, the SMTP service needs to be running. At that point we unplugged the DSL modem and started the service.

We extracted ExMerge and its .ini file to the Exchange bin folder (C:\Program Files\Exchsrvr\bin), once the above temporary permissions are set, we were required to log off then on again for them to take, we were able to successfully run the Exmerge GUI by double clicking on the application and initiating the Two Step Export.

We chose to export all mailbox related items but the folder permissions as no Outlook user was sharing anything from their profile. Note that in our case ExMerge choked on the Dumpster during the export for all 15 mailboxes.

Critical step: Make sure to manually export all Outlook clients to PST before beginning the Import Steps!

Once we had the ExMerge and Outlook PST files in hand, we made a backup copy of the Exchange MDBDATA folder contents. We then proceeded to rename the two .edb files and delete the rest of the files as per the instructions.

A broadcast email was sent via our test account to the organization caused the new store to fire up everyone's mailboxes just as the instructions state. This step is critical to getting ExMerge to recognize the mailboxes we want to import into.

Run ExMerge in import mode and we were eventually greeted with:

ExMerge Import succeeds with some errors

Most of the export and import errors were around the Dumpster, Deleted Items, and some Inbox issues.

The total volume of the stores increased by about 300MB once the process was finished.

A rebuild of the Offline Address Book will be required once the dust settles: KB 905813: You receive an error message when you try to synchronize the offline address list on an Exchange Server 2007 or Exchange Server 2003 server while you are using Outlook 2003: "0x8004010F".

For the Outlook clients themselves, once the process has completed, we need to kill the original Exchange Profile and reestablish it. There are keys associated with the user's Outlook OST and thus the existing OST will not work. Starting Outlook will only generate errors:

Outlook: Exchange is currently in recovery mode.

We made sure Outlook was not running, then via the Mail icon on both XP Pro and Vista we removed the default Exchange profile. In the case of those Outlook profiles that have not created an Archives.PST file yet, we created a Temp.PST file to point all new items to. The Outlook Profiles will complain if Mailbox - User Name is the only option.

Once an Exchange connection has been established by Outlook, and the post recovery email test message is in the user's inbox, we know that we are in the right place. For larger profiles the OST generation process will take some time, so be prepared to move onto another user profile while waiting.

If a Temp.PST file was created above, use the Data Files button to remove it once Outlook is happy.

One thing to keep in mind: When doing the manual PST creation from the Outlook clients, make sure to put those files somewhere on the network in one place. Once the entire organization's PST files are created, move those PST files, or rename the folder that they are in so that Outlook is not able to find them ... just in case.

The exported PST files will be needed ... that is almost 100% guaranteed.

Further blog posts related to this server crash:

• SBS, ShadowProtect, and an Event ID 55 NTFS Error ...
• Images No Good ... Catastrophic SBS Failure ... Now What?!?
• SBS Disaster Recovery - Finished
• SBS Disaster Recovery - Second DC SBS Restore Caveats

Further Exchange related reading and resources:

• Microsoft Exchange Server 2003 Disaster Recovery Operations Guide
• Microsoft Exchange Best Practices Analyzer v2.8

This post is put together in what little time we have ... so it is not meant to be an exact step-by-step. It does flow according to the process, but the indicated documentation will facilitate the necessary deep dive that is required to get a complete grasp of what needs to happen.

So far so good, our client's users are happy. And, hopefully, we will no longer be hearing about strange Outlook behaviours!

This is one of the biggest recovery situations in the history of our company, and, also hopefully, it is now closed! :D

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.