Hyper-V: Set Up A Permanent Host OS Flash Drive

All of our Hyper-V hosts whether standalone or clustered 1U/2U nodes have a permanent flash drive plugged into the server.

They also have an Intel RMM, Dell iDRAC Enterprise, or HP iLO Advanced set up for full remote KVM over IP access.

We can then flatten and restore that host to production worthy status in about 30-45 minutes. PowerShell has a lot to do with the ability to make this happen as far as post OS configuration.

Here's how we do it:

1. DiskPart
1. List Disk
2. Select Disk x (flash)
3. Clean
4. Create Partition Primary
5. Select Partition 1
6. Format FS=NTFS Quick Label=”HV_Node-01”
7. Active
8. Assign
2. Have the host OS ISO mounted and copy its ENTIRE contents to the root of the flash drive.
3. We then create the following folders in the root of the flash drive:
1. _Drivers
2. _Utilities
3. _Software

What goes into those folders:

• Drivers = obvious
• Utilities = things like HVRemote, the bind tools, ETC
• Software = Server Management software

We _never_ back up the host. Period.

Once the host OS has been set up please make sure that having that USB flash drive permanently plugged in does not interfere with the host's boot process by verifying the boot order and USB boot settings in the BIOS.

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
Third Tier: Enterprise Solutions for Small Business

Cause For Pause: Accounting Firm Possibly Done In Due to Technician Error and Cryptolocker (reddit)

This article came across one of the lists I am a part of and really brought home our own experiences back when Backup Exec and Symantec spent three days working with us to recover a backup that in the end proved to be unrecoverable.

In the above case we were fortunate to have other methods in place to protect the data but we did end up losing the domain and 24 of a partner's files out of 650GB of data (the failure was progressive - garbage in garbage out).

• reddit: TL;DR - Accounting firm gets Cryptolocker Virus. Tech wipes the server to clean it because he has Carbonite backups. He can't remember password to the privately managed encryption key file and can't download the firms backup. Everything lost. : talesfromtechsupport

The BUE fail taught us to advocate strongly for us to be the ones to rotate the backups (the person responsible in the above case failed to rotate the two magazines) and to do a quarterly _full_ bare metal or hypervisor restore of the backup.

It also drove us to find a different backup and restore method that gave us portability for the backed up server along with good recoverability. We came across and have been running with StorageCraft's ShadowProtect product ever since. Since then we have had some spectacular recoveries completed as a result of ShadowProtect and the skills learned via Jeff Middleton's SwingIT migration methods.

One of the other lessons we learned early in our IT careers and is exemplified in the above article is the thoroughness with which we keep our client's audit notes. We document absolutely _everything_ about their network setups. They get any updated versions after they have been updated. One can never be too sure!

A full bare metal/hypervisor restored backup is the ONLY known good backup. Period. Full Stop.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Windows Live Writer

Utility Feebie: ExFolders for Exchange 2010 SP1+

When there is a need to work with Public Folders in Exchange 2010 the ExFolders utility is the best.

We can manage all aspects of Public Folders _and_ recover deleted items!

• The Exchange Team Blog: Exchange, meet ExFolders

Please take a careful read of the blog post and the instructions on running the utility.

It just works.

Thanks to the Exchange team for this! :)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Windows Server 2012 Hyper-V How To: Convert VHD to VHDX

We stood up a VM on a Windows Server 2008 R2 Hyper-V server and are looking to test it on a newly installed Windows Server 2012 Beta Hyper-V.

First test, convert the VHD to VHDX:

1. In Hyper-V Manager on 2012 click Edit Disk
• 
2. Read then click Next on the Before You Begin window.
3. Click the Browse button and locate the VHD to be converted
• We put the VHD into the default storage location we set up on this server.
• 
• Note the caveat:
• 
4. In the Choose Action window click the Convert radio button and click Next.
5. In the Convert Virtual Hard Disk window choose the format which is VHDX in this case.
• 
6. Choose between Fixed or Dynamically expanding.
• 
7. Select the name and location.
• 
8. If Next is clicked then a summary screen will be presented along with the Finish button.
9. Click Finish.
10. The Editing process will start.
1. 
11. Check the folder the original VHD and the exported VHDX are in and:
• 
• We can see the progress being made. . .
• 
• On this particular system it looks as though we are seeing about 20GB of conversion over about 20 minutes.
• Intel Xeon X3350, 8GB ECC, Intel RAID, 3x Seagate ES in RAID 5.
• NOTE: There must be enough storage available for the exported VHDX!
• If storage is shared then it is doubly important that the full capacity of the dynamically expanding VHDX be taken into consideration.

We had some trouble running the ShadowProtect restore in Windows Server 2012 Hyper-V. The best recovery rate we could get was about 1.5MB/Second.

But, on Server 2008 R2 SP1 with all of the recent hotfixes we were able to get 21MB/Second across a Gigabit LAN using a single vCPU and 1GB of fixed RAM for the being restored VM. ShadowProtect version was 4.2.5 or thereabouts (most recent edition).

Check out the following blog for tips on avoiding BSODs in a P2V situation:

• MPECS Inc. Blog: Hyper-V Restore Tips using Hardware Independent Restore Blog Post

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

SQL Database Restore Error: Restore failed for Server 'Name'. (Microsoft.SqlServer.Smo) - Exclusive access could not be obtained because the database is in use.

We are in the process of troubleshooting a database corruption issue that happened due to a RAID 1 array failure on an older SBS 2003 R2 server configured with a pair of software RAID 1 arrays.

When we went to restore the database from the most recent .BAK file we always ended up with the following error:

Microsoft SQL Server Management Studio

Restore failed for Server 'Name'. (Microsoft.SqlServer.Smo)

Additional Information:
System.Data.SqlClient.SqlError: Exclusive access could not be obtained because the database is in use. (Microsoft.SqlServer.Smo)

It took a number of searches to finally come up with the answer:

• Microsoft Dynamics Forums: How to restore CRM 4.0 Database? (Restore Failed)

Following the steps listed by pogo69:

1. Right-click on the database name
2. Click on Properties context menu item
3. Click on Options menu item on the left
4. In the grid, scroll down to the bottom
5. Change 'Restrict Access' to SINGLE_USER

Once we ran the above steps we were able to restore the database successfully.

Microsoft SQL Server Management Studio

The restore of database 'NAME' completed successfully.

We were then back into our client's LoB.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Backups Should Be Bare Metal and/or Virtually Test Restored Right?

We provide a service to all of our clients for a fixed monthly fee where we rotate their backups for them every two weeks for local clients or FedEx the backup drives back and forth.

Every quarter we do a full bare metal or virtualized restore of their backup. A _FULL_ restore of their server or servers onto our own systems. We are then assured that the backup technology being used is working as it should.

For image based backups like StorageCraft or Acronis products is very important to note, “Garbage in, Garbage out”.

We do this so that we know the following:

1. The backups are good.
2. We will not be stressed about the backups if things do blow up.
3. The backups are good.
4. The client will not be _as_ stressed about their data and getting their business up and running after a catastrophe.
5. Their backups are known good.

The above service and mantra comes via many years of experience with that experience containing some extremely bizarre failures in that time. This blog accounts some of them in more recent years.

Disaster Recovery Planning

Disaster Recovery Planning should be one of the conversation steps for new clients and a semi-frequent nag for existing clients that have not studied and implemented a disaster recovery plan for their business.

The “It won’t happen to me” attitude does not wash . . . at least not in our experience.

One of the most amazing presentations I have ever sat through was with Jeff Middleton of SBS Migration sharing his experiences trying to decision make and get ready for the oncoming Hurricane Katrina to a group of us on one of his visits.

Up here we may not face hurricanes but we certainly do have enough to consider:

1. Break-in with servers removed.
2. Disgruntled employee data destruction.
3. Weather related disasters.
4. Power related equipment failures.
5. Fire.

Cloud Vendors and Data Preservation

Now, it seems that larger Cloud vendors are finding out the hard way that backups need to be fully test restored in order to certify their validity:

• Own Web Now: MPECS Inc. Blog: ExchangeDefender E-Mail Outage – Our Client E-mail and Some Thoughts
• Tieto: The Channel Register: Flash drive meltdown fingered in Swedish IT blackout
• Tieto: The Register: Titsup [sic] EMC VNX kit unleashes 5 days of chaos in Sweden - Crashed Tieto system took down bank, pharmacy, schools' website
• The Alaskan Government: ZDNet: Alaska revenue dept. lost two backups of taxpayer data
• Summary: Technician wiped out hard drives, backup drives and backup tapes were unreadable. After $200,000 in recovering records from paper originals, new procedures are in place

Obviously we are talking about a scale way beyond a single USB drive or High-Rely unit with a few Terabytes or 10 worth of data backing up a few standalone or clustered servers.

However, we believe that it goes without saying that at some point, preferably sooner than later, a Cloud Services Company, a larger Enterprise, or an SME/SMB company would schedule and go through a full restore of their backups to verify that things would work when needed.

With the advent of mainframe style computing, what we are now calling Cloud Computing or Cloud Services, there is a particular need to pay attention to all Cloud vendor’s SLA with particular attention on who is responsible for the data.

• Microsoft O365 SLA
• No clear mention of data rights and responsibilities.
• Own Web Now SLA
• Backups and Archives - Customer is solely responsible for all data on their servers, mailboxes, databases and other storage mediums. While Own Web Now Corp provides means and mechanisms for customers to perform backups, customers are responsible for establishing policies, monitoring backups, data restoration, and retention intervals. Customers data is customers responsibility, under no circumstance shall Own Web Now Corp provide drives or other media nor send them to a data retrieval facility.
• Google Apps SLA
• No mention of client/customer data.
• Google Apps for Business (Online Agreement)
• No mention of client/customer data.

In the end, where nothing is explicit in any agreements, the wording uses legalese, or the wording is just all-around vague it is up to us to provide a solution that protects the client’s data integrity.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Patching SBS 2008 and SBS 2011

We are noticing that some of the recent updates are requiring several auto reboots of the server along with a seemingly long hang at a blank or green screen.

• SBS 2011 Setup Guide

In our Setup Guide at step 24 as of this writing we outline how we go about patching our SBS servers:

• Update to the latest SBS Update Rollup first.
• Run updates according to the following product groups:
• Windows Server 2008 Standard R2
  • Run OS Updates at around 10-15 per reboot cycle.
  • Run OS Security Updates at around 5-10 per reboot cycle.
• Exchange SP1/2/3 or Exchange Rollup RU1/2/3/etc 
• .NET
  • If .NET v1 is present update first.
  • Do .NET v2 and v2.x updates one at a time.
  • Do .NET v3 and v3.x updates one at a time.
  • Do .NET v4 and v4.x updates one at a time.
  • Reboot between each cycle as requested.
• SQL
  • Start with 2005 versions.
  • Next to 2008 versions.
  • Next to 2008 R2 versions.
• SharePoint Foundation, WSUS, and others.

One of the things to be mindful of is which updates are selected when opening the Windows Update console for the first time. We are seeing that while most updates are selected by default some _are not_.

Why are they not selected?

To hazard a guess probably because there are patches that need to be in place prior to their being applied.

So, take note of any patches not selected and run them last.

We follow the above patching guideline religiously. For a server that may have 20, 30, or more patches waiting our method could mean tacking on an extra hour or more of time to get that server up to date.

We balance the risk versus the reward. Would we prefer to spend some extra time running updates on the server or take even more time restoring the server from backup because something blew up?

And, in this case out of all the update runs we have done over the years we have only had one bad patching victim on SBS 2008. SharePoint service pack 1 killed the Companyweb on a newly installed server.

From what we understand .NET updates somehow compile themselves on the fly. This is one of the causes of a long update cycle. Note that we update .NET patches in family first and only two or three at a time!

On SBS 2011 note that there is an extra manual step for SharePoint Foundation updates!

Make sure to run that command afterwards.

• Official SBS Blog: You Must Manually Run PSCONFIG after Installing SharePoint 2010 Patches

The command:

• PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures

Remember to run a backup of the server before starting any updates (yes, in between too!) and be prepared to restore that server!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

SBS 2011 Backup Error: Add Destination Drive Failure – Cannot Configure Backup Schedule

We went to add a destination drive to one of our newly migrated to SBS 2011 servers.

When we ran the wizard after plugging in the new backup drive we ended up with the wizard coughing up a generic failure message: Cannot Configure Backup Schedule.

We opened C:\Program Files\Windows Small Business Server\Logs\SBCW.log in NotePad to find:

[6572] 110627.121259.4347: Storage: Terminating error in Set-WBPolicy: System.Exception:
The filename, directory name, or volume label syntax is incorrect ---> Microsoft.SnapIns.Backup.UI.Proxy.BlockLevelBackupException: IBlbEngine::ModifyTemplate failed due to error: The filename, directory name, or volume label syntax is incorrect, (0x8007007b)..

We searched on that error and came up with:

• Microsoft TechNet Forums: Dreaded "Cannot Configure Backup Schedule" on SBS 2011

That in turn led us to this Microsoft KB:

• Microsoft KB 2009365: Error message when you try to add an additional disk to a scheduled backup: "The filename, directory name, or volume label syntax is incorrect"

We are presented with three options:

1. Attach the original backup destination.
• In this case we only have one drive dock so no go.
2. Remove the original backup destination altogether.
• Live disk for rotation so no go on this one.
3. Manually add the new disk using WBAdmin.
1. This was it.

The how-to from the KB:

1. Run the following command from an elevated command prompt to determine the Disk Identifier of the new disk:
        wbadmin get disks
2. Based on the output, locate the disk that will be added to the scheduled backup. Make a note of the Disk Identifier. The output will resemble the following:
   Disk name: xxxxxxxxxxx
   Disk number: x
   Disk identifier: {xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}
   Total space: xxx.xx GB
   Used space : xxx.xx GB
3. Run the following command to add the new disk to the Scheduled backup.  Use the Disk Identifier from the previous step as the "AddTarget" parameter.
   WBADMIN ENABLE BACKUP -addtarget:{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}
   
4. When you receive the following prompt, type Y for Yes.

"Do you want to enable scheduled backups with the above settings?"

Now, there were a few more prompts after point 4 that we said Yes to.

Of course the new disk now had the Windows Server Backup label so we changed that using Disk Management. But, the status under the Backup tab in the SBS Console still showed the WSB label.

For now, we have that additional disk in the rotation with our first backup to it successful:

We will keep this backup rotation short and introduce another drive at the end of the week so we can bring this one back to the shop and do a test restore. We have not yet needed to restore a backup from a destination that was incorporated into a backup routine via this method as of yet.

We _always_ want to know if a backup is good for a full bare metal restore. The only way to know is to actually restore it! :)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

P2V Using ShadowProtect V4.x – Quicker But with BSOD 0x0000007B for Windows Server 2003 R2 in Hyper-V

When we were running P2V processes using v3 of ShadowProtect we were waiting quite a while as the ShadowProtect Recovery Environment did not work too well on the Hyper-V side.

ShadowProtect version 4 uses the Win7/Win2K8 R2 codebase for its Recovery Environment so we get full keyboard and mouse functionality while working via RDP plus a pleasant 20MB/Second as shown above.

The P2V we were running was for a legacy Line of Business application running on Windows 2003 Standard R2 that cannot run on a current OS. Not only that, the LoB does an awesome job for what is does and there is no current application out there that comes even close.

On first boot after doing a straight restore we hit a BSOD: STOP: 0x0000007B

A problem has been detected and Windows has been shut down to prevent damage to your computer.

Nothing in our searches turned up anything really useful other than a similar error indicating an inaccessible boot device.

• Microsoft KB 324103: Advanced troubleshooting for "Stop 0x0000007B" errors in Windows XP

Since we did not do an HIR for this P2V we went back and ran the restore process again after extracting the Hyper-V Integration Services as explained here:

• winDeploy Blog: Extracting Hyper-V R2 Drivers

The commands are as follows:

1. msiexec /a d:\support\amd64\Windows5.x-HyperVIntegrationServices-x64.msi TARGETDIR=C:\AIPx64
2. msiexec /a d:\support\x86\Windows5.x-HyperVIntegrationServices-x86.msi TARGETDIR=C:\AIPx86

Once we had our x86 version drivers we were able to load all of the Integration Services drivers using the Hardware Independent Restore feature of ShadowProtect. About 20-30 minutes later we had our server back online.

We then ran through the Integration Services install via the Action drop down menu to load the ISO in the VM.

All was good to go from there.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

ShadowProtect 4.0.1–Windows 7 Ultimate x64 Restore Good To Go

We supplied a number of new machines into a new client when we did a new SBS 2008 install not too long ago.

One of the machine’s hard disks was causing some flaky behaviour and throwing Disk errors in the Event Logs.

We installed ShadowProtect 4.0.1 Desktop edition on the machine yesterday and set up a scheduled backup to run every hour to a network share.

When we arrived this morning we ran a final Incremental backup prior to shutting down the system. We swapped out the bad drive for a new one and then booted off of our ShadowProtect IT Edition’s USB flash drive.

Once we created the necessary disk signature, rebooted, we were able to restore the 100MB partition that Windows 7 creates and then the actual system partition.

There was some initial concern because that 100MB partition normally does not have a drive letter associated with it plus the restore process flagged a bad BCD Database restore in red.

Once we ran through the actual system drive’s restore process there was a note in the Details pane that showed that the BCD Database was edited and successfully corrected.

We then closed ShadowProtect IT Edition and the system rebooted. It complained about not being shut down properly, but Windows 7 Ultimate did indeed boot up without a hiccup.

A quick check in Disk Manager showed that 100MB partition as not having a drive letter and everything was good to go!

Total time to swap out a bad drive and restore the system on an Intel Core 2 Duo E8500 on an Intel DQ45EK Mini-ITX motherboard with a 320GB 7200 RPM Momentus was about an hour and 10 minutes. A full OS install, driver install, updates, and application install would have been about 3-4 hours.

Note that ShadowProtect must be installed and configured on a Backup Schedule in order to have access to creating Incremental backups. A manual backup (Backup Once) will not allow for any incremental backups.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

ShadowProtect – Garbage In = Garbage Out

The situation we are in is not the fault of ShadowProtect specifically, but one that we have encountered before where the system’s disks were starting to fail and the backup was image based. Thus, any image backup software can be found to have bad data within the image.

We are in the process of finalizing an SBS 2008 server setup for a new client on-site. Their “server” system was experiencing spontaneous reboots with Check Disk running on each start up. This morning the system refused to come back up.

We had set up a ShadowProtect backup on that machine to help protect the data that was on it.

However, as the above screenshot of a BeyondCompare data copy out of the mounted ShadowProtect image shows us, if the data going in is bad the data coming out will be bad.

We experienced a situation like this at an unprecedented level a few years ago where a client’s very large RAID 5 array decided to start dropping sectors on the drives. We ended up recovering that SBS domain using the Swing method and the data through a combination of backup recoveries across the two servers at that site.

In this case we have no other fallback methods other than some of the users may have a copy of the corrupted files on the own systems since the central “server” has been behaving quite flaky for a while now.

So, we will be mounting the ShadowProtect backups that have been running on the flaky system for a couple of weeks now to see if we can find any of the now corrupted files that still may be in good shape.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Intel Modular Server – Promise VTrak e310sD Has Arrived

A client that is running a three node Hyper-V Server 2008 R2 cluster on an Intel Modular Server (IMS) needs to have _a lot_ more storage configured for their highly available VMs.

Their current SPF file server is dying a harsh death and causing all sorts of problems with folder redirection and share availability.

• MPECS Inc Blog: Hyper-V Server 2008 R2 OS Storage Usage And Our IMS Storage Group Structure

In our initial investigations into augmenting storage for servers and/or clusters running on the IMS we ended up focusing in on the Promise VTrak series of direct attached storage units as indicated in the above blog post.

Well, our first VTrak unit, an e310sD 12 bay with dual controllers,  has arrived here at the shop. We are waiting on the 12 300GB 15K.7 Seagate SAS drives and the two Adaptec external mSAS cables that should be showing up tomorrow or early next week.

Once we have everything together, we will look to getting our own IMS fired up ASAP as we will be testing a few configurations on our own systems before looking to deploy at the client’s site.

Client IMS Storage Configuration

The client IMS in question is configured similarly to ours in that we have all 14 drive bays in the IMS configured with 146GB 15K Seagate Savvio SAS drives.

1. Storage Group 1: 2 physical disks – Low I/O Requirement
1. H-V 2K8 R2 OS volumes for nodes
2. Single volume for H-V configuration files (VM memory files etc).
2. Storage Group 2: 4 physical disks – Medium I/O Requirement
1. TS VM Remote Apps installation and other data volume
2. SBS VM secondary storage
3. Desktop OS VM volumes.
3. Storage Group 3: 6 physical disks – High I/O Requirement
1. SBS VM OS, Exchange, and other volumes
2. SQL VM OS and database volume
3. TS VM OS volume

The Promise VTrak will be used to add some high capacity medium I/O performance storage for the cluster to allow for the company’s redirected folders as well as folder shares to reside as an attached volume within the highly available SBS 2008 OS VM.

Hyper-V VM IDE Limitation

Keeping in mind that we only have 4 IDE connections to work with in a Hyper-V based VM, we usually only have 2 VHDs IDE connected at a time in a production scenario.

One IDE connection hosts the VHD OS volume, one hosts the VHD with data sets, and one IDE connection remains open.

If there is a need to recover the VM, we can attach a remote VHD to the open IDE connection and it will be seen by the backup software as a source to restore from.

This configuration leaves one IDE connection left over for the Hyper-V “optical” drive that is always connected.

VM Backup Destination VHD

Note that the remotely accessed VHD is actually connected via the Hyper-V SCSI bus which is hot swap capable in Hyper-V Server 2008 R2 thus allowing us to “rotate” the backup VHDs. Backups to those VHDs are encrypted using ShadowProtect.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

ShadowProtect Restore Of Windows 7 To Hyper-V Tips And More

We jumped the gun on the last post.

Using the Hardware Independent Restore feature of ShadowProtect we were able to get some Windows 7 OSs into Hyper-V via P2V along with some troublesome Windows Server 2003 OSs.

We needed to extract the .MSI contents of the VMGuest.iso into individual folders and ISO them. We inserted those drivers using the HIR feature of ShadowProtect and were greeted with the Windows 7 logon screen not too long afterwards.

The post is here:

• ShadowProtect Restore to Hyper-V VM Tips

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

ShadowProtect Restore to Hyper-V VM Tips

Here are a couple of tips to get the restore working relatively quick:

1. Use the Legacy NIC in Hyper-V if the restore files are on a network share.
2. Use only 1 virtual CPU to run the restore.
3. Use the Integration Services drivers as part of a Hardware Independent Restore for the guest OSs:
1. Extract the contents of the VMGuest.iso to an architecture specific folder.
1. msiexec /a Windows5.x-HyperVIntegrationServices-x86.msi /qb targetdir=c:\temp\x86
2. msiexec /a Windows5.x-HyperVIntegrationServices-x64.msi /qb targetdir=c:\temp\x64
2. Use a utility like CDBurnerXP to place the extracted content into an ISO file.
3. Mount that ISO file when it comes time to run the restore in SP.
4. Install _every_ driver that is in the \Hyper-V Integration Services folder for the required architecture.
• 
5. Boot into the OS and clean things up HIR style (previous blog post).

The WinRE used by ShadowProtect will not pick up the Hyper-V NIC by default. So, run the restore with the Hyper-V legacy NIC in place. Once the restore is complete, shut down the VM and switch out the Legacy NIC for the Hyper-V native NIC.

This is what a restore looks like with 2 virtual CPUs for throughput:

This is what a restore looks like with only 1 virtual CPU:

So, somehow having anymore than 1 virtual CPU enabled causes a real problem with not only the restoration process, but navigating around the ShadowProtect recovery GUI too. Things are a lot more responsive with just 1 vCPU.

Using the above procedure will yield the following on a Windows 7 P2V:

The 100MB system partition that Windows 7 creates during its install routine is not needed to restore the OS as a Hyper-V guest. We needed to use the Hardware Independent Restore process to get beyond a BSOD 0x0000007E.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

SBS 2003 to SBS 2008 Swing Migration

Up to now we have been using the Microsoft method for migrating existing SBS 2003 to SBS 2008. For all intents and purposes, this method has worked quite well for us.

We are now in a position with one of our clients, yesterday’s recovery (previous blog post), where we need to expedite their migration, but need to do so with minimal interruption to their operations and our own.

So, we are looking to run a Swing Migration for this particular case. We are doing so because the Swing method allows us to run most of the migration here in the shop while our client remains totally productive.

This in turn allows us to work on the processes as time permits and then insert the newly migrated SBS 2008 into their production network with minimal impact on their productivity and minimal need for on-site time.

Click on the Swing Migration category (blog category link) to read about some of our experiences with the method.

From migrating SBS 2003 onto new hardware or off existing hardware and back on again, the Swing method is an awesome way to bring about change on a production network with minimal impact on our clients.

The Swing method has also been instrumental in providing the resources and abilities to bring back SBS catastrophic failures seemingly from the dead and most importantly providing us with the _confidence_ to bring those networks back.

We strongly recommend that any IT technician working with SBS have an SBSMigration.com subscription to augment the needed skills to migrate and recover SBS networks.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

SBS 2008 Backup Recovery To Hyper-V OEM Services Caveat

One of the services we offer to our clients is the rotation of their backup drives along with a quarterly test restore of the full server to Hyper-V or one of our lab servers.

When restoring to Hyper-V, one of the things to watch out for on that first boot post recovery are runaway processes. In most cases those processes will be the server manufacturer’s management software searching for the server hardware.

The above screenshot is the Services.msc console showing the HP services that we needed to set to Manual before rebooting the now recovered SBS 2008 VM.

When the server comes back up after the reboot, the services running on the server will behave as expected . . . hopefully! :)

Depending on the complexity of the SBS 2008 setup, a full restore to production levels should not take more than 4 hours with 2 virtual CPUs, 6GB RAM, and the appropriate storage volume(s) applied.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Windows Server Backup Recovery Fail – A Handle to the driver control device could not be created. . .

We needed to spot recover some files on one of our client’s servers.

But, when we plugged in the required backup USB hard drive and ran the Recovery Wizard from within the Windows Server Backup (WSB) node we ran into a really strange error:

Recovery Wizard

Unable to browse ServerName (C:). A handle to the driver control device could not be created. Make sure the driver is installed correctly.

Now, a search for the error message turned up very little in the way of results.

Somehow, in the numerous searches we attempted on the problem, the following site held the clue:

• Microsoft Forums: Can’t install backup feature on Server 2K8 Standard.

In the above linked forum post SriramB[MSFT] runs the questioner through some steps to get WSB installed. But, the clue for us was found because of the following found in the Device Manager after revealing hidden devices:

We right clicked on the File as Volume Driver and Uninstalled it. We then right clicked on the root computer name and clicked on Scan for hardware changes and the File as Volume Driver got picked up:

File as Volume Driver

Device driver software installed successfully.

Once the driver was installed, we saw this in our Recovery process:

The + NetworkData (L:) volume was able to be browsed for the content we were looking to restore.

We are not entirely sure what caused the driver to get knocked out in the first place since the backup portion of WSB was working just fine.

This is one reason why regular spot recovery as well as a full recovery tests should be a mandatory part of any managed services offering. If the backups are not tested, then no one will know they were broken until there was a real emergency.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

RAID Controller Log: Unrecoverable Medium Error – Puncturing Bad Block?!?

Okay, so this is a new one and lead to a near stoppage of the heart last evening:

The errors in order:

Controller ID: 0 Unrecoverable medium error during rebuild: PD –|—:0 Location 0x26f1640

Controller ID: 0 Puncturing bad block: PD –|—:0 Location 0x26f1640

Controller ID: 0 Puncturing bad block: PD –|—:1 Location 0x26f1640

PD 0 is the last original disk in this server that is giving us headaches. Both PD 1 and PD 2 (there are three hot swap bays in the SR1560SFH Intel Server System) were replaced with new drives.

The original PD 1 had failed during a server firmware including BMC update (previous blog post). The original PD 2, the then global hot spare, was rebuilt into the array with no errors . . . until a consistency check that ran later that afternoon produced some unrecoverable fatal errors.

Last night we dropped original PD 1 out of the configuration, replaced it with a new drive, had the new drive picked up as a hot spare. We then failed out the original hot spare PD 2 now RAID 1 array member assuming that it was the source of the errors we saw in the consistency check yesterday afternoon.

So, the above screenshot was taken after the new PD 1 was being rebuilt into the array with PD 0 as the source. Needless to say the heart definitely skipped a few beats with visions of index $0 running through my head (previous blog post)!

The rebuild did eventually finish successfully though?!?

We will be going back this evening to fail out the bad PD 0 and replace it with a new drive which will then be designated the new hot spare.

Once the PD 2, currently a hot spare, rebuild into the RAID 1 array has finished, Intel indicated to us that we need to run a consistency check. From there, hopefully ShadowProtect will finally give us a backup!

And one more thing, just what does “Puncturing bad block” really mean?

• Bing Search: Puncturing bad block
• Yields nada.
• Google Search: Puncturing bad block
• Yields: NEC MegaRAID Storage Manager Manual (PDF)

The suggestion in the above NEC linked document is to take the preventative measure and swap out the indicated drive(s) promptly. :)

It looks as though the RAID controller has found some bad sectors on the PD 0 and puncturing means to set those sectors as off limits on both array members.

But part of this whole puzzle is the fact that the RAID controller (Intel SRCSASRB with firmware 470) shows a media error level of 0 for both array members and a predictive failure count of 0 for both members!

Hopefully tomorrow we can rest easy with a backup in hand!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Preparing for the Distinct Possibility of a Server Disaster

Besides being an awesome method for migrating SBS to SBS on the same or different hardware, Jeff Middleton’s Swing method brings about a whole new way of viewing disasters and disaster recovery situations.

Because of our previous experiences using the Swing method in past recoveries, we are in the process of preparing our OS backup-less server for the possibility of an all-out failure.

As mentioned in our earlier post this morning on the missing backup and the previous post to that we had a hardware hiccup on one of our client’s SBS servers. The hiccup is enough to cause ShadowProtect to not be able to take an image of the OS partition.

So far, despite the RAID controller throwing the fatal error, no NTFS 55 errors have shown up in SBS’s Event Logs. Hopefully this means that the bad section on the drive is in an empty area. We will be flipping out the array members this afternoon and rechecking the consistency to see if the error has disappeared.

Array member number one’s status:

Array member number two’s status:

Note the distinct lack of error counts for both drive’s media as well as a zero for the Predictive Fail Count on both drives. This is indeed a real puzzle.

So, here are a few of the things we are doing to prepare for the possibility of the main SBS crashing:

• We have moved the live Exchange private and public stores to the data partition that is being backed up.
• We have installed Win2k3 using the SBS media into Hyper-V on the second server and using the Swing method have introduced it as a DC on the network.
• We have the previous backup drive with ShadowProtect Images that can be used to restore SBS into Hyper-V thus providing us with a frame of reference for the setup.
• It is a little too far out of date to use as a recovery source though . . . maybe.

The second bullet will thus eliminate a step if we need to do a fresh build of the SBS box and re-introduce it back into the network using the Swing method.

We will be that much further ahead for recovery if this happens between the temporary DC and having the existing server recovered on our own lab Hyper-V box for reference.

In this particular case, our client’s second server also has a mirror of their own client data on it. We will be able to mount the ShadowProtect image from the good SBS backup and copy the changed files over using BeyondCompare.

While we are not in an ideal state with this particular server situation, we are in a position to expedite a recovery in short order thus leaving our client with very little down time.

ExchangeDefender will be very helpful in this situation too. If things do go sideways and the stores take a hit, we will have access to our client’s LiveArchive so that they can keep active with their e-mail.

More to come . . . hopefully a happy resolution with the new replacement drives cleaning things up!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Recovering Server 2003 with Dynamic Disks via ShadowProtect

One of our non-profit clients needed to replace an aging server with new hardware. Since their licensing covered moving the OS to another box, and the legacy app on that box would not run on Server 2008, we used ShadowProtect to move the OS to new hardware.

Unfortunately, virtualizing the legacy setup was not an option or we would have taken that approach.

The source server was set up with a pair of dynamic disks that were mirrored via an OS software RAID.

When recovering a server that uses a set of Dynamic disks for an OS based software RAID, there is an extra step to the image recovery process.

If the server is a member server, as was the case for us, make sure to reset the local admin password on the server and have that change reflected in the SP image before the image is used for a restore.

Once the SP image has been placed on the new drives, or in our case on a new server, we needed to boot to the Windows Server 2003 CD and log onto the recovery console.

At the recovery console we needed to run the following commands:

1. fixboot c: [Enter]
2. fixmbr [Enter]

This will reset the dynamic disks to basic and allow the OS to boot.

Remember to follow the required post Hardware Independent Restore steps (previous blog post) if moving the OS to new hardware as we were doing here.

StorageCraft’s white paper on dynamic disks:

• StorageCraft: Dynamic Disk – Restore from a ShadowProtect Image.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*All Mac on SBS posts will not be written on a Mac until we replace our now missing iMac! (previous blog post)

Windows Live Writer