Wednesday 30 April 2008

Intel SR2400SYSD2, SR1530AH, and SR1560SF Power Consumption

We have an APC BR1500LCD UPS unit that we leave sitting on our bench.

It has the facility to provide live readouts on its LCD panel of various power and UPS events.

One of them is the amount of watts that a device uses while connected to it.

So, to get an idea of what each unit uses for power we plugged each unit into the BR1500LCD on its own and fired it up.

Here are some of the results:

  • Intel SR1560SF 1U with Dual E5440 Quad Core Xeons and 3x 750GB Seagate ES SATA
    • Power Up POST: 205 Watts Peak
    • SBS Boot: 175 Watts on average
    • SBS Online: 145 Watts on average
    • SBS at rest: 110 Watts on average
  • Intel SR1530AH 1U with Intel Xeon X3230 Quad Core Xeon and 2x 750GB Seagate ES SATA
    • Power Up: 175 Watts Peak
    • Server 2008 Core Boot: 125 Watts on average
    • Server 2008 Core online: 80 Watts on average
    • Server off but plugged in: 8 Watts
  • Intel SR2400SYSD2 2U with dual Intel Xeon 3.0GHz HT and 6x 500GB Seagate ES
    • Power Up: 525 Watts Peak
    • POST Initialization phase: 235 Watts on Average
    • Windows Server 2003 scroll Portion of OS startup: 250 Watts on average
    • Windows Server 2003 network and computer startup portion: 205 Watts on average
    • SBS at work: 225 Watts on average
    • SBS at rest: 205 Watts on average
    • Server off but plugged in: 30 Watts
It looks as though the newer technology really shows its efficiency when running full tilt or at a relative idle as can be seen by the above numbers.

Obviously the 2U will consume a bit more power due to the additional hard drives and a more powerful fan setup. However, the older 2U still has a big hit against it in the form of power savings relative to the new 3000 and 5000 series Xeon setups.

In a high density data centre situation, that 50 watts at idle on the newer technologies spread across hundreds if not thousands of servers means a huge savings in power consumption for the server systems themselves as well as the power required to cool the centre.

The neat thing is, there are even better power efficiency enhancements coming down the pipe in future versions of Intel CPUs, the server boards they plug into, and the plug in peripherals. All of this translates into a bit of relief for those data centres already pushed to the limits on their floor space per compute capability, power consumption, and cooling capacity.

For us, it means we have seen a significant drop in our power bill with every NetBurst Xeon server we have replaced with a Core Xeon technologies based server.

That practical experience for us translates into a good selling point for the newer technologies with our clients. A server that saves them money even when they are not in the office is a good thing!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Tuesday 29 April 2008

Hyper-V - Migrating VS 2K5 R2 SP1 XP VMs to Hyper-V

There is a little bit of a process to get that XP Pro VM from Virtual Server 2005 R2 SP1 onto Hyper-V and get the Hyper-V Integration Services installed.
  1. Install Windows XP Service Pack 3 (Reboot required).
  2. Uninstall VS 2K5 R2 SP1 VM Additions.

  3. Reboot.
  4. Install Hyper-V Integration Services.
    1. Cancel the New Hardware Found dialogues.
    2. Update the HAL.

    3. Reboot.
    4. Installation of Integration Services runs as soon as the user logs on.
    5. Cancel any New Hardware Found dialogues as they will hang up the post reboot install.
    6. One final reboot.
When working with the VMs via the Hyper-V management console, one needs to CTRL+ALT+End to send the CTRL+ALT+DLTE key stroke pattern.

There are a few others, but once the Integration Services are installed and configured, things go pretty smooth.

So far, with just a single core assigned to each VM running on the 1U Xeon X3230 based server we are noticing a pretty good performance increase. The OS is Server 2008 x64 Standard Core with Hyper-V RC0 installed and configured.

The out of the box Hyper-V would not work with the Intel NICs in this particular box. We needed to update right away.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Server Core - Move that Swap/Page File - WMI Error Code = 0x80041017

UPDATED 2012-01-17: MPECS Inc. Blog: Server Core–How to Properly Move the Swap File

This one was a really tough find ...

From the Microsoft TechNet Server 2008 Core online documentation on managing Server Core we get (square brackets because Blogger can't seem to interpret greater than or less than symbols properly):

Disable paging file management: At a command prompt, type:

wmic computersystem where name="[computername]" set AutomaticManagedPagefile=False
So, in our Core command line we type on one line (ignore text wrapping):

wmic computersystem where name="MyCoreServerName" set AutomaticManagedPagefile=False
WMI tells us that the command was successful.

On to step 2:

Configure the paging file: At a command prompt, type:

wmic pagefileset where name=”[path/filename]” set InitialSize=[initialsize],MaximumSize=[maxsize]

Where:

path/filename is the path to and name of the paging file
initialsize is the starting size of the paging file in bytes.
maxsize is the maximum size of the page file in bytes.
Okay, no matter how we interpret the above command, we cannot get a success message from WMI.

The errors we were getting the following two errors consistently:

  • No Instances Available
  • Error Code = 0x80041017
    Description = Invalid query
    Facility = WMI
Any amount of searching either turned up a regurgitation of the TechNet article, or out of date instructions that did nothing to help us.

After a number of different searches using a number of different search engines, we finally stumbled on the following on the Server Core blog: Tips and Tricks Vol. 4:

  1. wmic computersystem where name=”[computername]” set AutomaticManagedPagefile=False
  2. wmic pagefileset where name=”C:\\pagefile.sys” set InitialSize=500,MaximumSize=1000
How refreshing. :) The second command worked for establishing an 8-16GB page file on the S: partition and it verifies after the reboot.

We are fortunate that we have blogs to turn to in this day and age. Here is a good example where the documentation for a product is not so good, but the product team's blog documentation is very good.

This is not the first time we have run into poor product documentation for a new Microsoft product. It probably will not be the last. This is truly unfortunate because we then need to waste a bunch of time searching out the answer to something that should be right in front of us in the first place. :(

UPDATE 2008-06-25: For clarification on the above: Assuming that we have created a partition with the S: label:
  1. wmic computersystem where name=”MyCoreServer” set AutomaticManagedPagefile=False
  2. wmic pagefileset where name=”S:\\pagefile.sys” set InitialSize=2500,MaximumSize=10240
The above commands will disable the system managed pagefile and create a new one using the sizes indicated on the S: partition. Note the lack of spaces between the Intial and Maximum size settings.

Also note, if the command continues to fail, reboot after disabling the automatic pagefile management and run the command again. We are finding that the command does not always run successfully after disabling the automatic management.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Monday 28 April 2008

SMB Nation - Up to Date Agenda

For those who will be attending, and are thinking about attending, this weekend's SMB Nation Toronto conference, we now have our finalized agenda:

SMB Nation Toronto Agenda

At this point, it looks like I will be at the following presentations:
  • Saturday
    • Migrating SMB Customers to Managed Services
    • Intro to Response Point
    • Windows SharePoint 3.0 on SBS
    • Automate and Optimize your IT Services Business
    • SharePoint 3.0 Continued
  • Sunday
    • FirstLook at SBS 2008
    • And SBS 2008 Continued
    • Response Point Business Models for SMB Consultants
    • SBS 2008 Opportunity
    • SBS 2008 Experts Panel Discussion
There is the third column indicating a "Hands On Lab". Depending on how things work, I will definitely want to swing in there to check it out.

I hope to see some of you there! It would be good to meet face to face and kibitz! :)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Server Core - Activation Error 0x8007232B

When we went to activate our Open License version of Server Core, we ended up with the following error:
Activating Windows Server(R), ServerStandardCore edition (xxx-xxx-xxx-xxx-xxxxx) ...
Run 'slui.exe 0x2a 0x8007232B' to display the error text.
Error: 0x8007232B
That error has been encountered before.

Server Core: Error 0x8007232B

The error was encountered when we went to activate our Open Licensing version of Windows Vista: Windows Vista - Activation Error: Code 0x8007232b DNS Name does not exist.

In the case of Windows Vista, it was because we installed Vista without a product key.

Well, when installing Server Core, no product key was requested and we are not using the unattend file to install yet.

So, the challenge now becomes, how do we get that key in there and activate the product?

The command we ran to attempt the activation was:
slmgr.vbs -ato
A little searching was required in order to find the right command line commands.

We found them on Sander Berkouwer's blog "The things that are better left unspoken" Activating Server Core.

The commands are as follows:
  1. start /w slmgr.vbs -ipk ABCDE-FGHIJ-KLMNO-PQRST-UVWXY
    • Format of the product key should be exact
  2. start /w slmgr.vbs -ato
If the activation process is successful, one will see:

Product activated successfully.

This process will also switch the Activation process from KMS to MAK which is the preferential activation method for us.

We were now able to go on and discover how to install and setup Hyper-V, configure the built-in firewall for remote management and Hyper-V management, and all sorts of fun stuff that we need to discover at the command line! ;)

The Server Core command line sure is both refreshing and challenging at the same time.

It brings back memories of the challenges of doing things like setting up the AutoExec.bat file for memory management to squeeze the most conventional memory we could (my personal record was 614KB) to get those games up and running.

Other very exciting and fun stuff in the DOS command line were things like getting the TCP/IP networking configuration to actually work and prepping a system for a Windows 3.xx install! :D

Resources for Server Core are out there, but are still rather spotty due to the newness of the OS.

We have one book on the subject so far, which is in the process of being read and evaluated.

Here are some resources on Server Core: Server Core is definitely a neat OS option in Server 2008!

And, it is looking like the same VMs that ran on W2K3 R2 hosting Virtual Server 2005 R2 SP1 are a lot faster on similar equipment in Hyper-V installed on Server Core. Having the option to specify the number of cores per VM is a pretty neat feature and can be very handy for things like hosting server OSs.

More to come and thanks for reading! :)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Hyper-V - Access to network shares for ISOs

When it came to setting up Virtual Server 2005 to allow VMs to access ISOs across the network, one needed to go through and setup constrained delegation on all of the systems running Virtual Server and the server hosting the ISO's shared folders.

Unfortunately, the error in Hyper-V when trying to mount the network located XP Service Pack 3 erred out with a "General Access Denied" message.

We tried the constrained delegation setup with the new Server Core system and that did not work.

This InfoWorld article provided a clue to how we needed to give our VMs on Server Core ISO access on our network: Next-gen virtualization from Microsoft.

For Hyper-V, network access for things like ISOs are simplified somewhat as one need only add the computer Hyper-V is installed on to the shared folder's permissions with read/write access.

Once this is done, we were able to mount the service pack ISO with no difficulties.

This eliminated the need to copy the ISO over to the host machine.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Windows XP SP3 RTM bits available on TechNet

The bits for Windows XP Service Pack 3 Release to Market are available to TechNet Plus subscribers now.

Windows XP Service Pack 3 (x86) ISOs

The download is only available via the Top Downloads section of the site.

Windows XP Service Pack 3 is a requirement for anyone planning on running an XP VM on Windows Server 2008 Hyper-V.

Time to get your bits, read the release notes (RC Version), and start testing!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

New Acer XP SP2 + Office 2007 SBE = 1GB Microsoft Updates!

This has to be a record for us!

Microsoft Update is 1GB!

Included in the updates is Office 2007 SP1, Office 2003 SP3, SQL 2005 Express SP2, and a couple of other larger ones.

This is a new Acer TravelMate 7720 that has been downgraded to XP Pro SP2 with Office 2007 Small Business Edition installed.

The actual download volume may be a lot less depending on how the Office 2007 SP1 download comes down. In the list it is indicated at 697MB!

We have mentioned about our Workbench Bandwidth Saver before, however what we have discovered is that it is only working with XP loads.

Windows Vista seems to have its own way about getting the updates and that way seems to bypass the caching feature of ISA 2000.

So, what that means is we are not plugging in any Vista machines that will be delivered to client sites where they have WSUS installed.

The other option for us is to use the suggestion that Chris mentioned in the comments of the above earlier post: Import the registry settings for our WSUS server and wuauclt /resetauthorization /detectnow until the system has gone through all of the necessary install and reboot cycles.

We place the above wuauclt command into a batch file that is on our Technician's Thumb Drive and copied to the desktop on any system we are installing. A copy of the batch file also resides on the client's WSS Companyweb site along with a share \\MySBS\Company\Network Admin\ folder that we keep somewhere on the network.

This is the registry settings for Windows Vista to import:
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate]
"WUServer"="http://MYSBS-SBS01:8530"
"WUStatusServer"="http://MYSBS-SBS01:8530"
"ElevateNonAdmins"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU]
"NoAutoUpdate"=dword:00000000
"AUOptions"=dword:00000004
"RescheduleWaitTimeEnabled"=dword:00000001
"RescheduleWaitTime"=dword:00000001
"RebootWarningTimeoutEnabled"=dword:00000001
"RebootWarningTimeout"=dword:00000005
"RebootRelaunchTimeoutEnabled"=dword:00000001
"RebootRelaunchTimeout"=dword:0000000a
"DetectionFrequencyEnabled"=dword:00000001
"DetectionFrequency"=dword:00000001
"AutoInstallMinorUpdates"=dword:00000001
"UseWUServer"=dword:00000001
"NoAutoRebootWithLoggedOnUsers"=dword:00000000
"ScheduledInstallDay"=dword:00000000
"ScheduledInstallTime"=dword:00000003
Note that MYSBS-SBS01 would be your own internal or workbench server.

You could always export the settings from one of the Vista workstations on the domain hosting WSUS v3 that is to be used as an update source. If the domain is workbench based, or a new setup for the workbench, and no Vista machines have touched it yet then a vanilla Vista VM would work to at least connect to the WSUS v3 server and get all Vista related updates cached.

Here is a screen shot of a vanilla Windows Vista registry before importing the above settings:

Vanilla Vista install. No Windows Update Key

Once all of the install cycles have completed, delete the above imported registry settings, force a GP refresh, reboot, and the system should connect to Microsoft's update site with no problems.

For those systems that are going to be installed in a peer to peer or stand-alone situation, make sure to run "Windows Update" from the Search field and upgrade the system's update source to Microsoft Update to catch updates for Microsoft Office and the like.

Thanks Chris for the initial pointer. It looks like we will be using this method to update Vista as we go forward. We just need to figure out if we are going to install a separate WSUS v3 instance on the workbench LAN.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Saturday 26 April 2008

SBS Disaster Recovery - Second DC SBS Restore Caveats

The SBS domain recovery we worked on for a client that began last week has presented us with a number of foreseen and unforeseen challenges.

Once we had them online with their backup systems, the goal was to Swing their SBS over to new hardware. In this particular situation, we have a secondary DC installed on the domain.

It was installed for this very reason: To provide Active Directory, DNS, and Internet access along with VPN access to company files if needed.

It was our goal to use the Swing method to introduce a new SBS box utilizing the old SBS box as the starting point for the Swing.

We had also grabbed the most recent ShadowProtect images of the backup DC for any unforeseen needs.

Once the battle to settle the old SBS server down to some form of stability happened, the first thing we did was attempt to join a system to the SBS domain. This is the message we received during the attempt:

Computer Name Changes
The following error occurred attempting to join the domain "mysbsdomain.lan":
The directory service was unable to allocate a relative identifier.
This did not necessarily yield any clues at first.

After a bunch of searching around, the closest thing we could find was at the Experts-Exchange: The directory service was unable to allcate a relative identifier (keep in mind they are now subscription based) which in turn led to this MS KB article: KB248410 Error Message: The Account-Identifier Allocator Failed to Initialize Properly which is for Server 2000 and this one: MS KB article: KB822053 Error message: "Windows cannot create the object because the Directory Service was unable to allocate a relative identifier" also for Server 2000/3.

The KB articles gave us some repadmin tool commands to test things out that lead to some clues as to the source of the problem.

At this point, the old SBS box was plugged into a stand-alone Gigabit switch. The NICs had the appropriate IP setups and teaming, and the Internet NIC is plugged into our Workbench network just in case we need outside access.

We knew that the old SBS box could not communicate with the backup DC. This is a given since the SBS box was sitting in our shop and not the client's site.

However, not having communication with the backup DC should not be a problem right?

So, we figured that since the Backup DC was nowhere to be found, we would try something simple like adding a user on the SBS box itself via the wizard.

We ran the Add User Wizard.

This is the message we received:
You must be a member of the Small Business Server Administrators or Power Users group to create computer accounts. Contact your administrator.
Oh. So, we tried the Add Computer wizard and received the same message. But, we were logged in as the domain admin.

In the SBS Console however, we were able to open ADUC and make changes to object properties or GPMC and modify policies. So, this at least confirmed that we were into the server with a domain admin account and our domain admin privileges.

A more detailed search into the SBS event logs brought us to this log entry:
Event Type: Error
Event Source: SAM
Event Category: None
Event ID: 16651
Date: 4/19/2008
Time: 1:56:52 PM
User: N/A
Computer: MYFailed-SBS01
Description: The request for a new account-identifier pool failed. The operation will be retried until the request succeeds. The error is " The requested FSMO operation failed. The current FSMO holder could not be contacted."
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
The error does not make sense since the SBS box holds all FSMO roles. There were consistent NTDS KCC warnings in the logs too:
Event Type: Warning
Event Source: NTDS
KCC Event Category: Knowledge Consistency Checker
Event ID: 1308
Date: 4/19/2008
Time: 2:01:32 PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: MySBSServer
Description: The Knowledge Consistency Checker (KCC) has detected that successive attempts to replicate with the following domain controller has consistently failed.
Attempts: 31
Domain controller: CN=NTDS Settings,CN=MyBackupDC,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=MySBSDomain,DC=LAN Period of time (minutes):
6902
The Connection object for this domain controller will be ignored, and a new temporary connection will be established to ensure that replication continues. Once replication with this domain controller resumes, the temporary connection will be removed.
Additional Data Error value: 8524
The DSA operation is unable to proceed because of a DNS lookup failure. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Again, these errors are to be expected as the backup DC was not online.

By this time we were firing up a Xeon 3070 based box to do a Hardware Independent Restore of our client's backup DC as it was looking like we were going to need it.

Finally, about an hour later, there was the final clue to the mess we found ourselves in:
Event Type: Warning
Event Source: NTDS Replication
Event Category: Replication
Event ID: 2092
Date: 4/19/2008
Time: 2:56:32 PM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: MySBSServer
Description: This server is the owner of the following FSMO role, but does not consider it valid. For the partition which contains the FSMO, this server has not replicated successfully with any of its partners since this server has been restarted. Replication errors are preventing validation of this role.

Operations which require contacting a FSMO operation master will fail until this condition is corrected.

FSMO Role: DC=MySBSDomain,DC=LAN
User Action:

1. Initial synchronization is the first early replications done by a system as it is starting. A failure to initially synchronize may explain why a FSMO role cannot be validated. This process is explained in KB article 305476.

2. This server has one or more replication partners, and replication is failing for all of these partners. Use the command repadmin /showrepl to display the replication errors. Correct the error in question. For example there maybe problems with IP connectivity, DNS name resolution, or security authentication that are preventing successful replication.

3. In the rare event that all replication partners being down is an expected occurance, perhaps because of maintenance or a disaster recovery, you can force the role to be validated. This can be done by using NTDSUTIL.EXE to seize the role to the same server. This may be done using the steps provided in KB articles 255504 and 324801 on http://support.microsoft.com.

The following operations may be impacted:

Schema: You will no longer be able to modify the schema for this forest.
Domain Naming: You will no longer be able to add or remove domains from this forest.
PDC: You will no longer be able to perform primary domain controller operations, such as Group Policy updates and password resets for non-Active Directory accounts.
RID: You will not be able to allocation new security identifiers for new user accounts, computer accounts or security groups.
Infrastructure: Cross-domain name references, such as universal group memberships, will not be updated properly if their target object is moved or renamed.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
We were still nowhere near having the backup DC restored on our box here in the shop. So, we created a VPN connection to the production backup DC and forced a replication across the VPN.

At first we were expecting the replication to take a while, but it was relatively quick, and we now had a viable SBS DC to work with.

We have now learned SBS Recovery with secondary DC valuable lesson number 1: When needing to recover an SBS DC that has other DCs in the SBS Active Directory forest, we need one of those DCs for the initial replication. Given that our recovered SBS DC was in good shape at that point, replicating with the production backup DC was okay to do.

However, if things were a lot more messed up, then the only option would be to have a recovered version of one of the other DCs attached to the recovered SBS' isolated network and forcing replication with it or using it as the source for the Swing Migration.

As soon as we saw a successful KCC message in the logs, we ran the Add User Wizard and sure enough we could create users and computers again.

Then came the sigh of relief. :)

Okay, so through the Swing steps we go to establish a new SBS instance on the new hardware we have in the shop.

We ran into a few initial hiccups in the Swing process, but they were relative to some the methodology itself.

Once we had the new SBS server finished, we delivered it to our client's site very early this last Tuesday morning. The intention was to bring it online while everyone was not in the office.

We shutdown DHCP on the backup server, reconnected the internal network cable to the box's second NIC, teamed them back up, and reset the IP and DNS settings on the team.

The new SBS box and the backup DC were not very happy to see each other at first. Replication failed either way.

Since they were not wanting to replicate, we needed to work with the highest priority which was to get the client machines moved over to the new SBS box. We created a startup script to do the following:
  • ipconfig /release (remove the IP settings given by the backup DC)
  • ipconfig /renew (reestablish IP settings to the new SBS box)
  • net use g: /delete (SBS Company folder)
  • net use h: /delete (Backup Company folder)
  • net use g: file://mysbsserver/company (data now online)
  • net use h: file://backupdc/companybu (now read only)
  • gpupdate /force (forces the client machines to pull GP from the SBS box)
The last step was critical for bringing things back together. We made sure all of the client machines that were online once we reached this point were sent into a reboot via a shutdown -r batch file on the server. We logged on as our test domain user account to verify share, Outlook, and Internet access. The ISA Firewall Client was connected and everything seemed to be working as it should.

We then made sure that the users that brought their laptops in that morning understood that they were to answer yes to the reboot question that came via the GPUpdate.

Their Office 2003 which is distributed via Group Policy Software Installation ran again causing a slow down on the initial boot.

But, once they were connected, their data and shares were as they expected them, and their Outlook was connected to Exchange and happy. Email was moving as it should.

We ran into a problem getting the new SBS DC and backup DC to replicate however.

The SBS box happily picked up the proper settings both in DNS and Active Directory to replicate with the backup DC. But, the backup DC would have nothing to do with the new SBS box.

In a way, this is expected behaviour given the new SBS box will have a totally different underlying identity to the original SBS box. We followed the entries in the following TechNet article: Troubleshooting GUID Discrepancies.

This is what we see from the NTDS properties of an SBS box and the corresponding AD DNS entry:

NTDS DNS Alias and its corresponding entry in _msdcs.mysbsdomain.lan

When we had a look in _msdcs.mysbsdomain.lan on the new SBS box and there were indeed multiple entries for the old SBS box, the new SBS box, and the existing backup DC.

DNS on the backup DC also had multiple entries, so we cleaned out the wrong ones and ran replication again. Still no go. The new SBS server happily tried to connect to the backup DC, but the Backup DC would not connect to the new SBS box. We were getting Access Denied messages in the logs whenever we forced replication or it ran on its own.

Another clue was in the fact that we could not access any shares or network resources on the new SBS box from the backup DC but we could the other way around.

Looking into the ServicePrincipalName cleanup suggested in the above article, we made the necessary changes.

ServicePrincipalName Cleanup: Remove the old entries, paste the new GUID in place, and Add

There are two entries in the SPN that needed to be changed.

After the cleanup and a reboot of the backup DC, they still would not replicate. That GUID alias in the NTDS properties under DssSites.msc would not change to the new SBS server's GUID. It still had the old SBS server's GUID there.

Given the amount of time we were fighting to get them to replicate by this point, we decided to DCPromo the backup DC to demote it and DCPromo it back in again.

That failed!

From the command line we had to DCPromo /forceremoval on the backup DC to get it to demote. That worked.

But that still left us with the new SBS server and all of the backup DC references in Active Directory. However, we knew that would be the case as the Swing Migration steps prepared us for what was next: Utilizing NTDSUtil to perform a metadata cleanup of the backup DC settings, and a cleanup of any reference to the backup DC in ADSIEdit.msc. We also needed to clean up DNS of any reference to the old backup DC's GUID.

We doubled back over our work to make sure there were absolutely no AD settings left for the backup DC. Once satisfied, we DCPromoed the backup DC back into the AD forest.

After a reboot, they were both happily replicating!

We have now learned SBS Recovery with secondary DC valuable lesson number 2: When we go to reintegrate a newly installed SBS server that was a part of a disaster recovery process, we may need to demote any and all secondary and tertiary DCs.

In this case, the secondary DC was in the same office, but, in the case of a branch office scenario where there are a couple of other offices out there, this could present a real rats nest to get things up to speed AD replication wise.

Now, what we experienced could very well be an anomaly where the edits done to the SPNs on the other DCs may in fact take and the issue stops there. They go on to replicate with no further issue.

For your reference: This was one of the more challenging disaster recoveries we have had to face yet.

We have been very fortunate that none of our clients have totally lost a location, but we came close once with one of our clients where the entire building's roof rained a deluge of rain water into a server closet of one of our clients at 03:30 in the morning. That was a scary call. It seemed that building maintenance has not gotten around to cleaning out the roof drains and the drain just above the closet was the one to give way. :(

Since we have both the old SBS and backup DC up and running and replicating happily here in the shop, we will be running a couple of test Swing Migrations to see if that second DC causes problems in a nondisaster recovery SBS domain migration too.

The step after that will be to see how the new Server 2008 Active Directory schema extensions for a Read Only DC at a branch site impacts our SBS 2003 to 2003 and 2003 to 2008 migrations.

Thanks for reading! :)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Vista - Enabling UAC with Credentials for Local Admins when managing Server 2008

The Remote Server Administration Tools for Windows Vista SP1 is available: Validation is required for both downloads. Once they are downloaded, both should be placed on the Technician's Thumb Drive for later installation.

When running the admin tools, we need to be prompted by UAC for credentials so that we can authenticate in the management console as domain admin.

If, however, we are working on a Windows Vista workstation where the user account has local admin priviledges, then the UAC will only prompt to continue on into the admin consoles.

This will not work for managing servers as that elevation is local admin only.

We need UAC to prompt for credentials every time. To do this we do the following:
  1. Start
  2. GPEdit.msc [Enter]
  3. Navigate to: Local Computer Policy --> Computer Configuration --> Windows Settings --> Security Settings --> Local Policies --> Security Options:

    • UAC Prompt: Behaviour for admins

  4. Change the default from "Prompt for Consent" to "Prompt for credentials"

    • UAC Behaviour Options

  5. Click Apply and OK
  6. Close the Local Group Policy Editor
One should now be prompted for credentials every time a UAC is presented.

This setting is machine specific, so any users of the machine will need to enter their domain credentials if they are presented with a UAC prompt. Keep this in mind for systems that are used when at the client site for server management.

Thanks to Josh's Windows Blog for providing us with the tip: Windows Vista Tip: Run as administrator.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Friday 25 April 2008

Intel SRCSASRB on Intel SR1560SF/S5400SF Series requires PS/2 Keyboard

During the first phase of the SBS setup, there is a point right around 34 minutes where the USB ports are knocked out.

That is also right around the time that the Intel SRCSASRB RAID card drivers will have a WHQL warning pop-up message that one will need to acknowledge.

If there is only a USB keyboard and mouse installed on the server, one will not be able to click on the Continue button. One will need to reset the server and plug in a PS/2 keyboard or mouse to get to the pop-up window.

Once the pop-up driver windows have been acknowledged, the setup can proceed without the need for any further input via a PS/2 keyboard.

This situation is one reason to keep a PS/2 keyboard around besides the need for one on KVMs.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Intel DX48BT2 - 1600MHz DDR3 Memory Limit Caveat

Intel's new Bonetrail board, the DX48BT2 supports the Intel Quad Core Extreme QX9770 3.2GHz processor with a 1600MHz (MT) Front Side Bus.

If one wants to build a system on one and utilize 1600MHz memory, there is a 4GB (2x 2GB) installed memory limit.

1333MHz memory can be installed in an 8GB (4x 2GB) configuration.

Keep this in mind if quoting on a new Extreme based system.

From Intel's support site:
System Memory Features
The board has four DIMM sockets and support the following memory features:


  • Dual or single channel memory configurations
  • Four 240-pin Double Data Rate 3 (DDR3) SDRAM Dual Inline Memory Module (DIMM) connectors with gold-plated contacts.
  • 1600/1333/1066/800 MHz DDR3 SDRAM interface
  • Unbuffered, non-registered single or double-sided DIMMs
  • Non-ECC DDR3 memory
  • Serial Presence Detect (SPD) memory only
  • Up to 4.0 GB utilizing 512 Mb or 1 Gb technology
  • Up to 8.0 GB utilizing 1 Gb technology
Notes:

To be fully compliant with all applicable Intel® SDRAM memory specifications, the board should be populated with DIMMs that support the Serial Presence Detect (SPD) data structure. If your memory modules do not support SPD, you will see a notification to this effect on the screen at power up. The BIOS will attempt to configure the memory controller for normal operation.

For 1600 MHz memory, only two DIMMs are supported for a maximum of 4 GB utilizing 1 Gb memory technology. DIMMs should be configured as a matched pair equal in speed and size in DIMM 0 and DIMM 1 of channel B.
Emphasis on the last paragraph is ours.

In the Tested Memory list further down the page, no 1600MHz memory shows up yet.

Kingston's site does not list the DX48BT2 board with Kingston's available memory products either.

This may be one case where a newer revision of the board will support 8GB DDR3 running at the 1600MHz front side bus speed.

Some product related links: Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Thursday 24 April 2008

Post SBS Migration - Outlook RPC/HTTPS - change that certificate

After completing a Swing Migration, one thing to remember after the new SBS server is in place: Users that use Outlook to connect to Exchange on their laptops via RPC/HTTPS will no longer be able to connect.

This is the case for those SBS installations that are using the self-signed certificate. One should not encounter this problem after importing their Internet trusted certificate.

If the user visits the Remote Web Workplace in IE, the SSL lock will show, and nothing will appear out of the ordinary.

IE does not notice that the previous SBS certificate signer is no longer in existence, but Outlook does.

So, we need to remove that certificate:
  1. Open IE
  2. Tools
  3. Internet Options
  4. Centent tab
  5. Click the certificates button
  6. Trusted Root Certification Authorities tab
  7. Click on the myrww.mysbsdomain.com certificate
  8. Click the Remove button
  9. Close
  10. Apply and OK
  11. Restart IE and import that certificate again.
The user will now be able to connect their Outlook while out of the office.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Wednesday 23 April 2008

SBS Disaster Recovery - Finished

Wow ...

After one full week of working on the failed SBS from last week, many attempts to Swing the old SBS onto new hardware, and recover data, we are finally finished.

The new SBS box was installed very early yesterday morning. We brought it online, shifted the client desktops off of the backup server, and verified their data.

Outlook, their shares, printers, and passwords were as they were with the exception of a couple of users that needed to change their passwords while on the backup DC.

A number of years ago, Jeff Middleton, the author of the SBS Swing Migration was up here in Edmonton sharing some of his Katrina Experiences. It was an amazing and very inspiring time for myself personally.

One of the questions he was asked was about introducing a second DC into the SBS network to provide some redundancy for Active Directory. His response as I recall it was something along the lines of, "That introduces a whole new can of worms".

He wasn't kidding. We ran into all sorts of hiccups, hurdles, and road blocks recovering and subsequently installing the new SBS because of the backup DC.

Since there are a number of things sitting here needing attention, some chronicles on the SBS disaster recovery (DR) will be forth coming.

This DR was probably one of the most challenging we have faced to date.

But, with the time, tools, and talent that we have we triumphed! And, most importantly: Our client is very pleased with the outcome.

We were able to salvage Active Directory - meaning no desktop profile impact at all, all of their data, and bring them back over to the new SBS server with a reboot and a couple of extra startup scripts.

More to come ... and a nod to Jeff Middleton: Thanks! Those Swing skills came in very handy with this recovery. :)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

Monday 21 April 2008

SBS Swing Migrations - Using W2K3 VMs during the process

We have mentioned in the past how one can utilize Virtual PC 2007 based Server 2003 VMs to facilitate a smooth SBS Swing Migration.

Using a Windows Server 2003 R2 VM setup can streamline the steps. One can have a W2K3 SP2/R2 base VM install that can be used to copy from and start a new VM instance for the Swing Migration. The SP2 install can be used in the case where the SBS 2K3 Active Directory has not been extended for the new Server 2003 R2 components.

One of the significant benefits to using a Server 2003 VM in the Swing process is the ability to both enable Undo Disks and also to Save State and subsequently copy the VM VHDs after each phase of the Swing.

This is a great way for someone who is new to the whole process to get comfortable with running through the Swing steps and not have to worry about starting from Ground Zero if they make a mistake.

Having the saved VMs also gives us the ability to go back to them and run through the various server components to see the how and where things were modified and/or deleted.

The same is true for those who are Swinging SBS for a client's production domain environment. Having that Server 2003 VM can be a huge time saver if something hiccups or dies during the Swing process or during the final phase of the SBS setup post-Swing.

Note that when we are working with client's SBS production domains, we always create a ShadowProtect image of any of the servers needed for the Swing before we start. This gives us a quick fall back if things don't work out as we expect.

We also tend to take a quick ShadowProtect image of the new SBS install at the various Swing stages to make sure that we do have a fallback in case things hiccup or blow up.

We use one of our Vista lab systems running VPC 2007. Besides the DQ35JOE's on board Gigabit adapter, the system has a low profile Intel PRO/1000 PT Dual Port (EXPI9402PT) server adapter installed as our lab systems are setup in the Antec Minuet chassis.

The JOE's on board NIC is connected to our internal network, and one of the Intel PT's ports is connected to a stand alone dedicated Gigabit switch that the old SBS and new SBS servers will also connect to depending on which stage we are at in the Swing Migration.

The Windows Server 2003 Virtual Machine setup has helped us out during the Swing process on countless occasions.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

SBS Setup and Intel (or other) Adapter Teaming

When the first phase of the SBS 2003 setup (blog post SBS setup checklist) is complete, one goes on to install all of the relevant drivers and configures the partitions for production service.

From there we move the swap file over to its dedicated partition and verify that the hardware and base OS are happy before proceeding on to second phase of the SBS setup.

One thing one should never do is team the LAN facing adapters before running SBS Setup Phase 2.

Remember that one of the questions we get asked once we have initiated the second phase is which adapter is the Internal one?

Since a teamed adapter is virtual in that it is comprised of two or more physical adapters, we run into a problem once SBS setup takes over: SBS setup only wants one physical adapter enabled during the process.

If SBS setup finds any other physical adapters enabled during phase 2, it disables them. The SBS setup routine does not figure out that when it disables the physical adapters, the virtual teamed one will disappear ... but that is exactly what happens.

The only time we will see any indication that there is a problem will be when we see the red X on Server Tools, and once SBS setup is finished with a list of errors including the inability to install SharePoint.

By then it is too late.

Timing wise and stability wise it is expedient to blow away the box and start fresh. Troubleshooting and trying to work around the errors presented is not advisable on a new SBS box.

This is one situation where if this mistake was made, we must choke on the cost of rebuilding that box ... and facing the music if we need to move the SBS install date at the client site.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists

*All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

SBS 2003 R2 - Setup steps and resources - Revised V2.1.1

This is a revision of the original post: SBS 2K3 R2 - Setup steps and resources.

It has been updated to include some new functionality as well as some available feature additions that can be installed during the various stages of the setup.

  1. Install the base OS to the primary OS drive formatted leaving 10 GB unused on the drive during setup. All other drives/partitions should be left untouched at this time.
  2. Stop the continuing setup after the reboot by cancelling it.
  3. In between Phase 1 and Phase 2: Install relevant drivers, format the partitions and drives.
  4. Install the following Windows Server Tools:
  5. Move the Swap File over to the partition created out of the above 10 GB. This partition can be used for the ISA URLCache as well.
  6. Verify all devices are recognized and happy.
  7. Run Phase 2 setup via the Setup icon on the desktop.
  8. We set the MSDE Engine, Users Shared Folders, Client Applications, and Sent Faxes to the same drive/partition.

  9. After the final SBS reboot, enter the Domain Admin account into the DHCP Manager's properties.
  10. Finish the install by completing the ToDo tasks.
    • OPTION: Install Access Based Enumeration for greater folder security. Enable ABE by share as opposed to globally.
    • OPTION: During the CEICW add the following attachment extensions to the Exchange exclusion list: .htm .html
    • OPTION: Revise your user template Quotas as appropriate by right clicking your data drive --> Properties --> Quota --> Quota Entries and change the template volume.
    • OPTION: Download and install .NET 2.0 in preparation for SyncToy.
      • OPTION: Download and install the Microsoft copy utility: SyncToy 1.4.
  11. Install the Premium Technologies with SQL Server 2005 Workgroup Edition followed by ISA 2004 SP1.
  12. Install ISA 2004 via the Premium Technologies CD
    • The CEICW will fail on the initial ISA install. ISA SP2 or later will fix that.
    • Note that during the ISA CEICW one will need to recheck any extra settings at the Web Services Configuration step as it tends to pull them back.
    • Make sure that the Create Certificate is also left at the setting it chooses.
  13. Install the components and updates from the R2 Technologies CD.
  14. Install IE7 for Windows Server 2003 (download link).
  15. Complete the Post Exchange SP2 Must Dos (previous blog post).
  16. Enable Exchange settings: "Display routing groups" and "Display administrative groups" by righ clicking on the Exchange "First Organization" --> Properties and tick both. Apply & OK.
  17. If there are no pre-Exchange 2000 servers on the domain switch to "Native Mode" while in the above Properties.
  18. Look at the client email Mailbox size limits. We tend to set the limit to 1GB out of the box.
    • Under Advanced Management: MySBSDomain (Exchange) --> Administrative Groups --> first administrative group --> Servers --> MySBSServer -->First Storage Group --> Right click Mailbox Store (MySBSServer) and Properties --> Limits Tab.
  19. Update the Exchange registry settings for the Greylist problem that Exchange seems to be having: Exchange may not be retrying! (previous blog post)
  20. Impliment Recipient Filtering to protect against a Reverse NDR Spam Attack (previous blog post) and avoid an SMTP Queue Length error in the inbox.
  21. Install WSS 3.x in side-by-side mode (official SBS blog post on the matter). Why? Because the WSS 3.x and Outlook two way functionality may be required for WSS hosted calendars, or the Recycle Bin may be for those who "accidentally" delete files.
  22. Install WSUS 3.x afterwards (previous blog post on WSUS 3.x install experience).
    • Note that WSUS v2 should be synchronized before installing WSUS v3.
  23. At the command prompt, run the appropriate osql commands to tame SQL's memory usage (previous blog post explanation).
  24. RipCurl the SBS R2 box for Vista and Office 2007 compatibility (previous blog post).
    • ISA SP3 to follow the RipCurl Patch for ISA
  25. Uncheck "Shutdown worker processes after being idle for (time in minutes)" under the Performance Tab for the DefaultAppPool in IIS.
    • This reduces the first visit to Companyweb browser stall in the morning.
  26. Setup any Software installs via Group Policy (previous blog post) making sure to modify the permissions on \ClientApps\.
  27. Create and setup any custom Domain or OU level Group Policy Objects.
    • We remove a domain user's ability to remotely shutdown any system as one example.
  28. OPTION: (May not be needed if you plan to use WSS v3 as principle) Upgrade the Companyweb SharePoint MSDE instance to SQL 2005 (smallbizserver.net article) and enable Full Text Search.
  29. OPTION: Backup the server using your preferred backup method. Ours is StorageCraft's ShadowProtect. Crash the drives. Restore from backup and then deliver.
  30. OPTION: Enable users ability to change their password while working in Outlook Web access (OWA) (previous blog post).
    In the case of the R2 Tech CD, verify that the Service Packs are still current before installing. If not, download and install the most current Service Pack.

    Post Updates:
    • 2008-01-18: Added steps 4a, 4b, and 4c above to download and install the appropriate support tools.
    • 2008-02-15: Added the step to implement Recipient Filtering in Exchange to avoid SMTP Queue Length errors and protect against a Reverse NDR Spam Attack.
    • 2008-03-24: Added IE 7 for Server 2003 download link and install recommendation. Also added the link to SyncToy 1.4 to download and install.
    • 2008-04-19: Added the need to apply a SQL 2005 post SP2 Hotfix
    • 2008-04-21: DHCP credentials required, SQL SP2 references, ISA not installing on more than 4 cores, and IIS setting for Companyweb.
    • 2008-08-11: Added the note that WSUS v2 should be synchronized prior to WSUS v3 being installed.
    • 2009-08-26: Added the option to allow users to change their password in OWA.
    Philip Elder
    MPECS Inc.
    Microsoft Small Business Specialists

    *All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

    Office 2007 OEM Part Numbers

    Here is a list of the common Office 2007 OEM single pack versions we deal with:
    • S55-02515 Office Basic 2007
    • 79G-01153 Office Home and Student 2007
    • 9QA-01757 Office Small Business Edition 2007
    • 269-14071 Office Professional Edition 2007
    We tend to stick with purchasing the singles since the discount per copy when purchasing the three pack is only pennies. For smaller shops like us there is no sense tying up capital in stock that may sit on the shelf for a period of time.

    If purchase volumes demonstrate the need for three packs, then here are the part numbers:

    • S55-02516 Office Basic 2007 (3 Pack)
    • 79G-01154 Office Home and Student 2007 (3 Pack)
    • 9QA-01758 Office Small Business Edition 2007 (3 Pack)
    • 269-14068 Office Professional Edition 2007 (3 Pack)
    Up to date part numbers along with geographically specific ones can be had from the Microsoft OEM Partner site.

    Philip Elder
    MPECS Inc.
    Microsoft Small Business Specialists

    *All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

    Friday 18 April 2008

    QuickBooks 2008 NonMultiCurrency vs. MultiCurrency ... pick MultiCurrency

    We got our "Your Subscription will expire today" message yesterday. Yes, we had been getting warning before, but when we went through the activation process, the product came back as activated without going through the actual activation process.

    So, a phone call into QuickBook's (QB) support line was in order.

    Given a number of client's perilous, and sometimes catastrophic, experiences with QB 2008 non-MultiCurrency version, there was a great amount of misgiving expressed by us to the QB support technician on upgrading.

    She made sure to reassure us that the QB MultiCurrency was actually a step up from the existing QB 2007 product as opposed to the non-MultiCurrency version which is an entirely new product.

    So, we are going to make sure our accounting VMs (XP Pro) are all backed up and the undo disk feature is enabled for them prior to giving it a shot.

    The main thing here is: With all of the problems the new non-MultiCurrency version (now in its 4th update edition) has, go for the MultiCurrency version which will hopefully step things up gracefully as the 05-06, 06-07 step-ups did.

    We will let you know!

    Philip Elder
    MPECS Inc.
    Microsoft Small Business Specialists

    *All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

    Thursday 17 April 2008

    Intel SRCSASRB RAID Controller and SR1560SF Caveat

    Intel's SRCSASRB RAID controller has a catch when it is to be installed on the S5400SF series server board that comes in the SR1560SF series server system: The PBA on the card needs to be D92806-153.

    The MM# on the box needs to be 892598.

    The PCN:

    Intel Product Change Notification: 108057-00

    The above mentioned PCN came to us via the Intel support technician we spoke with on the phone today.

    Now, we have mentioned in the past on a number of occasions how we need to research our system components to see if there were any compatibility issues relative to the revision level of the components.

    It is very important, because in the case of Intel desktop and server boards one runs into the board revision level being too low for a particular processor on a somewhat frequent basis.

    The same is true for compatibility between RAID controllers and a given desktop or server board.

    In the Tested Hardware and OS List document we see the following:

    Intel Server Board S5400SF Successfully Tested with SRCSASRB

    Note the BIOS, BMC, FRU/SDR, and HSC revision levels required.

    At the bottom of the Tested Hardware document we find:

    SRCSASRB Stepping C1 not compatible with Gen2 PCI-E

    Intel's products all of a PBA number on them. In this case, the product we pulled out of the SRCSASRB box had a PBA number of: D928906-152.

    As indicated in the above PCN, we needed to have -153 in order to have the correct stepping for the second generation PCI-E slot that the SR1560SF/S5400SF has.

    But, our box has an MM# 892598, which by the above PCN should have had a -153 in the box.

    We did not however find out about the above PCN and the correct PBA number until after the SR1560SF would freeze during POST with the SRCSASRB installed and the Intel technician supplied us with a link to the PCN.

    It is truly amazing how Murphy's Law finds its way into situations where timing is of the essence! ;)

    Our supplier has shipped us another SRCSASRB to replace this one. Hopefully it is correctly boxed! 8*O

    Product related links: Philip Elder
    MPECS Inc.
    Microsoft Small Business Specialists

    *All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

    Wednesday 16 April 2008

    Images No Good ... Catastrophic SBS Failure ... Now What?!?

    In what turned out to be an SBS catastrophic failure yesterday, we were purposed with installing some fresh hard drives and restoring the OS and data partitions via a ShadowProtect (SP) backup image set.

    Well, things did not go even close to plan. We had hoped that we would be in and out in under 4 hours under optimal conditions.

    It did not matter how many times, which image version we used, or combination of array sizes in the SRCS16's BIOS settings, we could not get a successful recovery. After one successful restore we went into the Recovery Console to run CHKDSK against the troubled partitions. After that, the OS choked on a missing sys file. :(

    Of the times that we did manage to get the SBS booted up, we found a plethora of Event ID 55s in the logs.

    On many of the OS boot attempts we were greeted with:

    Checking file system on H:

    So, it began to look like the corruption ran pretty far back into our backup image sets.

    We all know that hindsight is 20/20! ;)

    So, in hindsight, the most expedient method of recovering this server would have been to SwingIT off the original hardware and SwingIt back onto a fresh install of SBS on that original hardware. Given that we did not know we would end up being on-site for 12 hours making recovery attempts and eventually rolling out the backup DC setup to provide authentication and shares it was not a viable option until well into the wee hours of the morning.

    We now have the go ahead to Swing onto a new server instead of back onto the existing one. Since they are up and running, we took out the old SBS box that has a somewhat stable recovery on it ... though not stable enough for production ... to use in our Swing Migration.

    For now, they are running via the backup DC and data mirror, along with the backup DC providing Internet access via RRAS and a second NIC. It is not ideal as there are a number of network dependent applications that required some fiddling to get working, but at least they are not twiddling their thumbs and loosing money hand over fist.

    This is one scenario where having our client's email setup as follows pays off:
    • MX 100 ispmailserver1.myisp.com
    • MX 50 ispmailserver2.myisp.com
    • MX 25 mysbsmail.mysbsdomain.com
    The ISP email is pulled down to the SBS box via the POP3Connector which is set to 1 hour intervals.

    At least for now they still have access to the outside world via Webmail and the server will get all of their incoming mail when it comes back online. Any critical emails can be BCCd back to themselves for later download.

    While we have tried to keep the impact on our client's business down to a minimum, there have been a number of hiccups before things started to settle down. So, to provide our client some restitution for the lost time, we will provide some of our billable time to them at no cost.

    Since they are our longest running client at close to 10 years now, it only makes sense to have a little give-and-take in the business relationship.

    There are a couple of important lessons here:
    • Test those SP backup images by restoring them
    • Test their durability by restoring them to different hardware
    • Having a second DC can provide an Active Directory source for a Swing Migration in the event of a total SBS failure.
    In our case, we were guilty of not having enough time to run through their more recent image to do some restore tests. This too is another motivation to give our client a break on the otherwise very expensive I.T. week they are having.

    Given the volume of work with this situation, and others, there may be a smattering of blog posts for a while ...

    Thanks for reading and supporting us! :)

    Philip Elder
    MPECS Inc.
    Microsoft Small Business Specialists

    *All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.

    Tuesday 15 April 2008

    SBS, ShadowProtect, and an Event ID 55 NTFS Error ...

    We went in to a client site yesterday evening to finish a warranty swap of some flaky memory sticks.

    Up to the time that the server was shut down last night, there were no indications of any problems with the two RAID arrays, a 500GB RAID 1 and 1TB RAID 5, on the box. They were partitioned as follows:
    • RAID 1
      • 50GB OS partition
      • 430GB ServerData
      • 10GB SwapISACache
    • RAID 5
      • 1TB NetworkData
    The unit is a 3 year old SR2400 with dual 3.0GHz Xeons and 4GB of RAM. We had changed out the hard drives to give the server more storage about two years ago and did a warranty swap out of the Intel SE7520JR2 motherboard (previous blog post) about a month ago twice. The first swap was a struggle to get the replacement SE7520JR2 board to recognize the 4 sticks of Kingston RAM. The second swap was because the first warranty replacement board lost 2GB of RAM on a patch reboot. :(

    Once we had the new Kingston RAM sticks installed, we fired up the box we went straight into the BIOS to confirm that all 4 1GB sticks were there which they were.

    Subsequently, when booting into the OS, the initial Windows Server 2003 scroller kept going and going. While not necessarily a bad thing, after three years, we know how long this particular box takes to boot.

    The heart started to sink at that point.

    Then the kicker:


    Once of your disks needs to be checked for consistency

    Um, this NetworkData partition is a 1TB RAID 5 array! The previous ServerData partition took only a few minutes. For this partition, it looks like we are going to be here for a while ...

    I: NetworkData is 95 percent completed.

    Well, okay ... maybe not ... but then ...

    Inserting an index entry into index $0 of file 25.

    We have all had that one minute seems like an eternity experience when something stressful like this was going on. The amount of time that the above message scrolled on the screen seemed like one even though it may have lasted only 3 or 4 minutes.

    The only thing that kept us from hitting that reset button was the fact that the 4 drives in the RAID 5 array where this was occurring were pinned. Meaning, their lights were on constantly due to disk activity.

    Then a little light in what was seemingly turning into a catastrophic failure:

    Correcting error in index $I30 for file 9377.

    This went on for over 20 minutes.

    Then we faced something one hopes to never face that late at night expecting to pop in and pop out for a quick task ... the proverbial nail in the coffin - a catastrophic failure:

    An unspecified error occurred.
    .

    It is at this point that it has become pretty clear that we were in for the duration.

    But then ...

    Windows is starting up.

    Again ... must resist pushing buttons (best Captain Kirk voice) ... keeping those fingers tied up and away from the power and/or reset on the front of the server. Just in case, we left it alone. And, thankfully, the above screen is what we were greeted with.

    Soon we saw:

    The Active Directory is rebuilding indices. Please wait

    This stage took a couple of minutes. The Initializing Network Interfaces stage took another 10-15 minutes.

    We were eventually greeted with a Services Failed to Start error and subsequently the logon screen.

    *Phew*

    It looks as though the OS partition has made it through this relatively unscathed. The service chokes were for SQL, WSUS, WSS, and a LoB application that had their databases stored on one of the soon to be discovered absent partitions. Exchange had also choked.

    One lesson in all of this: A server may stay up and running almost indefinitely when experiencing a sector breakdown on a disk member or members of the array. To some degree the RAID controller will compensate. However, in our experience, as soon as the server is downed, or rebooted, those sector gremlins can jump out and make their presence known as was the case here.

    Another lesson from this: We keep the Exchange databases on the OS partition for this very reason. If the Exchange databases were on a different partition and/or array and it fails, we loose Exchange and email communication. If we have an SBS OS that boots with a relatively happy Exchange ... the databases intact too, then at least our client will not loose their ability to communicate with the outside world while we would be working on the data recovery side of things.

    Back to this SBS box: Once into the OS, we were able to eventually get into the Event Logs and sure enough, out of the four partitions, the three besides the OS partition were toast.

    Amazing ... simply amazing.

    From the server Event Log:
    Event Type: Error
    Event Source: Ntfs
    Event Category: Disk
    Event ID: 55
    Date: 4/15/2008
    Time: 7:01:32 AM
    User: N/A
    Computer: MY-SBS01
    Description:
    The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume .

    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
    Data:
    0000: 0d 00 04 00 02 00 52 00 ......R.
    0008: 02 00 00 00 37 00 04 c0 ....7..À
    0010: 00 00 00 00 02 01 00 c0 .......À
    0018: 00 00 00 00 00 00 00 00 ........
    0020: 00 00 00 00 00 00 00 00 ........
    0028: 81 01 22 00 .".
    The above Event Log messages were numerous.

    Just in case, we initiated the ChkDsk utility from within the GUI. It too crashed on the two critically needed partitions.

    We made sure that the relevant services that had folders on the ServerData partition were shutdown, and we fired up ShadowProtect to bring that partition back. We were fortunate that this particular partition recovery cooperated and we were able to fire up the relevant services and their LoB app that had the database server logs on that partition too.

    The 1TB RAID 5 array did not cooperate at all. Even after a 10 hour plus SBS OS initiated ShadowProtect restore attempt that we began very early this morning. It failed and caused the SBS server to spontaneously reboot around 10:30AM this morning. This also means no LoB application for now. We are fortunate that it was not critical to the daily functioning of our client's business.

    So, where does that leave us?

    In this client's case, we have a backup DC that also has a live data mirror on it! So, with SBS at least functional, we were able to email users a simple batch file to disconnect the original Company share and connect them to the backup server's Company share.

    On the SBS box, we made sure to restart and verify the services running on the now restored ServerDATA partition, and have left the RAID 5 array partition alone for now.

    The extra expense of having that backup DC/Data Mirror box sitting there has just paid for itself in spades. For this client, we are talking a hit against the firm in the magnitude of $1K/hour for down time. The share switch took a relatively small period of time. If SBS went down totally, the backup server is setup to bring DHCP, DNS, and a secondary ISP gateway online within very short order to at least keep the firm functional.

    If things happened to end up with a nonfunctional SBS OS, we would have also had the option of bringing down one of our Quad Core Xeon 3000 series servers that sits on our bench just for this task: A ShadowProtect Hardware Independent Restore of a client's entire SBS setup. We would bring them back online fully functional on newer, albeit temporary, hardware setup until such time as a new permanent server could be installed.

    Having the ShadowProtect backup setup in place gives us a good number of very flexible options to make sure that there is very little or no impact on our client's daily business operations in the event of a server failure.

    Given the age of this particular 2U system, we are now talking to the partners about a replacement SBS 1U to be Swung in by the end of this week.

    ShadowProtect

    There is definitely one thing that has been made especially clear in the midst of all of this: The last time we experienced a catastrophic failure of this magnitude, we had BackupExec and two 72GB x6 HP Tape Libraries to fall back on. The recovery took a whole long weekend because of the sheer volume of data and struggles with BUE.

    This time around, while the stress levels were and are still high, they were no where near the levels they were at when the last SBS catastrophic failure happened.

    Even if the partners decide on a set of replacement hard drives as a temporary measure until their high season dies down near the end of the summer, we will have them back online with solid storage tonight. Can't say the same for tape and BUE ... especially with the volumes of data we are talking about.

    ShadowProtect gives us the options ... with the hard drive replacement, between StorageCraft's ShadowProtect, and the disaster recovery training (one can connect the dots) one receives doing Swing Migrations, we will be able to do the following:
    • Restore a clean version of SBS from the previous evening's ShadowProtect image
    • Recover the Exchange databases from the ShadowProtect incremental we will do tonight on the SBS partition
    • Forklift those databases into the recovered SBS Exchange
    • Restore the ServerData and NetworkData partition data from last night's clean ShadowProtect image
    • Copy the backup server's live data mirror changes that were made by the client's users back to the SBS box.
    As it turns out, we just received a call back from our partner contact at the firm. They prefer to go with the hard drive replacement until their business slows down this summer to keep things as close to status quo as possible for now.

    We can do that! We have the technology and the skills! ;)

    A big thanks to both StorageCraft for a great product and Jeff Middleton of SBSMigration for the awesome skill set we have gained via the SwingIt! Kit. Without this product and those skills, we would be in a very bad situation getting worse by the minute ... and very likely out of a really good client ... or ...

    Philip Elder
    MPECS Inc.
    Microsoft Small Business Specialists

    *All Mac on SBS posts are posted on our in-house iMac via the Safari Web browser.