APC Network Management Error: Somone is currently logged into the ACP Management Web Server

We received an alarm from one of our network managed APC UPS systems. When we went to log on we received the following notice:

Notice

Someone is currently logged into the APC Management Web Server.

Please try again later.

To “fix” this “problem” we need to log on via Telnet. If the Telnet Client is not installed then go ahead and install it since it is also a good tool for testing connections and SMTP issues.

1. Install Windows Features
   • 
2. Once complete close the window.
   • 
3. From there Start –> Telnet [Enter]
4. Open 192.168.99.99
5. Log on using the admin username and password
6. Exit
   • 

Refresh the browser window and attempt to log on again. It should be successful.

If not, a paperclip reset or serial connection to the network management would be the next step.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

APC UPS Product Comparison Quick Access and Surge Let-Through

Sometimes it takes a while to root around and figure out where things are.

We were looking at comparing the two 1500VA rack mount UPS offerings from APC and initially found it difficult to find the comparison button:

• APC Smart-UPS On-Line

It is the third item down in the left column. :)

The number we are looking for is the Surge Let-Through. Both should be the same or very similar but for some reason finding the number for the SURTA1500RMXL2U has been more than difficult.

So, we called: 1-800-800-4272 and chose the Support Option for Smart-UPS. Our result:

• 0.3% Surge Let-Through (330 Volt).
• Wikipedia: An explanation of Clamping voltage

They are virtually identical.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Windows Live Writer

APC AP9630 Management Card: Error: The application was not able to load.

We have an APC UPS Network Management Card 2 that decided to go South on us:

Error: The application was not able to load.

You are attempting to access an APC device.

There was a problem loading the application. Please login to the device via telnet for more details.

Telnet told us absolutely nothing with the event logs being empty. All we saw was the exclamation mark behind the “A”.

apc>system
E000: Success
Name:     Unknown
Contact:  Unknown
Location: Unknown
DateTime: 06/13/2012:11:52:05
User:     Administrator
Up Time:  0 Days 0 Hours 13 Minutes
Stat:     P+ N4+ N6+ A!
Bootmon:  bootmon:v1.0.2
AOS:      aos:v5.1.7
App:      sumx:v5.1.7

APC Support page:

• APC Knowledge Base: How do I upgrade the firmware on an APC Network Management Card?

So, time to flash the firmware.

Firmware download page:

• APC Support: Firmware/Software Page
  • Choose the correct product part number.

We ran the flash in Windows 7 with no errors but the web application still failed to load:

After about four or five minutes we saw:

But, we still had the error. We called into APC Support to see if we could get any further.

They had us run the Windows version of the update from a Windows XP machine. It still failed.

While on the phone with the tech support folks we ran the following commands via Telnet (Tera Term):

1. reboot [Enter]
2. yes [Enter]

It still failed to come back online. So, we went into the following steps:

1. resetToDef –p keepip [Enter]
2. yes [Enter]
3. reboot [Enter]
4. yes [Enter]

With absolutely no results we then had one more option (Nuke and Pave):

1. format [Enter]
2. yes [Enter]
3. reboot [Enter]
4. yes [Enter]

Note that we used Tera Term via Serial connection to complete the final command set to format as the IP address info would be lost.

Once the card finished its final reboot we were able to log in and see the status A+ for the web server services.

American Power Conversion               Network Management Card AOS      v5.1.7
(c) Copyright 2010 All Rights Reserved  Smart-UPS & Matrix-UPS APP       v5.1.7
-------------------------------------------------------------------------------
Name      : Unknown                                   Date : 05/16/2011
Contact   : Unknown                                   Time : 00:17:48
Location  : Unknown                                   User : Administrator
Up Time   : 0 Days 0 Hours 14 Minutes                 Stat : P+ N4+ N6+ A+

We then used the tcpip command to set the IP, subnet, and gateway. Once the card rebooted we were into the web console successfully!

In the end the best possible solution after running a firmware update and having it fail is to nuke and pave.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Production Hyper-V Failover Cluster on an Intel Modular Server Pictures

The following are some shots of one of the Intel Modular Server based Hyper-V Failover Clusters that we have deployed.

 

This particular three node Hyper-V Failover Cluster is set up with 14x 146GB 15K Seagate SAS drives configured in three Storage Groups in the built-in IMS SAN. One for the Hyper-V OSs, one for VM OS partitions, and another for a high I/O SQL database setup.

The IMS has both SAS Storage Controllers and both Ethernet Switch Modules installed. Each Compute Module has a mezzanine card installed to bring the total Gigabit NIC connections to four.

The Promise VTrak E310sD was added about a year later to open storage up for the SBS 2008 VM and also add Windows Desktop OS based VMs for a VDI deployment.

There are two Trip-Lite Rack Mount enclosures in this deploy. Unfortunately APC did not have a 24U unit available when this deployment was done so we had to run with them. APC is the vendor of choice for our rack enclosure and UPS needs.

The IMS, Intel Server System SR1630HGP 1U 1P that provides standalone DC services, the Promise VTrak, and the backup destination drives are housed in one.

The second APC Rack Mount enclosure has all of the APC UPS components:

A 6000KV head unit with four additional 192V battery packs give the cluster about 8-10 hours of run time. Near the top of the stack is the 208V to 120V transformer. A network management card is installed in the UPS head unit for total power control and monitoring.

A separate set of SUA1500RM2U APC UPS units are used for the networking, 1U DC, and the backup destination drives to give us some power control granularity.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

APC Switched Rack PDU AP7900

We are in the process of moving some of our single plug systems over to a new APC PDU.

• APC Switched Rack PDU AP7900 (product page)

Note that in our case the AP7900 came with the PDU unit and a cable distribution bracket that mounts to the front of the unit. The piece shown in the first picture above must be a part of a different product SKU.

On the top of the unit is a sticker with the MAC address which we set up with a DHCP and BOOTP reservation on our SBS 2011 VM.

Note that out of the box the PDU uses BOOTP to obtain an IP address so it is important to make sure that the reservation includes BOOTP:

The default username and password for the unit is apc/apc.

Once we have connected to the device and renamed the admin user as well as reset the password we then moved on to naming each outlet after the device connected to it:

Once complete we will have each outlet identified with a quick link to the product’s support page (in this case):

Instead of the product support page we could link to the device’s own management IP address if it had one.

We would then go on to create power cycle schedules to automatically power cycle any device connected to the outlets:

Power cycling a device on a semi-regular basis can be quite helpful as we have all experienced flaky behaviours from all manner of devices that have been up and running in some cases for years.

In the end we have the ability to power cycle any device or group of devices that we need to. This is especially important for devices and older servers that do not have a Remote Management Module with remote power cycling capabilities.

A port redirection can be set up in the Internet gateway device to the PDU’s management console to allow for remote power cycling … so long as the gateway is up and running. In some cases the PDU could be plugged directly into the Internet to allow for direct management.

Conclusion

The APC PDU was quite simple to set up and get going. With the IP reservation set prior to plugging in an Ethernet cable and powering it up we were able to have it set up in a matter of minutes.

We then went on to install it into our NetShelter enclosure and move device power cords over to it.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Not All UPSs Are Created Equal – Surge Let-Through

As Chris Knight pointed out in a comment on our post on a failed APC BR1500LCD and both power supplies in an SC5295BRP chassis, the Surge Let-Through value is a very important one when it comes to determining what UPS needs to be put in front of expensive and critical IT equipment.

So, here we have three APC UPS products set up in an online comparison on APC’s site:

APC Smart-UPS 1000VA SMT1000RM2U $559.00

APC Smart-UPS 1500VA SUA1500RM2U $719.00

APC Smart-UPS 1500VA SMX1500RM2U $779.00

Now, out of the above units, which unit or units would have a better Surge Let-Through number?

The answer was actually a bit surprising:

APC Smart-UPS 1000VA SMT1000RM2U 0.3%

APC Smart-UPS 1500VA SUA1500RM2U 0.3%

APC Smart-UPS 1500VA SMX1500RM2U 5.0%

What caught us off guard was the fact that it was the most expensive unit that had the lesser surge let-through rating.

Now, considering that the BR1500LCD unit also shares in the 5% rating, if we had the SMX1500RM2U in its place we probably still would have had two blown power supplies.

Thanks to Chris, we will be paying a lot more attention to that rating on any of the UPSs we supply our clients or purchase for our own use.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Primary A/C Breaker Fail – APC BR1500LCD and Intel SC5299BRP Dual PSU Fail

One of our clients had a primary breaker pop on Friday last week when an appliance was turned on while someone was running an oven’s self-clean cycle.

When the power failed it somehow managed to kill the APC BR1500LCD UPS protecting the server. The end result of the UPS being taken out was that both power supplies in the redundant PSU setup also failed.

The server was configured as follows:

• Intel Xeon X3220 Series CPU
• Intel S3210SHLX Server Board
• 4GB RAM
• Intel RAID Controller SRCSASRB + BBU
• 500GB Seagate ES series drives in dual RAID 1 arrays.
• Intel SC5299BRP with both PSUs.

Intel sent out a pair of replacement PSUs which fired up the hardware with no issues.

The BR1500LCD’s onboard display normally shows about 20 to 30 minutes of runtime on any given day. It now reads 1 minute so it will also be replaced.

We were fortunate that the spontaneous shutdown of SBS 2003 R2 Premium with ISA installed did not in some way mess up Exchange or Active Directory.

We ran a few reboot cycles and an update cycle checking to make sure that the Exchange database was happy in between each cycle which it was. We had their ShadowProtect backup on standby if there was a need to recover the server.

The final reboot happened in about 1/3 the time the first boot up after plugging in the new PSUs did. We will be keeping a close eye on this box to make sure everything stays happy.

It just goes to show that even with a good UPS in front of a system that includes line levelling that there are no guarantees.

Nothing replaces having a good backup that is known to recover without issues. The stress levels are so much lower when we know we have a fall back in the midst of a server down situation.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

UPS Battery Life = Monitor Or Possible Problems

One of our own UPS units, an APC SUA1500RM2U threw a battery alarm code not too long ago. We received the replacement battery set this week.

This is what we found when we eventually pried the existing battery set out of the unit:

Note the bloating in the battery housings. The battery without the label was the one that caused the most grief when prying the tray out of the APC UPS.

Some of the terminals on the batteries themselves were also showing some signs of frosting along with leakage.

The APC is hot swap capable, so we were able to swap out the batteries while the servers were still up and running. This particular unit is about 3 years old.

Ironically, while working in the server closet at a client’s site not too long ago, one of their SUA1500RM2U units threw a battery code as well.

So, today we will be heading down there to swap out both battery sets as they have two of the UPSs and both are somewhere around the same age of 3-4 years old.

Even with the UPS self-tests running every two weeks, the above battery condition did not show itself until something inside one of the batteries collapsed enough to cause a ground condition between the plates. Depending on the failure method, things could have been a lot worse when the batteries failed.

We will take to visually inspecting the batteries in these units on a yearly basis now.

We have seen the results of a spontaneous breakdown of a UPS’s internal components . . . along with the panicked phone calls about the noise and smoke that brings if it happens during business hours.

Fortunately none of the SUA1500RM2U units we have out there have ever done that. The units that we have seen combust were smaller pedestal style 350-1200VA units. We now tend to recycle them after 2-3 years instead of putting in a new battery.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*All Mac on SBS posts will not be written on a Mac until we replace our now missing iMac! (previous blog post)

Windows Live Writer