Friday 1 January 2010

Intel Modular Server – Error Component Unfit And IMS Firmware Update Caveats And More

After getting the Intel Modular Server (IMS) together, we fired it up to see how things looked.

We set up one of our own management desktop VMs with an IP on the same subnet as the default IP for the IMS management module. The VM’s NIC was then connected to the switch the IMS was connected to.

This is what we were greeted with once into the Modular Server Control (MSC) console:

image

The three compute modules were sight unseen for the moment.

Now, flip over to the firmware management node and we see:

image

The current firmware build version is 2.6.100.20081120.14717. Dollars to donuts the new MFS5520VI compute modules are not being recognized by the older firmware.

Now, there is an important caveat in the most current firmware’s release notes (v5.0 as of this writing):

The system must be at firmware level v2.7, v3.0, v3.6, v4.0, or v4.1 prior to starting this update.

Given the above screenshot, it is pretty obvious that we will need at least two firmware update runs to get to the newest v5.0 version on this particular IMS (Intel part number MFSYS25).

Now, given our experience with pretty much any I.T. related product, it is _very_ important to read through the Release Notes thoroughly. There are a lot of caveats with regards to the various IMS configurations and firmware update processes. Those caveats span a new configuration as well as an in production configuration.

For example:

Prior to loading and using the V2.7 unified software update – all server modules in the system must be powered down.

In our case, they did not even power up so we are okay to go forward.

Or so we thought:

image

Okay, the compute modules are still plugged into the chassis but they are off as we power cycled them with video connected to verify.

We needed to pull all three compute modules out of the chassis before we received the following:

image

Start Update button is now available.

Note the absence of the compute modules in the component listing below the Start Update button.

One thing to take note of is the following firmware listings on the MFS5520VI Compute Module download page:

image

There are are definitely a number of earlier firmware versions missing from this list. Given that, it is pretty reasonable to conclude that the compute modules do not need to be installed until _after_ the next firmware update to v5.0.

Since this is our first experience with the IMS, there is definitely a learning curve involved with getting the hardware configured and ready for our Hyper-V Server 2008 R2 cluster.

In brief:

  1. If the IMS’s firmware is earlier than v2.7, then download v2.7 and v5.0 or whatever is the newest version from Intel’s MFSYS25 Support site (link through the product support page).
  2. Unzip the package and read the Release Notes.
  3. Make sure the MFS5520VI compute modules _are not_ plugged into the chassis.
  4. Connect and log onto the management module.
    1. https://192.168.150.150 (default IP).
    2. User/Pass: admin/admin (default user/pass).
    3. The management module’s status lights will indicate green when the update is finished and the fans will settle down to a fairly quiet hum depending on the room/shop temperature.
  5. Follow steps 1 through 16 to update to v2.7 if v2.6 and earlier firmware is installed.
    • Steps 1-16 are in firmware v2.7’s Release Notes.
    • Note that each round of firmware updates will take anywhere from 30-90 minutes.
    • The
  6. Follow steps 1 through 15 to update firmware v2.7 to v5.0.
    • Steps 1-15 are in firmware v5.0’s Release Notes.
    • Note that it will take quite a while for the reboot to happen. The fans will quiet down near the end of the updates. The MSC will be available, though one or both of the Storage Control Modules may still be updating.
    • Make sure no chassis components are updating before moving onto the next steps.
  7. Follow steps 16 and on to update the Compute Modules’ Firmware starting with Compute Module 1.
    • image
    • Note that the BIOS update for each of the Compute Modules will be a separate process once the firmware portion of the updates have completed.
  8. Once Compute Module 1 has completed its firmware update, plug in the balance of the Compute Modules for their firmware updates.
    • This process can be run simultaneously on the balance of the Compute Modules.
    • We do this as a precaution, not as a rule to make sure that things run as expected.
  9. If the MFSYS25 firmware updates started at an earlier version than v3.0, then follow steps 24 through 31 in v5.0’s Release Notes.
    • image
  10. Power up the Compute Modules either by pressing the power button on the front or via the power on required link.
  11. Once the update has been uploaded to the Compute Module, a reset will be required.
    • image
  12. Be patient, as the BIOS update will automatically start:
    • image
    • If a video connection is to be had, the scripts that are running on the actual Compute Modules can be watched as they execute.
  13. Each Compute Module will reboot once the BIOS updates complete. Allow the reboot to proceed all the way through to a dead stop at the Reboot and Select proper Boot device … message.
    • image
    • Update status will show the correct BIOS string _before_ the Compute Module has completed its boot cycle.
  14. Reset each Compute Module and F2 into the BIOS. Configure the settings as needed.

Once configured, we were ready to begin the next phase of configuring the IMS for our Hyper-V Server 2008 R2 cluster.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

6 comments:

Anonymous said...

Hello, excellent blog on the Intel Modular Server. Just one comment i thought i would add with regards to the firmware update. Even though you may be on a very old version of firmware, you do not need to step up through the various firmware packages like the release notes may tell you - IF your system is a new deployment or you don't care about any management module settings you may have prevsiously set such as SNMP settings etc, then you can go ahead and blast sraight to the newest firmware release. i.e. 2.6->5.0 etc. All that will happen is that the newest firmware package will look at the older firmware and if it doesn't understand the database in the old firmware version, it will simply replace with the latest version. so this can save a lot of time if you are deploying a brand new system or you simply doin't have any CMM settings that you need to keep.

Thanks

Philip Elder Cluster MVP said...

Thanks for the comments.

While I see what you are saying as far as the firmware updates and a new setup, the final word on how we process those updates comes via the Release Notes.

Even though there is a bit of extra time involved in stepping through the updates, it is our preference to do so given our past experiences.

Yes, something may work as far as cutting out a step or two, but it may not work _every_ time ... and ultimately it is up to us to evaluate that risk.

Given the complex nature of the Intel Modular Server and its interactions between the various components we have plugged into it, we see the risk as being too high to not follow the guidance in the Release Notes.

Philip

Евгений said...

Firmware update, it was updated all but one module. Now to me all time suggests to reboot it. Remote access does not work. Direct connection of the keyboard and a mouse gives nothing. What to do?

Philip Elder Cluster MVP said...

Voland,

Remove the module, then plug it back in again. If that does not settle things down, then pull the plugs on the IMS and let it sit.

Once you plug the A/C back in, watch for things to come back online.

Philip

Anonymous said...

hello, your blog was very helpfully
i've got very big problem
MFSYS35 Old firmware version was 2.6 and then we tried to install version 5.5 on that 2.6. That was my very big mistake cos i didn't read instruction that's why the update progress stopped. And i can’t continue it Is there any firmware package can upgrade from 2.6 to 4.1 or later? Or can I downgrade firmware from 5.5 to 2.6? if there is how? please help me! if you have 2.6 firmware package pls share with me please please

Philip Elder Cluster MVP said...

A.,

All firmware versions are posted on the Intel Download site.

Yes, if the IMS is already in production you would need to follow the Read Me instructions to step things up between firmware versions. Stepping things up between versions preserves the settings database as it gets modified with each version change.

There is a way to recover the CMM, but not your settings if they were not backed up. You will need to contact support for this. You will need a laptop and a serial cable if you have not already plugged the serial port into a local management server.

Make sure you have your SharedLUN key and any others on hand to drop into Modular Server Control.

If you don't have the keys, then ask the tech to have a temporary SharedLUN key or other key generated for you _before_ starting the procedure.

From there, TeraTerm is an excellent terminal software that allows you to log everything done on the IMS CMM console for future reference.

BTW, your storage and Ethernet modules along with the compute modules should maintain their setup during the recovery process.

The caveat though is that we have seen a one-off situation where the NIC GUIDs were flattened and thus we lost the cluster running on that particular IMS until we rebuilt the networking setup on the nodes. Fortunately, once that was done the cluster came back up and running as though nothing had happened. +1 for Hyper-V clusters!

Philip