Mellanox SwitchX-2 MLNX-OS Upgrade Stall via WebGUI and MLNX-OS to Onyx Upgrade?

Yesterday, we posted about our OS update process and the grids that indicated the proper path to the most current version.

A catch that became apparent was that there were two streams of updates available to us:

1. image-PPC_M460EX-3.6.4112.img
2. onyx-PPC_M460EX-3.6.4112.img
• ETC
• 

As can be seen in the snip above we read an extensive number of Release Notes (RN) and User Manuals (UM) trying to figure out what was what and which was which. :S

In the end, we opened a support ticket with Mellanox to figure out why our switches were stalling on the WebGUI upgrade process and why there was a dearth of documentation indicating anything about upgrade paths.

The technician mentioned that we should use CLI to clean-up any image files that may be left over. That's something we've not had to do before.

Following the process in the UM to connect via SSH using our favourite freebie tool TeraTerm we connected to both switches and found only one file to delete:

• WebImage.tbz

Once that file was deleted we were able to initiate the update from within the WebGUI without error on both switches.

Since we had MLNX-OS 3.6.4112 already installed the next question for the tech was, "How do we get to the most current version of Onyx?"

The process was as follows:

1. Up to MLNX-OS 3.6.4112
2. Up to Onyx 3.6.6000
3. Up to Onyx 3.6.8008

As always, check out the Release Notes (RN) to make sure that the update will not cause any problems especially with in-production NICs and their firmware!

Happy News! Welcome to Onyx

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
www.s2d.rocks !
Our Web Site
Our Cloud Service

Mellanox SwitchX-2 and Spectrum OS Update Grids

We're in the process of building out a new all-flash based Kepler-64 2-node cluster that will be running the Scale-Out File Server Role. This round of testing will have several different rounds to it:

1. Flat Flash Intel SSD DC S4500 Series
• All Intel SSD DC S4500 Series SATA SSD x24
2. Intel NVMe PCIe AIC Cache + Intel SSD DC S4500
• Intel NVMe PCIe AIC x4
• Intel SSD DC S4500 Series SATA SSD x24
3. Intel Optane PCIe AIC + Intel SSD DC S4500
1. Intel Optane PCIe AIC x4
2. Intel SSD DC S4500 Series SATA SSD x24

Prior to running the above tests we need to update the operating system on our two Mellanox SwitchX-2 MSX1012B series switches as we've been quite busy with other things!

Their current OS level is 3.6.4006 so just a tad bit out of date.

The current OS level for SwitchX-2 PPC switches is 3.6.8008. And, as per the Release Notes for this OS version we need to do a bit of a Texas Two-Step to get our way up to current.

Now, here's the kicker: There is no 3.6.5000 on Mellanox's download site. The closest version to that is 3.6.5009 which provides a clarification on the above:

Okay, so that gets us to 3.6.5009 that in turn gets us to 3.6.6106:

And that finally gets us to 3.6.8008:

Update Texas Two Step

To sum things up we need the following images:

1. 3.6.4122
2. 3.6.5009
3. 3.6.6106
4. 3.6.8008

Then, it's a matter of time and a bit of patience to run through each step as the switches can take a bit of time to update.

A quick way to back up the configuration is to click on the Setup button then Configurations then click the initial link.

Copy and paste the output into a TXT file as it can be used to reconfigure the switch if need-be via the Execute CLI commands window just below it.

As always, it pays to read that manual eh! ;)

NOTE: Acronym Finder: AIC = Add-in Card so not U.2.

Oh, and be patient with the downloads as they are _slow_ as molasses in December as of this writing. :(

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
www.s2d.rocks !
Our Web Site
Our Cloud Service

A Little Plug for Mellanox and RoCE RDMA

RoCE (RDMA over Converged Ethernet) via Mellanox NICs and switches is our primary fabric choice for Storage Spaces Direct (S2D) and Scale-Out File Server (SOFS) to Hyper-V compute cluster fabric.

With the Mellanox MSX1012X 10GbE switch we can deploy a pair of them along with a pair of ConnectX-4 Lx dual port NICs per node for about the same cost as a pair of NETGEAR XS716T 10GbE switches and a pair of Intel X540/X550-T2 10GbE RJ45 based NICs per node.

We have a great business relationship with Mellanox. They are great folks to work with and their product support is second to none.

I was honoured to be asked to use a portion of my presentation for MVPDays to create the following video that is resident on Mellanox's YouTube channel.

Hopefully the video comes out okay as embedding it was a bit of a chore.

Thanks for reading and have a great weekend!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Mellanox PPC SwitchX Update v3.6.4006

Mellanox has released a firmware update for their SwitchX switches: v3.6.4006.

We've already updated our two SX1012 switches to v3.6.3508 as per our blog post Mellanox Prep for RoCE RDMA. That means that we'll be able to upgrade without any intermediary steps as per the section Upgrade From Previous Versions.

When looking into the Release Notes for the new firmware version we see:


Note that in our case we are running ConnectX-3 Pro ( MCX354A) adapters. So, we'll be keeping firmware 2.4.5030 on those NICs until such time as Mellanox lets us know that we are able to bump them up to 2.4.7000.


Looking in the Changes and New Features section there doesn't seem to be anything specific to us however there are quite a few items listed for versions between v3.6.3508 and v3.6.4006!

There are a few items in the General Known Issues section that we need to be aware of.

• Point 32: Statistics files are reset which means graphs get reset.
• Point 49 indicates that a faulty cable may cause other ports to delay their "rise". 
• Point 50 is important. 40GbE passive copper cables 5m in length may experience "rise" issues if connected to a third party 40GbE NIC.
• Point 93: Break-out Cables
• Odd ports might suffer from Tx drops even when global flow control is enabled.
  Set the egress poll to 8M using the following command:
  “pool ePool0 direction egress-mc size 8M type dynamic”.
•  Point 128: QoS: ETS does not work on SN2100 switch system.

I suggest checking out the Bug Fixes section near the end of the document. ;)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc