Intel/LSI/Avago StorCli Error: syntax error, unexpected $end FIX

We're working with an Intel setup and needed to verify the setup on an Intel RAID Controller.

After downloading the command line utilities, since we're in Server Core, we hit this:

C:\Temp\Windows>storcli /cx show

syntax error, unexpected $end

     Storage Command Line Tool  Ver 007.0415.0000.0000 Feb 13, 2018

     (c)Copyright 2018, AVAGO Technologies, All Rights Reserved.

help - lists all the commands with their usage. E.g. storcli help
<command> help - gives details about a particular command. E.g. storcli add help

List of commands:

Commands   Description
-------------------------------------------------------------------
add        Adds/creates a new element to controller like VD,Spare..etc
delete     Deletes an element like VD,Spare
show       Displays information about an element
set        Set a particular value to a property
get        Get a particular value to a property
compare    Compares particular value to a property
start      Start background operation
stop       Stop background operation
pause      Pause background operation
resume     Resume background operation
download   Downloads file to given device
expand     expands size of given drive
insert     inserts new drive for missing
transform  downgrades the controller
/cx        Controller specific commands
/ex        Enclosure specific commands
/sx        Slot/PD specific commands
/vx        Virtual drive specific commands
/dx        Disk group specific commands
/fall      Foreign configuration specific commands
/px        Phy specific commands
/[bbu|cv]  Battery Backup Unit, Cachevault commands
/jbodx      JBOD drive specific commands

Other aliases : cachecade, freespace, sysinfo

Use a combination of commands to filter the output of help further.
E.g. 'storcli cx show help' displays all the show operations on cx.
Use verbose for detailed description E.g. 'storcli add  verbose help'
Use 'page=[x]' as the last option in all the commands to set the page break.
X=lines per page. E.g. 'storcli help page=10'
Use J as the last option to print the command output in JSON format
Command options must be entered in the same order as displayed in the help of
the respective commands.

What the Help does not make clear, and what our stumbling block was, is what exactly we're missing.

It turns out, that the correct command is:

C:\Temp\Windows>storcli /c0 show jbod
CLI Version = 007.0415.0000.0000 Feb 13, 2018
Operating system = Windows Server 2016
Controller = 0
Status = Success
Description = None

Controller Properties :
=====================

----------------
Ctrl_Prop Value
----------------
JBOD      ON
----------------

CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded

The /cx switch needed a number for the controller ID.

A quick search turned up the following:

• Broadcom: MegaCLI Commands to StorCLI Command Conversion
• Broadcom: StorCLI Reference Manual (PDF)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
www.commodityclusters.com
Our Web Site
Our Cloud Service

Flash Multiple Intel or LSI RAID and SAS HBAs Together

We ran into a bit of a question mark around updating the SAS HBAs (Host Bus Adapters) in our Scale-Out File Server nodes. The question being how do we update all of the SAS HBAs in the server without having to take things a part physically.

When looking at the HBA.NSH file that comes with the firmware update for EFI Shell we see the default command line in the batch:

• sas2flash -f Intel\gb.fw -b mptsas2.rom -b x64sas2.rom

A quick search on the SAS2Flash utility and we found the LSI manual here:

• LSI SAS2Flash Utility Quick Reference Guide (Direct link to PDF)

In the guide we find what we need:

• sas2flash -fwall Intel\gb.fw -b mptsas2.rom -b x64sas2.rom

Once we power cycled through POST and CTRL+C into the HBA BIOS we saw:

Happiness is not having to pull apart the server systems to update things one at a time!

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen ASP Project
Find out more at
Third Tier: Enterprise Solutions for Small Business

Repeat after me: SATA does not belong in servers.

One of the very last servers we deployed with SATA drives had yet another failure in it.

There is a new Intel R2208GZ4GC 2U server in place with eight 600GB 10K SAS drives configured in a RAID 6 array already installed and waiting for tax season to slow down for them (they are an accounting firm).

Our client recently moved to a new location with the servers now located in a dedicated room in the basement. The little A/C unit in that room was a leftover from the previous occupant that we were not too sure about.

Well, the hot spare in this server, an Intel Server System SR1560SFHS with three 750GB Seagate ES series SATA drives, died about four months ago. Since the system was slated for replacement we left the remaining two in a RAID 1 array alone.

Well, that ended this morning with one of the drives in the pair having gone full stop. This was probably due to the fact that the temp in the room upon arrival this afternoon was close to 90F.

Someone had fired up the A/C unit without realizing that the hose that puts the heat outside was not connected to the back of the unit. Thus all of the heat it was trying to pull out plus its own heat yielded a very high temperature in that room.

Once the hose was affixed to the back of the unit the temperature started to come down.

So, here we are writing this blog post at 2216Hrs on a Wednesday evening after having logged in to check on the progress of the array rebuild and the above was what we saw.

The RAID controller is an Intel RAID Controller SRCSASRB with battery backup.

SATA does not belong in a server when it comes to spindled hard drives. This experience with the blind failure and the dismal rebuild times, during off hours no less, are definitely a part of it.

SAS/SCSI was designed and engineered to run in server environments. SATA was not.

The firmware tweaks that the hard drive vendors have introduced, along with the pretty much failed NCQ effort, to try and mimic a SAS setup within the SATA controller do not come close to the performance, longevity, and stability that SAS drives offer.

By the way, this goes for NearLine SAS drives as well. These drive types are SATA internals with SAS electronics slapped on to the external of the drive. There is a very good reason why the drives are called "NearLine". :)

The cost on 2.5" 10K SAS drives in 300GB and 600GB sizes have come down quite a bit in the last year. The 900GB 10K SAS drives are still relatively expensive per Gigabyte but provide an opportunity for a large aggregate of storage when needed.

Another way to look at it is this: How many RMA efforts have gone in to server setups with SATA drives in them? Compare that with the servers that have SAS setups. In our case, where we have lots of servers deployed, there is virtually no comparison. Over time the SAS drives have completely trumped the SATA drives in all aspects.

Even with 24x7x365 by 4 hour response times most vendors require time wasted on the phone prior to initiating that on-site visit to replace the failed drive. This time is expensive and to some extent a waste.

Oh, and one more thing: If going with parity in an array go RAID 6 with at least eight 10K spindles and make sure the RAID controller has either flash backed cache or a battery backup.

Storage is almost always the weakest point in a server both for hardware failures and I/O bottlenecks. Kill both. Use a wide array of eight spindles or more and make sure the drives 10K SAS.

The risk when using SATA is just not worth the "savings" IMNSHO (in my not so humble opinion).

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Chef de partie in the SMBKitchen
Find out more at
www.thirdtier.net/enterprise-solutions-for-small-business/

Windows Live Writer

Server Uptime at 16 Months

So, is this a good thing or a bad thing?

Yeah, this particular box has been up for 16 months.

In August of 2011 switch one of two failed. That switch just happened to be the one this particular box was plugged in to. Prior to that the box had been up and running since Server 2008 R2 was installed (just prior to R2’s RTM).

Its sole purpose in life is to serve Terabytes of data via a teamed Gigabit NIC pair.

Note the CPU setup. It has a pair of Intel Xeon 5130 CPUs and 16GB of RAM. Yeah, it’s a bit long in the tooth. ;)

Given the server’s role serving files we tend to leave it alone. No Web browsing, no desktop access, or any other access is needed unless something happens with it.

Well, today the backplane decided to hiccup along with the 500GB Seagate ES series drive that died in it (we’ve had a _huge_ fail rate on these drives over the last two or more years).

It is time for this old box to be retired.

Its replacement will be a Hyper-V Failover Cluster based on two or three Intel Server Systems SR1695GPRX2AC 1U servers with an Intel Xeon X3470 and 32GB of ECC RAM.

Is it a good thing for one to leave a box up and running for months or even years at a time?

Is the risk worth it?

In some cases we have no choice where three shifts run 24/7/365. Coordinated downtime is about the only way in to these boxes. Though SBS tends to start choking around the 90 day mark for both SBS 2008 Standard and SBS 2011 Standard so these reboots with patch cycles tend to happen every quarter.

To mitigate this situation we need to make sure we have good monitoring in place for edge access, AD authentication attempts (especially failures), proper edge configuration blocking both inbound and outbound packets by default, and other strategies like no touching the box/VM.

In the end, the risks need to be evaluated beside the benefits of no reboot cycles and/or no patch cycles for a lengthy amount of time.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Windows Live Writer

Hyper-V: Creating a Fixed VHDX on 8x RAID 6 10K SAS Spindle Set: 130GB

Our previous blog posts on this subject:

• Hyper-V: Creating a Fixed VHDX on 8x RAID 6 SATA Spindle Set: 130GB
• Result: 158 Minutes
• Hyper-V: Creating a Fixed VHDX on SSD: 40GB Time
• Result: 5 Minutes
• 130GB Extrapolate: 19 Minutes

For today’s test we are running the following setup:

• Intel Server System R2208GZ4GC
• Dual Xeon E5-2630
• 128GB (16x 8GB) Kingston ECC
• Intel Integrated RAID RMS25CB080 with BBU
• 8x 600GB Seagate 10K SAS drives at 2.5”

The operating system is a freshly installed Windows Server 2012 Standard.

The RAID 6 array setup in the RAID controller’s BIOS is as follows:

• Logical/Virtual Disk 0: 120GB
• Logical/Virtual Disk 1: 3.1TB

We start the New Virtual Hard Disk Wizard in Hyper-V Manager and run through to create a fixed VHDX with a 130GB size.

Our test results are:

• Start time: 13:29
• End time: 13:31
• 
• Result: 2 Minutes

Yup. 2 Minutes.

As a just-in-case we ran this test again this time while watching the Performance Monitor:

• Start: 13:37
• Finish: 13:39
• Disk Queue Length:
• 
• Disk Throughput:
• 
• All three measures:
• 
• Result #2: 2 Minutes

Check out that throughput at 1GB/Second! And, Disk Queue Length hovering around 5 for the working virtual/logical disk.

This test demonstrates that a properly configured disk subsystem will perform as good or better than expected.

Now on to the SSD tests! :)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

Windows Live Writer

Intel Integrated RAID Modules - POST Hang and/or Data Loss Issues

A few Intel Technical Advisory documents to take note of with the first being the most critical.

• Intel Technical Advisory TA-1005-1: Intel Integrated RAID Module rework to resolve an AC power cycling cache backup issue (PDF Document)
• RMS25PB080
• RMS25PB040
• RMT3PB080
• RMS25CB080
• RMS25CB040
• RMT3PB080

Description

The PCB’s (Print Circuit Boards) for the controllers listed above are being modified to resolve an AC power cycling cache backup issue with the Intel RAID Smart Battery AXXRS8BU9 and the Intel RAID Maintenance Free Backup Unit AXXRMFBU2 cache backup solutions. In addition to some minor component changes a jumper wire will be added to the PCB’s during manufacturing. Intel has completed product qualification for the changes to verity that there is no impact to product quality and reliability.

Root Cause

During AC power cycle testing Intel found a rare occurrence where data yet to be written to disk from the RAID modules DDR3 memory may be lost even though cache protection is enabled by the Battery Backup Unit (Intel® RAID Smart Battery AXXRSBBU9) or the RAID Maintenance Free Backup Unit (Intel® RAID Maintenance Free Backup Unit A)(XRMFBU2) that is installed. The issue has a very low occurrence rate and is observed on a small percentage of modules. The component changes and jumper wire solution provides a resolution to this issue.

The units we just brought in for a couple of 2 node clusters we are building out do indeed have the jumper wire in place.

• Intel Technical Advisory TA-1014-1: System may hang during POST with "Quiet Boot" enabled in BIOS and default Intel Integrated RAID Module firmware (PDF Document)
• RMS25PB080, 080N
• RMS25PB040
• RMT3PB080
• RMS25CB080, 080N
• RMS25CB040
• RMT3PB080

With the factory default firmware in place the server will hang during POST. We update all firmware on the server builds and deploys that we do prior to introducing them into a production environment. Once updated we run a series of burn-in tests over a period of days to validate the build.

Firmware, drivers, and more can be had from Intel's Download Centre.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Hyper-V Error 41 – Hyper-V launch failed; Either VMX not present or not enabled in BIOS and BIOS Update Methodology

We have just finished setting up a lab server that has been sold to an IT Consultant we have been working with for years now.

The server was set up with the existing Intel Xeon 3070 CPU on a new Intel Server Board S3210SHLC with 8GB of Intel certified Kingston ECC RAM.

One of the first things that we do with a new server setup, or an existing server setup that we are working on, is to verify the firmware levels on all components.

• Server board BIOS, BMC, and FRU/SDR.
• Hardware RAID controller firmware.
• Hardware SAS controller firmware.
• Hot Swap backplane firmware level.

We ran through and updated the firmware on all components in the server setup as they were all out of date to some degree.

The process we run to update the BIOS and firmware on the server board is as follows:

1. Boot to EFI and run the update.nsh script.
2. Power down completely after the updates have finished.
3. After a minute power the server up and let it run all the way through to a “No OS Found” message.
4. Power the system down again.
5. Wait for a minute and power it up.
6. Enter the BIOS.
7. Make settings changes.
8. Save and reboot to a “No OS Found” message.
9. Power down.
10. Power up and boot to OS install DVD.
11. Install drivers, configure OS partition, install OS.

We followed that process or so we thought for this particular server.

Once we have Windows Server 2008 R2 Standard SP1 installed we enabled the Hyper-V Role and ran all available Windows Updates.

We configured a Windows 7 desktop OS VM and went to start it when:

Hyper-V Manager

An error occurred while attempting to start the selected virtual machine(s).

‘VirtualMachine’ could not initialize.

The virtual machine could not be started because the hypervisor is not running.

The server logs showed:

Event ID 41: Hyper-V-Hypervisor

Hyper-V launch failed; Either VMX not present or not enabled in BIOS.

When we went into the Win2K8 MMC we saw that all three Hyper-V services were indeed running. We ran a restart of all three with one failing to restart.

So, we rebooted the server to see where the BIOS settings were at:

They indicate that they are indeed enabled. But, note that the information section states that a full power off and on cycle is required before the settings change will take.

Well, we were sure that we did fully power cycle the box when we were into the BIOS settings after updating it.

Just in case, we shut the server down and pulled the plugs. We let it sit for about a minute and then plugged it in. The server fired up on its own and booted into the OS.

All three Hyper-V services were up and running and there were no more errors in the logs once it finished booting.

Sure enough, we were able to go on and create the VMs, make the necessary settings changes, and start them.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Configuring An Intel RAID Controller Via RMM JViewer

We are in the process of setting up a new server remotely.

We are logged in via the Intel Remote Management Module 3 that is installed on the Intel Server Board S3420GPLX.

In order to get the mouse to a liveable control level we had to close the KVM session and set the mouse configuration to RELATIVE:

Out of the box the mouse is set to ABSOLUTE mode which does not work very well at all.

Neither the Soft Keyboard nor the system’s keyboard that we are using allow for ALT+Key presses. The Keyboard menu at the top has some control over the ALT and CTRL key presses and holds but still it was not too reliable. It just worked to allow us to choose the drives for the disk group.

As long as we remained patient and moved the mouse about in a slow and fluid manner, controlling the RAID BIOS worked . . . just barely.

And finally we had our array:

So, it just takes a bit of patience to work through the process of setting up an array or arrays via the Intel Remote Management Console.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

SAS versus SATA and Hardware RAID versus Software RAID

In the last few years we have made some changes to the server configurations that we either build and deploy or Tier 1 provides:

• We install SAS 10K or SAS 15K drives over SATA.
• Performance for one is vastly superior on the SAS drives.
• SAS drives are a lot more sensitive to bad sector behaviour and are better able to recover from bad data being tossed up.
• SAS drives use many forms of ECC which protects the integrity of the data.
• We install a hardware RAID controller with battery backup or SSD Cache over using the on board software RAID.
• The on board “RAID” is software driven. All RAID calculations are completed by the server’s CPU and in many cases require the software driver to rebuild or function properly – meaning we need to boot into the OS.
• We have had difficulties with on board software RAID recoveries and ShadowProtect due to driver issues.
• Hardware RAID on Chip with the battery backup or SSD Cache virtually eliminates the parity write cost of RAID 5 and greatly reduces that cost for RAID 6. SSD Cache almost renders the whole discussion moot as the most frequently requested data sits on the SSD.
• Hardware RAID controllers have the ability to mitigate the failure of a drive by keeping the server up. In our experience software RAID tends to freeze the box if a drive fails.
• Hardware RAID controllers tied to SAS drives have a much better chance of mitigating or eliminating the possibility of data corruption if sectors on an array member are dying.

Intel has a couple of resources for the above discussion as do many other RAID related vendors.

One Intel resource is the following Intel support page: Intel Server Products: Choosing between SAS vs. SATA Hard Disk for your Server RAID System.

There we find this grid that gives us a pretty good idea of some of the significant differences between SAS and SATA drives:

We are given an extensive explanation in the following document which is linked to below the above table: Intel – Enterprise-class versus Desktop-class Hard Drives (Link to PDF document download).

We put our server configurations through a lot of testing before we deploy them to client sites or within our own organization. We do this because we want to make absolutely sure that the server configuration we are going to deploy will meet the needs of our client over the life of the box which is about 36 months.

The extra cost for the hardware RAID controller, battery backup, and 15K SAS drives (not much of a cost difference between 300GB 15K SAS and Seagate Enterprise SATA these days) when taken over the life of the box (divide that cost by 36) is actually quite small relative to the performance, data protection, and overall storage stability benefits.

Tier 1 Caveat

Keep in mind that cost inferences mentioned here are for our in-house Intel server solution components. When it comes to Tier 1 the costs of some server components and server storage can be extremely high relative to components supplied in our solutions.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Integrating An Intel RAID RS2BL040 Battery Backup AXXRSBBU7 With An Intel Server System SR1630HGPRX

The Intel Server System SR1630HGPRX has a spot on one side of the chassis to mount the Intel RAID Controller’s RAID Smart Battery.

Having the battery unit mounted elsewhere in the chassis is a good idea since having it mounted on the RAID controller itself causes an obstruction to the air flowing towards the RAID controller’s processor heatsink.

Tray Mounting

The plastic tray was a bit of a bear to mount in place:

We had to slip the guides in onto the high points of the mounting slot and somehow pivot it into the low point with a good amount of effort required to get the locking mechanism to snap in place. The lock is just above the white plastic zip tie in the above screenshot.

Note the star shaped pin near the bottom right hand side of the tray. One would initially gather that it was meant to guide the battery backup’s PCB board into the proper place. Especially since there looks to be a hole in the PCB board just for that purpose.

In fact, it is not a guide but a stand-off to prevent the PCB board from being pushed too far into the tray. Keep this in mind when mounting the battery and PCB.

Another catch to getting the battery/PCB in is in fact the catches themselves. There is one on either side of the tray’s lock. They provide a significant amount of resistance to getting the PCB to set in place. We needed to be mindful of just how much they were being pulled back while trying to slip the PCB into place.

And one more thing when it comes to the tray and batter/PCB combination, there is a requirement for the battery to sit above the PCB in order for the PCB to catch under the two mounting clips on the tray. The PCB clip that holds the battery down looks large enough to prevent this, but there is enough flex in the PCB itself to get things into place.

All in all, this is one snug fitting trio.

Remote PCB and RAID Controller

Once we ran the cable from the battery backup to the RAID controller, we needed to install a small daughter card onto the RAID controller itself.

The daughter card, shown above, has the socket that connects to the RAID controller as well as the black pivot pin that is used to set the mounting screw into on the RAID controller. The pivot pin is shown in the correct location above. We needed to loosen the screw on the daughter card and rotate the pivot pin into the correct location prior to installing it onto the RAID controller.

The daughter card installed:

After installing the power cable the RAID controller was connected to the PCI-E riser and finally installed in the system.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Western Digital RE3 Series SATA Drives In Intel Hot Swap Expander Backplanes May Error Or Go Offline

It seems that there is an issue between the Western Digital RE3 SATA drives running at 3 Gb/s in Intel’s hot swap expander backplanes.

• Intel Support: TA-958: Drive errors may occur when using certain 3rd party drives at 3.0 Gb/s

Root Cause
The SAS expander chip on Intel® Server Drive Enclosure AXX6DRV3GEXP and AXX4DRV3GEXP experience an undesired flow-control behavior when connected to a SATA disk drive running at 3.0 Gb/s. Drives operating at 3.0 Gb/s may not receive proper data or SATA flow control and may result in a timeout error. There is no timeout at 1.5 Gb/s. This interoperability limitation of the SAS expander with SATA disk drive is currently seen on Western Digital’s RE3 series SATA drives.

Products affected:

• Intel Server Drive Enclosure AXX6DRV3GEXP
• 6 drive expander backplane.
• Intel Server Drive Enclosure AXX4DRV3GEXP
• 4 drive expander backplane.

To mitigate the issue the drive needs to be set to run at 1.5 Gbp/s by setting the jumper at position 5/6.

• Western Digital RE3 Support Page
• Western Digital RE3 Jumper Positions

It is also important to make sure that the hot swap backplane has the most current firmware.

• Intel Server Chassis SC5650 Support Page

The hot swap backplane firmware download link can be found on the server chassis support page.

SATA vs. SAS

As a rule we no longer use SATA drives in any of our server configurations.

With the cost of SAS drives having come down quite substantially over the last few years along with the huge performance benefits we get with using SAS drives there is really no reason to use SATA drives in a server setting anymore.

The exception to this rule is for Intel Solid-State Drives that run with a SATA interface.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

On Board Software “RAID” No More

We stopped using the on board “RAID” feature of on all of our client servers.

Why?

Because the cost of a high performance 4 port RAID controller like the Intel RAID Controller RS2BL040 is under $450CA.

For $450 we get:

1. True RAID on Chip driven calculations.
   • No software interpreting the RAID calculations on the CPU.
2. Drive redundancy.
   • But, on board RAID is just that: RAID right? Well, no. If a drive fails in an on board based “RAID” array 9 times out of 10 the server is locked up.
   • If a drive fails on a true RAID controller based array the RAID controller marks the drive dead, logs the failure, but _keeps moving along_.
3. Data Protection
   • We have seen more failed OS loads after an on board based “RAID” array member failure by a factor of at least 10 compared to either a member or RAID controller failure in server settings.
4. Performance
   • On board RAID relies on the software drivers to do all of the calculations. They run on top of the CPU. There is a performance hit for this.
   • Add a battery backup for the RAID on Chip’s memory cache and the array performance steps up accordingly.
   • Array rebuild times to hot spare for the RAID on Chip solution will be superior to on board software “RAID”.
     • A failed drive is when the server is most vulnerable.
5. Maintenance
   • RAID on Chip solutions offer true hot swap compatibility.
   • Some software “RAID” setups may offer the same. But, test it first.

Given the overall benefits of an add-in RAID solution, in our opinion, no server should go out the door with an on board software “RAID” setup.

It is just not worth the risk.

Adaptec also makes some pretty good RAID on Chip solutions.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

160GB Pair of Intel G2 SSD’s RAID 1 and RAID 0 Performance On A W5590 Xeon Workstation

While putting together the trading station for our client, we did some bench testing of the Intel solid-state drives in two RAID array configurations.

The system configuration:

• Intel S5520SC Workstation Board
• BIOS 42, BMC 0.45, FRU/SDR 21
• Intel W5590 Xeon Processor (second is going into an identical setup)
• 12GB Kingston KVR1333 ECC Registered RAM
• 160GB Intel SSD -G2R5 Code pair of drives
• On Board Chipset LSI based RAID
• Intel SC5650WS Workstation Chassis with 1,000Watt PSU
• PNY nVidia Quadro NVS 450 PCI-E x16 (4 monitors)
• ATI FirePro 2450 PCI-E x16 (4 monitors)
• Windows 7 Ultimate x64 Edition

The Intel SSDs are using the factory default firmware and no trim tools have been installed. The RAID Web Console 2 was installed but the SSD trim tools built into it were not enabled either for both tests.

Here is the WinSat output for the RAID 1 configuration:

C:\Windows\system32>winsat disk
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-seq -read -n 0'
> Run Time 00:00:03.71
> Running: Storage Assessment '-ran -read -n 0'
> Run Time 00:00:00.31
> Running: Storage Assessment '-scen 2009 -drive C:'
> Run Time 00:00:55.51
> Running: Storage Assessment '-seq -write -drive C:'
> Run Time 00:00:07.55
> Running: Storage Assessment '-flush -drive C: -seq'
> Run Time 00:00:01.42
> Running: Storage Assessment '-flush -drive C: -ran'
> Run Time 00:00:01.37
> Running: Storage Assessment '-hybrid -ran -read -n 0 -ransize 4096'
NV Cache not present.
> Run Time 00:00:00.00
> Running: Storage Assessment '-hybrid -ran -read -n 0 -ransize 16384'
NV Cache not present.
> Run Time 00:00:00.00
> Disk  Sequential 64.0 Read               504.89 MB/s         7.9
> Disk  Random 16.0 Read                  242.46 MB/s         7.9
> Responsiveness: Average IO Rate     1.52 ms/IO          7.5
> Responsiveness: Grouped IOs          11.60 units          6.9
> Responsiveness: Long IOs                 3.62 units          7.8
> Responsiveness: Overall                  42.01 units          7.2
> Responsiveness: PenaltyFactor              0.0
> Disk  Sequential 64.0 Write             97.07 MB/s          6.5
> Average Read Time with Sequential Writes 0.997 ms     7.6
> Latency: 95th Percentile                     1.791 ms          7.8
> Latency: Maximum                             5.227 ms          7.9
> Average Read Time with Random Writes   0.879 ms          7.9
> Total Run Time 00:01:10.57

And here is the raw output for the RAID 0 configuration:

C:\Windows\system32>winsat disk
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-seq -read -n 0'
> Run Time 00:00:04.13
> Running: Storage Assessment '-ran -read -n 0'
> Run Time 00:00:00.26
> Running: Storage Assessment '-scen 2009 -drive C:'
> Run Time 00:00:51.11
> Running: Storage Assessment '-seq -write -drive C:'
> Run Time 00:00:04.30
> Running: Storage Assessment '-flush -drive C: -seq'
> Run Time 00:00:01.11
> Running: Storage Assessment '-flush -drive C: -ran'
> Run Time 00:00:01.09
> Running: Storage Assessment '-hybrid -ran -read -n 0 -ransize 4096'
NV Cache not present.
> Run Time 00:00:00.00
> Running: Storage Assessment '-hybrid -ran -read -n 0 -ransize 16384'
NV Cache not present.
> Run Time 00:00:00.02
> Disk  Sequential 64.0 Read           537.33 MB/s          7.9
> Disk  Random 16.0 Read              375.88 MB/s          7.9
> Responsiveness: Average IO Rate        0.76 ms/IO          7.9
> Responsiveness: Grouped IOs             8.99 units          7.3
> Responsiveness: Long IOs                     1.69 units          7.9
> Responsiveness: Overall                      15.21 units          7.9
> Responsiveness: PenaltyFactor                0.0
> Disk  Sequential 64.0 Write             197.47 MB/s          7.3
> Average Read Time with Sequential Writes    0.947 ms     7.7
> Latency: 95th Percentile                     1.719 ms          7.9
> Latency: Maximum                             5.686 ms          7.9
> Average Read Time with Random Writes   0.952 ms          7.9
> Total Run Time 00:01:02.78

Taking the above statistics we get:

• RAID 1 configuration:
• Disk  Sequential 64.0 Read        504.89 MB/s         7.9
• Disk  Random 16.0 Read           242.46 MB/s         7.9
• Disk  Sequential 64.0 Write         97.07 MB/s          6.5
• Total Run Time 00:01:10.57
• RAID 0 configuration:
• Disk  Sequential 64.0 Read         537.33 MB/s          7.9
• Disk  Random 16.0 Read            375.88 MB/s          7.9
• Disk  Sequential 64.0 Write        197.47 MB/s          7.3
• Total Run Time 00:01:02.78

It is pretty clear that having the SSDs in a RAID 0 or RAID 10 configuration will provide a significant performance advantage in real world usage.

The write speed more than doubled when the drives were striped and took a huge step for the random read tests as well.

So, what do the numbers translate to? They translate to extremely fast OS loads, application loads, RAM cache like behaviour for any swap file activity, and an all around great user computing experience.

For anyone that generates revenue on system setups and the speed that they accomplish their computing tasks, this is one rig to consider.

By the way, the peak power consumed by the box was 195Watts under load. The average power usage while running its various tasks was 110-120Watts as measured by the APC BR1500LCD UPS. Only the workstation was connected to the UPS.

When the components for the second workstation arrive, we will run the same tests but with an add-in RAID controller to see if that further improves disk I/O performance.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Intel Hot Swap 2.5” to 3.5” Adapter Tray Enables Intel SSD Hot Swap RAID – AXX25DRVADPTR

We had a really difficult time getting our hands on these adapters:

The adapter is actually upside down in the above image.

Flipped over, the SATA and power connectors on the 2.5” drive line up perfectly with the hot swap backplane connector to allow for the smaller drive to be used in 3.5” drive hot swap backplanes.

• Intel AXX25DRVADPTR Installation Guide (PDF Download)
• Intel AXX25DRVADPTR Compatible Intel Chassis and System List

With these trays we can look at configuring a RAID 1 array or RAID 10 array of SSDs for I/O intensive needs for things such as OS or Exchange databases.

We can also offer an option to update some of our existing client servers with these trays and a couple of Intel SSDs to speed things up significantly.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Intel RAID Controller RAID Smart Battery Backup Installation Caveat

Most Intel add-in RAID controllers have an option to add a battery backup unit for the onboard cache memory.

When the battery is installed, the RAID on Chip processor has the ability to cache data in memory and work on it on the fly while in cache RAM instead of passing all data plus parity bits through to the disks.

A list of the battery backup units and the RAID controller(s) they are compatible with can be found here:

• Intel Support: RAID controller and Battery Backup Unit compatibility list and product codes

Once the battery backup is installed on the RAID controller or connected to the RAID controller if the battery is remote to it, the actual battery charging process will not run until the server has booted up.

Simply plugging the server in after installing the battery backup will not initiate a charge.

So, for existing servers where the battery backup has been added in, the ability to enable Write Back Mode with BBU Present in the RAID controller’s BIOS will be available, but Write Back Mode will not happen until the battery is charged.

The server will need to be booted up and either idled or tested for at least 24 hours before the battery will be fully charged. Once charged, a reboot will allow the RAID controller to initiate Write Back Mode. Charge status is indicated in the RAID controller’s BIOS information screen during the boot process. It may also be indicated in the RAID Web Console.

For new servers, the point is fairly moot since the battery can charge while the server is being burned in. The final RAID configuration would be set after the burn-in period along with the Write Back Mode setting.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

RAID Controller Log: Unrecoverable Medium Error – Puncturing Bad Block?!?

Okay, so this is a new one and lead to a near stoppage of the heart last evening:

The errors in order:

Controller ID: 0 Unrecoverable medium error during rebuild: PD –|—:0 Location 0x26f1640

Controller ID: 0 Puncturing bad block: PD –|—:0 Location 0x26f1640

Controller ID: 0 Puncturing bad block: PD –|—:1 Location 0x26f1640

PD 0 is the last original disk in this server that is giving us headaches. Both PD 1 and PD 2 (there are three hot swap bays in the SR1560SFH Intel Server System) were replaced with new drives.

The original PD 1 had failed during a server firmware including BMC update (previous blog post). The original PD 2, the then global hot spare, was rebuilt into the array with no errors . . . until a consistency check that ran later that afternoon produced some unrecoverable fatal errors.

Last night we dropped original PD 1 out of the configuration, replaced it with a new drive, had the new drive picked up as a hot spare. We then failed out the original hot spare PD 2 now RAID 1 array member assuming that it was the source of the errors we saw in the consistency check yesterday afternoon.

So, the above screenshot was taken after the new PD 1 was being rebuilt into the array with PD 0 as the source. Needless to say the heart definitely skipped a few beats with visions of index $0 running through my head (previous blog post)!

The rebuild did eventually finish successfully though?!?

We will be going back this evening to fail out the bad PD 0 and replace it with a new drive which will then be designated the new hot spare.

Once the PD 2, currently a hot spare, rebuild into the RAID 1 array has finished, Intel indicated to us that we need to run a consistency check. From there, hopefully ShadowProtect will finally give us a backup!

And one more thing, just what does “Puncturing bad block” really mean?

• Bing Search: Puncturing bad block
• Yields nada.
• Google Search: Puncturing bad block
• Yields: NEC MegaRAID Storage Manager Manual (PDF)

The suggestion in the above NEC linked document is to take the preventative measure and swap out the indicated drive(s) promptly. :)

It looks as though the RAID controller has found some bad sectors on the PD 0 and puncturing means to set those sectors as off limits on both array members.

But part of this whole puzzle is the fact that the RAID controller (Intel SRCSASRB with firmware 470) shows a media error level of 0 for both array members and a predictive failure count of 0 for both members!

Hopefully tomorrow we can rest easy with a backup in hand!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Contracted to Build a Boss Dual W5580 Trading System

One of our long time clients has delved into currency trading and is using a proprietary monitoring and graphing application for the various world currencies being watched and traded.

Their current system is a high performance system we put together for them six years ago.

That system was based on a P4 with HyperThreading, Windows XP Pro and 1GB of RAM. We had our client bring the system here to the shop to run various process monitoring utilities on the PC while the proprietary application was up and running.

We were looking for a couple of specific things:

• How many simultaneous threads was the app capable of tossing towards the CPU.
• How many threads were involved in generating a new graph and what happens during the graph generation.

What we discovered was an application that is capable of taking advantage of as many “cores” as possible.

The last test we ran was the generation of a new graph with everything else going on in the background. The new graph being generated pegged one of the “cores” on the system, so that particular process was single threaded . . . maybe.

Now that we had a pretty good overview of how the application performed, we now knew some key performance needs:

• The ability to bin a core or multiple cores up for a demanding process was critical.
• The ability to handle more than 4 simultaneous threads was also a priority due to the way the software operated.
• The more RAM we had in the system, the better.
• We need the ability to drive 8 monitors.
• We need to eliminate the disk I/O bottleneck.

So, the configuration we have come up with for the system to be built to meet our client’s needs will be as follows:

• Dual Intel Xeon Processor W5580 or W5590
• Intel Workstation Board S5520SC
• 24GB Kingston Registered ECC RAM (6x 4GB Intel Tested)
• Intel RS2BL080 PCI-E 8x RAID Controller and Battery Backup
• 320GB (4x 160GB Intel 2nd Gen SSD) RAID 10
• Intel SC5650WS Workstation Chassis
• 6 Bay Hot Swap Option
• Dual ATI FirePro 2450 PCI-E Multi-View
• Up to 8 24” Wide Screen LCD Monitors
• ErgoTron Multi-Monitor Stands
• Windows 7 Ultimate x64 Edition

We expect the ability of each of the Xeon Processors to bin one or two cores upwards to gain the needed extra performance will make this rig fit our client’s needs perfectly.

Using the second generation Intel Solid-State Drives plus a second generation Intel RAID controller aught to help us virtually eliminate any disk subsystem bottlenecks.

The extra RAM and the 64bit capabilities of Windows 7 Ultimate round out the setup.

The workstation board S5520SC has a Trusted Platform Module version 1.2. Since we are installing Windows 7 Ultimate x64 Edition on this setup and due to the sensitivity of the tasks being accomplished by the user we will be encrypting the contents of any partition created with BitLocker.

In the end, the system should provide our client with exactly the responsiveness they are looking for in a system and also protect everything on the system via encryption.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

Intel On-Board RAID 1 Failed Array Member Replacement Method

A system we have on our bench has some serious gremlins working in it.

We have since replaced the original 150GB Raptors with 150GB VelociRaptors, we replaced the RAM, and we replaced the motherboard.

The replacement of the Raptors and then a subsequent OS install served to really stabilize the system. There were no more spontaneous reboots happening every couple or three random hours.

But, the system would still spontaneously reboot every two or three days. We have had the full Season 2 of 24 run through the box alternating with a local radio station for good measure. All multimedia runs through Windows Media Center on Windows Vista Ultimate with a Hauppauge PCI-E multimedia card providing the radio and a Radeon HD3870 providing the DVD decoding. A SoundBlaster X-Fi rounds out the configuration.

The onboard Intel Matrix Storage BIOS would not let us replace the defective hard drive no matter what methodology we tried.

A call into Intel support provided the guidance we needed to get things working as they should.

We had to delete the RAID 1 set the last good array member was on. From there, we booted into Windows Vista and needed to start the Intel Matrix Storage Console.

In the Console, we needed to switch to advanced mode, then select Create RAID Volume from Existing Hard Drive.

We chose to create a RAID 1 array.

The next step, which is choosing the source drive for the array, is critical. Since the failed drive was on Port 0, we knew that the drive on Port 1 need to be the source drive.

Once we selected the source and destination drives we let the Migration process run. It took about 45 minutes for it to complete. We then did a reboot to make sure that the OS was still good to go.

Hopefully the flaky behaviour we have seen so far will disappear now that one of the array members has outright failed … hopefully!

We will run Season 3 of 24 through the machine, balancing that with some local radio station tunes to make sure. :)

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*All Mac on SBS posts will not be written on a Mac until we replace our now missing iMac! (previous blog post)

Windows Live Writer

LSI Corp to purchase 3Ware/AMCC Storage Adapter Business

In an interesting twist of events:

• AMCC – Applied Micro Circuits Corporation Announces Sale of Its 3ware Storage Adapter Solutions Business.
• LSI – LSI To Purchase AMCC 3ware RAID Adapter Business.

Intel, Dell, and other manufacturers use LSI boards as the foundation for their RAID controller offerings.

How this purchase pans out for the RAID controller market has yet to be determined. We do hope that the 3ware products live on since they are as good as the Intel/LSI products we deal with and competition is a healthy thing for manufacturers and for our clients and us.

For us, Intel/LSI are the RAID controllers of choice. We have had great success with Intel’s cards over the years. So, we will continue to stick with them.

The only other RAID controller we would look at for specialized applications would be by Adaptec. Though that need has been virtually nonexistent for a few years now.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*All Mac on SBS posts will not be written on a Mac until we replace our now missing iMac! (previous blog post)

Windows Live Writer

Intel SR1625UR 1U Dual Xeon 5500 Series Configuration

The image looked good in Windows Live Writer when putting together yesterday’s post on the new Intel 5500 Series Xeons and related components (previous blog post).

But, I see now that the Intel Server Configurator screenshot turned out to be quite illegible.

So, there is that server configuration:

Description	Part Number	MM #	QTY
Intel® Server System SR1625UR (1U Passive Backplane)	SR1625URNA (Includes power cord for North America) and R1625UR	901132 (power cord included) or 900986	1
Intel® Xeon® Processor X5560 (Heat Sink not included)	BX80602X5560	901026	2
SR1625 Standard front panel accessory kit	ASR1625FP	900991	1
Quad port GbE I/O module	AXX4GBIOMOD2	901359	1
Intel® Remote Management Module 3	AXXRMM3	900961	1
Intel® RAID Controller SRCSASJV (8 internal/external SAS/SATA ports, up to 1GB cache)	SRCSASJV	892602	1
AXXRSBBU3 Intel® RAID Smart Battery Backup Module (Use with Intel® Integrated Server RAID)	AXXRSBBU3	900954	1
Slimline SATA DVD ROM - AXXSATADVDROM	AXXSATADVDROM	893637	1
Black Bezel (Standard Control Panel)	ASR1550BEZ	879406	1
Full extending Rail Kit	AXXHERAIL2	901082	1
RJ45 To DB9 Cable	AXXRJ45DB92	881944	1
650W Hot-Swap power supply module	ASR1625PS	901016	1
2.5" Hard Drive Carrier	FXX25HDDCAR	881681	2

Intel SR1625UR 1U 8 x 2.5” SAS Configuration

The above configuration would be implemented with the Hyper-V Role set up on Server Core where the VMs would have high load performance requirements.

Note that there are a few typos in the actual Server Configurator results. Make sure to verify the part numbers with distribution. We corrected the mistakes in the above grid.

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*All Mac on SBS posts will not be written on a Mac until we replace our now missing iMac! (previous blog post)

Windows Live Writer