Cluster: Troubleshooting an Issue Using Failover Cluster Manager Cluster Events

When we run into issues the first thing we can do is poll the nodes via the Cluster Events log in Failover Cluster Manager (FCM).

1. Open Failover Cluster Manager
2. Click on Cluster Events in the left hand column
3. Click on Query
• 
4. Make sure the nodes are ticked in the Nodes: section
5. In the Event Logs section:
• Microsoft-Windows-Cluster*
• Microsoft-Windows-FailoverClustering*
• Microsoft-Windows-Hyper-V*
• Microsoft-Windows-Network*
• Microsoft-Windows-SMB*
• Microsoft-Windows-Storage*
• Microsoft-Windows-TCPIP*
• Leave all defaults checked
• OPTION: Hardware Events
6. Critical, Error, Warning
7. Events On
• From: Events On: 2017-12-17 @ 0800
• To: Events On: 2017-12-18 @ 2000
8. Click OK
9. Click Save Query As...
10. Save it
• Copy the resultant .XML file for use on other clusters
• Edit the node value section to change the node designations or add more
11. Click on Save Events As... in FCM to save the current list of events for further digging

Use the Open Query option to get to the query .XML and tweak the dates for the current date and time, add specific Event IDs that we are looking for, and then click OK.

We have FCM and Hyper-V RSAT installed on our cluster's physical DC by default.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Web Site
Our Cloud Service

PowerShell TotD: Hyper-V Live Move a specific VHDX file

There are times when we need to move one of two VHDX files associated with a VM.

The following is the PowerShell to do so:

Poll Hyper-V Host/Node for VM HDD Paths

get-vm "*" | Select *path,@{N="HDD";E={$_.Harddrives.path}} | FL

Move a Select VHDX

Move-VMStorage -VMName VMName -VHDs @(@{"SourceFilePath" = "X:\Hyper-V\Virtual Hard Disks\VM-LALoB_D0-75GB.VHDX"; "DestinationFilePath" = "Y:\Hyper-V\Virtual Hard Disks\VM-LALoB_D0-75GB.VHDX"})

Move-VMStorage Docs

The Move-VMStorage Docs site. This site has the full syntax for the PowerShell command.

Conclusion

While the above process can be initiated in the GUI, PowerShell allows us to initiate a set of moves for multiple VMs. This saves on time bigtime versus mouse.

By the way, TotD means: Tip of the Day.

Thanks for reading! :)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Web Site
Our Cloud Service

Intel Server System R2224WFTZS Integration & Server Building Thoughts

We have a brand new Intel Server System R2224WFTZS that is the foundation for a mid to high performance virtualization platform.

Intel Server System R2224WFTZS 2U

Below it sits one of our older lab Intel Server System SR2625URLX 2U. Note the difference in the drive caddy.

That change is welcome as the caddy no longer requires a screwdriver to set the drive in place:

Intel 2.5" Tooless Drive Caddy

What that means is the time required to get 24 drives installed in the caddies went from half an hour or more to five or ten minutes. That, in our opinion, is a great leap ahead!

The processors for this setup are Intel Xeon Gold 6134s with 8 cores running at 3.2GHz with a peak of 3.7GHz. We chose the Gold 6134 as a starting place as most of the other CPUs have more than eight cores thus pushing up the cost of licensing Microsoft Windows Server Standard or Datacenter.

Intel Xeon Gold 6134, Socket, Heatsink, and Canadian Loonie $1 Coin

The new processors are huge!

The scale difference between the E3-1200 series, E5-2600 series is orders of magnitude larger. The jump in size reminds me of the Pentium Pro's girth next to the lesser desktop/server processors of the day.

Intel Xeon Processor E3-1270 sits on the Intel Xeon Gold 6134

The server is nearly complete.

Intel Server System R2224WFTZS Build Complete

Bill of Materials

In this setup the server's Bill of Materials (BoM) is as follows:

• (2) Intel Xeon Gold 6134
• 384GB via 12x 32GB Crucial DDR4 LRDIMM
• Intel Integrated RAID Module RMSP3CD080F with 7 Series Flash Cache Backup
• Intel 12Gbps RAID Expander Module RES3TV360
• (2) 150GB Intel DC S3520 M.2 SSDs for OS
• (5) 1.9TB Intel DC S4600 SATA SSDs for high IOPS tier
• (19) 1.8TB Seagate 10K SAS for low to mid IOPS tier
• Second Power Supply, TPM v2, and RMM4 Module

It's important to note that when setting up a RAID controller instead of a Host Bus Adapter (HBA) that does JBOD only we require the flash cache backup module. In this particular unit one needs to order the mounting bracket: AWTAUXBBUBKT

I'm not sure why we missed that, but we've updated our build guides to reflect the need for it going forward.

One other point of order is the rear 2.5" hot swap drive bay kit (A2UREARHSDK2) does not come installed from the factory in the R2224WFTZS as it did in the R2224WTTYS. I'm still not sold on M.2 for the host operating system as they are not hot swap capable. That means, if one dies we have to down a node in order to change it. With the rear hot swap bay we can do just that, swap out the 2.5" SATA SSD that's being used for the host OS.

For the second set of two 10GbE ports we used an Intel X540-T2 PCIe add-in card as the I/O modules are not in the distribution channel as of this writing.

NOTE: One requires a T30 hex screwdriver for the heatsinks! After installing the processor please make sure to start all four nuts prior to tightening. As a suggestion, from there snug each one up gradually starting with the two middle nuts then the outer nuts similar to the process for installing a head on an engine block. This process provides an even amount of pressure from the middle of the heatsink outwards.

Firmware Notes

Finally, make sure to update the firmware on all components before installing an operating system. There are some key fixes in the motherboard firmware updates as of this writing (BIOS 00.01.0009 ReadMe). Please make sure to read through to verify any caveats associated with the update process or the updates themselves.

• Intel Download Centre: Downloads for Intel Server System R2224WFTZS

Next up on our build process will be to update all firmware in the system, install the host operating system and drivers, and finally run a burn-in process. From there, we'll run some tests to get a feel for the IOPS and throughput we can expect from the two RAID arrays.

• Intel Server Edge Site
• Intel Server Configurator Site
• Intel Tested Hardware and Operating System List (THOL)
• This one is awkward. Just click through and log on as Guest.
• Intel Server Systems Site

Why Build Servers?

That's got to be the burning question on some minds. Why?

The long and the short of it is because we've been doing so for so many years it's a hard habit to kick. ;)

Actually, the reality is much more mundane. We continue to be actively involved in building out our own server solutions for a number of reasons:

• We can fine tune our solutions to specific customer needs
• Need more IOPS we can do that
• Need more throughput we can do that
• Need a blend of the two as is the case here, then we can do that too.
• Direct contact with firmware issues, interoperability, and stability
• Making the various firmware bits play nice together can be a challenge
• Driver issues, interoperability, and stability
• Drivers can be quite finicky about what's in the box with them
• Hardware interoperability
• Our parts bin is chalk full of parts that refused to work with one another
• On the other hand our solution sets are known good configurations
• Cost
• Our server systems are a fraction of the cost of Tier 1
• Overall system configuration
• As Designed Stability out of the box
• He said She said
• Since we test our systems extensively prior to deploying we know them well
• Software Vendors that point the finger have no leg to stand on as we have plenty of charts and graphs
• Performance issues are easier to pinpoint in software vendor's products
• We remove the guesswork around an already configured Tier 1 box

Business Case

The business case is fairly simple: There are _a lot_ of folks out there that do not want to cloud their business. We help customers with a highly available solution set and our business cloud to give them all of the cloud goodness but keep their data on-premises.

We also help I.T. Professional Shops who may not have the skill-set on board that have customers with a need for High Availability and a cloud like experience but want the solution deployed on-premises.

For those customers that do want to cloud their business we have a solution set for the Small to Medium I.T. Shops that want to provide multi-tenant solutions in their own data centres. We provide the solution and backend support at a very reasonable cost while they spend their time selling their cloud.

All in all, we've found ourselves a number of different great little niches for our highly available solutions (clusters) over the last few years.

Thanks for reading! :)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Web Site
Our Cloud Service
Twitter: @MPECSInc

A Little Plug for Mellanox and RoCE RDMA

RoCE (RDMA over Converged Ethernet) via Mellanox NICs and switches is our primary fabric choice for Storage Spaces Direct (S2D) and Scale-Out File Server (SOFS) to Hyper-V compute cluster fabric.

With the Mellanox MSX1012X 10GbE switch we can deploy a pair of them along with a pair of ConnectX-4 Lx dual port NICs per node for about the same cost as a pair of NETGEAR XS716T 10GbE switches and a pair of Intel X540/X550-T2 10GbE RJ45 based NICs per node.

We have a great business relationship with Mellanox. They are great folks to work with and their product support is second to none.

I was honoured to be asked to use a portion of my presentation for MVPDays to create the following video that is resident on Mellanox's YouTube channel.

Hopefully the video comes out okay as embedding it was a bit of a chore.

Thanks for reading and have a great weekend!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Error Fix: Event 7034 Service Control Manager - Server, BITS, Task Scheduler, Windows Management Instrumentation, Shell Hardware Detection Crashes

This has just recently started to pop up on networks we manage.

All of the following are Event ID 7034 Service Control Manager service terminated messages:

• The Windows Update service terminated unexpectedly. It has done this 3 time(s).
• The Windows Management Instrumentation service terminated unexpectedly. It has done this 3 time(s).
• The Shell Hardware Detection service terminated unexpectedly. It has done this 3 time(s).
• The Remote Desktop Configuration service terminated unexpectedly. It has done this 3 time(s).
• The Task Scheduler service terminated unexpectedly. It has done this 3 time(s).
• The User Profile Service service terminated unexpectedly. It has done this 3 time(s).
• The Server service terminated unexpectedly. It has done this 3 time(s).
• The IP Helper service terminated unexpectedly. It has done this 2 time(s).
• The Device Setup Manager service terminated unexpectedly. It has done this 3 time(s).
• The Certificate Propagation service terminated unexpectedly. It has done this 2 time(s).
• The Background Intelligent Transfer Service service terminated unexpectedly. It has done this 3 time(s).
• The System Event Notification Service service terminated unexpectedly. It has done this 2 time(s).

It turns out that all of the above are tied into SVCHost.exe and guess what:

Log Name: Application
Source: Application Error
Date: 10/23/2017 5:09:57 PM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
Computer: ABC-Server.domain.com
Description:
Faulting application name: svchost.exe_DsmSvc, version: 6.3.9600.16384, time stamp: 0x5215dfe3
Faulting module name: DeviceDriverRetrievalClient.dll, version: 6.3.9600.16384, time stamp: 0x5215ece7
Exception code: 0xc0000005
Fault offset: 0x00000000000044d2
Faulting process id: 0x138
Faulting application start time: 0x01d34c5c3f589fe7
Faulting application path: C:\Windows\system32\svchost.exe
Faulting module path: C:\Windows\System32\DeviceDriverRetrievalClient.dll

A contractor of ours that we deployed a greenfield AD and cluster for was the one who figured it out. WSUS and the Group Policy settings were deployed this last weekend with everything in our Cloud Stack running smoothly until then.

The weird thing is, we have had these settings in place for years now without any issues.

The following are the settings changed at both sites:

System/Device Installation
Specify search order for device driver source locations: Not Configured
2014-02-11: Enabled by Philip Elder.
2017-11-01: Not Configured by Philip Elder.
Specify the search server for device driver updates: Not Configured
2014-02-11: Enabled by Philip Elder.
2017-11-01: Not Configured by Philip Elder.

System/Driver Installation
Turn off Windows Update device driver search prompt: Not Configured
2017-10-28: Disabled by Philip Elder.
2017-11-1: Returned to Not Configured by Philip Elder

System/Internet Communication Management/Internet Communication settings
Turn off Windows Update device driver searching: Not Configured
2014-02-11: Disabled by Philip Elder.
2017-11-01: Not Configured by Philip Elder.

It is important to note that when working with Group Policy settings a comment should be made in each setting if at all possible. Then, when it comes to troubleshooting an errant behaviour that turns out to be Group Policy related we are better able to figure out where the setting is and when it was set. In some cases, a short description of the "Why" the setting was made helps.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Xeon Scalable Processor Motherboard CPU-Soft Lockup Fix

The new Intel Purley based Intel Server Boards S2600WF, S2600BP, and S2600ST Product Family use a new BMC (Baseboard Management Controller) video subsystem.

As a result, some operating systems, mostly *NIX based, will choke on install as they may not have the driver built-in.

Intel Technical Advisory: Intel® Server Board S2600WF, S2600BP and S2600ST Product Family fail to initialize the operating system video driver for the ASPEED* Base Management Controller (BMC).

That document point's to ASPEED's site for downloading an up to date driver that fixes the problem.

Root Cause
Full root cause of this issue has been determined. Intel has confirmed that the failure has no bearing on system performance, it only impacts local video graphics. In detail, when the operating system loads, the OS-embedded ASPEED* video driver is not able to access a portion of the BMC memory space, therefore the process stalls.

On Windows Server based configurations we need to update the driver once the OS is installed. The default VGA driver that comes built-in to the OS works just fine.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Fujitsu ScanSnap N1800: E-mail Button Greyed Out Fix

We have moved a ScanSnap N1800 onto a new greenfield setup in a side-by-side migration we've been running.

In this case, the Exchange server is on-premises with the appropriage Anonymous MFP Relay setup configured.

Searching about turned up what turned out to be a simple fix though not one we would prefer: Enable a mailbox in Exchange for the scanner's account.

Once we did that the e-mail button did indeed appear and work with subsequent scan and send tests being successful.

Note that the account being used has a rediculously long password that never changes and is restricted on the domain. So, the attack surface is relatively small.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Some thoughts on Windows Server 1709 and Where to Find It

We were looking for the new Software Assurance benefit based download of Windows Server 1709 in Microsoft's Volume Licensing Service Center.

It took a bit to realize that the download was not tied into "Windows Server 2016" and it's available downloads.

The following two items show up in search for "Windows Server":

When we click through and try to download either one they both point to the same download:

Keep in mind that 1709 is a Server Core only option and receives updates every six months. Plus, the service life of each release is 18 months.

That means that adopting the Semi-Annual Channel (SAC) release of Windows Server would require a significant investment in both testing prior to deployment and in deploying the OS on a regular basis.

Keep in mind that Software Assurance is required for access to SAC.

Is it of value? For those businesses that are looking to adopt newer/better features via quicker cadence then yes, there is value in it.

For those that are looking for long-term stability in their deployments then the Long Term Service Branch/Channel (LTSB/C) is the way to go.

For us, we are in a "Wait and see" mode as our focus is currently Storage Spaces and Hyper-V along with Storage Spaces Direct clusters.

As far as SAC being a Server Core option only we don't have a problem with that now do we? ;)

Realistically though, there may be a lot of really neat features and abilities that may only appear in the SAC branch of Windows Server as we go along. That has yet to be seen, but given Microsoft's push to add value to Software Assurance over the last number of years one can comfortably wager that there will be extra value in that branch of the OS.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Microsoft Groove to Spotify Move

When Microsoft announced the discontinuation of the Groove Music Pass there was a lot of dissappointment around here.

Groove offered the ability to check out all sorts of music over the years without having to fork out a buck or two on a song that ended up not being listened to.

With the ability to download music to four different devices for offline listening it was a really good value for the money.

So, off we go into the migration from Groove to Spotify.

First off, this was one of the smoothest transitions ever experienced. Everything moved over without a hitch. It took a bit but nothing was lost in the process!

Score one for Microsoft and Spotify!

Score two for Spotify: The $15/Month Premium Family Plan for up to five folks under one roof was the clincher.

We were looking to obtain two more Groove Music Pass accounts. One for my wife and the other for our daughter. That would have been expensive!

A couple of settings in the Spotify app to take note of after upgrading to Premium:

• Settings - Music Quality: Enable High quality streaming (Premium only)
• Settings - Social: Disable Automatically make new playlists public
• Settings - Social: Enable Private session

The last two are personal preference but as a rule we will make sure our kid's apps are set up this way.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Fix: Unable to Message Skype Contacts

This was a weird one. Skype on my Windows Phone stated "Messaging Unavailable" for my Dad.

The Skype apps would not present any form of his contact.

While logged in to the OneDrive site I saw a Chat bubble. I clicked on it and typed in his name.

Low and behold his contact came up BLOCKED in red.

Huh?

I didn't do that and nowhere else in the Skype ecosystem on any of my devices did that status come up.

So, I unblocked him but still no joy.

I had him log on to the OneDrive site, click the chat bubble beside the bell, and search for me and sure enough, I came up BLOCKED.

After his unblocking me our mutual contacts lit up.

Yo Skype, what the chicken?!?

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Storage Live Migration: Where's the Move Status?

We're in the process of re-working a client's cluster setup (Dell MD3220 DAS + (2) Dell R520 Hyper-V Nodes) by adding drives and a new LUN.

Note that until the new RAID array on the MD3220 has finished initializing the drives will remain Read-Only.

A Storage Live Migration (SLM) can be initiated from within Failover Cluster Manager, Hyper-V Manager, or via PowerShell (Move-VMStorage on TechNet).

To see how things are moving along we can check in Hyper-V Manager:

The VHDX in question is over 500GB in size and it's taking a while to move!

Note that the VM remains online all the while and also once the SLM completes.

Once we've completed our storage re-organization we have some new workloads to configure.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Client E-mail Warning for the Current Malware Campaigns

This went out this morning. The first place in any "security strategy" should be to train the human.

Folks,

I hope you had a great summer!

With anti-SPAM services getting better and better the malicious folks out there are getting a lot more subtle in their efforts plus we’re seeing an uptick of baddies in the Inbox.

Things to note in the message below:

1. The FROM domain @fmelaw.com does not match the domain in the link
2. After hovering the mouse over the Here link the URL listed contains a bunch of gibberish
3. Watch for language, spelling, and grammar errors as there tends to be a lot of them
4. Is the Subject and/or Sender legit? Call them first!
5. Do NOT open any Word documents and especially do NOT click Enable Macros if prompted!
6. Be cautious with any PDF attachments. If in doubt call the sender or forward to here with a question.

NOTE: We are seeing _a lot_ of compromised e-mail addresses and mailboxes as a result of users opening something or clicking on something they should not have.

One attack vector is via a Macro enabled Word document harvests both E-mail and Addresses to send out _replies_ to a legitimate e-mail thread/conversation. If the Word document gets clicked on and a prompt comes for enabling Macros the Word document is BAD. CLOSE Word and SHIFT+DELETE the e-mail!

If in doubt, don’t open or click on it! Do _not_ hesitate to call or forward the questionable content!

Thank you and have a wonderful day! :)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Enable 2FA (Two Factor Authentication) Everywhere It's Available!

Yes, it's a bit of an extra inconvenience.

But, that inconvenience may save the account and any data associated with it from being hijacked!

As an example, after logging into my Microsoft ID and heading into the Security section I can check and see if there is anything out of the ordinary.

And, low and behold what do I find? That I've attempted to log on from some interesting places!

2FA is enabled on this Microsoft ID and all others. Amazon, Blogger, Microsoft,and any other that offer 2FA has it enabled.

There's absolutely no way in this day and age that it should not be used.

Thanks for reading. :)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Server 2012 R2 BitLocker Post Install Error: Unspecified Error

The following error happened on a DC we recently set up and were going to encrypt via BitLocker:

C:\Users\USERNAME\AppData\Local\Packages\windows.immersivecont...
C:\Users\USERNAME\Classic_{GUID}.settingcontent-ms
Unspecified error

A quick search turned up a simple fix: Reboot the server a second time.

Sure enough, good to go:

As a rule, we deploy a TPM in all of our physical DCs that are deployed with our clusters. They are then encrypted using BitLocker. This greatly reduces the exposure to compromise if someone has physical access to that DC. For virtual DCs, we now have the ability to pass a vTPM through to the guests in Server 2016. We're still in the testing phase, but our plan is to have _all_ domain controllers on networks we manage encrypted!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

A Few Thoughts on the Intel Xeon Processor Scalable Family

The original article is here: Intel® Xeon® Processor Scalable Family Technical Overview.

This quick post is for the time challenged folks trying to figure things out as far as how the new Intel Xeon Processor Scalable Family relates to the previous generation Intel Xeon Processor E5-2600 series.

Please note that all of the images below are from the above article.

The above grid gives us an idea of which processor grade goes where. Our standard go-to has been the Intel Xeon Processor E5-2620 through the E5-2640 which were at one time the mainstream processors.

The next tier for us would be the E5-26*3 and E5-26*7 series that provided high bin counts (GHz) with low core counts.

Now we can see that the mainstream processors are Silver and the performance grade are Gold.

In the charge above 2S, 4S, 8S is the number of sockets the processor supports. DPC is DIMMs Per Channel.

As we can see, there are just a few new features included in the new processor family.

Some Thoughts

There is a definite glaring omission in this new processor family: Fourth Generation PCIe :(

As we all know, the data bus is playing catch-up (blog post) to storage and to some extent networking.

While the newly introduced Purley platform has integrated PCIe NVMe ports on the server boards and backplanes there is still a lack of clarity as far as what we need to make things work on the Intel Server System platform.

The PCIe channel count bump from 32 to 48 is most certainly not enough especially with the spec stuck in Generation 3. A pair of 100Gb Mellanox Ethernet cards and a few PCIe NVMe SSDs and we're pretty much saturating the bus ... again.

And one more thing as we've not had a chance to compare apples to apples yet, the new processors look to be more expensive than the previous generation E5-2600v4 equivalents. And, it seems as the core counts go up so do the prices in an almost exponential way.

We'll post some price comparisons in another blog post.

Have a great weekend and thanks for reading!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service

Some Thoughts on Writing in Digital and the Surface Pro 4

I personally love to write. With real pens and ink. :)

Currently, we have a pending order with a Taiwanese vendor for some TWSBI fountain pens.

The above image is from TWSBI's web site. It's a TWSBI Diamond 580 Clear. There's also a Mini version that travels a lot better.

Pilot makes some gorgeous inks. The above ajisai is a pretty neat colour that will be the go-to for regular note taking. We have several different colours on their way at the moment.

When there is a need to write in pencil the Platinum PRO-USE 03 (MSD-1500) is one of the best mechanical pencils ever made in my opinion.

When it comes to art, my primary medium is coloured pencil on various media or graphite pencil also on various media. I'm currently working on a Tiger Moth Orchid using Faber-Castell Polychromos oil based colour pencils.

All of the above is to bring about just how important the digital ink experience needs to be. While not a professional digital ink writer or artist by any means, the digital ink experience is quite important.

To date, my personal best digital pen experience for both writing and art has been with the Microsoft Surface Pro 3 with the Pro 4 (SP4) being even better.

The SP4 provides an excellent platform for one who prefers to write over type.

OneNote has an excellent recognition process that allows for hand written notes to be copied and pasted into Word. For those that take notes at meetings to provide minutes at a later date this feature works great!

It's also great for those that attend conferences to gain information. Writing the notes on the fly can be a lot faster, especially for those of us that developed a written shorthand while in university classes back in the day. ;)

Tie in the taking of pictures to use as a reference later in the day when re-working the handwritten notes into a final set and we have a pretty good method for building some pretty good written work such as articles, blog posts, or even books.

Side Note: Another aspect of writing versus typing is in memory retention. Retention seems to be _a lot_ better when notes are taken live with a pen versus typing those notes in. Retention gets even better if the "crib" notes and pictures are re-worked later that day into a final set of notes.

At some point time will be spent with the Microsoft Surface Studio. It seems to be about the best platform out there for the artistically inclined. We certainly know of quite a few engineering, architectural, and other such firms either switching or looking to switch to the Surface Studio.

The one catch though is that it is difficult to let go of pen and paper when it comes to art. For some, the "analog" versus "digital" art "discussion" can be quite "religious" in nature. ;)

Suffice it to say, if looking for a new ultra-portable system that will run most work related applications and provide an excellent platform for the written word the Surface Pro 4 is the one to choose.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Edge Browser: Reset After Malware How To

Every time a client of ours opened Edge they would receive a big red screen with "Edge has been compromised".

With the Edge option to open previous tabs/pages there is no real way to get out of the loop. We cleaned out the Edge temporary files folder and the problem still happened.

So, to fix it we needed to nuke & pave.

We do that by running the following two steps on the problematic machine:

1: Delete:
C:\Users\%username%\AppData\Local\Packages\Microsoft.MicrosoftEdge_8wekyb3d8bbwe

2: Elevated PowerShell all on one line:
Get-AppXPackage -AllUsers -Name Microsoft.MicrosoftEdge | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register "$($_.InstallLocation)\AppXManifest.xml" -Verbose}

3: Start Edge

With the above process complete the user should get the "Welcome to Edge" message and tabs.

NOTE: This process essentially removes and re-installs Edge. _ALL_ settings, saved passwords, and such are removed!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Exchange: ERROR: The internal transport certificate cannot be removed... FIX

We recently renewed an Exchange server's trusted certificate.

When we went to remove the old certificate in EAC we received the following error:

error
A special Rpc error occurs on server SERVERNAME: The internal transport certificate cannot be removed because that would cause the Microsoft Exchange Transport service to stop.
To replace the internal transport certificate, create a new certificate. The new certificate will automatically become the internal transport certificate. you can then remove the existing certificate.

Searching turned up a lot of suggestions to just delete the old certificate in the Personal certificates store. Somehow, that did not strike as being the correct methodology since the error makes it clear that the old certificate is still in use.

The proper methodology is to run the following PowerShell in the Exchange Shell to create and bind a new self-issued certificate. Since the certificate is bound to internal services there are no trust issues as indicated by the error message.

New-ExchangeCertificate -IncludeServerFQDN -IncludeServerNetBIOSName -Confirm:$False

The result would be something like this:

Once the command has completed we were able to delete the expired third party certificate in EAC.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Intel JBOD2224S2DP - Troubleshooting Redundant Path Fail

We have an Intel JBOD2224S2DP that has seemingly dropped one of its expanders as we are seeing a MPIO path error on both nodes in a Hyper-V/Storage Spaces cluster (2x nodes + 2x JBODs).

First step is to get the SAS IDs for the expanders by pulling the cover:

With IDs in-hand the next step is to figure out which one has failed.

We do this by downloading the latest firmware for the JBOD and copying the contents to a \TMP folder on the server or server node.

Open an elevated CMD on the server/node and:

C:
CD \TMP\Windows [ENTER]
cmdtool2_64 -adpsetprop ExposeEnclDevicesEnbl 1 -aall [ENTER]
xflash -I get avail [ENTER]

And, voila! We have our culprit:

The problematic expander is the one on the right.

The final step to run on the server/node:
cmdtool2_64 -adpsetprop ExposeEnclDevicesEnbl 0 -aall [ENTER]

Now, off to call Intel to see about a warranty replacement or to find one out there somewhere. ;)

UPDATE 2017-08-16: As it turns out, we replaced the seemingly problematic expander and still had the error. After swapping the RS25GB008 HBA pair between nodes the problem followed the HBAs. After a bit more testing we found that one of the RS25GB008 HBAs had a bad port.

Since Intel no longer supports them and distribution didn't have any in the channel we had to go out and find some via the regular channels. They just arrived the other day and we now have MPIO on both systems without an error.

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Windows Server 2016 July 18, 2017 CU is Important!

The July 18, 2017-KB4025334 (OS Build 14393.1532) Update is _important_!

There are fixes in there for a lot of cluster specific products.

• iSCSI
• S2D
• ReFS
• DeDup
• MPIO
• NTFS

The specifics are in the Microsoft page linked to above as is a download link.

We are in the process of updating our base Install.WIM image (blog post) with this update as I write this!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Windows 10: Installing on Intel Desktop Board DX79SR

Boy, did we get a lot of grief trying to get Windows 10 to install on an Intel Desktop Board DX79SR based system!

Windows Setup

Windows cannot be installed to this disk. This computer's hardware may not support booting to this disk. Ensure that the disk's controller is enabled in the computer's BIOS menu.

Some pointers:

• Windows 10 DX79SR Raid (thread 109296)
• Problems with intel (sic) DX79sr running windows (sic) 10 (thread 83709)

Neither post is available for comment thus this blog post plus a new discussion on the Intel Communities site: Windows 10 on Intel Desktop Board DX79SR.

What finally worked:

1. Set up RAID in RSTe (CTRL+I)
2. Set BIOS Boot Mode to UEFI
3. Plug in ISO mount type enclosure with Win10 ISO mounted (we use StarTech S2510BU3ISO)
4. NOTE: I had to use the USB2 ports as the USB3 ports did not power the enclosure during boot
5. F10 during POST
6. Choose DVDROM - UEFI (name may vary)
7. Click through and choose ADVANCED Setup
8. Click on the RAID array logical disk for the OS
   1. NOTE: If any MBR partitions exist they need to be cleaned prior to this step
   2. Use DiskPart in Repair --> CMD
9. Click NEXT

That should do it!

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc

Mellanox PPC SwitchX Update v3.6.4006

Mellanox has released a firmware update for their SwitchX switches: v3.6.4006.

We've already updated our two SX1012 switches to v3.6.3508 as per our blog post Mellanox Prep for RoCE RDMA. That means that we'll be able to upgrade without any intermediary steps as per the section Upgrade From Previous Versions.

When looking into the Release Notes for the new firmware version we see:


Note that in our case we are running ConnectX-3 Pro ( MCX354A) adapters. So, we'll be keeping firmware 2.4.5030 on those NICs until such time as Mellanox lets us know that we are able to bump them up to 2.4.7000.


Looking in the Changes and New Features section there doesn't seem to be anything specific to us however there are quite a few items listed for versions between v3.6.3508 and v3.6.4006!

There are a few items in the General Known Issues section that we need to be aware of.

• Point 32: Statistics files are reset which means graphs get reset.
• Point 49 indicates that a faulty cable may cause other ports to delay their "rise". 
• Point 50 is important. 40GbE passive copper cables 5m in length may experience "rise" issues if connected to a third party 40GbE NIC.
• Point 93: Break-out Cables
• Odd ports might suffer from Tx drops even when global flow control is enabled.
  Set the egress poll to 8M using the following command:
  “pool ePool0 direction egress-mc size 8M type dynamic”.
•  Point 128: QoS: ETS does not work on SN2100 switch system.

I suggest checking out the Bug Fixes section near the end of the document. ;)

Philip Elder
Microsoft High Availability MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book
Our Cloud Service
Twitter: @MPECSInc