Thursday 12 March 2015

Hyper-V: Broadcom Gigabit NICs and Virtual Machine Queues (VMQ)

Here is an explanation posted to the Expert’s Exchange forum that we believe needs a broader audience.

***

VMQ is a virtual networking structure allowing virtual Switch (vSwitch) networking to be processed by the various cores in a CPU. Without VMQ only one core would be processing those packets.

In a Gigabit setting the point is moot since the maximum of 100MB/Second or thereabouts per physical port is not going to tax any modern CPU core.

In a 10GbE setting where we have an abundance of throughput available to the virtual switch things change very quickly. We can then see a single core processing the entire virtual switch being a bottleneck.

In that setting, and beyond, VMQ starts tossing vSwitch processes out across the CPU's cores to distribute the load. Thus, we essentially eliminate the CPU core as a bottleneck source.

For whatever reason, Broadcom did not disable this setting in their 1Gb NIC drivers. As we understand things the specification for VMQ requires it to be disabled on 1GbE ports.

VMQ enabled on Broadcom NICs has caused no end of grief over the last number of years for countless Hyper-V admins. With Broadcom NICs one needs to disable Virtual Machine Queues (VMQ) on _all_ Broadcom Gigabit physical ports in a system to avoid what becomes a vSwitch traffic.

***

The above is a summary of conversations had with networking specialists. If there are any corrections or specifics that all y’all have about the VMQ structures please feel free to comment! :)

Philip Elder
Microsoft Cluster MVP
MPECS Inc.
Co-Author: SBS 2008 Blueprint Book

3 comments:

Anonymous said...

Just avoid Broadcom altogether. I've had trouble with them since physical server days. Intel all the way.

Philip Elder Cluster MVP said...

Concur 110%!

We are an all Intel house for our components and server systems.

stryqx said...

Hi Phil.

What you're describing here is actually RSS - Receive-Side Scale.

RSS distributes inbound traffic across logical processors.
VMQ (Virtual Machine Queue) provides direct plumbing between a virtual machine network interface and the physical network interface queues.