Monday 20 September 2010

Intel PROSet Adapter Team – Creating a VLAN Setup for Clustering on IMS

Our previous post on setting up an adapter team:

Once our adapter teams are created the next step for us is to create a dedicated VLAN for our Heartbeat network.

In this case we will have two adapter teams created:

  1. Management
    • NIC 0 Switch 1
    • NIC 0 Switch 2
  2. vSwitch
    1. NIC 1 Switch 1
    2. NIC 1 Switch 2

The above team names are what we are working with on our Intel Modular Server at this point.

So, we need to know what the index number is for each team:

  1. PROSetCL Team_Enumerate
    • image

Now that we know our index numbers, we can go ahead and create the two needed VLANs:

  1. PROSetCL Team_CreateVLAN 2 untagged Management
  2. PROSetCL Team_CreateVLAN 2 99 Heartbeat
  3. PROSetCL Team_EnumerateVLANS 2
    1. image

The “untagged” VLAN will remain connected to our management network. VLAN 99 will be our exclusive heartbeat network.

Now, in our Hyper-V Server 2008 R2 “console” we can see that we do indeed have 3 NICs to work with:

image

We will then go ahead and set our heartbeat subnet and bind the Hyper-V virtual switch to the vSwitch team.

Now that our teaming has been configured our node is ready to become a part of our cluster!

Philip Elder
MPECS Inc.
Microsoft Small Business Specialists
Co-Author: SBS 2008 Blueprint Book

*Our original iMac was stolen (previous blog post). We now have a new MacBook Pro courtesy of Vlad Mazek, owner of OWN.

Windows Live Writer

12 comments:

Axel Doms said...

great set of articles, thanks a lot.
I did notice for some reason that i had to create the tagged vlan first before it allowed me to create the untagged one.
starting with the untagged resulted in a non helpfull "VLAN cannot be created" message.
this happened on both the servers i tried to configure.

Philip Elder Cluster MVP said...

Axel,

That is our experience as well. We create the tagged/named setups before we can create the untagged one.

This is the default behaviour for both teamed and non-teamed NICs.

You are welcome! :)

Philip

Axel Doms said...

Philip,
I did the config on both servers, and checked for binding order as well.
Management : Untagged VLAN -> 192.168.10.x -> working fine. i can RDP to both servers from my workstation and servers can ping each other.
HeartBeat : Tagged 150 -> 192.168.150.x -> servers cannot ping each other
Since the teaming is in ALB mode I am assuming the switches themself do not need to be aware of it.
I would also assume that making both ports of the team, on which the HeartBeat VLAN is configured, tagged member of that VLAN, and untagged member of the default VLAN for management purposes, would be the way to go.
So I went into advanced switch config for both SW1 and SW2 and configured the relevant ports as tagged members of vlan id 150.
no luck however, they still don't see each other.
can you tell me what kind of additional configuration was required on the IMS switches to make this work for you ?
I've been wrestling with the advanced configuration control panel of both switches for a few hours now playing with various vlan settings while the servers are in a ping -t loop, but I've kind of hit a wall after trying all possible settings.
My plans to connect a machine to an external port for advanced troubleshooting were foiled by my laptop dying on me as I was walking to the server room :-)
thanks and cheers,
Axel

Philip Elder Cluster MVP said...

Axel,

If the NICs are being teamed using one from the compute and one from the Mezzanine, thus separate switches, then the team must use Switch Balance Mode from what we have been able to determine.

We have not delved into this any further as we have been too busy to rework things.

However, the advanced management for each switch also needs to be configured to allow any tagged traffic within or between the switches.

We were stalled at this point as well as in the "how" to configure the on board switches and the Cisco SA520s for tagged traffic on different subnets to run to a dedicated Internet connection. :|

We will hopefully have some time to get into this setup again soon!

Philip

Axel Doms said...

Philip,
I fixed it thanks to you pointing me in the right direction.
What I was not aware of yet was how the switches were interconnected exactly.
I opened the advanced switch config for SW1 and had a look at Layer 2 > Address Table > dynamic Addresses. From this data it becomes obvious that the switches interconnect through interface 10G.XC. So as soon as I made this interface a tagged member of the appropriate VLAN on boths switches things started working. In hindsight I feel rather stupid for not thinking of this myself but ok ;)
In my defence, I could be overlooking it but I consulted the AXXSW1GB user guide for this issue and it didntt seem to mention the 10G.XC or its function anywhere.
a few sidenotes :
-there is another 10 gigabit interface, 10G.SC, I wonder what that one does.
-I noticed the switch control panel behaves slightly less erratic when opened in chrome, in comparison to firefox. When I open it in FF it sometimes (usually first attempt) opens with user ReadOnly instead of user Admin. very annoying. In case you experience that as well you may want to give chrome a try.

Philip Elder Cluster MVP said...

Axel,

Once we have IE8 configured correctly we don't have an issue with using it for managing the IMS and working with its advanced consoles.

Philip

Nico said...

Dear Philip,

I have following your articles for setup the clustering on IMS (http://blog.mpecsinc.ca/2010/09/ims-cluster-nic-setup-worksheet.html).
We have in our IMS double switches. I have create with PROsetCL port, NIC0 with mezz NIC0 in one team and NIC1 with mezz NIC1 in a team. The first team is for management and heartbeat and the second is the virtual machine switch. On the first team we have a untagged management vlan and a vlan for heartbeat. I don't get the heartbeat vlan working. I have now configure one vlan 99 heartbeat on switch 2, and have tagged this on the right server ports. I have for testing Ext.2 port untagged and link this port with a router. In the router have I setup a DHCP scope. The DHCP scope don't give a IP-adres to the heartbeat NIC. How must I correct configure this on the switch to let it works?

I hope that you have any suggestion!

Thanks in advance.

Regards,
Nico

Axel Doms said...

Hi Nico,
i've tried to make a basic drawing of my setup, just showing one module for simplicity's sake, same principles apply for the other module(s).
Basically 1 and 3 make a team, with both untagged and tagged vlan membership as you described earlier;
2 and 4 make a team, which will be directly exposed to hyper-v to use as a virtual switch (vlan tagging will then be done in the hyper-v management interface)
Also you need to make 9 and 10 a member of the tagged vlans that you use, I forgot to do this the first time.
this should be enough for the heartbeat to work.
As soon as you put VM's in tagged VLAN's you may need to adjust VLAN membership of 11 and 12 (these symbolize the external ports) as well.
As far as i know, changing vlan settings for 9 and 10 can only be done through the advanced switch management interface.
Also, in my particular case, I noticed that as soon as I use the simple management interface to change a setting on one of the switches, it screws up the settings I may have done in the advanced management interface, even if that setting was not even exposed in the simple interface (this can be very troublesome to experience in a production environment !)
So for me personally, I stopped using the simple management interface all together.
kind regards,
Axel

Philip Elder Cluster MVP said...

Axel,

Excellent drawing. That hits the mark exactly.

Thanks for filling in the blanks. :)

Philip

Nico said...

Axel,

Thanks a lot for the explainable drawning. I have the heartbeat working now. :)

I have now only a question about the vSwitch. You write that the VLAN tagging must done on the Virtual Machine Manager, do you nothing to do on the switch?

Currently we have 2 nodes in the blades all working for a year. The nodes a not in a cluster. On SWM1 do I have a trunk created on port Ext.1 to a Sonicwall. Now I tagged many VLANS (different customers) on the trunk. This do I also on the virtual VLAN trunk in the Hyper-V manager. Than I tagged the right VLAN on the VHD. Is this the correct way of do you advice to do it on another way? It is the intention to bring the 2 nodes also in the cluster!!

Thank in advance.

Regards,
Nico

Nico said...

Axel,
Thanks a lot for the explainable drawning. I have the heartbeat working now. :)

I have now only a question about the vSwitch. You write that the VLAN tagging must done on the Virtual Machine Manager, do you nothing to do on the switch?
Currently we have 2 nodes in the blades all working for a year. The nodes a not in a cluster. On SWM1 do I have a trunk created on port Ext.1 to a Sonicwall. Now I tagged many VLANS (different customers) on the trunk. This do I also on the virtual VLAN trunk in the Hyper-V manager. Than I tagged the right VLAN on the VHD. Is this the correct way of do you advice to do it on another way? It is the intention to bring the 2 nodes also in the cluster!!

Thank in advance.

Regards,
Nico

Axel Doms said...

Nico,
you are welcome.
I created this setup a while ago as a test and am not running any hyper-v cluster right now, but I just went over my installation notes again to see how I configured the VLAN last time.
I believe I did not tag the physical interface (TEAM 2) but I did create trunks on the switches, matching tags that I used in the virtual machine manager.
So using the image from my earlier post that means tagging 6, 8, 9, 10, 11 and 12 with all the tags you want to use on your virtual machines. But not 2 and 4 themselves.
The instructions I got from this article which goes a bit more in depth on this matter.
hope this helps,
cheers,
Axel