redundancy and performance for our ESXi 5.5 server. HP got the IRF
technology from H3C.
Here a short summary, what need to be configured, to get a nice high availability setup, as always the documention and examples
found on the internet are not so good:
Prerequisite: same firmware version on both switches and two JD360B 2-port 10GbE Local Connect Modules.
step 1
The local connect modules are not build in, yet. Login on both switches.
step 2
Set the member id for both switches:
[HP-ESX-Master]irf member 1 renumber 1
[HP-ESX-Slave]irf member 1 renumber 2
Then on both:
[HP-All]quit
[HP-All]save
[HP-All]reboot
step 3
Configure IRF MAC persistency, so that the "virtual" switch is always available with the same MAC address.
[HP-ESX-Master]irf mac-address persistent always
Configuration of the priority:
[HP-ESX-Master]irf member 1 priority 32
[HP-ESX-Slave]irf member 1 priority 30
Then on both:
[HP-All]quit
[HP-All]save
[HP-All]reboot
step 4
Put the 10GE local connect modules into the switches and connect them cross. That means master switch port 1 connected to slave switch port 2 and
master switch port 2 connected to slave switch port 1.
step 5
Connect the physical IRF ports with the logical ones.
Shutdown the interfaces:
[HP-ESX-Master]interface Ten-GigabitEthernet 1/1/1
[HP-ESX-Master]shutdown
[HP-ESX-Master]quit
[HP-ESX-Master]interface Ten-GigabitEthernet 1/1/2
[HP-ESX-Master]shutdown
[HP-ESX-Master]quit
[HP-ESX-Slave]interface Ten-GigabitEthernet 2/1/1
[HP-ESX-Slave]shutdown
[HP-ESX-Slave]quit
[HP-ESX-Slave]interface Ten-GigabitEthernet 2/1/2
[HP-ESX-Slave]shutdown
[HP-ESX-Slave]quit
Create logical IRF ports:
[HP-ESX-Master]irf port 1/1
[HP-ESX-Master]port group interface Ten-GigabitEthernet 1/1/1
[HP-ESX-Master]irf port 1/2
[HP-ESX-Master]port group interface Ten-GigabitEthernet 2/1/1
[HP-ESX-Slave]irf port 2/1
[HP-ESX-Slave]port group interface Ten-GigabitEthernet 1/1/2
[HP-ESX-Slave]irf port 2/2
[HP-ESX-Slave]port group interface Ten-GigabitEthernet 2/1/2
Activate IRF:
[HP-ESX-Master]irf-port-configuration active
Before you reboot now, be sure you have no network cable connected between the master and slave switch, otherwise you get a loop!
[HP-All]quit
[HP-All]save
[HP-All]reboot
After the reboot you should only reach the master switch with the configured IP address:
[HP-ESX-Master]display irf
Switch Role Priority CPU-Mac Description
+1 Master 32 xxxx-yyyy-zzzz -----
2 Slave 30 xxxx-yyyy-zzzz -----
--------------------------------------------------
indicates the device is the master.
+ indicates the device through which the user logs in.
The Bridge MAC of the IRF is: xxxx-yyyy-zzzz
Auto upgrade : yes
Mac persistent : always
Domain ID : 0
[HP-ESX-Master]display irf topology
Topology Info
-------------------------------------------------------------------------
IRF-Port1 IRF-Port2
Switch Link neighbor Link neighbor Belong To
1 UP 2 UP 2 xxxx-yyyy-zzzz
2 UP 1 UP 1 xxxx-yyyy-zzzz
[HP-ESX-Master]display irf configuration
MemberID NewID IRF-Port1 IRF-Port2
1 1 Ten-GigabitEthernet1/1/1 Ten-GigabitEthernet1/1/2
2 2 Ten-GigabitEthernet2/1/1 Ten-GigabitEthernet2/1/2
Configuration of MAD (Multiple Access Detection)
Our switches do not support MAD BFD. For MAD LACP we would need another switch, with MAD LACP extensions. We do not have one.
We use the third possible method, MAD ARP with a direct network cable connection between master and slave switch.
For MAD ARP you need to change the MAC address persistence:
[HP-ESX-Master]undo irf mac-address persistent
Now we create a VLAN and activate MAD ARP for this link:
[HP-ESX-Master]vlan 7
[HP-ESX-Master-vlan7]description ARP-MAD-VLAN
[HP-ESX-Master-vlan7]port GigabitEthernet 1/0/48
[HP-ESX-Master-vlan7]port GigabitEthernet 2/0/48
[HP-ESX-Master-vlan7]quit
[HP-ESX-Master]interface vlan-interface 7
[HP-ESX-Master]ip address 192.168.168.168 24
[HP-ESX-Master]mad arp enable
[HP-ESX-Master]save
Now the switches can be connected with a network cable on port 48.
Configuration of an aggregation port for connection to a Linux server
Example for the connection to a Linux server, we use LACP (dynamic link aggregation):
[HP-ESX-Master]vlan 9
[HP-ESX-Master-vlan9]description ESX-DMZ
[HP-ESX-Master-vlan9]port GigabitEthernet 1/0/17
[HP-ESX-Master-vlan9]port GigabitEthernet 2/0/17
[HP-ESX-Master-vlan9]quit
[HP-ESX-Master]interface bridge-aggregation 17
[HP-ESX-Master-Bridge-Aggregation17]description fw-dmz
[HP-ESX-Master-Bridge-Aggregation17]link-aggregation mode dynamic
[HP-ESX-Master-Bridge-Aggregation17]quit
[HP-ESX-Master]interface GigabitEthernet 1/0/17
[HP-ESX-Master-GigabitEthernet1/0/17]port link-aggregation group 17
[HP-ESX-Master-GigabitEthernet1/0/17]quit
[HP-ESX-Master]interface GigabitEthernet 2/0/17
[HP-ESX-Master-GigabitEthernet2/0/17]port link-aggregation group 17
[HP-ESX-Master-GigabitEthernet2/0/17]quit
[HP-ESX-Master]interface bridge-aggregation 17
[HP-ESX-Master-Bridge-Aggregation17]port access vlan 9
[HP-ESX-Master-Bridge-Aggregation17]quit
Now the Linux system can be connected with two network cables to Port 17 on both switches.
Then ethernet bonding on Debian is configured via /etc/network/interfaces:
auto bond0
iface bond0 inet static
address 192.168.1.1
netmask 255.255.255.0
bond-slaves eth0 eth4
bond-mode 802.3ad
bond-miimon 100
xmit_hash_policy layer2+3
Execute following:
ifup bond0
On the switch it look like:
[HP-ESX-Master]display link-aggregation summary
Aggregation Interface Type:
BAGG -- Bridge-Aggregation, RAGG -- Route-Aggregation
Aggregation Mode: S -- Static, D -- Dynamic
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Actor System ID: 0x8000, xxxx-yyyy-zzzz
AGG AGG Partner ID Select Unselect Share
Interface Mode Ports Ports Type
-------------------------------------------------------------------------------
BAGG17 D 0xffff, xxxx-yyyy-zzzz 2 0 Shar
..
Configuration of an aggregation port for an ESX-Host
Example for the connection of an ESXi host to the DMZ VLAN, we use static link aggregation without LACP.
The teaming mode in ESXi must be configured, before the switches are configured.
[HP-ESX-Master]vlan 9
[HP-ESX-Master-vlan9]port GigabitEthernet 1/0/18
[HP-ESX-Master-vlan9]port GigabitEthernet 2/0/18
[HP-ESX-Master-vlan9]quit
[HP-ESX-Master]interface bridge-aggregation 18
[HP-ESX-Master-Bridge-Aggregation18]description esx1-dmz
[HP-ESX-Master-Bridge-Aggregation18]quit
[HP-ESX-Master]interface GigabitEthernet 1/0/18
[HP-ESX-Master-GigabitEthernet1/0/18]port link-aggregation group 18
[HP-ESX-Master-GigabitEthernet1/0/18]quit
[HP-ESX-Master]interface GigabitEthernet 2/0/18
[HP-ESX-Master-GigabitEthernet2/0/18]port link-aggregation group 18
[HP-ESX-Master-GigabitEthernet2/0/18]quit
[HP-ESX-Master]interface Bridge-Aggregation 18
[HP-ESX-Master-Bridge-Aggregation18]port access vlan 9
[HP-ESX-Master-Bridge-Aggregation18]quit
After that you can connect the ESXi host with two network cables to both switches on Port 18.
The link aggregation summary on the switch then looks like:
[HP-ESX-Master]display link-aggregation summary
Aggregation Interface Type:
BAGG -- Bridge-Aggregation, RAGG -- Route-Aggregation
Aggregation Mode: S -- Static, D -- Dynamic
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Actor System ID: 0x8000, xxxx-yyyy-zzzz
AGG AGG Partner ID Select Unselect Share
Interface Mode Ports Ports Type
-------------------------------------------------------------------------------
BAGG18 S none 2 0 Shar
..
To use the full performance of 2 GBit/s with VMotion, you should take a look at
Vmware KB 2007467.
This does only works since ESXi 5.5, when using ESXi 5.1 there is a bug, which does not allow any VMotion, while one switch is in maintenance or out of order.
Furthermore you should ignore the warnings about using IP hash. IP hash is required to use with HP IRF technology.
After the configuration you should verify that everything works, by disconnecting power of one switch while doing some machine migration.
For the measurement of the performance we used SNMP with PRTG, Cacti didn't worked well.
