VMware HA Error During VLT Failure

I have received following message in to my mailbox ...

Hi.
I have a customer that has been testing Force10 VLT with peer routing and VMWare and has encountered the warning message on all hosts during failover of the switches (S4810’s) only when the primary VLT node is failed
“vSphere HA Agent, on this host couldn’t not reach isolation address 10.100.0.1”
Does this impact HA at all?  Is there a solution?
Thanks
Paul 

Force10 is the legacy product name of DELL S-series datacenter networking. Force10 S4810's are datacenter L3 switches. If you don't know what Force10 VLT is look here. Generally it is something like CISCO virtual Port Channel (vPC), Juniper MC-LAG, Arista MLAG, etc.

I think my answer can be valuable for broader networking and virtualization community so here it is ...

First of all let’s make some assumptions:

• Force10 VLT is used for multi chassis LAG capability
• Force10 VLT peer routing is enabled in VLT domain to achieve L3 routing redundancy
• 10.100.0.1 is IP address of VLAN interface on Force10 S4810 (primary VLT node) and this particular VLAN is used for vSphere management.
• 10.100.0.2 is IP address on Force10 S4810 - secondary VLT node.
• vSphere 5.x and above is used.

Root cause with explanation:

When primary Force10 VLT node is down then ping to 10.100.0.1 doesn’t work because peer-routing is ARP proxy on L2. Secondary node will route L2 traffic on behalf of primary node but 10.100.0.1 doesn’t answer on L3 therefore ICMP doesn’t work.

VMware (vSphere 5 and above) HA Cluster use network and storage heartbeat mechanism. Network mechanism use two probe algorithms listed below. 

1. ESXi hosts in the cluster are sending heartbeat beacon to each other. This should work ok during primary VLT node failure.
2. ESXi hosts are also pinging HA isolation addresses (Default HA isolation address is default gateway therefore 10.100.0.1 in your particular case). This doesn’t work  during primary VLT node failure.

That’s the reason VMware HA Cluster will log about this situation.

Is there any impact?

There is no impact on HA Cluster because

•  It is just informative message because algorithm (1) works correctly and there is still network visibility  among ESXi hosts in the cluster.
• From vSphere 5 and above there is also storage heartbeat mechanism which can eliminate network invisibility among ESXi host in the cluster.

Are there any potential improvements?

Yes they are. You can configure multiple HA Isolation Addresses to mitigate default gateway unavailability. In your particular case I would recommend to use two IP addresses (10.100.0.1 and 10.100.0.2) because at least one VLT node will be always available.

For more information how to configure multiple HA isolation addresses look at http://kb.vmware.com/kb/1002117

DELL Force10 VLT and vSphere Networking

DELL Force10 VLT is multi chassis LAG technology. I wrote several blog posts about VLT so for VLT introduction look at http://blog.igics.com/2014/05/dell-force10-vlt-virtual-link-trunking.html. All Force10 related posts are listed here.  By the way DELL Force10 S-Series switches has been renamed to DELL S-Series switches with DNOS 9 (DNOS stands for DELL Network Operating System) however I’ll keep using Force10 and FTOS in my series to keep it uniform. 

In this blog post I would like to discuss Force10 VLT specific failure scenario when VLTi fails.

VLT Domain is actually cluster of two VLT nodes (peers). One node is configured as primary and second node as secondary. VLTi is a peer link between two VLT nodes. The main role of VLTi peer link is to synchronize MAC addresses interface assignments which is used for optimal traffic in VLT port-channels. In other words if everything is up and running data traffic over VLT port-channels (virtual LAGs)  is optimize and optimal link will be chosen to eliminate inter VLTi traffic. VLTi is used for data traffic only in case of some VLT link failure in one node and another VLT link still available on another node.

Now you can ask what happen in case of VLTi failure. In this situation backup link will kick in and act as a backup communication link for VLT Domain cluster. This situation is called Split-Brain scenario and exact behavior is nicely described in VLT Reference guide.

The backup heartbeat messages are exchanged between the VLT peers through the backup links of the OOB Management network. When the VLTI link (port-channel) fails, the MAC/ARP entries cannot be synchronized between the VLT peers through the failed VLTI link, hence the Secondary VLT Peer shuts the VLT port-channel forcing the traffic from the ToR switches to flow only through the primary VLT peer to avoid traffic black-hole. Similarly the return traffic on layer-3 also reaches the primary VLT node. This is Split-brain scenario and when the VLTI link is restored, the secondary VLT peer waits for the pre-configured time (delay-restore) for the MAC/ARP tables to synchronize before passing the traffic. In case of both VLTi and backup link failure, both the VLT nodes take primary role and continue to pass the traffic, if the system mac is configured on both the VLT peers. However there would not be MAC/ARP synchronization.

With all that being said let’s look at some typical VLT topologies with VMware ESXi host. Force10 S4810 is L3 switch therefore VLT domain can provide switching and routing services. Upstream router is single router for external connectivity. ESXi host has two physical NIC interfaces.

First topology

First topology is with VMware switch independent connectivity. This is very common and favorite ESXi network connectivity because of simplicity for vSphere administrator.

The problem with this topology is when VLTi peer-link has a failure (red cross in the drawing). We already know that in this scenario the backup link will kick in and VLT links from secondary node are intentionally disabled (black cross in the drawing). However our ESXi host is not connected via VLT therefore the server facing port will stay up.  VLT Domain doesn’t know anything about VMware vSwitch topology therefore it must keep port up which implies as a black hole scenario (black circle in the drawing) for virtual machines pinned into VMware vSwitch Uplink 2.

I hear you. You ask what the solution for this problem is.  I think there are two solutions.  First out-of-the-box solution is to use VLT down to the ESXi host which is depicted on second topology later in this post. Second solution could be to leverage UFD (Uplink Failure Detection) and track some VLT ports together with server facing ports. I did not test this scenario but I think it should work and there is big probability I’ll have to test it soon.   

Second topology

Second topology is leveraging VMware LACP. LACP connectivity is obviously more VLT friendly because VLT is established down to the server and downlink to ESXi host is correctly disabled. Virtual machines are not pinned directly to VMware vSwitch uplinks but they are connected through LACP virtual interface. That’s the reason you will not experience black hole scenario for some virtual machines.

Conclusion

Server virtualization is nowadays on every modern datacenter. That’s the reason why virtual networking has to be taken in to account for any datacenter network design. VMware switch independent NIC teaming is simple for vSphere administrator but it can negatively impact network availability in some scenarios. Unfortunately VMware standard virtual switch doesn’t support dynamic port-channel (LACP) but only static port-channel. Static port-channel should work correctly with VLT but LACP is recommended because of LACP keep-alive mechanism.  LACP is available only with VMware distributed virtual switch which requires the highest VMware licenses (vSphere Enteprise Plus edition). VMware’s distributed virtual switch with LACP uplink is the best solution for Force10 VLT.  In case of the budget or technical constraint you have to design an alternative solution leveraging either static port-channel (VMware call it “IP Hash load balancing”) or FTOS UFD (Uplink Failure Detection) to mitigate risk of black hole scenario. 

Update 2015-05-13:
I have just realized that NPAR is actually technical constraint avoiding to use port-channel technology on ESXi host virtual switch. NPAR technology allows switch independent network partitioning of physical NIC ports into more logical NICs. However port-channel cannot be configured on NPAR enabled NICs therefore UFD is probably the only solution to avoid black hole scenario when VLT peer-link fails. 

DELL Force10 : mVLT – Ethernet Loop Free Topology Design

Last week I have received following question from one of my reader …

I came to your blog post http://blog.igics.com/2014/05/dell-force10-vlt-virtual-link-trunking.html and I am really happy that you shared this information with us. However I was wondering if you have tested a scenario with 4 S4810 with VLT configured on 2 x 2 and connected together (somewhere called mLAG). How do you continue to add VLT couples to the setup? I would be really happy if you could provide any info regarding such setup.

So let’s deep dive into VLT port-channel between two Force10 VLT Domains also known as mVLT. Please note that VLT can be configured not only between two Force10 VLT domains but also between Force10 VLT domain and other multi chassis port-channel technology like for instance CISCO virtual Port Channel (vPC). However, this blog post is focused to single vendor solution mVLT on DELL S-Series Switches (previously known as Force10 S-Series).

If you are not familiar with DELL Force10 VLT technology read my previous blog post where is VLT described in detail. It is really important to understand VLT before you try to understand mVLT (Multi-domain VLT). By the way mVLT is called eVLT (Enhanced VLT) in Force10 documentation so it might be little bit confusing. Anyway mVLT is nothing else then regular virtual port channel (VLT) between  two VLT domains. Therefore mVLT is quite good term if you ask me.

mVLT Logical Design

mVLT logical design is pretty straight forward. It is required to achieve stretched L2 over two datacenters without any loops. This topology is often called loop free topology and it is depicted on figure below from spanning tree (STP) point of view.

However we would like to have hardware and link redundancy therefore multi chassis port-channel technology (Force10 VLT in our particular case) is used to still have simple loop free topology from spanning tree point of view but with switch unit and physical link redundancy. Force10 mVLT solution is logically depicted on figure below.

Please note, that each single VLT Domain act in spanning tree as a single logical switch.

DELL highly recommends using four links between VLT domains because of higher redundancy and optimal data flow. However, sometimes your are constraint with links between sites. Two links DCI is also supported design but not recommended because there is obviously lower link redundancy and therefore higher probability of communication over VLTi which adds hop and therefore latency. Two links mVLT DCI also known as square design is depicted on figure below. 

Even the topology is loop free and from logical view we have just one switch on each datacenter spanning tree protocol should be enabled and configured just in case of human error or VLT domain failure or split. Rapid Spanning Tree (RSTP) protocol is good enough therefore used later in physical configurations.

mVLT Physical Design and Configuration

Physical design below shows connectivity of four (2x two) Force10 S4810 switches leveraging four links for DCI port-channel (mVLT).

Physical design for just two links DCI is depicted on following schema.

And switch configuration snippets for four links mVLT are listed below for completeness. Two link DCI is just variation of similar configurations so you can simply reuse and slightly change four link configuration.

DCA-SWCORE-A – acts as primary Root Bridge in RSTP in case of loop

!

hostname DCA-SWCORE-A

!

protocol spanning-tree rstp

 no disable

 hello-time 1

 max-age 6

 forward-delay 4

 bridge-priority 4096

!

vlt domain 1

 peer-link port-channel 128

 back-up destination 172.16.201.2

 primary-priority 1

 system-mac mac-address 02:00:00:00:00:01

 unit-id 0

 peer-routing

!

 proxy-gateway lldp

  peer-domain-link port-channel 127

!

interface TenGigabitEthernet 0/46

 no ip address

 mtu 12000

! 

 port-channel-protocol LACP

  port-channel 127 mode active

 dampening 10 100 1000 60

 no shutdown

!

interface TenGigabitEthernet 0/47

 no ip address

 mtu 12000

! 

 port-channel-protocol LACP

  port-channel 127 mode active

 dampening 10 100 1000 60

 no shutdown

!

interface fortyGigE 0/56

 no ip address

 mtu 12000

 no shutdown

!

interface fortyGigE 0/60

 no ip address

 mtu 12000

 no shutdown

!

interface ManagementEthernet 0/0

 ip address 172.16.201.1/24

 no shutdown

!

interface Port-channel 127

 description "mVLT - interconnect link"

 no ip address

 mtu 12000

 switchport

 vlt-peer-lag port-channel 127

 no shutdown

!

interface Port-channel 128

 description "VLTi - interconnect link"

 no ip address

 mtu 12000

 channel-member fortyGigE 0/56,60

 no shutdown

!

DCA-SWCORE-B  – acts as secondary Root Bridge in RSTP in case of loop

!

hostname DCA-SWCORE-B

!

protocol spanning-tree rstp

 no disable

 hello-time 1

 max-age 6

 forward-delay 4

 bridge-priority 8192

!

vlt domain 1

 peer-link port-channel 128

 back-up destination 172.16.201.1

 primary-priority 8192

 system-mac mac-address 02:00:00:00:00:01

 unit-id 1

 peer-routing

!

 proxy-gateway lldp

  peer-domain-link port-channel 127

!

interface TenGigabitEthernet 0/46

 no ip address

 mtu 12000

! 

 port-channel-protocol LACP

  port-channel 127 mode active

 dampening 10 100 1000 60

 no shutdown

!

interface TenGigabitEthernet 0/47

 no ip address

 mtu 12000

! 

 port-channel-protocol LACP

  port-channel 127 mode active

 dampening 10 100 1000 60

 no shutdown

!

interface fortyGigE 0/56

 no ip address

 mtu 12000

 no shutdown

!

interface fortyGigE 0/60

 no ip address

 mtu 12000

 no shutdown

!

interface ManagementEthernet 0/0

 ip address 172.16.201.2/24

 no shutdown

!

interface Port-channel 127

 description "mVLT - interconnect link"

 no ip address

 mtu 12000

 switchport

 vlt-peer-lag port-channel 127

 no shutdown

!

interface Port-channel 128

 description "VLTi - interconnect link"

 no ip address

 mtu 12000

 channel-member fortyGigE 0/56,60

 no shutdown

!

DCB-SWCORE-A – acts as tertiary Root Bridge in RSTP in case of loop

!

hostname DCB-SWCORE-A

!

protocol spanning-tree rstp

 no disable

 hello-time 1

 max-age 6

 forward-delay 4

 bridge-priority 12288

!

vlt domain 2

 peer-link port-channel 128

 back-up destination 172.16.202.2

 primary-priority 1

 system-mac mac-address 02:00:00:00:00:02

 unit-id 0

 peer-routing

!

 proxy-gateway lldp

  peer-domain-link port-channel 127

!

interface TenGigabitEthernet 0/46

 no ip address

 mtu 12000

! 

 port-channel-protocol LACP

  port-channel 127 mode active

 dampening 10 100 1000 60

 no shutdown

!

interface TenGigabitEthernet 0/47

 no ip address

 mtu 12000

! 

 port-channel-protocol LACP

  port-channel 127 mode active

 dampening 10 100 1000 60

 no shutdown

!

interface fortyGigE 0/56

 no ip address

 mtu 12000

 no shutdown

!

interface fortyGigE 0/60

 no ip address

 mtu 12000

 no shutdown

!

interface ManagementEthernet 0/0

 ip address 172.16.202.1/24

 no shutdown

!

interface Port-channel 127

 description "mVLT - interconnect link"

 no ip address

 mtu 12000

 switchport

 vlt-peer-lag port-channel 127

 no shutdown

!

interface Port-channel 128

 description "VLTi - interconnect link"

 no ip address

 mtu 12000

 channel-member fortyGigE 0/56,60

 no shutdown

!

DCB-SWCORE-B – acts as quaternary Root Bridge in RSTP in case of loop

!

hostname DCB-SWCORE-B

!

protocol spanning-tree rstp

 no disable

 hello-time 1

 max-age 6

 forward-delay 4

 bridge-priority 16384

!

vlt domain 2

 peer-link port-channel 128

 back-up destination 172.16.202.1

 primary-priority 8192

 system-mac mac-address 02:00:00:00:00:02

 unit-id 1

 peer-routing

!

 proxy-gateway lldp

  peer-domain-link port-channel 127

!

interface TenGigabitEthernet 0/46

 no ip address

 mtu 12000

! 

 port-channel-protocol LACP

  port-channel 127 mode active

 dampening 10 100 1000 60

 no shutdown

!

interface TenGigabitEthernet 0/47

 no ip address

 mtu 12000

! 

 port-channel-protocol LACP

  port-channel 127 mode active

 dampening 10 100 1000 60

 no shutdown

!

interface fortyGigE 0/56

 no ip address

 mtu 12000

 no shutdown

!

interface fortyGigE 0/60

 no ip address

 mtu 12000

 no shutdown

!

interface ManagementEthernet 0/0

 ip address 172.16.202.2/24

 no shutdown

!

interface Port-channel 127

 description "mVLT - interconnect link"

 no ip address

 mtu 12000

 switchport

 vlt-peer-lag port-channel 127

 no shutdown

!

interface Port-channel 128

 description "VLTi - interconnect link"

 no ip address

 mtu 12000

 channel-member fortyGigE 0/56,60

 no shutdown

!

Conclusion

Force10 mVLT is great technology for loop free L2 network topology. It can be leveraged for local loop free topologies inside single datacenter or as L2 extension between datacenters. However our networks are usually built to support IP traffic therefore L3 considerations has to be addressed as well. Just think about default IP gateway behavior and potential DCI potential trombone.  That’s where other VLT features peer-routing and proxy-gateway come in to play and mitigate DCI trombone issue. You can see these technologies configured in VLT configurations above. But that’s another topic for another blog post.

To be absolutely honest I personally don't recommend L2 interconnects between datacenters without any good justification. I strongly recommend L3 datacenter interconnects and when stretched L2 is needed then some network overlay technology can be leveraged. L3 will guarantee independent availability zones and splitting L2 failure domain. But on the other hand such network overlay needs some other bits and pieces which in some cases increase complexity and cost. Therefore mVLT can be seriously considered for cost effective datacenter L2 extensions.  That's a typical "it depends" scenario where these two design decision options has to be compared and final decision clearly justified.   

If you want to know more about these technologies or use cases just ask and we can go deeper or broader. And as always any feedback and/or comment is highly appreciated.

Force10 VLT - Design Verification Test Plan

One my philosophical rule is "Trust, but Verify". Design Verification Test Plan is good approach to be sure how the system you have designed behaves. Typical design verification test plan contains Usability, Performance and Reliability tests.

Force10 VLT domain configuration is actually two node cluster (the system) providing L2/L3 network services. What network services your VLT domain should provides depends on customer requirements. However typical VLT customer requirement is to have high availability and eliminate network down times when some system component fails or is maintained by administrator. Planing and executing reliability tests is good approach to verify that customer's high availability requirements have been achieved.

Bellow are some reliability tests I'm thinking are worth to execute and when my gear will be back in my lab I'll try to find some time and execute tests described below and publish real test results.
 
If you know about some other tests which make sense to perform, please don't be shy, leave the comment and I'll do it for you.

Test #1
Description: 
Simulate VLT Domain secondary node failure impact on Ethernet traffic. How long (in ms) is traffic disrupted?
Tasks:
Use system A and system B both connected via VLT link to VLT Domain  
Ping from system A to system B at least 10x per second
Power Off secondary VLT node
Measure network disruption
Expected Results:
It should be sub second failure.

Test Result:
TBD

Test #2
Description: 
Simulate VLT Domain primary node failure impact on Ethernet traffic. How long (in ms) is traffic disrupted?
Tasks:
Use system A and system B both connected via VLT link to VLT Domain  
Ping from system A to system B at least 10x per second
Power Off primary VLT node
Measure network disruption
Expected Results:
It should be sub second failure.
Test Result:
TBD

Test #3
Description: 
Simulate one link from VLTi (ISL) port-channel failure.
Tasks:
Use system A and system B both connected via VLT link to VLT Domain  
Ping from system A to system B at least 10x per second
Pull out one cable participating in VLTi static port-channel
Measure network disruption
Expected Results:
VLT Domain should be still working without traffic impact.
Test Result:
TBD
Test #4
Description: 
Simulate all links from VLTi (ISL) port-channel failure.
Tasks:
Use system A and system B both connected via VLT link to VLT Domain  
Ping from system A to system B at least 10x per second
Pull out all cables participating in VLTi static port-channel
Measure network disruption
Expected Results:
Backup link should act as arbiter. VLT Domain should be still working  but in split brain mode and only primary VLT node should handle the traffic.
Test Result:
TBD

Test #5
Description: 
Simulate VLT Domain backup link failure. Backup link configured as IP heartbeat over out-of-band management.
Tasks:
Use system A and system B both connected via VLT link to VLT Domain  
Ping from system A to system B at least 10x per second
Pull out cable participating in backup link
Measure network disruption
Expected Results:
All traffic should work correctly but VLT should report backup link failure.
Test Result:
TBD 
Test #6
Description: 
Simulate one link failure on some virtual link trunk (aka VLT or virtual port-channel).
Tasks:
Use system A and system B both connected via VLT link to VLT Domain  
Ping from system A to system B at least 10x per second
Pull out cable participating in VLT
Measure network disruption
Expected Results:
Port channel should survive this failure.
Test Result:
TBD

These six tests should verify basic high availability and resiliency of Force10 VLT cluster. 

All problems should be notified by SNMP and/or syslog to central monitoring system in case it is configured properly. That can move us to Usability Tests .... but that's another set of tests ...

And please remember that TOO MUCH TESTING WOULD NEVER BE ENOUGH :-)