VMware vSphere DVS LACP timers

I have a customer who was planning a migration from Nexus 1000V (N1K) to VMware Distributed Virtual Switch (aka DVS). I assist their network team in testing DVS functionality and all was nice and shiny. However, they had few detailed LACP related questions because they would like to use LACP against Cisco vPC. I would like to highlight two questions for which I did not find any info in official documentation.

Q1: VMware's LACP is sending LACPDUs every 30 seconds. Is there any way how to configure LACPDU frequency to 1 second?

A1: Short answer is yes. It is possible to reconfigure LACPDU from 30 seconds (normal) to 1 second (fast).

Long answer ... Link Aggregation Control Protocol (LACP) allows the exchange of information with regard to the link aggregation between the two members of said aggregation. This information will be packetized in Link Aggregation Control Protocol Data Units (LACDUs). For further detail about LACP timers see blog post "LACP timer and what it means". In short, LACP timers can be set to "rate fast" - 1 second, or "rate normal" - 30 seconds.

The default value for VMware DVS LACP rate is 30 seconds (normal).  There is esxcli command to configure shorter LACP timer (1 second). See the command help bellow.

esxcli network vswitch dvs vmware lacp timeout set
It allows set advanced timeout settings for LACP
Description:
set Set long/short timeout for vmnics in one LACP LAG
Cmd options:
-l|--lag-id= The ID of LAG to be configured. (required)
-n|--nic-name= The nic name. If it is set, then only this vmnic in the lag will be configured.
-t|--timeout Set long or short timeout: 1 for short timeout and 0 for a long timeout. (required)
-s|--vds= The name of VDS. (required)

Q2: Do we have a possibility to display LACP settings of established LACP session in particular ESXi host? Something like "show lacp" on Cisco switch?

A2: Yes. There is esxcli command ... esxcli network vswitch dvs vmware lacp status get ... which is equivalent to "show lacp" on Cisco physical switch.


UPDATE 2019-03-26: I have been informed that LACP timers setting do not persist ESXi reboot. I'm trying to get more information internally within VMware about such unexpected behavior.  

Back to the basics - VMware vSphere networking

As a software-defined networking (VMware NSX) is getting more and more traction I have been recently often asked to explain the basics of VMware vSphere networking to networking experts who do not have experience with VMware vSphere platform. First of all, networking team should familiarize them self with vSphere platform at least from a high level. Following two videos can help them to understand what vSphere platform is.

vSphere Overview Video
https://youtu.be/EvXn2QiL3gs

What is vCenter (Watch the first two minutes)
https://youtu.be/J0pQ2dKFLbg

When they understand basic vSphere terms like vCenter and  ESXi we can start talking about virtual networking.

First thing first, VMware vSwitch is not a switch. Let me repeat it again ...

VMware vSwitch is not a typical ethernet switch.

It is not a typical network (ethernet) switch because not all switch ports are equal. In VMware vSwitch you have to configure switch uplinks (physical NICs) and internal switch ports (software constructs). If the ethernet frame is coming from the physical network via uplink, vSwitch will never forward such frame to any other uplink but only to internal switch ports, where virtual machines are connected. This behavior guarantees that vSwitch will never cause the L2 loop problem.  It also means that vSwitch does not need to implement and participate in spanning tree protocol (STP) usually running in your physical network. Another different vSwitch behavior compared to traditional ethernet switch is that vSwitch does not learn external MAC addresses. It only knows about MAC addresses of virtual machines running on particular ESXi host (hypervisor). Such devices are often called port extenders. For example, CISCO FEX (fabric extender) is a physical device having the same behavior.

Now let's talk about network redundancy. In production environments, we usually have a redundant network where multiple NICs are connected to different physical switches.

Each NIC connected to different physical switch

vSwitch network redundancy is achieved by NIC teaming. NIC teaming is also known as link aggregation, link bundling, port channeling, ethernet bonding or NIC teaming. VMware is using the term Network teaming or NIC teaming. So what teaming options do we have in VMware vSphere platform? It depends on what edition (license) you have and what vSwitch you want to use. VMware offers two types of vSwitches.

• VMware vSphere standard switch (aka vSwitch or vSS)
• VMware vSphere distributed virtual switch (aka dvSwitch or vDS)

Let's start with VMware's standard switch available on all editions.

VMware vSphere standard switch (vSS)

VMware vSphere standard switch supports multiple switch independent active/active and active/standby teaming methods and also one switch dependent active/active teaming method.

The standard switch can use following switch independent load balancing algorithms:

• Route based on originating virtual port - (default) switch independent active/active teaming where the traffic is load balanced in round-robin fashion across all active network adapters (NICs) based on internal vSwitch port id where virtual machine vNIC's or ESXi vmKernel ports are connected.
• Route based on source MAC hash - switch independent active/active teaming where the traffic is load balanced in round-robin fashion across all active network adapters (NICs) based on source MAC address identified in standard vSwitch.
• Use explicit failover order - is another switch independent teaming but active/passive. Only one adapter from all active adapters is used and if it fails the next one is used. In other words, it always uses the highest order uplink from the list of Active adapters which passes failover detection criteria.

and only one switch dependent load balancing algorithm

• Route based on IP hash - switch dependent active/active teaming where the traffic is load balanced based on a hash of the source and destination IP addresses of each packet. For non-IP packets, whatever is at those offsets is used to compute the hash. This is switch dependent teaming, therefore, the static port-channel (aka ether-channel) has to be configured on the physical switch side otherwise, it will not work.

It is worth to mention that for all active/active teaming methods you can add additional standby adapters which are used just in case the active adapter fails and you can also define unused adapters which you do not want to use at all. For further information, you can read VMware vSphere documentation.

VMware vSphere distributed switch (vDS)

If you have vSphere Enterprise Plus license or VSAN license you are eligible to use VMware vSphere distributed switch. VMware distributed switch key advantages are

• centralized management
• advanced enterprise functionality

When you use virtual distrubuted switch, you do not need to configure each vSwitch individually but instead, you have single distributed vSwitch across multiple ESXi hosts and you can manage it centrally. On top of centralized management you will get following advanced enterprise functionalities:

• NIOC (Network I/O Control) which allows QoS and marking (802.1p tagging, DSCP)
• PVLAN
• LBT (Load Based Teaming) - official name is "Route based on physical NIC load"
• LACP - dynamic switch dependent teaming
• Route based on physical NIC load - another switch independent teaming with optimized load balancing
• ACLs - Access Control Lists
• LLDP
• Port mirroring
• NetFlow
• Configuration backup and restore
• and more

So in vDS you have two additional teaming options

1. Switch Independent - LBT
2. Switch Dependent - LACP

LBT is proprietary VMware solution where vmkernel checks every 30sec the stats from the relevant pNIC's and the calculation use the stats for the 30sec interval and get an average over the 30sec (to normalize the average and to eliminate spikes). If the bandwidth is above 75%, it will mark the pNIC as saturated. If a pNIC have been marked as saturated, the vmkernel will not move any more traffic onto the saturated pNIC. Read blog post "LBT (Load Based Teaming) explained" for more details (see links to references below this article).

LACP is standardized link aggregation protocol. It is worth to mention, that when LACP is used you can leverage significantly enhanced load balancing algorithms to more optimal bandwidth usage of physical NICs. Theoretically, you can use for single VM more bandwidth than is the bandwidth of single pNIC. But to see it in the real world, multiple flows must be initiated from that single VM and LACP the result of the hash algorithm must route it across multiple links bundled in a LAG.

vSphere 6.0 LACP supports following twenty (20) hash algorithms:

1. Destination IP address
2. Destination IP address and TCP/UDP port
3. Destination IP address and VLAN
4. Destination IP address, TCP/UDP port and VLAN
5. Destination MAC address
6. Destination TCP/UDP port
7. Source IP address
8. Source IP address and TCP/UDP port
9. Source IP address and VLAN
10. Source IP address, TCP/UDP port and VLAN
11. Source MAC address
12. Source TCP/UDP port
13. Source and destination IP address
14. Source and destination IP address and TCP/UDP port
15. Source and destination IP address and VLAN
16. Source and destination IP address, TCP/UDP port and VLAN
17. Source and destination MAC address
18. Source and destination TCP/UDP port
19. Source port ID
20. VLAN

Note: Advanced LACP settings are available via esxcli commands.  
esxcli network vswitch dvs vmware lacp
esxcli network vswitch dvs vmware lacp config get
esxcli network vswitch dvs vmware lacp status get
esxcli network vswitch dvs vmware lacp timeout set

Hope this was informative and useful.

References to other useful resources

• VMware : vSphere Networking (official documentation)
• James Green : LAG vs. LBT for vSwitch Uplinks in vSphere
• Hugo Strydom : LBT (Load Based Teaming) explained - Part 1, Part 2,  Part 3
• Chris Wahl : Exploring Enhanced LACP Support with vSphere 5.5 
• Steven Kang : ADVANCED LACP CONFIGURATION USING ESXCLI

DELL Force10 : Virtual Routing and Forwarding (VRF)

VRF Overview

Virtual Routing and Forwarding (VRF) allows a physical router to partition itself into multiple Virtual Routers (VRs). The control and data plane are isolated in each VR so that traffic does NOT flow across VRs. Virtual Routing and Forwarding (VRF) allows multiple instances of a routing table to co-exist within the same router at the same time.

DELL OS 9.7 supports up 64 VRF instances. Number of instances can be increased in future versions therefore check current documentation for authoritative number of instances.

VRF Use Cases

VRF improves functionality by allowing network paths to be segmented without using multiple devices. Using VRF also increases network security and can eliminate the need for encryption and authentication due to traffic segmentation. 

Internet service providers (ISPs) often take advantage of VRF to create separate virtual private networks (VPNs) for customers; VRF is also referred to as VPN routing and forwarding.

VRF acts like a logical router; while a physical router may include many routing tables, a VRF instance uses only a single routing table. VRF uses a forwarding table that designates the next hop for each data packet, a list of devices that may be called upon to forward the packet, and a set of rules and routing protocols that govern how the packet is forwarded. These VRF forwarding tables prevent traffic from being forwarded outside a specific VRF path and also keep out traffic that should remain outside the VRF path.

VRF uses interfaces to distinguish routes for different VRF instances. Interfaces in a VRF can be either physical (Ethernet port or port channel) or logical (VLANs). You can configure identical or overlapping IP subnets on different interfaces if each interface belongs to a different VRF instance.

VRF Configuration

First of all you have to enable VRF feature.

conf
feature vrf

Next step is to create additional VRF instance

ip vrf  tenant-1 

vrf-id is assigned automatically however if you want to configure vrf-id explicitly you can by additional parameter. In example below we use vrf-id 1

ip vrf  tenant-1 1

We are almost done. The last step is interface assignment in to particular VRF. You can assign following interfaces

• Physical Ethernet interfaces (in L3 mode)
• Port-Channel interfaces (static and dynamic/lacp)
• VLAN interfaces
• Loopback interfaces 

Below is example how to assign LAN 100 in to VRF instance tenant-1.

interface vlan 100
  ip vrf forwarding tenant-1

Configuration is pretty easy, right?

Working in particular VRF instance

When you want to configure, show or troubleshoot in particular VRF instance you have to explicitly specify in what VRF you want to be.

So for example when you want to do ping from tenant-01 VRF instance you have to use following command

ping vrf tenant-01 192.168.1.1

Conclusion

VRF is great technology for L3 multi-tenancy. DELL Network Operating System 9 supports VRF therefore you can design interesting network solutions.

VMware vShiled Manager - VXLAN limit

We all know that all technologies has some limits. Only important thing is to know about particular limits limiting your solution.

Do you know VMware vShield Manager has limit for number of virtual networks?

There is the limit 5,000 networks even you use VXLAN network virtualization. So even VXLAN can have theoretically up to 16M segments (24-bit segment ID) you are effectively limited to 5,000 which is not significantly more then old VLAN ID limit of 4,096 (12-bit segment ID).

The most strange thing is that this limit is not documented on vSphere Configuration Maximums. There are documented only following network limits:

• Static/Dynamic port groups per distributed switch = 6,500
• Ports per distributed switch = 60,000
• vCloud Director "Number of networks" = 10,000

Thanks Tom Fojta for this information and link to VMware KB 2042799.

On top of that the current VMware VXLAN implementation provide VXLAN based network overlay only in single vCenter domain so it will not help you with network tunnel for DR (VMware SRM) use case where two vCenters are required.   

So only two benefits of current VMware VXLAN implementation I see today are:

• software defined network segments in single vCenter domain allowing automation of VXLAN provisioning. Nice blog about it is here.
• split between physical networking segments (system VLANs, internet VLANs, MPLS VLANs, ...) and multi-tenant virtual network segments used for tenant's private use.

To be honest even those two benefits are very useful and limits will increase year by year as technology evolves and matures. That's usual technology behavior.

Introduction to the Dell S6000 Layer 2 Gateway for VMware NSX

On following video you can see DELL Force10 S6000 integration with VMware NSX. That's beginning of real and usable software defined networking (SDN) or network virtualization if you wish.

 I'm looking forward for hands-on experience in the future.

CISCO Nexus 1000v - Quality Of Service configuration

class-map type queuing match-any n1kv_control_packet_mgmt_class
 match protocol n1k_control
 match protocol n1k_packet
 match protocol n1k_mgmt

class-map type queuing match-all vmotion_class
 match protocol vmw_vmotion

class-map type queuing match-all vmw_mgmt_class
 match protocol vmw_mgmt

class-map type queuing match-any vm_production
 match cos 0

policy-map type queuing uplink_queue_policy
 class type queuing n1kv_control_packet_mgmt_class
   bandwidth percent 10
 class type queuing vmotion_class
   bandwidth percent 30
 class type queuing vmw_mgmt_class
   bandwidth percent 10
 class type queuing vm_production
   bandwidth percent 40

port-profile type ethernet uplink
 service-policy type queuing output uplink_queue_policy

Software Defined Networking - SDN

SDN is another big topic in modern virtualized datacenter so it is worth to understand what it is and how it can help us to solve real datacenter challenges.

Brad Hedlund's explanation "What is Network Virtualization"
http://bradhedlund.com/2013/05/28/what-is-network-virtualization/
Bred Hedlund is very well known netwoking expert. Now he works for VMware | Nicira participating on VMware NSX product which should be next network virtualisation platform (aka network hypervisor). He is ex-CISCO and ex-DELL | Force10 so there is big probability he fully understand what is going on.

It is obvious that "dynamic service insertion" is the most important thing in SDN. OpenFlow and CISCO vPath is trying to do it but each in different way. Same goal but with different approach. What is better? Who knows? The future and real experience will show us what is better. Jason Edelman's blog post very nicely and clearly compares both approaches.
http://www.jedelman.com/1/post/2013/04/openflow-vpath-and-sdn.html

CISCO as long term networking leader and pioneer has of course its own vision of SDN. Nexus 1000V and Virtual Network Overlays Play for CISCO Pivotal Role in Software Defined Networks. Very nice explanation of CISCO approach is available at
http://blogs.cisco.com/datacenter/nexus-1000v-and-virtual-network-overlays-play-pivotal-role-in-software-defined-networks/

Network Overlays vs. Network Virtualization

Scott Lowe published very nice blog post (philosophy reflection) about "Network Overlays vs. Network Virtualization".

And this was my comment to his post ..

Scott, excellent write-up. As always. First of all I absolutely agree that good definitions, terminology, and conceptual view of particular layer is fundamental to fully understand any technology or system. Modern hardware infrastructure is complex and complexity is growing year on year.
Software programming has the same history. Who programs in assembler nowadays? Why we use object oriented programming more then 20 years? The answer is ... to avoid complexity and have control on system behavior. In software MVC model is often use and it stands for Model-View-Controller. Model is logical representation of something we want to run in software, View is simplified model presentation to end user and controller is engine behind the scene. The same concept apply to SDI (Software Defined Infrastructure) where SDN (Software Defined Network) is another example of the same story.
VMware did excellent job with infrastructure abstraction. Everything in VMware vSphere is object. Better to say managed object which has some properties and methods. So it is the model. vSphere Client or Web Client or vCLI or PowerCLI are different user interfaces into the system. So it is View. And who is Controller? Controller is vCenter because it orchestrates system behavior. vCenter controller includes prepackaged behavior (out-of-the-box) but it can be extended by custom scripts and orchestrated externally for example by vCenter Orchestrator. That's what I really love VMware vSphere. And it is from the begining architected to purely represent hardware infrastructure in software constructs.
Now back to Network Virtualization. In my opinion Network Overlay (for example VXLAN) is mandatory  component to abstract L2 from physical switches and have it in software. Particular Network overlay protocol must be implemented in "Network Hypervisor" which is software L2 switch. But "Network Hypervisor" has to implement also other protocols and components to be classified as "Network Virtualization" and not only as just another software vSwitch.
What Scott already mentioned in his post is that networking is not just L2 but also L3-7 network services so all network services must be available to speak about full "Network Virtualization". Am I correct Scott? And I feel the open question in this post ... who is the controller of "Network Virtualization"? :-)