VMware vShiled Manager - VXLAN limit

We all know that all technologies has some limits. Only important thing is to know about particular limits limiting your solution.

Do you know VMware vShield Manager has limit for number of virtual networks?

There is the limit 5,000 networks even you use VXLAN network virtualization. So even VXLAN can have theoretically up to 16M segments (24-bit segment ID) you are effectively limited to 5,000 which is not significantly more then old VLAN ID limit of 4,096 (12-bit segment ID).

The most strange thing is that this limit is not documented on vSphere Configuration Maximums. There are documented only following network limits:

• Static/Dynamic port groups per distributed switch = 6,500
• Ports per distributed switch = 60,000
• vCloud Director "Number of networks" = 10,000

Thanks Tom Fojta for this information and link to VMware KB 2042799.

On top of that the current VMware VXLAN implementation provide VXLAN based network overlay only in single vCenter domain so it will not help you with network tunnel for DR (VMware SRM) use case where two vCenters are required.   

So only two benefits of current VMware VXLAN implementation I see today are:

• software defined network segments in single vCenter domain allowing automation of VXLAN provisioning. Nice blog about it is here.
• split between physical networking segments (system VLANs, internet VLANs, MPLS VLANs, ...) and multi-tenant virtual network segments used for tenant's private use.

To be honest even those two benefits are very useful and limits will increase year by year as technology evolves and matures. That's usual technology behavior.

DELL Force10 configuration for VMware VXLAN transportation

Right now I work on vSphere Design where network virtualization is leveraged to simplify network management and provide segmentation of multiple tenants. Therefore I was tested VXLANs in my lab. I have equipment listed bellow:

• 1x DELL Blade Chassis  M1000e
• 2x DELL Force10 IOA (IO Aggregators - blade chassis network modules)
• 2x DELL Force10 S4810 as top of the rack switches
• 1x DELL Force10 S60 acting as physical router (L3 switch)
• 1x DELL EqualLogic storage PS-4110 (iSCSI storage module inside Blade Chassis)

Here are VXLAN physical switch requirements 

• Minimum MTU size requirement is 1600 however we will use maximum Jumbo Frames across physical network
• IGMP snooping should be enabled on L2 switches, to which VXLAN participating hosts are attached.
• IGMP Querier enabled on router or L3 switch with connectivity to the multicast enabled networks.
• Multicast routing (PIM-SM) must be enabled on routers.

Force10 switches are by default configured to allow Jumbo Frames. However physical interfaces, vlan interfaces and port-channels has to be configured explicitly. 

Force10 S-series switches interfaces MTU can be set up to 12000. In CISCO Nexus environments max MTU is 9216.

Force10 IOA (I/O Aggregator) is by default set to MTU 12000 so it is already prepared for VXLAN and nothing has to be configured.

Let's assume we use VLAN 14 for VXLAN transport.

Router config (Force10 S60)

config
igmp snooping enable
ip multicast-routing
interface vlan 14
  ip pim sparse-mode
  mtu 12000
  tagged gigabitethernet 0/46-47
  exit

! For all interfaces in VLAN 14 we have to set MTU at least 1600

interface range gigabitethernet 0/46 - 47
  mtu 12000
  end

Switch config (Force10 S4810)
! IGMP snooping must be enabled

conf
ip igmp snooping enable
interface vlan 14
  mtu 12000
  exit
interface range tengigabitethernet 0/46 , tengigabitethernet 0/48 - 51 , fortyGigE 0/56 , fortyGigE 0/60
  mtu 12000
  exit
interface range port-channel 1 , port-channel 128
  mtu 12000
  exit
end

IO Aggregator (Force10 IOA)

Force10 IOA default configuration has maximum MTU  already in default factory settings so it is VXLAN ready and no changes are required.

Here are Force10 IOA default values:

mtu 12000
ip mtu 11982
igmp snooping enabled

Check out these excellent blog articles for more details on VXLAN theory and implementation:

VXLAN requirements
http://www.yellow-bricks.com/2012/10/04/vxlan-requirements/

VXLAN on UCS and vSphere: from L3 to Nexus 1000V
http://vmtrooper.com/vxlan-on-ucs-and-vsphere-from-l3-to-nexus-1000v/

http://www.force10networks.com/CSPortal20/TechTips/0008_mtu-settings.aspx
Adjusting MTU and Configuring Jumbo Frame Settings

UPDATE 2015-02-02:
I have multicast router enabled on my VLAN 14 (see configuration of Force10 S60) therefore it works like IGMP querier. However if you have a need to have VXLAN overlay over the network without multicast router you should configure IGMP Querier on particular VLAN otherwise multicast traffic will be flooded into whole broadcast network (VLAN). IGMP querier can be configured by following command:

ip igmp snooping querier 

What type of NIC teaming, loadbalancing and physical switch configuration to use for VMware's VXLAN?

As a former CISCO UCS Architect I'm observing VXLAN initiative almost 2 years so I was looking forward to do the real customer project. Finally it is here. I'm working on vSphere design for vCloud Director (vCD). To be honest I'm responsible just for vSphere design and someone else is doing vCD Design because I'm not vCD expert and I have just conceptual and high-level vCD knowledge. I'm not planning to change it in near future because I'm more focused on next generation infrastructure and vCD is in my opinion just another software for selling IaaS. I'm not saying it is not important. It is actually very important because IaaS is not just technology but business process. However nobody knows everything and I leave some work for other architects :-)

We all know that vCD sits on top of vSphere providing multi-tenancy and other IaaS constructs and since vCD 5.1 the network multi-tenancy segmentation is done by VXLAN network overlay. Therefore I have finally opportunity to plan, design and implement VXLANs for real customer.

Right now I'm designing network part of vSphere architecture and I describe VXLAN oriented design decision point bellow.

VMware VXLAN Information sources:

• S1: VMware  vShiled Administration Guide [Official source]
• S2: VMware KB 2050697 [Official source]
• S3: Duncan Epping blog post here. [Unofficial source]
• S4: VMware VXLAN Deployment Guide available here. [Official source]

I would like to thanks Duncan for his blog post back in October 2012 right before Barcelona VMworld 2012 where VXLANs were officially introduced by VMware. Even it is unofficial information source it is very informative and I'm verifying it against official VMware documentation and white papers.  Unfortunately I have realized that there is a lack of trustful and publicly available technical information till today and some information are contradictory. See bellow what confusion I'm facing and I would be very happy if someone help me to jump out from the circle.

Design decision point:
What type of NIC teaming, loadbalancing and physical switch configuration to use for VMware's VXLAN?

Requirements:

• R1: Fully supported solution
• R2: vSphere 5.1 and vCloud Director 5.1
• R3: VMware vCloud Network & Security (aka vCNS or vShield) with VMware distributed virtual switch
• R4: Network Virtualization and multi-tenant segmentation with VXLAN network overlay 
• R5: Leverage standard access datacenter switches like CISCO Nexus 5000, Force10 S4810, etc.

Constraints:

• C1: LACP 5-tuple hash algorithm is not available on current standard access datacenter physical switches mentioned in requirement R5
• C2: VMware Virtual Port ID loadbalancing is not supported with VXLAN Source: S3
• C3: VMware LBT loadbalancing is not supported with VXLAN Source: S3
• C4: LACP must be used with 5-tuple hash algorithm Source: S3, S2, S1 on Page 48. [THIS IS STRANGE CONSTRAINT, WHY IT IS HASH DEPENDENT?] Updated 2013-09-11: It looks like there is a bug in VMware documentation and KB Article. Thanks @DuncanYB and @fojta for confirmation and internal VMware escalations.

Available Options:

• Option 1: Virtual Port ID
• Option 2: Load based Teaming
• Option 3: LACP
• Option 4: Explicit fail-over

Option comparison:

• Option 1: not supported because of C1
• Option 2: not supported because of C2
• Option 3: supported
• Option 4: supported but not optimal because only one NIC is used for network traffic. 

Design decision and justification:
Based on available information options 3 and 4 complies with requirements and constraints. Option 3 is better because network traffic is load balanced across physical NICs. That's not a case for option 4.

Other alternatives not compliant with all requirements:

• Alt 1: Use physical switches with 5-tuple hash loadbalancing. That means high-end switch models like Nexus 7000, Force10 E Series, etc.
• Alt 2: Use CISCO Nexus 1000V with VXLAN. They support LACP with any hash algorithm. 5-tuple hash is also recommended but not strictly required.

Conclusion:
I hope some information in constraints C2, C3, and C4 are wrong and will be clarified by VMware. I'll tweet this blog post to some VMware experts and hope someone will help me to jump out from the decision circle.
If you have any official/unofficial topic related information or you see anything where I'm wrong, please feel free to speak up in the comments.
Updated 2013-09-11: Constraint C4 doesn't exists and VMware doc will be updated.
Based on updated information LACP and "Explicit fail-over" teaming/load-balancing is supported for VXLANs. LACP is better way to go and  "Explicit fail-over" is alternative in case LACP is not achievable on your environment.

VMware VXLAN Deployment Guide

Vyenkatesh Deshpande recently published "VMware Network Virtualization Design Guide" which can be downloaded here. However deployment guide which is here is very valuable if you really want to implement VXLAN in your environment.