General recommendations for stretched vSphere HA Cluster aka Metro Cluster Storage (vMSC)

This is just a brief blog post with general recommendations for VMware vSphere Metro Cluster Storage (aka vMSC). For more holistic view, please read white paper "VMware vSphere Metro Storage Cluster Recommended Practices"

vSphere HA Cluster Recommended Configuration Settings:

• Set Admission Control - Failover capacity by defining percentage of the cluster (50% for CPU and Memory)
• Set Host Isolation Response - Power Off and Restart VMs
• Specify multiple host isolation addresses - Advanced configuration option das.isolationaddressX
• Disable default gateway as host isolation address - Advanced configuration option das.useDefaultIsolationAddress=false
• Change the default settings of vSphere HA and configure it to Respect VM to Host affinity rules during failover - Advanced configuration option das.respectVmHostSoftAffinityRules=true
• The minimum number of heartbeat datastores is two and the maximum is five. VMware recommends increasing the number of heartbeat datastores from two to four in a stretched cluster environment Advanced configuration option das.heartbeatDsPerHost=4
• VMware recommends using "Select any of the cluster datastores taking into account my preferences" for heartbeat datastores and choose two datastores (active distributed volumes/LUNs) on each site. 
• PDL and APD considerations depends on stretched cluster mode (uniform/non-uniform). However, VMware recommends to configure PDL/APD responses therefore VM Component Protection (VMCP) must be enabled and response should be set to "Power Off and Restart VMs - Conservative". Detail configuration should be discussed with particular storage vendor. 

vSphere DRS Recommended Configuration Settings:

• DRS mode - Fully automated
• Use DRS VM/Host rules to set VM per site locality
• Use DRS "Should Rules" and avoid the use of "Must Rules"

SIOC/SDRS

• Based on KB 2042596 SIOC is not supported
• Based on KB 2042596 SDRS is only supported when the IO Metric function is disabled.

Distributed (stretched) Storage Recommendations:

• Always consult your configuration with your storage vendor
• VMware highly recommends to use storage witness (aka arbitrator, tie-braker, etc.) in third site.

Custom automation for compliance check and / or operational procedures Recommendations:

• VMware recommends manually defining “sites” by creating a group of hosts that belong to a site and then adding VMs to these sites based on the affinity of the datastore on which they are provisioned. 
• VMware recommends automating the process of defining site affinity by using tools such as VMware vCenter OrchestratorTM or VMware vSphere PowerCLITM. 
• If automating the process is not an option, use of a generic naming convention is recommended to simplify the creation of these groups. 
• VMware recommends that these groups be validated on a regular basis to ensure that all VMs belong to the group with the correct site affinity.

Other relevant references:

1. VMware vSphere Metro Storage Cluster Recommended Practices
2. Setting Multiple Isolation Response Addresses for VMware High Availability (1002117)
3. vSphere HA respecting VM-Host should rules?
4. Using EMC VMAX Storage in VMware vSphere Environments

VMware HA Error During VLT Failure

I have received following message in to my mailbox ...

Hi.
I have a customer that has been testing Force10 VLT with peer routing and VMWare and has encountered the warning message on all hosts during failover of the switches (S4810’s) only when the primary VLT node is failed
“vSphere HA Agent, on this host couldn’t not reach isolation address 10.100.0.1”
Does this impact HA at all?  Is there a solution?
Thanks
Paul 

Force10 is the legacy product name of DELL S-series datacenter networking. Force10 S4810's are datacenter L3 switches. If you don't know what Force10 VLT is look here. Generally it is something like CISCO virtual Port Channel (vPC), Juniper MC-LAG, Arista MLAG, etc.

I think my answer can be valuable for broader networking and virtualization community so here it is ...

First of all let’s make some assumptions:

• Force10 VLT is used for multi chassis LAG capability
• Force10 VLT peer routing is enabled in VLT domain to achieve L3 routing redundancy
• 10.100.0.1 is IP address of VLAN interface on Force10 S4810 (primary VLT node) and this particular VLAN is used for vSphere management.
• 10.100.0.2 is IP address on Force10 S4810 - secondary VLT node.
• vSphere 5.x and above is used.

Root cause with explanation:

When primary Force10 VLT node is down then ping to 10.100.0.1 doesn’t work because peer-routing is ARP proxy on L2. Secondary node will route L2 traffic on behalf of primary node but 10.100.0.1 doesn’t answer on L3 therefore ICMP doesn’t work.

VMware (vSphere 5 and above) HA Cluster use network and storage heartbeat mechanism. Network mechanism use two probe algorithms listed below. 

1. ESXi hosts in the cluster are sending heartbeat beacon to each other. This should work ok during primary VLT node failure.
2. ESXi hosts are also pinging HA isolation addresses (Default HA isolation address is default gateway therefore 10.100.0.1 in your particular case). This doesn’t work  during primary VLT node failure.

That’s the reason VMware HA Cluster will log about this situation.

Is there any impact?

There is no impact on HA Cluster because

•  It is just informative message because algorithm (1) works correctly and there is still network visibility  among ESXi hosts in the cluster.
• From vSphere 5 and above there is also storage heartbeat mechanism which can eliminate network invisibility among ESXi host in the cluster.

Are there any potential improvements?

Yes they are. You can configure multiple HA Isolation Addresses to mitigate default gateway unavailability. In your particular case I would recommend to use two IP addresses (10.100.0.1 and 10.100.0.2) because at least one VLT node will be always available.

For more information how to configure multiple HA isolation addresses look at http://kb.vmware.com/kb/1002117

vSphere HA Cluster Redundancy

All vSphere administrators and implementers know how easily vSphere HA Cluster can be configured. However sometimes quick and simple configuration doesn't do exactly what is expected. You can, and typically you should, enable Admission Control in vSphere HA Cluster configuration settings. VMware vSphere HA Admission Control is control mechanism checking if another VM can be powered on in HA enabled cluster and still satisfy redundancy requirement. So far so good however complexity starts from here because you have several options what algorithm you will use to fulfill your spare capacity redundancy requirement. So what options do you have?

Admission Control can be configured for following three algorithms:

1. Define fail-over capacity by static number of hosts
2. Define fail-over capacity by reserving a percentage of cluster resources
3. Use dedicated fail-over hosts

Let's deep dive into each option ...

Algorithm 1 is generally N+X host redundancy 

When N+X redundancy is required most vSphere designers go with this option because it looks like most suitable choice. However, it is important to know that this particular algorithm is working with HA Slot Size. HA Slot Size is calculated based on defined reservations on powered VMs. If you don't use CPU/MEM reservations per VM than default reservation values (32 MHz, memory virtualization overhead)  are used for HA Slot Size calculation. By the way, VMware recommends to set  reservations per resource pools and not per VM so there is relatively high probability you don't have VM reservations and you will have very low HA Slot Size which means that Admission Control will allow to power on lot of VMs which introduce high resource over-allocation and your N+1 redundancy can significantly suffer. On the other hand, if you have just one VM with huge CPU/MEM reservations it can significantly impact and skew HA Slot Size with a negative impact on your VM consolidation ratio.  

How can we solve this problem? One solution is HA Cluster Advanced Options described below.

Maximum HA Slot size can be limited to two following advanced options.

• das.slotcpuinmhz - Defines the maximum bound on the CPU slot size. If this option is used, the slot size is the smaller of this value or the maximum CPU reservation of any powered-on virtual machine in the cluster.
• das.slotmeminmb - Defines the maximum bound on the memory slot size. If this option is used, the slot size is the smaller of this value or the maximum memory reservation plus memory overhead of any powered-on virtual machine in the cluster.

It helps in a situation when you have one VM with high CPU or RAM reservations. Such VM will not increase HA Slot Size but it consumes smaller HA Slots.

Default VM reservation values for HA slot calculation can be defined by another two advanced options.

• das.vmcpuminmhz - Defines the default CPU resource value assigned to a virtual machine if its CPU reservation is not specified or zero. This is used for the Host Failures Cluster Tolerates admission control policy. If no value is specified, the default is 32MHz.
• das.vmmemoryminmb - Defines the default memory resource value assigned to a virtual machine if its memory reservation is not specified or zero. This is used for the Host Failures Cluster Tolerates admission control policy. If no value is specified, the default is 0 MB.

Default VM reservation values can help you to define HA Slot Size you want but it doesn't automatically correspond with required overbooking and planed spare fail-over capacity because HA Slot Size is not proportional to VM sizes on a particular cluster. If you really want to have one real spare host fail-over capacity you have to go with option 3 (Use dedicated fail-over hosts).

Algorithm 2 : percentage cluster spare capacity

This algorithm doesn't use HA Slot size but it simply calculates total cluster CPU/MEM resources and decrease these cluster resources by spare capacity defined in percentage.  The rest of cluster available resources is also decreased by powered on VM reservations and new VMs can be powered on only when some cluster resources are available. Quite clear and simple, right? However, it also requires to have VM reservations otherwise you will end up with over-allocated cluster and your overbooking ratio will be too high which can introduce some performance issues. So once again, if you really want to have one real spare host fail-over capacity without dealing with VM reservations the best way is to go with option 3 (Use dedicated fail-over hosts).
Note that algorithm 2 doesn't use HA Cluster Advanced Options related to HA Slot mentioned above. However das.vmCpuMinMHz and das.vmMemoryMinMB can be used  to set default reservations. For more details read this.

Algorithm 3 : dedicated fail-over hosts

This algorithm simply dedicates specified hosts to be unused during normal conditions and used only in case of ESXi host failure. Multiple fail-over dedicated hosts are supported since vSphere 5.0. This algorithm will keep your capacity and performance absolutely predictable and independent on VM reservations. You'll get exactly what you configure.

UPDATE 2018-01-09: for some additional details about dedicated fail-over hosts read the blog post Admission Control - Dedicated fail-over hosts.

CONCLUSION
So what option to use? The correct answer is, as usually , ...  it depends :-)   

However, if VM reservations are not used and absolutely predictable N+X redundancy is required I currently recommend Option 3.

If you have a mental problem with not using some ESXi host during non-degraded cluster state (isn't it exactly what is required?) I recommend Option 1 but VM reservations must be used to have a realistic size of HA Slot. In this options, artificial HA Slot can be designed leveraging advanced options.

If you don't want elaborate with HA Slot and use all ESXi hosts in the cluster you can use Option 2 but VM reservations must be used for some capacity guarantee to avoid high overbooking ratio.

FEATURE REQUEST
It would be great if VMware vSphere has some kind of Cluster Reservation policy for VMs. For example, if you want to guarantee cluster resources overbooking 2:1 you would set up 50% CPU and 50% RAM reservations for each VM running in HA Cluster. This policy should be dynamic so if someone changes VM size from CPU or RAM perspective reservations would be recalculated automatically.

Let's break down our example above. We are assuming following HA CLUSTER RESERVATION POLICY => CPU 50%, RAM 50% assigned to our HA Cluster. Let's powered on VM with 2x vCPUs and 6GB RAM. Dynamic reservation calculation is quite easy from RAM perspective because memory reservation would be 3GB (50% from 6GB). It is a little bit more complicated from CPU reservation perspective. CPU dynamic reservation has to be calculated based on physical CPU where VM is running. So let's assume we have Intel Xeon E5-2450 @ 2.1GHz. So 50% from 2.1GHz is 1.05GHz but we have 2 vCPUs so we have to multiply it by 2. Therefore dynamic CPU reservation for our VM is 2.1GHz.  I believe with such dynamic reservation policy we would be able to guarantee overbooking ratio and define cluster redundancy more predictable from overbooking and performance degradation point of view.

FEEDBACK 
I would like to know what is your preferred HA Cluster Admission Control setting. So, don't hesitate to leave a comment and share your thoughts with the community. Any feedback is very welcome and highly appreciated.