Undocumented SDRS Advanced Options

Almost two years ago, I was challenged by one VMware customer who has experienced failed VM's provisioning in case of parallel VM deployments. SDRS default behavior is not optimized for fast multiple parallel deployments because it returns just SDRS recommendations (step 1) and later (step 2) these recommendations are applied by someone else who is executing VM provisioning. Back in the days, when SDRS was designed and developed, it was optimized for few VM provisioning, however nowadays when we are in a cloud era, more parallel provisionings are common.

Disclaimer: Please note that default behavior works best for most of the cases, do use advanced options only when it is required. Below are some of the Storage DRS advanced options user can configure on Storage DRS cluster. These options may or may not have been properly documented from my research/digging and it is most likely not supported by VMware. Please take caution if you decide to play with this advanced settings.

SDRS Advanced settings

VraInitPlacement
The reason behind this option is explained in this video - "VMware vSphere SDRS VM provisioning process". Based on my customer feedback, VMware engineering has introduced 'VraInitialPlacement' - SDRS advanced setting - which reconfigures SDRS to generate only one recommendation and reserve the resource till "lease" expires. Lease time is defined in API call as property resourceLeaseDurationSec. It is by default 0 (no lease) but it can be reconfigured on VRA side. It is used in conjunction with VRA custom property "VirtualMachine.Admin.Datastore.Cluster.ResourceLeaseDurationSec" at VMware VRA Documentation here - https://docs.vmware.com/en/vRealize-Automation/7.4/com.vmware.vra.prepare.use.doc/GUID-FA2ED665-4973-435C-A93B-8E4EAB5D1F8A.html

Custom property description: When provisioning to multiple VMs and using SDRS, specifies a value in seconds, in the range of 30 to 3600, for reserving storage resources during the RecommendDataStore API call. You can add this property to a business group or blueprint or when you request provisioning. The lease lock is only applied to the datastore that is used by the deployment, not all datastores in the storage cluster. The lease lock is released when provisioning either completes or fails. If not specified, no lock is applied to the storage resources at provisioning time.

Long story short, when 'VraInitPlacement' is set to "1", it generates only one recommendation. This is VMware internal undocumented option but in theory, it can be done by vSphere admin. But Lease Time is not configurable by vSphere Admin and it is the parameter (property) during vSphere SOAP API call asking for SDRS recommendations.

To set this advanced parameter choose Datastore Cluster > Edit Settings > SDRS Automation > Advanced Options (SDRS) > insert property  'VraInitialPlacement' with value 1

VraExpandDisk 
This is another SDRS Advanced option. When this is enabled and an expand disk will exceed the size of the datastore the disk will be storage vMotioned to another datastore which can service the request.

To set this advanced parameter choose Datastore Cluster > Edit Settings > SDRS Automation > Advanced Options (SDRS) > insert property  'VraExpandDisk' with value 1

EnforceStorageProfiles
To configure Storage DRS interop with SPBM, below options need to be set.

• 0 – disabled (default)
• 1 – soft enforcement
• 2 – hard enforcement

percentIdleMBinSpaceDemand
The PercentIdleMBinSpaceDemand setting defines the percentage of IdleMB that is added to the allocated space of a VMDK during free space calculation of the datastore. The default value is set to 25%. This value can range from 0 to 100.

For more info look at http://frankdenneman.nl/2012/10/01/avoiding-vmdk-level-over-commitment-while-using-thin-disks-and-storage-drs/

EnforceCorrelationForAffinity
Use datastore correlation while enforcing/fixing anti-affinity rules

• 0 – disabled (default)
• 1 – soft enforcement
• 2 – hard enforcement

OTHER VMWARE VSPHERE ADVANCED SETTINGS
Back in the days, William Lam published the blog post "New vSphere 5 HA, DRS and SDRS Advanced/Hidden Options". It is worth to read it.

What's new in vSphere 6.7

VMware vSphere 6.7 has been released and all famous VMware bloggers released their blog posts about new features and capabilities. It is worth to read all of these blog posts as each blogger is focused on a different area of SDDC so it can give you a broader context to newly available product features and capabilities. Anyway, industry veterans should start reading product Release Notes and official VMware blog posts first.

Please note, that this blog post is just an aggregation of information published in other places. All used sources are listed below.

Release Notes:
vSphere 6.7 Release Notes

VMware KB:
Important information before upgrading to vSphere 6.7 

VMware official blog posts:
Introducing VMware vSphere 6.7!
Introducing vCenter Server 6.7
Introducing Faster Lifecycle Management Operations in VMware vSphere 6.7
Introducing vSphere 6.7 Security
What’s new with vSphere 6.7 Core Storage
vSphere 6.7 Videos

Community blog posts:
Emad Younis : vCenter Server 6.7 What’s New Rundown
Duncan Epping : vSphere 6.7 announced!
Cormac Hogan : What's new in vSphere and vSAN 6.7 release?
Cody Hosterman : What's new in core storage in vSphere 6.7 part I: in-guest unmap and snapshots
Cody Hosterman :  What's new in core storage in vSphere 6.7 part V: Rate control for automatic VMFS unmap
William Lam : All vSphere 6.7 release notes & download links
Florian Greh (Virten) : VMware vSphere 6.7 introduces Skylake EVC Mode
Florian Greh (Virten) : New ESXCLI Commands in vSphere 6.7

So after reading all resources above let's aggregate and document interesting features area by area.

vSphere Management

vCenter with embedded platform services controller in enhanced linked mode. This is nice because you can leverage "vCenter Server High Availability" to achieve higher availability for PSC without the external load balancer. All benefits listed below.

• No load balancer required for high availability and fully supports native vCenter Server High Availability.
• SSO Site boundary removal provides flexibility of placement.
• Supports vSphere scale maximums.
• Allows for 15 deployments in a vSphere Single Sign-On Domain.
• Reduces the number of nodes to manage and maintain.

vSphere 6.7 introduces vCenter Server Hybrid Linked Mode, which makes it easy and simple for customers to have unified visibility and manageability across an on-premises vSphere environment running on one version and a vSphere-based public cloud environment, such as VMware Cloud on AWS, running on a different version of vSphere.

vSphere 6.7 also introduces Cross-Cloud Cold and Hot Migration, further enhancing the ease of management across and enabling a seamless and non-disruptive hybrid cloud experience for customers.

vSphere 6.7 enables customers to use different vCenter versions while allowing cross-vCenter, mixed-version provisioning operations (vMotion, Full Clone and cold migrate) to continue seamlessly.

vCenter Server Appliance (VCSA) Syslog now supports up to three syslog forwarding targets.

The HTML5-based vSphere Client provides a modern user interface experience that is both responsive and easy to use and includes 95% of functionality available in Flash Client. Some of the newer workflows in the updated vSphere Client release include:

• vSphere Update Manager
• Content Library
• vSAN
• Storage Policies
• Host Profiles
• vDS Topology Diagram
• Licensing

PSC/SSO CLI (cmsso-util) has some improvements. Repointing an external vCenter Server Appliance across SSO Sites within a vSphere SSO domain is supported. Repoint of vCenter Server Appliance across vSphere SSO domains is also supported. This is huge! It seems that SSO domain consolidation is now possible. The domain repoint feature only supports external deployments running vSphere 6.7. The repoint tool can migrate licenses, tags, categories, and permissions from one vSphere SSO Domain to another.

Brand-new Update Manager interface that is part of the HTML5 Web Client. The new UI provides a much more streamlined remediation process. 

New vROps plugin for the vSphere Client. This plugin is available out-of-the-box and provides some great new functionality. When interacting with this plugin, you will be greeted with 6 vRealize Operations Manager (vROps) dashboards directly in the vSphere client! 

Compute

vSphere 6.7 delivers a new capability that is key for the hybrid cloud, called Per-VM EVC. Per-VM EVC enables the EVC (Enhanced vMotion Compatibility) mode to become an attribute of the VM rather than the specific processor generation it happens to be booted on in the cluster. This allows for seamless migration across different CPUs by persisting the EVC mode per-VM during migrations across clusters and during power cycles.

A new EVC mode (Intel Skylake Generation) has been introduced.  Compared to Intel "Broadwell " EVC mode, the Skylake EVC mode exposes following additional CPU features:

• Advanced Vector Extensions 512
• Persistent Memory Support Instructions
• Protection Key Rights
• Save Processor Extended States with Compaction
• Save Processor Extended States Supervisor

Single Reboot when updating ESXi hosts. It is reducing maintenance time by eliminating one of two reboots normally required for major version upgrades.

vSphere Quick Boot is a new innovation that restarts the ESXi hypervisor without rebooting the physical host, skipping time-consuming hardware initialization (aka POST, Power-On Self Tests).

New ESXCLI Commands. In vSphere 6.7 the command line interface esxcli has been extended with new features. vSphere 6.7 introduced 62 new ESXCLI commands including:

• 3 Device
• 6 Hardware
• 1 iSCSI
• 14 Network
• 14 NVMe
• 2 RDMA
• 9 Storage
• 6 System
• 7 vSAN

for more information look here.

Fault Tolerance maximums increased. Up to 8 Virtual CPUs per virtual machine and up to 128 vRAM per FT VM. For more info look at https://configmax.vmware.com/ (ESXi Host Maximums)

Storage

Support for 4K native HDD. Customers may now deploy ESXi on servers with 4Kn HDDs used for local storage (SSD and NVMe drives are currently not supported). ESXi providing a software read-modify-write layer within the storage stack allowing the emulation of 512B sector drives. ESXi continues to expose 512B sector VMDKs to the guest OS. Servers having UEFI BIOS can boot from 4Kn drives.

XCOPY enhancement. XCOPY is used to offload storage-intensive operations such as copying, cloning, and zeroing to the storage array instead of the ESXi host. With the release of vSphere 6.7, XCOPY will now work with specific vendor VAAI primitives and any vendor supporting the SCSI T10 standard. Additionally, XCOPY segments and transfer sizes are now configurable. By default, the Maximum Transfer Size of an XCOPY ranges between 4MB-16MB. In vSphere 6.7, through the use of PSA claim-rules, this functionality is extended to additional storage arrays. Further details should be documented by particular storage vendor.

Configurable Automatic UNMAP. Automatic UNMAP was released with vSphere 6.5 with a selectable priority of none or low. Storage vendors and customers have requested higher, configurable rates rather than a fixed 25MBps. With vSphere 6.7 we’ve added a new method, “fixed” which allows you to configure an automatic UNMAP rate between 100MBps and 2000MBps, configurable both in the UI and CLI. I recommend reading this blog post for details how it works on Pure Storage.

UNMAP for SESparse. SESparse is a sparse virtual disk format used for snapshots in vSphere as a default for VMFS-6. In this release, automatic space reclamation for VM’s with SESparse snapshots on VMFS-6 is provided. This only works when the VM is powered on and only affect the top-most snapshot.

VVols enhancements. As VMware continues the development of Virtual Volumes, in this release is added support for IPv6 and SCSI-3 persistent reservations. With end-to-end support of IPv6, this enables organizations, including government, to implement VVols using IPv6. With SCSI-3 reservations, this substantial feature allows shared disks/volumes between virtual machines across nodes/hosts. Often used for Microsoft WSFC clusters, with this new enhancement it allows for the removal of RDMs!

Increased maximum number of LUNs/Paths (1K/4K LUN/Path). The maximum number of LUNs per host is now 1024 instead of 512 and the maximum number of paths per host is 4096 instead of 2048. Customers may now deploy virtual machines with up to 256 disks using PVSCSI adapters. Each PVSCSI adapter can support up to 64 devices. Devices can be virtual disks or RDMs. A major change in 6.7 is the increased number of LUNs supported for Microsoft WSFC clusters. The number increased from 15 disks to 64 disks per adapter, PVSCSI only. This changes the number of LUNs available for a VM running MICROSOFT WSFC from 45 to 192 LUNs.

The increased maximums for Virtual SCSI adapter (PVSCSI only). Up to 64 Virtual SCSI Targets Per Virtual SCSI Adapter and up to 256 Virtual SCSI Targets Per Virtual Machine.

VMFS-3 EOL. Starting with vSphere 6.7, VMFS-3 will no longer be supported. Any volume/datastore still using VMFS-3 will automatically be upgraded to VMFS-5 during the installation or upgrade to vSphere 6.7. Any new volume/datastore created going forward will use VMFS-6 as the default.

Support for PMEM /NVDIMMs. Persistent Memory or PMem is a type of non-volatile DRAM (NVDIMM) that has the speed of DRAM but retains contents through power cycles. It’s a new layer that sits between NAND flash and DRAM providing faster performance and it’s non-volatile unlink DRAM.

Intel VMD (Volume Management Device). With vSphere 6.7, there is now native support for Intel VMD technology to enable the management of NMVe drives. This technology was introduced as an installable option in vSphere 6.5. Intel VMD currently enables hot-swap management, as well as NVMe drive, LED control allowing similar control used for SAS and SATA drives.

RDMA (Remote Direct Memory Access) over Converged Ethernet (RoCE). This release introduces RDMA using RoCE v2 support for ESXi hosts. RDMA provides low latency, and higher-throughput interconnects with CPU offloads between the end-points. If a host has RoCE capable network adaptor(s), this feature is automatically enabled.

Para-virtualized RDMA (PV-RDMA). In this release, ESXi introduces the PV-RDMA for Linux guest OS with RoCE v2 support. PV-RDMA enables customers to run RDMA capable applications in the virtualized environments. PV-RDMA enabled VMs can also be live migrated.

iSER (iSCSI Extension for RDMA). Customers may now deploy ESXi with external storage systems supporting iSER targets. iSER takes advantage of faster interconnects and CPU offload using RDMA over Converged Ethernet (RoCE). We are providing iSER initiator function, which allows ESXi storage stack to connect with iSER capable target storage systems.

SW-FCoE (Software Fiber Channel over Ethernet). In this release, ESXi introduces software-based FCoE (SW-FCoE) initiator than can create FCoE connection over Ethernet controllers. The VMware FCoE initiator works on lossless Ethernet fabric using Priority-based Flow Control (PFC). It can work in Fabric and VN2VN modes. Please check VMware Compatibility Guide (VCG) for supported NICs.

Performance

vSphere 6.7 VCSA delivers phenomenal performance improvements (all metrics compared at cluster scale limits, versus vSphere 6.5):

• 2X faster performance in vCenter operations per second
• 3X reduction in memory usage
• 3X faster DRS-related operations (e.g. power-on virtual machine)

Security

vSphere 6.7 adds support for Trusted Platform Module (TPM) 2.0 hardware devices and also introduces Virtual TPM 2.0, significantly enhancing protection and assuring integrity for both the hypervisor and the guest operating system.

vSphere 6.7 introduces support for the entire range of Microsoft’s Virtualization Based Security technologies aka “Credential Guard” support.

Recoverability

vCenter Server Appliance (VCSA) File-Based Backup introduced in vSphere 6.5 now has a scheduler. Now customers can schedule the backups of their vCenter Server Appliances and select how many backups to retain. Another new section for File-Based backup is Activities. Once the backup job is complete it will be logged in the activity section with detailed information. The Restore workflow now includes a backup archive browser. The browser displays all your backups without having to know the entire backup path.

Conclusion

It seems that vSphere 6.7 is the continuous evolution of the best x86 virtualization platform with a lot of interesting improvements, features, and capabilities. Keep in mind, that this is just a list of features and capabilities which have to be very carefully planned, designed and tested before implementation into production.

Just FYI, I did not finish the reading of all vSphere 6.7 documents so I will update this blog post when find something interesting.

Deploying vCenter High Availability with network addresses in separate subnets

VMware vCenter High Availability is a very interesting feature included in vSphere 6.5. Generally, it provides higher availability of vCenter service by having three vCenter nodes (active/passive/witness) all serving the single vCenter service.

This is written in the official vCenter HA documentation

vCenter High Availability (vCenter HA) protects vCenter Server Appliance against host and hardware failures. The active-passive architecture of the solution can also help you reduce downtime significantly when you patch vCenter Server Appliance.
After some network configuration, you create a three-node cluster that contains Active, Passive, and Witness nodes. Different configuration paths are available.   

The last sentence is very true. The simplest VCHA deployment is within the same SSO domain and within the single datacenter with two Layer2 networks, one for management and second for the heartbeat. Such design can be deployed in fully automated manner and you just need to provide dedicated network (portgroup/VLAN) for the heartbeat network and use 3 IP addresses from separated heartbeat subnet. Easy. But is it what you are expecting from vCenter HA? To be honest, the much more attractive use case is to spread vCenter HA nodes across three datacenters to keep vSphere management up and running even one of two datacenters experiences some issue. Conceptually it is depicted in the figure below.

Conceptual vCenter HA Design

In this particular concept, I have embedded PSC controllers because of simplicity and vCenter HA can increase availability even of PSC services. The most interesting challenge in this concept is networking so let's look into the intended network logical design.

vCenter HA - networking logical design

Networking logical design:

• Each vCenter Server Appliance node has two NICs
• One NIC is connected to management network and second NIC to heartbeat network
• Layer 2 Management network (VLAN 4) is stretched across datacenters A and B because vCenter IP address must work without human intervention in datacenter B after VCHA fail-over.
• In each datacenter we have independent heartbeat network (VCHA-HB-A, VCHA-HB-B, VCHA-HB-C) with different IP subnets to not stretch Layer 2 across datacenters, especially not to datacenter C where is the witness. This requires specific static routes in each vCenter Server Appliance node to have IP reachability over heartbeat network.
• Specific VCHA network tcp/udp ports must be allowed among VCHA nodes across a heartbeat network.

Helpful documents:

• FAQ: vCenter High Availability (2148003) - https://kb.vmware.com/s/article/2148003
• Deploying vCenter High Availability with network addresses in separate subnets (2148442) - https://kb.vmware.com/kb/2148442
• Required ports for VCHA 6.5 (52835) - https://kb.vmware.com/kb/52835
• Errors "Failed to ssh connect peer node x.x.x.x" and "sshConnect Authentication (publickey) failed" while configuring vCenter HA (2150715) - https://kb.vmware.com/kb/2150715

Implementation Notes: 

Note 1:

VMware KB 2148442 (Deploying vCenter High Availability with network addresses in separate subnets) is very important to deploy such design but one information is missing there. After cloning of vCenter Server Appliances, you have to go to passive node and configure on eth0 the same IP address you use in active node. Configuration is in file  /etc/systemd/network/10-eth0.network.manual

Note 2:

In case of badly destroyed VCHA cluster use following commands to destroy VCHA from the command line

cd /etc/systemd/network 

mv 10-eth0.network.manual 20-eth0.networkdestroy-vchareboot

The solution was found at https://communities.vmware.com/thread/552084

Link to the official documentation (Resolving Failover Failures) - https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.avail.doc/GUID-FE5106A8-5FE7-4C38-91AA-D7140944002D.html

Note 3:

In case, you will see the error message MethodFault.summary error during the finalization process it is because a hostname mismatch is detected. The hostname assigned to the Passive node must be the same as the hostname of the Active node. The solution was found  at https://www.altaro.com/vmware/how-to-deploy-a-vcenter-ha-cluster-part-2/ but also written in KB https://kb.vmware.com/kb/2148442

How to check I/O device on VMware HCL

VMware has Hardware Compatibility List of supported I/O devices is available here
https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io

VMware HCL for I/O devices

The best identification of I/O device is VID (Vendor ID), DID (Device ID), SVID (Sub-Vendor ID), SSID (Sub-Device ID). VID, DID, SVID and SSID can be simply entered into VMware HCL and you will find if it is supported and what capabilities have been tested. You can also find supported firmware and driver.

The get these identifiers you have to log in to ESXi via SSH and use command "vmkchdev -l".  This command shows VID:DID SVID:SSID for PCI devices and you can use grep to filter just VMware NICs (aka vmnic)

vmkchdev -l | grep vmnic

You should get similar output

[dpasek@esx01:~] vmkchdev -l | grep vmnic

0000:02:00.0 14e4:1657 103c:22be vmkernel vmnic0

0000:02:00.1 14e4:1657 103c:22be vmkernel vmnic1

0000:02:00.2 14e4:1657 103c:22be vmkernel vmnic2

0000:02:00.3 14e4:1657 103c:22be vmkernel vmnic3

0000:05:00.0 14e4:168e 103c:339d vmkernel vmnic4

0000:05:00.1 14e4:168e 103c:339d vmkernel vmnic5

0000:88:00.0 14e4:168e 103c:339d vmkernel vmnic6

0000:88:00.1 14e4:168e 103c:339d vmkernel vmnic7

So, in case of vmknic4 there is
·       VID:DID SVID:SSID
·       14e4:168e 103c:339d


The same applies to HBAs and disk controllers.  For HBA and local disk controllers use

vmkchdev -l | grep vmhba

This is the output from my Intel NUC at my home lab

[root@esx02:~] vmkchdev -l | more

0000:00:00.0 8086:0a04 8086:2054 vmkernel 

0000:00:02.0 8086:0a26 8086:2054 vmkernel 

0000:00:03.0 8086:0a0c 8086:2054 vmkernel 

0000:00:14.0 8086:9c31 8086:2054 vmkernel vmhba32

0000:00:16.0 8086:9c3a 8086:2054 vmkernel 

0000:00:19.0 8086:1559 8086:2054 vmkernel vmnic0

0000:00:1b.0 8086:9c20 8086:2054 vmkernel 

0000:00:1d.0 8086:9c26 8086:2054 vmkernel 

0000:00:1f.0 8086:9c43 8086:2054 vmkernel 

0000:00:1f.2 8086:9c03 8086:2054 vmkernel vmhba0

0000:00:1f.3 8086:9c22 8086:2054 vmkernel 

vmhba0 is local disk controller
vmhba32 is USB storage controller
vmnic0 is network interface

Hope this helps.

vSphere 6.5 - DRS CPU Over-Commitment

A lot of VMware vSphere architects and engineers are designing their vSphere clusters for some overbooking ratios to define some level of the service (SLA or OLA) and differentiate between different compute tiers. They usually want to achieve something like

• Tier 1 cluster (mission-critical applications) - 1:1 vCPU / pCPU ratio
• Tier 2 cluster (business-critical applications) - 3:1 vCPU / pCPU ratio
• Tier 3 cluster (supporting applications) - 5:1 vCPU / pCPU ratio
• Tier 4 cluster (virtual desktops) - 10:1 vCPU / pCPU ratio

Terminology: 

• vCPU - virtual CPU configured for Virtual Machine
• pCPU - physical CPU core (hyper-threading is not considered) 

 

Before vSphere 6.5 we have to monitor it externally by vROps or some other monitoring tool. Some time ago I have blogged how to achieve it with PowerCLI and LogInsight - ESXi host vCPU/pCPU reporting via PowerCLI to LogInsight.

vSphere 6.5 DRS has introduced additional option to set maximum CPU over-commitment. It limits the number of vCPUs per pCPU in particular DRS cluster. However, it is good to know that there are two different advanced DRS options (configuration parameters) how to specify vCPU:pCPU ratio and each setting behaves differently. See table below ...

DRS Advanced Option	Scope	Min-Max Value
MaxVcpusPerClusterPct	cluster	0% - 500%
MaxVCPUsPerCore	host	0 - 32

It is worth to mention a little bit tricky setting of these additional options via GUI. It is good to know how GUI setting of "CPU Over-Commitment" is mapped to DRS cluster advanced options.

In my lab, I have VCSA 6.5 U1c (build 7119157) so I did some tests.

If I set "CPU Over-Commitment" in vSphere Web Client (Flash/Flex) it sets MaxVcpusPerClusterPct so it is the setting per the whole vSphere Cluster.

However, in vSphere Client (HTML5) it sets MaxVCPUsPerCore so it is per ESXi host.

Therefore, it is good to know what you would like to achieve and double check DRS Advanced Options.

See different behavior in screenshots below

6.5 U1c (build 7119157) vSphere Web Client (Flash/Flex) sets MaxVcpusPerClusterPct

6.5 U1c (build 7119157) vSphere Client (HTML5) sets MaxVCPUsPerCore

So this is how it should work. Now let's do some test to understand real behavior.

TEST 1: MaxVcpusPerClusterPct = 0 

Let's set MaxVcpusPerClusterPct to 0 so we are saying to allow 0 : 1 vCPU / pCPU ratio. In other words, no VM can be running in the cluster. And it works as expected. When I try to run VM I get the error "The total number of virtual CPUs present or requested in virtual machines' configuration has exceeded the limit on the host: 0". Well, it is a little bit misleading because it should be cluster-wide rule but it works as expected.

Error when MaxVcpusPerClusterPct is set to 0

TEST 2: MaxVcpusPerClusterPct = 100 

Let's set MaxVcpusPerClusterPct to 100% so we are saying to allow 1 : 1 vCPU / pCPU ratio. I have 4 node DRS cluster where each ESXi host has two cores (pCPUs), therefore I have 8 pCPUs available in the cluster. And I can really start only four VMs because each has 2 vCPUs so I can run up to 8 vCPUs in DRS cluster.

Error when MaxVcpusPerClusterPct is set to 100

It is great, but it is worth to mention that VMs were started on single ESXi host even I have 4 ESXi hosts in DRS cluster. So vCPU / pCPU ratio is compliant per cluster but not per ESXi host as I have 8 vCPUs on single ESXi hosts having just 2 pCPUs. But that's expected behavior. So far so good.

TEST 3: MaxVcpusPerCore = 0 

MaxVcpusPerCore should solve the problem observed in previous two test because it should set vCPU / pCPU ratio per host which is much better from the predictability point of view.

Let's set MaxVcpusPerCore to 0. My expectation was that I will not be able to start any VM but that was NOT the case. I was able to start a lot of VMs and exceed the expected vCPU / pCPU ratio. This is unexpected behavior.

TEST 4: MaxVcpusPerCore = 1 

Let's set MaxVcpusPerCore to 1.  My expectation was that I will not be able to start more than 2 vCPUs per ESXi host so only one VM with 2 vCPUs. Unfortunately, I was able to start much more vCPUs per ESXi host. This is again unexpected behavior.

TEST 5: MaxVcpusPerCore = 4 

I have been informed by DRS Engineer that the minimum value allowed by host over-commitment ratio option (MaxVCPUsPerCore) is 4:1.

So, let's set MaxVcpusPerCore to 4.  I have prepared nine VMs with 4 vCPUs each and my expectation is that I will not be able to start more than 8 vCPUs per ESXi host so only two VMs with 4 vCPUs per ESXi host. And because I have 4 ESXi hosts per cluster I should be able to start the maximum of 8 VMs.

Expected error message when MaxVcpusPerCore is set to 4 and vCPU:pCPU is over 4:1 per ESXi host.

UPDATE 2018-04-20: The issue with MaxVcpusPerCore is fixed in vSphere 6.5 U2. This is written in Release Notes: The advanced vSphere DRS parameter MaxVcpusPerCore might not work as expected and the desired ratio of virtual CPUs per physical CPU or core will not take effect in configurations below 4:1. MaxVcpusPerCore supported ratios now start from 1:1.
Please note, that MaxVcpusPerCore does not support ratio 0:1 in contrast with MaxVcpusPerClusterPct where 0:1 is possible and it will effectively disable to run any VM on the cluster.

Conclusion

Advanced DRS setting MaxVcpusPerClusterPct supports values between 0 and 500 and represents percentage between vCPUs and pCPUs across the whole DRS cluster. If the value is higher then 500, enforcing does not work. So, vCPU / pCPU percentage ratio can be enforced between 0:1 to 5:1.  vCPU:pCPU ratio 0:1 is a special setting where no VM can be PowerOn on vSphere Cluster. This is little bit risky setting but it can be used to put the whole cluster in kind of "maintenance mode" and forbid anyone to run VMs there.

Advanced DRS setting MaxVcpusPerCore currently supports values between 0 and 32 but it works only with values between 4 and 32. This setting enforces vCPU / pCPU per each ESXi host within DRS cluster. This means that the minimum vCPU / pCPU ration configurable by this option is 4 : 1.

To be honest, I think for vSphere architects/designers MaxVcpusPerCore makes more sense than MaxVcpusPerClusterPct because vCPU/pCPU overbooking ratio defines CPU quality per ESXi host.

VMware is internally considering to align MaxVcpusPerClusterPct and MaxVcpusPerCore behavior and allow lower vCPU / pCPU ratios when MaxVcpusPerCore is used. I will track how this topic will evolve in the future. Stay tuned.

SDRS Initial Placement - interim storage lease between recommendation and provisioning

Every day we learn something new. In the past, I blogged about SDRS behavior on these blog posts

• Storage DRS Design Considerations
• VMware vSphere SDRS - test plan of SDRS initial placement
• VMware vSphere SDRS VM provisioning process
• Storage DRS integration with storage profiles

Recently (a few months ago),  I have been informed about interesting SDRS behavior which is not exposed through standard GUI nor advanced settings but available through API. Such functionality was not very well known even within VMware so I have decided to blog about it.

Long story short ...

vSphere API Call for SDRS Initial Placement can lease recommended storage resource for some time. 

What does it mean? Just after recommendations, SDRS can lease the storage space on recommended datastores to have an interim reservation for somebody who is, most probably, going to do provisioning. By default, SDRS do not lease storage space on recommended datastores, therefore, you can observe provisioning failures in some situations. I have simulated such situation in Test #3 of test plan available here. Such situations are not very common when you do manual provisioning but there is higher probability when automated provisioning is in use so you can experience such issues on environments with VMware vRealize Automation (vRA) or vCloud Director (vCD).

And now the secret I did not know ... SDRS has the solution for such issues since vSphere 5.1. When somebody (vRA, vCD, anybody else who wants to deploy VM) asked for SDRS recommendation via API call, that API call can include a specific parameter (resourceLeaseDurationSec) which will instruct vSphere to block the recommended storage space on datastores only for provisioning of that specific SDRS recommendation. It's worth to mention, the resource leasing is released immediately after provisioning, therefore the time defined in resourceLeaseDurationSec is actually the maximum reservation time of the resource just in case somebody who wanted to do provisioning change his mind and decided to not deploy VM. This is to avoid unnecessary storage space reservations.
 
If you want to know details check API documentation. Here is what is written in vSphere API documentation about placeSpec.resourceLeaseDurationSec - Resource lease duration in seconds. If the duration is within bounds, Storage DRS will hold onto resources needed for applying recommendations generated as part of that call. Only initial placement recommendations generated by storage DRS can reserve resources this way.

Parameter resourceLeaseDurationSec is used in StoragePlacementSpec which encapsulates all of the inputs passed to the VcStorageResourceManager method recommendDatastores. VcStoragePlacementSpec is documented here.

So that sounds good, right? Well, there is one issue with this approach. SDRS can give provisioning application more recommendations (multiple datastores) which would lead to blocking more storage space than is really needed. VMware engineering is aware of this issue and at the moment works at least with vRA BU to solve it. As far as I know, the final solution will be a special SDRS setting to return a single recommendation. However, this is planned as a specific integration optimization between SDRS and vRA provisioning. 

UPDATE: details about special SDRS setting is described at https://www.vcdx200.com/2018/06/undocumented-sdrs-advanced-options.html

The challenge with VRA storage (vSphere vDisk) provisioning and SDRS is depicted in the figure bellow

To be honest, there is another design consideration and potential risk associated with this solution. If resourceLeaseDurationSec is used and an external application (VRA, vCloud Director, or other) is using it incorrectly, it can eventually block a storage space in the Datastore cluster and cause Deny of Service (DoS). Incorrect usage would be to leverage SDRS for recommendations that would block the recommended storage space but do not provision anything, therefore storage would be in block state for some defined time and not available for other provisionings until the lease expires.

What is the difference between VMware vRealize Suite and vCloud Suite

Several times I have been asked by my customers what is the difference between VMware vRealize Suite and vCloud Suite. Both are actually licensing packaging suits. VMware vCloud Suite suite is the superset of VMware vRealize Suite. In other words, vCloud Suite includes everything as vRealize Suite plus vSphere Infrastructure (ESXi Enterprise Plus licenses).

VMware vRealize Suite is a purpose-built management solution for the heterogeneous data center and the hybrid cloud. It is designed to deliver and manage infrastructure and applications to increase business agility while maintaining IT control. It provides the most comprehensive management stack for private and public clouds, multiple hypervisors, and physical infrastructure.

vRealize Suite editions comparison is available here and visualized on figure below.

vRealize Suite editions comparison

More specifically vRealize Suite 2017 includes following components:

• vSphere Replication 6.5.1
• vSphere Data Protection 6.1.5
• vSphere Big Data Extensions 2.3.2
• vRealize Orchestrator Appliance 7.3.0
• vRealize Suite Lifecycle Manager 1.0
• vRealize Operations Manager 6.6.1
• vRealize Business for Cloud 7.3.1
• vRealize Log Insight 4.5.0
• vRealize Automation 7.3.0
• vRealize Code Stream Management Pack for IT DevOps 2.2.1

VMware vCloud Suite brings together VMware’s industry-leading vSphere hypervisor with vRealize Suite, the complete cloud management solution.

vCloud Suite 2017 currently includes vRealize Suite 2017 and vSphere Enterprise Plus and NSX-V:

• Everything included in vRealize Suite 2017
• plus vSphere Enterprise Plus Infrastructure for vCloud Suite (ESXi licenses only, vCenter is not included)
• ESXi 6.5 U1
• plus NSX-V
• NSX-V 6.3.3

What's new in 2017 Suites?

1. VMware has introduced new overall manager called vRealize Suite Lifecycle Manager to simplify deployment and on-going management of the vRealize products.
2. NSX-V (NSX for vSphere) is included in vCloud Suite 2017 ???

Please note, that vCenter Server is not included in vCloud Suite, therefore additional license for vCloud Suite is required.

Hope this helps to understand VMware license packaging.

...........................................................................................

UPDATE 2017-09-20:  The note about NSX-V licensing in vCloud Suite 2017 Release announcement is a little bit misleading. On VMware WebSite here is written

NSX is available as an optional component that can be purchased with vCloud Suite. 

So, VMware vCloud Suite 2017 customers are not entitled to use full NSX-V. It is just an entitlement to use NSX Manager for AV (replacement for vCNS Endpoint). NSX Manager for AV has been available for free even before vCloud Suite 2017 so that's nothing new. And of course, any VMware customer can purchase full NSX-V to extend their vSphere infrastructure for a network virtualization and datacenter modernization.

CLI for VMware Virtual Distributed Switch - implementation procedure

Some time ago I have blogged about perl scripts emulating well known physical network switch CLI commands (show mac-address-table and show interface status) for VMware Distributed Virtual Switch (aka VDS). See the blog post here "CLI for VMware Virtual Distributed Switch".

Now is the time to operationalize it. My scripts are written in Perl leveraging vSphere Perl SDK which is distributed by VMware as vCLI. vCLI is available for Linux and Windows OS. I personally prefer Linux over Windows, therefore, the implementation procedure below is for Centos 7.

Implementation procedure for Centos 7

1/ Install Centos 7 minimal OS installation

2/ Install Perl 

yum install perl

3/ Install vCLI Prerequisite Software for Linux Systems with Internet Access

yum install e2fsprogs-devel libuuid-devel openssl-devel perl-devel
yum install glibc.i686 zlib.i686
yum install perl-XML-LibXML libncurses.so.5 perl-Crypt-SSLeay

4/ Install the vCLI Package on a Linux System with Internet Access

Download vSphere Perl SDK (Use link here)
tar –zxvf VMware-vSphere-CLI-6.X.X-XXXXX.x86_64.tar.gz
sudo vmware-vsphere-cli-distrib/vmware-install.pl

5/ Install VDSCLI scripts

# Create directory for vdscli
mkdir vdscli

# Change directory to vdscli
cd vdscli

# Install wget to be able to download VDSCLI files
yum install wget

# Download vdscli.pl
wget https://raw.githubusercontent.com/davidpasek/vdscli/master/vdscli.pl

# Download supporting shell wraper for show mac-address-table
wget https://raw.githubusercontent.com/davidpasek/vdscli/master/show-mac-address-table.sh

# Download supporting shell wraper for show interface status
wget https://raw.githubusercontent.com/davidpasek/vdscli/master/show-interface-status.sh

# Change mod of all files in VDSCLI directory to allow execution
chmod 755 *

# Edit shell wrappers with your specific vCenter hostname and credentials

It is highly recommended to create a specific readonly (AD or vSphere SSO) account for VDSCLI as depicted on screenshot below

vSphere SSO account for VDSCLI

6/ VDSCLI validation

# You must be in vdscli directory
# Run command below
./show-mac-address-table.sh

You should get mac address table for VMware distributed virtual switch. Something like on screenshot below

Output from ./show-mac-address-table.sh 

So now we have Perl scripts to get information from VMware Distributed Virtual Switch. So far so good. However, we would like to have Interactive CLI to have the same user experience as we have on physical switches CLI, right? For Interactive CLI I have decided to use Python ishell (https://github.com/italorossi/ishell).

7/ iShell installation

# Install python
yum install python

# Install and upgrade pip which is Python package manager
yum install epel-release
yum install python-pip
pip install --upgrade pip

# We need gcc therefore we install all development tools
yum group install "Development Tools"
yum install python-devel
yum install ncurses-devel

# Now we can finally install ishell
pip install ishell

8/ Interactive VDSCLI validation

# You must be in vdscli directory
# Run command bellow
./vdscli-ishell.py

You should be able to use an interactive shell with just two show commands. See. the screenshot below to get the impression how it works.

VDSCLI Interactive Shell

9/ Remote SSH or Telnet

The last step is to expose VDSCLI interactive shell over SSH or Telnet. I will show you how to do it for SSH but if you enable Telnet on your linux server it will work as well.

Let's add specific OS user
adduser admin
passwd admin

You have to copy all vdscli scripts to home dir of the newly created user (/home/admin). It is user admin in our case but you can create any username you want.

We have to add our interactive shell /home/admin/vdscli-ishell.py into /etc/shells because only programs configred there can be used as shells.

chsh admin
and use /home/vdscli/vdscli-ishell.py as a shell

CONCLUSION

So at the end you can simply ssh to Linux system we have build and immediately use CLI to VMware Distributed Switch as depicted on screenshot below

VDSCLI Interactive Shell over SSH

And that was the goal. Hope somebody else in VMware community will find it useful.

CLI for VMware Virtual Distributed Switch

A few weeks ago I have been asked by one of my customers if VMware Virtual Distributed Switch (aka VDS) supports Cisco like command line interface. The key idea behind was to integrate vSphere switch with open-source tool Network Tracking Database (NetDB) which they use for tracking MAC addresses within their network. I have been told by customer that NetDB can telnet/ssh to Cisco switches and do screen scraping so would not it be cool to have the most popular switch CLI commands for VDS? These commands are

• show mac-address-table
• show interface status

The official answer is NO, but wait a minute. Almost anything is possible with VMware API. So my solution is leveraging VMware's vSphere Perl SDK to pull information out of Distributed Virtual Switches. I have prepared PERL script vdscli.pl which currently supports two commands mentioned above. It goes through all VMware Distributed Switches on single vCenter.

Script along with shell wrappers are available on GITHUB here https://github.com/davidpasek/vdscli

See screenshots below to get an idea what script does.

The output of the command

vdscli.pl --server=vc01.home.uw.cz --username readonly --password readonly --cmd show-port-status

looks as depicted in screenshot below.

and output of the command
vdscli.pl --server=vc01.home.uw.cz --username readonly --password readonly --cmd show-mac-address-table

So now we have Perl scripts to get information from VMware Distributed Virtual Switch which is nice, however, we would like to have Interactive CLI to have the same user experience as we have on physical switches CLI, right? For Interactive CLI I have decided to use Python ishell (https://github.com/italorossi/ishell) to emulate Cisco like CLI. To start interactive VDSCLI shell you must have Python with iShell installed and then you can simply run script

./vdscli-ishell.py

which is just a wrapper around vdscli.pl The screenshot of VDSCLI shell is in the figure below

VDSCLI Interactive Shell

And the last step is to allow SSH or Telnet access to VDSCLI shell. It can be very easily done via standard Linux possibility to change a shell for the particular user. The VDSCLI over ssh is depicted on the screenshot below.

VDSCLI Interactive Shell over SSH

To operationalize all these scripts, I would highly encourage you to read my another blog post ...
"CLI for VMware Virtual Distributed Switch - implementation procedure".

Hope somebody else in VMware community will find it useful.

VMware VVOLs scalability

I'm personally a big fan of VMware Virtual Volumes concept. If you are not familiar with VVOLs check this blog post with the recording of VMworld session and read VMware KB Understanding Virtual Volumes (VVols) in VMware vSphere 6.0

We all know that the devil is always in details. The same is true with VVOLs. VMware prepared the conceptual framework but implementation always depends on storage vendors thus it varies around storage products.

Recently, I have had VVOLs discussion with one of my customers and he was claiming that their particular storage vendor supports a very small number of VVOLs. That discussion inspired me to do some research.

Please, note that numbers below are valid at the moment of writing this article. You should always check current status with your particular storage vendor.

Vendor / Storage Array	Maximum VVOLs / Snapshots or Clones
DELL / Compellent SC 8000	2,000 / TBD
EMC / Unity 300	9,000 / TBD
EMC / Unity 400	9,000 / TBD
EMC / Unity 500	13,500 / TBD
EMC / Unity 600	30,000 / TBD
EMC / VMAX 3	64,000 / TBD
HPE / 3PAR	I have been told that it depends on specific model and firmware versions but it is roughly about 10,000 (the info gathered at November 2018)
Hitachi / VSP G200	2,000 / 100,000
Hitachi / VSP G400	4,000 / 100,000
Hitachi / VSP G600	4,000 / 100,000
Hitachi / VSP G800	16,000 / 100,000
Hitachi / VSP G1000	64,000 / 1,000,000

Numbers above are very important because single VM have minimally 3 VVOLs (home, data, swap) and usually even more (snapshot) or more data disks. If you will assume 10 VVOls for single VM you will end up with just 200 VMs on Dell Compellent or Hitachi VSP G200. On the other hand, EMC Unity 600 would give you up to 3,000 VMs which is not bad and enterprise storage systems (EMC VMAX and Hitachi G1000) would give you up to 6,400 VMs which is IMHO very good scalability.

So as always, it really depends on what storage system do you have or planning to buy.

If you know numbers for other storage systems, please share it in comments below this blog post.

Keywords: vvol, vvols, virtual volumes 

Maximum client sessions vCenter server can accept

I work as VMware TAM (Technical Account Manager) and one my customer had recently significant incident when clients (vSphere admins) was not able connect to vCenter server. It did not work nighter from old C# client nor new Web Client. It was interesting that sometimes some admins were able to connect and stay connected but others where not able to connect.

The error message was very general saying ...

Call "ServiceInstance.RetrieveContent" for object "ServiceInstance" on Server "vc01.example.com" failed.

C# Client returned another further explanation ...

The server 'vc01.example.com' could not interpret the client's request. (The remote server returned an error: (503) Server Unavailable.) 

See error messages in screenshot below ...

C# Client error messages

As you can see, both error messages are very general and further holistic troubleshooting was necessary. After multiple theories, one customer's vSphere/Windows administrator did a Windows OS analysis with Windows perfmon tool and realized that during the incident there were more then 1400 open threads with client connections to vCenter server. This turned in to the hypothesis that we have reached the maximum of client sessions vCenter can accept.

The hypothesis is always very important but even more important is the proof that hypothesis is valid and it is the root cause of particular issue.

Unfortunately, the maximum of total client sessions to vCenter server is not documented. The only numbers documented in "Configuration Maximums - vSphere 5.5" are ..

Concurrent vSphere Client connections to vCenter Server = 100Concurrent vSphere Web Clients connections to vCenter Server = 180

However, my customer is using automation extensively, therefore PowerCLI can have additional connections. The only way how to know the maximum is to test it.

My customer is still on vCenter 5.5 but I have prepared and executed the test in my home lab where I have vCenter 6.0 U2. I prepared PowerCLI script to create 2000 new client sessions and keep sessions open. The purpose of script is to find the maximum of established sessions vCenter can accept and see what will be the error message when maximum will be achieved.

The PowerCLI script is available on GitHub here
https://github.com/davidpasek/powercli-scripts/blob/master/vcenter-sessions.ps1
and it is based on excellent blog post and scripts "List and Disconnect vCenter Sessions" prepared by Alan Renouf.

I run the script in my lab and waited when it fails to find the maximum. You can see the expected failure on screenshot below ...

Expected connection failure to find what is the maximum

And the result is ...

vCenter Server 6.0 U2 accepts maximally 1995 established client sessions

When the above maximum is exceeded you are not able to connect to vCenter server any more and you will see the error messages mentioned at the beginning of this article.

Business impact and visibility

It is good to mention that this technical issue was observed during Disaster Recovery fail-over test and it silently disappeared after fail back of all services. That's the reason why this incident had very high internal business visibility and the issue was escalated to top IT management which required very quick Root Cause Analysis and proper problem management.

That's just another proof how vCenter and vSphere platform is critical in modern IT environments.

It seems, that my customer is using some automation script which establish connection to vCenter server, but because of some circumstances which happening only when services are running on disaster recovery backup site, the script does not disconnect sessions and the vCenter server maximum is exceeded and it does not accept any new connections. In such situation, vSphere platform is unmanageable.

This is good to know, especially in the age of automation, where single badly written automation script, can crash vSphere manageability.

As VMware TAM, I can communicate and justify my customer's product feature requests internally inside VMware organization.  That's another benefit of VMware TAM Program.

So here is publicly written vCenter Product Feature Request which I will open with our Product Management.

Feature Request: Maximum of supported client sessions should be documented in "vSphere Configuration Maximums" document. When the maximum is exceeded, vCenter server should accept at least one more connection for vSphere Administrator (for example administrator@vsphere.local) which should be used as last resort or back door if you wish. Such special "back door" connection should be terminated and re-established by the most recent connection of vSphere Admin to allow manageability in such situation.

VMware vSphere 6.0 PSC and SSO Domain useful resources

I do not have real numbers but it seems obvious and logical that SMB and midrange customers are adopting the latest VMware software much quicker then large enterprise customers. To be more precise, they are probably already running vSphere 6.0 and planing to upgrade to 6.5 now or soon. Some of them just waiting for 6.5 U1 which is expected soon.

On the other hand, the largest VMware customers are logically more conservative and starting migrations from vSphere 5.5 to 6.0 just now, in time of writing this article (beginning of 2017). These large customers have significantly larger scale therefore their PSC/SSO topology is much more complex.

During last few weeks I have discussed some vSphere 6 PSC/vCenter topology design decision points with these customers and I have decided to write down blog post about few useful, publicly available, resources / documents for such discussions.

First and foremost,  FAQ below is the most comprehensive VMware KB article about this topic.

FAQ: VMware Platform Services Controller in vSphere 6.0 (2113115)
https://kb.vmware.com/kb/2113115

The most surprised information, even for long time VMware customers, are following two Q&A's from FAQ above.

Q: Can I merge two vSphere Domains together?
A: No, there is no way to merge two vSphere domains together.

Q: Can I get Enhanced Linked Mode (ELM) between two, separate vSphere domains?
A: No, Enhanced Linked Mode requires that all PSCs be in the same domain and replicating. Since two separate vSphere Domains do not have a means of replicating, the new APIs that provide ELM cannot display the contents of both domains.

What does it mean?
Well, if you have multiple independent vSphere 5.5 SSO domains and you want to merge them, you have to do it in vSphere 5.5 before upgrade to 6.0 because you will not be able to do so in vSphere 6 and later.
Note: I do not know how it will change in longer term but it is the true even for vSphere 6.5 which is the latest version in time of writing this blog post.

Q: One of my customers asked me if the same vSphere SSO name (vsphere.local) in their two separate datacenters means that it is the same vSphere domain.
A: No. If you do not have replication between domains, there are not the same domain even they have the same name.

Another good question, you have to ask yourselves is, if you should or should not merge your vSphere domains. The typical reason for single vSphere domain is requirement for Enhanced Linked Mode (ELM). What Enhanced Linked Mode will give you? Below are several benefits of ELM:

• You can log in to all linked vCenter Server systems simultaneously with a single user name and password.
• With Enhanced Linked Mode, you can view and search across all linked vCenter Server systems. This mode replicates roles, permissions, licenses, and other key data across systems.
• You can view and search the inventories of all linked vCenter Server systems within the vSphere Web Client.
• Roles, permission, licenses, tags, and policies are replicated across linked vCenter Server systems.
• You can use WebClient GUI to do cross vCenter vMotion

However, any technology has some limits. In case of vSphere, we should always look at vSphere Configuration Maximums. The relevant information from configuration maximums are

• Maximum PSCs per vSphere Domain - 8
• Maximum PSCs per site, behind a load balancer - 4
• Maximum number of VMware Solutions connected to a single PSC - 4
• Maximum number of VMware Solutions in a vSphere Domain - 10

What are VMware Solutions?

A VMware Solution is defined as a product that creates a Machine Account and one or more Solution User (a collection of vSphere services) within the VMware Directory Service when the product is joined to the PSC, thus the vSphere Domain. The Machine Account and Solution User(s) are used to broker and secure communication between other Solutions available within the vSphere environment. In order to count against these maximums, the Machine Account and Solution Users must be fully integrated with all of the PSC's available feature sets (Identity Management and Authentication Brokering, Certificate Management, Licensing, etc.) such that the product makes full use of the PSC. At this time, only vCenter Server is defined as a fully integrated solution and counts against these maximums. Partially integrated solutions, such as vCenter Site Recovery Manager, vCloud Director vRealize Orchestrator, vRealize Automation Center, and vRealize Operations, do not count against these defined maximums.

So in other words, vCenters are currently the only solutions which counts into maximum of 10 VMware solutions. 

Now, when you know if you really need and want to merge vSphere domains it must be done in vSphere 5.5 because in vSphere 6 it is not possible.

I was asked by one of my customers, where is written that vSphere domain merging is supported and how it can be done.

Bellow are two blog post written by blogger Thom Greene ...

Merging SSO Domains in vCenter 5.5 part 1: Why?
https://www.thomgreene.com/blog/2016/11/4/merging-sso-domains-in-vcenter-55-part-1-why

Merging SSO Domains in vCenter Server 5.5 pt 2: How?
https://www.thomgreene.com/blog/2016/11/7/merging-sso-domains-in-vcenter-server-55-pt-2-how

and very detailed blog post of Andreas Peetz referred by Thom in his posts.

Re-pointing vCenter Server 5.5: A Survival Guide to KB2033620
https://www.v-front.de/2016/03/re-pointing-vcenter-server-55-survival.html

... but resources above are not VMware official documents so where are VMware official documents? Andreas' blog posts are referring to following VMware KB's

Migrating two VMware vCenter Single Sign-On embedded VMware vCenter Servers in the same VMware vCenter Single Sign-On domain (2130433)
https://kb.vmware.com/kb/2130433

How to repoint and re-register vCenter Server 5.1 / 5.5 and components (2033620)
https://kb.vmware.com/kb/2033620

VMware vCenter Server 5.1/5.5 fails to start after re-registering with vCenter Single Sign-On (2048753)
https://kb.vmware.com/kb/2048753

Old but still informative blog post ... vSphere Datacenter Design – vCenter Architecture Changes in vSphere 6.0 – Part 1

https://blogs.vmware.com/consulting/2015/03/vsphere-datacenter-design-vcenter-architecture-changes-vsphere-6-0-part-1.html

Additional VMware resources:

Platform Services Controller Topology Decision Tree
https://blogs.vmware.com/vsphere/2016/04/platform-services-controller-topology-decision-tree.html

vCenter Server Topology Considerations
https://youtu.be/6mh_FGlQ2lk

Reconfigure a Standalone vCenter Server with an Embedded Platform Services Controller to a vCenter Server with an External Platform Services Controller - link

How to repoint vCenter Server 6.x between External PSC within a site (2113917)
https://kb.vmware.com/kb/2113917

Using the cmsso command to unregister vCenter Server from Single Sign-On (2106736)
https://kb.vmware.com/kb/2106736

and just another related blog post from William Lam
How to split vCenter Servers configured in an Enhanced Linked Mode (ELM)?
http://www.virtuallyghetto.com/2017/03/how-to-split-vcenter-servers-configured-in-an-enhanced-linked-mode-elm.html

Understanding the Impacts of Mixed-Version vCenter Server Deployments

Using esxtop to identify storage performance issues for ESX / ESXi

ESXi performance are exposing to administrators through vSphere Clients. You can see real-time performance statistics which are collected in 5 minute intervals where each interval consists of fifteen 20 seconds samples. It is obvious that 20 second sample is pretty large for storage performance where we are working in mili or even micro second scale.

20 seconds contains 20,000 milliseconds

Let's be clear here, we will never have full visibility but smaller monitoring sample will give as better clue what is really happening inside the system. It is similar to microscope device.

The smallest monitoring samples can be achieved by ESXi utility ESXTOP. The default esxtop delay between monitoring points (sample) is 5 seconds. However, it can be lowered up to 2 seconds by parameter -d 2

For real analytics the esxtop data must be exprted to external file. In esxtop terminology it is batch mode and it is achieved by parameter -b 

Another important factor is what statistics (metrics) we are going to collect. The best is to collect all statistics because during performance analytics you have to correlate multiple values against each other. It is achieved by parameter -a

And last parameter is -n which defines how many iterations you want to perform in batch mode. So in example below we will have 30 iterations with delay between each other 2 seconds. So we will do total monitoring for 60 seconds.

esxtop -b -a -d 2 -n 30 > esxtop-data.csv

For all esxtop parameters see screenshot below.

 [root@esx11:~] esxtop -h  
 usage: esxtop [-h] [-v] [-b] [-l] [-s] [-a] [-c config file] [-R vm-support-dir-path]   
         [-d delay] [-n iterations]  
        [-export-entity entity-file] [-import-entity entity-file]   
        -h prints this help menu.  
        -v prints version.  
        -b enables batch mode.  
        -l locks the esxtop objects to those available in the first snapshot.  
        -s enables secure mode.  
        -a show all statistics.  
        -c sets the esxtop configuration file, which by default is .esxtop60rc  
        -R enables replay mode.  
        -d sets the delay between updates in seconds.  
        -n runs esxtop for only n iterations. Use "-n infinity" to run esxtop forever.  
        -----Experimental Features-------------  
        -export-entity writes the entity ids into a file, which can be modified  
         to select interesting entities.  
        -import-entity reads the file of selected entities. If this opion   
         is used, esxtop only shows the data for the selected entities.  

It is important to know, that esxtop will give you significantly more statistics you can see in vSphere Client level. That's another important benefit of esxtop. But each benefit has also some drawbacks or impact. The impact is, that single esxtop output line can have several thousands statistic counters. For example ESXi 6.0 host with just 2 running VMs in my home lab has 27,314 counters. My customer's product ESXi host has over 330,000 counters! So the output file can be pretty large in case you run it for 24 hours. Count on it.

In the file are very interesting counters. Following counters for physical disk devices are the most interesting

### Reponse times
Average Guest MilliSec/Command
Average Kernel MilliSec/Command
Average Queue MilliSec/Command
Average Queue MilliSec/Read
Average Driver MilliSec/Command
Average Driver MilliSec/Write
### Queue
Adapter Q Depth
### IOPS
Reads/sec
Writes/sec
Commands/sec
### MB/s
MBytes Read/sec
MBytes Written/sec"
### Split commands
Split Commands/sec
### SCSI Reservations
Reserves/sec
Failed Reserves/sec
Conflicts/sec
### Failures
Failed Commands/sec
Failed Reads/sec
Failed Writes/sec
Failed Bytes Read/sec
Failed Bytes Written/sec
Aborts/sec
Resets/sec

Some of above counters are not available in vSphere Client but the big benefit is that esxtop will give you data in 2 second interval which is much better granularity.

I hear your questions - So what now? How to analyze esxtop output file?
Well, you can replay it back in esxtop or you can use any of following tools

• VisualEsxtop
• perfmon
• excel
• esxplot

To be honest, none of tools above fulfilled my requirements therefore I'm writing my own Python script for esxtop output analysis.

The Python script is available on github here.