Network benchmark (iperf) of 10Gb Data Center Interconnect

I wanted to test 10Gb ethernet link I have got as data center interconnect between two datacenters. I generally do not trust anything I have not tested.

If you want test something, it is important to have good testing methodology and toolset.

Toolset

OS: FreeBSD 14.2 is IMHO the best x86-64 operating system in terms of networking. Your mileage may vary.

Network benchmark testing tool: IPERF (iperf2) is weel known tool to benchmark network performance and bandwidth.

Hypervisor: VMware ESXi 8.0.3 is the best in class hypervisor to test varios virtual machines

Methodology

I have use two Virtual Machines. At the end I will test network throughput between two VMs, where one VM is in each end of network link (DC Interconnect). However before the final test (Test 4) of DC interconnect throughput, I will test network throughput (Test 1) within the same VM to test localhost throughput, (Test 2) between VMs within single hypervisor (ESXi) host to avoid using physical network, (Test 3) VMs across two hypervisors (ESXi) within single VLAN in one datacenter to test local L2 throughput.

Results

Test 1: Network throughput within the same VM to test localhost throughput

VMware Virtual Machines have following hardware specification:

• 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
• 8 GB RAM
• 8 GB vDisk
  
• 1 vNIC (vmxnet) 

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -t 60

Network Throughput: 75.4Gb/s - 83Gb/s

CPU usage on server/client: 23%

MEM usage on server/client: ~500MB 

 

2 iperf connections / 4 CPU Threads (-P 2)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 2 -t 60

Network Throughput: 90.8Gb/s - 92Gb/s

CPU usage on server/client: 28%

MEM usage on server/client: ~500MB

4 iperf connections / 8 CPU Threads (-P 4)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 4 -t 60

Network Throughput: 88.5Gb/s - 89.1Gb/s

CPU usage on server/client: 29%

MEM usage on server/client: ~500MB 

 

8 iperf connections / 16 CPU Threads (-P 8)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 8 -t 60

Network Throughput: 91.6Gb/s - 95.3Gb/s

CPU usage on server/client: 30%

MEM usage on server/client: ~500MB 

 

Tests with Higher TCP Windows Size (800kB)

 

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -w 800k -t 60

Network Throughput: 69.8Gb/s - 81.0Gb/s

CPU usage on server/client: 28%

MEM usage on server/client: ~500MB

 

2 iperf connections / 4 CPU Threads (-P 2 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 2 -w 800k -t 60

Network Throughput: 69.8Gb/s - 69.9Gb/s

CPU usage on server/client: 28%

MEM usage on server/client: ~500MB

 

4 iperf connections / 8 CPU Threads (-P 4 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 2 -w 800k -t 60

Network Throughput: 69.2Gb/s - 70.0Gb/s

CPU usage on server/client: 28%

MEM usage on server/client: ~500MB

 

8 iperf connections / 16 CPU Threads (-P 8 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 8 -w 800k -t 60

Network Throughput: 72.6Gb/s - 74.0Gb/s

CPU usage on server/client: 28%

MEM usage on server/client: ~500MB

Test 2: Network throughput between VMs within hypervisor (no physical network)

VMware Virtual Machines with following hardware specification:

• 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
• 8 GB RAM
• 8 GB vDisk
  
• 1 vNIC (vmxnet) 

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -t 60

Network Throughput: 6.5Gb/s - 6.71Gb/s

CPU usage on server: 70%

CPU usage on client: 30-50%

MEM usage on server/client: ~500MB 

 

2 iperf connections / 4 CPU Threads (-P 2)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 2 -t 60

Network Throughput: 8.42Gb/s -8.62Gb/s

CPU usage on server: ~33%

CPU usage on client: ~30%

MEM usage on server/client: ~500MB

 

4 iperf connections / 8 CPU Threads (-P 4)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 4 -t 60

Network Throughput: 19.5Gb/s - 20.2Gb/s

CPU usage on server: 85%

CPU usage on client: 48%

MEM usage on server/client: ~500MB

 

 

8 iperf connections / 16 CPU Threads (-P 8)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 8 -t 60

Network Throughput: 17.1Gb/s - 18.4Gb/s

CPU usage on server: ~85%

CPU usage on client: ~30%

MEM usage on server/client: ~500MB

 

Tests with Higher TCP Windows Size (800kB)

 

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -w 800k -t 60

Network Throughput: 6.57Gb/s - 6.77Gb/s

CPU usage on server: 24%

CPU usage on client: 24%

MEM usage on server/client: ~500MB

 

2 iperf connections / 4 CPU Threads (-P 2 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 2 -w 800k -t 60

Network Throughput: 7.96Gb/s -8.0Gb/s

CPU usage on server: ~30%

CPU usage on client: ~28%

MEM usage on server/client: ~500MB

 

4 iperf connections / 8 CPU Threads (-P 4 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 4 -w 800k -t 60

Network Throughput: 15.8Gb/s -18.8Gb/s

CPU usage on server: ~85%

CPU usage on client: ~40%

MEM usage on server/client: ~500MB

 

8 iperf connections / 16 CPU Threads (-P 8 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 8 -w 800k -t 60

Network Throughput: 19.1Gb/s - 22.8Gb/s

CPU usage on server: ~98%

CPU usage on client: ~30%

MEM usage on server/client: ~500MB

 

Test 3: Network throughput between VMs across two hypervisors within VLAN (25Gb switch ports) in one DC

VMware Virtual Machines have following hardware specification:

• 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
• 8 GB RAM
• 8 GB vDisk
  
• 1 vNIC (vmxnet) - connected to 25Gb physical switch ports

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -t 60

Network Throughput: 6.1Gb/s - 6.34Gb/s

CPU usage on server: 23%

CPU usage on client: 17%

MEM usage on server/client: ~500MB 

 

2 iperf connections / 4 CPU Threads (-P 2)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 2 -t 60

Network Throughput: 9.31Gb/s -10.8Gb/s

CPU usage on server: ~43%

CPU usage on client: ~30%

MEM usage on server/client: ~500MB

 

4 iperf connections / 8 CPU Threads (-P 4)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 4 -t 60

Network Throughput: 19.5Gb/s - 20.2Gb/s

CPU usage on server: 85%

CPU usage on client: 48%

MEM usage on server/client: ~500MB

8 iperf connections / 16 CPU Threads (-P 8)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 8 -t 60

Network Throughput: 17.1Gb/s - 18.4Gb/s

CPU usage on server: ~80%

CPU usage on client: ~50%

MEM usage on server/client: ~500MB

Tests with Higher TCP Windows Size (800kB)

 

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -w 800k -t 60

Network Throughput: 6.11Gb/s - 6.37Gb/s

CPU usage on server: 16%

CPU usage on client: 22%

MEM usage on server/client: ~500MB

 

2 iperf connections / 4 CPU Threads (-P 2 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 2 -w 800k -t 60

Network Throughput: 9.81Gb/s -10.9Gb/s

CPU usage on server: ~39%

CPU usage on client: ~25%

MEM usage on server/client: ~500MB

 

4 iperf connections / 8 CPU Threads (-P 4 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 4 -w 800k -t 60

Network Throughput: 16.5Gb/s -19.8Gb/s

CPU usage on server: ~85%

CPU usage on client: ~40%

MEM usage on server/client: ~500MB

 

8 iperf connections / 16 CPU Threads (-P 8 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 8 -w 800k -t 60

Network Throughput: 17.7Gb/s - 18.2Gb/s

CPU usage on server: ~80%

CPU usage on client: ~50%

MEM usage on server/client: ~500MB

Test 4: Network throughput between VMs across two hypervisors across two interconnected VLANs across two DCs

VMware Virtual Machines have following hardware specification:

• 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
• 8 GB RAM
• 8 GB vDisk
  
• 1 vNIC (vmxnet) 

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 4 -t 60

Network Throughput: 9.74 Gb/s

Conclusion

Network throughput requires CPU cycles, therefore number of CPU cores matters.

 

iperf client by default uses one connection for generating network traffic where each connection uses 2 vCPUs (hyper-threading threads). In such default configuration I was able to achieve ~6.65 Gb/s in VM with at least 2 vCPU, which is not enough to test 10Gb/s datacenter interconnect. 

 

By using parameter -P 4, four parallel iperf client connections are initiated where each iperf connection uses 2 vCPUs (hyper-threading threads), therefore it can leverage all 8 vCPUs we have in testing VM.

 

By using parameter -P 8 in VM, eight parallel iperf client connections are initiated where each iperf client connection uses 2 vCPUs (hyper-threading threads), therefore it can leverage 16 vCPUs, but us we use only 8 vCPUs in our test machine, it only make bigger stress on existing CPUs and therfore it can have negative impact on overall network throughput.

 

The best practice is to use -P 4 for iperf client on machine with 8 CPUs as iperf client connections can be balanced across all 8 available CPUs. If you have more CPUs available, parameter -P should be the half of number of available CPUs.

• 1 CPUs VM can achieve network traffic up to 5.83 Gb/s. During such network traffic, CPU is fully used (100% usage) and maximum single iperf connection throughput of 6.65 Gb/s cannot be acieved duw to CPU constraint.
• 2 CPUs VM can achieve network traffic up to 6.65 Gb/s. During such network traffic, CPU is fully used (100% usage).
• 4 CPUs VM with -P 2 is necessary to achieve network traffic up to 10 Gb/s.
• 8 CPUs VM with -P 4 is necessary to achieve network traffic over 10 Gb/s. These 8 threads can generate 20 Gb/s which is good enough to test my 10Gb/s data center interconnect. 

Another iperf parametr which in theory could improve network throughput is the parameter -w which defines TCP Window Size. iperf by default uses TCP Window Size between 32kB and 64kB. By increasing TCP Window Size to 800kB (-w 800k) can slightly improve (~10%) performance during higher stress on CPU (-P 8 = 8 Processes / 16 Threads) across VMs. However, higher TCP Window Size (-w 800k) has negative impact (in some cases almost 30%) on localhost network throughput performance.

What real network throughput I have measured during this testing excercise? 

Localhost network throughput is significantly higher than network throughput across Virtual Machines or accross physical network and servers. We can achieve between 75 Gb/s and 95 Gb/s on Localhost. Network traffic does not need across virtual and physical hardware. It is logical that virtual and physical hardware introduces some bottlenecks.

 

Network throughput between VMs within single hypervisor can achieve 6.5 Gb/s with single process and two threads.  Up to 22.8 Gb/s (eight processes / sixteen threads and higher TCP Windows Size - 800kB) and 20.2 Gb/s with eight processes / sixteen threads and default TCP Windows Size. 

Network throughput between VMs within VLAN (25 Gb switch ports) in one data center can achieve up to 20.2 Gb/s (eight processes / sixteen threads and standard TCP Windows Size).

 

When you would need higher throughput than 20 Gb/s between VMware virtual machines, more CPU cores and special performance tuning of vNIC/vmxnet driver would need to be done. Such performance tunning would be about enabling Jumbo Frames (MTU 9,000, ifconfig_vmx0="inet <IP> netmask <NETMASK> mtu 9000") into guest OS, increasing Network Buffers in FreeBSD kernel (kern.ipc.maxsockbuf, net.inet.tcp.sendspace, net.inet.tcp.recvspace=4194304), Enable TCP Offloading (ifconfig_vmx0="inet <IP> netmask <NETMASK> mtu 9000 txcsum rxcsum tso4 tso6 lro"), Tune Interrupt Moderation, Use Multiple Queues aka RSS (sysctl net.inet.rss.enabled=1, sysctl net.inet.rss.bits=4). Fortunatelly enough, 20 Gb throughput is good enough to test my 10 Gb data center interconnect. 

Network throughput between VMs accros 10 Gb data center interconnect can achieve 9.74 Gb/s (four iperf connections / eight vCPUs in use). 9.74 Gb/s TCP throughput over 10 Gb/s data center ethernet interconnect is acceptable throughput.

VMware PowerCLI (PowerShell) on Linux

VMware PowerCLI is very handy and flexible automation tool allowing automation of almost all VMware features. It is based on Microsoft PowerShell. I do not have any Microsoft Windows system in my home lab but I would like to use Microsoft PowerShell. Fortunately enough, Microsoft PowerShell Core is available for Linux. Here is my latest runbook how to leverage PowerCLI in Linux management workstation leveraging Docker Application packaging.

Install Docker in your Linux Workstation

This is out of scope of this runbook. 

Add yourself to User Group docker

sudo usermod -aG docker $USER

Note: you must logout and login back to be able to run docker.

Pull official and verified VMware PowerCLI

docker pull vmware/powercli:latest

Pull official and verified Microsoft Powershell

docker pull mcr.microsoft.com/powershell:latest

List container images

docker image ls

How to run container?

Now you can run any of two above powershell/powercli containers interactively (-i) and in allocated pseudo-TTY (-t).

docker run  -it  CONTAINER-NAME

docker run  -it vmware/powercli 

or

docker run  -it mcr.microsoft.com/powershell

 

VMware PowerCLI container

Using VMware PowerCLI container is easier because it contains PowerShell with PowerCLI prepared by VMware

This command will run VMware PowerCLI and you have everything ready to go.

docker run  -it \

-v /home/dpasek/powercli/scripts:/tmp/scripts \

-v /home/dpasek/powercli/settings:/root/.local/share/VMware/PowerCLI \ 

vmware/powerclicore

The -v option in Docker is used to mount volumes, allowing data to persist or be shared between the host and the container. 

We map our scripts at /home/dpasek/powercli/scripts directory to container directory /tmp/scripts.

 

We are keeping PowerCLI config at /home/dpasek/powercli/settings directory. This allows us to keep PowerCLI configuration persistent and we can run the following configuration commands just once and they stay persistent. Following commands are necessary to initialize PowerCLI configuration.

Set-PowerCLIConfiguration -Scope User -ParticipateInCEIP $true -Confirm:$false
Set-PowerCLIConfiguration -InvalidCertificateAction:ignore -Confirm:$false

... and now you can connect to vCenter and list VMs

Connect-VIServer -Server <vcenter-server> -User <username> -Password <password> | Out-Null

Get-VM | Select-Object -ExpandProperty Name

... if we have PowerShell script prepared, we can run it directly from the host system. Following example will run script /tmp/scripts/get-vms.ps1

docker run  -it \

-v /home/dpasek/powercli/scripts:/tmp/scripts \

-v /home/dpasek/powercli/settings:/root/.local/share/VMware/PowerCLI \

--entrypoint='/usr/bin/pwsh'  \

vmware/powerclicore \

/tmp/scripts/get-vms.ps1

 

Microsoft PowerShell container

If you want use Microsoft PowerShell container, you can, but you have to install PowerCLI manualy. 

Following command will run Microsoft PowerShell

docker run -it -v /home/dpasek/scripts:/root/scripts mcr.microsoft.com/powershell

Inside Microsoft PowerShell container, you must install PowerCLI manualy

Install-Module -Name VMware.PowerCLI -Scope CurrentUser -Force

You must allow untrusted certificates and enable (or disable) participation in VMware CEIP.

Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false

Set-PowerCLIConfiguration -Scope User -ParticipateInCEIP $true

And now you can connect to vCenter, display vCenter Instance UUID, list ESXi hosts, and list VMs

Connect-VIServer -Server <vcenter-server> -User <username> -Password <password>

(Get-View ServiceInstance).Content.About.InstanceUuid

Get-VMhost

Get-VM

 

How to update ESXi with unsupported CPU?

I have old unsupported servers in my lab used for ESXi 8.0.3. In such configuration, you cannot update ESXi by default procedure in GUI.

vSphere Cluster Update doesn't allow remediation

ESXi host shows unsupported CPU

Solution is to allow legacy CPU and update ESXi from shell with esxcli.

Allow legacy CPU

The option allowLegacyCPU is not available in the ESXi GUI (DCUI or vSphere Client). It must be enabled using the ESXi shell or SSH. Bellow are command to allow legacy CPU.

esxcli system settings kernel set -s allowLegacyCPU -v TRUE

You can verify it by command ...

esxcli system settings kernel list | grep allowLegacyCPU

If above procedure fails, the other option is to edit file /bootbank/boot.cfg and add allowLegacyCPU=true to the end of kernelopt line.

In my case, it look like ...

kernelopt=autoPartition=FALSE allowLegacyCPU=true

After modifying /bootbank/boot.cfg, ESXi configuration should be saved to make changes persistent across reboots.

 /sbin/auto-backup.sh

Reboot of ESXi is obviously required to make kernel option active.

reboot

After reboot, you can follow by standard system update procedure by ESXCLI method as documented below.

ESXi update procedure (ESXCLI method)

1. Download appropriate ESXi offline depot. You can find URL of depot in Release Notes of particular ESXi version. You will need Broadcom credentials to download it from Broadcom support site.
2. Upload (leveraging Datastore File Browser, scp, winscp, etc.) ESXi offline depot to some Datastore
• in my case /vmfs/volumes/vsanDatastore/TMP
3. List profiles in ESXi depot
• esxcli software sources profile list -d /vmfs/volumes/vsanDatastore/TMP/VMware-ESXi-8.0U3d-24585383-depot.zip 
4. Update ESXi to particular profile with no hardware warning
• esxcli software profile update -d /vmfs/volumes/vsanDatastore/TMP/VMware-ESXi-8.0U3d-24585383-depot.zip -p ESXi-8.0U3d-24585383-no-tools --no-hardware-warning
5. Reboot ESXi
•   reboot
  

Hope this helps other folks in their home labs with unsupported CPUs.

Broadcom (VMware) Useful Links for Technical Designer and/or Architect

Lot of URLs have been changed after Broadcom acquisition of VMware. That's the reason I have started to document some of useful links for me.

VMware Product Configuration Maximums - https://configmax.broadcom.com

Network (IP) ports Needed by VMware Products and Solutions - https://ports.broadcom.com/

VMware Compatibility Guide - https://compatibilityguide.broadcom.com/ (aka https://www.vmware.com/go/hcl)

VMware Product Lifecycle - https://support.broadcom.com/group/ecx/productlifecycle (aka https://lifecycle.vmware.com/)

Product Interoperability Matrix - https://interopmatrix.broadcom.com/Interoperability

VMware Hands-On Lab - https://labs.hol.vmware.com/HOL/catalog

Broadcom (VMware) Education / Learning - https://www.broadcom.com/education

VMware Validated Solutions - https://vmware.github.io/validated-solutions-for-cloud-foundation/

If you are independent consultant and have to open support ticket related to VMware Education or Certification you can use form at https://broadcomcms-software.wolkenservicedesk.com/web-form  

VMware Health Analyzer

• Full VHA download: https://docs.broadcom.com/docs/VHA-FULL-OVF10
• Collector VHA download: https://docs.broadcom.com/docs/VHA-COLLECTOR-OVF10
• Full VHA license Register Tool: https://pstoolhub.broadcom.com/

 Do you know any other helpful link? Use comments below to let me know. Thanks.

How my Microsoft Windows OS syncing the time?

This is very short post with the procedure how to check time synchronization of Microsoft Windows OS in VMware virtual machine.

There are two options how time can be synchronized

1. via NTP 
2. via VMware Tools with ESXi host where VM is running 
   

The command w32tm /query /status shows the current configuration of time sync.

 Microsoft Windows [Version 10.0.20348.2582]  
 (c) Microsoft Corporation. All rights reserved.  
 C:\Users\david.pasek>w32tm /query /status  
 Leap Indicator: 0(no warning)  
 Stratum: 6 (secondary reference - syncd by (S)NTP)  
 Precision: -23 (119.209ns per tick)  
 Root Delay: 0.0204520s  
 Root Dispersion: 0.3495897s  
 ReferenceId: 0x644D010B (source IP: 10.77.1.11)  
 Last Successful Sync Time: 2/4/2025 10:14:10 AM  
 Source: DC02.example.com  
 Poll Interval: 7 (128s)  
 C:\Users\david.pasek>   

If Windows OS is connected to Active Directory (this is my case), it synchronize time with AD via NTP by default. This is visible in the output of command w32tm /query /status.

You are dependent on Active Directory Domain Controllers, therefore, the correct time in Active Directory Domain Controllers is crucial. I was blogging how to configure time in virtualized Active Directory Domain Controller back in 2011. Is is very old post but it still should work.

To check if VMware Tools are syncing time with ESXi host use following command

 C:\>"c:\Program Files\VMware\VMware Tools\VMwareToolboxCmd.exe" timesync status  
 Disabled  

VMware Tools time sync is disabled by default, which is the VMware best practice. It is highly recommended to not synchronize time with underlaying ESXi host and leverage NTP sync over network with trusted time provider. This will help you in case someone will make configuration mistake and time is not configured properly in particular ESXi.  

Hope you find this useful.

CPU cycles required for general storage workload

I recently published a blog post about CPU cycles required for network and VMware vSAN ESA storage workload. I realized it would be nice to test and quantify CPU cycles needed for general storage workload without vSAN ESA backend operations like RAID/RAIN and compression.

Performance testing is always tricky as it depends on guest OS, firmware, drivers, and application, but we are not looking for exact numbers and approximations are good enough for a general rule of thumb helping pure designer during capacity planning. 

My test environment was old Dell PowerEdge R620 (Intel Xeon CPU E5-2620 @ 2.00GHz), with ESXi 8.0.3 and Windows Server 2025 in a Virtual Machine (2 vCPU @ 2 GHz, 1x para-virtualized SCSI controller/PVSCSI, 1x vDisk). Storage subsystem was VMware VMFS datastore on local NVMe consumer-grade disk (Kingston SNVS1000GB flash).

Storage tests were done using an old good Iometer.

VMware Desktop Products direct download links

UPDATE: Direct links below do not work anymore. They are redirected to https://support.broadcom.com

Main URL for all desktop products: https://softwareupdate.vmware.com/cds/vmw-desktop/

VMware Fusion: https://softwareupdate.vmware.com/cds/vmw-desktop/fusion/

VMware Workstation: https://softwareupdate.vmware.com/cds/vmw-desktop/ws/

VMware Remote Console (VMRC): https://softwareupdate.vmware.com/cds/vmw-desktop/vmrc/

You do not need to have a Broadcom account. All VMware desktop products are directly downloadable without signing in.

VMware Health Analyzer - how to download and register the tool

Are you looking for VMware Health Analyzer? It is not easy to find it so here are links to download and register the tool to get the license.

Full VHA download: https://docs.broadcom.com/docs/VHA-FULL-OVF10

Collector VHA download: https://docs.broadcom.com/docs/VHA-COLLECTOR-OVF10

Full VHA license Register Tool: https://pstoolhub.broadcom.com/

I publish it mainly for my own reference but I hope other VMware community folks find it useful.

Every I/O requires CPU Cycles - vSAN ESA is not different

This is the follow-up blog post to my recent blog post about "benchmark results of VMware vSAN ESA".

It is obvious and logical that every computer I/O requires CPU Cycles. This is not (or better to say should not be) a surprise for any infrastructure professional. Anyway, computers are evolving year after year, so some rules of thumb should be validated and sometimes redefined from time to time.

Every bit transmitted/received over the TCP/IP network requires CPU cycles. The same applies to storage I/O. vSAN is a hyper-converged software-defined enterprise storage system, therefore, it requires TCP/IP networking for data striping across nodes (vSAN is RAIN - Redundant Array of Independent Nodes) and storage I/Os to local NVMe disks. 

VMware vSAN ESA - storage performance testing

I have just finished my first VMware vSAN ESA Plan, Design, and Implement project and had a chance to test vSAN ESA performance. By the way, every storage should be stressed and benchmarked before being put into production. VMware's software-defined hyperconverged storage (vSAN) is no different. It is even more important because the server's CPU, RAM, and Network usually used only for VM workloads are leveraged to emulate the enterprise-class storage.

What is the core dump size for ESXi 8.0 U3?

Nine years ago, I wrote the blog "How large is my ESXi core dump partition?". Back then, it was about core dumps in ESXi 5.5. Over the years, a lot has changed in ESXi which is true for core dumps too. 

Let's write a new blog post about the same topic but right now for ESXi 8.0 U3. The behavior should be the same in ESXi 7.0. In this blog post, I will use some data from ESXi 7.0 U3 because we are still running ESXi 7.0 U3 in production and I plan and design the upgrade to vSphere 8. That's why I have ESXi 8.0 U3 just in the lab where some hardware configurations are unavailable. We use ESXi hosts with 1.5 TB RAM in production but I don't have hosts with such memory capacity in my lab.

What is a core dump? It boils down to PSOD. ESXi host Purple Screen of Death (PSOD) happens when VMkernel experiences a critical failure. This can be due to hardware issues, driver problems, deadlock, etc. During the PSOD event, the ESXi hypervisor captures a core dump to help diagnose the cause of the failure. Here’s what happens during this process:

After a PSOD, ESXi captures a core dump, which includes a snapshot of the hypervisor memory and the state of the virtual machines. The core dump is stored based on the host configuration (core dump partition, file, or network), and it helps diagnose the cause of the critical failure by providing insights into the state of the system at the time of the crash. A core dump is crucial for troubleshooting and resolving the issues leading to PSOD. And here is the change. In ESXi 6.7, the core dump was stored in a disk partition but since ESXi 7, it has been stored in the precreated file.

For the detailed vSphere design, I would like to know the typical core dump file size to allocate optimal storage space for core dumps potentially redirected to shared datastore (by default, in ESXi 7 and later, the core dumps are stored in ESX-OSData partition, typically on boot disk). Of course, the core dump size depends on multiple factors, but the main factor should be the memory used by vmKernel.   

ESXi host memory usage is split into three buckets

1. vmKernel memory usage (core hypervisor)
2. Other memory usage
• BusyBox Console including
• Core BusyBox Utilities (e.g., ls, cp, mv, ps, top, etc.)
• Networking and Storage Tools (ifconfig, esxcfg-nics, esxcfg-vswitch, esxcli, etc.)
• Direct Console User Interface (DCUI)
• Management Agents and Daemons (hostd, vpxa, network daemons like SSH, DNS, NTP, and network file copy aka NFC)
3. Free memory

So let's go to the lab and test it. Here is data from three different ESXi host configurations I have access to. 

ESXi, 8.0.3 (24022510) with 256 GB (262 034 MB) physical RAM

vSAN is disabled, NSX is installed

In Production mode running 10 Powered On VMs having 24 GB vRAM:

• vmKernel memory usage:  1544 MB
• Other memory usage: 21 498 MB
• Free memory: 238 991 MB

In Maintenance mode (no VMs):

• vmKernel memory usage:  1453 MB
• Other memory usage: 4 207 MB
• Free memory: 256 373 MB

Let's try PSOD on the ESXi host in maintenance mode.

In ESXi 8.0.3 with 256 GB RAM, the core dump is set to be stored into a 3.6 GB file (3,882,876,928 bytes) at the ESX-OSData.

 [root@dp-esx02:~] esxcli system coredump file list  
 Path                                                   Active Configured    Size  
 ------------------------------------------------------------------------------------------------------- ------ ---------- ----------  
 /vmfs/volumes/66d993b7-e9cd83a8-b129-0025b5ea0e15/vmkdump/00000000-00E0-0000-0000-000000000008.dumpfile  true    true 3882876928  

It is configured and active. 

 [root@dp-esx02:~] esxcli system coredump file get  
   Active: /vmfs/volumes/66d993b7-e9cd83a8-b129-0025b5ea0e15/vmkdump/00000000-00E0-0000-0000-000000000008.dumpfile  
   Configured: /vmfs/volumes/66d993b7-e9cd83a8-b129-0025b5ea0e15/vmkdump/00000000-00E0-0000-0000-000000000008.dumpfile  

The core dump file has 3.6 GB

 [root@dp-esx02:~] ls -lah /vmfs/volumes/66d993b7-e9cd83a8-b129-0025b5ea0e15/vmkdump/00000000-00E0-0000-0000-000000000008.dumpfile  
 -rw-------  1 root   root    3.6G Oct 29 13:07 /vmfs/volumes/66d993b7-e9cd83a8-b129-0025b5ea0e15/vmkdump/00000000-00E0-0000-0000-000000000008.dumpfile  

Now let's try the first PSOD on the ESXi host in maintenance mode and watch what happens. Below is the command to initiate PSOD and the screenshot

 vsish -e set /reliability/crashMe/Panic 1  

VMware Support will ask you for zdump file (VMware proprietary bin file) which can be generated by command esxcfg-dumppart

 [root@dp-esx02:~] esxcfg-dumppart --file --copy --devname /vmfs/volumes/66d993b7-e9cd83a8-b129-0025b5ea0e15/vmkdump/00000000-00E0-0000-0000-000000000008.dumpfile --zdumpname /vmfs/volumes/DP-STRG02-Datastore01/zdump/zdump-coredump.dp-esx02.1  
 Created file /vmfs/volumes/DP-STRG02-Datastore01/zdump/zdump-coredump.dp-esx02.1.1  
 [root@dp-esx02:~] ls -lah /vmfs/volumes/DP-STRG02-Datastore01/zdump/zdump-coredump.dp-esx02.1.1  
 -rw-r--r--  1 root   root   443.9M Oct 29 13:07 /vmfs/volumes/DP-STRG02-Datastore01/zdump/zdump-coredump.dp-esx02.1.1  

The extracted VMkernel zdump file from the first PSOD has 443.9 MB.

Now let's try the second PSOD.

 vsish -e set /reliability/crashMe/Panic 1  

Let's extract the core dump.

 [root@dp-esx02:~] esxcfg-dumppart --file --copy --devname /vmfs/volumes/66d993b7-e9cd83a8-b129-0025b5ea0e15/vmkdump/00000000-00E0-0000-0000-000000000008.dumpfile --zdumpname /vmfs/volumes/DP-STRG02-Datastore01/zdump/zdump-coredump.dp-esx02.2  
 Created file /vmfs/volumes/DP-STRG02-Datastore01/zdump/zdump-coredump.dp-esx02.2.1  
 [root@dp-esx02:~] ls -lah /vmfs/volumes/DP-STRG02-Datastore01/zdump/zdump-coredump.dp-esx02.2.1  
 -rw-r--r--    1 root     root      311.2M Nov  4 09:33 /vmfs/volumes/DP-STRG02-Datastore01/zdump/zdump-coredump.dp-esx02.2.1

The extracted VMkernel zdump file from the 2nd PSOD has 311.2 MB.

Only one core dump exists in the core dump file; therefore, multiple core dumps are not stored in the system. Thus the best practice is to extract every core dump ( esxcfg-dumppart --file --copy)  from the core dump file to an external storage location to allow core dump analysis of older PSODs. 

 

Let's continue in our PSOD test with additional PSOD's to find out if the system can manage sequential core dumps having a total size bigger than 3.6 GB, which is the size of a core dump file. Let's assume the single core dump size is always around 300 MB, we would need 13 PSODs, so let's do 14 PSODs. 

_3rd PSOD:    304.9 MB

_4th PSOD:    285.4 MB

_5th PSOD:    303.2 MB

_6th PSOD:    316.6 MB

_7th PSOD:    322.9 MB

_8th PSOD:    288.3 MB

_9th PSOD:    283.0 MB

10th PSOD:    276.7 MB

11th PSOD:    292.5 MB

12th PSOD:    289.7 MB

13th PSOD:    281.8 MB

14th PSOD:    290.3 MB

TOTAL:           4.2 GB

So even though we have a 3.6 GB core dump file for the ESXi host with 256 GB RAM, we can collect more than 4 GB core dumps. This is the proof, that the core dump file is used for a single core dump and the next core dump rewrites the old one. 

ESXi, 8.0.3 (24022510) with 128 GB (131 008 MB) physical RAM

vSAN is disabled, NSX is not installed

In Maintenance mode (no VMs):

• vmKernel memory usage:  694 MB
• Other memory usage: 1 660 MB
• Free memory: 128 653 MB

In ESXi 8.0.3 with 128 GB RAM, the core dump is set to be stored into a 2.27 GB file (2,441,084,928 bytes) at the ESX-OSData partition.

 [root@esx21:~] esxcli system coredump file list  
 Path                                                                                                     Active  Configured        Size
-------------------------------------------------------------------------------------------------------  ------  ----------  ----------
/vmfs/volumes/6727594d-c447be9c-5a0e-90b11c13fc14/vmkdump/4C4C4544-0054-5810-8033-B3C04F48354A.dumpfile    true        true  2441084928

Now let's try PSOD on ESXi host in maintenance mode.

 vsish -e set /reliability/crashMe/Panic 1  

Let's extract the core dump.

[root@esx21:~] esxcfg-dumppart --file --copy --devname /vmfs/volumes/6727594d-c447be9c-5a0e-90b11c13fc14/vmkdump/4C4C4544-0054-5810-8033-B3C04F48354A.dumpfile --zdumpname /vmfs/volumes/ESX21
-FLASH-01/coredump.esx21.1
Created file /vmfs/volumes/ESX21-FLASH-01/coredump.esx21.1.1
[root@esx21:~] ls -lah /vmfs/volumes/ESX21-FLASH-01/coredump.esx21.1.1
-rw-r--r--    1 root     root      111.0M Dec  2 08:26 /vmfs/volumes/ESX21-FLASH-01/coredump.esx21.1.1

The VMkernel zdump file extracted from the PSOD on an ESXi 8.0 U3 host with 128 GB of RAM is 111 MB in size, which is significantly smaller than the zdump file from an ESXi 8.0 U3 host with 256 GB of RAM.

Let's compare it to ESXi host with 256 GB RAM

• 128 GB RAM is half of 256 GB RAM
• vmKernel memory usage 694 MB is ~half of 1453 MB
• coredump file 2.27 GB file is ~60% of 3.6 GB file
• zdump file 111 MB is  ~4x smaller than 443.9 MB

ESXi, 7.0.3 (23794027) with 512 GB (524 178 MB) physical RAM

In Production mode running 38 Powered On VMs having 310.37 GB vRAM:

• vmKernel memory usage:  3 227 MB
• Other memory usage: 366 140 MB
• Free memory: 154 810 MB

In Maintenance mode (no VMs):

• vmKernel memory usage:  2 776 MB
• Other memory usage: 25 402 MB
• Free memory: 495 998 MB

In ESXi 7.0.3 with 512 GB RAM, the core dump is set to be stored into an 8.16 GB file at the ESX-OSData partition.

 [root@prg03t0-esx05:~] esxcli system coredump file list  
 Path                                                        Active Configured    Size  
 ------------------------------------------------------------------------------------------------------------------ ------ ---------- ----------  
 /vmfs/volumes/6233a3c2-58e4bf62-94e7-0025b5ea0e13/vmkdump/00000000-00E0-0000-0000-000000000006-8162115584.dumpfile  true    true 8162115584  

Now let's try PSOD on ESXi host in maintenance mode.

 vsish -e set /reliability/crashMe/Panic 1  

Let's extract the core dump.

[root@prg03t0-esx05:~] esxcfg-dumppart --file --copy --devname /vmfs/volumes/6233a3c2-58e4bf62-94e7-0025b5ea0e13/vmkdump/00000000-00E0-0000-0000-000000000006-8162115584.dumpfile --zdumpname /vmfs/volumes/PRG03T0-HDD01/coredump.esx05.1
Created file /vmfs/volumes/PRG03T0-HDD01/coredump.esx05.1.1  
[root@prg03t0-esx05:~] ls -lah /vmfs/volumes/PRG03T0-HDD01/coredump.esx05.1.1
-rw-r--r--    1 root     root        4.6G Nov  5 18:11 /vmfs/volumes/PRG03T0-HDD01/coredump.esx05.1.1

The extracted VMkernel zdump file from the PSOD of ESXi 7.0 U3 with 512 GB RAM has 4.6 GB, which is significantly bigger than the zdump file from ESXi 8.0 U3 with 256 GB RAM. 

The VMkernel zdump file extracted from the PSOD on an ESXi 7.0 U3 host with 512 GB of RAM is 4.6 GB in size, which is significantly larger than the zdump file from an ESXi 8.0 U3 host with 256 GB of RAM.

Let's compare it to ESXi 8.0 U3 with 256 GB RAM

• 512 GB RAM is 2x bigger than 256 GB RAM
• vmKernel memory usage 2 776 MB is ~2x larger than 1453 MB (it makes sense)
• coredump file 8.16 GB file is ~2.25x larger than 3.6 GB file (it makes sense)
• zdump file 4.6 GB (4 710 MB) is ~10x larger than 443.9 MB (hmm, interesting)

Why is the zdump file 10x bigger on ESXi 7.0 U3 with 512 GB RAM and not just 2x bigger than I would expect? To be honest, I don't know. I have to retest it on ESXi 8.0 U3 with 512 GB RAM and 1.5 TB RAM when possible.

ESXi, 7.0.3 (23794027) with 1.5 TB (1 571 489 MB) physical RAM

This ESXi host (1.5 GB RAM) is only in production so it is managed by the operational team and we want to avoid testing PSOD in production. However, we checked the memory usage and core dump file size. 

In Maintenance mode (no VMs):

• vmKernel memory usage:  2 705 MB
• Other memory usage: 2 705 MB
• Free memory: 1 561 570 MB

In ESXi 7.0.3 with 1.5 TB RAM, the core dump is set to be stored into a 16.1 GB file at the ESX-OSData partition.

 [root@prg0301-esx36:~] esxcli system coredump file list  
 Path                                                         Active Configured     Size  
 ------------------------------------------------------------------------------------------------------------------- ------ ---------- -----------  
 /vmfs/volumes/5dec0956-3d83cd8b-de10-0025b52ae000/vmkdump/00000000-0021-0000-0000-000000000024.dumpfile        true    true 16106127360  

I cannot test PSOD in the production system by myself, so I have to wait until our operation team schedules the vSphere 8 upgrade and we can test it together. 

Conclusion

Core dump files for ESXi 6.7 and lower are stored in a disk partition. ESXi 7 and higher store core dumps into the core dump file. The core dump file is used for a single core dump, therefore it should be extracted (esxcfg-dumppart --file --copy) by vSphere administrator immediately after the PSOD otherwise it will be lost when another PSOD occurs.

In the current ESXi 8, the core file is located in ESX-OSData partition which can be on a boot disk or an additional disk.

If the boot disk is higher than 128 GB ESXi 8 the standard layout is

1. 101 MB   - Boot Loader partition
2. 4 GB        - Boot Bank 1 partition
3. 4 GB        - Boot Bank 2 partition
4. 119.9 GB - ESX-OSData partition

This is the disk usage of ESXi 8.0 U3 with 256 GB RAM and 128 GB boot disk - vSAN disabled, NSX installed.

 Filesystem  Size  Used Available Use% Mounted on  
 VMFSOS   119.8G  5.2G  114.6G  4% /vmfs/volumes/OSDATA-66d98185-2bceed00-72c5-0025b5ea0e0d  
 vfat     4.0G 274.1M   3.7G  7% /vmfs/volumes/BOOTBANK1  
 vfat     4.0G 338.9M   3.7G  8% /vmfs/volumes/BOOTBANK2  

As you see, 5.2 GB is used in ESX-OSData partition. 

We use ESXi with 1.5 TB RAM booting from SAN (Fibre Channel) in our production environment. The boot disk (LUN on shared storage) is a size of 32 GB. In such case, the partition layout looks as described below

1. 101 MB - Boot Loader partition
2. 4 GB      - Boot Bank 1 partition
3. 4 GB      - Boot Bank 2 partition
4. 23.9 GB - ESX-OSData partition

The ESX-OSData volume takes on the role of the legacy /scratch partition, locker partition for VMware Tools, and core dump destination. In ESX-OSData having 23.9 GB, there is still space for a core dump file (16.1 GB), log files, and trace files. If we want to keep 20% free space on ESX-OSData partition, we have 19.1 GB available. 16.1 GB is preallocated for the core dump file and 3 GB is available for logs and traces. This should be enough. 

Note: Logs and traces are also configured to be sent to a remote syslog server (Aria Operations for Logs / aka LogInsight).

Even though core dumps can be redirected to a shared datastore (by changing Scratch Partition location), keeping the core dump in the boot device is a relatively good design choice when using a 32 GB or higher capacity durable boot disk device (HDD, NVMe, SATADOM, etc.). 

The VMware minimum recommended boot disk size is 32 GB, while 128 GB is considered ideal. VMware also recommends using durable boot devices, such as local disks or NVMe drives, instead of SD cards or USB sticks for ESXi 7.0 and later versions.

Note: In my home lab, I still boot from USB and have ESX-OSData on NVMe disk because my old equipment does not support booting from NVMe.

IPv4 Addresses Cheat Sheet

Bellow is my cheat sheet about IPv4 addresses and subnetting.

The cheat sheet is primarily for myself :-), but somebody else can find it helpful and use it.

Description: The math binary representation of IP octets (bytes) and relation to Net Subnetting.

Keywords: Class Addressing, Classless Addressing

Monolithic versus Microservices application architecture consideration

Microservices application architecture is very popular nowadays, however, it is important to understand that everything has advantages and drawbacks. I absolutely understand advantages of micro-services application architecture, however, there is at least one drawback. Of course, there are more, but let's show at least the potential impact on performance. The performance is about latency.

Monolithic application calls functions (aka procedures) locally within a single compute node memory (RAM). Latency of RAM is approximately 100 ns (0.0001 ms) and Python function call in decent computer has latency ~370 ns (0.00037 ms). Note: You can test Python function latency in your computer with the code available at https://github.com/davidpasek/function-latency/tree/main/python

Microservices application is using remote procedure calls (aka RPC) over network. Typically as REST or gRPC call over https, therefore, it has to traverse the network. Even the latency of modern 25GE Ethernet network is approximately 480 ns (0.00048 ms is still 5x slower than latency of RAM), and RDMA over Converged Ethernet latency can be ~3,000 ns (0.003 ms), the latency of microservice gRPC function call is somewhere between 40 and 300 ms. [source] 

Conclusion

Python local function call latency is ~370 ns. Python remote function call latency is ~280 ms. That's the order of magnitude (10^6) higher latency of micro-services application. RPC in low-level programming languages like C++ can be 10x faster, but it is still 10^5 slower than local Python function call.

I'm not saying that micro-services application is bad. I just recommend to consider this negative impact on performance during your application design and specification of application services.

Grafana - average size of log line

As I'm currently participating on Grafana observability stack Plan & Design exercise, I would like to know what is the average size of log line ingested into the observability stack. Such information is pretty useful for capacity planning and sizing.

Log lines are stored on Loki log database and Loki itself is exposing metrics into Mimir time series database for self monitoring purpose. Grafana Loki and Promtail metrics are documented here.

The following formula calculates average size of log message:

sum(rate(loki_distributor_bytes_received_total [7d])) / sum(rate(loki_distributor_lines_received_total [7d]))

The result is visualized in the screenshot below.

 Hope this tip will be useful for someone else.

Farewell VMware

The clever people and Buddhists know that the only constant thing in the world is change. The change is usually associated with transition, and as we all know, transitions are not easy, but generally good and inevitable things. All transitions are filled with anticipation and potential risks, however, any progress and innovations are only achieved by accepting the risk and going outside of the comfort zone. That's one of the reasons I have decided to leave VMware, even though VMware organization and technologies are very close to my heart, and I truly believe that the VMware software stack is one of the most important IT technology stacks for the future of humans. 

As I prepare to move on, I have to say goodbye and a big thank you to the VMware organization. VMware technologies are part of my daily life for a long time, using the technology since 2006, and I’ll really miss the VMware family I joined back in 2015. It was a great time, and I will especially miss VMware core technical folks transforming the industry and building one of the best software-defined infrastructure stacks humans have done so far. 

And where do I actually go? Back in 2001, I was the co-founder of the software start-up, where I started my professional career by architecting, developing, and operating the air ticket online booking platform, which was later acquired by Galileo Travelport https://www.travelport.com. Now, after 20 years, I have got the proposal to help Kiwi.com to become the # 1 digital system in the modern digitalized travel industry. For those who do not know Kiwi.com, Kiwi.com was originally a Czech start-up growing into a worldwide #3 air ticketing booking platform. They were acquired by General Atlantic back in 2019 https://www.generalatlantic.com/media-article/general-atlantic-announces-strategic-partnership-with-travel-platform-kiwi-com/, and General Atlantic’s past and current investments in the global online travel industry include Priceline, Airbnb, Meituan, Flixbus, Uber, Despegar, Smiles and Mafengwo can tell you where the online travel industry is heading. Those who can read between the lines understand that such a mix allows to build optimal door-to-door traveling for the next human generation(s). I have decided that I would like to be part of such travel industry transformation! Not only because Kiwi.com really does multi-cloud with Kubernetes at a large scale, but mainly to be part of a very young, innovative, and inspiring team including hundreds of software developers and dozens of infrastructure platform and DevOps engineers operating everything as cloud computing.

I'm expecting big fun and you can expect more blog posts about DevOps, multi-cloud, Docker, Kubernetes, CI/CD, Observability, and infrastructure for modern applications because I have to learn and test a lot of new technologies and writing the blog post is the great way to share new knowledge and getting the feedback from other folks in various communities.

Hope my blog will be still useful for my current readers who are typically very IT infrastructure-oriented, however, the software eats the world, and the IT infrastructure is here to support software, isn't it? 

VMware vSphere DRS/DPM and iDRAC IPMI

I have four Dell server R620 in my home lab. I'm running some workloads which have to run 24/7 (DNS/DHCP server, Velocloud SD-WAN gateway, vCenter Server, etc.), however,  there are other workloads just for testing and Proof of Concepts purposes. These workloads are usually powered off. As electricity costs will most probably increase in near future, I realized VMware vSphere DRS/DPS (Distributed Resource Scheduler/Distributed Power Management) could be a great technology to keep the bill of electricity at an acceptable level.

VMware vSphere DPM is using IPMI protocol to manage physical servers. IPMI has to be configured per ESXi server as depicted in the screenshot below.

I have iDRAC Enterprise in my Dell servers and I thought it will be a simple task to configure iDRAC by just entering the iDRAC username, password, IP address, and MAC address.

However, I have realized that the configuration operation fails with an error message "A specified parameter was not correct: ipmiInfo".

During troubleshooting, I tested ipmi (ipmitool -I lanplus -H 192.168.4.222 -U root -P calvin chassis status) from FreeBSD operating system, and I have realized it does not work as well.

That led me to do some further research and to find, that iDRAC doesn't have IPMI enabled by default. iDRAC command to get the IPMI status is "racadm get iDRAC.IPMILan"

iDRAC command "racadm set iDRAC.IPMILan.Enable 1" enables IPMI over LAN and the command "racadm get iDRAC.IPMILan" can be used to validate the IPMI over LAN status.

After such iDRAC configuration, I was able to use IPMI from FreeBSD operating system.

And it worked correctly in VMware vSphere as well as depicted in the screenshot below. 

When IPMI is configured correctly on ESXi, the ESXi host can be switched into Standby Mode manually from vSphere Client as ESXi action.   

The ESXi Standby Mode is used for vSphere DRS/DPM automation. 

Job done!

Hope this helps some other folks in the VMware community.