VMware Cloud Foundation 9.0 Architecture and Fleet Components Latency

VMware Cloud Foundation (VCF) 9.0 Architecture is prepared to cover the whole planet. If your business covers the whole globe you proably have datacenters at least in three regions where these regions are typically located at EMEA (Europe / EU), AMER (America / United States), APJ (Asia / Malaysia, India, etc.). 

For such deployments, you have to consider network latency and the following diagram is for you.

VMware Cloud Foundation 9.0 Fleet Latency

However, there is also VCF Minimal Architecture starting with just 4 ESXi hosts (servers). 

In the rest of this blog post I will describe VCF terminology, VCF components and relationship between them.

VMware Certified Distinguished Expert (VCDX)

VMware is excited to announce the evolution of this iconic certification into a new, broader, and more inclusive framework: the VMware Certified Distinguished Expert (VCDX). This updated program extends beyond traditional design specializations and now welcomes a wider community of top-tier professionals, including Architects, Administrators, and Support specialists. The name change reflects a larger vision: to recognize excellence across all expert roles that shape, operate, and safeguard VMware Cloud Foundation environments at scale.

Introducing the VMware Certified Distinguished Expert (VCDX): A New Era for Elite Private Cloud Professionals.

 

How to expand ZFS on FreeBSD

Running out of disk space is one of the leading causes of IT outages. In this blog post, I will show you how to expand storage on FreeBSD with ZFS. ZFS works as volume manager and filesystem.

Datacenter Power Costs and Their Impact in the Virtualized World

I recently conducted a quick analysis of a VMware vSphere–based virtual datacenter for a customer, and here’s what I found.

The average monthly electricity consumption of a single vCPU with ~3 GB vRAM is 1.4 kWh, which translates to approximately $0.4  

The datacenter of my customer is located in Central Europe, and they pay 0.33 USD for 1 kWh of electricity in a Tier 3 datacenter facility (UPS + cooling included in energy cost).

Here are my questions for the broader worldwide infrastructure community.

• Q1: How much do you pay for electricity in your data center or server facility?
• Q2: What are the statistics of your cluster (CPU, memory, # of VMs, # of vCPUs, # of vRAM)?
• Q3: How much power do your physical servers consume on average?

For vSphere Cluster statistics, you can use the PowerCLI script Get-ClusterDensity. I use it to compare virtual cluster metrics (# of VMs, # of vCPUs) with the actual power consumption of physical servers, which you can obtain from your hardware’s management tools.

If you want to dive deeper into my quick analysis, read on.

Password expiration for both the VCSA root user and the vSphere administrator

Password expiration for both the VCSA root account and the vSphere administrator (typically administrator@vsphere.local) is a common issue, especially if the default 90-day expiration settings are overlooked. It recently happened to me in one lab environment. Fortunately, both passwords can be recovered. This blog post outlines the recovery methods that worked in my case.

vSAN ESA RAID5 issue? Not really, but ...

I was observing unexpected behavior in my vSAN ESA cluster. I have a 6-node vSAN ESA cluster and a VM with a Storage Policy configured for RAID-5 (Erasure Coding). Based on the cluster size, I would expect a 4+1 stripe configuration. However, the system is using 2+1 striping, which typically applies to clusters with only 3 to 5 nodes.

RAID-5 (2+1) striping is using 133% of the raw storage

RAID-5 (4+1) striping is using 120% of the raw storage

A 13% difference is worth investigating.

vSAN ESA RAIDs

vSAN ESA is VMware’s software-defined storage solution. Each virtual hard disk (vDisk) is represented as an object within the vSAN datastore. The properties of these vSAN objects are governed by vSAN VM Storage Policies, which define data placement and protection rules. While these policies may emulate traditional RAID (Redundant Array of Independent Disks), vSAN actually implements RAIN (Redundant Array of Independent Nodes). This is because data components, such as stripes and replicas, are distributed across failure domains, which by default correspond to vSphere/vSAN cluster nodes (ESXi hosts). The specific striping and distribution are determined by the configured failures-to-tolerate policy and vSAN cluster size.

vSAN ESA supports multiple levels of RAID/RAIN (Redundant Array of Independent Nodes) for data protection:

• RAIN-0: No redundancy (data is not protected)

• RAIN-1: Mirroring (1+1) across two nodes

• RAIN-5: Erasure coding with a 2+1 or 4+1 configuration (minimum 4 or 6 hosts)

• RAIN-6: Erasure coding with higher fault tolerance, typically 4+2, but can also be 6+2 or 8+2 depending on cluster size

These options allow you to balance storage efficiency, performance, and fault tolerance based on your specific workload and cluster topology.

How do I check the build or version number of VMware ESX?

The ESX build (version number) information is available in the Summary tab of the vSphere Client, but in larger environments it is worth to use some kind of automation. PowerShell/PowerCLI is well know scripting tool for VMware vSphere.

Below is PowerCLI one-liner to easily query all vCenters where you are connected ...

Get-VMhost | Select-Object Name,Version,Build 

If you want connect to vCenter(s) interactively, you can use following script ...

# Connect to vCenter
Write-Host "Connecting to vCenter ..."
$VC = Read-Host "Enter one vCentre Server or multiple vCenter servers delimted by comma."
Write-Host "Enter vCenter credentials ..."
$CRED = Get-Credential
Connect-VIServer -Server $VC -Credential $CRED -ErrorAction Stop | Out-Null

VMwareOpsGuide.com has been retired

I'm an architect and designer, not involved in day-to-day operations, but I firmly believe that any system architecture must be thoughtfully designed for efficient operations, otherwise the Ops team will go mad in no time.

Over the years, I’ve been learning a lot from the book VMware Operations Management by  Iwan E1 Rahabok, which covers everything related to vROps, Aria Operations, and now VCF Operations.

Veeam Backup & Replication on Linux v13 [Beta]

I have finally found some spare time and I decided to test Veeam Backup & Replication on Linux v13 [Beta] in my home lab. It is BETA, so it is good to test it and be prepared for the final release, even anything can change before the final release is available. 

There is clear information that update and upgrade into newer versions will not be possible, but I'm really curious how Veeam transition from Windows to Linux is doing. 

Anyway, let's test it and get the feeling about the Veeam future with Linux based systems.

PureStorage has 150TB DirectFlash Modules

I have just realized that PureStorage has 150TB DirectFlash Modules. 

That got me thinking. 

Flash capacity is increasing year by year. What are performance/capacity ratios?

The reason I'm thinking about it is that poor Tech Designer (like me) need some rule-of-thumb numbers for capacity/performance planning and sizing.

Virtual NIC Link Speed - is it really speed?

This will be a quick blog post, prompted by another question I received about VMware virtual NIC link speed. In this blog post I’d like to demonstrate that the virtual link speed shown in operating systems is merely a reported value and not an actual limit on throughput.

I have two Linux Mint (Debian based) systems mlin01 and mlin02 virtualized in VMware vSphere 8.0.3. Each system has VMXNET3 NIC. Both virtual machines are hosted on the same ESXi host, so they are not constraint by physical network. Let's test network bandwidth between these two systems with iperf.

How to troubleshoot virtual disk high latencies in VMware Virtual Machine

In VMware vSphere environments, even the most critical business applications are often virtualized. Occasionally, application owners may report high disk latency issues. However, disk I/O latency can be a complex topic because it depends on several factors, such as the size of the I/O operations, whether the I/O is a read or a write and in which ratio, and of course, the performance of the underlying storage subsystem. 

One of the most challenging aspects of any storage troubleshooting is understanding what size of I/O workload is being generated by the virtual machine. Storage workload I/O size is the significant factor to response time. There are different response times for 4 KB I/O and 1 MB I/O. Here are examples from my vSAN ESA performance testing.

• 32k IO, 100% read, 100% random - Read Latency: 2.03 ms Write Latency: 0.00 ms
• 32k IO, 100% write, 100% random - Read Latency: 0.00 ms Write Latency: 1.74 ms
• 32k IO, 70% read - 30% write, 100% random - Read Latency: 1.55 ms Write Latency: 1.99 ms
• 1024k IO, 100% read, 100% sequential - Read Latency: 6.38 ms Write Latency: 0.00 ms
• 1024k IO, 100% write, 100% sequential - Read Latency: 0.00 ms Write Latency: 8.30 ms
• 1024k IO, 70% read - 30% write, 100% sequential - Read Latency: 5.38 ms Write Latency: 8.68 ms

You can see that response times vary based on storage profile. However, application owners very often do not know what is the storage profile of their application workload and just complain that storage is slow. 

As one storage expert (I think it was Howard Marks [1] [2]) once said, there are only two types of storage performance - good enough and not good enough.

Fortunately, on an ESXi host, we have a useful tool called vscsiStats. We have to know on which ESXi host VM is running and ssh into that particular ESXi host.

The vSCSI monitoring procedure is

1. List all running virtual machines on particular ESXi host, and identify our Virtual Machine and its identifiers (worldGroupID and Virtual SCSI Disk handleID)
2. Start vSCSI statistics collection in ESXi host
3. Collect vSCSI statistics histogram data
4. Stop vSCSI statistics collection

The procedure is documented in VMware KB - Using vscsiStats to collect IO and Latency stats on Virtual Disks 

Let's test it in lab.

Are you looking for VMware SRM and cannot find it?

Here is what happened with VMware Site Recovery Manager. It was repackaged into VMware Live Recovery.

UPDATE 2025-07-07: Nice VCF 9 Disaster Recovery / Business Continuity (DRBC) solution overview is explained at VMware official blog post "VMware Cloud Foundation Recovery Improvements with VMware Live Recovery".

What is VMware Live Recovery?

VMware Live Recovery is the latest version of disaster and ransomware recovery from VMware. It combines VMware Live Site Recovery (previously Site Recovery Manager) with VMware Live Cyber Recovery (previously VMware Cloud Disaster Recovery) under a single shared management console and a single license. Customers can protect applications and data from modern ransomware and other disasters across VMware Cloud Foundation environments on-premises and in public clouds with flexible licensing for changing business needs and threats. 

For more details see the VMware Live Recovery FAQ and the VMware Live Recovery resource page.

In this blog post I will just copy information from Site Recovery Manager FAQ PDF, because that's what old good on-prem SRM is, and it is good to have it in HTML form in case Broadcom/VMware PDF from what ever reasons disapeer.

Here you have it ...

How to run IPERF on ESXi host?

iperf is great tool to test network throughput.There is iperf3 in ESXi host, but there are restrictions and you cannot run it.

There is the trick.

First of all, you have to disable ESXi advanced option execInstalledOnly=0. This enables you to run executable binaries which were not preinstalled by VMware.

Second step is to make a copy of iperf binary, because installed version os estricted and cannot be run.

The third step is to disable ESXi firewall to allow cross ESXi communication between iperf client and iperf server.

After finishing performance testing, you should clean ESXi environment

• delete your copy of iperf
• re-enable ESXi firewall to allow only required tcp/udp ports for ESXi services
  
• re-enable ESXi advanced option (execInstalledOnly=1) to keep ESXi hypervisor secure by default
  

ESXi Commands 

# Allow execute binaries which are not part of base installation

localcli system settings advanced set -o /User/execInstalledOnly -i 0
 

# Make a copy of iperf

cp /usr/lib/vmware/vsan/bin/iperf3 /usr/lib/vmware/vsan/bin/iperf3.copy

 

# Disable firewall

esxcli network firewall set --enabled false

# Run iperf server
./iperf3.copy -s -B 192.168.123.22

# Run iperf client (typically in another ESXi host than iperf server is running)
./iperf3.copy -c -B 192.168.123.22

After iperf benchmarking you should enable firewall and disallow execution of binaries which are not part of base installation

 

# Cleaning
rm /usr/lib/vmware/vsan/bin/iperf3.copy

esxcli network firewall set --enabled true
localcli system settings advanced set -o /User/execInstalledOnly -i 1

 

Test Jumbo Frames (MTU 9000) between ESXi hosts

When we want to enable Jumbo-Fames on VMware vSphere, it must be enabled on

• physical switches
• virtual switches - VMware Distributed Switch (VDS) or VMware Standard Switch (VSS)
• VMkernel interfaces where you would like to use Jumbo-Frames (typically NFS, iSCSI, NVMeoF, vSAN, vMotion)

Let's assume it is configured by network and vSphere administrators and we want to validate that vMotion network between two ESXi hosts supports Jumbo Frames. Let's say we have these two ESXi hosts

• ESX11 has IP address 10.160.22.111 on vMotion vmk interface within vMotion TCP/IP stack. 
• ESX12 has IP address 10.160.22.112 on vMotion vmk interface within vMotion TCP/IP stack.

Ping is a good network diagnostic tool for this purpose. It uses ICMP protocol within IP protocol. So, what is the maximum size of IP/ICMP packet? With Jumbo Frame we have 9000 bytes for Layer 2 (Ethernet Frame) payload and within Ethernet Frame is IP packet cariing ICMP packet with Echo Request. So, here is the calculation. 

9000 (MTU) - 20 (IP header) - 8 (ICMP header) = 8972 bytes

 

Let's do ping with 8972 bytes payload and with flag -d (fragmentation disabled).

[root@esx11:~] ping -I vmk1 -S vmotion -s 8972 -d 10.160.22.112
PING 10.160.22.112 (10.160.22.112): 8972 data bytes
8980 bytes from 10.160.22.112: icmp_seq=0 ttl=64 time=0.770 ms
8980 bytes from 10.160.22.112: icmp_seq=1 ttl=64 time=0.637 ms
8980 bytes from 10.160.22.112: icmp_seq=2 ttl=64 time=0.719 ms

We can see succesful test of large ICMP packets without fragmentation. We validated that ICMP packets with size 8972 bytes can be transfered over the network without fragmentation. That's the indication that Jumbo Frames (MTU 9000) are enabled end-to-end.

Now let's try to cary ICMP packets with size 8973 bytes.

[root@esx11:~] ping -I vmk1 -S vmotion -s 8973 -d 10.160.22.112
PING 10.160.22.112 (10.160.22.112): 8973 data bytes
sendto() failed (Message too long)
sendto() failed (Message too long)
sendto() failed (Message too long)

We can see that ICMP packets with size 8973 bytes cannot be transfered over the network without fragmentation. This is expected behavior and it proves that we know what we do.

Business Continuity and Disaster Recovery Terminology

Almost 10 years ago, I gave a presentation at the local VMware User Group (VMUG) meeting in Prague, Czechia, on Metro Cluster High Availability and SRM Disaster Recovery. The slide deck is available here on Slideshare. I highly recommend reviewing the slide deck, as it clearly explains the fundamental concepts and terminology of Business Continuity and Disaster Recovery (BCDR), along with the VMware technologies used to plan, design, and implement effective BCDR solutions.

Let me briefly outline the key BCDR concepts, documents, and terms below.

• Business Continuity Basic Concepts
• Resilience (High Availability)
• Recovery (Disaster Recovery)
• Mitigation (Disaster Avoidance)
• Business Continuity Essential Documents
• BIA (Business Impact Analysis) is essential document for any Business Continuity Initiative
  
• Risk Management Plan (Prevention and mitigation)
• Contingency Plan (Response and recovery)
• Business Continuity Basic Terms
• RPO (Recovery Point Objective) – Data level
• RTO (Recovery Time Objective) – Infrastructure level
• WRT (Work Recovery Time) – Application level
• MTD (Maximum Tolerable Downtime) = RTO + WRT – Business level
  

RPO (Recovery Point Objective) and RTO (Recovery Time Objective) are the most known terms in Business Continuity and Disaster Recovery world and I hope all IT professionals know at least these two terms. However, the repetition is the mother of wisdom so let's repeat what RPO and RTO are. The picture is worth 1,000 words so look at the picture bellow.

RPO and RTO

 

RPO - The maximum acceptable amount of data loss measured in time. In other words, "How far back in time can we afford to go in our backups?"

RTO - The maximum acceptable time to restore systems and services (aka infrastructure) after a disaster.

WRT - The time needed after systems are restored (post-RTO) to make applications fully operational (e.g., data validation, restarting services). It’s a subset of MTD and follows RTO.

MTD - How much time does our business accept before the company is back and running after a disaster. In other words, it is the total time a business can be unavailable before causing irrecoverable damage or significant impact.

Easy, right? Not really, Disaster Recovery projects are, based on my experience, the most complex project in IT infrastructure. 

Are you ready to support your business after the disaster and guarantee business continuity? Are you sure? 

 

Do you test your BCDR regularly? How often? Once a year? Twice a year? Or even more? 

 

Feel free to leave a comment.

How to check your public IP address from FreeBSD or Linux console

Web service available at https://ifconfig.me/ will expose the client IP address. This is useful when you do not know your public IP address as you are behind the NAT (Network Address Translation) in some public Wi-Fi access point or even in your home behing CGNAT (Carrier-Grade NAT) very often used by Internet Service Providers using IPv4.

How we can leverage it from FreeBSD? 

It is easy. Just use old good fetch command available on FreeBSD by default to retrieve a file by Uniform Resource Locator (URL).

Oneliner:

fetch -qo - https://ifconfig.me | grep "ip_addr:"

How we can leverage it from Linux? 

It is easy. I use Debian and there is old good wget command available by default. It can be used to retrieve a file by Uniform Resource Locator (URL) similarly to fetch on FreeBSD.

Oneliner:

wget -o /dev/null -O - https://ifconfig.me | grep "ip_addr:"

 

Another even simpler possibility is to use dig

Use this command

dig +short myip.opendns.com @resolver1.opendns.com

dig to get public IP

Conclusion

In Unix-like operating systems is very easy to leverage standard tools to get job done.

What is and how to identify ESX OSDATA partition?

I have two VMware vSphere home labs with relatively old hardware (10+ years old). Even I have upgraded the old hardware to use local SATA SSD disks or even NVMe disks the old systems does not support boot from NVMe. That's the reason I still boot my homelab ESXi hosts from USB flash disks, even it is highly recommended to not use USB flash disks or SD cards as boot media for ESXi 7 and later. The reason for this recommendation is to keep boot medium in healthy state even ESXi 7 and later writes more frequently to boot media then earlier ESXi versions. If you want to know more details about reasons for this recommendation, please read my older post vSphere 7 - ESXi boot media partition layout changes.

During the ESXi installation you can choose USB disk as boot medium and local NVMe disk as a disk for ESX OSDATA partition, which will be used to write ESXi system data like config backups, logs (ESX, vSAN, NSX), vSAN traces, core dumps, VM tools (directory /productLocker), and other ephemeral or persistent system files.

How it looks like and how you can identify ESX OSDATA partition on local disks?

vCenter / vSphere Client

Interestingly enough, if you check disk partition layout form vCenter, you will see this 128 GB partition identified as "Legacy MBR". See the screenshot below.

ESX OSDATA Partition details in vSphere Client - Legacy MBR (128 GB)

 

This is, IMHO, little bit misleading.

ESXi host Web Management

You can connect directly into particular ESXi host and check the partition diagram in ESXi host web management. See the screenshot below.

ESXi OSDATA partition details about NVMe disk - VMFSL (128 GB)

There is no information about "Legacy MBR" partition but it is identified as "VMFSL" partition.  So again, there is not explicit information about ESXi OSDATA partition but it is identified as VMFSL file system. What is VMFSL? VMFSL in ESXi stands for VMware File System Logical — it's not an officially documented term in VMware's main literature, but it refers to an internal filesystem layer used by ESXi for managing system partitions and services that aren’t exposed like traditional datastores (VMFS or NFS).

It is, IMHO better information than vSphere Client provides via vCenter.

ESXi Shell

We can also list partition tables in ESXi Shell.

 [root@esx22:~] partedUtil getptbl /vmfs/devices/disks/eui.0000000001000000e4d25cc9a0325001  
 gpt  
 62260 255 63 1000215216  
 7 2048 268435455 4EB2EA3978554790A79EFAE495E21F8D vmfsl 0  
 8 268435456 1000212480 AA31E02A400F11DB9590000C2911D1B8 vmfs 0  

It is also listed as VMFSL partition, but we can also see partition GUID.

The GUID (Globally Unique Identifier) that identifies the partition type. This GUID is unique to the partition and corresponds to a specific partition format (e.g., VMFS6, OSDATA, etc.).

4EB2EA3978554790A79EFAE495E21F8D is OSDATA

AA31E02A400F11DB9590000C2911D1B8 is VMFS6

And that's it. Now you know all details about ESX OSDATA partition and how to identify it.

 

 

SAP HANA on VMware

Recently Broadcom announced that vSAN ESA support for SAP HANA was introduced. Erik Rieger is Broadcom's Principal SAP Global Technical Alliance Manager and Architect. Erik was the guest in Duncan Epping's podcast Unexplored Territory and you can listen their discussion on all major podcast platforms. The episode name is "#094 - Discussing SAP HANA support for vSAN ESA 8.x with Erik Rieger!"

 

For technical details you can check following SAP and VMware resources.

SAP note 3406060

SAP HANA on VMware vSphere 8 and vSAN 8 for details - https://me.sap.com/notes/3406060

SAP HANA and VMware support pages

https://help.sap.com/docs/SUPPORT_CONTENT/virtualization/3362185786.html

SAP HANA on HCI powered by vSAN
vSphere and SAP HANA best practices - https://www.vmware.com/docs/vmw-sap-hana-vmware-vsphere-best-practices