Network benchmark (iperf) of 10Gb Data Center Interconnect

I wanted to test 10Gb ethernet link I have got as data center interconnect between two datacenters. I generally do not trust anything I have not tested.

If you want test something, it is important to have good testing methodology and toolset.

Toolset

OS: FreeBSD 14.2 is IMHO the best x86-64 operating system in terms of networking. Your mileage may vary.

Network benchmark testing tool: IPERF (iperf2) is weel known tool to benchmark network performance and bandwidth.

Hypervisor: VMware ESXi 8.0.3 is the best in class hypervisor to test varios virtual machines

Methodology

I have use two Virtual Machines. At the end I will test network throughput between two VMs, where one VM is in each end of network link (DC Interconnect). However before the final test (Test 4) of DC interconnect throughput, I will test network throughput (Test 1) within the same VM to test localhost throughput, (Test 2) between VMs within single hypervisor (ESXi) host to avoid using physical network, (Test 3) VMs across two hypervisors (ESXi) within single VLAN in one datacenter to test local L2 throughput.

Results

Test 1: Network throughput within the same VM to test localhost throughput

VMware Virtual Machines have following hardware specification:

• 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
• 8 GB RAM
• 8 GB vDisk
  
• 1 vNIC (vmxnet) 

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -t 60

Network Throughput: 75.4Gb/s - 83Gb/s

CPU usage on server/client: 23%

MEM usage on server/client: ~500MB 

 

2 iperf connections / 4 CPU Threads (-P 2)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 2 -t 60

Network Throughput: 90.8Gb/s - 92Gb/s

CPU usage on server/client: 28%

MEM usage on server/client: ~500MB

4 iperf connections / 8 CPU Threads (-P 4)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 4 -t 60

Network Throughput: 88.5Gb/s - 89.1Gb/s

CPU usage on server/client: 29%

MEM usage on server/client: ~500MB 

 

8 iperf connections / 16 CPU Threads (-P 8)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 8 -t 60

Network Throughput: 91.6Gb/s - 95.3Gb/s

CPU usage on server/client: 30%

MEM usage on server/client: ~500MB 

 

Tests with Higher TCP Windows Size (800kB)

 

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -w 800k -t 60

Network Throughput: 69.8Gb/s - 81.0Gb/s

CPU usage on server/client: 28%

MEM usage on server/client: ~500MB

 

2 iperf connections / 4 CPU Threads (-P 2 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 2 -w 800k -t 60

Network Throughput: 69.8Gb/s - 69.9Gb/s

CPU usage on server/client: 28%

MEM usage on server/client: ~500MB

 

4 iperf connections / 8 CPU Threads (-P 4 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 2 -w 800k -t 60

Network Throughput: 69.2Gb/s - 70.0Gb/s

CPU usage on server/client: 28%

MEM usage on server/client: ~500MB

 

8 iperf connections / 16 CPU Threads (-P 8 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c localhost -P 8 -w 800k -t 60

Network Throughput: 72.6Gb/s - 74.0Gb/s

CPU usage on server/client: 28%

MEM usage on server/client: ~500MB

Test 2: Network throughput between VMs within hypervisor (no physical network)

VMware Virtual Machines with following hardware specification:

• 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
• 8 GB RAM
• 8 GB vDisk
  
• 1 vNIC (vmxnet) 

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -t 60

Network Throughput: 6.5Gb/s - 6.71Gb/s

CPU usage on server: 70%

CPU usage on client: 30-50%

MEM usage on server/client: ~500MB 

 

2 iperf connections / 4 CPU Threads (-P 2)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 2 -t 60

Network Throughput: 8.42Gb/s -8.62Gb/s

CPU usage on server: ~33%

CPU usage on client: ~30%

MEM usage on server/client: ~500MB

 

4 iperf connections / 8 CPU Threads (-P 4)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 4 -t 60

Network Throughput: 19.5Gb/s - 20.2Gb/s

CPU usage on server: 85%

CPU usage on client: 48%

MEM usage on server/client: ~500MB

 

 

8 iperf connections / 16 CPU Threads (-P 8)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 8 -t 60

Network Throughput: 17.1Gb/s - 18.4Gb/s

CPU usage on server: ~85%

CPU usage on client: ~30%

MEM usage on server/client: ~500MB

 

Tests with Higher TCP Windows Size (800kB)

 

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -w 800k -t 60

Network Throughput: 6.57Gb/s - 6.77Gb/s

CPU usage on server: 24%

CPU usage on client: 24%

MEM usage on server/client: ~500MB

 

2 iperf connections / 4 CPU Threads (-P 2 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 2 -w 800k -t 60

Network Throughput: 7.96Gb/s -8.0Gb/s

CPU usage on server: ~30%

CPU usage on client: ~28%

MEM usage on server/client: ~500MB

 

4 iperf connections / 8 CPU Threads (-P 4 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 4 -w 800k -t 60

Network Throughput: 15.8Gb/s -18.8Gb/s

CPU usage on server: ~85%

CPU usage on client: ~40%

MEM usage on server/client: ~500MB

 

8 iperf connections / 16 CPU Threads (-P 8 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 8 -w 800k -t 60

Network Throughput: 19.1Gb/s - 22.8Gb/s

CPU usage on server: ~98%

CPU usage on client: ~30%

MEM usage on server/client: ~500MB

 

Test 3: Network throughput between VMs across two hypervisors within VLAN (25Gb switch ports) in one DC

VMware Virtual Machines have following hardware specification:

• 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
• 8 GB RAM
• 8 GB vDisk
  
• 1 vNIC (vmxnet) - connected to 25Gb physical switch ports

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -t 60

Network Throughput: 6.1Gb/s - 6.34Gb/s

CPU usage on server: 23%

CPU usage on client: 17%

MEM usage on server/client: ~500MB 

 

2 iperf connections / 4 CPU Threads (-P 2)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 2 -t 60

Network Throughput: 9.31Gb/s -10.8Gb/s

CPU usage on server: ~43%

CPU usage on client: ~30%

MEM usage on server/client: ~500MB

 

4 iperf connections / 8 CPU Threads (-P 4)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 4 -t 60

Network Throughput: 19.5Gb/s - 20.2Gb/s

CPU usage on server: 85%

CPU usage on client: 48%

MEM usage on server/client: ~500MB

8 iperf connections / 16 CPU Threads (-P 8)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 8 -t 60

Network Throughput: 17.1Gb/s - 18.4Gb/s

CPU usage on server: ~80%

CPU usage on client: ~50%

MEM usage on server/client: ~500MB

Tests with Higher TCP Windows Size (800kB)

 

1 iperf connection / 2 CPU Threads (-P not specified, default setting in use -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -w 800k -t 60

Network Throughput: 6.11Gb/s - 6.37Gb/s

CPU usage on server: 16%

CPU usage on client: 22%

MEM usage on server/client: ~500MB

 

2 iperf connections / 4 CPU Threads (-P 2 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 2 -w 800k -t 60

Network Throughput: 9.81Gb/s -10.9Gb/s

CPU usage on server: ~39%

CPU usage on client: ~25%

MEM usage on server/client: ~500MB

 

4 iperf connections / 8 CPU Threads (-P 4 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 4 -w 800k -t 60

Network Throughput: 16.5Gb/s -19.8Gb/s

CPU usage on server: ~85%

CPU usage on client: ~40%

MEM usage on server/client: ~500MB

 

8 iperf connections / 16 CPU Threads (-P 8 -w 800k)

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 8 -w 800k -t 60

Network Throughput: 17.7Gb/s - 18.2Gb/s

CPU usage on server: ~80%

CPU usage on client: ~50%

MEM usage on server/client: ~500MB

Test 4: Network throughput between VMs across two hypervisors across two interconnected VLANs across two DCs

VMware Virtual Machines have following hardware specification:

• 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
• 8 GB RAM
• 8 GB vDisk
  
• 1 vNIC (vmxnet) 

iperf server command: iperf -s

iperf client comand: iperf -c 10.202.201.6 -P 4 -t 60

Network Throughput: 9.74 Gb/s

Conclusion

Network throughput requires CPU cycles, therefore number of CPU cores matters.

 

iperf client by default uses one connection for generating network traffic where each connection uses 2 vCPUs (hyper-threading threads). In such default configuration I was able to achieve ~6.65 Gb/s in VM with at least 2 vCPU, which is not enough to test 10Gb/s datacenter interconnect. 

 

By using parameter -P 4, four parallel iperf client connections are initiated where each iperf connection uses 2 vCPUs (hyper-threading threads), therefore it can leverage all 8 vCPUs we have in testing VM.

 

By using parameter -P 8 in VM, eight parallel iperf client connections are initiated where each iperf client connection uses 2 vCPUs (hyper-threading threads), therefore it can leverage 16 vCPUs, but us we use only 8 vCPUs in our test machine, it only make bigger stress on existing CPUs and therfore it can have negative impact on overall network throughput.

 

The best practice is to use -P 4 for iperf client on machine with 8 CPUs as iperf client connections can be balanced across all 8 available CPUs. If you have more CPUs available, parameter -P should be the half of number of available CPUs.

• 1 CPUs VM can achieve network traffic up to 5.83 Gb/s. During such network traffic, CPU is fully used (100% usage) and maximum single iperf connection throughput of 6.65 Gb/s cannot be acieved duw to CPU constraint.
• 2 CPUs VM can achieve network traffic up to 6.65 Gb/s. During such network traffic, CPU is fully used (100% usage).
• 4 CPUs VM with -P 2 is necessary to achieve network traffic up to 10 Gb/s.
• 8 CPUs VM with -P 4 is necessary to achieve network traffic over 10 Gb/s. These 8 threads can generate 20 Gb/s which is good enough to test my 10Gb/s data center interconnect. 

Another iperf parametr which in theory could improve network throughput is the parameter -w which defines TCP Window Size. iperf by default uses TCP Window Size between 32kB and 64kB. By increasing TCP Window Size to 800kB (-w 800k) can slightly improve (~10%) performance during higher stress on CPU (-P 8 = 8 Processes / 16 Threads) across VMs. However, higher TCP Window Size (-w 800k) has negative impact (in some cases almost 30%) on localhost network throughput performance.

What real network throughput I have measured during this testing excercise? 

Localhost network throughput is significantly higher than network throughput across Virtual Machines or accross physical network and servers. We can achieve between 75 Gb/s and 95 Gb/s on Localhost. Network traffic does not need across virtual and physical hardware. It is logical that virtual and physical hardware introduces some bottlenecks.

 

Network throughput between VMs within single hypervisor can achieve 6.5 Gb/s with single process and two threads.  Up to 22.8 Gb/s (eight processes / sixteen threads and higher TCP Windows Size - 800kB) and 20.2 Gb/s with eight processes / sixteen threads and default TCP Windows Size. 

Network throughput between VMs within VLAN (25 Gb switch ports) in one data center can achieve up to 20.2 Gb/s (eight processes / sixteen threads and standard TCP Windows Size).

 

When you would need higher throughput than 20 Gb/s between VMware virtual machines, more CPU cores and special performance tuning of vNIC/vmxnet driver would need to be done. Such performance tunning would be about enabling Jumbo Frames (MTU 9,000, ifconfig_vmx0="inet <IP> netmask <NETMASK> mtu 9000") into guest OS, increasing Network Buffers in FreeBSD kernel (kern.ipc.maxsockbuf, net.inet.tcp.sendspace, net.inet.tcp.recvspace=4194304), Enable TCP Offloading (ifconfig_vmx0="inet <IP> netmask <NETMASK> mtu 9000 txcsum rxcsum tso4 tso6 lro"), Tune Interrupt Moderation, Use Multiple Queues aka RSS (sysctl net.inet.rss.enabled=1, sysctl net.inet.rss.bits=4). Fortunatelly enough, 20 Gb throughput is good enough to test my 10 Gb data center interconnect. 

Network throughput between VMs accros 10 Gb data center interconnect can achieve 9.74 Gb/s (four iperf connections / eight vCPUs in use). 9.74 Gb/s TCP throughput over 10 Gb/s data center ethernet interconnect is acceptable throughput.

VMware PowerCLI (PowerShell) on Linux

VMware PowerCLI is very handy and flexible automation tool allowing automation of almost all VMware features. It is based on Microsoft PowerShell. I do not have any Microsoft Windows system in my home lab but I would like to use Microsoft PowerShell. Fortunately enough, Microsoft PowerShell Core is available for Linux. Here is my latest runbook how to leverage PowerCLI in Linux management workstation leveraging Docker Application packaging.

Install Docker in your Linux Workstation

This is out of scope of this runbook.

Pull official and verified VMware PowerCLI

sudo docker pull vmware/powercli:latest

Pull official and verified Microsoft Powershell

sudo docker pull mcr.microsoft.com/powershell:latest

List container images

sudo docker image ls

How to run container?

Now you can run any of two above powershell/powercli containers interactively (-i) and in allocated pseudo-TTY (-t). 

The -v option in Docker is used to mount volumes, allowing data to persist or be shared between the host and the container.  We can map our Linux host directory /home/dpasek/scripts in to container directory /tmp/scripts This allow us to store our scripts permanently even we exit and remove the PowerShell container.

sudo docker run  -it  CONTAINER-NAME

sudo docker run  -it vmware/powercli 

or

sudo docker run  -it mcr.microsoft.com/powershell

 

VMware PowerCLI container

Using VMware PowerCLI container is easier because it contains PowerShell with PowerCLI prepared by VMware

This command will run VMware PowerCLI and you have everything ready to go.

sudo docker run  -it -v /home/dpasek/scripts:/root/scripts vmware/powerclicore

The -v option in Docker is used to mount volumes, allowing data to persist or be shared between the host and the container. We map our Linux host directory /home/dpasek/scripts in to container directory /tmp/scripts This allow us to store our scripts permanently even we exit or even remove the PowerShell container.

To work with PowerCLI, the following commands are necessary to initialize PowerCLI configuration.

Set-PowerCLIConfiguration -Scope User -ParticipateInCEIP $true
Set-PowerCLIConfiguration -InvalidCertificateAction:ignore

... and now you can connect to vCenter and list VMs

Connect-VIServer -Server <vcenter-server> -User <username> -Password <password>

Get-VM

 

Microsoft PowerShell container

If you want use Microsoft PowerShell container, you can, but you have to install PowerCLI manualy. 

Following command will run Microsoft PowerShell

sudo docker run -it -v /home/dpasek/scripts:/root/scripts mcr.microsoft.com/powershell

Inside Microsoft PowerShell container, you must install PowerCLI manualy

Install-Module -Name VMware.PowerCLI -Scope CurrentUser -Force

You must allow untrusted certificates and enable (or disable) participation in VMware CEIP.

Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false

Set-PowerCLIConfiguration -Scope User -ParticipateInCEIP $true

And now you can connect to vCenter and list VMs

Connect-VIServer -Server <vcenter-server> -User <username> -Password <password>

Get-VM

 

 

 

How to update ESXi with unsupported CPU?

I have old unsupported servers in my lab used for ESXi 8.0.3. In such configuration, you cannot update ESXi by default procedure in GUI.

vSphere Cluster Update doesn't allow remediation

ESXi host shows unsupported CPU

Solution is to allow legacy CPU and update ESXi from shell with esxcli.

Allow legacy CPU

The option allowLegacyCPU is not available in the ESXi GUI (DCUI or vSphere Client). It must be enabled using the ESXi shell or SSH. Bellow are command to allow legacy CPU.

esxcli system settings kernel set -s allowLegacyCPU -v TRUE

You can verify it by command ...

esxcli system settings kernel list | grep allowLegacyCPU

If above procedure fails, the other option is to edit file /bootbank/boot.cfg and add allowLegacyCPU=true to the end of kernelopt line.

In my case, it look like ...

kernelopt=autoPartition=FALSE allowLegacyCPU=true

After modifying /bootbank/boot.cfg, ESXi configuration should be saved to make changes persistent across reboots.

 /sbin/auto-backup.sh

Reboot of ESXi is obviously required to make kernel option active.

reboot

After reboot, you can follow by standard system update procedure by ESXCLI method as documented below.

ESXi update procedure (ESXCLI method)

1. Download appropriate ESXi offline depot. You can find URL of depot in Release Notes of particular ESXi version. You will need Broadcom credentials to download it from Broadcom support site.
2. Upload (leveraging Datastore File Browser, scp, winscp, etc.) ESXi offline depot to some Datastore
• in my case /vmfs/volumes/vsanDatastore/TMP
3. List profiles in ESXi depot
• esxcli software sources profile list -d /vmfs/volumes/vsanDatastore/TMP/VMware-ESXi-8.0U3d-24585383-depot.zip 
4. Update ESXi to particular profile with no hardware warning
• esxcli software profile update -d /vmfs/volumes/vsanDatastore/TMP/VMware-ESXi-8.0U3d-24585383-depot.zip -p ESXi-8.0U3d-24585383-no-tools --no-hardware-warning
5. Reboot ESXi
•   reboot
  

Hope this helps other folks in their home labs with unsupported CPUs.