I wanted to test 10Gb ethernet link I have got as data center interconnect between two datacenters. I generally do not trust anything I have not tested.
If you want test something, it is important to have good testing methodology and toolset.
Toolset
OS: FreeBSD 14.2 is IMHO the best x86-64 operating system in terms of networking. Your mileage may vary.
Network benchmark testing tool: IPERF (iperf2) is weel known tool to benchmark network performance and bandwidth.
Hypervisor: VMware ESXi 8.0.3 is the best in class hypervisor to test varios virtual machines
Methodology
I
have use two Virtual Machines. At the end I will test network
throughput between two VMs, where one VM is in each end of network link
(DC Interconnect). However before the final test (Test 4) of DC interconnect throughput, I will test network throughput (Test 1) within the same VM to test localhost throughput, (Test 2) between VMs within single hypervisor (ESXi) host to avoid using physical network, (Test 3) VMs across two hypervisors (ESXi) within single VLAN in one datacenter to test local L2 throughput.
Results
Test 1: Network throughput within the same VM to test localhost throughput
VMware Virtual Machines have following hardware specification:
- 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
- 8 GB RAM
- 8 GB vDisk
- 1 vNIC (vmxnet)
1 iperf connection / 2 CPU Threads (-P not specified, default setting in use)
iperf server command: iperf -s
iperf client comand: iperf -c localhost -t 60
Network Throughput: 75.4Gb/s - 83Gb/s
CPU usage on server/client: 23%
MEM usage on server/client: ~500MB
2 iperf connections / 4 CPU Threads (-P 2)
iperf server command: iperf -s
iperf client comand: iperf -c localhost -P 2 -t 60
Network Throughput: 90.8Gb/s - 92Gb/s
CPU usage on server/client: 28%
MEM usage on server/client: ~500MB
4 iperf connections / 8 CPU Threads (-P 4)
iperf server command: iperf -s
iperf client comand: iperf -c localhost -P 4 -t 60
Network Throughput: 88.5Gb/s - 89.1Gb/s
CPU usage on server/client: 29%
MEM usage on server/client: ~500MB
8 iperf connections / 16 CPU Threads (-P 8)
iperf server command: iperf -s
iperf client comand: iperf -c localhost -P 8 -t 60
Network Throughput: 91.6Gb/s - 95.3Gb/s
CPU usage on server/client: 30%
MEM usage on server/client: ~500MB
Tests with Higher TCP Windows Size (800kB)
1 iperf connection / 2 CPU Threads (-P not specified, default setting in use -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c localhost -w 800k -t 60
Network Throughput: 69.8Gb/s - 81.0Gb/s
CPU usage on server/client: 28%
MEM usage on server/client: ~500MB
2 iperf connections / 4 CPU Threads (-P 2 -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c localhost -P 2 -w 800k -t 60
Network Throughput: 69.8Gb/s - 69.9Gb/s
CPU usage on server/client: 28%
MEM usage on server/client: ~500MB
4 iperf connections / 8 CPU Threads (-P 4 -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c localhost -P 2 -w 800k -t 60
Network Throughput: 69.2Gb/s - 70.0Gb/s
CPU usage on server/client: 28%
MEM usage on server/client: ~500MB
8 iperf connections / 16 CPU Threads (-P 8 -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c localhost -P 8 -w 800k -t 60
Network Throughput: 72.6Gb/s - 74.0Gb/s
CPU usage on server/client: 28%
MEM usage on server/client: ~500MB
Test 2: Network throughput between VMs within hypervisor (no physical network)
VMware Virtual Machines with following hardware specification:
- 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
- 8 GB RAM
- 8 GB vDisk
- 1 vNIC (vmxnet)
1 iperf connection / 2 CPU Threads (-P not specified, default setting in use)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -t 60
Network Throughput: 6.5Gb/s - 6.71Gb/s
CPU usage on server: 70%
CPU usage on client: 30-50%
MEM usage on server/client: ~500MB
2 iperf connections / 4 CPU Threads (-P 2)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 2 -t 60
Network Throughput: 8.42Gb/s -8.62Gb/s
CPU usage on server: ~33%
CPU usage on client: ~30%
MEM usage on server/client: ~500MB
4 iperf connections / 8 CPU Threads (-P 4)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 4 -t 60
Network Throughput: 19.5Gb/s - 20.2Gb/s
CPU usage on server: 85%
CPU usage on client: 48%
MEM usage on server/client: ~500MB
8 iperf connections / 16 CPU Threads (-P 8)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 8 -t 60
Network Throughput: 17.1Gb/s - 18.4Gb/s
CPU usage on server: ~85%
CPU usage on client: ~30%
MEM usage on server/client: ~500MB

Tests with Higher TCP Windows Size (800kB)
1 iperf connection / 2 CPU Threads (-P not specified, default setting in use -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -w 800k -t 60
Network Throughput: 6.57Gb/s - 6.77Gb/s
CPU usage on server: 24%
CPU usage on client: 24%
MEM usage on server/client: ~500MB
2 iperf connections / 4 CPU Threads (-P 2 -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 2 -w 800k -t 60
Network Throughput: 7.96Gb/s -8.0Gb/s
CPU usage on server: ~30%
CPU usage on client: ~28%
MEM usage on server/client: ~500MB
4 iperf connections / 8 CPU Threads (-P 4 -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 4 -w 800k -t 60
Network Throughput: 15.8Gb/s -18.8Gb/s
CPU usage on server: ~85%
CPU usage on client: ~40%
MEM usage on server/client: ~500MB
8 iperf connections / 16 CPU Threads (-P 8 -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 8 -w 800k -t 60
Network Throughput: 19.1Gb/s - 22.8Gb/s
CPU usage on server: ~98%
CPU usage on client: ~30%
MEM usage on server/client: ~500MB
Test 3: Network throughput between VMs across two hypervisors within VLAN (25Gb switch ports) in one DC
VMware Virtual Machines have following hardware specification:
- 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
- 8 GB RAM
- 8 GB vDisk
- 1 vNIC (vmxnet) - connected to 25Gb physical switch ports
1 iperf connection / 2 CPU Threads (-P not specified, default setting in use)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -t 60
Network Throughput: 6.1Gb/s - 6.34Gb/s
CPU usage on server: 23%
CPU usage on client: 17%
MEM usage on server/client: ~500MB
2 iperf connections / 4 CPU Threads (-P 2)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 2 -t 60
Network Throughput: 9.31Gb/s -10.8Gb/s
CPU usage on server: ~43%
CPU usage on client: ~30%
MEM usage on server/client: ~500MB
4 iperf connections / 8 CPU Threads (-P 4)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 4 -t 60
Network Throughput: 19.5Gb/s - 20.2Gb/s
CPU usage on server: 85%
CPU usage on client: 48%
MEM usage on server/client: ~500MB
8 iperf connections / 16 CPU Threads (-P 8)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 8 -t 60
Network Throughput: 17.1Gb/s - 18.4Gb/s
CPU usage on server: ~80%
CPU usage on client: ~50%
MEM usage on server/client: ~500MB
Tests with Higher TCP Windows Size (800kB)
1 iperf connection / 2 CPU Threads (-P not specified, default setting in use -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -w 800k -t 60
Network Throughput: 6.11Gb/s - 6.37Gb/s
CPU usage on server: 16%
CPU usage on client: 22%
MEM usage on server/client: ~500MB
2 iperf connections / 4 CPU Threads (-P 2 -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 2 -w 800k -t 60
Network Throughput: 9.81Gb/s -10.9Gb/s
CPU usage on server: ~39%
CPU usage on client: ~25%
MEM usage on server/client: ~500MB
4 iperf connections / 8 CPU Threads (-P 4 -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 4 -w 800k -t 60
Network Throughput: 16.5Gb/s -19.8Gb/s
CPU usage on server: ~85%
CPU usage on client: ~40%
MEM usage on server/client: ~500MB
8 iperf connections / 16 CPU Threads (-P 8 -w 800k)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 8 -w 800k -t 60
Network Throughput: 17.7Gb/s - 18.2Gb/s
CPU usage on server: ~80%
CPU usage on client: ~50%
MEM usage on server/client: ~500MB
Test 4: Network throughput between VMs across two hypervisors across two interconnected VLANs across two DCs
VMware Virtual Machines have following hardware specification:
- 8 vCPU (INTEL XEON GOLD 6544Y @ 3.6 Ghz)
- 8 GB RAM
- 8 GB vDisk
- 1 vNIC (vmxnet)
iperf server command: iperf -s
iperf client comand: iperf -c 10.202.201.6 -P 4 -t 60
Network Throughput: 9.74 Gb/s
Conclusion
Network throughput requires CPU cycles, therefore number of CPU cores matters.
iperf
client by default uses one connection for generating network traffic
where each connection uses 2 vCPUs (hyper-threading threads). In such
default configuration I was able to achieve ~6.65 Gb/s in VM with at least 2 vCPU, which is not enough to test 10Gb/s datacenter interconnect.
By using parameter -P 4,
four parallel iperf client connections are initiated where each iperf
connection uses 2 vCPUs (hyper-threading threads), therefore it can
leverage all 8 vCPUs we have in testing VM.
By using parameter -P 8 in VM,
eight parallel iperf client connections are initiated where each iperf
client connection uses 2 vCPUs (hyper-threading threads), therefore it
can leverage 16 vCPUs, but us we use only 8 vCPUs in our test machine,
it only make bigger stress on existing CPUs and therfore it can have
negative impact on overall network throughput.
The
best practice is to use -P 4 for iperf client on machine with 8 CPUs as
iperf client connections can be balanced across all 8 available CPUs. If you have more CPUs available, parameter -P should be the half of number of available CPUs.
- 1 CPUs VM can achieve network traffic up to 5.83 Gb/s.
During such network traffic, CPU is fully used (100% usage) and maximum
single iperf connection throughput of 6.65 Gb/s cannot be acieved duw
to CPU constraint.
- 2 CPUs VM can achieve network traffic up to 6.65 Gb/s. During such network traffic, CPU is fully used (100% usage).
- 4 CPUs VM with -P 2 is necessary to achieve network traffic up to 10 Gb/s.
- 8 CPUs VM with -P 4 is necessary to achieve network traffic over 10 Gb/s. These 8 threads can generate 20 Gb/s which is good enough to test my 10Gb/s data center interconnect.
Another iperf parametr which in theory could improve network throughput is the parameter -w which defines TCP Window Size.
iperf by default uses TCP Window Size between 32kB and 64kB. By
increasing TCP Window Size to 800kB (-w 800k) can slightly improve
(~10%) performance during higher stress on CPU (-P 8 = 8 Processes / 16
Threads) across VMs. However, higher TCP Window Size (-w 800k) has
negative impact (in some cases almost 30%) on localhost network
throughput performance.
What real network throughput I have measured during this testing excercise?
Localhost network throughput is
significantly higher than network throughput across Virtual Machines or
accross physical network and servers. We can achieve between 75 Gb/s and 95 Gb/s
on Localhost. Network traffic does not need across virtual and physical
hardware. It is logical that virtual and physical hardware introduces
some bottlenecks.
Network throughput between VMs within single hypervisor can achieve 6.5 Gb/s with single process and two threads. Up to 22.8 Gb/s (eight processes / sixteen threads and higher TCP Windows Size - 800kB) and 20.2 Gb/s with eight processes / sixteen threads and default TCP Windows Size.
Network throughput between VMs within VLAN (25 Gb switch ports) in one data center can achieve up to 20.2 Gb/s (eight processes / sixteen threads and standard TCP Windows Size).
When you would need higher throughput than 20 Gb/s
between VMware virtual machines, more CPU cores and special performance
tuning of vNIC/vmxnet driver would need to be done. Such performance
tunning would be about enabling Jumbo Frames (MTU 9,000, ifconfig_vmx0="inet <IP> netmask <NETMASK> mtu 9000") into guest OS, increasing Network Buffers in FreeBSD kernel (kern.ipc.maxsockbuf, net.inet.tcp.sendspace, net.inet.tcp.recvspace=4194304), Enable TCP Offloading (ifconfig_vmx0="inet <IP> netmask <NETMASK> mtu 9000 txcsum rxcsum tso4 tso6 lro"), Tune Interrupt Moderation, Use Multiple Queues aka RSS (sysctl net.inet.rss.enabled=1, sysctl net.inet.rss.bits=4). Fortunatelly enough, 20 Gb throughput is good enough to test my 10 Gb data center interconnect.
Network throughput between VMs accros 10 Gb data center interconnect can achieve 9.74 Gb/s
(four iperf connections / eight vCPUs in use). 9.74 Gb/s TCP throughput
over 10 Gb/s data center ethernet interconnect is acceptable
throughput.