How to check I/O device on VMware HCL

VMware has Hardware Compatibility List of supported I/O devices is available here
https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io

VMware HCL for I/O devices

The best identification of I/O device is VID (Vendor ID), DID (Device ID), SVID (Sub-Vendor ID), SSID (Sub-Device ID). VID, DID, SVID and SSID can be simply entered into VMware HCL and you will find if it is supported and what capabilities have been tested. You can also find supported firmware and driver.

The get these identifiers you have to log in to ESXi via SSH and use command "vmkchdev -l".  This command shows VID:DID SVID:SSID for PCI devices and you can use grep to filter just VMware NICs (aka vmnic)

vmkchdev -l | grep vmnic

You should get similar output

[dpasek@esx01:~] vmkchdev -l | grep vmnic

0000:02:00.0 14e4:1657 103c:22be vmkernel vmnic0

0000:02:00.1 14e4:1657 103c:22be vmkernel vmnic1

0000:02:00.2 14e4:1657 103c:22be vmkernel vmnic2

0000:02:00.3 14e4:1657 103c:22be vmkernel vmnic3

0000:05:00.0 14e4:168e 103c:339d vmkernel vmnic4

0000:05:00.1 14e4:168e 103c:339d vmkernel vmnic5

0000:88:00.0 14e4:168e 103c:339d vmkernel vmnic6

0000:88:00.1 14e4:168e 103c:339d vmkernel vmnic7

So, in case of vmknic4 there is
·       VID:DID SVID:SSID
·       14e4:168e 103c:339d


The same applies to HBAs and disk controllers.  For HBA and local disk controllers use

vmkchdev -l | grep vmhba

This is the output from my Intel NUC at my home lab

[root@esx02:~] vmkchdev -l | more

0000:00:00.0 8086:0a04 8086:2054 vmkernel 

0000:00:02.0 8086:0a26 8086:2054 vmkernel 

0000:00:03.0 8086:0a0c 8086:2054 vmkernel 

0000:00:14.0 8086:9c31 8086:2054 vmkernel vmhba32

0000:00:16.0 8086:9c3a 8086:2054 vmkernel 

0000:00:19.0 8086:1559 8086:2054 vmkernel vmnic0

0000:00:1b.0 8086:9c20 8086:2054 vmkernel 

0000:00:1d.0 8086:9c26 8086:2054 vmkernel 

0000:00:1f.0 8086:9c43 8086:2054 vmkernel 

0000:00:1f.2 8086:9c03 8086:2054 vmkernel vmhba0

0000:00:1f.3 8086:9c22 8086:2054 vmkernel 

vmhba0 is local disk controller
vmhba32 is USB storage controller
vmnic0 is network interface

Hope this helps.

Using esxtop to identify storage performance issues for ESX / ESXi

ESXi performance are exposing to administrators through vSphere Clients. You can see real-time performance statistics which are collected in 5 minute intervals where each interval consists of fifteen 20 seconds samples. It is obvious that 20 second sample is pretty large for storage performance where we are working in mili or even micro second scale.

20 seconds contains 20,000 milliseconds

Let's be clear here, we will never have full visibility but smaller monitoring sample will give as better clue what is really happening inside the system. It is similar to microscope device.

The smallest monitoring samples can be achieved by ESXi utility ESXTOP. The default esxtop delay between monitoring points (sample) is 5 seconds. However, it can be lowered up to 2 seconds by parameter -d 2

For real analytics the esxtop data must be exprted to external file. In esxtop terminology it is batch mode and it is achieved by parameter -b 

Another important factor is what statistics (metrics) we are going to collect. The best is to collect all statistics because during performance analytics you have to correlate multiple values against each other. It is achieved by parameter -a

And last parameter is -n which defines how many iterations you want to perform in batch mode. So in example below we will have 30 iterations with delay between each other 2 seconds. So we will do total monitoring for 60 seconds.

esxtop -b -a -d 2 -n 30 > esxtop-data.csv

For all esxtop parameters see screenshot below.

 [root@esx11:~] esxtop -h  
 usage: esxtop [-h] [-v] [-b] [-l] [-s] [-a] [-c config file] [-R vm-support-dir-path]   
         [-d delay] [-n iterations]  
        [-export-entity entity-file] [-import-entity entity-file]   
        -h prints this help menu.  
        -v prints version.  
        -b enables batch mode.  
        -l locks the esxtop objects to those available in the first snapshot.  
        -s enables secure mode.  
        -a show all statistics.  
        -c sets the esxtop configuration file, which by default is .esxtop60rc  
        -R enables replay mode.  
        -d sets the delay between updates in seconds.  
        -n runs esxtop for only n iterations. Use "-n infinity" to run esxtop forever.  
        -----Experimental Features-------------  
        -export-entity writes the entity ids into a file, which can be modified  
         to select interesting entities.  
        -import-entity reads the file of selected entities. If this opion   
         is used, esxtop only shows the data for the selected entities.  

It is important to know, that esxtop will give you significantly more statistics you can see in vSphere Client level. That's another important benefit of esxtop. But each benefit has also some drawbacks or impact. The impact is, that single esxtop output line can have several thousands statistic counters. For example ESXi 6.0 host with just 2 running VMs in my home lab has 27,314 counters. My customer's product ESXi host has over 330,000 counters! So the output file can be pretty large in case you run it for 24 hours. Count on it.

In the file are very interesting counters. Following counters for physical disk devices are the most interesting

### Reponse times
Average Guest MilliSec/Command
Average Kernel MilliSec/Command
Average Queue MilliSec/Command
Average Queue MilliSec/Read
Average Driver MilliSec/Command
Average Driver MilliSec/Write
### Queue
Adapter Q Depth
### IOPS
Reads/sec
Writes/sec
Commands/sec
### MB/s
MBytes Read/sec
MBytes Written/sec"
### Split commands
Split Commands/sec
### SCSI Reservations
Reserves/sec
Failed Reserves/sec
Conflicts/sec
### Failures
Failed Commands/sec
Failed Reads/sec
Failed Writes/sec
Failed Bytes Read/sec
Failed Bytes Written/sec
Aborts/sec
Resets/sec

Some of above counters are not available in vSphere Client but the big benefit is that esxtop will give you data in 2 second interval which is much better granularity.

I hear your questions - So what now? How to analyze esxtop output file?
Well, you can replay it back in esxtop or you can use any of following tools

• VisualEsxtop
• perfmon
• excel
• esxplot

To be honest, none of tools above fulfilled my requirements therefore I'm writing my own Python script for esxtop output analysis.

The Python script is available on github here.

How to convert thick zeroed virtual disk to thin and save storage space

Last week I've been notified by my colleague about long term VMware vSphere issue described in VMware KB 2048016. The issue is that vSphere Data Protection restores a thin-provisioned disk as a thick-provisioned disk. This sounds like relatively big operational impact. However after reading VMware KB I've explained to my colleague that this is not typical issue or bug but it is rather expected behavior of VMware's CBT (change block tracking) technology and VADP (VMware API for data protection) framework. It's important to mention that it should behave like that only when you do initial full backup of thin provisioned VM which has never been powered on. In other words, if VM was ever powered on before initial backup procedure you shouldn't experience this issue.

After above explanation another logical question appeared.  

"How you can convert thick zeroed virtual disk to thin" ... when you experience weird behavior explained above and you restore your originally thin provisioned VM as thick VM. The obvious objection is to save storage space again leveraging VM thin provisioning.

My answer was to use "storage vMotion" which allows change ot vDisk type during migration. But just after my quick answer I realized there can be another potential issue with storage vMotion. If you use VAAI capable storage then storage vMotion is offloaded to the storage and it may not reclaim even zeroed vDisk space. This behavior and resolution is describe in VMware KB 2004155 named as "Storage vMotion to thin disk does not reclaim null blocks". The workaround mentioned in KB is to use offline method leveraging vmkfstools. If you want live storage migration (conversion) without downtime you would need another datastore with different block size. You can do it with legacy VMFS3 filesystem.

I decided to do a test to prove real storage vMotion behavior and know the truth. Everything else would be just speculations. Therefore I’ve test storage vMotion behavior of thick to thin migration and space reclamation in my lab where I have vSphere/ESX 5.5 and EqualLogic storage with VAAI support. To be honest the result surprised me in positive way. It seems that svMotion can save the space even I do svMotion between datastore with the same block and there is VAAI enabled.

You can see thick eager zeroed 40GB disk in screenshot below ...

Provisioned size is 44GB because VM has 4 GB RAM and therefore 4 GB swap file on VMFS.
Used storage is 40GB.

After live storage vMotion with conversion to Thin it saved the space. 
Used storage is just 22 GB.  You can see result at screenshot below ...

So I have just verified that svMotion can do what you need without downtime. And I don’t even need to migrated between datastores with different block size.

It was tested on ESX 5.5, EqualLogic firmware 6.x., and VMFS5 datastores created on thin provisioned LUNs by EqualLogic.  Storage thin provisioning is absolutely transparent to vSphere so this should not have impact on vSphere thin provisioning.


I know that this is just a workaround to the problem of VADP restore of never powered on VMDK but it works.  It converts thick to thin and is able to reclaim unused (zeroed) space insight virtual disks.

Conclusion:
vSphere 5.5 storage vMotion can convert thick VM to thin even between datastores having same block size. At least in tested configuration. Good to know. If someone else can do the test in your environment just leave the comment. It can be beneficial for others.

Difference between ESXi Shell time-outs

ESXi Advanced Settings have two timeout parameters to manage ESXi Shell timeout:

• UserVars.ESXiShellTimeOut 
• UserVars.ESXiShellInteractiveTimeOut

Both parameters are by default set to 0 which means the time-outs are disabled. However, it is good practice to set these timeouts as it has a positive impact on security.

But what values should be set there?
What is the difference between ESXiShellTimeOut and ESXiShellInteractiveTimeOut?

Let's start with description of both parameters available in ESXi hosts.

UserVars.ESXiShellTimeOut
Key: UserVars.ESXiShellTimeOut
Description: Time before automatically disabling local and remote shell access (in seconds, 0 disables). Takes effect after the services are restarted.
Value: 0
Default: 0
Read only: No
Range: 0 ≤ x ≤ 86400

UserVars.ESXiShellInteractiveTimeOut
Key: UserVars.ESXiShellInteractiveTimeOut
Description: Idle time before an interactive shell is automatically logged out (in seconds, 0 disables). Takes effect only for newly logged in sessions.
Value: 0
Default: 0
Read only: No
Range: 0 ≤ x ≤ 86400

Is it clear? No? So, here is a more human-readable explanation ...

On ESXi host are following two shells:

• ESXi Shell (Local Troubleshooting Shell available in Local Console) 
• SSH Service (Remote Shell)

What does UserVars.ESXiShellTimeOut setting?
If you want to troubleshoot ESXi locally or remotely, you have to enable services ESXi Shell or SSH. We are talking about ESXi services highlighted in the figure below.

ESXi Shell and SSH services

Advanced setting parameter UserVars.ESXiShellTimeOut is the time in seconds how long are these services (ESXi Shell and SSH) running when explicitly started by vSphere admin for troubleshooting. The default value of UserVars.ESXiShellTimeOut is 0, therefore these services will stay up and running until vSphere admin stop them manually. There is a potential risk that vSphere admin will forget to stop these services after troubleshooting and it would have a negative impact on security as these services should be running only for troubleshooting. That's the reason for best practice to set this timeout setting to some reasonable time for troubleshooting and to stop services automatically after configure time. I think the reasonable time for troubleshooting is for example 4 hours which is 14400 seconds.

What does UserVars.ESXiShellInteractiveTimeOut setting?
If you have started ESXi services ESXi Shell or SSH Service you can do local or remote troubleshooting. Advanced settings parameter UserVars.ESXiShellInteractiveTimeOut configures timeout from the active (interactive) session. In other words, if you do a troubleshooting via SSH session (SSH service must be running) this setting will close the active SSH session in case of idle (no interaction) is higher then UserVars.ESXiShellInteractiveTimeOut value. So, it will close your interactive SSH session but SSH service will stay up and running so you can reconnect and continue with troubleshooting. The same principle applies to local troubleshooting with ESXi Shell.

Do these settings have any impact on Direct Console UI?

No. None of the above two advanced settings have any impact on Direct Console UI (aka DCUI, ESXi Text-based Menu available in Local Console). DCUI is not considered as a shell.

Hope the explanation above help you to understand the difference. If not, please leave the comment below the blog post.

VMware vSphere 5 Memory Management and Monitoring

Do you think you fully understand VMware vSphere ESXi memory management?
Compare your understanding with memory diagram at  VMware KB 2017642.

Now another question. Do you still think you are able to exactly know how much memory is used and how much is available? Do you? It is very important to know that this task is complex in any operating system because of lot of memory virtualization layers, memory garbage algorithm, caching, buffering, etc .... therefore nobody is able to know exact numbers. Of course you can monitor ESX memory usage but that is always estimated number.

Real memory over allocation and potential memory issue can be monitored by several mechanisms

• Running VMs ballooning - because ballooning starts only when there is not enough memory
• VMs (ESX) swapping - mainly swap in/out rate higher then 0 because that's the real indication you have memory problem

DELL NPAR and VMware vSphere

DELL NPAR is Network Partitioning of single 10Gb NIC or better to say 10Gb CNA (Converged Network Adapter). NPAR technology is implemented on modern Broadcom and QLogic CNAs which allows to split single physical NIC up to 4 logical NICs. More about NPAR can be found for example here or here.

Please be aware that

• NPAR is not implemented on Intel 10G NIC (X520, X540)
• NPAR is not SR-IOV. More about SR-IOV is here and here.

The biggest NPAR value propositions are

• More logical interfaces partitioned from single interface which appears in the OS as normal PCI-e adapter.
• Switch independent solution. I'll explain what does it mean in the minute.

I have seen several customers complaining about NPAR. NPAR is just another technology and each technology has to be used correctly with respect for what purpose it was invented and designed. I have depicted NPAR architecture in the drawing bellow.

Let's describe the picture. On the picture you can see one physical server with ESXi hypervisor and two CNAs. Each CNA is divided into four logical partitions where each partition act as independent NIC with unique MAC address. You can see two physical wires interconnecting CNA ports with switch ports. Inside each physical wire are four "virtual wires" interconnecting CNA logical interfaces with single physical switch port. That's important!!! Four virtual ports on CNA are connected into single switch port. You can imagine it like four connectors on one side of the wire and just single connector on the other side.

That's not common, right?
The benefit of this architecture is switch independence.
The drawback is that ethernet flows between NPAR interfaces on single CNA port will fail.

So with this information in the mind let's explain NPAR architecture behavior in bigger detail.

Physical switch will never forward Ethernet frame back to the port from which the frame is coming. So, if src-mac and  dst-mac is on the same physical port switch (these are entries in switch mac-address-table) the L2 communication will be broken. That’s standard Ethernet switch behavior.
So what happen in NPAR architecture where are 4 virtual cables (NPAR interfaces with independent MAC addresses) connected into single physical switch port? No communication.

It is shown on picture below.

That’s the reason CISCO has VN-TAG (802.1Qbh) and HP has multi-channel VEPA (802.1Qbg)
These solutions multiplex Ethernet on both sides of the wire.

Note:
I have hands-on experirence with CISCO VN-TAG so I can admit it works correctly but I have never tested HP VEPA.

NPAR is relatively good technology to separate and prioritize Storage and Ethernet traffic on unified (converged) ethernet networks. It can be also used to separate and prioritize L2 traffic. But it will not work if L2 communication between logical NPAR interfaces are required.

Problematic scenarios can be for example following configurations

• vCenter in VM <-> ESX vmkernel management port in the same L2 segment but different portgroups routed through separated NPAR interfaces (uplinks) as depicted above.
• Cisco Nexus 1000v VSM in VM <-> ESX VEM communicate over L2 protocol routed through separated NPAR interfaces.

Hope this helps DELL and VMware community.

Deploying ESXi 5.x using the Scripted Install feature

Unfortunately I had no chance to design and implement automated vSphere deployment for any customer. I tried several automated deployment possibilities in the lab but I have never met the customer with such requirement. That's probably because right now I do vSphere consulting for small country in the middle of Europe where 32 ESX farm is "PRETTY BIG" vSphere environment ;-)
 
Nevertheless, excellent VMware KB article about PXE & KickStart file method of ESXi scripted installation is here.

VMware All Paths Down (aka APD)

All Paths Down (APD), a feature of the VMware ESXi host used in cases where all paths to the VM
go down because of storage failure or administrative error, is properly handled in ESX 5.1 as a
result of feature enhancement performed by VMware. Previously, in ESX versions 5.0 or 4.1, the
host would try continuously to revive the storage links and, as a result, performance would be
impacted for working VMs. A host reboot was required to clear this error.

I was engaged by several customers impacted with APD issue and it was always disaster. If you operate ESX 5.0 and older consider upgrade to ESX 5.1 or even better to ESX 5.5.

VMware ESXi vim-cmd Command: A Quick Tutorial

Very nice blog post on www.doublecloud.org ...

Command lines are very important for system administrors when it comes to automation. Although GUIs are more likely (not always as I’ve seen too many bad ones) to be more intuitive and easier to get started with, sooner or later administrators will use command lines more for better productivity. There are a few command line options in VMware ESXi, among which is the vim-cmd ... READ MORE.

DELL EqualLogic valuable resources on the web

I've got an email from one DELL EqualLogic expert and he has in the mail signature links to very valuable DELL EqualLogic web resources. Here there are:

• EqualLogic Compatibility Matrix
• EqualLogic Configuration Guide
• Rapid EqualLogic Configuration Portal
• EqualLogic Best Practices Whitepapers
• EqualLogic Best Practices ESX

Also see my another blog post DELL Storage useful links.

I'm sharing these links here because I'm sure it is interested for other people.

iSCSI NetGear datastore issues

Yesterday I had a phone call from my neighbor who work as vSphere admin for one local system integrator. He was in the middle of upgrade from vSphere 4.1 to vSphere 5.5 and had a trouble.

He decided to use vSphere 5.5 but not by in place upgrade but as having two environments. The legacy one (vSphere 4.1) and new one (vSphere 5.5). Each environment had their own vCenter and he used one iSCSI datastore connected to both environments as transfer datastore.  He called me because he experienced issues with powering on particular VM stored on transfer datastore and registered on ESXi 5.5 managed by vCenter 5.5. When VM power on was initiated it took some time and the task failed in - he told me - 25%.

I remember  we were discussing some time ago if is better to use vSphere 5.1 or go directly to very new vSphere 5.5. My answer was "it depends" but at  the end we agreed that in small environment is possible to go directly to vSphere 5.5 and accept some risk.  That's the reason why I felt little bit guilty.

As we are neighbors he came to my garden. He smoked several cigarettes probably to organize his thoughts and we were discussing potential root cause and other best practices including migration possibilities. All those general ideas and recommendations were just best practices and hypothesis. At the end we agreed that we have to look at log files to understand what is really happening and what issue he is experiencing.

I have to say I like troubleshooting ...  the first log file to check in such situations is obviously /var/log/vmkernel.log

As he is more Microsoft (GUI) then *nix (CLI) oriented I navigated him over the phone how to enable ssh, login to ESXi  and check the log file.

When we start the command

tail -f /var/log/vmkernel.log 

the troubleshooting was almost done. Lot of SCSI errors were continuously logged into vmkernel.log. SCSI errors included following useful information 

H:0x0 D:0x2 P:0x0  SCSI sense keys: 0x0B 0x24 0x00

Let's translated log file information into human language ... device returns "aborted command" (0x0B) and additional sense code (0x24) is undocumented so it is probably device specific.
However  root cause was obvious ... it is storage related issue. We tried to create directory on affected datastore and it took almost 30 seconds which prove our  assumption of storage issue. Problematic datastore was backed by iSCSI NetGear storage. The same operation in another datastore backed by another storage connected directly by SAS was, of course, immediate.

So I asked him again (we talk about HCL at the beginning general discussion) if he checked HCL and he confirmed he does it but he will double check it. In one hour later he send me a message that storage model is supported but the firmware must be upgraded to work correctly with ESX 5.5

All my "ad-hoc consulting" was done just like quick help to friend of mine so I don't even know what NetGear iSCSI storage my neighbor has but I will ask him and update this post because it can help other people.

Update 10/16/2013:
I have been informed that exact NetGear iSCSI storage model is "NetGear Ready NAS 3100". I checked VMware HCL by my self and at the moment it is supported just for ESX 5.1 with firmware RAIDiator-x86 4.2.21. So I warn my neighbor that even it will work with new firmware this configuration will be unsupported.  Another lesson from this - don't trust anybody and validate everything by your self :-)

So what is the conclusion of this story? Plan, plan and plan again before any vSphere upgrade. Don't check just hardware models on HCL but check also firmwares. Modern hardware and operating systems (including hypervisors) are very software dependent so firmware versions matters.

Two (2) or four (4) socket servers for vSphere infrastructure?

Last week I had interesting discussion with customer subject matter experts and VMware PSO experts about using 2-socket versus 4-socket servers for VMware vSphere infrastructure in IaaS cloud environment. I was impressed how difficult is to persuade infrastructure professionals about 4-socket server benefits in some cases.

Although it seems as pretty easy question it is actually more complex if we analyze it deeper. It is common question from many of my customers and because it is a typical "it depends" answer I've decided to blog about it.

Let's start with some general statements:

• 2-socket servers are designed and used for general business workloads
• 2-socket servers are less expensive
• 4-socket servers are very expensive
• 4-socket servers are designed and used for high performance and mission-critical workloads
• failure of single 4-socket server node in vSphere cluster has bigger impact on capacity 

All these general statements are relative so what is really better for particular environment depends on customer's requirements, expected workload size and chosen hardware platform.

It is important to note that at the time of writing this post there are two types of 4-socket Intel servers in the market. Servers with Intel E7 CPU Family and servers with Intel E5-4600 Family. Comparing the Intel E7-4780 (10 core, 2.4GHz) with an Intel E5-4650 (8 core, 2.7 GHz), you’ll find that the E5 server outperforms against the E7 server in the following benchmarks:

•  CAE
•  SPECfp*_rate_base2006
•  Numerical Weather
•  Financial Services
•  Life Sciences
•  Linpack AVX
•  SPECint*_rate_base2006

The E7 server outperforms the E5 server in the following benchmarks:

•  java* Middleware
•  OLTP Database
•  Enterprise Resource Planning

 CPU family comparison is taken from here.

Intel E7 are designed for mission critical workloads and E5-4600 family for general workloads with big CPU performance requirements. Therefore E7 processors are "very" (I would say more) expensive but price difference between E5-4600 (4-socket) and E5-2600 (2-socket) servers is usually less than 10 or 20 percent but it can vary among different hardware vendors.


Server consolidation is the most common use case of server virtualization. Before any server consolidation it is highly recommended to do "ASIS" capacity monitoring and "TOBE" capacity planning with consolidation scenarios. There are plenty of different tools for such exercise. For example VMware Capacity Planner, Platespin Recon, CIRBA, etc. However if we design green field environment and there is not legacy environment which can be monitored we have to define expected average and maximum VM. So, let's define our average and maximum workload we are planning to virtualize in single VM.

Let's assume our typical VM is configured as

• 1 vCPU consuming 333 MHz CPU
• 1 vCPU consuming 1/3 of one CPU Thread
• 4GB RAM

and maximal VM (aka monster VM) is configured as

• 8 vCPU
• 128 GB RAM

So what physical servers to use for virtualization in such environment? E7 CPUs are significantly more expensive therefore let's compare 2-socket servers with E5-2665 (2.4GHz) and 4-socket server with E5-4640 (2.4GHz). So here are our server options in detail.

4S-SERVER: Single 4-socket server E5-4640 (8 cores) has 32 cores and 64 CPU Threads (logical CPU) in case hyper-threading is enabled. Total CPU capacity is 76.8 GHz. From RAM perspective it can accommodate 48 DIMMs (4 sockets x 4 channels x 3 DIMMs).

2S-SERVER: Single 2-socket server E5-2665 (8 cores) has 16 cores and 32 CPU Threads (logical CPUs) in case hyper-threading is enabled. Total CPU capacity is 38.4 GHz. From RAM perspective it can accommodate 24 DIMMs (2 sockets x 4 channels x 3 DIMMs).

So in first look 8 x 4-socket server has same compute capacity and performance as 16 x 2-socket servers, right? 4-socket server can accommodate double number of DIMMs, so total RAM capacity of 8 x 4-socket servers and 16 x 2-socket servers is also the same. It is 768GB RAM in 16GB DIMMs or 1536GB RAM in 32GB DIMMs.

If we will use vSphere Cluster with 8 x 4S-SERVER or 16 x 2S-SERVER we have same total raw capacity and performance but 16 x 2S-SERVERs will beet 8 x 4S-SERVERx in real available capacity because in case of single server fail we will lose just 1/16 of capacity and performance unlike 1/8 of capacity.

Is it true or not?

Yes, from memory perspective.
Yes and sometimes No, from CPU performance.

Let's concentrate on CPU performance and compare CPU performance of DELL 2-socket server  M620 (E5-2665/2.4GHz) with DELL 4-socket server M820 (E5-4640/2.4GHz). We all know that 1MHz on two different systems doesn't represent comparable performance, so the question is how to compare CPU performance. The answer is CPU normalization. Good, good ... but wait how we can normalize CPU performance. The answer is CPU benchmark. Good ... but what benchmark?

Below are listed different benchmark results for single host so based on results we can deeply discuss what system is better for particular environment.  Please note that some benchmark results are not available or published so I use results from similar systems. I believe it's enough accuracy for our comparison.

2S-SERVER: M620 (E5-2665/2.4GHz)

• SPECint_rate2006: 611
• SPECfp_rate2006: 467
• VMmark: 5.1  Calculation: 2x M620 VMmark (E5-2680) has 10.20 @ 10 Tiles = 10.2 / 2
• SPECvirt_sc2013: 236.15 Calculation: 1x HP DL380p G8 SPECvirt_sc2013 (E5-2690) 472.3 @ 27 =472.3 / 2

4S-SERVER: M820 (E5-4640/2.4GHz)

• SPECint_rate2006: 1080
• SPECfp_rate2006: 811
• VMmark: 10.175 Calculation: 2x HP DL560 VMmark (E5-4650) 20.35 @ 18 Tiles =20.35 / 2
• SPECvirt_sc2013: 454.05 Calculation: 1x HP DL560 SPECvirt_sc2013 (E5-4650) 908.1 @ 53 =908.1 / 2

Note 1: DELL 4S-SERVER VMware results are not published so I use results for HP DL560 servers

Note 2: Some SPECvirt_sc2013 results are not available for VMware vSphere so I use results for Redhat KVM.

Based on results above I prepared performance benchmark comparison table:

Benchmark	2x 2S	1x 4S	4S against 2S
SPECint	1222	1080	88.38%
SPECfp	934	811	86.83%
VMmark	10.2	10.175	99.75%
SPECvirt_sc	472.3	454.05	96.13%

So what does it mean? I explain it by way of 2-socket servers are better for RAW mathematical operations (integer and flouting point) but for more real live workloads 4-socket servers have generally same performance like 2-socket servers and more cores/threads per single system.

BTW: It seems to me that CPU performance normalization based on SPECint and/or SPECfp is not fair to 4-socket servers. That's what Platespin Recon use for CPU normalization.

We can say that there is not 1MHz performance difference between our 2S-SERVER and 4S-SERVER. So what is the advantage of 4-Socket servers based on E5-4600 CPUs? The CPU performance is not only about MHz performance but also about CPU scheduling (aka multi-threading). The 4S-SERVER advantage is bigger count of  logical CPUs which has positive impact on co-scheduling vCPUs of vSMP virtual machines. Although vCPU co-scheduling has been dramatically improved in ESX 3.0 some co-scheduling is required anyway. Co-scheduling executes a set of threads or processes at the same time to achieve high performance. Because multiple cooperating threads or processes frequently synchronize with each other, not executing them concurrently would only increase the latency of synchronization. For more information  about co-scheduling look at https://communities.vmware.com/docs/DOC-4960

In our example we are planning to have monster VMs with 8 vCPUs so 64 logical CPUs in 4S-SERVER offers potentially more scheduling opportunities against 32 logical CPUs in 2S-SERVER. As far as I know virtualization benchmarks tiles (tiles are group of VMs) usually have up to 2 vCPUs so I think co-scheduling issue is not covered by these benchmarks.

So final decision depends on expected number of  monster VMs which can affect real performance of workloads inside these monster VMs. CPU performance overloading can be monitored by ESX metric %RDY (vCPU READY but pCPU not) and co-scheduling execution delays by metric %CSTP (vCPU stoped because of co-scheduling). Recommended thresholds are discussed here but every environment has different quality requirements so your thresholds can be different and it depends what quality SLA you want to offer and what type of application you want to run on top of virtual infrastructure.

Anyway, co-scheduling of monster VMs is serious issue for IaaS Cloud Providers because it is really hard to explain customers that less vCPUs can give them paradoxically better CPU performance. I call this phenomenon "VIRTUALIZATION PARADOX".   

The final hardware selection or recommendation is always dependent on justification of vSphere Architect who has to carefully analyze specific requirements and constraints in particular environment and reasonably justify selected decision. We should remember there can be other requirements favoring a specific platform. Example of "other" requirement (sometime the constraint) can be the situation when blade servers want to be used. In 2-socket blade servers is usually very difficult and sometimes even impossible to avoid single point of failure of NIC/HBA/CNA adapter. 4-socket server are usually full height (dual slot) and therefore I/O cards are doubled ... but that's another topic.

NFS or Fibre Channel Storage for VMware vSphere?

Final decision depends what do you want to get from your storage. Check out my newly uploaded presentation on SlideShare:  http://www.slideshare.net/davidpasek/design-decision-nfsversusfcstorage-v03 where I'm trying to compare both options with special requirements from real customer engagement.

If you have any storage preference, experience or question please feel free to speak up in the comments.

Quick troubleshooting of ESX and 10Gb Broadcom NeXtreme II negotiated only to 1Gb

I have just realized that my vmnic(s) in one DELL blade server M620 (let's call him BLADE1) is connected only at 1Gb speed even I have 10Gb NIC(s) connected to Force10 IOA blade module(s). It should be connected at 10Gb and another blade  (let's call him BLADE2) with the same config is really connected at 10Gb speed.

So quick troubleshooting ... we have to find where is the difference

Let's go step by step ...

1. NIC ports on ESX vSwitch in BLADE1 are configured to  use auto negotiation so no problem here
2. Ports on Force 10 IOA are also configured for auto negotiation and configuration is consistent across all ports in switch modules so that's not a problem.
3. ESX builds are the same on both blade servers.
4. What are NIC firmwares? On BLADE1 there is 7.2.14 and on BLADE2 7.6.15

Bingo!!! Let's upgrade NIC firmwares on BLADE1 and check if this was the root cause of the problem ...

ESXi Advanced Settings for NetApp NFS

Here are NetApp 
Net.TcpipHeapSize=30
Net.TcpipHeapMax=120
NFS.MaxVolumes=64
NFS.HeartbeatMaxFailures=10
NFS.HeartbeatFrequency=12
NFS.HeartbeatTimeout=5

Enabled SIOC or if you don't have Entrprise+ license set NFS.MaxQueueDepth=64, 32 or 16 based on storage workload and utilization

DELL EqualLogic general recommendations for VMware vSphere ESXi

Bellow are eight major recommendations for DELL EqualLogic implementation with vSphere ESXi:

1. Delayed ACK disabled
2. LRO disabled
3. If using Round Robin, set IOs to 3
4. If  ENTERPRISE or ENTERPRISE+ license and ONLY with Enterprise/Enterprise+  install MEM 1.1.2
5. Extend login_timeout to 60 seconds.
6. Don’t have multiple VMDKs on a single Virtual SCSI controller.  (major cause of latency alerts)
7. Align partitions on 64K boundary
8. Format with 64K cluster (allocation unit) size with Windows

ESX host remote syslog configuration

For remote CLI you can use vMA or vCLI. Here is the example how to configure ESX host (10.10.1.71) to send logs remotely to syslog server listening on IP address 10.10.4.72 on tcp port 514. 

First of all we have to instruct ESX where is the syslog server.

esxcli -s 10.10.1.71 -u root -p Passw0rd. system syslog config set --loghost='tcp://10.10.4.72:514'

Then syslog service on ESX host must be restarted to accept configuration change.

esxcli -s 10.10.1.71 -u root -p Passw0rd. system syslog reload

ESX firewall must be reconfigured to allow syslog traffic

esxcli -s 10.10.1.71 -u root -p Passw0rd. network firewall ruleset set --ruleset-id=syslog --enabled=true
esxcli -s 10.10.1.71 -u root -p Passw0rd. network firewall refresh

If you want to test or troubleshoot syslog logging you can login to ESX host and use logger command to send test message to syslog.

logger "Test syslog over network"

How to change default path selection policy for particular storage array?

Sometimes the firmware in storage array has some problems and you have to "downgrade" functionality to achieve operable system. That's sometimes happen for some ALUA storage systems where Round Robin path policy or Fixed path policy (aka FIXED) should work but doesn't because of firmware issue.

So relatively simple solution is to switch back from more advanced round robin policy to legacy - but properly functioning -  Most Recently Used path policy (aka MRU) normally used for active/passive storage systems. 

Note: Please be aware that some storage vendors saying they have active/active storage even they have not. Usually and probably more precisely they call it "dual-active storage" which is not same as active/active. Maybe I should write another post about this topic.

You can change Path Selection Policy by several ways and as always the best option depends on your specific requirements and constrains.

However, if you have only one  instance of some storage type connected to your ESX hosts you can simply change default path selection policy for this particular SATP type. Let's assume you have some LSI storage.

Below is simple esxcli command how to do it ...

esxcli storage nmp satp set --default-psp=VMW_PSP_MRU --satp=VMW_SATP_LSI

... and then your default PSP for VMW_SATP_LSI is now VMW_PSP_MRU

One thing you must be aware … if you in the past explicitly changed any devices (disks) to another path selection policy then default PSP will not change even you have another default path policy. There is not esxcli mechanism how to change devices back to accept  default PSP for particular SATP type. Only solution is to edit /etc/vmware/esx.conf

All previous explicit changes are written in /etc/vmware/esx.conf  so it is pretty simple to find it and remove these lines form config file.  I silently assume you do such operations in maintenance mode so after ESX reboot all your paths for your devices will follow default SATP path selection policy. 

BTW: That’s why I generally don’t recommend to change PSP for particular device when not necessary. Sometimes it is necessary for example for RDM’s participating in MSCS cluster. But usually it is abused by admins and implementation engineers.  I strongly believe it is always better to set default PSP to behave as required.