VMware VVOLs scalability

I'm personally a big fan of VMware Virtual Volumes concept. If you are not familiar with VVOLs check this blog post with the recording of VMworld session and read VMware KB Understanding Virtual Volumes (VVols) in VMware vSphere 6.0

We all know that the devil is always in details. The same is true with VVOLs. VMware prepared the conceptual framework but implementation always depends on storage vendors thus it varies around storage products.

Recently, I have had VVOLs discussion with one of my customers and he was claiming that their particular storage vendor supports a very small number of VVOLs. That discussion inspired me to do some research.

Please, note that numbers below are valid at the moment of writing this article. You should always check current status with your particular storage vendor.

Vendor / Storage Array	Maximum VVOLs / Snapshots or Clones
DELL / Compellent SC 8000	2,000 / TBD
EMC / Unity 300	9,000 / TBD
EMC / Unity 400	9,000 / TBD
EMC / Unity 500	13,500 / TBD
EMC / Unity 600	30,000 / TBD
EMC / VMAX 3	64,000 / TBD
HPE / 3PAR	I have been told that it depends on specific model and firmware versions but it is roughly about 10,000 (the info gathered at November 2018)
Hitachi / VSP G200	2,000 / 100,000
Hitachi / VSP G400	4,000 / 100,000
Hitachi / VSP G600	4,000 / 100,000
Hitachi / VSP G800	16,000 / 100,000
Hitachi / VSP G1000	64,000 / 1,000,000

Numbers above are very important because single VM have minimally 3 VVOLs (home, data, swap) and usually even more (snapshot) or more data disks. If you will assume 10 VVOls for single VM you will end up with just 200 VMs on Dell Compellent or Hitachi VSP G200. On the other hand, EMC Unity 600 would give you up to 3,000 VMs which is not bad and enterprise storage systems (EMC VMAX and Hitachi G1000) would give you up to 6,400 VMs which is IMHO very good scalability.

So as always, it really depends on what storage system do you have or planning to buy.

If you know numbers for other storage systems, please share it in comments below this blog post.

Keywords: vvol, vvols, virtual volumes 

Metro Cluster High Availability or SRM Disaster Recovery?

Several years I continuously try to explain my customers that metro cluster is not disaster recovery. I have finally found some time and summarize my thoughts into slide deck which I published on SlideShare. I'm planning to present it at Czech VMUG local meeting on 6 December this year. More info about this particular Czech VMUG event is here.

The goal of my presentation is to explain the difference between multi site high availability (aka metro cluster) and disaster recovery. General concepts are same for any products but presentation is obviously more tailored for specific VMware products and technologies.

You can look at presentation here on SlideShare ...

Metro Cluster High Availability or SRM Disaster Recovery? from David Pasek

It would be great to see you at the event if you will be in the town. But in the meantime don't hesitate to write any comment or feedback here and we can have good discussion as there are still two months till the event.

BTW: Kudos to Stanislav Jurena @stan_jurena who already did several reviews and gave me some comments and feedback before first public release.  

ESXi : How to mask storage device causing some issues

I have heard about the issue with ESXi 6 Update 2 and HP 3PAR storage where VVOLs are enabled. I have been told that the issue is caused by issuing unsupported SCSI command to PE LUN (256). PE stands for Protocol Endpoint and it is VVOL technical LUN for data path between ESXi and remote storage system.

Observed symptoms:

• ESX 6 Update 2 – issues (ESXi disconnects from vCenter, console is very slow)
• Hosts may take a long time to reconnect to vCenter after reboot or hosts may enter a "Not Responding" state in vCenter Server
• Storage-related tasks such as HBA rescan may take a very long time to complete
• I have been told that ESX 6 Update 1 doesn't experience such issues (there are entries are in log file but no other symptoms occur)

Below is a snippet from a log file ..

 2016-05-18T11:31:27.319Z cpu1:242967)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.2ff70002ac0150c3" - issuing command 0x43a657470fc0  
 2016-05-18T11:31:27.320Z cpu31:33602)WARNING: NMP: nmpCompleteRetryForPath:352: Retry cmd 0x28 (0x43a657470fc0) to dev "naa.2ff70002ac0150c3" failed on path "vmhba0:C0:T2:L256" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.  
 2016-05-18T11:31:27.320Z cpu31:33602)WARNING: NMP: nmp_PathDetermineFailure:2973: Cmd (0x28) PDL error (0x5/0x25/0x0) - path vmhba0:C0:T2:L256 device naa.2ff70002ac0150c3 - triggering path failover  
 2016-05-18T11:31:27.320Z cpu31:33602)WARNING: NMP: nmpCompleteRetryForPath:382: Logical device "naa.2ff70002ac0150c3": awaiting fast path state update before retrying failed command again.  

Possible workarounds

• ESXi hostd restart helps therefore SSH to ESXi hosts was enabled for quick resolution in case of problem
• LUN masking of LUN 256

UPDATE 2016-09-30: There is most probably another workaround. Changing the Disk.MaxLUN parameter on ESXi Hosts as described in VMware KB 1998

Final solution

• Application of HP 3PAR firmware patch (unfortunately patch is not available for current firmware thus firmware upgrade has to be planned and executed)
• Investigation of root cause why ESXi 6 Update 2 is more sensitive then ESXi 6 Update 1

Immediate steps

• Application of workarounds mentioned above  

HOME LAB EXERCISE
I have tested in my home lab how to mask particular LUN on ESXi host just to be sure I know how to do it.

Below is quick solution for impatient readers.

Let's sat we have following device with following path.

• Device: naa.6589cfc000000bf5e731ffc99ec35186
• Path: vmhba36:C0:T0:L1

LUN Masking
esxcli storage core claimrule add -P MASK_PATH -r 500 -t location -A vmhba36 -C 0 -T 0 -L 1
esxcli storage core claimrule load
esxcli storage core claiming reclaim -d naa.6589cfc000000bf5e731ffc99ec35186

LUN Unmasking
esxcli storage core claimrule remove --rule 500
esxcli storage core claimrule load
esxcli storage core claiming unclaim --type=path --path=vmhba36:C0:T0:L1
esxcli storage core claimrule run

... continue reading for details.

LUN MASKING DETAILS
Exact LUN masking procedure is documented in vSphere 6 Documentation here. It is also documented in these KB articles 1009449 and 1014953.

List storage devices

 [root@esx02:~] esxcli storage core device list  
 naa.6589cfc000000bf5e731ffc99ec35186  
   Display Name: FreeNAS iSCSI Disk (naa.6589cfc000000bf5e731ffc99ec35186)  
   Has Settable Display Name: true  
   Size: 10240  
   Device Type: Direct-Access  
   Multipath Plugin: NMP  
   Devfs Path: /vmfs/devices/disks/naa.6589cfc000000bf5e731ffc99ec35186  
   Vendor: FreeNAS  
   Model: iSCSI Disk  
   Revision: 0123  
   SCSI Level: 6  
   Is Pseudo: false  
   Status: degraded  
   Is RDM Capable: true  
   Is Local: false  
   Is Removable: false  
   Is SSD: true  
   Is VVOL PE: false  
   Is Offline: false  
   Is Perennially Reserved: false  
   Queue Full Sample Size: 0  
   Queue Full Threshold: 0  
   Thin Provisioning Status: yes  
   Attached Filters:  
   VAAI Status: supported  
   Other UIDs: vml.010001000030303530353661386131633830300000695343534920  
   Is Shared Clusterwide: true  
   Is Local SAS Device: false  
   Is SAS: false  
   Is USB: false  
   Is Boot USB Device: false  
   Is Boot Device: false  
   Device Max Queue Depth: 128  
   No of outstanding IOs with competing worlds: 32  
   Drive Type: unknown  
   RAID Level: unknown  
   Number of Physical Drives: unknown  
   Protection Enabled: false  
   PI Activated: false  
   PI Type: 0  
   PI Protection Mask: NO PROTECTION  
   Supported Guard Types: NO GUARD SUPPORT  
   DIX Enabled: false  
   DIX Guard Type: NO GUARD SUPPORT  
   Emulated DIX/DIF Enabled: false
  
 naa.6589cfc000000ac12355fe604028bf21  
   Display Name: FreeNAS iSCSI Disk (naa.6589cfc000000ac12355fe604028bf21)  
   Has Settable Display Name: true  
   Size: 10240  
   Device Type: Direct-Access  
   Multipath Plugin: NMP  
   Devfs Path: /vmfs/devices/disks/naa.6589cfc000000ac12355fe604028bf21  
   Vendor: FreeNAS  
   Model: iSCSI Disk  
   Revision: 0123  
   SCSI Level: 6  
   Is Pseudo: false  
   Status: degraded  
   Is RDM Capable: true  
   Is Local: false  
   Is Removable: false  
   Is SSD: true  
   Is VVOL PE: false  
   Is Offline: false  
   Is Perennially Reserved: false  
   Queue Full Sample Size: 0  
   Queue Full Threshold: 0  
   Thin Provisioning Status: yes  
   Attached Filters:  
   VAAI Status: supported  
   Other UIDs: vml.010002000030303530353661386131633830310000695343534920  
   Is Shared Clusterwide: true  
   Is Local SAS Device: false  
   Is SAS: false  
   Is USB: false  
   Is Boot USB Device: false  
   Is Boot Device: false  
   Device Max Queue Depth: 128  
   No of outstanding IOs with competing worlds: 32  
   Drive Type: unknown  
   RAID Level: unknown  
   Number of Physical Drives: unknown  
   Protection Enabled: false  
   PI Activated: false  
   PI Type: 0  
   PI Protection Mask: NO PROTECTION  
   Supported Guard Types: NO GUARD SUPPORT  
   DIX Enabled: false  
   DIX Guard Type: NO GUARD SUPPORT  
   Emulated DIX/DIF Enabled: false  

So we have two device with following NAA IDs

• naa.6589cfc000000bf5e731ffc99ec35186
• naa.6589cfc000000ac12355fe604028bf21

Now let's list paths of both of my iSCSI devices

[root@esx02:~] esxcli storage nmp path list
iqn.1998-01.com.vmware:esx02-096fde38-00023d000001,iqn.2005-10.org.freenas.ctl:test,t,257-naa.6589cfc000000bf5e731ffc99ec35186
   Runtime Name: vmhba36:C0:T0:L1
   Device: naa.6589cfc000000bf5e731ffc99ec35186
   Device Display Name: FreeNAS iSCSI Disk (naa.6589cfc000000bf5e731ffc99ec35186)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1,TPG_state=AO,RTP_id=3,RTP_health=UP}
   Path Selection Policy Path Config: {current path; rank: 0}

iqn.1998-01.com.vmware:esx02-096fde38-00023d000001,iqn.2005-10.org.freenas.ctl:test,t,257-naa.6589cfc000000ac12355fe604028bf21
   Runtime Name: vmhba36:C0:T0:L2
   Device: naa.6589cfc000000ac12355fe604028bf21
   Device Display Name: FreeNAS iSCSI Disk (naa.6589cfc000000ac12355fe604028bf21)
   Group State: active
   Array Priority: 0
   Storage Array Type Path Config: {TPG_id=1,TPG_state=AO,RTP_id=3,RTP_health=UP}
   Path Selection Policy Path Config: {current path; rank: 0}

Let's mask iSCSI devices exposed as a LUN 1.
So our path we want to mask is vmhba36:C0:T0:L1 and device UID is naa.6589cfc000000bf5e731ffc99ec35186

So let's create masking rule of path above. In this particular case we have just a single path because it is local device. In real environment we have usually multiple paths and all paths should be masked.

 esxcli storage core claimrule add -P MASK_PATH -r 500 -t location -A vmhba36 -C 0 -T 0 -L 1
 esxcli storage core claimrule load  

We can list our claim rules to see the result

 [root@esx02:~] esxcli storage core claimrule list  
 Rule Class  Rule Class  Type    Plugin   Matches                  XCOPY Use Array Reported Values XCOPY Use Multiple Segments XCOPY Max Transfer Size  
 ---------- ----- ------- --------- --------- ---------------------------------------- ------------------------------- --------------------------- -----------------------  
 MP       0 runtime transport NMP    transport=usb                            false            false            0  
 MP       1 runtime transport NMP    transport=sata                           false            false            0  
 MP       2 runtime transport NMP    transport=ide                            false            false            0  
 MP       3 runtime transport NMP    transport=block                           false            false            0  
 MP       4 runtime transport NMP    transport=unknown                          false            false            0  
 MP      101 runtime vendor   MASK_PATH vendor=DELL model=Universal Xport                  false            false            0  
 MP      101 file   vendor   MASK_PATH vendor=DELL model=Universal Xport                  false            false            0  
 MP      500 runtime location  MASK_PATH adapter=vmhba36 channel=0 target=0 lun=1              false            false            0  
 MP      500 file   location  MASK_PATH adapter=vmhba36 channel=0 target=0 lun=1              false            false            0  
 MP     65535 runtime vendor   NMP    vendor=* model=*                          false            false            0  

We can see that new claim rule (500) is in configuration file (/etc/vmware/esx.com) and also loaded in runtime.

However, to really mask our particular device without ESXi host reboot we have to reclaim device

 [root@esx02:~] esxcli storage core claiming reclaim -d naa.6589cfc000000bf5e731ffc99ec35186  

The particular device disappear from ESXi host immediately. ESXi host reboot is not needed.
So we are done. Particular device is not visible to ESXi host anymore.

Note: I was unsuccessful when I was testing LUN masking with local device. Therefore I assume that LUN masking works only with remote disks (iSCSI, Fibre Channel). 

LUN UNMASKING
Just in case you would like to unmask device and use it again here is the procedure.

Let's start with removing claimrules for our previously masked path.

 [root@esx02:~] esxcli storage core claimrule remove --rule 500  
 [root@esx02:~] esxcli storage core claimrule list  
 Rule Class  Rule Class  Type    Plugin   Matches                  XCOPY Use Array Reported Values XCOPY Use Multiple Segments XCOPY Max Transfer Size  
 ---------- ----- ------- --------- --------- ---------------------------------------- ------------------------------- --------------------------- -----------------------  
 MP       0 runtime transport NMP    transport=usb                            false            false            0  
 MP       1 runtime transport NMP    transport=sata                           false            false            0  
 MP       2 runtime transport NMP    transport=ide                            false            false            0  
 MP       3 runtime transport NMP    transport=block                           false            false            0  
 MP       4 runtime transport NMP    transport=unknown                          false            false            0  
 MP      101 runtime vendor   MASK_PATH vendor=DELL model=Universal Xport                  false            false            0  
 MP      101 file   vendor   MASK_PATH vendor=DELL model=Universal Xport                  false            false            0  
 MP      500 runtime location  MASK_PATH adapter=vmhba36 channel=0 target=0 lun=1              false            false            0  
 MP     65535 runtime vendor   NMP    vendor=* model=*                          false            false            0  
 [root@esx02:~]   

You can see that rule is removed from file configuration but it is still running. We have to re-load claimrules from file to runtime.

 [root@esx02:~] esxcli storage core claimrule load  
 [root@esx02:~] esxcli storage core claimrule list  
 Rule Class  Rule Class  Type    Plugin   Matches              XCOPY Use Array Reported Values XCOPY Use Multiple Segments XCOPY Max Transfer Size  
 ---------- ----- ------- --------- --------- --------------------------------- ------------------------------- --------------------------- -----------------------  
 MP       0 runtime transport NMP    transport=usb                        false            false            0  
 MP       1 runtime transport NMP    transport=sata                        false            false            0  
 MP       2 runtime transport NMP    transport=ide                        false            false            0  
 MP       3 runtime transport NMP    transport=block                       false            false            0  
 MP       4 runtime transport NMP    transport=unknown                      false            false            0  
 MP      101 runtime vendor   MASK_PATH vendor=DELL model=Universal Xport              false            false            0  
 MP      101 file   vendor   MASK_PATH vendor=DELL model=Universal Xport              false            false            0  
 MP     65535 runtime vendor   NMP    vendor=* model=*                       false            false            0  
 [root@esx02:~]   

Here we go. Now there is no rule with id 500.

But the device is still not visible and we cannot execute command

esxcli storage core claiming reclaim -d naa.6589cfc000000bf5e731ffc99ec35186

because such device is not visible to ESXi host. We mask it, right? So it is exactly how it should behave.

ESXi host would probably help but can we do it without ESXi host reboot?
The answer is yes we can.
We have to unclaim the path to our device and re-run claim rules.

 esxcli storage core claiming unclaim --type=path --path=vmhba36:C0:T0:L1  
 esxcli storage core claimrule run  

and now we can see both paths to iSCSI LUNs again.

 [root@esx02:~] esxcli storage nmp path list  
 iqn.1998-01.com.vmware:esx02-096fde38-00023d000001,iqn.2005-10.org.freenas.ctl:test,t,257-naa.6589cfc000000bf5e731ffc99ec35186  
   Runtime Name: vmhba36:C0:T0:L1  
   Device: naa.6589cfc000000bf5e731ffc99ec35186  
   Device Display Name: FreeNAS iSCSI Disk (naa.6589cfc000000bf5e731ffc99ec35186)  
   Group State: active  
   Array Priority: 0  
   Storage Array Type Path Config: {TPG_id=1,TPG_state=AO,RTP_id=3,RTP_health=UP}  
   Path Selection Policy Path Config: {current path; rank: 0}  
 iqn.1998-01.com.vmware:esx02-096fde38-00023d000001,iqn.2005-10.org.freenas.ctl:test,t,257-naa.6589cfc000000ac12355fe604028bf21  
   Runtime Name: vmhba36:C0:T0:L2  
   Device: naa.6589cfc000000ac12355fe604028bf21  
   Device Display Name: FreeNAS iSCSI Disk (naa.6589cfc000000ac12355fe604028bf21)  
   Group State: active  
   Array Priority: 0  
   Storage Array Type Path Config: {TPG_id=1,TPG_state=AO,RTP_id=3,RTP_health=UP}  
   Path Selection Policy Path Config: {current path; rank: 0}  

Hope this helps to other vmware users having a need for LUN masking / unmasking.

VMware vSphere SDRS - test plan of SDRS initial placement

VMware vSphere Storage DRS (aka SDRS) stands for Storage Distributed Resource Scheduler. It continuously balances storage space usage and storage I/O load while avoiding resource bottlenecks to meet application service levels.

Lab environment:
5x10GB Datastores formed into Datastore Cluster with SDRS enabled.
It is configured to balance based on storage space usage and also I/O load.

• Storage Space threshold is 1 GB
• I/O latency threshold is kept on default 15 ms.
• Each "empty" datastore has real capacity 9.75 GB where real free capacity is 8.89 GB because 882 MB is used. 

You can see configuration details on screenshot below.  

Capacity of one particular 10 GB datastore is depicted below.

Used space (882 MB) is occupied by following system files (.sf) ...

In VMFS 5 every datastore gets its own hidden files to save the file-system structure.

Test 1

Test description: Does SDRS Initial Placement algorithm take into account VM swap file capacity?

Test prerequisites:

• All 5 datastores in datastore cluster are empty
• That's mean that each datastore has free capacity 8.89 GB
• Provisioned VM doesn't have any RAM reservation

Test steps:

• Deploy Virtual Machine with 4 GB RAM and 8 GB Disk manually (through Web Client)
• Start deployed Virtual Machine
• Observe behavior 

Test expectations:

• I want to test if swap file is considered during SDRS initial placement
• We have only 8.89 GB free space on datatastores therefore if VM swap file is considered new VM with 8GB disk and 4 GB RAM wont be provisioned because we would need 12 GB space on some datastore which is not our case.
• In other words, if provisioning fails then we will proof that SDRS doesn't take VM swap file into account.

Test screenshots:

Deployed Virtual Machine.

VM PowerOn Failure 

Test Result:

• Virtual machine was successfully provisioned and 8GB was decreased from Datastore 5 available space.
• Virtual machine power on action failed because of not enough storage space for 4 GB swap file. This is expected behavior in case that SDRS doesn't take VM swap into account.

Test Summary:

• We have tested that SDRS Initial Placement algorithm does NOT take VM swap file capacity into account.
• Virtual Machine memory (RAM) reservation would have impact on such test because if VM has for example 100% memory reservation it doesn't need any disk space for VM swap.

Test 2

Test description: How SDRS defragmentation is efficient when datastore cluster is running out of storage space?

Test prerequisites:

• 4 datastores in datastore cluster are almost full
• 1 datastore (Datastore4) has 7.58 GB free space
• In one datastore (Datastore5) we have virtual machine (test1_big) having 8GB disk

Test steps:

• Clone Virtual Machine (test1_big) to datstore cluster (through Web Client)
• Observe behavior 

Test expectations:

• SDRS will free up Datastore4 to have enough space for clone of virtual machine (test1_big)
• Provisioning of virtual machine clone will be successful 

Test screenshots:

Before SDRS defragmentation

After SDRS defragmentation and clone provisioning

Test Result:

• SDRS freed up Datastore4 as expected
• Provisioning of virtual machine clone FAILED because of insufficient disk space on Datastore4. 
• That's unexpected behavior because Datastore4 is empty (thanks to SDRS defragmentation) and another machine with same configuration was successfully provisioned on Datastore5.

Test Summary:

• SDRS successfully freed up the only datastore where virtual machine clone can be placed but VM clone deployment started before storage vMotion finished therefore clone provisioning failed.
• SDRS defragmentation works but there can be some cases when initial placement fails even the storage was freed up and there will be free continuous space in some datastore after defragmentation.
• It is important to understand how VM provisioning to datastore cluster really works. Datastore Cluster is nothing else then the group of single datastores where SDRS is "just" a scheduler on top of Datastore Cluster. You can imagine a scheduler as a placement engine which prepare placement recommendations for initial placement and continuous balancing. That means that other software component (C# Client, Web Client, PowerCLI, vRealize Automation, vCloud Director, etc) is responsible for initial placement provisioning and SDRS give them recommendations where is the best place to put a new storage objects (vmdk file or VM config file).
• In other words, Initial VM provisioning doesn’t have nothing to do with SDRS initial placement. VM initial provisioning process is managed by vSphere Client, vRA, vRO, PowerCLI or other software component over vSphere API. SDRS is just a placement engine gives recommendation where is the best place at the moment when is asked for recommendations. Provisioning process selects one particular SDRS recommendation and continue with provisioning (API method ApplyStorageDrsRecommendation_Task). However, in the mean time there can be some other software doing VM provisioning and selected datastore can be filled by somebody else. There is always some probability for vm provisioning failure and it is exactly where good vSphere / Storage design has crucial role to decrease probability of provisioning failure. 

Test 3

Test description: How is SDRS initial placement balancing among different datastores?

Test prerequisites:

• Storage Space threshold is 1 GB
• I/O latency threshold is kept on default 15 ms.
• Each "empty" datastore has real capacity 9.75 GB where real free capacity is 8.89 GB because 882 MB is used. 
• Usage of PowerCLI script to provision multiple VMs. PowerCLI script is available here.

Test steps:

• Run PowerCLI script to generate 50 virtual machines with following specification (1 vCPU, 512 MB RAM, 1GB Disk - thick) in to datastore cluster with SDRS enabled.
• Observe behavior

Test expectations:

• We have datastore cluster with 5 datastores each having 8.89 GB (9,103 MB) available storage.
• We are deploying VMs with 1000 MB each.
• It is deployed in not power on state - so swap file doesn't need to be considered. 
• Therefore we would expect to end up with 45 VMs balanced in round robin fashion across 5 datastores.  

Test screenshots:

Single datastore capacity

PowerCLI Automated Provisioning.

Datastore free space after automatic sequential provisioning

Test Result:

• 45 VMs was successfully provisioned and 46th-50th VM failed because of "Insufficient disk space on datastore 'Datastore1'." This was expected behavior.
• Following VMs are provisioned on datastores

• Datastore 1: TEST-05, TEST-06, TEST-15, TEST-20, TEST-21, TEST-30, TEST-31, TEST-40, TEST-45 

• Datastore 2: TEST-04, TEST-10, TEST-11, TEST-19, TEST-25, TEST-26, TEST-35, TEST-36, TEST-44

• Datastore 3: TEST-03, TEST-09, TEST-14, TEST-18, TEST-24, TEST-29, TEST-34, TEST-39, TEST-43

• Datastore 4: TEST-02, TEST-08, TEST-13, TEST-17, TEST-23, TEST-28, TEST-33, TEST-38, TEST-42

• Datastore 5: TEST-01, TEST-07, TEST-12, TEST-16, TEST-22, TEST-27, TEST-32, TEST-37, TEST-41

Test Summary: Test passed as expected. Only few details are worth to mention.

• I would expect VMs evenly distributed across datastores.  Recall that we are using artificial sequence provisioning of 1GB vDisks per VM. I would expect VMs TEST-01, TEST-06, TEST-11, TEST-16, TEST-21, TEST-26, TEST-31, TEST-36, TEST-41 on Datastore 5. And similar VM numbering on other datastores. But at the end of the day it doesn't seems to be a big deal.
• Please, note that I observed that different provisioning runs can end-up with slightly different machine placement. I have suspicious that it is because other factors (I/O load, storage usage trend) then are also considered in SDRS algorithm.
• Datastore free space is 103 MB on all datastores. Recall that we have Storage Space threshold set to 1 GB. That's expected behavior. Storage Space threshold is just a threshold (soft limit) used by SDRS for balancing and defragment. 

Test 4

Test description: Will be new VM provisioned to the datastore with the biggest frees space?

Test prerequisites:

• Storage Space threshold is 1 GB
• I/O latency threshold is kept on default 15 ms.
• One datastore (Datastore1) is "empty" has real capacity 9.5 GB where real free capacity is 8.64 GB because 882 MB is used. 
• One datastore (Datastore2) has 5.71 GB free capacity.
• All other datastores (Datastore3, Datastore4, Datastore5) are almost full having only 848 MB empty.

Test steps:

• Usage of vSphere Web Client to provision one VM with 2GB disk into Datastore Cluster.
• Observe behavior. We are interested where new VM will be placed.

Test expectations:

• We expect that new VM disk will be placed on Datastore1 because there is the biggest free (available) space.

Test screenshots:

Datastore cluster capacity before VM provisioning.

Datastore cluster capacity after VM provisioning.

Test Result:

• New virtual machine was provisioned into Datastore1 where was the bigest available storage capacity.

Test Summary: Initial placement behaves as expected. New VM is placed to the datastore with less used space. However, we should be aware that this test was done just for single VM provisioning. Multiple VM provisioning can behaves differently because of other SDRS calculation factors (I/O load, capacity usage trend) and also because of particular provisioning workflow and exact timing when SDRS recommendation is called and when datastore space is really consumed for next SDRS recommendations.

Next steps

See blog post "Storage DRS Design Considerations".

And as always, any comment is appreciated.

ESXi 6 - manual partitioning for multiple VMFS filesystems on single disk device

I have to test SDRS initial placement exact behavior (blog post here) therefore I need multiple VMFS datastores to form an Datastore Cluster with SDRS. Unfortunately, I'm constraint with storage resources in my home lab therefore I would like to use one local 220GB SSD to simulate multiple VMFS datastores.

Warning: This is not recommended practice for productional systems. It is recommended to have single partition (LUN) per single device.

I did not find GUI way how to create multiple partitions per single disks therefore I used CLI.

You cannot use old good fdisk for VMFS partitions. Instead PartedUtil has to be used because of GPT. PartedUtil is included in ESXi so you can login to ESXi over SSH and use partedUtil.

Note: It is worth to mention that you can have maximally 16 partitions in GUID Partition Table (GPT) in ESXi 6. This is what I have tested. When I tried to create 17 partitions the partedUtil failed with message - "Too many partitions (17)". 

In my lab I have Intel NUC with local SSD identified as "t10.ATA_____INTEL_SSDMCEAW240A4_____________________CVDA4426003E240M____".

To get partition table use following command

 partedUtil getptbl "t10.ATA_____INTEL_SSDMCEAW240A4_____________________CVDA4426003E240M____"  

If disk is empty you should see follwoing output:

msdos
29185 255 63 468862128

I would like to have 5 x 10Gb LUNs.

10Gb is 20971519 in sectors. Sector is 512Bytes

First 2048 should be skipped to keep space for GPT and to be aligned with 512B sectors. Below is my partition plan in format [Partition number, Start sector, End Sector]

P1 2048-20973567
P2 20973568-41945087
P3 41945088-62916607
P4 62916608-83888127
P5 83888128-104859647

Following command is used to create these 5 partitions.

 partedUtil setptbl "t10.ATA_____INTEL_SSDMCEAW240A4_____________________CVDA4426003E240M____" gpt "1 2048 20973567 AA31E02A400F11DB9590000C2911D1B8 0" "2 20973568 41945087 AA31E02A400F11DB9590000C2911D1B8 0" "3 41945088 62916607 AA31E02A400F11DB9590000C2911D1B8 0" "4 62916608 83888127 AA31E02A400F11DB9590000C2911D1B8 0" "5 83888128 104859647 AA31E02A400F11DB9590000C2911D1B8 0"  

Now you can list partitions again and you should see following output:

gpt
29185 255 63 468862128
1 2048 20973567 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
2 20973568 41945087 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
3 41945088 62916607 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
4 62916608 83888127 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
5 83888128 104859647 AA31E02A400F11DB9590000C2911D1B8 vmfs 0 

So we have 5 partitions and the last step is to format it to VMFS5 file system. We have to leverage vmkfstools which is also included in ESXi 6 system.

Let's start with first partition and we will use datastore name Datastore1.

 vmkfstools -C vmfs5 -S Datastore1 t10.ATA_____INTEL_SSDMCEAW240A4_____________________CVDA4426003E240M____:1  

The same procedure has to be repeated for each partition.

Here is Perl script which generates commands for partedUtil and vmkfstools based on several variables.

 #!/usr/bin/perl  
 #  
 my $disk_device = "t10.ATA_____INTEL_SSDMCEAW240A4_____________________CVDA4426003E240M____";  
 my $starting_sector = 2048;  
 my $partition_size_in_sectors = 4000000;  
 my $datastore_prefix = "Datastore";  
 my $num_of_partitions = 16; # Maximum is 16 partitions  
 my $partitions;  
 my $start = $starting_sector;  
 my $end;  
 my $i;  
 # Generate command for GPT partitions  
 for ($i=1; $i<$num_of_partitions; $i=$i+1) {  
     $end = $start + $partition_size_in_sectors;  
     $partitions .= "\"$i $start $end AA31E02A400F11DB9590000C2911D1B8 0\" ";  
     $start = $end + 1;  
 }  
 print "partedUtil setptbl \"$disk_device\" gpt $partitions\n";  
 # Generate commands to format partitions with VMFS5 file system  
 #  
 for ($i=1; $i<$num_of_partitions; $i=$i+1) {  
     print "vmkfstools -C vmfs5 -S $datastore_prefix$i $disk_device:$i\n";  
 }  

If you want to unmount 16 datastores it is very handy to use following command line:

 esxcli storage filesystem unmount -l Datastore1   

Hope this helps at least one other vSphere dude in his lab exercises. 

Don't use 4K Native drives for VMware vSphere ESXi nor VSAN

First of all, let's be absolutely clear. Disks with 4K sector size are not currently supported by VMware. See VMware KB- Support statement for 512e and 4K Native drives for VMware vSphere and VSAN (2091600)

UPDATE: vSphere 6.5 and VSAN 6.5 introduced 512e support so 4K native drives with 512 emulation (512e) are supported. In other words, 4K native drives without 512e are still not supported. 

UPDATE 2018-04-18: vSphere 6.7 introdiuced support of 4K native drives.

IMPORTANT STATEMENTS FROM KB

Does current GA version of vSphere and VSAN support 4K Native drives?

No. 4K Native drives are not supported in current GA releases of vSphere and VSAN.

Does current GA version of vSphere and VSAN support 512e drives?

No. 512e drives are not supported with the current versions of vSphere and VSAN due to potential performance issues when using these drives. 

Therefore only 512n (native) drives are supported on any ESXi (5.x, 6.x) at the moment.

It is usually not big deal on shared storage systems (aka disk arrays) because logical volumes are virtually emulated and sector size is usually 512 by default or configurable (512 or 4K). 

I thought that it is the same with RAID controllers because virtual volumes are also emulated by RAID controller. However, I have just recently learned that it is not true. At least not for all RAID controllers. For example DELL PERC H730 (the best RAID controller DELL currently offers) doesn't allow you to choose sector size for virtual volume. Instead, sector size is passed from physical disks to operating system - ESXi hypervisor in our case.

Here is one real customer story with 4K native disks. 

The customer was not able to create datastore on some disks. The error message was ...

Call "HostDatastoreSystem.QueryVmfsDatastoreCreateOptions" for object "ha-datastoresystem" on ESXi "esxi-test" failed.  

The error message is depicted on screenshot below.

Server Model: PowerEdge R530 – System Revision I

Operating System: VMware ESXi 5.5.0 build-3343343

BIOS Version: 1.5.4

Lifecycle Controller Firmware: 2.21.21.21

RAID Controller: Perc H730Mini 

RAID1 (SYSTEM)

Physical Disk 0:1:0 Online 0 278.88 GB Not Capable SAS HDD No

Physical Disk 0:1:1 Online 1 278.88 GB Not Capable SAS HDD No

DATASTORE – successfully created during ESXi installation  – OK

RAID1 (DATA)

Physical Disk 0:1:2 Online 2 558.38 GB Not Capable SAS HDD No

Physical Disk 0:1:3 Online 3 558.38 GB Not Capable SAS HDD No

DATASTORE – Datastore cannot be created because 4K - FATAL ISSUE

RAID5 (DATA)

Physical Disk 0:1:4 Online 4 1862.50 GB Not Capable SAS HDD No

Physical Disk 0:1:5 Online 5 1862.50 GB Not Capable SAS HDD No

Physical Disk 0:1:6 Online 6 1862.50 GB Not Capable SAS HDD No

DATASTORE – datastore can be created but not officially supported by VMware because 512e – RISK

RAID Controller list of disks

T17: C0:PD   Flags    State Type Size          S N F P Vendor   Product          Rev  P C ID SAS Addr         Port Phy DevH WU BFw  BRev

T17: C0:------------------------------------------------------------------------------------------------------------------------------

T17: C0:0    f1400005 00020 00   22ecb25b      0 0 0 1 TOSHIBA  AL13SXB30EN      DK02 0 0 500003969802a076 03   04  0010   1  NA   NA - 512b (512 Native)

T17: C0:1    f1400005 00020 00   22ecb25b      0 0 0 1 TOSHIBA  AL13SXB30EN      DK02 0 0 500003969802a062 00   00  000a   1  NA   NA - 512b (512 Native)

T17: C0:2    f1400005 00020 00   8bba5f5       0 0 0 1 HGST     HUC156060CS4204  EK11 0 0 5000cca059596e59 05   06  000e   1  NA   NA - 4kn (4k Native)

T17: C0:3    f1400005 00020 00   8bba5f5       0 0 0 1 HGST     HUC156060CS4204  EK11 0 0 5000cca0595aa1cd 02   02  000c   1  NA   NA - 4kn (4k Native)

T17: C0:4    f1400005 00020 00   e8e088af      0 0 0 1 SEAGATE  ST2000NX0273     NS28 0 0 5000c5008f3a8efd 04   05  000d   1  NA   NA - 512e (512 Emulation)

T17: C0:5    f1400005 00020 00   e8e088af      0 0 0 1 SEAGATE  ST2000NX0273     NS28 0 0 5000c5008f3ae545 01   01  000b   1  NA   NA - 512e (512 Emulation)

T17: C0:6    f1400005 00020 00   e8e088af      0 0 0 1 SEAGATE  ST2000NX0273     NS28 0 0 5000c5008f3b0661 06   07  000f   1  NA   NA - 512e (512 Emulation)

T17: C0:20   01400005 00020 0d   0             0 0 0 0 DP       BP13G+           2.23 0 0 524180704c645200 00   08  0009   0  NA   NA

T17: C0:100  00400005 00020 03   0             0 0 0 0 LSI      SMP/SGPIO/SEP    4402 0 0                0 00   ff  ffff   0  NA   NA

CONCLUSION AND  LESSONS LEARNED

It is obvious that VMware will support 4K disks sometimes in the future because industry is moving there but if you are planning to use directly attached disks choose disks with 512 sector. It is ESXi limitation at the moment. VMware VSAN is also impacted by this limitation because VSAN relies on ESXi.

Update 2016-01-21:
I have just received following question from one reader ...

"How can I list physical disks connected to internal PERC8?"

Unfortunately I don't have access to any 13G Dell server but I did a research and there should be three available methods.

Method 1/ racadm
If you have DRAC (Dell Remote Access/management Card) you can leverage racadm.
Based on racadm documentation it should be possible

• Storage.PhysicalDisk.BlockSizeInBytes (Read Only) Description This is readonly attribute. This property indicates the logical block size of the physical drive that this virtual disk belongs to. Legal Values Values: 512 or 4096

Method 2/ Export PERC Raid Controller Log with Dell Support Live Image Version 2.0

http://de.community.dell.com/techcenter/support-services/w/wiki/369.export-perc-raid-controller-log-with-dell-support-live-image-version-2-0-englisch

Method 3/ perccli
Dell has utility called perccli. You can check perccli documentation for all details but there is command for viewing physical drive details for the specified slot in the controller.

• Syntax is perccli /c0/e32/s4 show all

Downside of this method is that perccli binaries exist just for windows or linux so you cannot use it directly from ESXi and you have to boot for example Linux live CD.

Method 4/ megacli (not supported)

Third method is leveraging LSI megacli utility. Dell PERC is manufactured by LSI so it should work. LSI has megacli VIB for ESXi but it is not officially supported by VMware nor Dell.
See details at http://de.community.dell.com/techcenter/support-services/w/wiki/909.how-to-install-megacli-on-esxi-5-x

How to restore deleted vmdk from VMFS5

Yesterday I have got an E-mail from somebody asking me how to restore deleted vmdk from VMFS5. They deleted VM but realised there are very important data.

Typical answer would be - "Restore from backup" - however they wrote that they don't have backup.

Fortunately, I have never had a need to restore deleted vmdk so I was starting to do some quick research (aka googling :-) )

I found VMware KB 1015413 with following statement ...

"VMware does not offer data recovery services. This article provides contact information for data recovery service providers.
Note: VMware does not endorse or recommend any particular third-party utility, nor is this list meant to be exhaustive."

I was sitting in the VMware office so I have asked colleagues if they have any practical experience with undeleting vmdk from VMFS.  One colleague of mine suggested utility "VMFS Recovery" from DiskInternals.com. He had positive experience with this particular tool. His suggestion was to use trial version which should help to identify if recovery is possible and buy full version for recovery.

Warning: Use any third party tool for recovery on your own risk.

I absolutely understand that if you lose your important data you would like to try anything to recover it back however here are my general thoughts:

• Clone or snapshot your raw disk (LUN) with VMFS before any recovery (use storage array capabilities, third party imaging tool or *nix dd command) 
• If your data are very valuable for you consider engagement of data recovery expert services
• When  you do recovery by your self I wish you good luck.

If you have some other experience with this topic please share it with community in comments. 

Storage related problems with LSI 12Gb SAS HBA card

Our Dell field engineer experienced strange storage problems with SAS storage connected to ESXi hosts having LSI 12Gb SAS HBAs. Datastores were inaccessible after ESXi reboot, paths were temporarily unavailable, etc. In this particular case it was DELL Compellent storage with SAS front-end ports but the problem was not related to particular storage and similar issue can be experienced on other SAS storage systems like DELL PowerVault MD, etc.

He has found (thanks Google) Christopher Glemot's blog post describing other issues with LSI 12Gb SAS HBA. However the key point was that mpt3sas driver should be used and not msgpt3 driver which was on these particular ESXi hosts.

The solution is relatively easy:

• Uninstall msgpt3 driver - esxcli software vib remove –n lsi-msgpt3
• Download the latest mpt3sas driver (currently mpt3sas-10.00.00.00-6.0-2803883.zip)
• Unpack and upload mpt3sas driver  to ESXi host
• Install  mpt3sas driver on ESXi host - esxcli software vib install –d /path/mpt3sas-10.00.00.00-6.0-offline_bundle-2803883.zip
• Restart ESXi host

Hope this helps.

DELL Compellent Best Practices for Virtualization

All DELL Compellent Best Practices has been moved here.

The most interesting best practice document for me is "Dell Storage Center Best Practices with VMware vSphere 6.x".

Dell Virtual Racks

Virtual racks with Dell equipment are available at http://esgvr.dell.com/

Dell Server Virtual Rack

Direct link to DELL Server Virtual Rack where you can see how particular compute systems physically looks.

Dell Storage Virtual Rack

Direct link to DELL Storage Virtual Rack where you can see how particular storage systems physically looks.

Dell Networking Virtual Rack

Direct link to DELL NetworkingVirtual Rack where you can see how particular network systems physically looks.

Disk queue depth in an ESXi environment

On the internet, there are a lot of information and documents about ESXi and Disk Queue Depth but I didn't find any single document explained all details I would like to know and in the format for easy consumption. Different vendors have their specific recommendations and best practices but without deeper principal explanation and holistic view. Some documents are incomplete and some others have, in my opinion, wrong or at least strange information. That's the reason I have decided to write this blog post mainly for my own purpose but I hope it helps other people from the VMware community.

When I did research I found Frank Denneman blog post from March 2009 where almost everything is very well and accurately explained but technology little bit changed over time. Another great document is here on VMware Community Docs.

Disclaimer: I work for DELL Services, therefore, I work on a lot of projects with DELL Compellent Storage Systems, DELL Servers and QLogic HBA's. Therefore this blog post is based on "DELL Compellent Best Practices with VMware vSphere 5.x" and "Qlogic Best Practices for VMware vSphere 5.x". I have all this equipment in the lab so I was able to test it and verify what's going on. However, although this blog has information about specific hardware general principles are the same for any other HBAs and storage systems.

Some blog sections are just copied and paste from public documents mentioned above so all credits go there. Some other sections are based on my lab tests, experiments and thoughts.

What is queue depth?

Queue depth is defined as the number of disk transactions that are allowed to be “in flight” between an initiator and a target (parallel I/O transaction), where the initiator is typically an ESXi host HBA port/iSCSI initiator and the target is typically the Storage Center front-end port. Since any given target can have multiple initiators sending it data, the initiator queue depth is generally used to throttle the number of parallel transactions being sent to a target to keep it from becoming “flooded”. When this happens, the transactions start to pile up causing higher latencies and degraded performance. That being said, while increasing the queue depth can sometimes increase performance, if it is set too high, there is an increased risk of over-driving (over-loading and saturating) the storage array. As data travels between the application and the storage array, there are several places that the queue depth can be set to throttle the number of concurrent (parallel) disk transactions. The most common places queue depth can be modified are:

• The application itself (Default=dependent on application)
• The virtual SCSI card driver in the guest (Default=32)
• The VMFS layer (DSNRO) (Default=32)
• The HBA VMkernel Module driver (Default=64)
• The HBA BIOS (Default=Varies)

HBA Queue Depth = AQLEN 

(Default Varies but in my QLogic card it is 2176)

Here are Compellent recommended BIOS settings for QLogic Fibre Channel card:

1. The “connection options” field should be set to 1 for point to point only
2. The “login retry count” field should be set to 60 attempts
3. The “port down retry” count field should be set to 60 attempts
4. The “link down timeout” field should be set to 30 seconds
5. The “queue depth” (or “Execution Throttle”) field should be set to 255. This queue depth can be set to 255 because the ESXi VMkernel driver module and DSNRO can more conveniently control the queue depth

I don't think the point (5) is correct and up to date in Compellent Best Practices. Based on QLogic document HBA "Queue Depth" doesn't exist in QLogic BIOS and "Execution Throttle" is not used anymore. QLogic Adapter Queue Depth is only managed by the driver and default value is 64 for vSphere 5.x. For more info look at here.

I believe QLogic Host Bus Adapter has Queue Depth 2176 because this number is visible in esxtop as AQLEN (Adapter Queue Length) for each particular HBA (vmhba).

HBA LUN Queue Depth 

(Default=64)
HBA LUN Queue Depth is by default 64 but can be changed via VMkernel Module driver. Compellent recommends to set HBA max queue depth to 255 but this queue depth can be used just for single VM running on single LUN (disk). If more VMs are running on LUN than DSNRO has precedence. However, when SIOC (Storage I/O Control) is used then DSNRO is not used at all and HBA VMkernel Module Driver Queue Depth value is used for each device queue depth  (DQLEN). I think SIOC is beneficial in this situation because you can have deeper queues with dynamic queue management across all ESXi hosts connected to particular LUN. Please note, that special attention has to be taken for correct SIOC latency threshold especially on LUNs with sub-lun tiering.

If you want to check your Disk (aka LUN) Queue Depth value it is nicely visible in esxtop as DQLEN.

So here is the procedure on how to change HBA VMkernel Module Driver Queue Depth.

Find the appropriate driver name for the module that is loaded for QLogic HBA:

~ # esxcli system module list | grep ql
qlnativefc                          true        true

In our case, the name of module for QLogic driver is qlnativefc.

Set the QLogic driver queue depth and timeouts using the esxcli command:

esxcli system module parameters set -m qlnativefc -p "ql2xmaxqdepth=255 ql2xloginretrycount=60 qlport_down_retry=60"

You have to reboot ESXi host to apply module changes.

Below is a description of QLogic Parameters we have just change

• ql2xmaxqdepth (int) - Maximum queue depth to report for target devices. 
• ql2xloginretrycount (int) - Specify an alternate value for the NVRAM login retry count. 
• qlport_down_retry (int) - Maximum number of command retries to a port that returns a PORT-DOWN status.

You can verify QLogic driver options by the following command:

esxcfg-module --get-options qlnativefc 

And affected change (after ESXi reboot) is visible on esxtop on disk devices as DQLEN.

VMFS Layer DSNRO 

(Default=32)
When two or more virtual machines share a LUN (logical unit number), this parameter controls the total number of outstanding commands permitted from all virtual machines collectively on the host to that LUN (this setting is not per virtual machine). For more information see KB

The HBA LUN queue depth (DQLEN) determines how many commands the HBA is willing to accept and process per LUN and target. If a single VM virtual machine is issuing IO, the HBA LUN Queue Depth (DQLEN) setting is applicable. When multiple VM’s are simultaneously issuing IO’s to the LUN, the ESX parameter Disk.SchedNumReqOutstanding (DSNRO) value becomes the leading parameter, and HBA LUN Queue Depth is ignored. The above statement is only true when SIOC is not enabled on the LUN (datastore).

Parameter can be changed to the value between 1 and 256 so when 

esxcli storage core device set -d -O

No of outstanding IOs with competing worlds (DSNRO) can be listed for each device

~ # esxcli storage core device list -d naa.6000d3100025e7000000000000000098
naa.6000d3100025e7000000000000000098
   Display Name: COMPELNT Fibre Channel Disk (naa.6000d3100025e7000000000000000098)
   Has Settable Display Name: true
   Size: 1048576
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.6000d3100025e7000000000000000098
   Vendor: COMPELNT
   Model: Compellent Vol
   Revision: 0504
   SCSI Level: 5
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: supported
   Other UIDs: vml.02000200006000d3100025e7000000000000000098436f6d70656c
   Is Local SAS Device: false
   Is Boot USB Device: false
   No of outstanding IOs with competing worlds: 32

The issue with DSNRO is that it must be set per devices (LUNs) on all ESXi hosts so it has a negative impact on implementation and overall manageability. It is usually left on default value anyway because 32 outstanding (async) IOs is good enough on low latency SANs and properly designed storage devices with good response times. Bigger queues are beneficial on higher latency SANs and/or storage systems which need more parallelism (more parallel threads) to draw out more I/O's storage is physically capable of. It doesn't help in situations when storage doesn't have enough back end performance (usually spindles). Bigger queues can generate more outstanding IOs to the storage system which can overload the storage and everything will be even worse. That's the reason why I think it is a good idea to leave DSNRO on the default value.

And once again. DSNRO is not used when SIOC is enabled because in that case "HBA LUN queue depth" is the leading parameter for disk queue depth (DQLEN).

Storage Target Port Queue Depth

Queues are on a lot of places and you have to understand the path from end to end. A queue exist on the storage array controller port as well, this is called the “Target Port Queue Depth“. Modern midrange storage arrays, like most EMC and HP arrays, can handle around 2048 outstanding IO’s.
2048 IO’s sounds a lot, but most of the time multiple servers communicate with the storage controller at the same time. Because a port can only service one request at a time, additional requests are placed in the queue and when the storage controller port receives more than 2048 IO requests, the queue gets flooded. When the queue depth is reached, this status is called (QFULL), the storage controller issues an IO throttling command to the host to suspend further requests until space in the queue becomes available. The ESXi host accepts the IO throttling command and decreases the LUN queue depth to the minimum value, which is 1! The QFULL condition is handled by the QLogic driver itself. The QFULL status is not returned to the OS. But some storage devices return BUSY rather than QFULL. BUSY errors are logged in the /var/log/vmkernel. Not every busy error is a Qfull error! Check the SCSI sense codes indicated in the vmkernel message to determine what type of error it is. The VMkernel will check every 2 seconds to check if the QFULL condition is resolved. If it is resolved, theVMkernel will slowly increase the LUN queue depth to its normal value, usually, this can take up to 60 seconds.

Compellent SC8000 front-end ports act as targets and have Target Port Queue Depth 1900+. What 1900+ means? Compellent is excellent storage operating system on top of commodity servers, IO cards, disk enclosures and disks. By the way, is it software-designed-storage or not? But back to queues, 1900+ means that Queue Depth depends on IO card used for front-end port but 1900 is the guaranteed minimum. So for our case let's assume we have Storage Target Port Queue Depth 2048.

All together will give as the holistic view
Here are end to end disk queue parameters:

• DSNRO is 32 - this is default ESXi value.
• HBA LUN Queue Depth is 64 by default in QLogic HBAs. Compellent recommends to change it  to 256 by parameter of HBA VMkernel Module Driver.
• HBA Queue Depth is 2176.
• Compellent Target Ports Queue Depths are 2048.

Everything is depicted in the figure below.

Disk Queue Depth - Holistic View

What is the optimal HBA LUN Queue Depth?

We already know 64 is the default value for QLogic and you can change it via VMkernel Module driver. So what is the optimal HBA LUN Queue Depth?

To prevent flooding the target port queue depth, the result of the combination of a number of host paths + HBA LUN Queue Depth+ number of presented LUNs through the host port must be less than the target port queue depth. In short T >= P * Q * L

T = Target Port Queue Depth (2048)
P = Paths connected to the target port (1)
Q = HBA LUN Queue Depth (?)
L = number of LUN presented to the host through this port (20)

So when I have 20 LUNs exposed from Compellent storage system HBA LUN Queue Depth should be maximally Q = T / P /L = 2048 / 1 / 20 = 102.4 for a single ESXi host. When I have 16 hosts HBA LUN Queue Depth must be divided by 16 so the value for this particular scenario is  6.4.
This number is without any overbooking which is not the real case. Current QLogic HBA LUN Queue Depth default value (64) introduces fan in fan-out ratio 10:1 which is probably based on long term experience with virtualized workloads. In the past, Qlogic has lower default values 16 in ESX 3.5 and 32 in ESX 4.x which had lower fan-in / fan-out ratios (2.5:1 and 5:1).

With Compellent recommended QLogic HBA LUN Queue Depth 255 we will have, in this particular case, fan-in / fan-out ratio 40 : 1. But only when SIOC is used. And when SIOC is used and normalized datastore latency is higher than SIOC threshold dynamic queue management kick in and disk queues are automatically throttled. So all is good.

What is my real Disk Queue Depth?

VMkernel Disk Queue Depth (DQLEN) is the number that matters. However, this number is dynamic and it depends on several factors. Here are scenarios you can have:

Scenario 1 - SIOC enabledWhen you have datastore with SIOC enabled then your DQLEN will be set to HBA LUN Queue Depth. It is by default 64 but Compellent recommends to set it to 255. But Compellent doesn't recommend to use SIOC.
Scenario 2 - SIOC disabled and you have only one VM on datastoreWhen you have just one VM on datastore your DQLEN will be set o HBA LUN Queue Depth. But how often do you have just one VM on datastore?
Scenario 3 - SIOC disabled and you have two or more VMs on datastoreIn this scenario your DQLEN will be set to DSNRO which is by default 32 and it has precedence over HBA LUN Queue Depth.

ESXi Disk Queue Management - None, Adaptive Queuing or Storage I/O Control?

By default, ESXi doesn't use any Disk Queue Length (DQLEN) throttling mechanism and your DQLEN is set to DSNRO (32) when two or more VM (vDisks) are on the datastore. When only one VM disk is on the datastore your DQLEN will be set to HBA LUN Queue Depth (default=64, Compellent recommends 255).

When you have enabled Adaptive Queuing or Storage I/O Control your Disk Queue Length (DQLEN) will be throttled automatically when I/O congestion occurs. The difference between Adaptive Queuing and SIOC is how I/O congestion is detected.

Adaptive queuing is waiting for storage SCSI Sense Code of BUSY or QUEUE FULL status on the I/O path. For adaptive queueing, there are some advanced parameters that must be set on a per-host basis. From the Configuration tab, you have to select Software Advanced Settings. Navigate to Disk and the two parameters you need are Disk.QFullSampleSize and Disk.QFullThreshold. By default, QFullSampleSize is set to 0, meaning that it is disabled. When this value is set, adaptive queueing will kick in and half the queue depth when this number of queue full conditions is reported by the array. The QFullThreshold is the number of good statuses to receive before incrementing the queue once again.

Storage I/O Control uses the concept of a congestion threshold, which is based on latency. But not normal latency of datastore on only one particular ESXi host but there is a sophisticated algorithm preparing normalized datastore latency of all datastore latencies across ESXi hosts using one particular datastore. On top of datastore wise normalization, there is another type of normalization which takes in to account just "normal" I/O sizes. What is a normal IO size? I cannot find this detail but I think I read somewhere that the normal I/O size is between 2KB and 16KB.  Update 2016-10-07: Now when I work for VMware I have access to ESXi source codes and if I read the code correctly it seems to me that IO size normalization works a little bit differently. I cannot publish the exact formula as it is VMware intellectual property and implementation details but the whole idea are that for example 1MB I/Os would significantly skewed latency against normally accepted latency values in the storage industry, therefore, latency for bigger I/O's are somehow adjusted based on I/O size.

Information in the above section is based on Cormag Hogan blog post.

Conclusion

Based on the information above I think that default ESXi and QLogic values are the best for general workloads and tuning queues is not something I would recommend to do. It is important to know that queues tuning does not help with performance in most cases. Queues are used for latency improvements during transient I/O burst. If you have storage performance issues, it is usually because of an overloaded storage system and not about queues in the network path. Bigger Queue Depth can help you to handle more parallel I/Os to the storage subsystem. This could be beneficial when your target storage system can handle more I/Os. It cannot help in situations when the target storage system is overloaded and you expect more performance just by increasing queue depth.

The question I still need the answer to my self is why Compellent doesn't recommend VMware Storage I/O Control and Adaptive Queuing for dynamic queue management which is in my opinion very good thing.

UPDATE 2014-12-16: I have downloaded the latest Compellent Best Practices and there is a new statement about SIOC. SIOC can be enabled but you must know what's happening and if it is beneficial for you. Here is the snippet from the document ...

SIOC is a feature that was introduced in ESX/ESXi 4.1 to help VMware administrators regulate storage performance and provide fairness across hosts sharing a LUN. Due to factors such as Data Progression and the fact that Storage Center uses a shared pool of disk spindles, it is recommended that caution is exercised when using this feature. Due to how Data Progression migrates portions of volumes into different storage tiers and RAID levels at the block level, this could ultimately affect the latency of the volume, and trigger the resource scheduler at inappropriate times. Practically speaking, it may not make sense to use SIOC unless pinning particular volumes into specific tiers of disk.

I still believe SIOC is the way to go and special attention has to be paid to SIOC latency threshold. Compellent recommends keeping it on default value 30 milliseconds which makes perfect sense. Storage System will do all the hard work for you but when there are huge congestion and normalized latency is too high dynamic ESXi disk queue management can kick in. It makes sense to me.

I also believe that Adaptive Queuing is a really good and practical safety mechanism when your storage array has full queues in storage front-end ports or LUNs. However, it is applicable only with storage arrays supporting Adaptive Queuing and sending back to ESXi SCSI sense codes about full queue condition. If Adaptive Queueing is not used, even SIOC cannot help you with LUNs/Datastores issues (Datastore Disconnections) because SIOC algorithm is based on device response time but not on queue full storage response. Therefore, I strongly recommend enabling SIOC together with Adaptive Queuing unless your storage vendor has a really good justification to not do so. SIOC will help you with storage traffic throttling during high device response times and Adaptive Queuing when storage array queues are full and the device cannot accept new I/O's. However, Adaptive Queuing should be configured in concert with your storage vendor. For more information on how to enable Adaptive Queuing read VMware KB 1008113. Please note, that SIOC and Adaptive Queuing are just safety mechanisms on how to mitigate impacts of storage issues but the root cause is the Capacity Planning on the storage array.

UPDATE 2020-10-22: SIOC control throttles queues based on hitting a datastore latency threshold. This is the disk (device) latency not including any additional latency induced by additional VMkernel queueing. SIOC is using a minimal threshold 5 ms response time to kick in SIOC, this can be good for traditional storage arrays with rotational disks, however, in the all-flash era, you expect response times between 1 and 3 ms, right? Sometimes even sub-millisecond response time. SIOC will not help you to achieve fairness and defined SLA/OLA in such all-flash environments, but can you do dynamic queue management at least in situations the response times increase above 5 ms. 

If any of you with a deep understanding of vSphere and storage architectures see an error in my analysis, please let me know so that I can correct it appropriately.

Useful links:

• VirtualGeek : VMware I/O queues, “micro-bursting”, and multipathing
• Cormac Hogan : Adaptive Queueing vs. Storage I/O Control 
  
• VMware KB: Checking the queue depth of the storage adapter and the storage device
• Mariusz Kaczorek : What is Storage Queue Depth (QD) and why is it so important?
• Duncan Epping : Why Queue Depth matters!
• Duncan Epping : Storage I/O Fairness
• Duncan Epping : DQLEN changes, what is going on?
• Cody Hosterman : UNDERSTANDING VMWARE ESXI QUEUING AND THE FLASHARRAY
• Jason Massae, Cody Hosterman : Core Storage Best Practice Deep Dive (VMworld 2020 - HCI1691)
• HPE : HPE 3PAR Storage System - Handling Port Target_qlength_above_threshold Alerts on ESX Servers