SDRS Initial Placement - interim storage lease between recommendation and provisioning

Every day we learn something new. In the past, I blogged about SDRS behavior on these blog posts

• Storage DRS Design Considerations
• VMware vSphere SDRS - test plan of SDRS initial placement
• VMware vSphere SDRS VM provisioning process
• Storage DRS integration with storage profiles

Recently (a few months ago),  I have been informed about interesting SDRS behavior which is not exposed through standard GUI nor advanced settings but available through API. Such functionality was not very well known even within VMware so I have decided to blog about it.

Long story short ...

vSphere API Call for SDRS Initial Placement can lease recommended storage resource for some time. 

What does it mean? Just after recommendations, SDRS can lease the storage space on recommended datastores to have an interim reservation for somebody who is, most probably, going to do provisioning. By default, SDRS do not lease storage space on recommended datastores, therefore, you can observe provisioning failures in some situations. I have simulated such situation in Test #3 of test plan available here. Such situations are not very common when you do manual provisioning but there is higher probability when automated provisioning is in use so you can experience such issues on environments with VMware vRealize Automation (vRA) or vCloud Director (vCD).

And now the secret I did not know ... SDRS has the solution for such issues since vSphere 5.1. When somebody (vRA, vCD, anybody else who wants to deploy VM) asked for SDRS recommendation via API call, that API call can include a specific parameter (resourceLeaseDurationSec) which will instruct vSphere to block the recommended storage space on datastores only for provisioning of that specific SDRS recommendation. It's worth to mention, the resource leasing is released immediately after provisioning, therefore the time defined in resourceLeaseDurationSec is actually the maximum reservation time of the resource just in case somebody who wanted to do provisioning change his mind and decided to not deploy VM. This is to avoid unnecessary storage space reservations.
 
If you want to know details check API documentation. Here is what is written in vSphere API documentation about placeSpec.resourceLeaseDurationSec - Resource lease duration in seconds. If the duration is within bounds, Storage DRS will hold onto resources needed for applying recommendations generated as part of that call. Only initial placement recommendations generated by storage DRS can reserve resources this way.

Parameter resourceLeaseDurationSec is used in StoragePlacementSpec which encapsulates all of the inputs passed to the VcStorageResourceManager method recommendDatastores. VcStoragePlacementSpec is documented here.

So that sounds good, right? Well, there is one issue with this approach. SDRS can give provisioning application more recommendations (multiple datastores) which would lead to blocking more storage space than is really needed. VMware engineering is aware of this issue and at the moment works at least with vRA BU to solve it. As far as I know, the final solution will be a special SDRS setting to return a single recommendation. However, this is planned as a specific integration optimization between SDRS and vRA provisioning. 

UPDATE: details about special SDRS setting is described at https://www.vcdx200.com/2018/06/undocumented-sdrs-advanced-options.html

The challenge with VRA storage (vSphere vDisk) provisioning and SDRS is depicted in the figure bellow

To be honest, there is another design consideration and potential risk associated with this solution. If resourceLeaseDurationSec is used and an external application (VRA, vCloud Director, or other) is using it incorrectly, it can eventually block a storage space in the Datastore cluster and cause Deny of Service (DoS). Incorrect usage would be to leverage SDRS for recommendations that would block the recommended storage space but do not provision anything, therefore storage would be in block state for some defined time and not available for other provisionings until the lease expires.

CLI for VMware Virtual Distributed Switch

A few weeks ago I have been asked by one of my customers if VMware Virtual Distributed Switch (aka VDS) supports Cisco like command line interface. The key idea behind was to integrate vSphere switch with open-source tool Network Tracking Database (NetDB) which they use for tracking MAC addresses within their network. I have been told by customer that NetDB can telnet/ssh to Cisco switches and do screen scraping so would not it be cool to have the most popular switch CLI commands for VDS? These commands are

• show mac-address-table
• show interface status

The official answer is NO, but wait a minute. Almost anything is possible with VMware API. So my solution is leveraging VMware's vSphere Perl SDK to pull information out of Distributed Virtual Switches. I have prepared PERL script vdscli.pl which currently supports two commands mentioned above. It goes through all VMware Distributed Switches on single vCenter.

Script along with shell wrappers are available on GITHUB here https://github.com/davidpasek/vdscli

See screenshots below to get an idea what script does.

The output of the command

vdscli.pl --server=vc01.home.uw.cz --username readonly --password readonly --cmd show-port-status

looks as depicted in screenshot below.

and output of the command
vdscli.pl --server=vc01.home.uw.cz --username readonly --password readonly --cmd show-mac-address-table

So now we have Perl scripts to get information from VMware Distributed Virtual Switch which is nice, however, we would like to have Interactive CLI to have the same user experience as we have on physical switches CLI, right? For Interactive CLI I have decided to use Python ishell (https://github.com/italorossi/ishell) to emulate Cisco like CLI. To start interactive VDSCLI shell you must have Python with iShell installed and then you can simply run script

./vdscli-ishell.py

which is just a wrapper around vdscli.pl The screenshot of VDSCLI shell is in the figure below

VDSCLI Interactive Shell

And the last step is to allow SSH or Telnet access to VDSCLI shell. It can be very easily done via standard Linux possibility to change a shell for the particular user. The VDSCLI over ssh is depicted on the screenshot below.

VDSCLI Interactive Shell over SSH

To operationalize all these scripts, I would highly encourage you to read my another blog post ...
"CLI for VMware Virtual Distributed Switch - implementation procedure".

Hope somebody else in VMware community will find it useful.

Best Practices for Faster vSphere SDK Scripts

Source at http://www.virtuin.com/2012/11/best-practices-for-faster-vsphere-sdk.html 

The VMware vSphere API is one of the more powerful vendor SDKs available in the Virtualization Ecosystem.  As adoption of VMware vSphere has grown over the years, so has the size of Virtual Infrastructure environments.  In many larger enterprises, the increasing number of VirtualMachines and HostSystems is driving the architectural requirement to deploy multiple vCenter Servers.
In response, the necessity for automation tooling has grown just as quickly.  Automation to create daily reports, perform bulk operations, and aggregate data from large, distributed Virtual Infrastructure environments is a common requirement for managing the increasing virtual sprawl.
In a Virtual Infrastructure comprised of thousands of objects, even a simple script to list all VirtualMachines and their associated HostSystem and Datastores can result in very slow runtime execution.  Developing automation with the following, simple best practices can take orders of magnitude off your vSphere API tool's runtime.

 READ FULL ARTICLE