Start order of software services in VMware vCenter Server Appliance 6.0 U2

vCenter Server Appliance 6.0 U2 services are started in the following order ...

1. vmafdd (VMware Authentication Framework)
2. vmware-rhttpproxy (VMware HTTP Reverse Proxy)
3. vmdird (VMware Directory Service)
4. vmcad (VMware Certificate Service)
5. vmware-sts-idmd (VMware Identity Management Service)
6. vmware-stsd (VMware Security Token Service)
7. vmware-cm (VMware Component Manager)
8. vmware-cis-license (VMware License Service)
9. vmware-psc-client (VMware Platform Services Controller Client)
10. vmware-sca (VMware Service Control Agent)
11. applmgmt (VMware Appliance Management Service)
12. vmware-netdumper (VMware vSphere ESXi Dump Collector)
13. vmware-syslog (VMware Common Logging Service)
14. vmware-syslog-health (VMware Syslog Health Service)
15. vmware-vapi-endpoint (VMware vAPI Endpoint)
16. vmware-vpostgres (VMware Postgres)
17. vmware-invsvc (VMware Inventory Service)
18. vmware-mbcs (VMware Message Bus Configuration Service)
19. vmware-vpxd (VMware vCenter Server)
20. vmware-eam (VMware ESX Agent Manager)
21. vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter)
22. vmware-sps (VMware vSphere Profile-Driven Storage Service)
23. vmware-vdcs (VMware Content Library Service)
24. vmware-vpx-workflow (VMware vCenter Workflow Manager)
25. vmware-vsan-health (VMware VSAN Health Service)
26. vmware-vsm (VMware vService Manager)
27. vsphere-client ()
28. vmware-perfcharts (VMware Performance Charts)
29. vmware-vws (VMware System and Hardware Health Manager) 

CLI for VMware Virtual Distributed Switch

A few weeks ago I have been asked by one of my customers if VMware Virtual Distributed Switch (aka VDS) supports Cisco like command line interface. The key idea behind was to integrate vSphere switch with open-source tool Network Tracking Database (NetDB) which they use for tracking MAC addresses within their network. I have been told by customer that NetDB can telnet/ssh to Cisco switches and do screen scraping so would not it be cool to have the most popular switch CLI commands for VDS? These commands are

• show mac-address-table
• show interface status

The official answer is NO, but wait a minute. Almost anything is possible with VMware API. So my solution is leveraging VMware's vSphere Perl SDK to pull information out of Distributed Virtual Switches. I have prepared PERL script vdscli.pl which currently supports two commands mentioned above. It goes through all VMware Distributed Switches on single vCenter.

Script along with shell wrappers are available on GITHUB here https://github.com/davidpasek/vdscli

See screenshots below to get an idea what script does.

The output of the command

vdscli.pl --server=vc01.home.uw.cz --username readonly --password readonly --cmd show-port-status

looks as depicted in screenshot below.

and output of the command
vdscli.pl --server=vc01.home.uw.cz --username readonly --password readonly --cmd show-mac-address-table

So now we have Perl scripts to get information from VMware Distributed Virtual Switch which is nice, however, we would like to have Interactive CLI to have the same user experience as we have on physical switches CLI, right? For Interactive CLI I have decided to use Python ishell (https://github.com/italorossi/ishell) to emulate Cisco like CLI. To start interactive VDSCLI shell you must have Python with iShell installed and then you can simply run script

./vdscli-ishell.py

which is just a wrapper around vdscli.pl The screenshot of VDSCLI shell is in the figure below

VDSCLI Interactive Shell

And the last step is to allow SSH or Telnet access to VDSCLI shell. It can be very easily done via standard Linux possibility to change a shell for the particular user. The VDSCLI over ssh is depicted on the screenshot below.

VDSCLI Interactive Shell over SSH

To operationalize all these scripts, I would highly encourage you to read my another blog post ...
"CLI for VMware Virtual Distributed Switch - implementation procedure".

Hope somebody else in VMware community will find it useful.

Storage DRS integration with storage profiles

This is a very quick blog post. In vSphere 6.0, VMware has introduced Storage DRS integration with storage profiles (aka SPBM - Storage Policy Based Management).

Here is the link to official documentation.

Generally, it is about SDRS advanced option EnforceStorageProfiles. Advanced option EnforceStorageProfiles takes one of these integer values, 0,1 or 2 where the default value is 0.

• When option is set to 0, it indicates that there is NO storage profile or policy enforcement on the SDRS cluster.
• When option is set to 1, it indicates that there is storage profile or policy SOFT enforcement on the SDRS cluster. It is analogous with DRS soft rules. SDRS will comply with storage profile/policy in the optimum level. However if required, SDRS will violate the storage profile compliant.
• When option is set to 2, it indicates that there is storage profile or policy HARD enforcement on the SDRS cluster. It is analogous with DRS hard rules. In any case, SDRS will not violate the storage profile or policy compliant.

Please note that at the time of writing this post, SDRS Storage Profiles Enforcement works only during initial placement and NOT for already provisioned VMs during load balancing. Therefore, when iVM Storage Policy is changed for particular VM, SDRS will not make it automatically compliant nor throw any recommendation.

Another limitation is that vCloud Director (vCD) backed by SDRS cluster does NOT support  Soft (1) or Hard (2) storage profile enforcements. vCloud Director (vCD) will work well with Default (0) option

Relevant references to other resources:

• VMware KB : Storage DRS integration with storage profiles
• Vikas Shitole : One of cool vSphere features: Tutorial:How Storage DRS works with storage policies in SPBM?
• Robert Kloosterhuis : Metro-Cluster SDRS datastore tagging and the EnforceStorageProfiles advanced setting

VMware Photon OS with PowerCLI

Photon OS is linux distribution maintained by VMware with multiple benefits for virtualized form factor, therefore any virtual appliance should be based on Photon OS.

I have recently tried to play with Photon OS and here are some my notes.

IP Settings

Network configuration files are in directory

/etc/systemd/network/

IP settings are leased from DHCP by default. It is configured in file  /etc/systemd/network/10-dhcp-en.network

File contains following config

[Match]
Name=e*
[Network]
DHCP=yes

To use static IP settings it is good to move DHCP config file down in alphabetical order and create config file with static IP settings.

mv 10-dhcp-en.network 99-dhcp-en.networkcp 99-dhcp-en.network 10-static-en.network

file  /etc/systemd/network/10-static-en.network should looks similar to

[Match]Name=eth0

[Network] 

Address=192.168.4.56/24 

Gateway=192.168.4.254 

DNS=192.168.4.4

Network can be restarted by command

systemctl restart systemd-networkd

and network settings can be checked by command

networkctl

Package management

Photon OS uses TDNF  (Tiny DNF) package manager. It is based on Fedora's DNF.  This is a development by VMware that comes with compatible repository and package management capabilities. Note that not every dnf command is available but the basic ones are there.

Examples:

• tdnf install libxml2
• tdnf install openssl-devel
• tdnf install binutils
• tdnf install pkg-config
• tdnf perl-Crypt-SSLeay
• tdnf install cpan
• tdnf libuuid-devel
• tdnf install make

Update of the whole operating system can be done by command

tdnf update

Log Management

You will not find typical linux /var/log/messages
Instead, journald is used and you have to use command journalctl

Equivalent to tail -f /var/log/messages is

journalctl -f 

System services

System services are control by command systemctl

To check service status use

systemctl status docker

To start service use

systemctl start docker

To enable service after system start use

systemctl enable docker

Docker and containerized PowerCLI

One of key use cases for Photon OS is to be a docker host, therefore, docker is preinstalled in Photon OS. You can see further Docker information by command

docker info

If Docker is running on your system, you can very quickly spin up docker container. Let's use example of containerized PowerCLI. To download container image from DockerHup use command

docker pull vmware/powerclicore

to check all downloaded images use the command

docker images -a   

 root@photon-machine [ ~ ]# docker images -a    
 REPOSITORY      TAG         IMAGE ID      CREATED       SIZE  
 vmware/powerclicore  latest       a8e3349371c5    6 weeks ago     610 MB  
 root@photon-machine [ ~ ]#   

Now you can run powercli container interactively (-i) and in allocated pseudo-TTY (-t). Option -rm stands for "Automatically remove the container when it exits".

docker run --rm -it vmware/powerclicore 

 root@photon-machine [ ~ ]# docker run --rm -it --name powercli vmware/powerclicore         
 PowerShell   
 Copyright (C) Microsoft Corporation. All rights reserved.erclicore --name powercl  
      Welcome to VMware vSphere PowerCLI!  
 Log in to a vCenter Server or ESX host:       Connect-VIServer  
 To find out what commands are available, type:    Get-VICommand  
 Once you've connected, display all virtual machines: Get-VM  
     Copyright (C) VMware, Inc. All rights reserved.  
 Loading personal and system profiles took 3083ms.  
 PS /powershell#   

Now you can use PowerCLI running on linux container. The very first PowerCLI command is usually Connect-VIServer, but you can get following warning and error messages

 PS /powershell> Connect-VIServer                                                                         
 cmdlet Connect-VIServer at command pipeline position 1  
 Supply values for the following parameters:  
 Server: vc01.home.uw.cz  
 Specify Credential  
 Please specify server credential  
 User: cdave  
 Password for user cdave: *********  
 WARNING: Invalid server certificate. Use Set-PowerCLIConfiguration to set the value for the InvalidCertificateAction option to Prompt if you'd like to connect once or to add  
  a permanent exception for this server.  
 Connect-VIServer : 06/07/2017 19:25:44     Connect-VIServer          An error occurred while sending the request.       
 At line:1 char:1  
 + Connect-VIServer  
 + ~~~~~~~~~~~~~~~~  
   + CategoryInfo     : NotSpecified: (:) [Connect-VIServer], ViError  
   + FullyQualifiedErrorId : Client20_ConnectivityServiceImpl_Reconnect_Exception,VMware.VimAutomation.ViCore.Cmdlets.Commands.ConnectVIServer  
 PS /powershell>   

To solve the problem you have to adjust PowerCLI configuration by

Set-PowerCLIConfiguration -InvalidCertificateAction ignore -confirm:$false -scope All

The command above changes PowerCLI configuration for all users.

To use other docker commands you can open another ssh session, and for example list running containers

 root@photon-machine [ ~ ]# docker ps -a     
 CONTAINER ID    IMAGE         COMMAND       CREATED       STATUS       PORTS        NAMES  
 6ecccf77891e    vmware/powerclicore  "powershell"    7 minutes ago    Up 7 minutes              powercli  
 root@photon-machine [ ~ ]#   

... or issue any other docker command.

That's cool, isn't it?

VMware VVOLs scalability

I'm personally a big fan of VMware Virtual Volumes concept. If you are not familiar with VVOLs check this blog post with the recording of VMworld session and read VMware KB Understanding Virtual Volumes (VVols) in VMware vSphere 6.0

We all know that the devil is always in details. The same is true with VVOLs. VMware prepared the conceptual framework but implementation always depends on storage vendors thus it varies around storage products.

Recently, I have had VVOLs discussion with one of my customers and he was claiming that their particular storage vendor supports a very small number of VVOLs. That discussion inspired me to do some research.

Please, note that numbers below are valid at the moment of writing this article. You should always check current status with your particular storage vendor.

Vendor / Storage Array	Maximum VVOLs / Snapshots or Clones
DELL / Compellent SC 8000	2,000 / TBD
EMC / Unity 300	9,000 / TBD
EMC / Unity 400	9,000 / TBD
EMC / Unity 500	13,500 / TBD
EMC / Unity 600	30,000 / TBD
EMC / VMAX 3	64,000 / TBD
HPE / 3PAR	I have been told that it depends on specific model and firmware versions but it is roughly about 10,000 (the info gathered at November 2018)
Hitachi / VSP G200	2,000 / 100,000
Hitachi / VSP G400	4,000 / 100,000
Hitachi / VSP G600	4,000 / 100,000
Hitachi / VSP G800	16,000 / 100,000
Hitachi / VSP G1000	64,000 / 1,000,000

Numbers above are very important because single VM have minimally 3 VVOLs (home, data, swap) and usually even more (snapshot) or more data disks. If you will assume 10 VVOls for single VM you will end up with just 200 VMs on Dell Compellent or Hitachi VSP G200. On the other hand, EMC Unity 600 would give you up to 3,000 VMs which is not bad and enterprise storage systems (EMC VMAX and Hitachi G1000) would give you up to 6,400 VMs which is IMHO very good scalability.

So as always, it really depends on what storage system do you have or planning to buy.

If you know numbers for other storage systems, please share it in comments below this blog post.

Keywords: vvol, vvols, virtual volumes