VCSA - This appliance cannot be used or repaired ...

I have just got an email from my customer describing the weird issue with VMware vCenter Server Appliance (aka VCSA).

The customer is doing weekly native backups of VCSA manually via VAMI. He wanted to run VCSA native backup again but when he tried to log into virtual appliance management interface (VAMI) he is getting the following error message
 

Error message - This appliance cannot be used or repaired because of failure was encountered. You need to deploy a new appliance.

 

The error message includes a resolution. Deploy a new appliance. The recommended solution is the last thing a typical vSphere admin would like to resolve such an issue. Fortunately enough, there is another solution/workaround.

To resolve this issue stop and start all the services on the vCSA,

• Putty/SSH to vCenter server appliance.
• Login to VCSA using the root credentials.
• Enabled "shell".
• Restart VCSA services

To restart VCSA services run the following commands:
service-control --stop --all
service-control --start --all

In case, simple services restart does not help, you can have an issue with some recent backup job. In such a case, there is another resolution with an additional workaround

• Putty/SSH to vCenter server appliance.
• Login to vCSA using the root credentials.
• Enabled "shell".
• Move the /var/vmware/applmgmt/backupRestore-history.json file to /var/tmp/.
• Restart the vCenter Server Appliance.

Hope this helps other folks in VMware community.    

What is vCenter PNID?

Today I have got the question what is PNID in vCenter.

Well, PNID (primary network identifier) is a VMware internal term and it is officially called "system name".

But my question to the questioner was why he needs to know something about PNID. I have got the expected answer. The questioner did some research how to change vCenter IP address and hostname.

So let's discuss these two requests independently.

First thing first, vCenter hostname cannot be changed. At least for vCenter 6.0 and 6.5. It may or may not change in the future.

On the other hand, vCenter IP can be changed.  However, system name (aka PNID) is very important when you are trying to change vCenter IP address. vCenter IP address can be changed only when you have entered FQDN during vCenter installation. In such case, PNID is the hostname. In case, you did not enter FQDN during vCenter installation, the IP address is used as PNID which would end up in the inability to change the vCenter IP Address.

Below is the command how to check in VCSA what is your vCenter PNID.

root@vc01 [ ~ ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

vc01.home.uw.cz

root@vc01 [ ~ ]# 

In this case above, PNID is the hostname (vc01.home.uw.cz) so I would be able to change IP address.

Deploying vCenter High Availability with network addresses in separate subnets

VMware vCenter High Availability is a very interesting feature included in vSphere 6.5. Generally, it provides higher availability of vCenter service by having three vCenter nodes (active/passive/witness) all serving the single vCenter service.

This is written in the official vCenter HA documentation

vCenter High Availability (vCenter HA) protects vCenter Server Appliance against host and hardware failures. The active-passive architecture of the solution can also help you reduce downtime significantly when you patch vCenter Server Appliance.
After some network configuration, you create a three-node cluster that contains Active, Passive, and Witness nodes. Different configuration paths are available.   

The last sentence is very true. The simplest VCHA deployment is within the same SSO domain and within the single datacenter with two Layer2 networks, one for management and second for the heartbeat. Such design can be deployed in fully automated manner and you just need to provide dedicated network (portgroup/VLAN) for the heartbeat network and use 3 IP addresses from separated heartbeat subnet. Easy. But is it what you are expecting from vCenter HA? To be honest, the much more attractive use case is to spread vCenter HA nodes across three datacenters to keep vSphere management up and running even one of two datacenters experiences some issue. Conceptually it is depicted in the figure below.

Conceptual vCenter HA Design

In this particular concept, I have embedded PSC controllers because of simplicity and vCenter HA can increase availability even of PSC services. The most interesting challenge in this concept is networking so let's look into the intended network logical design.

vCenter HA - networking logical design

Networking logical design:

• Each vCenter Server Appliance node has two NICs
• One NIC is connected to management network and second NIC to heartbeat network
• Layer 2 Management network (VLAN 4) is stretched across datacenters A and B because vCenter IP address must work without human intervention in datacenter B after VCHA fail-over.
• In each datacenter we have independent heartbeat network (VCHA-HB-A, VCHA-HB-B, VCHA-HB-C) with different IP subnets to not stretch Layer 2 across datacenters, especially not to datacenter C where is the witness. This requires specific static routes in each vCenter Server Appliance node to have IP reachability over heartbeat network.
• Specific VCHA network tcp/udp ports must be allowed among VCHA nodes across a heartbeat network.

Helpful documents:

• FAQ: vCenter High Availability (2148003) - https://kb.vmware.com/s/article/2148003
• Deploying vCenter High Availability with network addresses in separate subnets (2148442) - https://kb.vmware.com/kb/2148442
• Required ports for VCHA 6.5 (52835) - https://kb.vmware.com/kb/52835
• Errors "Failed to ssh connect peer node x.x.x.x" and "sshConnect Authentication (publickey) failed" while configuring vCenter HA (2150715) - https://kb.vmware.com/kb/2150715

Implementation Notes: 

Note 1:

VMware KB 2148442 (Deploying vCenter High Availability with network addresses in separate subnets) is very important to deploy such design but one information is missing there. After cloning of vCenter Server Appliances, you have to go to passive node and configure on eth0 the same IP address you use in active node. Configuration is in file  /etc/systemd/network/10-eth0.network.manual

Note 2:

In case of badly destroyed VCHA cluster use following commands to destroy VCHA from the command line

cd /etc/systemd/network 

mv 10-eth0.network.manual 20-eth0.networkdestroy-vchareboot

The solution was found at https://communities.vmware.com/thread/552084

Link to the official documentation (Resolving Failover Failures) - https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.avail.doc/GUID-FE5106A8-5FE7-4C38-91AA-D7140944002D.html

Note 3:

In case, you will see the error message MethodFault.summary error during the finalization process it is because a hostname mismatch is detected. The hostname assigned to the Passive node must be the same as the hostname of the Active node. The solution was found  at https://www.altaro.com/vmware/how-to-deploy-a-vcenter-ha-cluster-part-2/ but also written in KB https://kb.vmware.com/kb/2148442

How to downsize vCenter Server Appliance 6.5 storage?

Last week I have been asked by one partner how to downsize vCenter Server Appliance (VCSA) 6.5 storage.

Well, let's start with upsizing. To add CPU and RAM resources is very easy. VCSA 6.5 supports CPU Hot Add and Memory Hot Plug, therefore you do not need to even shut down VCSA to increase CPU and RAM resources.

CPU Hot Add and RAM Hot Plug

Storage expansion though is a little bit more difficult. You still do not have to shut down VCSA because virtual disk can hot-extended, however after a disk is extended you have to grow disk partitions within the operating system, Photon OS in this particular case. William Lam wrote the blog post here about it. Generally you have to run script  /usr/lib/applmgmt/support/scripts/autogrow.sh within VCSA shell so it is not a rocket science you just need to know what script to execute.

So upsize is easily doable. But what about downsize? VCSA 6.5 supports CPU Hot Remove but RAM cannot be downsized, therefore for RAM downsizing you have to shut down VCSA decrease memory resources and power on VM. Not a big deal, just small downtime so it can be done during a maintenance window. But the storage downsize is not possible. Well, in theory, it is possible but it is definitely not supported to decrease the size of disk partitions and the virtual disk itself because it is hard and also very risky as you do not know where data are located within the disk.

Warning: I have been told by someone that downsizing method described below does not work and restore options will allow you to choose just bigger form factor than originally backed up  VCSA. Unfortunately, I have the smallest form factor in my home lab so I cannot verify it but I believe him. Sorry for the misleading idea but there might be some unsupported method how to tweak backup files to have the impression it is a backup from smaller VCSA form factor.  

We have another downsizing option for VCSA 6.5. You are most probably aware that VCSA 6.5 has introduced application-based backup where vCenter inventory, identity, and even the database is backed up to a remote location via following protocols HTTP, HTTPS, SCP, FTP, FTPS. The restore of VCSA 6.5 backup is done as a new VCSA deployment executed in a restore mode where a previously created backup is used as a restore point. The nice thing is, that during VCSA restore you can choose different VCSA form factor with different storage footprint. So this is a potential way how to downsize vCenter Server Appliance 6.5 storage in a supported way. The only assumption is that your backup data will fit into new storage size.

If you will plan such downsizing exercise, please do not forget to keep your original vCenter Server Appliance somewhere to have a simple way how to roll back.

Hope this is helpful and informative.

Start order of software services in VMware vCenter Server Appliance 6.0 U2

vCenter Server Appliance 6.0 U2 services are started in the following order ...

1. vmafdd (VMware Authentication Framework)
2. vmware-rhttpproxy (VMware HTTP Reverse Proxy)
3. vmdird (VMware Directory Service)
4. vmcad (VMware Certificate Service)
5. vmware-sts-idmd (VMware Identity Management Service)
6. vmware-stsd (VMware Security Token Service)
7. vmware-cm (VMware Component Manager)
8. vmware-cis-license (VMware License Service)
9. vmware-psc-client (VMware Platform Services Controller Client)
10. vmware-sca (VMware Service Control Agent)
11. applmgmt (VMware Appliance Management Service)
12. vmware-netdumper (VMware vSphere ESXi Dump Collector)
13. vmware-syslog (VMware Common Logging Service)
14. vmware-syslog-health (VMware Syslog Health Service)
15. vmware-vapi-endpoint (VMware vAPI Endpoint)
16. vmware-vpostgres (VMware Postgres)
17. vmware-invsvc (VMware Inventory Service)
18. vmware-mbcs (VMware Message Bus Configuration Service)
19. vmware-vpxd (VMware vCenter Server)
20. vmware-eam (VMware ESX Agent Manager)
21. vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter)
22. vmware-sps (VMware vSphere Profile-Driven Storage Service)
23. vmware-vdcs (VMware Content Library Service)
24. vmware-vpx-workflow (VMware vCenter Workflow Manager)
25. vmware-vsan-health (VMware VSAN Health Service)
26. vmware-vsm (VMware vService Manager)
27. vsphere-client ()
28. vmware-perfcharts (VMware Performance Charts)
29. vmware-vws (VMware System and Hardware Health Manager) 

Maximum client sessions vCenter server can accept

I work as VMware TAM (Technical Account Manager) and one my customer had recently significant incident when clients (vSphere admins) was not able connect to vCenter server. It did not work nighter from old C# client nor new Web Client. It was interesting that sometimes some admins were able to connect and stay connected but others where not able to connect.

The error message was very general saying ...

Call "ServiceInstance.RetrieveContent" for object "ServiceInstance" on Server "vc01.example.com" failed.

C# Client returned another further explanation ...

The server 'vc01.example.com' could not interpret the client's request. (The remote server returned an error: (503) Server Unavailable.) 

See error messages in screenshot below ...

C# Client error messages

As you can see, both error messages are very general and further holistic troubleshooting was necessary. After multiple theories, one customer's vSphere/Windows administrator did a Windows OS analysis with Windows perfmon tool and realized that during the incident there were more then 1400 open threads with client connections to vCenter server. This turned in to the hypothesis that we have reached the maximum of client sessions vCenter can accept.

The hypothesis is always very important but even more important is the proof that hypothesis is valid and it is the root cause of particular issue.

Unfortunately, the maximum of total client sessions to vCenter server is not documented. The only numbers documented in "Configuration Maximums - vSphere 5.5" are ..

Concurrent vSphere Client connections to vCenter Server = 100Concurrent vSphere Web Clients connections to vCenter Server = 180

However, my customer is using automation extensively, therefore PowerCLI can have additional connections. The only way how to know the maximum is to test it.

My customer is still on vCenter 5.5 but I have prepared and executed the test in my home lab where I have vCenter 6.0 U2. I prepared PowerCLI script to create 2000 new client sessions and keep sessions open. The purpose of script is to find the maximum of established sessions vCenter can accept and see what will be the error message when maximum will be achieved.

The PowerCLI script is available on GitHub here
https://github.com/davidpasek/powercli-scripts/blob/master/vcenter-sessions.ps1
and it is based on excellent blog post and scripts "List and Disconnect vCenter Sessions" prepared by Alan Renouf.

I run the script in my lab and waited when it fails to find the maximum. You can see the expected failure on screenshot below ...

Expected connection failure to find what is the maximum

And the result is ...

vCenter Server 6.0 U2 accepts maximally 1995 established client sessions

When the above maximum is exceeded you are not able to connect to vCenter server any more and you will see the error messages mentioned at the beginning of this article.

Business impact and visibility

It is good to mention that this technical issue was observed during Disaster Recovery fail-over test and it silently disappeared after fail back of all services. That's the reason why this incident had very high internal business visibility and the issue was escalated to top IT management which required very quick Root Cause Analysis and proper problem management.

That's just another proof how vCenter and vSphere platform is critical in modern IT environments.

It seems, that my customer is using some automation script which establish connection to vCenter server, but because of some circumstances which happening only when services are running on disaster recovery backup site, the script does not disconnect sessions and the vCenter server maximum is exceeded and it does not accept any new connections. In such situation, vSphere platform is unmanageable.

This is good to know, especially in the age of automation, where single badly written automation script, can crash vSphere manageability.

As VMware TAM, I can communicate and justify my customer's product feature requests internally inside VMware organization.  That's another benefit of VMware TAM Program.

So here is publicly written vCenter Product Feature Request which I will open with our Product Management.

Feature Request: Maximum of supported client sessions should be documented in "vSphere Configuration Maximums" document. When the maximum is exceeded, vCenter server should accept at least one more connection for vSphere Administrator (for example administrator@vsphere.local) which should be used as last resort or back door if you wish. Such special "back door" connection should be terminated and re-established by the most recent connection of vSphere Admin to allow manageability in such situation.

How to show vCenter Instance configuration?

Login to vCenter Server Appliance (VCSA) via ssh.

Enable BASH access: "shell.set --enabled True"
Launch BASH: "shell"

Run following command to list vCenter Instance configuration.

vc01:/etc/vmware-vpx # cat /etc/vmware-vpx/instance.cfg 
applicationDN=dc\=virtualcenter,dc\=vmware,dc\=int
instanceUuid=b7cc1468-6d27-4117-943f-7b1b4485028b
ldapPort=389
ldapInstanceName=VMwareVCMSDS
ldapStoragePath=/etc/vmware-vpx/

vCenter UUID is very important identifier which is unique identification of particular instance in external systems like Vmware Platform Service Controller (PSC), vROps, SRM, etc.

UUID is in our example b7cc1468-6d27-4117-943f-7b1b4485028b

How to identify from the guest OS on which vCenter is virtual machine registered?

One my customer asked me how to identify - from the VM guest operating system - in which vCenter server is that particular virtual machine registered.

They use VM deployment from VM Templates with Customization Specifications and they would like to use vCenter locality information for additional tasks during VM deployment process.

I was thinking about several possibilities. Considered options are listed below.

Considered options:

• OPTION 1: Define specific customization profile for each vCenter and have a special guest OS specific command in Customization Specification to run after sysprep and save vCenter identification somewhere to the guest file system.
• OPTION 2: Use VM mac address for identification of vCenter Server Instance.
• OPTION 3: leverage PowerCLI or vCLI running in guest os to communicate with vCenter.
• OPTION 4: leverage custom VM guestinfo properties which can be read inside Guest OS.

Option 3 is not good option at all because you would need to have network connectivity from production VMs to vCenter (management network) and therefore it has negative impact on overall security.

Option 4 is described by William Lam here. It would need special VM templates having custom VM property like guestinfo.vcenter=VC01 which is visible in the guest info through vmtools. The command in the guest would look like

vmtoolsd --cmd "info-get guestinfo.vcenter"

Option 1 is relatively easy and it is leveraging the fact that Customization Specification for deployment of VM templates can run some script in guest after template deployment. I think that Option 1 is relatively good option. The only drawback is that vSphere admin would need to manage more customization specification and specific scripts to store vCenter identification somewhere in guest os filesystem which introduces some additional management overhead but it is acceptable if you ask me.

Option 2 intrigued me technically so let's elaborate on this option. Option 2 is leveraging the fact that "vCenter Server instance ID" is used for generating virtual machine MAC addresses and MAC address is well known digital identifier which can be relatively simply identified in any operating system.  So what this "vCenter Server instance ID" is? Each vCenter Server system has a vCenter Server instance ID. This ID is a number between 0 and 63 which is randomly generated at installation time, but can be reconfigured after installation. Here in vSphere 6.0 documentation is written that ... According to this scheme, a MAC address has the following format:

00:50:56:XX:YY:ZZ
where 00:50:56 represents the VMware OUI,
XX is calculated as (80 + vCenter Server Instance ID),
and YY:ZZ is a random number.

Note 1:

The formula above (80 + vCenter Server Instance ID) is in hexadecimal format therefore in decimal format it is 128 + vCenter Server Instance ID.

Note 2:

vCenter Server unique ID is generated randomly during vCenter installation. It can be changed after installation in the Runtime Settings section from the General settings of the vCenter Server instance and restart it. Please be aware, that existing Virtual Machines MAC addresses are not changed automatically after ID reconfiguration therefore it is good idea to change vCenter Server unique ID immediately after vCenter Server installation. There are methods how to regenerate VM mac addresses but it requires VM downtime. For more information look at VMware KB 1024025. 

Below is PowerShell script example of in-guest calculation of vCenter Server Instance ID.

$mac_str = Get-CimInstance win32_networkadapterconfiguration | where {$_.ServiceName -eq "vmxnet3ndis6"} | select macaddress | Out-String
$mac_arr = $mac_str.split(':')
$XX_hex = $mac_arr[3];
$XX_dec = [Convert]::ToInt32($XX, 16)
$VC_instance_ID = $XX_dec - 128
$VC_instance_ID

The script above is just an example written for Windows OS (Win2012R2) and PowerShell (4.0) to show how to automate the trick described in this blog post. Similar scripts can be prepared for other guest operating systems.

Disclaimer:

The script above is just an example and it works in my lab environment. You should carefully test if your script inspired by this blog post works correctly in your particular environment. I don't take any responsibility for the script and you use it in your own risk. I have spent just few minutes to write this script and I would definitely recommend to invest some more time on development and test if you want to use such script in production environment.

Know caveats of option 2 (vCenter identification based on VM MAC address) :

• This solution will only work for dynamically assigned MAC addresses by vCenter and not for statically configured MAC addresses by administrator
• This solution will not work correctly for cross vCenter vMotioned VMs because they are keeping the MAC address from original vCenter
• I didn't test how behaves VM's recovered by VMware SRM (Site Recovery Manager). If recovered VM's keep original MAC address then this solution will not work for these recovered VMs. Unfortunately, I don't have access to SRM lab to verify SRM behavior. 

I would recommend to my customer to consider between options (1) and (2).

Hope this helps to broader IT community and as always ... your feedback is very welcome so don't hesitate to use comments, twitter or e-mail to share your opinions and other solution alternatives.

SQL commands for information of vCenter Server Database usage

Let's assume we have simple installation of vCenter Server database leveraging MS SQL Express and we want to know how much database space is currently used. The simplest way is to use existing sqlcmd program. Connect to MS Windows server where vCenter is installed. Open command prompt or PowerShell and use following SQL commands ...

sqlcmd -E -Slocalhost\VIM_SQLEXP
1>use VIM_VCDB
2> go
Changed database context to 'VIM_VCDB'.
1> sp_spaceused
2> go
database_name
         database_size      unallocated space
---------------------------------------------------------------------------------------- ------------------ ------------------
VIM_VCDB
         244.94 MB          0.93 MB
reserved           data               index_size         unused
------------------ ------------------ ------------------ ------------------
234824 KB          152800 KB          63200 KB           18824 KB

1>

There you can see that VIM_VCDB database size is 244.94 MB.

Note: In this particular environment I have just two ESX hosts with eight virtual machines.

vCenter SSO: Active Directory as a LDAP Server

Recently I had a need to use secondary Active Directory (VPOD02.example.com) to my vCenter SSO in the lab which is already integrated with Active Directory (VPOD01.example.com).

Here are several facts just to give you brief overview of my lab.

I have two independent vPODs in my lab. Each vPOD has everything what's needed for VMware vSphere infrastructure. I have there dedicated hardware (Compute, Storage, Network), vSphere components like vCenter, SSO, ESXi hosts, Site Recovery Manager, vSphere Replication Appliance, and also Domain Controllers and DNS servers.

vCenter SSO placed in VPOD01 is using Integrated Windows Authentication with Microsoft Active Directory "VPOD01.example.com". Therefore another integration with Microsoft Active Directory "VPOD02.example.com" can be done only via LDAP. Configuration of additional identity source is depicted on the screenshot below.

SSO: Add identity source

Identity source type: Active Directory as a LDAP Server
Identity source settings:
  Name: vpod02.example.com
  Base DN for users: dc=vpod02,dc=example,dc=com
  Domain name: vpod02.example.com
  Domain alias: vpod02
  Base DN for groups: dc=vpod02,dc=example,dc=com
  Primary server URL: ldap://10.2.22.51:389
  Secondary server URL: empty
  Username: administrator@vpod02.example.com

I know that two Microsoft domains can be integrated in to the single "Domain Trust" but because I'm not to much familiar and experienced with Microsoft Active Directory I think that vCenter Single Sign-On capability of multiple identity sources is another nice design option.

Simpler manageability for non-Microsoft oriented vSphere Admin was the primary reason and justification to use this option in my vSphere lab :-)

vCenter, Windows 2012 R2, .NET 3.5 issue

It is well know that vCenter Server 5.5 requires .NET Framework 3.5. It is quite easy to install it by Server Manager GUI or by following command: 

dism /online /enable-feature /featurename:NetFX3 /all /Source:d:\sources\sxs /LimitAccess

Command above assumes Windows 2012 DVD in drive d:
 
... but i had an issue with installation getting following error.

PS C:\Users\Administrator> dism /online /enable-feature /featurename:NetFX3 /all /Source:d:\sources\sxs /LimitAccess

Deployment Image Servicing and Management tool
Version: 6.3.9600.17031

Image Version: 6.3.9600.17031

Enabling feature(s)
[===========================66.4%======                    ]

Error: 0x800f081f

The source files could not be found.
Use the "Source" option to specify the location of the files that are required to restore the feature. For more informat
ion on specifying a source location, see http://go.microsoft.com/fwlink/?LinkId=243077.

The DISM log file can be found at C:\Windows\Logs\DISM\dism.log
PS C:\Users\Administrator>

I discuss this issue with our Microsoft Specialist and he already knew the root cause and fix. The root cause was some bad Windows update. It is already fixed by Microsoft and if you didn't do update in bad time you should not experience this issue. However, when you hit this bug the only solution is to run following Microsoft fix.  

NDPFixit-KB3005628-X64.exe

http://go.microsoft.com/fwlink/?LinkId=513775

Some more information about this issue:

KB: http://support2.microsoft.com/kb/3005628

HowTo

http://blogs.technet.com/b/askpfeplat/archive/2014/09/29/attempting-to-install-net-framework-3-5-on-windows-server-2012-r2-fails-with-error-code-0x800f0906-or-the-source-files-could-not-be-downloaded-even-when-supplying-source.aspx

Performance Data charts for datastore LUNs report the message: No data available

Performance Data charts for datastore LUNs are extremely useful to have clue to understand storage performance trend.

However sometimes you can see message like this

"Performance Data charts for datastore LUNs report the message: No data available"

I didn't know the root cause. Recently colleague of mine told me he has found what is the root cause which is described at VMware KB 2054403.

Workaround is to not use network virtual adapter E1000E. If you have larger environment it's not big fun to search these adapters. My colleague wrote useful PowerCLI one-liner to find VM with E1000E which should be manually changed. Here is the my colleague's script:

Get-VM | Get-NetworkAdapter | Where-object {$_.Type -like "Unknown" -or $_.Type -like "E1000E" } | Select @{N="VM";E={$_.Parent.Name}},Name,Type | export-Csv  c:\VM-Network_Interface.csv -NoTypeInformation 

He asked me to share this information with community so enjoy it.

Working with VCSA embedded database

It's not often but sometimes you have to work with vCenter database. Usually it should be done only if you are instructed by VMware Support or there is VMware KB article (like this one http://kb.vmware.com/kb/1005680) solving your problem.

Please do it very carefully in production systems.

VMware vSphere admin veterans usually have experience with MS-SQL but what about vCenter Server Appliance (VCSA) with embedded database? It is not very different. VMware uses Postgresql database (aka vPostgres) so logically it is the same as in any other SQL database. I would say even easier than in MS-SQL but that's highly dependent on administrator background and previous experience. I'm probably biases due to my *nix history and open-source (GNU) general preference.

Here are basic logical steps how to work with vCenter database.

• Connect to database server
• Discover database tables
• Issue SQL commands
• Exit from database server

CONNECT TO DATABASE SERVER

Change working directory to vpostgres

cd /opt/vmware/vpostgres/current/bin/

Display database configuration

cat /etc/vmware-vpx/embedded_db.cfg

output should looks like

EMB_DB_INSTALL_DIR='/opt/vmware/vpostgres/9.0'
EMB_DB_TYPE='PostgreSQL'
EMB_DB_SERVER='127.0.0.1'
EMB_DB_PORT='5432'
EMB_DB_INSTANCE='VCDB'
EMB_DB_USER='vc'
EMB_DB_PASSWORD='WZL2^y<-k8boy br="" fa="">EMB_DB_STORAGE='/storage/db/vpostgres'

connect to database

./psql VCDB -U vc

Update 2015-09-15: For VCSA 6 use /opt/vmware/vpostgres/current/bin/psql 
-d VCDB -U postgres (password is not required)

and you are in.

DISCOVER DATABASE TABLES

It's really good to know what tables are in the database. You need table names to compose SQL commands allowing you to select, insert and update data in the database.

Postgresql have special DBA (database administrator) commands witch start with character \ (slash). You can list all DBA commands by sequence \?

The output looks like this

vc01:/opt/vmware/vpostgres/current/bin # ./psql VCDB -U vc
psql.bin (9.0.13)
Type "help" for help.

VCDB=> \?
  \d[S+]                 list tables, views, and sequences
  \d[S+]  NAME           describe table, view, sequence, or index
  \da[S]  [PATTERN]      list aggregates
  \db[+]  [PATTERN]      list tablespaces
  \dc[S]  [PATTERN]      list conversions
  \dC     [PATTERN]      list casts
  \dd[S]  [PATTERN]      show comments on objects
  \ddp    [PATTERN]      list default privileges
  \dD[S]  [PATTERN]      list domains
  \des[+] [PATTERN]      list foreign servers
  \deu[+] [PATTERN]      list user mappings
  \dew[+] [PATTERN]      list foreign-data wrappers

We want list database tables so the command we are looking for is

\dt

where output looks like

                    List of relations
 Schema |              Name              | Type  | Owner
--------+--------------------------------+-------+-------
 vpx    | vpx_access                     | table | vc
 vpx    | vpx_alarm                      | table | vc
 vpx    | vpx_alarm_action               | table | vc
 vpx    | vpx_alarm_disabled_actions     | table | vc
 vpx    | vpx_alarm_expr_comp            | table | vc
 vpx    | vpx_alarm_expression           | table | vc
 vpx    | vpx_alarm_repeat_action        | table | vc
 vpx    | vpx_alarm_runtime              | table | vc
 vpx    | vpx_alarm_state                | table | vc
 vpx    | vpx_binary_data                | table | vc
 vpx    | vpx_bulletin_operation         | table | vc
 vpx    | vpx_change_tag                 | table | vc
 vpx    | vpx_compliance_status          | table | vc
 vpx    | vpx_compute_res_failover_host  | table | vc
 vpx    | vpx_compute_res_user_hb_ds     | table | vc
 vpx    | vpx_compute_resource           | table | vc
 vpx    | vpx_compute_resource_das_vm    | table | vc
 vpx    | vpx_compute_resource_dpm_host  | table | vc
 vpx    | vpx_compute_resource_drs_vm    | table | vc
 vpx    | vpx_compute_resource_vsan_host | table | vc

ISSUE SQL COMMANDS

If we want select and view some data from database we use SQL statement SELECT. As an example  we will use first table from the list an it is vpx_access. Table vpx_access contains all vCenter users/groups who has access to vCenter and their roles. Here is SELECT statement:

select * from vpx_access

and output

 id  |          principal          | role_id | entity_id | flag
-----+-----------------------------+---------+-----------+------
   1 | root                        |      -1 |         1 |    1
 101 | VSPHERE.LOCAL\Administrator |      -1 |         1 |    1
 201 | VPOD01\vsphere-admins       |      -1 |         1 |    3
(3 rows)

Update and delete statements can be composed in similar manner following ANSI SQL Standard. Postgresql is ANSI-SQL:2008 standard.

EXIT FROM DATABASE SERVER

To exit from database server simply use DBA command \q

That's it pretty easy, isn't it? Working with vCenter database is not daily task of vSphere admin however we all know that sometimes you can be instructed by VMware support or KB to change something in the database. Don't be afraid - it's easy.

vCenter Single Sign-On Design Decision Point

When you designing vSphere 5.1 you have to implement vCenter SSO. Therefore you have to make design decision what SSO mode to choose.

There are actually three available options

1. Basic
2. HA (don't mix with vSphere HA)
3. Multisite

Justin King wrote excellent blog post about SSO here and it is worth source of information to make right design decision. I fully agree with Justin and recommending Basic SSO to my customers if possible. SSO Server protection  can be achieved by standard backup/restore methods and SSO High Availability can be increased by vSphere HA. All these methods are well known and long time used.

You have to use Multisite SSO when vCenter linked-mode is required but think twice if you really need it and benefits overweight drawbacks.

Reduced vCenter DB by deleting old events and tasks from vCenter database

In vCenter MS-SQL Database is storage procedure called cleanup_events_tasks_proc which deletes old data based on event and task retention settings. vCenter retention settings can be setup in vCenter Settings though vSphere Client or changed directly in database. Using vSphere Client  is recommended.

Following example is copied from: http://communities.vmware.com/thread/191227?start=0&tstart=0

c:> "C:\Program Files\Microsoft SQL Server\90\Tools\Binn\OSQL.EXE" -S \SQLEXP_VIM -E
1> use VIM_VCDB
2> go
1> update vpx_parameter set value='' where name='event.maxAge'
2> update vpx_parameter set value='' where name='task.maxAge'
3> update vpx_parameter set value='true' where name='event.maxAgeEnabled'
4> update vpx_parameter set value='true' where name='task.maxAgeEnabled'
5> go
(1 row affected)
(1 row affected)
(1 row affected)
(1 row affected)
1> exec cleanup_events_tasks_proc
2> go
1> dbcc shrinkdatabase ('VIM_VCDB')
2> go
DbId   FileId      CurrentSize MinimumSize UsedPages   EstimatedPages
------ ----------- ----------- ----------- ----------- --------------
      5           1       81080         280       78776          78776
      5           2         128         128         128            128

(2 rows affected)
DBCC execution completed. If DBCC printed error messages, contact your system
administrator.
1> quit

How to get Managed Object Reference ID ( aka MoRef ) from vSphere

If you've already scripted vSphere infrastructure you probably already know that everything has software representation also known as Managed Object. Each Managed Object has unique identifier referenced as Managed Object ID. Sometimes this Managed Object ID is needed.

In PowerCLI you can get it via following two lines

$VM = Get-VM -Name $VMName 

$VMMoref = $VM.ExtensionData.MoRef.Value

You can also use Perl script leveraging VMware vSphere Perl SDK to get Managed Object ID for particular virtual machine or datastore. If you need MOID for another entity it's pretty easy to slightly change the script below.

Script is developed and tested on vMA (VMware management Assistant) in directory /usr/lib/vmware-vcli/apps/general and script name is getmoid.pl

Here is usage example how to get MOID of datastore called FreeNAS-iSCSI-01:
./getmoid.pl --server --username --password --dsname FreeNAS-iSCSI-01

Manage Object ID: datastore-162

Here is usage example how to get MOID of virtual machine called VMA:
./getmoid.pl --server --username --password --vmname VMA

Manage Object ID: vm-122

Any feedback or comments are welcome.

vCenter Server 5.1 Required Ports

http://www.vladan.fr/vcenter-server-5-1-required-ports/

The Trouble with CA SSL Certificates and vCenter 5

http://longwhiteclouds.com/2012/02/07/the-trouble-with-ca-ssl-certificates-and-vcenter-5/

http://www.wooditwork.com/2011/11/30/vsphere-5-certificates-1-installing-a-root-certificate-authority-3/

Migrating to vCenter Server 5.0 using the Data Migration Tool

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2005328

VMware vCenter Orchestrator

http://www.vcoteam.info/newsflash/vmware-released-the-vcenter-orchestrator-plug-in-update-for-vcloud-director-15.html