VCSA - This appliance cannot be used or repaired ...

I have just got an email from my customer describing the weird issue with VMware vCenter Server Appliance (aka VCSA).

The customer is doing weekly native backups of VCSA manually via VAMI. He wanted to run VCSA native backup again but when he tried to log into virtual appliance management interface (VAMI) he is getting the following error message
 

Error message - This appliance cannot be used or repaired because of failure was encountered. You need to deploy a new appliance.

 

The error message includes a resolution. Deploy a new appliance. The recommended solution is the last thing a typical vSphere admin would like to resolve such an issue. Fortunately enough, there is another solution/workaround.

To resolve this issue stop and start all the services on the vCSA,

• Putty/SSH to vCenter server appliance.
• Login to VCSA using the root credentials.
• Enabled "shell".
• Restart VCSA services

To restart VCSA services run the following commands:
service-control --stop --all
service-control --start --all

In case, simple services restart does not help, you can have an issue with some recent backup job. In such a case, there is another resolution with an additional workaround

• Putty/SSH to vCenter server appliance.
• Login to vCSA using the root credentials.
• Enabled "shell".
• Move the /var/vmware/applmgmt/backupRestore-history.json file to /var/tmp/.
• Restart the vCenter Server Appliance.

Hope this helps other folks in VMware community.    

How to downsize vCenter Server Appliance 6.5 storage?

Last week I have been asked by one partner how to downsize vCenter Server Appliance (VCSA) 6.5 storage.

Well, let's start with upsizing. To add CPU and RAM resources is very easy. VCSA 6.5 supports CPU Hot Add and Memory Hot Plug, therefore you do not need to even shut down VCSA to increase CPU and RAM resources.

CPU Hot Add and RAM Hot Plug

Storage expansion though is a little bit more difficult. You still do not have to shut down VCSA because virtual disk can hot-extended, however after a disk is extended you have to grow disk partitions within the operating system, Photon OS in this particular case. William Lam wrote the blog post here about it. Generally you have to run script  /usr/lib/applmgmt/support/scripts/autogrow.sh within VCSA shell so it is not a rocket science you just need to know what script to execute.

So upsize is easily doable. But what about downsize? VCSA 6.5 supports CPU Hot Remove but RAM cannot be downsized, therefore for RAM downsizing you have to shut down VCSA decrease memory resources and power on VM. Not a big deal, just small downtime so it can be done during a maintenance window. But the storage downsize is not possible. Well, in theory, it is possible but it is definitely not supported to decrease the size of disk partitions and the virtual disk itself because it is hard and also very risky as you do not know where data are located within the disk.

Warning: I have been told by someone that downsizing method described below does not work and restore options will allow you to choose just bigger form factor than originally backed up  VCSA. Unfortunately, I have the smallest form factor in my home lab so I cannot verify it but I believe him. Sorry for the misleading idea but there might be some unsupported method how to tweak backup files to have the impression it is a backup from smaller VCSA form factor.  

We have another downsizing option for VCSA 6.5. You are most probably aware that VCSA 6.5 has introduced application-based backup where vCenter inventory, identity, and even the database is backed up to a remote location via following protocols HTTP, HTTPS, SCP, FTP, FTPS. The restore of VCSA 6.5 backup is done as a new VCSA deployment executed in a restore mode where a previously created backup is used as a restore point. The nice thing is, that during VCSA restore you can choose different VCSA form factor with different storage footprint. So this is a potential way how to downsize vCenter Server Appliance 6.5 storage in a supported way. The only assumption is that your backup data will fit into new storage size.

If you will plan such downsizing exercise, please do not forget to keep your original vCenter Server Appliance somewhere to have a simple way how to roll back.

Hope this is helpful and informative.

Start order of software services in VMware vCenter Server Appliance 6.0 U2

vCenter Server Appliance 6.0 U2 services are started in the following order ...

1. vmafdd (VMware Authentication Framework)
2. vmware-rhttpproxy (VMware HTTP Reverse Proxy)
3. vmdird (VMware Directory Service)
4. vmcad (VMware Certificate Service)
5. vmware-sts-idmd (VMware Identity Management Service)
6. vmware-stsd (VMware Security Token Service)
7. vmware-cm (VMware Component Manager)
8. vmware-cis-license (VMware License Service)
9. vmware-psc-client (VMware Platform Services Controller Client)
10. vmware-sca (VMware Service Control Agent)
11. applmgmt (VMware Appliance Management Service)
12. vmware-netdumper (VMware vSphere ESXi Dump Collector)
13. vmware-syslog (VMware Common Logging Service)
14. vmware-syslog-health (VMware Syslog Health Service)
15. vmware-vapi-endpoint (VMware vAPI Endpoint)
16. vmware-vpostgres (VMware Postgres)
17. vmware-invsvc (VMware Inventory Service)
18. vmware-mbcs (VMware Message Bus Configuration Service)
19. vmware-vpxd (VMware vCenter Server)
20. vmware-eam (VMware ESX Agent Manager)
21. vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter)
22. vmware-sps (VMware vSphere Profile-Driven Storage Service)
23. vmware-vdcs (VMware Content Library Service)
24. vmware-vpx-workflow (VMware vCenter Workflow Manager)
25. vmware-vsan-health (VMware VSAN Health Service)
26. vmware-vsm (VMware vService Manager)
27. vsphere-client ()
28. vmware-perfcharts (VMware Performance Charts)
29. vmware-vws (VMware System and Hardware Health Manager) 

How to manage VCSA services via CLI

VMware vCenter Server Appliance (aka VCSA) is composed from several services. These services are  manageable through Web Client but in case you would need or want to use CLI here are some tips.

First of all you have to connect to VCSA via ssh and enable shell.

shell.set –enabled True
shell

Run the below command to list the services currently present on the VCSA.

service-control --list

If you want to check the status of the services, then run the below command.

service-control --status

Command above will list all the services that are present on the VCSA, even the ones that are not running listed at the end.

If you want start particular service you will have to use following syntax

service-control --start

to stop service

service-control --stop

If you want to start or stop all service use following commands

service-control --start --all
service-control --stop --all

Not very difficult, right?

VCSA is the only VMware vSphere management of the future, so don't be afraid and go VCSA!

How to show vCenter Instance configuration?

Login to vCenter Server Appliance (VCSA) via ssh.

Enable BASH access: "shell.set --enabled True"
Launch BASH: "shell"

Run following command to list vCenter Instance configuration.

vc01:/etc/vmware-vpx # cat /etc/vmware-vpx/instance.cfg 
applicationDN=dc\=virtualcenter,dc\=vmware,dc\=int
instanceUuid=b7cc1468-6d27-4117-943f-7b1b4485028b
ldapPort=389
ldapInstanceName=VMwareVCMSDS
ldapStoragePath=/etc/vmware-vpx/

vCenter UUID is very important identifier which is unique identification of particular instance in external systems like Vmware Platform Service Controller (PSC), vROps, SRM, etc.

UUID is in our example b7cc1468-6d27-4117-943f-7b1b4485028b

Working with VCSA embedded database

It's not often but sometimes you have to work with vCenter database. Usually it should be done only if you are instructed by VMware Support or there is VMware KB article (like this one http://kb.vmware.com/kb/1005680) solving your problem.

Please do it very carefully in production systems.

VMware vSphere admin veterans usually have experience with MS-SQL but what about vCenter Server Appliance (VCSA) with embedded database? It is not very different. VMware uses Postgresql database (aka vPostgres) so logically it is the same as in any other SQL database. I would say even easier than in MS-SQL but that's highly dependent on administrator background and previous experience. I'm probably biases due to my *nix history and open-source (GNU) general preference.

Here are basic logical steps how to work with vCenter database.

• Connect to database server
• Discover database tables
• Issue SQL commands
• Exit from database server

CONNECT TO DATABASE SERVER

Change working directory to vpostgres

cd /opt/vmware/vpostgres/current/bin/

Display database configuration

cat /etc/vmware-vpx/embedded_db.cfg

output should looks like

EMB_DB_INSTALL_DIR='/opt/vmware/vpostgres/9.0'
EMB_DB_TYPE='PostgreSQL'
EMB_DB_SERVER='127.0.0.1'
EMB_DB_PORT='5432'
EMB_DB_INSTANCE='VCDB'
EMB_DB_USER='vc'
EMB_DB_PASSWORD='WZL2^y<-k8boy br="" fa="">EMB_DB_STORAGE='/storage/db/vpostgres'

connect to database

./psql VCDB -U vc

Update 2015-09-15: For VCSA 6 use /opt/vmware/vpostgres/current/bin/psql 
-d VCDB -U postgres (password is not required)

and you are in.

DISCOVER DATABASE TABLES

It's really good to know what tables are in the database. You need table names to compose SQL commands allowing you to select, insert and update data in the database.

Postgresql have special DBA (database administrator) commands witch start with character \ (slash). You can list all DBA commands by sequence \?

The output looks like this

vc01:/opt/vmware/vpostgres/current/bin # ./psql VCDB -U vc
psql.bin (9.0.13)
Type "help" for help.

VCDB=> \?
  \d[S+]                 list tables, views, and sequences
  \d[S+]  NAME           describe table, view, sequence, or index
  \da[S]  [PATTERN]      list aggregates
  \db[+]  [PATTERN]      list tablespaces
  \dc[S]  [PATTERN]      list conversions
  \dC     [PATTERN]      list casts
  \dd[S]  [PATTERN]      show comments on objects
  \ddp    [PATTERN]      list default privileges
  \dD[S]  [PATTERN]      list domains
  \des[+] [PATTERN]      list foreign servers
  \deu[+] [PATTERN]      list user mappings
  \dew[+] [PATTERN]      list foreign-data wrappers

We want list database tables so the command we are looking for is

\dt

where output looks like

                    List of relations
 Schema |              Name              | Type  | Owner
--------+--------------------------------+-------+-------
 vpx    | vpx_access                     | table | vc
 vpx    | vpx_alarm                      | table | vc
 vpx    | vpx_alarm_action               | table | vc
 vpx    | vpx_alarm_disabled_actions     | table | vc
 vpx    | vpx_alarm_expr_comp            | table | vc
 vpx    | vpx_alarm_expression           | table | vc
 vpx    | vpx_alarm_repeat_action        | table | vc
 vpx    | vpx_alarm_runtime              | table | vc
 vpx    | vpx_alarm_state                | table | vc
 vpx    | vpx_binary_data                | table | vc
 vpx    | vpx_bulletin_operation         | table | vc
 vpx    | vpx_change_tag                 | table | vc
 vpx    | vpx_compliance_status          | table | vc
 vpx    | vpx_compute_res_failover_host  | table | vc
 vpx    | vpx_compute_res_user_hb_ds     | table | vc
 vpx    | vpx_compute_resource           | table | vc
 vpx    | vpx_compute_resource_das_vm    | table | vc
 vpx    | vpx_compute_resource_dpm_host  | table | vc
 vpx    | vpx_compute_resource_drs_vm    | table | vc
 vpx    | vpx_compute_resource_vsan_host | table | vc

ISSUE SQL COMMANDS

If we want select and view some data from database we use SQL statement SELECT. As an example  we will use first table from the list an it is vpx_access. Table vpx_access contains all vCenter users/groups who has access to vCenter and their roles. Here is SELECT statement:

select * from vpx_access

and output

 id  |          principal          | role_id | entity_id | flag
-----+-----------------------------+---------+-----------+------
   1 | root                        |      -1 |         1 |    1
 101 | VSPHERE.LOCAL\Administrator |      -1 |         1 |    1
 201 | VPOD01\vsphere-admins       |      -1 |         1 |    3
(3 rows)

Update and delete statements can be composed in similar manner following ANSI SQL Standard. Postgresql is ANSI-SQL:2008 standard.

EXIT FROM DATABASE SERVER

To exit from database server simply use DBA command \q

That's it pretty easy, isn't it? Working with vCenter database is not daily task of vSphere admin however we all know that sometimes you can be instructed by VMware support or KB to change something in the database. Don't be afraid - it's easy.