VCSA - This appliance cannot be used or repaired ...

I have just got an email from my customer describing the weird issue with VMware vCenter Server Appliance (aka VCSA).

The customer is doing weekly native backups of VCSA manually via VAMI. He wanted to run VCSA native backup again but when he tried to log into virtual appliance management interface (VAMI) he is getting the following error message
 

Error message - This appliance cannot be used or repaired because of failure was encountered. You need to deploy a new appliance.

 

The error message includes a resolution. Deploy a new appliance. The recommended solution is the last thing a typical vSphere admin would like to resolve such an issue. Fortunately enough, there is another solution/workaround.

To resolve this issue stop and start all the services on the vCSA,

• Putty/SSH to vCenter server appliance.
• Login to VCSA using the root credentials.
• Enabled "shell".
• Restart VCSA services

To restart VCSA services run the following commands:
service-control --stop --all
service-control --start --all

In case, simple services restart does not help, you can have an issue with some recent backup job. In such a case, there is another resolution with an additional workaround

• Putty/SSH to vCenter server appliance.
• Login to vCSA using the root credentials.
• Enabled "shell".
• Move the /var/vmware/applmgmt/backupRestore-history.json file to /var/tmp/.
• Restart the vCenter Server Appliance.

Hope this helps other folks in VMware community.    

How to downsize vCenter Server Appliance 6.5 storage?

Last week I have been asked by one partner how to downsize vCenter Server Appliance (VCSA) 6.5 storage.

Well, let's start with upsizing. To add CPU and RAM resources is very easy. VCSA 6.5 supports CPU Hot Add and Memory Hot Plug, therefore you do not need to even shut down VCSA to increase CPU and RAM resources.

CPU Hot Add and RAM Hot Plug

Storage expansion though is a little bit more difficult. You still do not have to shut down VCSA because virtual disk can hot-extended, however after a disk is extended you have to grow disk partitions within the operating system, Photon OS in this particular case. William Lam wrote the blog post here about it. Generally you have to run script  /usr/lib/applmgmt/support/scripts/autogrow.sh within VCSA shell so it is not a rocket science you just need to know what script to execute.

So upsize is easily doable. But what about downsize? VCSA 6.5 supports CPU Hot Remove but RAM cannot be downsized, therefore for RAM downsizing you have to shut down VCSA decrease memory resources and power on VM. Not a big deal, just small downtime so it can be done during a maintenance window. But the storage downsize is not possible. Well, in theory, it is possible but it is definitely not supported to decrease the size of disk partitions and the virtual disk itself because it is hard and also very risky as you do not know where data are located within the disk.

Warning: I have been told by someone that downsizing method described below does not work and restore options will allow you to choose just bigger form factor than originally backed up  VCSA. Unfortunately, I have the smallest form factor in my home lab so I cannot verify it but I believe him. Sorry for the misleading idea but there might be some unsupported method how to tweak backup files to have the impression it is a backup from smaller VCSA form factor.  

We have another downsizing option for VCSA 6.5. You are most probably aware that VCSA 6.5 has introduced application-based backup where vCenter inventory, identity, and even the database is backed up to a remote location via following protocols HTTP, HTTPS, SCP, FTP, FTPS. The restore of VCSA 6.5 backup is done as a new VCSA deployment executed in a restore mode where a previously created backup is used as a restore point. The nice thing is, that during VCSA restore you can choose different VCSA form factor with different storage footprint. So this is a potential way how to downsize vCenter Server Appliance 6.5 storage in a supported way. The only assumption is that your backup data will fit into new storage size.

If you will plan such downsizing exercise, please do not forget to keep your original vCenter Server Appliance somewhere to have a simple way how to roll back.

Hope this is helpful and informative.

How to restore deleted vmdk from VMFS5

Yesterday I have got an E-mail from somebody asking me how to restore deleted vmdk from VMFS5. They deleted VM but realised there are very important data.

Typical answer would be - "Restore from backup" - however they wrote that they don't have backup.

Fortunately, I have never had a need to restore deleted vmdk so I was starting to do some quick research (aka googling :-) )

I found VMware KB 1015413 with following statement ...

"VMware does not offer data recovery services. This article provides contact information for data recovery service providers.
Note: VMware does not endorse or recommend any particular third-party utility, nor is this list meant to be exhaustive."

I was sitting in the VMware office so I have asked colleagues if they have any practical experience with undeleting vmdk from VMFS.  One colleague of mine suggested utility "VMFS Recovery" from DiskInternals.com. He had positive experience with this particular tool. His suggestion was to use trial version which should help to identify if recovery is possible and buy full version for recovery.

Warning: Use any third party tool for recovery on your own risk.

I absolutely understand that if you lose your important data you would like to try anything to recover it back however here are my general thoughts:

• Clone or snapshot your raw disk (LUN) with VMFS before any recovery (use storage array capabilities, third party imaging tool or *nix dd command) 
• If your data are very valuable for you consider engagement of data recovery expert services
• When  you do recovery by your self I wish you good luck.

If you have some other experience with this topic please share it with community in comments. 

Good or Bad Backup Job?

Veeam is very good backup software specialized on agent-less VM backups. But we all  know that bugs are everywhere and Veeam is not the exception. If you have VMware vSphere VM with independent disk Veeam cannot successfully perform a backup. That's logical because independent disks cannot have snapshots which are mandatory for agent-less VM backups leveraging VMware API for Data Protection (aka VADP). However the problem is that backup job of independent virtual disk is green. That can give you impression that everything is OK. But it is not. You have false expectation that you have correct backup. But you haven't and if you don't check logs you can find it really late ... during restore procedure which is not possible.

You can see what happen below on the screenshot.     

Click to enlarge

The correct behavior would be if backup job fails and backup administrator can repair the issue. This behavior was seen in Veeam version 6.5. Veeam support has been informed about this wrong behavior so it hopefully will be repaired in the future.

Veeam Backup Components Requirements

Veeam is excellent backup software for virtualized environments. Veeam is relatively easy to install and use. However when you have bigger environment and looking for better backup performance is really important to know infrastructure requirements and size appropriately your backup infrastructure.

Here are hardware requirements for particular Veeam components.
 
Veeam Console
  Windows Server 2008 R2
  4GB RAM + 0.5GB per concurrent backup job

Veeam Proxy
  Windows Server 2008 R2
  2GB RAM + 0.2GB per concurrent task

Veeam WAN Accelerator
  Windows Server 2008 R2
  8GB RAM
  Disk for cache

Veeam Repository
  Windows Server 2008 R2, Linux or NAS (CIFS) 
  4GB RAM + 2GB per each concurrent ingress backup job
 

EMC Storage and Backup solutions for VMware

ftp://Bloglink:vgeekb1og@vspecialist.emc.com/Events/VMworld/2011/Americas/Sessions/SPO3977/SPO3977_sakac_final_v1.2_w_fonts.pptx

get 2 free VEEAM products for VCP

Get 2 free VEEAM products for VCP, vexpert people (NFR licence only)
http://www.veeam.com/nfr/free-nfr-license

Open Source VMFS Driver

This driver enables read-only access to files and folders on partitions formatted with the Virtual Machine File System (VMFS). VMFS is a clustered file system that is used by the VMware ESX hosts to store virtual machines and virtual disk files.

http://code.google.com/p/vmfs/

How to copy (backup) files from ESX3i?

You can use RCLI and vifs command. More info at
http://www.vm-help.com/esx/esx3i/esx_3i_rcli/vifs.php

Examples:
List files in datastore directory:
vifs --server 192.168.4.4 --username root --password ***** --dir "[Datastore1]/win2k3"

Download file from ESX3i Datastore:
vifs --server 192.168.4.4 --username root --dc ha-datacenter --password ***** --get "[Datastore1] /win2k3/win2k3-flat.vmdk" my-clone.vmdk

Notes:
File write speed around 1MBps only