How to fix a vCenter Appliance partition with Veeam

/, Virtualization/How to fix a vCenter Appliance partition with Veeam

How to fix a vCenter Appliance partition with Veeam

In this blog post, I will try to explain how to fix a vCenter Appliance partition with Veeam. In this case, is the vCenter Application log partition.

This week I needed to evacuate all VMs from one particular Datastore to a new Datastore, and vCenter was one of those Virtual Machines. When applying a vStorage (vMotion) to move this specific vCenter from one Volume to another, I get an error (around 95% before did finish).

Checking error was just a generic error, nothing special, but the vCenter did migrate to the new Datastore, but the task was not complete. Checking Datastore / Folder where vCenter was moved found that was missing some Virtual Disks files.

Listing all Virtual Disks from the vCenter, I notice that disk four didn’t move correctly. First, vmdk file in the Virtual Disk four was too small (only 0,65Kb) and there was no flat file on it (you need to check in ESXi console using ls -lah command to check all files).

How to fix a vCenter Appliance partition with Veeam

Note: When I took the screenshot I had already changed the files to .old so that I can restore the same files with the same name.

Just a small explanation about vCenter Appliance Virtual Disks.

First, each Virtual Disk consists of two files (three if CBT is enabled on the VM). These files are stored in the same directory as the .vmx file.

  • .vmdk – Virtual disk files, which store the contents of the virtual machine hard disk drive.
  • -ctk.vmdk – Virtual Disk file is created when CBT is enabled in the VM. CBT is a VMware feature used by backups. This file tracks all changes in the Virtual Disk for incremental backups.
  • -flat.vmdk – Virtual Disk file that contains all Data (if we could compare to physical, this would be the physical disk).

So when listing all files in the vCenter Datastore  “/vmfs/volumes/5c89bb3b-99692b70-f34b-9c8e9916e56c/Main vCenter-30” I notice that flat was missing. Since this is the data file, this was a problem.

Then I check the previous Datastore folder to see if vStorage did leave any files behind(this sometimes happen) and notice that the file was still in the old folder “VMware vCenter Server Appliance_4-flat.vmdk”.

So I try to move manually, error. Try to copy to the destination folder, error. Next, I need to check the consistency of this vmdk using vmkfstools.

[root@DL360-ESXi05:/vmfs/volumes/59d14489-f76c91b4-49f9-78e7d1f8a1c8/Main vCenter-30] vmkfstools –fix check VMware\ vCenter\ Server\ Appliance_4-flat.vmdk
DiskLib_Check() failed for source disk ‘VMware vCenter Server Appliance_4-flat.vmdk’: The file specified is not a virtual disk (15).
[root@DL360-ESXi05:/vmfs/volumes/59d14489-f76c91b4-49f9-78e7d1f8a1c8/Main vCenter-30]

Then surprisingly I had a vmdk corrupted. This was a problem since my last backup from this vCenter was almost three months old, and I had a lot of changes in this vCenter over the previous three months. Restoring a vCenter with a backup that old will have many inconsistencies between vCenter and ESXi hosts (new VMs, new Virtual Networks, new Datastores, etc.). All this information is saved in the vCenter DB.

So the question is, how to fix a vCenter Appliance partition with Veeam backup?

Next, I need to check how the 12 Virtual Disks from vCenter are composed and what represents each one in the vCenter structure. After some research about this, I found the vCenter Virtual Disk structure (thanks William Lam from virtuallyGhetto ).

How to fix a vCenter Appliance partition with Veeam

In this case was vmdk4 that represents the virtual disk number 5 (SCSI0:4) is the partition for the logs. So is not a boot partition or a partition where is vCenter DB.  Is the vCenter Appliance VM logs, not the DB logs partition. I will lose three months logs, but that is ok and affordable.

So the decision now here is to restore this particular Virtual Disk using the Veeam Backup that has three months old.

How to restore the vCenter Appliance partition with Veeam.

The first option that makes sense to restore a Virtual Disk in Veeam is the option “Restore virtual disks…”

How to fix a vCenter Appliance partition with Veeam

But unfortunately, this option doesn’t work. Try to troubleshoot why, but the time I had I have not found why Veeam was not able to restore the Virtual Disk, so I change for a second option “Restore VM files…”.

Since a Virtual Disk is just a file that belongs to the VM, this option is suitable for the restore.

How to fix a vCenter Appliance partition with Veeam

Select your restore point and then add the location for the restore. By default restore is pointing to the Veeam Server itself, you need to change the settings.

How to fix a vCenter Appliance partition with Veeam

Select the destination (ESXi host or vCenter and the Datastore folder) and select the file to restore. Like displayed above, the vmdk to restore the vmdk4.

How to fix a vCenter Appliance partition with Veeam

You will get a warning if the original files still exist, for the safe side I did not delete the files, just created a copy of both files (vmdk and ctk.vmdk) where I added .old to name, then select yes. Remember the -flat.vmdk file was not in this Datastore folder.

How to fix a vCenter Appliance partition with Veeam

Next click finish and restore will start, and in a couple of minutes, you have your vCenter back.

After restore I Power On vCenter and did some checks on the partition and mount points of vCenter Appliance.

All seems ok in partitions and mount points. After I login to vCenter through Web Client (Flash and HTML5) and all seems ok. No hosts disconnected or out of sync and no orphaned Virtual Machines. So all was ok.

And final test that I did to check if the latest changes in the vCenter done in the last days were in the DB I run a simple PowerCli script.

All updates where listed, so I was 100% sure that my vCenter Appliance is working and up to date.

You can find Get-VMConfigChanges function in GitHub, created by William Lam.

Note: I have done some test with other partitions, like Core with the same procedure and did also work.

Hope this blog post help you fix your vCenter Appliance without the need to restore the full Appliance.

Note: Share this article, if you think it is worth sharing.

©2019 ProVirtualzone. All Rights Reserved
By | 2019-03-21T12:56:36+02:00 March 21st, 2019|Partners, Virtualization|0 Comments

About the Author:

I am over 20 years’ experience in the IT industry. Working with Virtualization for more than 10 years (mainly VMware). I am an MCP, VCP6.5-DCV, VMware vSAN Specialist, Veeam Vanguard 2018/2019, vExpert vSAN 2018/2019 and vExpert for the last 4 years. Specialties are Virtualization, Storage, and Virtual Backups. I am working for Elits a Swedish consulting company and allocated to a Swedish multinational networking and telecommunications company as a Teach Lead and acting as a Senior ICT Infrastructure Engineer. I am a blogger and owner of the blog ProVirtualzone.com

Leave a Reply

%d bloggers like this: