Today one of our vCenter DB VMs we had a warning with the following message: “Virtual Disk Consolidation is required.” This is normal in a VMware farm and happens sometimes (this could be caused by too many snapshot, very old snapshots or snapshot that were not properly removed). All we need to do is click in the VM and the “Snapshot – Consolidate” option we can fix this issue. Sometimes is needed to power off the VM so that all vmdk locks are free and VM can consolidate all snapshots.

When trying to consolidate (with VM power down) the disks, I get: “An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED”.

With this message, this issue needs further troubleshooting.

First, need to check the vmdk files.

Note: Since this issue happen in my vCenter DB and vCenter was down, I had to connect with vSphere Client to the ESXi host directly where this VM was allocated ( if this issue is in any of your vCenter VMs, you should check which ESXi host is the VMS using, before power off the vCenter).

Browsing the Datastore where the VM is located and in the VM folder, I found 18 Snapshots files and 18 ctk vmdk files. This is not a reasonable number of snapshots and not a reasonable number of ctk files.

What are ctk files? CTK files are used by Changed Block Tracking (CBT) to track disk sectors that have been changed since the last backup. So that Backup tools only backup the altered disk sectors since the last changed ID (or backup). So after this, I suspected that was an issue originated by Veeam Backup tool.

Sometimes all that is necessary is to remove all these ctk files and delete all Snapshots or Consolidate the disks, and this will fix the problem.

A faster and safe way is to move (not delete) all ctk files to a temp folder and try to delete all Snapshots.

First, you need to power off the VM. Then go to ESXi host shell console and do the following tasks:

First, let’s list all ctk files.

root@ESXihost-001:/vmfs/volumes/datastore-01/db-003] ls -l *ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000001-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000004-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000005-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000006-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000007-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000008-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000009-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000010-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000011-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000012-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000013-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000014-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000015-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000016-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:48 db-003-000017-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 06:16 db-003-000018-ctk.vmdk
-rw——- 1 root root 5243392 Mar 4 04:49 db-003-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000001-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000004-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000005-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000006-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000007-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000008-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000009-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000010-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000011-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000012-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000013-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000014-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000015-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000016-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:49 db-003_1-000017-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 06:16 db-003_1-000018-ctk.vmdk
-rw——- 1 root root 4260352 Mar 4 04:50 db-003_1-ctk.vmdk
-rw——- 1 root root 6554112 Mar 4 06:16 db-003_2-ctk.vmdk
[root@ESXihost-001:/vmfs/volumes/datastore-01/db-003]

Then create a temporary folder inside of the VM folder

[root@ESXihost-001:/vmfs/volumes/datastore-01/db-003] mkdir cbt-temp

Then move all the files into the folder

[root@ESXihost-001:/vmfs/volumes/datastore-01/db-003] mv *-ctk.vmdk /vmfs/volumes/datastore-01/db-003/cbt-temp/

Then try to delete all Snapshots, or consolidate the Disks.

In my case, when I tried that and I still got: “An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED”

Something is locking these vmdk disks. I then focused my attention on the Veeam side.

In this environment, Veeam Backup Server is a physical machine, but Veeam Proxy is a VM. When using any Backup as a VM (or a Backup VM appliance), Virtual Disks from VMs are added to the appliance so that the appliance can create a backup. When the backup finish, the Virtual Disks must be released/removed from the VM appliance. Sometimes this is not removed.

Investigating the Veeam Proxy VM, I notice that two disks from the vCenter DB were added into the Veeam Proxy VM.

As we can see, those 2 Virtual Disks are in the Veeam Proxy VMs, that is why they are locked. In this case, we need to remove these virtual disks from this Veeam Proxy to delete the snapshots or consolidate the disks.

Let’s remove the Virtual Disks from this VM.

Important Note: Choose the option “Remove from Virtual Machine”, not the “Remove from Virtual Machine and delete files from disk”. Like is shown in the above images. If you choose the second option, Virtual Disks will be deleted from your original VM (in our case is the vCenter DB).

After this, now try again to delete all snapshots, or consolidate the disks. In this case, I chose the consolidate option.

The consolidation of the Virtual Disks then started. This is a procedure that can take a while (in this case took 45m), so please be patient.

For the final test, I started the Backup Job for this vCenter and double checked that it did finish properly and no Virtual Disks were locked in the Veeam Proxy or any snapshot were left in the fault VM.

Don’t forget to delete the temp folder and the files, when the CBT files were moved.

Hope this information was useful.

Note: Share this article, if you think it is worth sharing.

©2017 ProVirtualzone. All Rights Reserved