One of our had an issue vCloud Director stop working with partition root full.

When the team informs me that one vCD (a Linux version) was down, I needed to troubleshoot what was happen (since it was working 1h before).

The first thing was to login with ssh to vCD and check services.

So service was down, and even before I check the logs, I check partitions.

Immediately I know what the problem was, the lack of space in the root partition. But why? What was consuming 50Gb of space?

I check the log and tmp area to see if big logs were consuming this space, not consuming much space. So the problem was not here.

But /opt partition was consuming 46Gb, which is a huge size for this type of vCD (not small, but not so big).

So I start drilling down until I arrive in this folder. I notice this was the one that was filling up the /root partition /opt/vmware/vcloud-director/data/transfer.

I immediately suspect that there was no Shared Folder mounted on the vCD /data/transfer folder. Mount partition that I confirmed in the above initial df -h. But was looking for the full partition and did not even remember to check a mount partition.

Checking the mount and the folder, I confirm the problem was in the transfer folder and confirmed that the transfer folder was mounted on the /root partition.

Checking the transfer folder was a lot of temporarily vmdk files.

So it was obvious that was not added a mount in the initial install in this vCloud Director, and the install did set the root partition for the transfer folder. Obviously, one day users will start to use many uploads, this partition would be without space, and then service will stop because there is no space in the system to run the services, logs, etc.

I double-check vCenter tasks to see if there were any uploads before the team complains that vCD was down.

As we can see above, several OVF templates export, which triggers the problem, and /root partition was full because of these temporary files that should have been in a shared folder with enough space for this type of task.

So now I need to create a new Shared Folder in one of our QNAPs and mount it in this vCD.

I change permission and ownership of the /opt/vmware/vcloud-director/data/transfer

Added to mount to fstab to be mounted on a vCD reboot.

Tested fstab mount with mount -a, all was ok. Then I reboot the vCD(not mandatory, but I want to double-check the mount on a reboot).

After the reboot, all services were ok, and the cell was running without any issues.

So as we can see in this simple blog post, the /opt/vmware/vcloud-director/data/transfer continues to be one of the first issues in the vCD. I can’t count how often vCD had issues because of this mount (not mounted, wrong permissions, wrong ownership, etc.).

In this case, it was easy to identify and very easy (if you know where to look) to fix and start the services. Since most of the time, partition full is the reason for services not to start or stop.

I hope this blog post vCloud Director stop working with partition root full was useful to troubleshoot and find the issues in your vCloud Director infrastructures.

Share this article if you think it is worth sharing. If you have any questions or comments, comment here or contact me on Twitter.

©2021 ProVirtualzone. All Rights Reserved