Today, our new vCenter Appliance 6.0(VCSA) had a very strange problem, and in this, Unable to remove permissions in vCenter Appliance 6.0 U2, how to fix, I will try to explain how to fix it.
First query users and Groups in the Domain(searching to add in the permissions) was very slow, and in the Web Client, sometimes we see the Domain with double entries or could not find the users/groups.
Also, when we look at the vCenter permissions(vCenter level, or just Folder / Cluster level), we see the user/group with double entries(like domain.com\user and domain\user) and then try to remove any of these permissions one was always resident and will not be deleted. No errors, no warnings, just can’t delete.
Note: vsphere.local users and groups did not show this behavior. So my focus needs to be on the AD and Identify Sources to troubleshoot.
Try to Stop/restart the Services to see if it fixes the issue, but no luck. Rebooting the VCSA also did not work. While troubleshooting the issue, and knowing that there were double AD entries, somehow Domain alias and Domain Full Name were set in the system identity. Checking logs did not find anything related that could help(just some references to domain and domain.com), not the issue itself.
Googling the issue did not find anything related to the same issue(exactly), but did find a VMware KB regarding some VCSA upgrade(that was not the case here, since it is a new installation), but it shows a workaround to fix the issue regarding vCenter Server Appliance database to have the full domain name instead of an alias. Since we had both, it makes sense that this could fix the issue.
How to fix the issue:
Before starting, do a full backup of your VCSA, or at least do a Snapshot. In case of problems, you can always roll back.
So login to your VCSA using shell and root user.
Note: Don’t forget that after you log in, you need to enable and run the BASH shell. If you get: Shell is disabled, you need to run the shell.set to enabled.
Using username "root".
VMware vCenter Server Appliance 220.127.116.1100
Type: vCenter Server with an embedded Platform Services Controller
Last login: Thu Sep 22 22:09:53 UTC 2016 from xx.xx.xx.xx on ssh
Last login: Thu Sep 22 22:22:34 2016 from xx.xx.xx.xx
Connected to service
* List APIs: "help api list"
* List Plugins: "help pi list"
* Enable BASH access: "shell.set --enabled True"
* Launch BASH: "shell"
Command> shell.set --enabled True
---------- !!!! WARNING WARNING WARNING !!!! ----------
Your use of "pi shell" has been logged!
The "pi shell" is intended for advanced troubleshooting operations and while
supported in this release, is a deprecated interface, and may be removed in a
future version of the product. For alternative commands, exit the "pi shell"
and run the "help" command.
The "pi shell" command launches a root bash shell. Commands within the shell
are not audited, and improper use of this command can severely harm the
Help us improve the product! If your scenario requires "pi shell," please
submit a Service Request, or post your scenario to the
https://communities.vmware.com/community/vmtn/vcenter/vc forum and add
After you are inside your VCSA console, you need to connect to the postgress database and set your DB to use Full Domain Name.
hostname:~ # cd /opt/vmware/vpostgres/current/bin
hostname:~ # ./psql -d VCDB -U postgres
psql.bin (9.3.9 (VMware Postgres 18.104.22.168-2921310 release))
Type "help" for help.
VCDB=# update CIS_KV_KEYVALUE set KV_KEY = replace(kv_key,'DOMAIN%5C','DOMAIN.COM%5C');
VCDB=# update CIS_KV_KEYVALUE set KV_VALUE = replace(kv_value,'DOMAIN\','DOMAIN.COM\');
Note: Just replace the ‘DOMAIN’ for your alias Domain and ‘DOMAIN.COM’ for your Domain Full Name.
Just reboot your VCSA, and the issue is fixed.
What did trigger this issue? In this particular case, I could not identify the problem’s source. Still, I suspect that changing our Identity Sources to a different LDAP(that is in a different location and a different country) could have a different trust in the Forest and trigger the issue. Since VMware informs us that one of the sources for this type of issue could be problems connecting to the LDAP server and updating the changes that we make in our permissions.
I hope this can help you bypass this issue.
Share this article if you think it is worth sharing. If you have any questions or comments, comment here or contact me on Twitter.