Today had a very strange problem with our new vCenter Appliance 6.0(VCSA).
First query users and Groups in the Domain(searching to add in the permissions) was very slow and in the Web Client sometimes we see the Domain with double entries, or could not find the users/groups.
Also when we look at the vCenter permissions(vCenter level, or just Folder / Cluster level) we see also the user/group with double entries(like domain.com\user and domain\user) and when try to remove any of these permissions one was always resident and will not be deleted. No errors, no warnings, just cant delete.
Note: vsphere.local users and groups did not show this behavior. So my focus needs to be in the AD and Identify Sources to troubleshoot.
Try to Stop/restart the Services to see if it fix the issue, but no luck. Rebooting the VCSA also did not work. While troubleshooting the issue, and knowing that there was double AD entries, somehow Domain alias and Domain Full Name was set in the system identity. Checking logs did not find anything related that could help(just some references to domain and domain.com), not the issue itself.
Googling the issue, did not find anything related to the same issue(exactly), but did find a VMware KB regarding some VCSA upgrade(that was not the case here, since is a new installation), but it shows a workaround to fix the issue regarding vCenter Server Appliance database to have the full domain name instead of an alias. Since we had both, it makes since to me that this could fix the issue.
How to fix the issue:
Before started, do a full backup of your VCSA, or at least do a Snapshot. In case of problems, you can always rollback.
So login to your VCSA using shell and root user.
Note: Don;t forget that after you login you need to enable the BASH shell and run it. If you get: Shell is disabled, then you need to run the shell.set to enabled.
Using username "root".
VMware vCenter Server Appliance 220.127.116.1100
Type: vCenter Server with an embedded Platform Services Controller
Last login: Thu Sep 22 22:09:53 UTC 2016 from xx.xx.xx.xx on ssh
Last login: Thu Sep 22 22:22:34 2016 from xx.xx.xx.xx
Connected to service
* List APIs: "help api list"
* List Plugins: "help pi list"
* Enable BASH access: "shell.set --enabled True"
* Launch BASH: "shell"
Command> shell.set --enabled True
---------- !!!! WARNING WARNING WARNING !!!! ----------
Your use of "pi shell" has been logged!
The "pi shell" is intended for advanced troubleshooting operations and while
supported in this release, is a deprecated interface, and may be removed in a
future version of the product. For alternative commands, exit the "pi shell"
and run the "help" command.
The "pi shell" command launches a root bash shell. Commands within the shell
are not audited, and improper use of this command can severely harm the
Help us improve the product! If your scenario requires "pi shell," please
submit a Service Request, or post your scenario to the
https://communities.vmware.com/community/vmtn/vcenter/vc forum and add
After you are inside your VCSA console, you need to connect to the postgress database and set your DB to use Full Domain Name.
hostname:~ # cd /opt/vmware/vpostgres/current/bin
hostname:~ # ./psql -d VCDB -U postgres
psql.bin (9.3.9 (VMware Postgres 18.104.22.168-2921310 release))
Type "help" for help.
VCDB=# update CIS_KV_KEYVALUE set KV_KEY = replace(kv_key,'DOMAIN%5C','DOMAIN.COM%5C');
VCDB=# update CIS_KV_KEYVALUE set KV_VALUE = replace(kv_value,'DOMAIN\','DOMAIN.COM\');
Note: Just replace the ‘DOMAIN’ for your alias Domain, and ‘DOMAIN.COM’ for your Domain Full Name.
Just reboot your VCSA and the issue is fixed.
Final Note: What did trigger this issue? In this particular case I was not able to identify the source of the issue, but I suspect that by changing our Identity Sources to a different LDAP(that is in a different location and in a different country) could have a different trust in the Forest and did trigger the issue. Since VMware informs that one of the source for this type of issues could be problems connecting to LDAP server and update the changes that we make in our permissions.
Hope this can help you.
Note: Share this article, if you think is worth sharing.