When we have a maintenance window and need to do our work in the Virtual Environment, we always find new issues. This time was in one of the ESXi 6.0 hosts.
This is an HP DL360 G7 with the ESXi 6.0 build 3620759.
When trying to use VMware update manager in this host, the scan runs without any problem, but when trying to Stage the updates, we get:
Stage patches to entity esxi01.localdomain. The host returns esxupdate error code:99.
An unhandled exception was encountered. Check the Update Manager log files and esxupdate log files for more details.
Looking at the esxpudate.log(in ESXi var/log/), I see a lot of failed tries to upload VIB files to the repository. Also this:
2016-04-24T15:43:54Z esxupdate: root: ERROR: Traceback (most recent call last):
2016-04-24T15:43:54Z esxupdate: root: ERROR: File “/usr/lib/vmware/esxcli-software”, line 462, in <module>
2016-04-24T15:43:54Z esxupdate: root: ERROR: main()
2016-04-24T15:43:54Z esxupdate: root: ERROR: File “/usr/lib/vmware/esxcli-software”, line 453, in main
2016-04-24T15:43:54Z esxupdate: root: ERROR: ret = CMDTABLE
2016-04-24T15:43:54Z esxupdate: root: ERROR: File “/usr/lib/vmware/esxcli-software”, line 173, in ProfileGetCmd
2016-04-24T15:43:54Z esxupdate: root: ERROR: raise Exception(“No host image profile defined”)
2016-04-24T15:43:54Z esxupdate: root: ERROR: Exception: No host image profile defined
I needed to check the Host Software profile(not the same as the Host Profile) and check the issue.
Then I noticed that the ESXi somehow lost its software profile, and it was not possible to stage any update or even install a VIB manually.
When troubleshooting the issue, I found out that there was no Software profile in this host.
root@esxi01:~] esxcli software profile get Exception] No host image profile defined Please refer to the log file for more details.
[root@esxi01:~] esxcli software vib list Name Version Vendor Acceptance Level Install Date ----------- ------------------ ------ ---------------- ------------ tools-light 6.0.0-2.34.3620759 VMware VMwareCertified 2016-04-23
I needed to check the vibs folder and profiles folder in /var/db/esximg. I noticed the two folders were empty. Also, checking the VIB list in this host, I get this:
[root@esx01:~] ls -l /var/db/esximg/* /var/db/esximg/profiles: total 0 /var/db/esximg/vibs: total 0 [root@esx01:~]
When checking the same information from a working ESXi host, I get this:
[root@esxi02:~] ls -l /var/db/esximg/profiles/
total 24
-r--r--r-- 1 root root 23998 Apr 23 12:05 %28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855
[root@esxi02:~] esxcli software profile get
(Updated) HP-ESXi-6.0.0-iso-600.9.1.39
Name: (Updated) HP-ESXi-6.0.0-iso-600.9.1.39
Vendor: esxi02.localdomain
Creation Time: 2016-04-23T12:05:31
Modification Time: 2016-04-23T12:05:31
Stateless Ready: True
Description:
(Original Vendor):Hewlett-Packard
2016-04-23T12:05:30.688280+00:00: The following VIBs are
installed:
hpssacli 2.40.13.0-6.0.0.1854445
hponcfg 6.0.0.04-00.14.4.2494585
conrep 6.0.0.01-01.00.7.2494585
hptestevent 6.0.0.01-01.00.5.2494585
hpbootcfg 6.0.0.02-02.00.6.2494585And also this:
[root@esxi02:~] esxcli software vib list Name Version Vendor Acceptance Level Install Date ---------------------- ------------------------------------- --------------- ---------------- ------------ net-tg3 3.137l.v60.1-1OEM.600.0.0.2494585 BRCM VMwareCertified 2016-01-23 elxnet 10.7.110.13-1OEM.600.0.0.2768847 EMU VMwareCertified 2016-04-23 ima-be2iscsi 10.7.110.10-1OEM.600.0.0.2159203 EMU VMwareCertified 2016-04-23 lpfc 10.7.110.4-1OEM.600.0.0.2768847 EMU VMwareCertified 2016-04-23 scsi-be2iscsi 10.7.110.10-1OEM.600.0.0.2159203 EMU VMwareCertified 2016-04-23 amsHelper 600.10.4.0-22.2494585 HPE PartnerSupported 2016-04-23 .... .... scsi-megaraid-mbox 2.20.5.1-6vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23 scsi-megaraid-sas 6.603.55.00-2vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23 scsi-megaraid2 2.00.4-9vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23 scsi-mptsas 4.23.01.00-9vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23 scsi-mptspi 4.23.01.00-9vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23 uhci-usb-uhci 1.0-3vmw.600.0.0.2494585 VMware VMwareCertified 2015-08-23 vsan 6.0.0-2.34.3563498 VMware VMwareCertified 2016-04-23 vsanhealth 6.0.0-3000000.3.0.2.34.3544323 VMware VMwareCertified 2016-04-23 xhci-xhci 1.0-3vmw.600.2.34.3620759 VMware VMwareCertified 2016-04-23 tools-light 6.0.0-2.34.3620759 VMware VMwareCertified 2016-04-23 scsi-qla2xxx 911.k1.1-26OEM.500.0.0.472560 qlogic VMwareCertified 2015-08-23
Searching on the internet for the same type of issue, I cannot find anything similar. VMware KB2043170 has a solution for the “error code 99” by Initializing VUM without reinstalling. But since this is a problem in one host, this could not be the solution for this issue.
Then found an article with a similar issue in
Since I didn’t want to reinstall the ESXi host at the moment, so I needed to try this solution. Even was not 100% similar, it was almost the same.
I followed Patrick’s recommendation to copy the esximg files/folders from a working host(esxi-02) to the corrupted ESXi host(esxi-01).
First, if you don’t have your SSH Client enabled in the host firewall, you need to allow to do the next task using the SCP command.
To enable SSH Client in the source ESXi host:
[root@esx02:~] esxcli network firewall ruleset set --enabled true --ruleset-id=sshClient
Note: Don’t forget to disable SSH Client after finishing these tasks.
After you run the SCP command, you will be prompted for the root password of the remote host, and once you have successfully authenticated, the file will copy.
[root@esxi02:~] scp -r /var/db/esximg/ root@esxi01:/var/db The authenticity of host 'esxi02 (esxi02)' can't be established. RSA key fingerprint is SHA256:isiF8md0Q6GDazZ97fbJ/4ZiqxOrf9tE4mHv0XN64kM. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'esxi02' (RSA) to the list of known hosts. Password: sata-sata-sil-1748273158.xml 100% 1717 1.7KB/s 00:00 lsi-mr3-989864457.xml 100% 1504 1.5KB/s 00:00 scsi-ips--1979861494.xml 100% 1619 1.6KB/s 00:00 char-hpcru--1874046437.xml 100% 1638 1.6KB/s 00:00 net-tg3--917722591.xml 100% 1461 1.4KB/s 00:00 ipmi-ipmi-devintf-1862766627.xml 100% 1719 1.7KB/s 00:00 sata-sata-promise--860290007.xml 100% 1742 1.7KB/s 00:00 esx-ui--629271967.xml 100% 53KB 53.2KB/s 00:00 cpu-microcode-1845960806.xml 100% 2381 2.3KB/s 00:00 scsi-bnx2i--630528917.xml 100% 1407 1.4KB/s 00:00 scsi-fnic-248026735.xml 100% 1784 1.7KB/s 00:00 scsi-hpdsa--570772364.xml 100% 1412 1.4KB/s 00:00 qlnativefc--775726474.xml 100% 1408 1.4KB/s 00:00 hpe-ilo--120862598.xml 100% 1550 1.5KB/s 00:00 net-cnic-1577337473.xml 100% 1389 1.4KB/s 00:00 scsi-qla4xxx-1755174022.xml 100% 1517 1.5KB/s 00:00 hpe-smx-provider-1463103112.xml 100% 5426 5.3KB/s 00:00 ... ... net-bnx2x-272405923.xml 100% 1534 1.5KB/s 00:00 net-bnx2-984265687.xml 100% 1507 1.5KB/s 00:00 net-ixgbe--494567451.xml 100% 1415 1.4KB/s 00:00 scsi-mpt2sas--1969732120.xml 100% 1679 1.6KB/s 00:00 scsi-megaraid-mbox--26860043.xml 100% 1775 1.7KB/s 00:00 scsi-aic79xx-757558775.xml 100% 1643 1.6KB/s 00:00 %28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855 100% 23KB 23.4KB/s 00:00 [root@esxi02:~]
After copying the files, check if the files are in the ESXi host(esxi01).
Host Software profile:
[root@esx01:~] esxcli software profile get
(Updated) HP-ESXi-6.0.0-iso-600.9.1.39
Name: (Updated) HP-ESXi-6.0.0-iso-600.9.1.39
Vendor: deacvm-dsc-002
Creation Time: 2016-04-23T12:05:29
Modification Time: 2016-04-23T12:05:29
Stateless Ready: False
Description:
(Original Vendor):Hewlett-Packard
2016-04-23T12:05:29.506598+00:00: The following VIBs are
installed:
hpssacli 2.40.13.0-6.0.0.1854445
hponcfg 6.0.0.04-00.14.4.2494585
conrep 6.0.0.01-01.00.7.2494585
hptestevent 6.0.0.01-01.00.5.2494585
hpbootcfg 6.0.0.02-02.00.6.2494585
[root@esx01:~] ls -l /var/db/esximg/profiles/
total 24
-r--r--r-- 1 root root 23998 Apr 23 12:05 %28Updated%29%20HP-ESXi-6.0.0-iso-600.9.1.39-687127855
[root@esx01:~]After checking that the files are copied into the destination, I rerun the VUM Scan into the host and after Stage, and no errors were found. After that, I remediated the ESXi host, and it is fully updated.
Again, thanks to Patrick for his solution that put me on the right track.
I hope this can help.
Share this article if you think it is worth sharing. If you have any questions or comments, comment here or contact me on Twitter.
[…] a previous article we have fix some issues regarding the VUM errors and also with ESXi […]
[…] a previous article we have fix some issues regarding the VUM errors and also with ESXi […]
Great article, just wanted to double check that your copy command is correct, as you seem to be copying to same destination as the source?
[root@esxi02:~] scp -r /var/db/esximg/ root@esxi02:/var/db
Hi Chris,
Thank you for your reply.
Yes you are correct, was a typo error(I have already updated). Is [root@esxi02:~] scp -r /var/db/esximg/ root@esxi01:/var/db
Is to login to source ESXi, copy the folder esximg(from esxi02) to the foder /var/db (in esxi01).
Thanks again for your comment.
Luciano Patrao
Thanks for this saved my ass!
Happened on a BL460c Gen9 upgrading from build 3825889.
Hi James,
Glad I could help.
Luciano Patrao