In a previous article, we have fixed some VUM errors and the ESXi profile issues.
Again, we had some issues with the host when applying updates, and it was impossible to remediate them.
In the initial reboot(after applying the updates) host did not power up. It froze with this error:
We did a problem before. I wrote another article regarding this “fatal error 15”.
Need to reboot the ESXi host again, then the ESXi did start, but no updates were installed. When we try again to stage the updates, we get the “error code 15”. I discuss this issue in this article. I see this type of error in exupdate.log
Cannot locate source for payload uc_amd of VIB VMware_bootbank_cpu-microcode
Failed to send vob install.stage.error: [Errno 27] File too largeesxupdate: BootBankInstaller.pyc: DEBUG: Creating an empty ImageProfile for bootbank /bootbank
esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.
esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.Cannot locate source for payload b of VIB VMware_bootbank_esx-base_6.0.0-2.37.3825889
So it seems that the ESXi image profile was corrupted. Following the tasks from the previous articles did not fix the issue. The problem is in the host boot image.
So I did start to troubleshoot the host image profile.
1 2 3 4 5 6 |
[root@esxi01:/vmfs/volumes] esxcli software profile get [Exception] No host image profile defined Please refer to the log file for more details. |
No host profile in the ESXi host, even when copy files from a working ESXi host after a reboot, did not fix the issue. Always get the same problem.
So we need then check the /bootbank image (imgdb.tgz).
First, we need to check where the imgdb.tgz is located but run a find command.
1 2 3 4 5 6 |
[root@esxi01:/vmfs/volumes] find * | grep imgdb.tgz 35d740e9-b0c37c7e-1bc8-d524645d2758/imgdb.tgz 5e19abe0-a45004e7-7155-78541aa217bc/imgdb.tgz |
The system found two imgdb, need to check which one is the correct one (the other is just a backup) and see if the file is corrupted or not.
1 2 3 4 5 6 |
[root@esxi01:/vmfs/volumes] ls -l 35d740e9-b0c37c7e-1bc8-d524645d2758/imgdb.tgz -rwx------ 1 root root 183 Apr 23 09:31 35d740e9-b0c37c7e-1bc8-d524645d2758/imgdb.tgz [root@esxi01:/vmfs/volumes] ls -l 5e19abe0-a45004e7-7155-78541aa217bc/imgdb.tgz -rwx------ 1 root root 55580 Jan 23 13:07 5e19abe0-a45004e7-7155-78541aa217bc/imgdb.tgz |
I see the one with only 183 bytes is the one used in the /bootbank. So I immediately suspected that this one was a corrupted file since the file is at least 60Kb.
So I check the same image from a working ESXi and see the size.
1 2 3 4 |
[root@esxi02:/vmfs/volumes] ls -l 8858c8e4-432e17ef-bd6b-967f85396adc/imgdb.tgz -rwx------ 1 root root 63075 Apr 23 12:41 8858c8e4-432e17ef-bd6b-967f85396adc/imgdb.tgz |
So as we can see, this is the standard size for a working imgdb file from a working ESXi.
So to fix this, we will copy the same file from a working ESXi to our fault ESXi.
Before copying the image from the working ESXi host, let’s make a copy of the corrupt image to the tmp folder and remove the corrupt file.
1 2 3 4 |
[root@esxi01:/vmfs/volumes] cp /bootbank/imgdb.tgz /tmp [root@esxi01:/vmfs/volumes] rm /bootbank/imgdb.tgz |
Before copying the files between hosts, don’t forget again to set the firewall rule to allow the copy. For security reasons, after you do this task, disable it again.
1 2 3 |
esxcli network firewall ruleset set --enabled true --ruleset-id=sshClient |
Just copy the file over to the corrupted host.
1 2 3 4 5 6 7 8 9 10 |
[root@esxi02:/vmfs/volumes/6a0a3de6-986a7975-1f51-b4f1cbe99dbc] scp -r /bootbank/imgdb.tgz root@esxi01:/bootbank/ The authenticity of host 'esxi01 (esxi01)' can't be established. RSA key fingerprint is SHA256:isiF8md0Q6GDazZ97fbJ/4ZiqxOrf9tE4mHv0XN64kM. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'esxi01' (RSA) to the list of known hosts. Password: imgdb.tgz 100% 61KB 60.9KB/s 00:00 [root@esxi02:/vmfs/volumes/6a0a3de6-986a7975-1f51-b4f1cbe99dbc] |
Note: Even we have already copied the Profile and VIBs files from the working ESXi (tasks explained in the previous articles and above), we can also do the next step to copy those files into the corrupted ESXi host using the imgdb.tgz file. For that, we need to extract the file and copy the folders.
You could skip the next step if you copied the files already following the previous articles and tasks.
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[root@esxi01:~] cd /bootbank/ [root@esxi01:/vmfs/volumes/6a0a3de6-986a7975-1f51-b4f1cbe99dbc] ls -l imgdb.tgz -rwx------ 1 root root 62403 May 21 19:54 imgdb.tgz [root@esxi01:/vmfs/volumes/6a0a3de6-986a7975-1f51-b4f1cbe99dbc] cp imgdb.tgz /tmp [root@esxi01:/vmfs/volumes/6a0a3de6-986a7975-1f51-b4f1cbe99dbc] cd /tmp [root@esxi01:/tmp] tar -xzf imgdb.tgz [root@esxi01:/tmp] ls -l v* total 4 drwxr-xr-x 1 root root 512 Jul 3 20:07 db [root@esxi01:/tmp] cp -r /tmp/var/db* /var/db [root@esxi01:/tmp] |
Next, do a backup of the changes we made to the bootbank host image profiles.
1 2 3 4 5 6 7 8 |
[root@esxi01:~] /sbin/auto-backup.sh Files /etc/vmware/dvsdata.db and /tmp/auto-backup.40596//etc/vmware/dvsdata.db differ Saving current state in /bootbank Clock updated. Time: 20:06:58 Date: 07/03/2016 UTC [root@esxi01:~] |
Next, reboot the ESXi host, apply the patches, or use the VUM again.
In this case, since we copy an image and profile from an ESXi that had already the bugfix ESXi600-20160500 applied, now in this fixed host, it recognizes this bugfix is already installed. That is not correct. The option was to use the bugfix again manually.
Just upload to our ISO Datastore and run the vib install command.
But then we did get another issue: [Errno 2] No such file or directory: ‘/bootbank/conrep.v00’
1 2 3 4 5 6 7 8 9 |
[root@esxi01:/tmp] esxcli software vib install -d /vmfs/volumes/ISO-FILES/vmware/VIBs/ESXi600-201605001.zip -f [InstallationError] Error in copying from /bootbank/conrep.v00 to /tmp/stagebootbank/conrep.v00: [Errno 2] No such file or directory: '/bootbank/conrep.v00' vibs = set(['VMware_bootbank_sata-sata-sil_2.3-4vmw.600.0.0.2494585', 'VMware_bootbank_lsi-mr3_6.605.08.00-7vmw.600.1.17.3029758'... ...'Intel_bootbank_net-ixgbe_4.1.1.1-1OEM.550.0.0.1331820', 'VMware_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.600.0.0.2494585', 'VMware_bootbank_scsi-aic79xx_3.1-5vmw.600.0.0.2494585']) Please refer to the log file for more details. |
Checking the file “conrep.v00” I see that doesn’t exist in the /bootbank of the corrupted ESXi. So decided to copy this again from the working ESXi this file.
1 2 3 |
[root@esxi02:~] scp -r /bootbank/conrep.v00 root@esxi01:/bootbank |
Run the patch and install it again.
1 2 3 4 5 6 7 8 9 10 11 |
[root@esxi01:/tmp] esxcli software vib install -d /vmfs/volumes/ISO-FILES/vmware/VIBs/ESXi600-201605001.zip -f Installation Result Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective. Reboot Required: true VIBs Installed: VMware_bootbank_elxnet_10.2.309.6v-1vmw.600.0.0.2494585, VMware_bootbank_ima-qla4xxx_2.02.18-1vmw.600.0.0.2494585... ...VMware_bootbank_scsi-mptsas_4.23.01.00-9vmw.600.0.0.2494585, VMware_bootbank_scsi-mptspi_4.23.01.00-9vmw.600.0.0.2494585, VMware_bootbank_uhci-usb-uhci_1.0-3vmw.600.0.0.2494585, VMware_bootbank_vsan_6.0.0-2.37.3825891, VMware_bootbank_vsanhealth_6.0.0-3000000.3.0.2.37.3825892, VMware_bootbank_xhci-xhci_1.0-3vmw.600.2.34.3620759, VMware_locker_tools-light_6.0.0-2.34.3620759 |
No errors were found this time.
Reboot the host, and everything was ok again with this ESXi and is up to date.
I hope this can help you bypass this error in the future.
Share this article if you think it is worth sharing. If you have any questions or comments, comment here or contact me on Twitter.
Thanks for the post! The corrupt imgdb.tgz was my problem. Running ESXi 6.0 Update 2, HP custom ESXi image.
Glade to help Brandon.
Hi Brandon,
Only today I notice that my reply is not sending emails to users that comment on my blog. So now I am just FYI to you regarding my comment.
Thank You
Luciano Patrao
Thanks for this! This solved my issue quickly in a production environment!
Hi Josh,
Thanks. Glade that I could help.
PS: Share if you think can help others.
Thank You
Luciano Patrao
Hi Josh,
Only today I notice that my reply is not sending emails to users that comment on my blog. So now I am just FYI to you regarding my comment.
Thank You
Luciano Patrao
We have two HP Proliant BL460c Gen8 and one HP Proliant DL380 Gen( configured in a HA cluster. We upgraded BL460c from ESXi 5.5 to ESXi 6.0 (HP Customized Image) successfully. But we could not upgrade HP DL380 to ESXi 6.0 as it kept giving an error message ” Could not obtain module order from esx image db: (“/vmfs/columes/e88c0deb-2ab95e1a-d61f-0b6dbf28e3a5/imgdb.tgz’, “Error writing tar database: [Errno 2] No such file or directory: ‘/vmfs/volumes/e88c0deb-2ab95e1a-d61f-0b6dbf28e3a5/imgdb.tgz.new’.”) We tried to upgarde HP DL380 to HP Custom image and also tried Vmware normal ESXi 6.0 image. And we also tried a fresh install of ESXi 6.0 but we get the same error again and again.
Could you please assist us in resolving the issue and what could be the cause of issue.? I would be grateful to you.
Hi,
Have you check that partition? Does the file exist?
Are you doing a manual upgrade, or using VMware Update manager?
Did you follow the recommendations and procedures of this article to correct the imgdb.tgz file?
Can you provide the logs to investigate deeper?
God bless you sir! This was a lifesaver. I’d been hopping from one code 99 error (used your other article – http://www.provirtualzone.com/esxi-6-0-reports-vm-during-stage-patches-to-entity-vum-operation/), to error code 15 (used your other article – http://www.provirtualzone.com/esxi-6-0-reports-error-code-15-during-remediate-update-in-vum-operation/ ), to finally here to resolve the last error code 15. After rebuilding the bootbank (https://kb.vmware.com/kb/2033564), I ran over this article to help me with the last point.
VMware support was poking at it with me and I found that my NetAppNa.v00 file needed to be moved over from a good host to the problematic one. Once I did that we were finally able to stage the updates and install the updates normally using VUM.
For the record, I have HP BL460c G9 servers which used the HP U3 iso to install ESXi.
Hi Ytsejamer1.
Glad to help.
Thank you for your feedback.
Luciano Patrao
Thank you, thank you, thank you! Had a 6.5.0 host that VUM (on VCSA 6.5.0u1) kept failing to scan. Host’s /bootbank/imgdb.tgz was about 150 bytes (not kBytes!) and both /var/db/profiles and /var/db/vibs were empty.
Solved it by copying over /bootbank/imgdb.tgz from a “good” host to the bad one, extract the profile and VIBs into those empty directories in /var/db, run /sbin/auto-backup.sh as you described in this article, and everything worked again.
Hi Peter,
Glad to help.
Don’t forget to backup, reboot and then rerun the scan(apply any updates needed) and again restart the ESXi host. Also always copy the profile and VIBs from an ESXi host that is running in the same build number(that as not been updated yet). If you copy from a newer build, the corrupted ESXi host thinks is updated and is not.
Because sometimes the issue with host image profile is not fixed. Only after some reboots, we are sure that is fix.
Thank you for feedback.
Luciano Patrao