VMware finally launched ESXi 7.0 Update 2c to fix the huge bug about USB/SD cards. This subject I have already discussed HERE in my workaround to fix the problem.

I did not have time to write this mandatory blog post about this new patch to fix the vSphere 7.0.U2a bug. I will try to drill down this patch and show some examples of how to apply, what is new, and what does fix it.

This subject was so important for so many customers that my previous post had a record of 25,000 clicks and 172 messages in just a couple of months, when my maximum was around 9,000 clicks in 6 months in the same post.

Customers were kind of “abandoned” for at least a month before VMware starts to acknowledge the issue and address this subject properly and provide temporary solutions, or even proposing my workaround 😉

For an issue serious as this one and that had impacted a lot in our running systems, for a lot of customers was not the best moment for VMware. But now we have the patch, time to move on.

As we all know(everyone suffering from this bug), this bug was addressed KB83647, KB2077516, and KB2149257. But the patch did also address and fix others issues that were in the last version. Like slow boot, issues with VM VMDK, some PSOD issues, and Hosts iSCSI Volumes.

Here is the list of all the fixes addressed in this patch.

Resolved Issues

The resolved issues are grouped as follows.

• ESXi_7.0.2-0.20.18426014
• esx-update_7.0.2-0.20.18426014
• Broadcom-bnxt-Net-RoCE_216.0.0.0-1vmw.702.0.20.18426014
• MRVL-E4-CNA-Driver-Bundle_1.0.0.0-1vmw.702.0.20.18426014
• Intel-i40en_1.8.1.137-1vmw.702.0.20.18426014
• Broadcom-ELX-lpfc_12.8.298.3-2vmw.702.0.20.18426014
• VMware-vmkusb_0.1-4vmw.702.0.20.18426014
• ESXi_7.0.2-0.15.18295176
• esx-update_7.0.2-0.15.18295176
• VMware-VM-Tools_11.2.6.17901274-18295176
• VMware-vmkusb_0.1-1vmw.702.0.15.18295176
• ESXi-7.0U2c-18426014-standard
• ESXi-7.0U2c-18426014-no-tools
• ESXi-7.0U2sc-18295176-standard
• ESXi-7.0U2sc-18295176-no-tools
• ESXi Image – ESXi70U2c-18426014
• ESXi image – ESXi70U2sc-18295176

Full Release Notes

Where can we find the patch?

Before explaining how to install or find this patch, I need to explain it is a patch. Not a new version, but a new build(there is no vSphere ISO to install/upgrade). So if you have any customized image/install, like HPE, Dell or Lenovo, etc., no problems applying this patch. Since you are not upgrading your vSphere.

So is safe to apply the patch in any vSphere 7.0.x customize build that you may have.

For those asking if HPE, or Dell, or others will launch an ISO with this, my answer is NO. If this is not a new version and there is no new ISO with this version(at the moment), companies will not create/provide a new Customize ISO. You need to install the patch.

Install the latest ISO (from VMware or Customized) for a new install and then install the patch.

I am pretty sure VMware will launch a new version soon. But since this patch was urgent, they decided to launch the patch first, and the new ISO version should be next.

Note: By aware that this patch and issues to be fixed, is only if are running version ESXi 7.0 Update 2a build 17867351 and to fix those issues on that version. If you run previous versions, your system will also be updated to this VMware ESXi, 7.0.2, 18426014 build.

vCenter Lifecycle Manager will apply updates from your version eq 7.0.1 to 7.02c.

Important Note: Before you apply this patch, I recommend first upgrading your vCenter to the latest version. The new version was also launched on 24/08 and is also fixing some critical issues. So upgrade first your vCenter and only apply the patches. With this, we have your system up to date in using the same build. If not, you have your ESXi hosts with a higher version than vCenter.

Since this is a patch and not a new version, it is not mandatory but is recommended.

After you upgrade your vCenter to version 7.0.2.00400 build 18356314 you should apply the patches in the ESXi hosts.

You can update your ESXi hosts using vCenter Lifecycle Manager (previous VUM), or you can dot it manually one by one in the ssh command line.

In your vCenter Lifecycle Manager, Image Depot there are two ESXi versions:

• ESXi 7.0.U2c 18426014 – Security and Bugfix patch.
• ESXi 7.0U2sc 18295176 – Security only patch.

Are VMware patch and update releases contain general and critical image profiles. For this particular case, I don’t see any differences between both, maybe because it was a critical and bugfix patch release.

When running vCenter updates, these are the updates that will be applied from this vSphere 7.0 Update 2c patch. There are around 56 updates(vibs and ESXi updates) in my case, are 8 or 9(the extra update is if you need to update your VMware Tools version).

Full list of the vibs updates and ESXi updates:

• Broadcom NetXtreme-E Network and ROCE/RDMA Drivers 216.0.0.0-1vmw
• Broadcom Emulex Connectivity Division fcoe driver for FCoE adapters 12.0.1500.1-2vmw
• Broadcom Emulex Connectivity Division NVMe over Fabric driver for FC adapters 12.8.298.1-1vmw
• Broadcom IMA plugin for Emulex OneConnect iSCSI Driver 12.0.1200.0-4vmw
• Emulex OneConnect iSCSI Driver 12.0.1200.0-8vmw
• Broadcom Emulex Connectivity Division lpfc driver for FC adapters 12.8.298.3-2vmw
• Emulex OneConnect Network Driver 12.0.1250.0-5vmw
• Broadcom Emulex OneConnect Network Driver 11.4.62.0-1vmw
• Broadcom 12Gbps SAS/PCIe MegaRAID Driver 7.716.03.00-1vmw
• Broadcom 6Gbps SAS HBA Driver 20.00.06.00-3vmw
• Broadcom 12Gbps SAS HBA Driver 17.00.10.00-2vmw
• Broadcom 12Gbps SAS/PCIe HBA Driver 17.00.02.00-1vmw
• Broadcom LSI NATIVE DRIVERS LSU Management Plugin v2 1.0.0-5vmw
• Broadcom NetXtreme I ESX VMKAPI ethernet driver 4.1.5.0-0vmw
• Cisco VIC Ethernet Driver 1.0.33.0-1vmw
• Cisco VIC FCoE Driver 4.0.0.63-1vmw
• ESXi 7.0.2 Build – 18426014
• ESXi Install/Upgrade Component 7.0.2 Build 18426014
• HPE HPSA LSU V2 Management Plugin 1.0.0-3vmw
• HPE Smart Array Controller Driver 70.0051.0.100-2vmw
• Intel Network Driver for Intel(R) X710/XL710/XXV710/X722 Adapters 1.8.1.137-1vmw
• Intel Network Driver for Intel(R) 1 Gigabit Server Adapters 1.4.11.2-1vmw
• Network driver for Intel(R) X722 and E810 based RDMA Adapters 1.3.1.19-1vmw
• Intel Network Driver for Intel(R) 10 Gigabit Adapters 1.7.1.35-1vmw
• Intel Gigabit Ethernet Connection Driver 0.8.4-11vmw
• Intel(R) NVMe Volume Management Device enabled Driver Plugin v2 2.0.0-2vmw
• Intel SCU SAS/SATA for VMware ESX 2.0.2.0088-7vmw
• Intel(R) Volume Management Device enabled NVME Driver 2.0.0.1152-1vmw
• Mellanox Technologies ConnectX-3/Pro Core Ethernet and RoCE Drivers 3.19.16.8-2vmw
• Mellanox Technologies ConnectX-4/5 Core Ethernet and RoCE Drivers 4.19.16.10-1vmw
• Microsemi Storage Solution Smart Array Storage Controller Driver 70.4000.0.100-6vmw
• Microsemi Storage Solution Smart Array Storage Controller Driver Plugin v2 1.0.0-6vmw
• Micron RealSSD PCIe Native Driver 3.9.8-1vmw
• Marvell AQtion Ethernet Controllers Network driver for VMware ESXi 1.0.3.0-8vmw
• QLogic NetXtreme II 1 Gigabit Ethernet Driver 1.1.0.11-1vmw
• QLogic NetXtreme II 10 Gigabit Ethernet FCoE and iSCSI E3 Drivers 1.0.0.0-1vmw
• QLogic FastLinQ 10/25/40/50/100 GbE Ethernet and RoCE/RDMA Drivers 1.0.0.0-1vmw
• QLogic Fibre Channel HBA Driver 4.1.14.0-5vmw
• Solarflare XtremeScale NIC Driver 2.4.0.2010-4vmw
• VMware AHCI SATA Storage Controller Driver 2.0.9-1vmw
• Native icen network driver for VMware ESXi 1.0.0.10-1vmw
• VMware iSCSI Extensions over RDMA Driver 1.1.0.1-1vmw
• VMware NVMe over Fabrics – RDMA Driver 1.0.2.1-1vmw
• VMware NVMe PCI Express Storage Driver 1.2.3.11-1vmw
• VMware NVME PCIe LSU Management Plugin 1.0.0-1vmw
• esxcli plugin for VMware PCI Express NVMe 1.2.0.42-1vmw
• VMware VMXNET3 Virtual Ethernet Driver 2.0.0.30-1vmw
• VMWare VMXNET3 Virtual Network ENS Driver 2.0.0.22-1vmw
• VMware OEM DELL LSU Management Plugin 1.0.0-1vmw
• VMware OEM HP LSU Management Plugin 1.0.0-1vmw
• VMware OEM LENOVO LSU Management Plugin 1.0.0-1vmw
• VMware Paravirtual Storage Controller Driver 0.1-2vmw
• VMWare ATA Storage Controller Driver 0.1-1vmw
• VMware native Software FCoE Driver 1.0.0.2-1vmw
• VMWare USB Driver 0.1-4vmw
• ESXi VM Tools 11.2.6

As we can see in the two examples above, vmkusb (VMWare USB Driver) has two patches. One is Security, and the second is a Bugfix. As I discuss in my previous blog post, this was one of the USB/SD cards issue triggers.

If you want to apply this patch manually, you need to do it in the ESXi shell console. For example, in my case, I had one vSAN Cluster that could not fetch these updates, so I needed to use the manual option. Or import the patch to the vCenter Lifecycle Manager.

• Import Patch

First, you need to download the patch to import. Download from HERE (need VMware Customer account).

You need to select ESXi and version 7.0. If not, the specific patch will not be listed.

Second, import the patch.

In vCenter Lifecycle Manager, in the option actions, select Import Patch and select the patch file you downloaded.

Then after the patch is imported, and you can see it in the image depot. Then you can scan your ESXi/Cluster and then remediate.

• Apply patch manually in ESXi Console.

For applying the patch manually to your ESXi hosts through the ESXi host console, open an SSH connection and run the following commands.

If you want to apply as an offline Bundle, then you need to copy the file inside your ESXi host (using for example WinSCP) and then run the following command.

Note: You need always to use the file full path so that command can find the file to run the update.

Reboot, and your ESXi host is updated.

Updated 09/09/2021

Even is not related to the USB/SD bug issues, I would like to provide this information.

In one of my Clusters that are running Dell with the CPU

• • Model: PowerEdge R630
  • Processor Type: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz

I got this message when I try to apply the patch manually:

HardwareError]
Hardware precheck of profile ESXi-7.0U2c-18426014-standard failed with warnings: <CPU_SUPPORT WARNING: The CPU in this host may not be supported in future ESXi releases. Please plan accordingly.>

To bypass this warning and apply the patch we need to add the parameter –no -hardware-warning

Updated 26/09/2021

I will continue to post here some issues that we can encounter when applying this patch in several ESXi hosts(Dell, HPE, different models in vSAN Cluster, and without vSAN).

I still did not update all my ESXi (too many) so I will time to time encounter some issues.

This time with trying to apply the update to some HPE DL430 blades with vSAN in one ESXi host I get this error.

<em>[InstallationError]
Error untarring to /locker/packages/, return code: 1
stdout: b”
stderr: b’tar: invalid magic\ntar: short read\n’
cause = Error untarring to /locker/packages/, return code: 1
stdout: b”
stderr: b’tar: invalid magic\ntar: short read\n’
vibs = [‘VMware_locker_tools-light_11.2.6.17901274-18295176’]
Please refer to the log file for more details.</em>

In the logs I have:

<em>BootBankInstaller.py: WARNING: Ignoring error when loading bootbank: Error in loading boot.cfg from bootbank /tmp/stagebootbank: [Errno 2] No such file or directory: ‘/tmp/stagebootbank/boot.cfg’
vmware.runcommand: INFO: runcommand called with: args = ‘[‘/usr/lib/vmware/vob/bin/addvob’, ‘vob.user.esximage.install.stage.error’, ‘(Updated) ESXi-7.0U2c-18426014-standard’, “VMware_locker_tools-light_11.2.6.17901274-18295176: Error untarring to /locker/packages/, return code: 1\nstdout: b”\nstderr: b’tar: invalid magic\\ntar: short read\\n'”]’, outfile = ‘None’, returnoutput = ‘True’, timeout = ‘0.0’.</em>

In this case, the ESXi profile was corrupted and was not possible to apply the update. If I use the offline patch, I was able to apply without errors by launching the offline command. But then in the esxupdate.log I have some errors and was not possible to stage or remediate any other updates.

When this happens is common that we have an ESXi profile issue and most of the time is possible to fix as I explain HERE and HERE, but for this case, I don’t want to do this. Because of this SD cards issue, I may have also an SD corrupted. So the best choice here is to do a fresh install.

I always thought that the problem could be the VMTools RamDisk, but ESXi was properly set with the new guidelines for SD Cards, as I talked about in my initial workaround blog post about these SD cards issues. I use a script to move/migrate RamDisk,

Also, I was able to make sure was an ESXi profile issue when I try to create a Host Profile so that I can apply after I reinstall the ESXi it was not possible I always get errors. So, the profile on this ESXi was corrupted.

So I decided to remove the vSAN ESXi from Cluster format vSAN disks and then reinstall ESXi with vSphere 7.0.1 and then apply the U2c patch. The reason I apply the patch and did not stick to the 7.0.1 version is because of the disks format version (all needs to be in the same versions in vSAN).

After all was good and ESXi is running without any issues in the vSAN Cluster.

UPDATE 22/01/2022

There is a new patch version for this issue. So please download the new patch version

ESXi 7.0 Update 2d | 14 SEP 2021 | Build 18538813

The patch is from the 14th of September but had some updates on the 10th of January this year.

Does this patch fix the USB/SD Cards problem?

At the moment, I apply this patch in 2 Clusters. One is HPE DLs, and the other is also HPE but Blades DL that are running vSAN.

Since I apply the patch (25/08/2021), I do not see any issues. There are some log entries, but nothing special and nothing that we need to worry about. At least in my systems.

For my vSphere 7 U2a infrastructure, I will not apply the patch yet. I will leave these 2 Clusters running for more than a week, and if all is ok, I will apply to all the ESXi hosts next weekend.

I hope this blog post, VMware finally launched ESXi 7.0 Update 2c to address the bug in vSphere 7 Update 2 that loses connection with SD Cards. It can be useful to apply the patch and understand the problem and what was fixed in this new patch.

Share this article if you think it is worth sharing. If you have any questions or comments, comment here or contact me on Twitter.

©2021 ProVirtualzone. All Rights Reserved