/VMware finally launched ESXi 7.0 Update 2c

VMware finally launched ESXi 7.0 Update 2c

VMware finally launched ESXi 7.0 Update 2c to fix the huge bug about USB/SD cards. This subject I have already discussed HERE in my workaround to fix the problem.

I did not have time to write this mandatory blog post about this new patch to fix the vSphere 7.0.U2a bug. I will try to drill down this patch and show some examples of how to apply, what is new, and what does fix it.

This subject was so important for so many customers that my previous post had a record of 25,000 clicks and 172 messages in just a couple of months, when my maximum was around 9,000 clicks in 6 months in the same post.

Customers were kind of “abandoned” for at least a month before VMware starts to acknowledge the issue and address this subject properly and provide temporary solutions, or even proposing my workaround 😉

For an issue serious as this one and that had impacted a lot in our running systems, for a lot of customers was not the best moment for VMware. But now we have the patch, time to move on.

As we all know(everyone suffering from this bug), this bug was addressed KB83647, KB2077516, and KB2149257. But the patch did also address and fix others issues that were in the last version. Like slow boot, issues with VM VMDK, some PSOD issues, and Hosts iSCSI Volumes.

Here is the list of all the fixes addressed in this patch.

Resolved Issues

The resolved issues are grouped as follows.

Full Release Notes

Where can we find the patch?

Before explaining how to install or find this patch, I need to explain it is a patch. Not a new version, but a new build(there is no vSphere ISO to install/upgrade). So if you have any customized image/install, like HPE, Dell or Lenovo, etc., no problems applying this patch. Since you are not upgrading your vSphere.

So is safe to apply the patch in any vSphere 7.0.x customize build that you may have.

For those asking if HPE, or Dell, or others will launch an ISO with this, my answer is NO. If this is not a new version and there is no new ISO with this version(at the moment), companies will not create/provide a new Customize ISO. You need to install the patch.

Install the latest ISO (from VMware or Customized) for a new install and then install the patch.

I am pretty sure VMware will launch a new version soon. But since this patch was urgent, they decided to launch the patch first, and the new ISO version should be next.

Note: By aware that this patch and issues to be fixed, is only if are running version ESXi 7.0 Update 2a build 17867351 and to fix those issues on that version. If you run previous versions, your system will also be updated to this VMware ESXi, 7.0.2, 18426014 build.

vCenter Lifecycle Manager will apply updates from your version eq 7.0.1 to 7.02c.

Important Note: Before you apply this patch, I recommend first upgrading your vCenter to the latest version. The new version was also launched on 24/08 and is also fixing some critical issues. So upgrade first your vCenter and only apply the patches. With this, we have your system up to date in using the same build. If not, you have your ESXi hosts with a higher version than vCenter.

Since this is a patch and not a new version, it is not mandatory but is recommended.

VMware finally launched ESXi 7.0 Update 2c

VMware finally launched ESXi 7.0 Update 2c

After you upgrade your vCenter to version build 18356314 you should apply the patches in the ESXi hosts.

You can update your ESXi hosts using vCenter Lifecycle Manager (previous VUM), or you can dot it manually one by one in the ssh command line.

In your vCenter Lifecycle Manager, Image Depot there are two ESXi versions:

  • ESXi 7.0.U2c 18426014 – Security and Bugfix patch.
  • ESXi 7.0U2sc 18295176 – Security only patch.

Are VMware patch and update releases contain general and critical image profiles. For this particular case, I don’t see any differences between both, maybe because it was a critical and bugfix patch release.

When running vCenter updates, these are the updates that will be applied from this vSphere 7.0 Update 2c patch. There are around 56 updates(vibs and ESXi updates) in my case, are 8 or 9(the extra update is if you need to update your VMware Tools version).

VMware finally launched ESXi 7.0 Update 2c

Full list of the vibs updates and ESXi updates:

  • Broadcom NetXtreme-E Network and ROCE/RDMA Drivers
  • Broadcom Emulex Connectivity Division fcoe driver for FCoE adapters 12.0.1500.1-2vmw
  • Broadcom Emulex Connectivity Division NVMe over Fabric driver for FC adapters
  • Broadcom IMA plugin for Emulex OneConnect iSCSI Driver 12.0.1200.0-4vmw
  • Emulex OneConnect iSCSI Driver 12.0.1200.0-8vmw
  • Broadcom Emulex Connectivity Division lpfc driver for FC adapters
  • Emulex OneConnect Network Driver 12.0.1250.0-5vmw
  • Broadcom Emulex OneConnect Network Driver
  • Broadcom 12Gbps SAS/PCIe MegaRAID Driver 7.716.03.00-1vmw
  • Broadcom 6Gbps SAS HBA Driver
  • Broadcom 12Gbps SAS HBA Driver
  • Broadcom 12Gbps SAS/PCIe HBA Driver
  • Broadcom LSI NATIVE DRIVERS LSU Management Plugin v2 1.0.0-5vmw
  • Broadcom NetXtreme I ESX VMKAPI ethernet driver
  • Cisco VIC Ethernet Driver
  • Cisco VIC FCoE Driver
  • ESXi 7.0.2 Build – 18426014
  • ESXi Install/Upgrade Component 7.0.2 Build 18426014
  • HPE HPSA LSU V2 Management Plugin 1.0.0-3vmw
  • HPE Smart Array Controller Driver 70.0051.0.100-2vmw
  • Intel Network Driver for Intel(R) X710/XL710/XXV710/X722 Adapters
  • Intel Network Driver for Intel(R) 1 Gigabit Server Adapters
  • Network driver for Intel(R) X722 and E810 based RDMA Adapters
  • Intel Network Driver for Intel(R) 10 Gigabit Adapters
  • Intel Gigabit Ethernet Connection Driver 0.8.4-11vmw
  • Intel(R) NVMe Volume Management Device enabled Driver Plugin v2 2.0.0-2vmw
  • Intel SCU SAS/SATA for VMware ESX
  • Intel(R) Volume Management Device enabled NVME Driver
  • Mellanox Technologies ConnectX-3/Pro Core Ethernet and RoCE Drivers
  • Mellanox Technologies ConnectX-4/5 Core Ethernet and RoCE Drivers
  • Microsemi Storage Solution Smart Array Storage Controller Driver 70.4000.0.100-6vmw
  • Microsemi Storage Solution Smart Array Storage Controller Driver Plugin v2 1.0.0-6vmw
  • Micron RealSSD PCIe Native Driver 3.9.8-1vmw
  • Marvell AQtion Ethernet Controllers Network driver for VMware ESXi
  • QLogic NetXtreme II 1 Gigabit Ethernet Driver
  • QLogic NetXtreme II 10 Gigabit Ethernet FCoE and iSCSI E3 Drivers
  • QLogic FastLinQ 10/25/40/50/100 GbE Ethernet and RoCE/RDMA Drivers
  • QLogic Fibre Channel HBA Driver
  • Solarflare XtremeScale NIC Driver
  • VMware AHCI SATA Storage Controller Driver 2.0.9-1vmw
  • Native icen network driver for VMware ESXi
  • VMware iSCSI Extensions over RDMA Driver
  • VMware NVMe over Fabrics – RDMA Driver
  • VMware NVMe PCI Express Storage Driver
  • VMware NVME PCIe LSU Management Plugin 1.0.0-1vmw
  • esxcli plugin for VMware PCI Express NVMe
  • VMware VMXNET3 Virtual Ethernet Driver
  • VMWare VMXNET3 Virtual Network ENS Driver
  • VMware OEM DELL LSU Management Plugin 1.0.0-1vmw
  • VMware OEM HP LSU Management Plugin 1.0.0-1vmw
  • VMware OEM LENOVO LSU Management Plugin 1.0.0-1vmw
  • VMware Paravirtual Storage Controller Driver 0.1-2vmw
  • VMWare ATA Storage Controller Driver 0.1-1vmw
  • VMware native Software FCoE Driver
  • VMWare USB Driver 0.1-4vmw
  • ESXi VM Tools 11.2.6

As we can see in the two examples above, vmkusb (VMWare USB Driver) has two patches. One is Security, and the second is a Bugfix. As I discuss in my previous blog post, this was one of the USB/SD cards issue triggers.

If you want to apply this patch manually, you need to do it in the ESXi shell console. For example, in my case, I had one vSAN Cluster that could not fetch these updates, so I needed to use the manual option. Or import the patch to the vCenter Lifecycle Manager.

  • Import Patch

First, you need to download the patch to import. Download from HERE (need VMware Customer account).

You need to select ESXi and version 7.0. If not, the specific patch will not be listed.

VMware finally launched ESXi 7.0 Update 2c

Second, import the patch.

In vCenter Lifecycle Manager, in the option actions, select Import Patch and select the patch file you downloaded.

Then after the patch is imported, and you can see it in the image depot. Then you can scan your ESXi/Cluster and then remediate.

  • Apply patch manually in ESXi Console.

For applying the patch manually to your ESXi hosts through the ESXi host console, open an SSH connection and run the following commands.

If you want to apply as an offline Bundle, then you need to copy the file inside your ESXi host (using for example WinSCP) and then run the following command.

Note: You need always to use the file full path so that command can find the file to run the update.

Reboot, and your ESXi host is updated.

VMware finally launched ESXi 7.0 Update 2c

Updated 09/09/2021

Even is not related to the USB/SD bug issues, I would like to provide this information.

In one of my Clusters that are running Dell with the CPU

    • Model: PowerEdge R630
    • Processor Type: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz

I got this message when I try to apply the patch manually:

Hardware precheck of profile ESXi-7.0U2c-18426014-standard failed with warnings: <CPU_SUPPORT WARNING: The CPU in this host may not be supported in future ESXi releases. Please plan accordingly.>

To bypass this warning and apply the patch we need to add the parameter –no -hardware-warning

Updated 26/09/2021

I will continue to post here some issues that we can encounter when applying this patch in several ESXi hosts(Dell, HPE, different models in vSAN Cluster, and without vSAN).

I still did not update all my ESXi (too many) so I will time to time encounter some issues.

This time with trying to apply the update to some HPE DL430 blades with vSAN in one ESXi host I get this error.

Error untarring to /locker/packages/, return code: 1
stdout: b”
stderr: b’tar: invalid magic\ntar: short read\n’
cause = Error untarring to /locker/packages/, return code: 1
stdout: b”
stderr: b’tar: invalid magic\ntar: short read\n’
vibs = [‘VMware_locker_tools-light_11.2.6.17901274-18295176’]
Please refer to the log file for more details.</em>

In the logs I have:

<em>BootBankInstaller.py: WARNING: Ignoring error when loading bootbank: Error in loading boot.cfg from bootbank /tmp/stagebootbank: [Errno 2] No such file or directory: ‘/tmp/stagebootbank/boot.cfg’
vmware.runcommand: INFO: runcommand called with: args = ‘[‘/usr/lib/vmware/vob/bin/addvob’, ‘vob.user.esximage.install.stage.error’, ‘(Updated) ESXi-7.0U2c-18426014-standard’, “VMware_locker_tools-light_11.2.6.17901274-18295176: Error untarring to /locker/packages/, return code: 1\nstdout: b”\nstderr: b’tar: invalid magic\\ntar: short read\\n'”]’, outfile = ‘None’, returnoutput = ‘True’, timeout = ‘0.0’.</em>

In this case, the ESXi profile was corrupted and was not possible to apply the update. If I use the offline patch, I was able to apply without errors by launching the offline command. But then in the esxupdate.log I have some errors and was not possible to stage or remediate any other updates.

When this happens is common that we have an ESXi profile issue and most of the time is possible to fix as I explain HERE and HERE, but for this case, I don’t want to do this. Because of this SD cards issue, I may have also an SD corrupted. So the best choice here is to do a fresh install.

I always thought that the problem could be the VMTools RamDisk, but ESXi was properly set with the new guidelines for SD Cards, as I talked about in my initial workaround blog post about these SD cards issues. I use a script to move/migrate RamDisk,

Also, I was able to make sure was an ESXi profile issue when I try to create a Host Profile so that I can apply after I reinstall the ESXi it was not possible I always get errors. So, the profile on this ESXi was corrupted.

So I decided to remove the vSAN ESXi from Cluster format vSAN disks and then reinstall ESXi with vSphere 7.0.1 and then apply the U2c patch. The reason I apply the patch and did not stick to the 7.0.1 version is because of the disks format version (all needs to be in the same versions in vSAN).

After all was good and ESXi is running without any issues in the vSAN Cluster.

UPDATE 22/01/2022

There is a new patch version for this issue. So please download the new patch version

ESXi 7.0 Update 2d | 14 SEP 2021 | Build 18538813

The patch is from the 14th of September but had some updates on the 10th of January this year.

Does this patch fix the USB/SD Cards problem?

At the moment, I apply this patch in 2 Clusters. One is HPE DLs, and the other is also HPE but Blades DL that are running vSAN.

Since I apply the patch (25/08/2021), I do not see any issues. There are some log entries, but nothing special and nothing that we need to worry about. At least in my systems.

For my vSphere 7 U2a infrastructure, I will not apply the patch yet. I will leave these 2 Clusters running for more than a week, and if all is ok, I will apply to all the ESXi hosts next weekend.

I hope this blog post, VMware finally launched ESXi 7.0 Update 2c to address the bug in vSphere 7 Update 2 that loses connection with SD Cards. It can be useful to apply the patch and understand the problem and what was fixed in this new patch.

Share this article if you think it is worth sharing. If you have any questions or comments, comment here or contact me on Twitter.

©2021 ProVirtualzone. All Rights Reserved


By | 2022-01-20T15:08:48+01:00 September 1st, 2021|VMware Posts, vSAN, vSphere|22 Comments

About the Author:

I have over 20 years of experience in the IT industry. I have been working with Virtualization for more than 15 years (mainly VMware). I recently obtained certifications, including VCP DCV 2022, VCAP DCV Design 2023, and VCP Cloud 2023. Additionally, I have VCP6.5-DCV, VMware vSAN Specialist, vExpert vSAN, vExpert NSX, vExpert Cloud Provider for the last two years, and vExpert for the last 7 years and a old MCP. My specialties are Virtualization, Storage, and Virtual Backup. I am a Solutions Architect in the area VMware, Cloud and Backup / Storage. I am employed by ITQ, a VMware partner as a Senior Consultant. I am also a blogger and owner of the blog ProVirtualzone.com


  1. Juha-Pekka Lehtonen 09/09/2021 at 09:36

    You have a typo there in build number ESXi 7.0.U2c 184426014 – Security and Bugfix patch.
    It is ESXi 7.0.U2c 18426014

  2. Oliver 10/09/2021 at 13:55

    Have you seen any errors with updating to this patch on HPE hosts? Using Lifecycle Manager gives me error ” The host returns esxupdate error codes: -1″ and updating manually via SSH I get error “Error while waiting for untar process ‘[‘/bin/tar’, ‘xzf’, ‘-‘, ‘-C’, ‘/locker/packages/’]’: Timeout (30 seconds) expired waiting for output from command ‘[‘/bin/tar’, ‘xzf’, ‘-‘, ‘-C’, ‘/locker/packages/’]’, pid ‘40843937’.”

    I tried purging /locker/packages/, I tried reinstalling previous versions of 7.0.2 and 7.0.1, but still same error messages. I am using HPE builds if that matters at all.

    • Luciano Patrao 13/09/2021 at 15:12

      Hi Oliver,

      Never that one. Do you have this in one servers, or all that you have this HPE builts?

      • Oliver 21/09/2021 at 14:18

        It’s really weird. The cluster exists of 8 servers running the HPE build. 6/8 updated just fine. 2 servers got this exact error. 1 server got fixed when reinstalling 7.0.0, tried the exact same process on the 2nd problem server, but got same error again.

  3. Philipp Gruhn 12/09/2021 at 13:43

    I just tried to update my lone ESXi 7 via
    esxcli software vib update -d /vmfs/volumes/…/VMware-ESXi-7.0U2c-18426014-depot.zip
    and got the error message:
    This upgrade transaction would skip ESXi Base Image VIB(s) VMW_bootbank_i40enu_1.8.1.137-1vmw.702.0.20.18426014, which could cause failures post upgrade. Please use an alternative upgrade method that would install the above VIB(s). For example, use an image profile instead of VIBs to perform the upgrade.
    Please refer to the log file for more details.”

    Any idea, which logfile to look for? A quick look at vmkwarning and vmkernel didn’t bring up anyting useful.
    This is the FJT-Addon-for-FujitsuCustomImage_7.0.2-520.1.0 (Fujitsu) installation I commented on on the previous thread.

    The issue happened every 24hours the last week, so I finally decided to try the update, but fell short sadly.

    Couldn’t download VUM or Lifecycle manager yet, as we purchased our license through a third party.

    • Philipp Gruhn 12/09/2021 at 13:53

      According to https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u2a-release-notes.html this is as intended.
      “However, to avoid the errors with the esxcli software vib update command, you can use the esxcli software profile update command instead to perform an ESXi upgrade by using an image profile from a VMware or third-party depot.”

      I just wonder, if I can apply the same patch just via software profile update instead of software vib update. I’m a bit afraid to break this installation I guess. Read somewhere, that you can’t roll back an online update.

    • Luciano Patrao 13/09/2021 at 15:15

      Have you tried the online update through the command line? Never liked your type of upgrade for such a patch with this size and importance.
      But you could also remove the vib that is getting this error.

      • Luciano Patrao 13/09/2021 at 15:17

        try to check esxupdate.log for more information.

      • Philipp Gruhn 13/09/2021 at 15:40

        Didn’t pull the trigger on the online update (I guess via software profile update) yet as I would rather try that on a weekend, in case the server doesn’t boot up properly anymore. Bootbank vibs sound like possible trouble to me.

  4. Alain Sylvestre 24/09/2021 at 22:26

    Thank for your great work.

    It seem that the bug corrupt my usbkey because I am unable to patch it with 2c. I have this eerror message : ….
    Failed to copy files from /tmp/stagebootbank to /altbootbank: [Errno 116] Stale file handle: ‘/altbootbank/boot.cfg’
    cause = [Errno 116] Stale file handle: ‘/altbootbank/boot.cfg’

    anyone have see this ?

  5. Martin Pika 01/10/2021 at 09:50

    I would like to ask if there’s anyone using vROps and is seeing that VM/ESXi host CPU IOwait is rising after upgrade?

    • Luciano Patrao 01/10/2021 at 10:15

      Unfortunately, I cannot help you here, I do not use vROPs. We have teams that are working with that in a different infrastructure, but they are not running vSphere 7.0.2

  6. Stefan Kuhn 15/10/2021 at 06:44

    Now ESXi-7.0U3 build 18644231 is out
    I have downloaded the Dell ISO and installed it on one of our dual SD card of our Dell host.
    No problems so far, but will it fix our SD card problem that we had with previous releases?
    Did you get any feedback yet?

    • Luciano Patrao 15/10/2021 at 14:04

      Hi Stefan,

      Did not install any of the new vendor ISP for update 3 yet, so cannot answer your question.
      Plan to do some updates during this weekend and next week some new install using HPE ISO.

  7. David Webster 19/01/2022 at 22:51

    Nice articles. I hit the issue last night, and found your work around page. It may be helpful to put at the top of your work around page that the patch is released. I did not know until after hours waiting for VMware to call me back…Thanks for the great articles, very technical.

    • Luciano Patrao 20/01/2022 at 14:59

      Hi David,

      Thanks for your comment, and yes I agreed this information about the patch should be in the workaround blog post also.
      I will update the page.

      Also be aware that the patch has a new version, VMware-ESXi-7.0U2d-18538813. Will also update this page with the new information.

Leave A Comment