/VMware finally launched ESXi 7.0 Update 2c

VMware finally launched ESXi 7.0 Update 2c


VMware finally launched ESXi 7.0 Update 2c to fix the huge bug about USB/SD cards. This subject I have already discussed HERE in my workaround to fix the problem.

I did not have time to write this mandatory blog post about this new patch to fix the vSphere 7.0.U2a bug. I will try to drill down this patch and show some examples of how to apply, what is new, and what does fix it.

This subject was so important for so many customers that my previous post had a record of 25,000 clicks and 172 messages in just a couple of months, when my maximum was around 9,000 clicks in 6 months in the same post.

Customers were kind of “abandoned” for at least a month before VMware starts to acknowledge the issue and address this subject properly and provide temporary solutions, or even proposing my workaround πŸ˜‰

For an issue serious as this one and that had impacted a lot in our running systems, for a lot of customers was not the best moment for VMware. But now we have the patch, time to move on.

As we all know(everyone suffering from this bug), this bug was addressed KB83647, KB2077516, and KB2149257. But the patch did also address and fix others issues that were in the last version. Like slow boot, issues with VM VMDK, some PSOD issues, and Hosts iSCSI Volumes.

Here is the list of all the fixes addressed in this patch.

Resolved Issues

The resolved issues are grouped as follows.

Full Release Notes

Where can we find the patch?

Before explaining how to install or find this patch, I need to explain it is a patch. Not a new version, but a new build(there is no vSphere ISO to install/upgrade). So if you have any customize image/install, like HPE, Dell or Lenovo, etc., no problems applying this patch. Since you are not upgrading your vSphere.

So is safe to apply the patch in any vSphere 7.0.x customize build that you may have.

For those asking if HPE, or Dell, or others will launch an ISO with this, my answer is NO. If this is not a new version and there is no new ISO with this version(at the moment), companies will not create/provide a new Customize ISO. You need to install the patch.

Install the latest ISO (from VMware or Customized) for a new install and then install the patch.

I am pretty sure VMware will launch a new version soon. But since this patch was urgent, they decided to launch the patch first, and the new ISO version should be next.

Note: By aware that this patch and issues to be fixed, is only if are running version ESXi 7.0 Update 2a build 17867351 and to fix those issues on that version. If you run previous versions, your system will also be updated to this VMware ESXi, 7.0.2, 18426014 build.

vCenter Lifecycle Manager will apply updates from your version eq 7.0.1 to 7.02c.

Important Note: Before you apply this patch, I recommend first upgrade your vCenter to the latest version. The new version was also launched on 24/08 and is also fixing some critical issues. So upgrade first your vCenter and only apply the patches. With this, we have your system up to date in using the same build. If not, you have your ESXi hosts with a higher version than vCenter.

Since this is a patch and not a new version, it is not mandatory but is recommended.

VMware finally launched ESXi 7.0 Update 2c

VMware finally launched ESXi 7.0 Update 2c

After you upgrade your vCenter to version 7.0.2.00400 build 18356314 you should apply the patches in the ESXi hosts.

You can update your ESXi hosts using vCenter Lifecycle Manager (previous VUM), or you can dot it manually one by one in the ssh command line.

In your vCenter Lifecycle Manager, Image Depot there are two ESXi versions:

  • ESXi 7.0.U2c 18426014 – Security and Bugfix patch.
  • ESXi 7.0U2sc 18295176 – Security only patch.

Are VMware patch and update releases contain general and critical image profiles. For this particular case, I don’t see any differences between both, maybe because it was a critical and bugfix patch release.

When running vCenter updates, these are the updates that will be applied from this vSphere 7.0 Update 2c patch. There are around 56 updates(vibs and ESXi updates) in my case, are 8 or 9(the extra update is if you need to update your VMware Tools version).

VMware finally launched ESXi 7.0 Update 2c

Full list of the vibs updates and ESXi updates:

  • Broadcom NetXtreme-E Network and ROCE/RDMA Drivers 216.0.0.0-1vmw
  • Broadcom Emulex Connectivity Division fcoe driver for FCoE adapters 12.0.1500.1-2vmw
  • Broadcom Emulex Connectivity Division NVMe over Fabric driver for FC adapters 12.8.298.1-1vmw
  • Broadcom IMA plugin for Emulex OneConnect iSCSI Driver 12.0.1200.0-4vmw
  • Emulex OneConnect iSCSI Driver 12.0.1200.0-8vmw
  • Broadcom Emulex Connectivity Division lpfc driver for FC adapters 12.8.298.3-2vmw
  • Emulex OneConnect Network Driver 12.0.1250.0-5vmw
  • Broadcom Emulex OneConnect Network Driver 11.4.62.0-1vmw
  • Broadcom 12Gbps SAS/PCIe MegaRAID Driver 7.716.03.00-1vmw
  • Broadcom 6Gbps SAS HBA Driver 20.00.06.00-3vmw
  • Broadcom 12Gbps SAS HBA Driver 17.00.10.00-2vmw
  • Broadcom 12Gbps SAS/PCIe HBA Driver 17.00.02.00-1vmw
  • Broadcom LSI NATIVE DRIVERS LSU Management Plugin v2 1.0.0-5vmw
  • Broadcom NetXtreme I ESX VMKAPI ethernet driver 4.1.5.0-0vmw
  • Cisco VIC Ethernet Driver 1.0.33.0-1vmw
  • Cisco VIC FCoE Driver 4.0.0.63-1vmw
  • ESXi 7.0.2 Build – 18426014
  • ESXi Install/Upgrade Component 7.0.2 Build 18426014
  • HPE HPSA LSU V2 Management Plugin 1.0.0-3vmw
  • HPE Smart Array Controller Driver 70.0051.0.100-2vmw
  • Intel Network Driver for Intel(R) X710/XL710/XXV710/X722 Adapters 1.8.1.137-1vmw
  • Intel Network Driver for Intel(R) 1 Gigabit Server Adapters 1.4.11.2-1vmw
  • Network driver for Intel(R) X722 and E810 based RDMA Adapters 1.3.1.19-1vmw
  • Intel Network Driver for Intel(R) 10 Gigabit Adapters 1.7.1.35-1vmw
  • Intel Gigabit Ethernet Connection Driver 0.8.4-11vmw
  • Intel(R) NVMe Volume Management Device enabled Driver Plugin v2 2.0.0-2vmw
  • Intel SCU SAS/SATA for VMware ESX 2.0.2.0088-7vmw
  • Intel(R) Volume Management Device enabled NVME Driver 2.0.0.1152-1vmw
  • Mellanox Technologies ConnectX-3/Pro Core Ethernet and RoCE Drivers 3.19.16.8-2vmw
  • Mellanox Technologies ConnectX-4/5 Core Ethernet and RoCE Drivers 4.19.16.10-1vmw
  • Microsemi Storage Solution Smart Array Storage Controller Driver 70.4000.0.100-6vmw
  • Microsemi Storage Solution Smart Array Storage Controller Driver Plugin v2 1.0.0-6vmw
  • Micron RealSSD PCIe Native Driver 3.9.8-1vmw
  • Marvell AQtion Ethernet Controllers Network driver for VMware ESXi 1.0.3.0-8vmw
  • QLogic NetXtreme II 1 Gigabit Ethernet Driver 1.1.0.11-1vmw
  • QLogic NetXtreme II 10 Gigabit Ethernet FCoE and iSCSI E3 Drivers 1.0.0.0-1vmw
  • QLogic FastLinQ 10/25/40/50/100 GbE Ethernet and RoCE/RDMA Drivers 1.0.0.0-1vmw
  • QLogic Fibre Channel HBA Driver 4.1.14.0-5vmw
  • Solarflare XtremeScale NIC Driver 2.4.0.2010-4vmw
  • VMware AHCI SATA Storage Controller Driver 2.0.9-1vmw
  • Native icen network driver for VMware ESXi 1.0.0.10-1vmw
  • VMware iSCSI Extensions over RDMA Driver 1.1.0.1-1vmw
  • VMware NVMe over Fabrics – RDMA Driver 1.0.2.1-1vmw
  • VMware NVMe PCI Express Storage Driver 1.2.3.11-1vmw
  • VMware NVME PCIe LSU Management Plugin 1.0.0-1vmw
  • esxcli plugin for VMware PCI Express NVMe 1.2.0.42-1vmw
  • VMware VMXNET3 Virtual Ethernet Driver 2.0.0.30-1vmw
  • VMWare VMXNET3 Virtual Network ENS Driver 2.0.0.22-1vmw
  • VMware OEM DELL LSU Management Plugin 1.0.0-1vmw
  • VMware OEM HP LSU Management Plugin 1.0.0-1vmw
  • VMware OEM LENOVO LSU Management Plugin 1.0.0-1vmw
  • VMware Paravirtual Storage Controller Driver 0.1-2vmw
  • VMWare ATA Storage Controller Driver 0.1-1vmw
  • VMware native Software FCoE Driver 1.0.0.2-1vmw
  • VMWare USB Driver 0.1-4vmw
  • ESXi VM Tools 11.2.6

As we can see in the two examples above, vmkusb (VMWare USB Driver) has two patches. One is Security, and the second is a Bugfix. As I discuss in my previous blog post, this was one of the USB/SD cards issue triggers.

If you want to apply this patch manually, you need to do it in the ESXi shell console. For example, in my case, I had one vSAN Cluster that could not fetch these updates, so I needed to use the manual option. Or import the patch to the vCenter Lifecycle Manager.

  • Import Patch

First, you need to download the patch to import. Download from HERE (need VMware Customer account).

You need to select ESXi and version 7.0. If not, the specific patch will not be listed.

VMware finally launched ESXi 7.0 Update 2c

Second, import the patch.

In vCenter Lifecycle Manager, in the option actions, select Import Patch and select the patch file you downloaded.

Then after the patch is imported, and you can see it in the image depot. Then you can scan your ESXi/Cluster and then remediate.

  • Apply patch manually in ESXi Console.

For applying the patch manually to your ESXi hosts through the ESXi host console, open an SSH connection and run the following commands.

If you want to apply as an offline Bundle, then you need to copy the file inside your ESXi host (using for example WinSCP) and then run the following command.

Note: You need always to use the file full path so that command can find the file to run the update.

Reboot, and your ESXi host is updated.

VMware finally launched ESXi 7.0 Update 2c

Updated 09/09/2021

Even is not related to the USB/SD bug issues, I would like to provide this information.

In one of my Clusters that are running Dell with the CPU

    • Model: PowerEdge R630
    • Processor Type: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz

I got this message when I try to apply the patch manually:

HardwareError]
Hardware precheck of profile ESXi-7.0U2c-18426014-standard failed with warnings: <CPU_SUPPORT WARNING: The CPU in this host may not be supported in future ESXi releases. Please plan accordingly.>

To bypass this warning and apply the patch we need to add the parameter –no -hardware-warning

Updated 26/09/2021

I will continue to post here some issues that we can encounter when applying this patch in several ESXi hosts(Dell, HPE, different models in vSAN Cluster, and without vSAN).

I still did not update all my ESXi (too many) so I will time to time encounter some issues.

This time with trying to apply the update to some HPE DL430 blades with vSAN in one ESXi host I get this error.

<em>[InstallationError]
Error untarring to /locker/packages/, return code: 1
stdout: b”
stderr: b’tar: invalid magic\ntar: short read\n’
cause = Error untarring to /locker/packages/, return code: 1
stdout: b”
stderr: b’tar: invalid magic\ntar: short read\n’
vibs = [‘VMware_locker_tools-light_11.2.6.17901274-18295176’]
Please refer to the log file for more details.</em>

In the logs I have:

<em>BootBankInstaller.py: WARNING: Ignoring error when loading bootbank: Error in loading boot.cfg from bootbank /tmp/stagebootbank: [Errno 2] No such file or directory: ‘/tmp/stagebootbank/boot.cfg’
vmware.runcommand: INFO: runcommand called with: args = ‘[‘/usr/lib/vmware/vob/bin/addvob’, ‘vob.user.esximage.install.stage.error’, ‘(Updated) ESXi-7.0U2c-18426014-standard’, “VMware_locker_tools-light_11.2.6.17901274-18295176: Error untarring to /locker/packages/, return code: 1\nstdout: b”\nstderr: b’tar: invalid magic\\ntar: short read\\n'”]’, outfile = ‘None’, returnoutput = ‘True’, timeout = ‘0.0’.</em>

In this case, the ESXi profile was corrupted, and was not possible to apply the update. If I use the offline patch, I was able to apply without errors by launching the offline command. But then in the esxupdate.log I have some errors and was not possible to stage or remediate any other updates.

When this happens is common that we have an ESXi profile issue and most of the time is possible to fix as I explain HERE and HERE, but for this case, I don’t want to do this. Because of this SD cards issue, I may have also an SD corrupted. So the best choice here is to do a fresh install.

I always thought that the problem could be the VMTools RamDisk, but ESXi was properly set with the new guidelines for SD Cards, as I talked about in my initial workaround blog post about these SD cards issues. I use a script to move/migrate RamDisk,

Also, I was able to make sure was an ESXi profile issue when I try to create a Host Profile so that I can apply after I reinstall the ESXi it was not possible I always get errors. So, the profile on this ESXi was corrupted.

So I decided to remove the vSAN ESXi from Cluster format vSAN disks and then reinstall ESXi with vSphere 7.0.1 and then apply the U2c patch. The reason I apply the patch and did not stick to the 7.0.1 version is because of the disks format version (all needs to be in the same versions in vSAN).

After all was good and ESXi is running without any issues in the vSAN Cluster.

Does this patch fix the USB/SD Cards problem?

At the moment, I apply this patch in 2 Clusters. One is HPE DLs, and the other is also HPE but Blades DL that are running vSAN.

Since I apply the patch (25/08/2021), I do not see any issues. There are some log entries, but nothing special and nothing that we need to worry about. At least in my systems.

For my vSphere 7 U2a infrastructure, I will not apply the patch yet. I will leave these 2 Clusters running for more than a week, and if all is ok, I will apply to all the ESXi hosts next weekend.

I hope this blog post, VMware finally launched ESXi 7.0 Update 2c to address the bug in vSphere 7 Update 2 that loses connection with SD Cards. It can be useful to apply the patch and understand the problem and what was fixed in this new patch.

Share this article if you think it is worth sharing. If you have any questions or comments, comment here or contact me on Twitter.

Β©2021 ProVirtualzone. All Rights Reserved

 

By | 2021-09-26T17:56:25+02:00 September 1st, 2021|VMware, vSAN, vSphere|16 Comments

About the Author:

I am over 20 years’ experience in the IT industry. Working with Virtualization for more than 10 years (mainly VMware). I am an MCP, VCP6.5-DCV, VMware vSAN Specialist, Veeam Vanguard 2018/2019, vExpert vSAN 2018/2019 and vExpert for the last 4 years. Specialties are Virtualization, Storage, and Virtual Backups. I am working for Elits a Swedish consulting company and allocated to a Swedish multinational networking and telecommunications company as a Teach Lead and acting as a Senior ICT Infrastructure Engineer. I am a blogger and owner of the blog ProVirtualzone.com

16 Comments

  1. Juha-Pekka Lehtonen 09/09/2021 at 09:36 - Reply

    You have a typo there in build number ESXi 7.0.U2c 184426014 – Security and Bugfix patch.
    It is ESXi 7.0.U2c 18426014

  2. Oliver 10/09/2021 at 13:55 - Reply

    Have you seen any errors with updating to this patch on HPE hosts? Using Lifecycle Manager gives me error ” The host returns esxupdate error codes: -1″ and updating manually via SSH I get error “Error while waiting for untar process ‘[‘/bin/tar’, ‘xzf’, ‘-‘, ‘-C’, ‘/locker/packages/’]’: Timeout (30 seconds) expired waiting for output from command ‘[‘/bin/tar’, ‘xzf’, ‘-‘, ‘-C’, ‘/locker/packages/’]’, pid ‘40843937’.”

    I tried purging /locker/packages/, I tried reinstalling previous versions of 7.0.2 and 7.0.1, but still same error messages. I am using HPE builds if that matters at all.

    • Luciano Patrao 13/09/2021 at 15:12 - Reply

      Hi Oliver,

      Never that one. Do you have this in one servers, or all that you have this HPE builts?

      • Oliver 21/09/2021 at 14:18 - Reply

        It’s really weird. The cluster exists of 8 servers running the HPE build. 6/8 updated just fine. 2 servers got this exact error. 1 server got fixed when reinstalling 7.0.0, tried the exact same process on the 2nd problem server, but got same error again.

  3. Philipp Gruhn 12/09/2021 at 13:43 - Reply

    I just tried to update my lone ESXi 7 via
    esxcli software vib update -d /vmfs/volumes/…/VMware-ESXi-7.0U2c-18426014-depot.zip
    and got the error message:
    “[InstallationError]
    This upgrade transaction would skip ESXi Base Image VIB(s) VMW_bootbank_i40enu_1.8.1.137-1vmw.702.0.20.18426014, which could cause failures post upgrade. Please use an alternative upgrade method that would install the above VIB(s). For example, use an image profile instead of VIBs to perform the upgrade.
    Please refer to the log file for more details.”

    Any idea, which logfile to look for? A quick look at vmkwarning and vmkernel didn’t bring up anyting useful.
    This is the FJT-Addon-for-FujitsuCustomImage_7.0.2-520.1.0 (Fujitsu) installation I commented on on the previous thread.

    The issue happened every 24hours the last week, so I finally decided to try the update, but fell short sadly.

    Couldn’t download VUM or Lifecycle manager yet, as we purchased our license through a third party.

    • Philipp Gruhn 12/09/2021 at 13:53 - Reply

      According to https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u2a-release-notes.html this is as intended.
      “However, to avoid the errors with the esxcli software vib update command, you can use the esxcli software profile update command instead to perform an ESXi upgrade by using an image profile from a VMware or third-party depot.”

      I just wonder, if I can apply the same patch just via software profile update instead of software vib update. I’m a bit afraid to break this installation I guess. Read somewhere, that you can’t roll back an online update.

      • Luciano Patrao 13/09/2021 at 15:16 - Reply

        As long the build number changes(and in this case changes) is always possible o rollback.

    • Luciano Patrao 13/09/2021 at 15:15 - Reply

      Have you tried the online update through the command line? Never liked your type of upgrade for such a patch with this size and importance.
      But you could also remove the vib that is getting this error.

      • Luciano Patrao 13/09/2021 at 15:17 - Reply

        try to check esxupdate.log for more information.

      • Philipp Gruhn 13/09/2021 at 15:40 - Reply

        Didn’t pull the trigger on the online update (I guess via software profile update) yet as I would rather try that on a weekend, in case the server doesn’t boot up properly anymore. Bootbank vibs sound like possible trouble to me.

  4. Alain Sylvestre 24/09/2021 at 22:26 - Reply

    Thank for your great work.

    It seem that the bug corrupt my usbkey because I am unable to patch it with 2c. I have this eerror message : ….
    [InstallationError]
    Failed to copy files from /tmp/stagebootbank to /altbootbank: [Errno 116] Stale file handle: ‘/altbootbank/boot.cfg’
    cause = [Errno 116] Stale file handle: ‘/altbootbank/boot.cfg’

    anyone have see this ?

    • Luciano Patrao 26/09/2021 at 15:21 - Reply

      Yes, I had one with a similar issue, I just updated the blog post with the information.

Leave a Reply

%d bloggers like this: