Today when we try to upgrade one of our ESXi host we had a strange error.

There was an issue with one of the partitions and somehow the ESXi install was not able to reformat the partition for the upgrade. Maybe because a previous wrong configuration on the partitions.

VMware KB says: “The upgrade attempts to reformat the scratch partition (partition #2) with a VFAT filesystem. This mostly likely fails, as the maximum size allowed for VFAT is 4GB, and most VMFS datastores are larger than 4GB. A 4GB datastore may potentially be erased.”

Note: You can check this issue in the VMware KB: KB2015828

So need to investigate which type of partition is this and correct the problem before re-run the upgrade again.

This tasks can be done during the upgrade, with the Alt+F1 we can go to console during the upgrade, or just cancel the upgrade and reboot the server to start normally.

But unfortunately in this case was not possible, the upgrade re-started and ESXi begin loop and never went to normal boot. So go for the first option, correct the problem during the upgrade, or rollback the upgrade and start the ESXi with normal boot.

I decided to cancel and rollback the upgrade. To do this, I need to start the ESXi with the option recovery mode.

After the ESXi started I press SHIFT+R(check image)

After the recovery mode restart, the ESXi will start normally and then we can start the troubleshooting.

Check VMware KB for recovery mode: KB1033604

Connecting to ESXi console(with SSH) started to check the devices/partitions

Since this was an upgrade from ESXi 5.5 to 6.0, to check the partitions I need to use esxcfg-scsidevs command to check the device and partition that was in the error(mpx.vmhba32:C0:T0:L0:2)

# esxcfg-scsidevs -c

Device UID                            Device Type      Console Device                                            Size      Multipath PluginDisplay Name
mpx.vmhba32:C0:T0:L0                  Direct-Access    /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0                  7600MB    NMP     Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)
naa.600508b1001c4118e0e0f8b71eb9b654  Direct-Access    /vmfs/devices/disks/naa.600508b1001c4118e0e0f8b71eb9b654  1144609MB NMP     HP Serial Attached SCSI Disk (naa.600508b1001c4118e0e0f8b71eb9b654)
naa.60a980002d676739503f426f7675504d  Direct-Access    /vmfs/devices/disks/naa.60a980002d676739503f426f7675504d  768062MB  NMP     NETAPP iSCSI Disk (naa.60a980002d676739503f426f7675504d)

As we can see in the image error the device/partition that is preventing the upgrade is the vmhba32:C0:T0:L0. So I need to check the partition #2 in this device.

So we need to use partedUtil to check this.

# partedUtil getptbl /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0
gpt
968 255 63 15564800
1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128
5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0
8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0
2 15357952 15562751 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0 

The problem is in the #2 partition that is identify as the vmkDiagnostic partition. vmkDiagnostic partition is the coredump partition. So we need to fix the coredump partition.

Just to check the coredump partition run: esxcli system coredump.

# esxcli system coredump partition list

Name                    Path                                        Active  Configured
———————-  ——————————————  ——  ———-
mpx.vmhba32:C0:T0:L0:2  /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:2    true        true
mpx.vmhba32:C0:T0:L0:7  /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:7   false       false

I get 2 partitions in this ESXi for coredump. One active and other not active.

A previous ESXi installation and configuration was not very well configured.

So nothing to fix here, just delete the partition that is freezing the upgrade and re-run the upgrade.

To delete the partition we need to use the partedUtil again.

# partedUtil delete /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0 2

Error: Read-only file system during write on /dev/disks/mpx.vmhba32:C0:T0:L0

Unable to delete partition 2 from device /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

So since this a coredump and read-only partition we cannot delete before we disable the coredump.

# esxcli system coredump partition set –enable false

Lets check the coredump partitions again

# esxcli system coredump partition list
Name                    Path                                        Active  Configured
———————-  ——————————————  ——  ———-
mpx.vmhba32:C0:T0:L0:2  /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:2   false        true
mpx.vmhba32:C0:T0:L0:7  /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:7   false       false

Now coredump partition is disable, then we can delete the partition.

# partedUtil delete /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0 2

Checking partitions again, we can check that #2 was deleted.

# esxcli system coredump partition list

Since partition #7 is also set in the ESXi, I deleted also, then after upgrade we can create a new one, or the upgrade itself will create a new one.

Now we can re-run the upgrade again and will finish without any issues.

Hope this article can you help fixing this issue that you may encounter in your ESXi upgrade.