After some blog posts about the latest versions of Backup Tools is time for some technical blogging with this vMotion failed because of duplicated IP.

This week had a strange issue with one of our vCenters while trying to migrate some VMs from one of the vCenter Clusters. The migrations always failed when trying to put one host in maintenance mode, and it was not possible to evacuate the ESXi host.

The error itself on the tasks was not very helpful. Just that was not possible to migrate the xpto VM from ESXi host A to ESXi host B. But notice that ESXi had a warning that lost redundancy network.

So with a vmnic down need to check where the problem is.

Checking vCenter tasks and events to see if I get more information about this problem. Then I saw a lot of issues in the event, with vmnic4 and vmnic5 (Storage and vMotion interfaces) flapping up and down. But since these Interfaces, it is configured with LACP, that is normal behavior when you have an issue with one of the interfaces.

1. Check events.

Then I notice the main issue for this problem: “A duplicate IP address was detected for 100.10.58.43 on the interface vmk4. The current owner is 00:50:56:62:19:77.”

The root cause vMotion failed because duplicated IP, that is why some of the vMotion migrations are failing, and others are finishing with success.

Next, I need to check DNS to see if anyone made any mistake to delete/add an entry that was already in use. Found no issues in the DNS, vMotion vmkernel IP is set in the DNS for this ESXi host.

2. Check logs.

Next, I check the logs from VOB to check if I can get more information in /var/log.

The information in the VOB log is similar to what I have seen in the events, so I need to go to the next check.

3. Check the arp table.

So need to find where this mac address is. Checking arp table using the command “esxcli network ip neighbor list” you can check mac addresses, but you need to check each host one by one.

This is an example:

Checking a couple of ESXi hosts did not find anything about that duplicated IP and mac address. If you have dozens or hundreds of hosts, this is enormous work, so why not using PowerShell and search per vCenter.

4. Using PowerShell.

Open PowerShell connect to your vCenter and run the following command: Get-VMHost | Get-VMHostNetworkAdapter | Where-Object {$_.Mac -eq “00:50:56:62:19:77”} | Select VMHost, PortGroupName, DeviceName, Mac

Running the command in vCenter still nothing. So the problem must be in another vCenter. Run the PowerShell in all vCenters that we have in our infrastructure and in one vCenter… voila!

Found the vCenter vs ESXi host and the vmkernel that was using the same IP address for vMotion. It is strange since both ESXi hosts are in production for a couple of years now, and this never happens.

Checking the DNS entries again, this ESXi host for vMotion IP address have 100.10.58.12. So I don’t know who did make this mistake. So someone must have to make this change manually and add this used IP address on this ESXi host. Who did, is something for the next step, for now, I need to fix this asap.

Finally, to prevent this happen and I am not aware, I create a new alarm rule for duplicate IP addresses for all vCenters in the infrastructure. I created a post just for this, you can check here, How to create a vCenter Alarm for Duplicated IPs.

I hope this post will help you to troubleshoot and identify where is the duplicated IP.

Note: Share this article if you think it is worth sharing.

©2020 ProVirtualzone. All Rights Reserved