vMotion failed because duplicated IP

//vMotion failed because duplicated IP

vMotion failed because duplicated IP

After some blog posts about the latest versions of Backup Tools is time for some technical blogging with this vMotion failed because of duplicated IP.

This week had a strange issue with one of our vCenters while trying to migrate some VMs from one of the vCenter Clusters. The migrations always failed when trying to put one host in maintenance mode, and it was not possible to evacuate the ESXi host.

The error itself on the tasks was not very helpful. Just that was not possible to migrate the xpto VM from ESXi host A to ESXi host B. But notice that ESXi had a warning that lost redundancy network.

vMotion failed because duplicated IP

So with a vmnic down need to check where the problem is.

Checking vCenter tasks and events to see if I get more information about this problem. Then I saw a lot of issues in the event, with vmnic4 and vmnic5 (Storage and vMotion interfaces) flapping up and down. But since these Interfaces, it is configured with LACP, that is normal behavior when you have an issue with one of the interfaces.

1. Check events.

Then I notice the main issue for this problem: “A duplicate IP address was detected for 100.10.58.43 on the interface vmk4. The current owner is 00:50:56:62:19:77.”

vMotion failed because duplicated IP

The root cause vMotion failed because duplicated IP, that is why some of the vMotion migrations are failing, and others are finishing with success.

Next, I need to check DNS to see if anyone made any mistake to delete/add an entry that was already in use. Found no issues in the DNS, vMotion vmkernel IP is set in the DNS for this ESXi host.

2. Check logs.

Next, I check the logs from VOB to check if I can get more information in /var/log.

The information in the VOB log is similar to what I have seen in the events, so I need to go to the next check.

3. Check the arp table.

So need to find where this mac address is. Checking arp table using the command “esxcli network ip neighbor list” you can check mac addresses, but you need to check each host one by one.

This is an example:

Checking a couple of ESXi hosts did not find anything about that duplicated IP and mac address. If you have dozens or hundreds of hosts, this is enormous work, so why not using PowerShell and search per vCenter.

4. Using PowerShell.

Open PowerShell connect to your vCenter and run the following command: Get-VMHost | Get-VMHostNetworkAdapter | Where-Object {$_.Mac -eq “00:50:56:62:19:77”} | Select VMHost, PortGroupName, DeviceName, Mac

Running the command in vCenter still nothing. So the problem must be in another vCenter. Run the PowerShell in all vCenters that we have in our infrastructure and in one vCenter… voila!

Found the vCenter vs ESXi host and the vmkernel that was using the same IP address for vMotion. It is strange since both ESXi hosts are in production for a couple of years now, and this never happens.

Checking the DNS entries again, this ESXi host for vMotion IP address have 100.10.58.12. So I don’t know who did make this mistake. So someone must have to make this change manually and add this used IP address on this ESXi host. Who did, is something for the next step, for now, I need to fix this asap.

Finally, to prevent this happen and I am not aware, I create a new alarm rule for duplicate IP addresses for all vCenters in the infrastructure. I created a post just for this, you can check here, How to create a vCenter Alarm for Duplicated IPs.

I hope this post will help you to troubleshoot and identify where is the duplicated IP.

Note: Share this article if you think it is worth sharing.

©2020 ProVirtualzone. All Rights Reserved

 

 

 

By | 2020-02-08T13:14:01+01:00 February 8th, 2020|Virtualization|0 Comments

About the Author:

I am over 20 years’ experience in the IT industry. Working with Virtualization for more than 10 years (mainly VMware). I am an MCP, VCP6.5-DCV, VMware vSAN Specialist, Veeam Vanguard 2018/2019, vExpert vSAN 2018/2019 and vExpert for the last 4 years. Specialties are Virtualization, Storage, and Virtual Backups. I am working for Elits a Swedish consulting company and allocated to a Swedish multinational networking and telecommunications company as a Teach Lead and acting as a Senior ICT Infrastructure Engineer. I am a blogger and owner of the blog ProVirtualzone.com

Leave a Reply

%d bloggers like this: