Spent the better part of two days chasing an issue only to find it was no issue at all. So irritated! Wasn’t a total waste of time as I got to read stuff, but it side tracked me from the main issue.
Here’s my setup. I have a Windows Server 2012R2 physical server. This runs VMware Workstation 12.5. Within it I have XenServer and VMware ESXi (the hypervisor isn’t relevant to the story but I mention it anyways). Within the hypervisor I have a Windows 8.1 VM – well two of them actually, but again it doesn’t matter much to the story.
Within VMware Workstation I have a couple of other servers too – a mix of Windows Server 2012 R2, 2016, and FreeBSD.
Let’s call the VMs within VMware Workstation as “VMs” while the VMs within the nested hypervisors as “VVMs”. The issue was that from the VVMs I was able to ping the VMs and get info from them (e.g. IP addresses) but I couldn’t ping the VVMs from the VMs. It didn’t matter which hypervisor the VVM was on. Also, the VVMs couldn’t ping each other.
There’s a lot of forum and blog posts on theis topic but their issue seems to be different. Their issue is that the VVMs are unable to see the outside world (i.e. the VMs). But my issue was that the VVMs could see the outside world; it was the outside world that couldn’t see them. All the forum and blog posts pointed to it being a case of the virtual switch not allowing promiscuous mode or forged MAC addresses, and the fix was to enable these. In my case I couldn’t find any such setting on VMware Workstation so I began suspecting it as the culprit.
Some good links I found while reading on these; putting them here as info for myself:
- Understanding XenServer Networking – The Linux Perspective
- Best Practices for using Open vSwitch
- Why is Promiscuous Mode & Forged Transmits required for Nested ESXi?
- How The VMware Forged Transmits Security Policy Works
- New VMware Fling to improve Network/CPU performance when using Promiscuous Mode for Nested ESX
Oh, and if you are on a Linux host (where VMware Workstation is running) then you need to do some extra stuff to enable Promiscuous mode.
Nowhere could I find anything on what to do for VMware Workstation running on Windows and whether it had promiscuous mode enabled or not.
Finally I resorted to using
tcpdump (on XenServer)/
tcpdump-uw (on ESXi) to see if the nested hypervisor is receiving the ICMP packets – it was. The ARP requests had the correct MAC addresses too. Next I installed Wireshark on a VM and VVM to see what was happening, and I could see that the VVM was receiving packets but not replying. So the switch in VMware Workstation was definitely in promiscous mode – the problem was in the VVM. I didn’t suspect a VVM firewall at all as I had disabled the Windows firewall service; but just for the heck of it I enabled the firewall service and simply turned off the firewall. And what do you know – suddenly the VVM is responding to ICMP packets!!
I have no idea why this is so. I had always thought disabling the firewall service is enough to … well, disable the firewall. But looks like actually disabling the firewall for each of the network profiles is the important thing. Weird.
Anyways – after two days of scratching my head I now have connectivity from my VMs to VVMs.