{"id":2769,"date":"2015-08-24T10:14:45","date_gmt":"2015-08-24T06:14:45","guid":{"rendered":"https:\/\/rakhesh.com\/?p=2769"},"modified":"2015-08-29T11:11:34","modified_gmt":"2015-08-29T07:11:34","slug":"vmotion-is-using-the-management-network-and-failing","status":"publish","type":"post","link":"https:\/\/rakhesh.com\/infrastructure\/vmotion-is-using-the-management-network-and-failing\/","title":{"rendered":"vMotion is using the Management Network (and failing)"},"content":{"rendered":"

Was migrating one of our offices to a new IP scheme the other day and vMotion started failing. I had a good idea what the problem could be (coz I encountered something similar a few days ago in another context<\/a>) so here’s a blog post detailing what I did.<\/p>\n

For simplicity let’s say the hosts have two VMkernel NICs – vmk0 and vmk1. vmk0 is connected to the Management Network. vmk1 is for vMotion. Both are on separate VLANs.<\/p>\n

When our Network admins gave out the new IPs they gave IPs from the same range for both functions. That is, for example, vmk0 had an IP 10.20.1.2\/24 (and 10.20.1.3\/24 and 10.20.4\/24 on the other hosts) and vmk1 had an IP of 10.20.12\/24 (and 10.20.1.13\/24 and 10.20.1.14\/24 on the other hosts).<\/p>\n

Since both interfaces are on separate VLANs (basically separate LANs) the above setup won’t work. That’s because as far as the hosts are concerned both interfaces are on the same network yet physically they are on separate networks. Here’s the routing table on the hosts:<\/p>\n

~ # esxcli network ip route ipv4 list\r\nNetwork      Netmask        Gateway      Interface  Source\r\n-----------  -------------  -----------  ---------  ------\r\ndefault      0.0.0.0        10.20.1.254  vmk0       MANUAL\r\n10.20.1.0    255.255.255.0  0.0.0.0      vmk0       MANUAL<\/pre>\n
Notice that any<\/em> traffic to the 10.20.1.0\/24 network goes via vmk0. And that includes the vMotion traffic because that too is in the same network! And since the network that vmk0 is on is physically a separate network (because it is a VLAN) this traffic will never reach the vMotion interfaces of the other hosts because they don’t know of it.<\/p>\n
So even though you have specific vmk1 as your vMotion traffic NIC, it never gets used because of the default routes.<\/p>\n
If you could force the outgoing traffic to specifically use vmk1 it will work. Below are the results of vmkping<\/code> using the default route vs explicitly using vmk1:<\/p>\n
~ # vmkping 10.20.1.13\r\nPING 10.20.1.13 (10.20.1.13): 56 data bytes\r\n\r\n--- 10.20.1.13 ping statistics ---\r\n2 packets transmitted, 0 packets received, 100% packet loss\r\n~ # vmkping -I vmk1 10.20.1.13\r\nPING 10.20.1.13 (10.20.1.13): 56 data bytes\r\n64 bytes from 10.20.1.13: icmp_seq=0 ttl=64 time=0.433 ms\r\n64 bytes from 10.20.1.13: icmp_seq=1 ttl=64 time=0.198 ms\r\n\r\n--- 10.20.1.13 ping statistics ---\r\n2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min\/avg\/max = 0.198\/0.316\/0.433 ms<\/pre>\n
The solution here is to either remove the VLANs and continue with the existing IP scheme, or to keep using VLANs but assign a different IP network for the vMotion interfaces.<\/p>\n
Update<\/strong>: Came across the following from this blog post<\/a> while searching for something else:<\/p>\n
\n
If the management network (actually the first VMkernel NIC) and the vMotion network share the same subnet (same IP-range) vMotion sends traffic across the network attached to first VMkernel NIC. It does not matter if you create a vMotion network on a different standard switch or distributed switch or assign different NICs to it, vMotion will default to the first VMkernel NIC if same IP-range\/subnet is detected.<\/p>\n
Please be aware that this behavior is only applicable to traffic that is sent by the source host. The destination host receives incoming vMotion traffic on the vMotion network!<\/p>\n<\/blockquote>\n
That answered another question I had but didn’t blog about in my post above. You see, my network admins had also set the iSCSI networks to be in the same subnet as the management network – but separate VLANs – yet the iSCSI traffic was correctly flowing over that VLAN instead of defaulting to the management VMkernel NIC. Now I understand why! It’s only vMotion that defaults to the first<\/em> VMkernel NIC in the same IP range\/ subnet as vMotion.\u00a0<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
Was migrating one of our offices to a new IP scheme the other day and vMotion started failing. I had a good idea what the problem could be (coz I encountered something similar a few days ago in another context) so here’s a blog post detailing what I did. For simplicity let’s say the hosts … Continue reading vMotion is using the Management Network (and failing)<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[333],"tags":[389,529,583,184],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/2769"}],"collection":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/comments?post=2769"}],"version-history":[{"count":0,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/2769\/revisions"}],"wp:attachment":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/media?parent=2769"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/categories?post=2769"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/tags?post=2769"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}