{"id":3394,"date":"2017-04-15T15:29:49","date_gmt":"2017-04-15T11:29:49","guid":{"rendered":"https:\/\/rakhesh.com\/?p=3394"},"modified":"2017-04-15T15:32:36","modified_gmt":"2017-04-15T11:32:36","slug":"nsx-firewall-no-working-on-layer3-openbsd-vmware-tools-ip-discovery-etc","status":"publish","type":"post","link":"https:\/\/rakhesh.com\/virtualization\/nsx-firewall-no-working-on-layer3-openbsd-vmware-tools-ip-discovery-etc\/","title":{"rendered":"NSX Firewall no working on Layer3; OpenBSD VMware Tools; IP Discovery, etc."},"content":{"rendered":"

I have two security groups. Network 1 VMs (a group that contains my VMs in the 192.168.1.0\/24) and Network 2 VMs (similar, for 192.168.2.0\/24 network).\u00a0<\/p>\n

\"\"\"\"<\/p>\n

Both are dynamic groups. I select members based on whether the VM name contains -n1 or -n2. (The whole exercise is just for fun\/ getting to know this stuff).\u00a0<\/p>\n

I have two firewall rules making use of these rules. Layer 2 and Layer 3.\u00a0<\/p>\n

\"\" \"\"<\/p>\n

The Layer 2 rule works but the Layer 3 one does not! Weird.\u00a0<\/p>\n

I decided to troubleshoot this via the command line. Figured it would be a good opportunity.<\/p>\n

To troubleshoot I have to check the rules on the hosts (because remember, that’s where the firewall is; it’s a kernel module in each host). For that I need to get the host-id. For which I need to get the cluster-id. Sadly there’s no command to list all hosts (or at least I don’t know of any).\u00a0<\/p>\n

# get a list of clusters\r\nnsxmgr63-01> show dfw cluster all\r\nNo.  Cluster Name   Cluster Id               Datacenter Name   Firewall Status\r\n1    Cluster One    domain-c15               DC01              Enabled\r\n2    Cluster Two    domain-c17               DC01              Enabled\r\n3    Mgt Cluster    domain-c7                DC01              Enabled\r\n\r\n# check each cluster to get the host-ids\r\nnsxmgr63-01> show cluster domain-c15\r\nDatacenter: DC01\r\nCluster: Cluster One\r\nNo.  Host Name            Host Id                  Installation Status\r\n1    esx65-11.my.domain   host-19                  Enabled\r\n\r\nnsxmgr63-01> show cluster domain-c17\r\nDatacenter: DC01\r\nCluster: Cluster Two\r\nNo.  Host Name            Host Id                  Installation Status\r\n1    esx65-21.my.domain   host-21                  Enabled\r\n<\/pre>\n
So now I have my host-ids.<\/p>\n
Let’s also take a look the my VMs (thankfully it’s a short list! I wonder how admins do this in real life):<\/p>\n
nsxmgr63-01> show vm vm-40\r\nDatacenter: DC01\r\nCluster: Cluster One\r\nHost: esx65-11.my.domain\r\nHost-ID: host-19\r\nVM: obsd-n2h11_1\r\nVirtual Nics List:\r\n1.\r\nVnic Name      obsd-n2h11_1 - Network adapter 1\r\nVnic Id        50364847-a9d1-3326-a41a-5c62706edc9e.000\r\nFilters        nic-110513-eth0-vmware-sfw.2 <---- (host-19)\r\n\r\nnsxmgr63-01> show vm vm-35\r\nDatacenter: DC01\r\nCluster: Cluster One\r\nHost: esx65-11.my.domain\r\nHost-ID: host-19\r\nVM: obsd-n1h11_1\r\nVirtual Nics List:\r\n1.\r\nVnic Name      obsd-n1h11_1 - Network adapter 1\r\nVnic Id        50369375-2fc4-1f85-b85e-f15613751fbf.000\r\nFilters        nic-111445-eth0-vmware-sfw.2 <---- (host-19)\r\n\r\nnsxmgr63-01> show vm vm-64\r\nDatacenter: DC01\r\nCluster: Cluster Two\r\nHost: esx65-21.my.domain\r\nHost-ID: host-21\r\nVM: obsd-n3h21_1\r\nVirtual Nics List:\r\n1.\r\nVnic Name      obsd-n3h21_1 - Network adapter 1\r\nVnic Id        503698c0-363b-2c5d-2c73-6182766cd6f1.000\r\nFilters        nic-464408-eth0-vmware-sfw.2 <---- (host-21)\r\n\r\nnsxmgr63-01> show vm vm-39\r\nDatacenter: DC01\r\nCluster: Cluster Two\r\nHost: esx65-21.my.domain\r\nHost-ID: host-21\r\nVM: obsd-n1h21_1\r\nVirtual Nics List:\r\n1.\r\nVnic Name      obsd-n1h21_1 - Network adapter 1\r\nVnic Id        50365bd9-f536-095e-eeef-d9f6d81f733b.000\r\nFilters        nic-464142-eth0-vmware-sfw.2 <---- (host-21)<\/pre>\n
We can see the filters applying to each VM. \u00a0To summarize:<\/p>\n
nic-110513-eth0-vmware-sfw.2 | obsd-n2h11_1 | host-19 (esx65-11)\r\nnic-111445-eth0-vmware-sfw.2 | obsd-n1h11_1 | host-19 (esx65-11)\r\nnic-464408-eth0-vmware-sfw.2 | obsd-n1h11_1 | host-21 (esx65-21)\r\nnic-464142-eth0-vmware-sfw.2 | obsd-n3h21_1 | host-21 (esx65-21)<\/pre>\n
And are these filters applying on the hosts themselves?<\/p>\n
# checking host-19\r\nnsxmgr63-01> show dfw host host-19 summarize-dvfilter\r\n\r\n<snip>\r\n\r\nFilters:\r\nworld 0 <no world>\r\n port 50331650 vmk0\r\n  vNic slot 0\r\n  name: nic-0-eth4294967295-ESXi-Firewall.0\r\n agentName: ESXi-Firewall\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failOpen\r\n   slowPathID: none\r\n   filter source: Invalid\r\n port 50331663 vmk1\r\n  vNic slot 0\r\n  name: nic-0-eth4294967295-ESXi-Firewall.0\r\n agentName: ESXi-Firewall\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failOpen\r\n   slowPathID: none\r\n   filter source: Invalid\r\n\r\nworld 110513 vmm0:obsd-21 vcUuid:'50 36 48 47 a9 d1 33 26-a4 1a 5c 62 70 6e dc 9e'\r\n port 50331666 obsd-21.eth0\r\n  vNic slot 2\r\n  name: nic-110513-eth0-vmware-sfw.2 <----- applied!\r\n agentName: vmware-sfw\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failClosed\r\n   slowPathID: none\r\n   filter source: Dynamic Filter Creation\r\n  vNic slot 1\r\n  name: nic-110513-eth0-dvfilter-generic-vmware-swsec.1\r\n agentName: dvfilter-generic-vmware-swsec\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failClosed\r\n   slowPathID: none\r\n   filter source: Alternate Opaque Channel\r\n\r\nworld 111445 vmm0:obsd-11 vcUuid:'50 36 93 75 2f c4 1f 85-b8 5e f1 56 13 75 1f bf'\r\n port 50331668 obsd-11.eth0\r\n  vNic slot 2\r\n  name: nic-111445-eth0-vmware-sfw.2 <----- applied!\r\n agentName: vmware-sfw\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failClosed\r\n   slowPathID: none\r\n   filter source: Dynamic Filter Creation\r\n  vNic slot 1\r\n  name: nic-111445-eth0-dvfilter-generic-vmware-swsec.1\r\n agentName: dvfilter-generic-vmware-swsec\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failClosed\r\n   slowPathID: none\r\n   filter source: Alternate Opaque Channel\r\n\r\n# checking host-21\r\nnsxmgr63-01> show dfw host host-21 summarize-dvfilter\r\n\r\n<snip>\r\n\r\nFilters:\r\nworld 0 <no world>\r\n port 50331650 vmk0\r\n  vNic slot 0\r\n  name: nic-0-eth4294967295-ESXi-Firewall.0\r\n agentName: ESXi-Firewall\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failOpen\r\n   slowPathID: none\r\n   filter source: Invalid\r\n port 50331654 vmk1\r\n  vNic slot 0\r\n  name: nic-0-eth4294967295-ESXi-Firewall.0\r\n agentName: ESXi-Firewall\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failOpen\r\n   slowPathID: none\r\n   filter source: Invalid\r\n\r\nworld 464142 vmm0:obsd-n1h21_1 vcUuid:'50 36 5b d9 f5 36 09 5e-ee ef d9 f6 d8 1f 73 3b'\r\n port 50331656 obsd-n1h21_1.eth0\r\n  vNic slot 2\r\n  name: nic-464142-eth0-vmware-sfw.2 <----- applied!\r\n agentName: vmware-sfw\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failClosed\r\n   slowPathID: none\r\n   filter source: Dynamic Filter Creation\r\n  vNic slot 1\r\n  name: nic-464142-eth0-dvfilter-generic-vmware-swsec.1\r\n agentName: dvfilter-generic-vmware-swsec\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failClosed\r\n   slowPathID: none\r\n   filter source: Alternate Opaque Channel\r\n\r\nworld 464408 vmm0:obsd-n3h21_1 vcUuid:'50 36 98 c0 36 3b 2c 5d-2c 73 61 82 76 6c d6 f1'\r\n port 50331657 obsd-n3h21_1.eth0\r\n  vNic slot 2\r\n  name: nic-464408-eth0-vmware-sfw.2 <----- applied!\r\n agentName: vmware-sfw\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failClosed\r\n   slowPathID: none\r\n   filter source: Dynamic Filter Creation\r\n  vNic slot 1\r\n  name: nic-464408-eth0-dvfilter-generic-vmware-swsec.1\r\n agentName: dvfilter-generic-vmware-swsec\r\n   state: IOChain Attached\r\n   vmState: Detached\r\n   failurePolicy: failClosed\r\n   slowPathID: none\r\n   filter source: Alternate Opaque Channel<\/pre>\n
Hmm, that too looks fine.\u00a0<\/p>\n
Next I picked up one of the rule sets and explored it further:<\/p>\n
nsxmgr63-01> show dfw host host-19 filter nic-111445-eth0-vmware-sfw.2 rules\r\nruleset domain-c15 {\r\n  # Filter rules\r\n  rule 1003 at 1 inout protocol ipv6-icmp icmptype 135 from any to any accept;\r\n  rule 1003 at 2 inout protocol ipv6-icmp icmptype 136 from any to any accept;\r\n  rule 1002 at 3 inout protocol udp from any to any port 68 accept;\r\n  rule 1002 at 4 inout protocol udp from any to any port 67 accept;\r\n  rule 1010 at 5 inout protocol any from addrset src1010 to addrset src1010 drop; <------ L3\r\n  rule 1001 at 6 inout protocol any from any to any accept;\r\n}\r\n\r\nruleset domain-c15_L2 {\r\n  # Filter rules\r\n  rule 1009 at 1 inout ethertype any from addrset src1009 to addrset src1009 drop;  <------ L2\r\n  # internal # rule 1009 at 2 in ethertype any from addrset src1009 to addrset src1009 drop;\r\n  rule 1004 at 3 inout ethertype any from any to any accept;\r\n}\r\n<\/pre>\n
The Layer 3 & Layer 2 rules are in separate rule sets. I have marked the ones\u00a0which I am interested in. One works, the other doesn’t. So I checked the address sets used by both:<\/p>\n
nsxmgr63-01> show dfw host host-19 filter nic-111445-eth0-vmware-sfw.2 addrsets\r\naddrset src1009 {\r\nmac 00:50:56:b6:25:a4,\r\nmac 00:50:56:b6:2a:e2,\r\nmac 00:50:56:b6:ad:b3,\r\nmac 00:50:56:b6:f2:9a,\r\n}\r\naddrset src1010 {\r\n}\r\n<\/pre>\n
Tada! And there we have the problem. The address set for the Layer 3 rule is empty.\u00a0<\/p>\n
I checked this for the other rules too – same situation. I modified my Layer 3 rule to specifically target the subnets:<\/p>\n
\"\"<\/p>\n
And the address set for that rule is not empty:<\/p>\n
nsxmgr63-01> show dfw host host-19 filter nic-111445-eth0-vmware-sfw.2 addrsets\r\naddrset rdst1012 {\r\nip 192.168.1.0\/24,\r\nip 192.168.2.0\/24,\r\n}\r\naddrset rsrc1012 {\r\nip 192.168.1.0\/24,\r\nip 192.168.2.0\/24,\r\n}\r\n\r\n<snip><\/pre>\n
And because of this the firewall rules do work as expected. Hmm.<\/p>\n
I modified this rule to be a group with my OpenBSD VMs from each network explicitly added to it (i.e. not dynamic membership in case that was causing an issue). But nope, same result – empty address set!<\/p>\n
\"\"<\/p>\n
But the address set is now empty. :o)<\/p>\n
nsxmgr63-01> show dfw host host-19 filter nic-111445-eth0-vmware-sfw.2 addrsets\r\n<snip>\r\n\r\naddrset src1012 {\r\n}\r\n<\/pre>\n
So now I have an idea of the problem. I am not too surprised by this because I vaguely remember reading something about VMware Tools and IP detection inside a VM (i.e. NSX makes use of VMware Tools to know the IP address of a VM) and also because I am aware OpenBSD does not use the official VMware Tools package (it has its own<\/a> and that only provides a subset of functions).<\/p>\n
Googling a bit on this topic I came across the IP address Discovery<\/a> section in the NSX Admin guide – prior to NSX 6.2 if VMware Tools wasn’t installed (or was stopped) NSX won’t be able to detect the IP address of the VM. Post NSX 6.2 it can do DHCP & ARP snooping to work around a missing\/ stopped VMware Tools. We configure the latter in the host installation page:<\/p>\n
\"\"<\/p>\n
I am going to go ahead and enable both on all my clusters.\u00a0<\/p>\n
\"\"<\/p>\n
That helped. But it needs time. Initially the address set was empty. I started pings from one VM to another and the source VM IP was discovered and put in the address set; but since the destination VM wasn’t in the list traffic was still being allowed. I stopped pings, started pings, waited a while … tried again … and by then the second VM IP to was discovered and put in the address set – effectively blocking communication between them.\u00a0<\/p>\n
addrset src1012 {\r\nip 192.168.1.1,\r\nip 192.168.2.1,\r\n}<\/pre>\n
Side by side I installed a Windows 8.1 VM with VMware Tools etc and tested to see if it was being automatically picked up (I did this before enabling the snooping above). It was. In fact its IPv6 address too was discovered via VMware Tools and added to the list:<\/p>\n
addrset src1010 {\r\nip 192.168.1.1,\r\nip 192.168.1.2,\r\nip 192.168.1.11, <--- Windows 8.1 + VMware Tools\r\nip 192.168.2.1,\r\nip fe80::2570:6577:3d58:f198,  <--- Windows 8.1 + VMware Tools\r\n}\r\n<\/pre>\n
Nice! Picked up something interesting today.\u00a0<\/p>\n","protected":false},"excerpt":{"rendered":"
I have two security groups. Network 1 VMs (a group that contains my VMs in the 192.168.1.0\/24) and Network 2 VMs (similar, for 192.168.2.0\/24 network).\u00a0 Both are dynamic groups. I select members based on whether the VM name contains -n1 or -n2. (The whole exercise is just for fun\/ getting to know this stuff).\u00a0 I … Continue reading NSX Firewall no working on Layer3; OpenBSD VMware Tools; IP Discovery, etc.<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[196,183],"tags":[706,59,660,317],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/3394"}],"collection":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/comments?post=3394"}],"version-history":[{"count":0,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/3394\/revisions"}],"wp:attachment":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/media?parent=3394"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/categories?post=3394"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/tags?post=3394"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}