{"id":3361,"date":"2017-04-08T23:56:28","date_gmt":"2017-04-08T19:56:28","guid":{"rendered":"https:\/\/rakhesh.com\/?p=3361"},"modified":"2017-04-11T09:55:01","modified_gmt":"2017-04-11T05:55:01","slug":"notes-to-self-while-installing-nsx-6-3-part-4","status":"publish","type":"post","link":"https:\/\/rakhesh.com\/virtualization\/notes-to-self-while-installing-nsx-6-3-part-4\/","title":{"rendered":"Notes to self while installing NSX 6.3 (part 4)"},"content":{"rendered":"<p>Reading through the <a href=\"http:\/\/pubs.vmware.com\/nsx-63\/topic\/com.vmware.ICbase\/PDF\/nsx_63_install.pdf\">VMware NSX 6.3 Install Guide<\/a> after having installed the DLR and ESG in my home lab. Continuing from the DLR section.<\/p>\n<p>As I had <a href=\"https:\/\/rakhesh.com\/virtualization\/notes-to-self-while-installing-nsx-6-3-part-1\/\">mentioned earlier<\/a> NSX provides routing via DLR or ESG. \u00a0<\/p>\n<ul>\n<li>DLR == Distributed Logical Router.<\/li>\n<li>ESG == Edge Services Gateway<\/li>\n<\/ul>\n<p>DLR consists of an appliance that provides the control plane functionality. This appliance does not do any routing itself. The actual routing is done by the VIBs on the ESXi hosts. The appliance uses the NSX Controller to push out updates to the ESXi host. (<strong>Note<\/strong>: Only DLR. ESG does not depend on the Controller to push out route). Couple of points to keep in mind:<\/p>\n<ul>\n<li>A DLR instance cannot connect to logical switches in different transport zones.\u00a0<\/li>\n<li>A DLR cannot connect to a dvPortgroup with VLAN ID 0.<\/li>\n<li>A DLR cannot connect to a dvPortgroup with VLAN ID if that DLR also connects to logical switches spanning more than one VDS.\u00a0\n<ul>\n<li><em>This confused me. Why would a logical switch span more than one VDS? I dunno. There are reasons probably, same way you could have multiple clusters in same data center having different VDSes instead of using the same one.\u00a0<\/em><\/li>\n<\/ul>\n<\/li>\n<li>If you have portgroups on different VDSes with the same VLAN ID, and these VDSes share some hosts, then DLR cannot connect these.\u00a0<\/li>\n<\/ul>\n<p>I am not entirely clear with the above points. It&#8217;s more to enforce the transport zones and logical switches align correctly, but I haven&#8217;t entirely understood it so I am simply going to make note as above and move on &#8230;<\/p>\n<p>In a DLR the firewall rules only apply to the uplink interface and are limited to traffic destined for the edge virtual appliance. In other words they don&#8217;t apply to traffic between the logical switches a DLR instance connects. (Note that this refers to the firwall settings found under the DLR section, not in the Firewall section of NSX).\u00a0<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-3366 aligncenter\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/Firewall-Settings-DLR.png?resize=660%2C362&#038;ssl=1\" alt=\"\" width=\"660\" height=\"362\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/Firewall-Settings-DLR.png?w=1055&amp;ssl=1 1055w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/Firewall-Settings-DLR.png?resize=300%2C165&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/Firewall-Settings-DLR.png?resize=768%2C421&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/Firewall-Settings-DLR.png?resize=1024%2C562&amp;ssl=1 1024w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/Firewall-Settings-DLR.png?resize=150%2C82&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/Firewall-Settings-DLR.png?resize=400%2C220&amp;ssl=1 400w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>A DLR has many interfaces. The one exposed to VMs for routing is the Logical InterFace (LIF). Here&#8217;s a screenshot from the interfaces on my DLR.\u00a0<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-3367 aligncenter\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIFs.png?resize=660%2C146&#038;ssl=1\" alt=\"\" width=\"660\" height=\"146\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIFs.png?w=921&amp;ssl=1 921w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIFs.png?resize=300%2C66&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIFs.png?resize=768%2C170&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIFs.png?resize=150%2C33&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIFs.png?resize=400%2C89&amp;ssl=1 400w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>The ones of type &#8216;Internal&#8217; are the LIFs. These are the interfaces that the DLR will route between. Each LIF connects to a separate network &#8211; in my case a logical switch each. The IP address assigned to this LIF will be the address you set as gateway for the devices in that network. So for example: one of the LIFs has an IP address 192.168.1.253 and connects to my 192.168.1.0\/24 segment. All the VMs there will have 192.168.1.253 as their default gateway. Suppose we ignore the &#8216;Uplink&#8217; interface for now (it&#8217;s optional, I created it for the external routing to work), and all our DLR had were the two &#8216;Internal&#8217; LIFs, and VMs on each side had the respective IP address set as their default gateway, then our DLR will enable routing between these two networks.\u00a0<\/p>\n<p>Unlike a physical router though, which exists outside the virtual network and which you can point to as &#8220;here&#8217;s my router&#8221;,\u00a0there&#8217;s no such concept with DLRs. The DLR isn&#8217;t a VM which you can point to as your router. Nor is it a VM to which packets between these networks (logical switches) are sent to for routing. The DLR, as mentioned above, is simply your ESXi hosts. Each ESXi host that has logical switches which a DLR connects into has this LIF created in them with that LIF IP address assigned to it and a virtual MAC so VMs can send packets to it. The DLR is your ESXi host. (That is pretty cool, isn&#8217;t it! I shouldn&#8217;t be amazed because I had mentioned it earlier when reading about all this, but it is still cool to actually &#8220;see&#8221; it once I have implemented).<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-3368 aligncenter\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIF-MAC.png?resize=660%2C254&#038;ssl=1\" alt=\"\" width=\"660\" height=\"254\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIF-MAC.png?w=740&amp;ssl=1 740w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIF-MAC.png?resize=300%2C116&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIF-MAC.png?resize=150%2C58&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/LIF-MAC.png?resize=400%2C154&amp;ssl=1 400w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>Above screenshot is from my two VMs on the same VXLAN but on different hosts. Note that the default gateway (192.168.1.253) MAC is the same for both. Each of their hosts will respond to this MAC entry.\u00a0<\/p>\n<p>(<strong>Note to self<\/strong>: Need to explore the <code>net-vdr<\/code> command sometime. Came across it as I was Googling on how to find the MAC address table seen by the LIF on a host. Didn&#8217;t want to get side-tracked so didn&#8217;t explore too much. There&#8217;s something called a VDR (not encountered it yet in my readings).<\/p>\n<ul>\n<li><code>net-vdr -I -l<\/code> will list all the VDRs on a host.<\/li>\n<li><code>net-vdr -L -l &lt;vdrname&gt;<\/code> will list the LIFs.<\/li>\n<li><code>net-vdr -N -l &lt;vdrname&gt;<\/code> will list the MAC addresses (ARP info)<\/li>\n<\/ul>\n<p>)<\/p>\n<p>When creating a DLR it is possible to create it with or without the appliance. Remember that the appliance provides the control plane functionality. It is the appliance that learns of new routes etc and pushes to the DLR modules in the ESXi hosts. Without an appliance the DLR modules will do static routing (which might be more than enough, especially in a test environment like my nested lab for instance) so it is ok to skip it if your requirements are such. Adding an appliance means you get to (a) select if it is deployed in HA config (i.e. two appliance), (b) their locations etc, (c) IP address and such for the appliance, as well as enabling SSH. The appliance is connected to a different interface for HA and SSH &#8211; this is independent of the LIFs or Uplink interfaces. That interface isn&#8217;t used for any routing.\u00a0<\/p>\n<p>Apart from the control plane, the appliance also controls the firewall on the DLR. If there&#8217;s no appliance you can&#8217;t make any firewall changes to the DLR &#8211; makes sense coz there&#8217;s nothing to change. You won&#8217;t be connecting to the DLR for SSH or anything coz you do that to the appliance on the HA interface.\u00a0<\/p>\n<p>According to <a href=\"http:\/\/pubs.vmware.com\/nsx-63\/index.jsp#com.vmware.nsx.install.doc\/GUID-0B3BD895-8037-48A8-831C-8A8986C3CA42.html\">the docs<\/a> you can&#8217;t add an appliance once a DLR instance is deployed. Not sure about that as I do see an option to deploy an appliance on my non-appliance DLR instance. Maybe it will fail when I actually try and create the appliance &#8211; I didn&#8217;t bother trying.\u00a0<\/p>\n<p>Discovered <a href=\"http:\/\/www.routetocloud.com\/2014\/06\/nsx-distributed-logical-router\/\">this blog post<\/a> while Googling for something. I&#8217;ve encountered &amp; linked to his posts previously too. He has a lot of screenshots and step by step instructions. So worth a check out if you want to see some screenshots and much better explanation than me. :) Came across some commands from his blog which can be run on the NSX Controller to see the DLRs it is aware of and their interfaces. Pasting the output from my lab here for now, I will have to explore this later &#8230;<\/p>\n<pre class=\"lang:ps decode:true \">nsx-controller # show control-cluster logical-routers instance all\r\nLR-Id      LR-Name                                            Universal Service-Controller Egress-Locale                        In-Sync    Sync-Category\r\n0x1389     default+edge-3                                     false     10.10.1.51         local                                Yes        NORMAL\r\n0x1388     default+edge-1                                     false     10.10.1.51         local                                Yes        NORMAL\r\nnsx-controller # show control-cluster logical-routers interface-summary 0x1389\r\nInterface                        Type   Id                       IP[]\r\n13890000000a                     vxlan  5003(0x138b)             192.168.3.253\/24\r\n13890000000b                     vxlan  5004(0x138c)             192.168.4.253\/24\r\n138900000002                     vxlan  5005(0x138d)             192.168.0.249\/30\r\nnsx-controller # show control-cluster logical-routers interface-summary 0x1388\r\nInterface                        Type   Id                       IP[]\r\n13880000000b                     vxlan  5002(0x138a)             192.168.2.253\/24\r\n13880000000a                     vxlan  5001(0x1389)             192.168.1.253\/24\r\n138800000002                     vxlan  5000(0x1388)             192.168.0.253\/30\r\n<\/pre>\n<p>I have two DLRs. One has an appliance, other doesn&#8217;t. I made these two, and a bunch of logical switches to hook these to, to see if there&#8217;s any difference in functionality or options.<\/p>\n<p>One thing I realized as part of this exercise is that a particular logical switch can only connect to one DLR. Initially I had one DLR which connected to 192.168.1.0\/24 and 192.168.2.0\/24. Its uplink was on logical switch 192.168.0.0\/24 which is where the ESG too hooked into. Later when I made one more DLR with its own internal links and tried to connect its uplink to the 192.168.0.0\/24 network used by the previous DLR, I saw that it didn&#8217;t even appear in the list of options. That&#8217;s when I realized its better to use a smaller range logical switch for the uplinks &#8211; like say a \/30 network. This way each DLR instance connects to an ESG on its own \/30 network logical switch (as in the output above).\u00a0<\/p>\n<p>A DLR can have up to 8 uplink interfaces and 1000 internal interfaces.<\/p>\n<hr \/>\n<p>Moving on to ESG. This is a virtual appliance. While a DLR provides East-West routing (i.e. within the virtual environment), an ESG provides North-South routing (i.e. out of the virtual environment). The ESG also provides services such as DHCP, NAT, VPN, and Load Balancing. (Note to self: DLR does not provide DHCP or Load Balancing as one\u00a0<em>might<\/em> expect (at least I did! :p). DLR provides DHCP Relay though).\u00a0<\/p>\n<p>The uplink of an ESG will be a VDS (Distributed Switch) as that&#8217;s what eventually connects an ESXi environment to the physical network.\u00a0<\/p>\n<p>An ESG\u00a0<em>needs<\/em> an appliance to be deployed. You can enable\/ disable SSH into this appliance. If enabled you can SSH into the ESG appliance from the uplink address or from any of the internal link IP addresses. In contrast, you can only SSH into a DLR instance if it has an associated appliance. Even then, you cannot SSH into the appliance from the internal LIFs (coz these don&#8217;t really exist, remember &#8230; they are on each ESXi host). With a DLR we have to SSH into the interface used for HA (this can be used even if there&#8217;s only one appliance and hence no HA).\u00a0<\/p>\n<p>When deploying an ESG appliance HA can be enabled. This deploys two appliances in an active\/passive mode (and the two appliances will be on separate hosts). These two appliances will talk to each other to keep in sync via one of the internal interfaces (we can specify one, or NSX will just choose any). On this internal interface the appliances will have a link local IP address (a \/30 subnet from 169.254.0.0\/16) and communicate over that (doesn&#8217;t matter that there&#8217;s some other IP range actually used in that segment, as these are link local addresses and unlikely anyone&#8217;s going to actually use them). In contrast, if a DLR appliance is deployed with HA we need to specify a <em>separate<\/em> network from the networks that it be routing between. This can be a logical switch or a DVS, and as with ESG the two appliances will have link local IP addresses (a \/30 subnet from 169.254.0.0\/16) for communication. Optionally, we can specify an IP address in this network via which we can SSH into the DLR appliance (this IP address will not be used for HA, however).<\/p>\n<p>After setting up all this, I also created two NAT rules just for kicks.\u00a0<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-3377 aligncenter\" src=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/NAT.png?resize=660%2C83&#038;ssl=1\" alt=\"\" width=\"660\" height=\"83\" srcset=\"https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/NAT.png?w=1148&amp;ssl=1 1148w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/NAT.png?resize=300%2C38&amp;ssl=1 300w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/NAT.png?resize=768%2C96&amp;ssl=1 768w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/NAT.png?resize=1024%2C128&amp;ssl=1 1024w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/NAT.png?resize=150%2C19&amp;ssl=1 150w, https:\/\/i0.wp.com\/rakhesh.com\/wp-content\/uploads\/2017\/04\/NAT.png?resize=400%2C50&amp;ssl=1 400w\" sizes=\"(max-width: 660px) 100vw, 660px\" data-recalc-dims=\"1\" \/><\/p>\n<p>And with that my basic setup of NSX is complete! (I skipped OSPF as I don&#8217;t think I will be using it any time soon in my immediate line of work; and if I ever need to I can come back to it later). Next I need to explore firewalls (micro-segmentation) and possibly load balancing etc &#8230; and generally fiddle around with this stuff. I&#8217;ve also got to start figuring out the troubleshooting and command-line stuff. But the base is done &#8211; I hope!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Reading through the VMware NSX 6.3 Install Guide after having installed the DLR and ESG in my home lab. Continuing from the DLR section. As I had mentioned earlier NSX provides routing via DLR or ESG. \u00a0 DLR == Distributed Logical Router. ESG == Edge Services Gateway DLR consists of an appliance that provides the &hellip; <a href=\"https:\/\/rakhesh.com\/virtualization\/notes-to-self-while-installing-nsx-6-3-part-4\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Notes to self while installing NSX 6.3 (part 4)<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[196,183],"tags":[686,700,660,184],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/3361"}],"collection":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/comments?post=3361"}],"version-history":[{"count":0,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/3361\/revisions"}],"wp:attachment":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/media?parent=3361"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/categories?post=3361"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/tags?post=3361"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}