 © Rakhesh Sasidharan
|
Posted: August 28th, 2019 | Tags: load balancing, nsx | Category: Exchange, Virtualization | § This is going to be a quick and dirty post with not much details. Sorry. Spent some time figuring this out today and I wanted to put it here as a reference for anyone else.
At work we needed to setup two Exchange 2016 servers behind an NSX Edge load balancer. We wanted to capture the source IP too so this meant we had to use the load balancer in transparent mode. A colleague had set it up already but the Exchange servers weren’t seeing the source IP so I took a look to see what was missing. I had to make two changes primarily to get it working.
First: the Edge had two interfaces assigned to it. One for HA, another that connected to the backend servers. The VIP for the Edge was also in the second subnet. (This needn’t always be the case. I think a usual scenario is for the VIP to be on a different subnet so YMMV). For the Edge to pass on the source IP to the Exchange servers I knew I had to set it up in a transparent mode. This means the Edge passes on any packets it gets off to the appropriate backend server. It does not change the IP in the packet to be itself, so the backend server see the correct source IP. The problem with this though is that the backend server will then send this packet to its default router and from there to the source IP … which we don’t want. We want the source to only see the Edge VIP always and this means we need the return traffic to come back to the Edge, and the Edge will do a NAT to change the backend server IP to be the VIP.
(This blog post might be a good starting point to read more on the above).
For an Edge to be setup in this fashion we have to set the Edge as the default gateway for the backend servers. (Which is straight-forward – just change the default gateway in the server OS). But for the Edge to then act also as a router for the backend servers we have to tell it that such and such IP is on an internal network and it is ok to do routing on that. In our case the HA network had been set as of type Internal, but the network that was connecting to the backend servers was incorrectly set as Uplink. I changed its type to Internal. This is important as this is what tells the Edge that it can now route any traffic coming to the IP addresses defined on that interface.
This done, I went to the two backend servers and changed their default gateway to be the IP defined on the above Internal interface (vNIC0 in my case). I tested connectivity and also did a trace route to confirm it is going via the Edge. Great!
Second: By default an Edge load balancer is in L7 mode. When in L7 mode the Edge doesn’t forward on the packets it receives. It initiates a new connection to the Exchange servers, so the source IP is itself and defeats what we are trying to do. L7 mode is useful if we are doing any L7 manipulation such as SSL termination, cookie based persistence, URL rewriting, header insertion, etc. We are not doing anything like that here so I changed the mode to L4. This is done by enabling acceleration.
Did the same in the virtual server too (this option is only available after enabling it in the above section).
Lastly, I enabled transparent mode on the pool.
That’s all!
Coming from a NetScaler background I was used to the concept of a failover server. As in a virtual server would have a pool of servers it would load balance amongst and if all of them are down I can define a failover server that could be used. You would define the failover server as a virtual server with no IP, and tell the primary virtual server to failover to this virtual server in case of issues.
Looking around for a similar option with NSX I discovered it’s possible using application rules. Instead of defining two virtual servers though, here you define two pools. One pool for the primary servers you want to load balance, the other pool for the failover server(s).
Then you create an application rule this:
|
|
acl adfs_pri_down nbsrv(pool-adfs-https-443) eq 0 use_backend pool-bkpadfs-https-443 if adfs_pri_down |
Once again, the syntax is that of HAProxy. You define an ACL – adfs_pri_down is what I am defining for my purposes as this is for load balancing some ADFS servers – and the criterion is nbsrv(pool-adfs-https-443) eq 0. The nbsrv criterion checks the pool you pas on to it (pool-adfs-https-443 in my case) and returns the number of servers that are up. So the ACL basically is a boolean one that is true if the number of usable servers is 0.
Next, the use_backend rule switches to using the backup pool I have defined (pool-bkpadfs-https-443 in this case) if the ACL is true.
That’s all. Pretty straightforward!
Not a big deal, but something I tried today mainly as an excuse to try this with NSX.
So here’s the situation. I have a pair of ADFS WAP servers that are load balanced using NSX. ADFS listens on port 443 so that’s the only port my VIP needs to handle.
However, we are also using Verisign DNS to failover between sites. So I want it such that if say both the ADFS servers in Site A are down, then Verisign DNS should failover my external ADFS records to Site B. Setting this up in Verisign DNS is easy but you need to be able to monitor the ADFS / WAP services externally from Verisign for this to work properly. Thus I had to setup my NSX WAP load balancer to listen on port 80 too and forward those to my internal ADFS servers. To monitor the health of ADFS Verisign will periodically query http://myadfsserver/adfs/probe. If that returns a 200 response all is good.
Now here’s the requirement I came up with for myself. I don’t want any and every HTTP query on port 80 to be handled. Yes, if I try http://myadfserver/somerandomurl it gives a 404 error but I don’t want that. I want any HTTP queries to any URL except /adfs/probe to be redirected to /adfs/probe. Figured this would be a good place to use application rules. Here’s what I came up with:
|
|
acl allowed_url url /adfs/probe redirect location /adfs/probe if !allowed_url |
NSX application rules follow the same format as HAProxy so their config guide is a very handy reference.
The acl keyword defines an Access Control List. In my case the name of this ACL is allowed_url (this is a name I can define) and it matches URLs (hence the url keyword). Since I used url it does an exact match, but there are derivatives like url_beg and url_end and url_len and url_dir etc. that I could have used. I can even do regexp matches. In my case I am matching for the exact URL /adfs/probe. I define this as the ACL called allowed_url.
In the next line I use the redirect keyword to redirect any requests to the /adfs/probe location if it does not match the allowed_url ACL. That’s all!
This of course barely touches what one can do with application rules, but I am pleased to get started with this much for now. :)
Posted: February 20th, 2018 | Tags: ADFS, load balancing, nsx | Category: Networks, Virtualization | § Was looking at setting up monitoring of my ADFS servers on NSX.
I know what to monitor on the ADFS and WAP servers thanks to this article.
http://<Web Application Proxy name>/adfs/probe
http://<ADFS server name>/adfs/probe
http://<Web Application Proxy IP address>/adfs/probe
http://<ADFS IP address>/adfs/probe
Need to get an HTTP 200 response for these.
So I created a service monitor in NSX along these lines:
And I associated it with my pool:
Bear in mind the monitor has to check port 80, even though my pool might be on port 443, so be sure to change the monitor port as above.
The “Show Pool Statistics” link on the “Pools” section quickly tells us whether the member servers are up or not:
The show service loadbalancer pool command can be used to see what the issue is in case the monitor appears down. Here’s an example when things aren’t working:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
NSX-edge-15-1> show service loadbalancer pool pool-adfs-https_443 ----------------------------------------------------------------------- Loadbalancer Pool Statistics: POOL pool-adfs-https_443 | LB METHOD round-robin | LB PROTOCOL L7 | Transparent disabled | SESSION (cur, max, total) = (0, 0, 0) | BYTES in = (0), out = (0) +->POOL MEMBER: pool-adfs-https_443/ADFS01, STATUS: DOWN | | HEALTH MONITOR = MONITOR SERVICE, default_adfs_monitor:CRITICAL | | | LAST STATE CHANGE: 2018-02-20 10:02:13 | | | LAST CHECK: 2018-02-20 12:47:15 | | | FAILURE DETAIL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 400 Bad Request | | SESSION (cur, max, total) = (0, 0, 0) | | BYTES in = (0), out = (0) |
Here’s an example when all is well:
|
|
NSX-edge-15-1> show service loadbalancer pool pool-adfs-https_443 ----------------------------------------------------------------------- Loadbalancer Pool Statistics: POOL pool-adfs-https_443 | LB METHOD round-robin | LB PROTOCOL L7 | Transparent disabled | SESSION (cur, max, total) = (0, 0, 0) | BYTES in = (0), out = (0) +->POOL MEMBER: pool-adfs-https_443/ADFS01, STATUS: UP | | HEALTH MONITOR = BUILT-IN, default_adfs_monitor:L7OK | | | LAST STATE CHANGE: 2018-02-20 12:51:02 | | SESSION (cur, max, total) = (0, 0, 0) | | BYTES in = (0), out = (0) |
Thanks to this document for pointing me in the right troubleshooting direction. Quoting from that document, the list of error codes:
UNK: Unknown
INI: Initializing
SOCKERR: Socket error
L4OK: Check passed on layer 4, no upper layers testing enabled
L4TOUT: Layer 1-4 timeout
L4CON: Layer 1-4 connection problem. For example, “Connection refused” (tcp rst) or “No route to host” (icmp)
L6OK: Check passed on layer 6
L6TOUT: Layer 6 (SSL) timeout
L6RSP: Layer 6 invalid response – protocol error. May caused as the:
Backend server only supports “SSLv3” or “TLSv1.0”, or
Certificate of the backend server is invalid, or
The cipher negotiation failed, and so on
L7OK: Check passed on layer 7
L7OKC: Check conditionally passed on layer 7. For example, 404 with disable-on-404
L7TOUT: Layer 7 (HTTP/SMTP) timeout
L7RSP: Layer 7 invalid response – protocol error
L7STS: Layer 7 response error. For example, HTTP 5xx
Nice!
Posted: February 20th, 2018 | Tags: ESG, load balancing, nsx | Category: Networks, Virtualization | § Inline mode == Transparent mode (the latter is the terminology in the UI).
In this mode the load balancer is usually the default gateway for the servers it load balances. Traffic comes to the load balancer, it sends to the appropriate server (after changing the destination IP of the packet – hence DNAT), and replies come to it as it is the default gateway for the server. Note that as far as the destination server is concerned the source IP address is not the load balancer but the client who made the request. Thus the destination server knows who is making the request.
When the load balancer replies to the client who made the request it changes the source IP of the reply from the selected server to its own IP (hence SNAT when replying only).
—
One-Armed mode == Proxy mode
In this mode the load balancer is not the default gateway. The servers it load balance don’t have any changes required to be made to them. The load balancer does a DNAT as before, but also changes the source IP to be itself rather than the client (hence SNAT). When the selected server replies this time, it thinks the source is the load balancer and so replies to it rather than the client. Thus there’s no changes required on the server side. Because of this though, the server doesn’t know who made the request. All requests appear to come from the load balancer (unless you use some headers to capture the info).
As before, when the load balancer replies to the client who made the request it changes the source IP of the reply from the selected server to its own IP (hence SNAT when replying too).
—
You set the inline/ transparent vs. one-armed/ proxy mode per pool.
—
To have load balancing in NSX you need to deploy an ESG (Edge Services Gateway). I don’t know why, but I always associated an ESG with just external routing so it took me by surprise (and still does) when I think I need to deploy an ESG for load balancing, DHCP, and other edge- sort of services (VPN, routing, etc). I guess the point to remember is that it’s not just a gateway – it’s an edge services gateway. :)
Anyways, feel free to deploy as many ESGs as you feel like. You can have one huge ESG that takes care of all your load balancing needs, or you can have multiple small ones and hand over control to the responsible teams.
This is a good starting point doc from VMware.
—
You can have L4 and L7 load balancing. If you need only L4 (i.e. TCP, UDP, port number) the UI calls it acceleration. It’s a global configuration, on the ESG instance itself, so bear that in mind.
If you enable acceleration on an ESG, you have to also enable it per virtual server.
L4 load balancing is packet based (obviously, coz it doesn’t need to worry about the application as such). L7 load balancing is socket based. Quoting from this doc (highlight mine):
Packet-based load balancing is implemented on the TCP and UDP layer. Packet-based load balancing does not stop the connection or buffer the whole request, it sends the packet directly to the selected server after manipulating the packet. TCP and UDP sessions are maintained in the load balancer so that packets for a single session are directed to the same server. You can select Acceleration Enable in both the global configuration and relevant virtual server configuration to enable packet-based load balancing.
Socket-based load balancing is implemented on top of the socket interface. Two connections are established for a single request, a client-facing connection and a server-facing connection. The server-facing connection is established after server selection. For HTTP socket-based implementation, the whole request is received before sending to the selected server with optional L7 manipulation. For HTTPS socket-based implementation, authentication information is exchanged either on the client-facing connection or on the server-facing connection. Socket-based load balancing is the default mode for TCP, HTTP, and HTTPS virtual servers.
Also worth noting:
The L4 VIP (“acceleration enabled” in the VIP configuration and no L7 setting such as AppProfile with cookie persistence or SSL-Offload) is processed before the edge firewall, and no edge firewall rule is required to reach the VIP. However, if the VIP is using a pool in non-transparent mode, the edge firewall must be enabled (to allow the auto-created SNAT rule).
The L7 HTTP/HTTPS VIPs (“acceleration disabled” or L7 setting such as AppProfile with cookie persistence or SSL-Offload) are processed after the edge firewall, and require an edge firewall allow rule to reach the VIP.
—
Application Profiles define common application behaviors client SSL, server SSL, x-forwarded-for, and persistence. These can be reused across virtual server and is mandatory when defining a virtual server. This is also where you can do HTTP redirects.
The other day I blogged about how I had a host whose vMotion VMkernel interface seemed to be broken. Any vMotion attempts to it would hang at 14%.
At that time I logged on to the destination host, then used vmkping with the -I switch (to explicitly specify the vMotion VMkernel interface of the destination host), and found that I couldn’t ping the VMkernel interface of the other hosts. These hosts could ping each other but couldn’t ping the destination host.
The VMKernel interface is backed by two physical NICs. I found that if I remove one of the physical NICs from the VMkernel it works. Interestingly this link wasn’t showing any CDP info either, so it looked like something was wrong with it (the physical NIC shows as unclaimed coz the screenshot was taken after I moved it to unclaimed).
 So the first question is why did the VMkernel fail when only one of the physical NICs failed? Since the other physical NIC backing the VMkernel NIC is still active shouldn’t it have continued working?
The reason why it failed is that by default network failover detection is via “Link status only”. This only detects failures to the link – like say the cable is broken, the switch is down, or the NIC has failed – while failures such as the link being connected but blocked by switch are not detected. In my case as you can see from the screenshot above the link status is connected – so the host doesn’t consider the link failed even though it isn’t actually working, thus continues to use it.
Next I discovered that other hosts too similarly had their second vMotion physical NIC in a failed state as above yet they weren’t failing like this host. The simple explanation for this is that the host above somehow selected the faulty physical NIC as the one to use, didn’t detect it as failed and so continued to use it; whereas other hosts were more lucky and chose the physical NIC that works alright, so didn’t have any issues.
I am not sure that’s the entire answer though. For once the host that failed was ESXi 5.5 and using a distributed switch, while the other two hosts were ESXi 4.0 and using standard switches. Did that make a difference?
The default load balancing method for both standard and distributed switches is the same. (For a standard switch you check this under the vSwitch properties on the host. For a distributed switch you check this under the portgroup in the Networking section of vSphere (web) client).
 Load balancing is what I am concerned about here because that’s what the hosts should be using to balance between both the NICs. That’s what the host will be using to select the physical NIC to use for that particular traffic flow. The load balancing method is same between standard and distributed switches yet why were the distributed switch/ ESXi 5.5 hosts behaving differently?
I am still not sure of an answer but I have my theory. My theory is that since a distributed switch is across multiple hosts the load balancing method (above) of choosing a route based on virtual port ID comes into play. Here’s screenshots from two of my hosts connected to the same distributed switch port group for instance:
 As you can see the virtual port number is different for the VMkernel NIC of each host. So each host could potentially use a different underlying physical NIC depending on how the load balancing algorithm maps it.
But what about a standard switch? Since the standard switch is only on the host, and the only VMkernel NIC connected to it (in the case of vMotion) is the single VMKernel NIC I have assigned for vMotion, there is no load balancing algorithm coming into play! If, instead of a VMkernel I had a Virtual Machine network, then the virtual port number matters because there are multiple VMs connecting to the various port numbers; but that doesn’t matter for VMkernel NICs as there is only one of them. And so my theory is that for a VMkernel NIC (such as vMotion) backed by multiple physical NICs and using the default load balancing algorithm of virtual port ID – all traffic by default goes into one of the physical NICs and the other physical NIC is never used unless the chosen one fails. And that is why my hosts using the standard switches were always using the same physical NIC (am guessing the lower numbered one as that’s what both hosts chose) while hosts using distributed switches would have chosen different physical NICs per host.
That’s all! Just thought I’d put this out there in case anyone else has the same question.
One of the things you can do with a portgroup is define teaming for the underlying physical NICs.
If you don’t do anything here, the default setting of “Route based on originating virtual port” applies. What this does is quite obvious. Each virtual port on the virtual switch is mapped to a physical NIC behind the scenes; so all traffic to & from that virtual port goes & comes via that physical NIC. Since your virtual NIC connects to a virtual port this is equivalent to saying all traffic for that virtual NIC happens via a particular physical NIC.
In the screenshot above, for instance, I have two physical NICs dvUplink1 and dvUplink2. If I left teaming at the default setting and say I had 4 VMs connecting to 4 virtual ports, chances are two of these VMs will use dvUplink1 and two will use dvUplink2. They will continue using these mappings until one of the dvUplinks dies, in which case the other will take over – so that’s how you get failover.
This is pretty straightforward and easy to set up. And the only disadvantage, if any, is that you are limited to the bandwidth of a single physical NIC. If each of dvUplink1 & dvUplink2 were 1Gb NICs it isn’t as though the underlying VMs had 2Gb (2 NICs x 1Gb each) available to them. Since each VM is mapped to one uplink, 1Gb is all they get.
Moreover, if say two VMs were mapped to an uplink, and one of them was hogging up all the bandwidth of this uplink while the remaining uplink was relatively free, the other VM on this uplink won’t automatically be mapped to the free uplink to make better use of resources. So that’s a bummer too.
A neat thing about “Route based on originating virtual port” is that the virtual port is fixed for the lifetime of the virtual machine so the host doesn’t have to calculate which physical NIC to use each time it receives traffic to & from the virtual machine. Only if the virtual machine is powered off, deleted, or moved to a different host does it get a new virtual port.
The other options are:
- Route based on MAC hash
- Route based on IP hash
- Route based on physical NIC load
- Explicit failover
We’ll ignore the last one for now – that just tells the host to use the first physical NIC in the list and use that for all VMs.
“Route based on MAC hash” is similar to “Route based on originating virtual port” in that it uses the MAC address of the virtual NIC instead of virtual port. I am not very clear on how this is better than the latter. Since the MAC address of a virtual machine is usually constant (unless it is changed or a different virtual NIC used) all traffic from that MAC address will use the same physical NIC always. Moreover, there is the additional overhead in that the host has to check each packet for the MAC address and decide which physical NIC to use. VMware documentation says it provides a more even distribution of traffic but I am not clear how.
“Route based on physical NIC load” a good one. It starts off with “Route based on originating virtual port” but if a physical NIC is loaded, then the virtual ports mapped to it are moved to a physical NIC with less load! This load balancing option is only available for distributed switches. Every 30s the distributed switch checks the physical NIC load and if it exceeds 75% then the virtual port of the VM with highest utilization is moved to a different physical NIC. So you have the advantages of “Route based on originating virtual port” with one of its major disadvantages removed.
In fact, except for “Route based on IP hash” none of the other load balancing mechanisms have an option to utilize more than a single physical NIC bandwidth. And “Route based on IP hash” does not do this entirely as you would expect.
“Route based on IP hash”, as the name suggests, does load balancing based on the IP hash of the virtual machine and the remote end it is communicating with. Based on a hash of these two IP addresses all traffic for the communication between these two IPs is sent through one NIC. So if a virtual machine is communicating with two remote servers, it is quite likely that traffic to one server goes through one physical NIC while traffic to the other goes via another physical NIC – thus allowing the virtual machine to use more bandwidth than that of one physical NIC. However – and this is an often overlooked point – all traffic between the virtual server and one remote server is still constrained by the bandwidth of the physical NIC it happens via. Once traffic is mapped to a particular physical NIC, if more bandwidth is required or the physical NIC is loaded, it is not as though an additional physical NIC is used. This is a catch with “Route based on IP hash” that’s worth remembering.
If you select “Route based on IP hash” as a load balancing option you get two warnings:
- With IP hash load balancing policy, all physical switch ports connected to the active uplinks must be in link aggregation mode.
- IP hash load balancing should be set for all port groups using the same set of uplinks.
What this means is that unlike the other load balancing schemes where there was no additional configuration required on the physical NICs or the switch(es) they connect to, with “Route based on IP hash” we must combine/ bond/ aggregate the physical NICs as one. There’s a reason for this.
In all the other load balancing options the virtual NIC MAC is associated with one physical NIC (and hence one physical port on the physical switch). So incoming traffic for a VM knows which physical port/ physical NIC to go via. But with “Route based on IP hash” there is no such one to one mapping. This causes havoc with the physical switch. Here’s what happens:
- Different outgoing traffic flows choose different physical NICs. With each of these packets the physical switch will keep updating its MAC address table with the port the packet was got from. So for instance, say the two physical NICs are connected to physical switch Port1 and Port2 and the virtual NIC MAC address is VMAC1. When an outgoing traffic packet goes via the first physical NIC, the switch will update its tables to reflect that VMAC1 is connected to Port1. Subsequent traffic flows might continue using the first physical NIC so all is well. Then say a traffic flow uses the second physical NIC. Now the switch will map VMAC1 to Port2; then a traffic flow could use Port1 so the mapping gets changed to Port1, and then Port2, and so on …
- When incoming traffic hits the physical switch for MAC address VMAC1, the switch will look up its tables and decide which port to send traffic on. If the current mapping is Port1 traffic will go out via that; if the current mapping is Port2 traffic will go out via that. The important thing to note is that the incoming traffic flow port chosen is not based on the IP hash mapping – it is purely based on whatever physical port the switch currently has mapped for VMAC1.
- So what’s required is a way of telling the physical switch that the two physical NICs are to be considered as bonded/ aggregated such that traffic from either of those NICs/ ports is to be treated accordingly. And that’s what EtherChannel does. It tells the physical switch that the two ports/ physical NICs are bonded and that it must route incoming traffic to these ports based on an IP hash (which we must tell EtherChannel to use while configuring it).
- EtherChannel also helps with the MAC address table in that now there can be multiple ports mapped to the same MAC address. Thus in the above example there would now be two mappings VMAC1-Port1 and VMAC1-Port2 instead of them over-writing each other!
“Route based on IP hash” is a complicated load balancing option to implement because of EtherChannel. And as I mentioned above, while it does allow a virtual machine to use more bandwidth than a single physical NIC, an individual traffic flow is still limited to the bandwidth of a single physical NIC. Moreover there is more overhead on the host because it has to calculate the physical NIC used for each traffic flow (essentially each packet).
Prior to vCenter 5.1 only static EtherChannel was supported (unless you use a third party virtual switch such as the Cisco Nexus 1000V). Static EtherChannel means you explicitly bond the physical NICs. But from vCenter 5.1 onwards the inbuilt distributed switch supports LACP (Link Aggregation Control Protocol) which is a way of automatically bonding physical NICs. Enable LACP on both the physical switch and distributed switch and the physical NICs will automatically be bonded.
(To enable LACP on the physical NICs go to the uplink portgroup that these physical NICs are connected to and enable LACP).
 That’s it for now!
Update
Came across this blog post which covers pretty much everything I covered above but in much greater detail. A must read!
|