I setup a pair of Kemp LoadMasters at work earlier this week to load balance Exchange 2016. Now I know if you want Exchange 2016 to see the source IP of the clients you must enable transparency on the Kemps (or any load balancers for that matter; I’ve had to do this in the past with NSX for instance) and also set the Kemps as the default gateway for all the Exchange servers so traffic goes back through the Kemps; and I also know that this will not work if the client is in the same subnet as the Exchange servers as then the return traffic is simply sent to the clients directly because a gateway isn’t required.
We didn’t want to change the default gateway of the Exchange servers and we didn’t care about the source IP of the clients. So far so good. I should have unticked the “Transparency” option on the Kemp Virtual Service for Exchange but I didn’t ask I don’t know I was curious I suppose to see what would happen. So what we had was Kemps set to “Transparency” but Exchange 2016 not having Kemps as the default gateway – a situation which in theory should not work.
Funnily enough when I set this up and we tested by doing a telnet to port 25 of the Kemp VIP for Exchange, I was successfully able to send an email. So I figured hmm that’s odd, and left it as it is. Today, however, I got reports that while some clients were able to send emails others weren’t… and digging in that I found that clients in the same subnet as the Kemps & Exchange 2016 (both of which are in the same subnet) are succesfully able to send emails while clients in a different subnet fail. I can understand why they fail – because the Kemps aren’t the default gateway – but I have no idea why clients in the same subnet work. I did some tcpdumps from the Kemps and looked at them and still can’t figure out. Odd.
I made some pictures too in Lucidchart (coz why not, I enjoy making pictures) so I’ll add them to this post.
Non-Transparent Mode
When a load balancer is in non-transparent mode here’s the flow of traffic. As you can see when the Kemp sends the traffic over to the backend server, it changes the source IP to be itself… thus each section of the traffic (1 & 4, and 2 & 3) are contained to themselves and succeed. The only catch is the server sees the load balancer as the source IP.
Transparent Mode
In this mode, as I said earlier, if clients are in a different subnet and the load balancer is the default gateway for the backend servers things work fine. As far as the backend servers are considered the load balancers are transparent – they don’t see it at all. The only catch is when clients from the same subnets try to access via the load balancer the connections fail – or at least they are supposed to fail, and the fair bit of Googling I did today to double check my understanding confirms it too, but for whatever reason I can’t get them to fail here!
Here’s output from the tcpdump though. The .2.30 IP is my client, .2.173 is the Kemp, and .2.74 is the Exchange server. 
The first three conversations are the three-way handshake from client to Kemp. The next three are from Kemp to the backend Exchange 2016. I would have expected the source IP for these to be the client as the Kemp is supposed to be transparent, but the source IP is the Kemp instead… so that’s why things were working for clients in the same subnet.
When I do a similar tcpdump for clients not in the same subnet, I see the first 3 conversations and after that I see the Kemp initiating a handshake with the backend Exchange 2016 with the source IP changed to be that of the client, as expected. It doens’t get a response and there’s a bunch of retransmissions. Makes sense why it doens’t get a response; but doesn’t make sense why the Kemp didn’t change the source IP in the case where the client was in the same subnet.
Oh well… *shrug*