At work one of my colleagues is setting up NetScalers as load balancers for our new Exchange environment. He is replicating the existing setup but found that the RPC 60001 & 60002 Service Groups on the NetScalers were being marked as down. Curious, I took a look.
After SSH-ing into the NetScaler I could see the following via show serviceGroup <serviceGroupName>:
|
1 2 3 |
Monitor Name: tcp State: DOWN Passive: 0 Probes: 93936 Failed [Total: 93936 Current: 93936] Last response: Failure - TCP syn sent, reset received. |
My colleague too had seen this and pointed me to a good blog post from Citrix on what the reset codes mean. That blog post is a good one (that’s why I am linking it here, as a reference to myself) but I don’t think he was looking at the trace via a NetScaler trace so we had no idea of the codes. (Speaking of which, here’s a good post on NetScaler and Wireshark. Here’s a KB article on how to collect traces from NetScaler. And here’s a KB article on how to collect traces from the CLI. Whilst I have briefly read them, I haven’t tried them out currently).
Back to the issue at hand. I could see that the individual servers (Exchange 2010 Client Access) were up on RPC 135 and HTTPS, but only RPC 60001 & 60002 were down. I decided to do a portQry against a server in the older environment and compare against the new. Here’s the relevant bits from an older server:
|
1 2 3 4 5 6 7 8 9 10 11 |
UUID: a4f1db00-ca47-1067-b31f-00dd010662da ncacn_ip_tcp:EXHTCA01[60001] UUID: 5261574a-4572-206e-b268-6b199213b4e4 ncacn_ip_tcp:EXHTCA01[60001] UUID: f5cc5a18-4264-101a-8c59-08002b2f8426 Microsoft Exchange NSP Interface ncacn_ip_tcp:EXHTCA01[60002] UUID: 1544f5e0-613c-11d1-93df-00c04fd7bd09 Microsoft Exchange RFR Interface ncacn_ip_tcp:EXHTCA01[60002] |
As expected, something is listening on ports 60001 and 60002. When I tried the same against the new server, however, there was nothing listening on either of these ports. I searched the output based on the UUIDs and found the port numbers were different:
|
1 2 3 4 5 6 7 8 9 10 11 |
UUID: a4f1db00-ca47-1067-b31f-00dd010662da ncacn_ip_tcp:NEW-EXHTCA01[15212] UUID: 5261574a-4572-206e-b268-6b199213b4e4 ncacn_ip_tcp:NEW-EXHTCA01[15212] UUID: f5cc5a18-4264-101a-8c59-08002b2f8426 Microsoft Exchange NSP Interface ncacn_ip_tcp:NEW-EXHTCA01[15168] UUID: 1544f5e0-613c-11d1-93df-00c04fd7bd09 Microsoft Exchange RFR Interface ncacn_ip_tcp:NEW-EXHTCA01[15168] |
So that’s why the NetScalers were getting a reset. Nothing was listening on those ports! Solution is simple. Configure these RPC ports as static.
That’s all! :)