Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Recent Posts

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

NetScaler/ Exchange RPC – TCP syn sent, reset received

At work one of my colleagues is setting up NetScalers as load balancers for our new Exchange environment. He is replicating the existing setup but found that the RPC 60001 & 60002 Service Groups on the NetScalers were being marked as down. Curious, I took a look.

After SSH-ing into the NetScaler I could see the following via show serviceGroup <serviceGroupName>:

My colleague too had seen this and pointed me to a good blog post from Citrix on what the reset codes mean. That blog post is a good one (that’s why I am linking it here, as a reference to myself) but I don’t think he was looking at the trace via a NetScaler trace so we had no idea of the codes. (Speaking of which, here’s a good post on NetScaler and Wireshark. Here’s a KB article on how to collect traces from NetScaler. And here’s a KB article on how to collect traces from the CLI. Whilst I have briefly read them, I haven’t tried them out currently). 

Back to the issue at hand. I could see that the individual servers (Exchange 2010 Client Access) were up on RPC 135 and HTTPS, but only RPC 60001 & 60002 were down. I decided to do a portQry against a server in the older environment and compare against the new. Here’s the relevant bits from an older server:

As expected, something is listening on ports 60001 and 60002. When I tried the same against the new server, however, there was nothing listening on either of these ports. I searched the output based on the UUIDs and found the port numbers were different:

So that’s why the NetScalers were getting a reset. Nothing was listening on those ports! Solution is simple. Configure these RPC ports as static.

That’s all! :)

WMI Access Denied for remote machine etc

This isn’t going to be a coherent post really (unlike my usual posts which are more coherent, I hope!). I came across a bunch of new stuff as I was troubleshooting this WMI issue and thought I should put them all somewhere.

The issue is that we are trying to get Solarwinds to monitor one of our DMZ servers via WMI but it keeps failing.

solarwinds

Other servers in the DMZ work, it’s just this one that fails. WMI ports aren’t fixed like I had mentioned earlier but I don’t think that matters coz they aren’t fixed for the other servers either. Firewall configuration is the same between both servers.

I thought of running Microsoft Network Monitor but realized that it’s been replaced with Microsoft Message Analyzer. That looks nice and seems to do a lot more than just network monitoring – I must explore it sometime. For now I ran it on the DMZ and applied a filter to see traffic from our Solarwinds server to it. The results showed that access was being denied, so may not a port issue after all.

analyzer message

Reading up more on this pointed me to a couple of suggestions. None of them helped but I’d like to mention them here for future reference.

First up is the command wbemtest. Run this from the local machine (the DMZ server in my case), click “Connect”, and then “Connect” again.

wbemtest1

If all is well with WMI you should get no error.

wbemtest2

Now try the same from the Solarwinds server, but this time try connecting to the DMZ server and enter credentials if any.

wbemtest3

That worked with the local administrator account but failed with the account I was using from Solarwinds to monitor the server. Error 0x80070005.

wbemtest4

So now I know the issue is one of permissions and not ports.

Another tool that can be used is WmiMgmt. Just type wmimgmt.msc somewhere to launch it. I tried this on the DMZ machine to confirm WMI works fine. I also tried it from SolarWinds machine to the DMZ machine. (Right click to connect to a remote computer).

wmimgmt

Problem with WmiMgmt is that unlike wbemtest you can’t a different account to use. If the two machines are domain joined or have the same local accounts, then it’s fine – you can run as from one machine and connect to the other – but otherwise there’s nothing much you can do. WmiMgmt is good to double check the permissions though. Click “Properties” in the screenshot above, go to the “Security” tab, select the “Root” namespace and click the “Security” button. The resulting window should show the permissions.

wmimgmt2

In my case the Administrators group members had full permissions as expected. The account I was using from the Solarwinds server was a member of this group too yet had access denied.

Another place to look at is Component Services > Computers > My Computer > DCOM Config > “Windows Management and Instrumentation” – right click and “Properties”.

componentservices

Make sure “Authentication Level” is “Default”. Then go to the “Security” tab and make sure the account/ group you want has permissions.

componentservices2

 

Also right click on “My Computer” and go to “Properties”.

componentservices3

Under the “COM Security” tab go to the “Edit Limits” of both access & launch and activation permissions and ensure the permissions are correct. My understanding is that the limits you specify here over-ride everything else.

componentservices4

In my case none of the above helped as they were all identical between both servers and at the correct setting. What finally helped was this Serverfault post.

Being DMZ servers these were on a workgroup and the Solarwinds server was connecting via a local account. Turns out that:

In a workgroup, the account connecting to the remote computer is a local user on that computer. Even if the account is in the Administrators group, UAC filtering means that a script runs as a standard user.

That is a good link to refer to. It is about Remote UAC and WMI. The solution is to go to the following registry key – HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System – and create a new name called LocalAccountTokenFilterPolicy (of type DWORD_32) with a value of 1. (Check the Serverfault post for one more solution if you are using a Server 2012).

Remote UAC. Ah! Am surprised I didn’t think of that in the beginning itself. If the LocalAccountTokenFilterPolicy has a value of 0 (the default) then whenever a member of the Administrators group connects remotely, the security tokens of that account and filtered to remove admin access. This is only for local admin accounts, not domain admin accounts – mind you. If LocalAccountTokenFilterPolicy has a value of 1, then no filtering happens and the account connects with full admin rights. I have encountered Remote UAC in the past when accessing admin shares (e.g. \\computer\drive$) in my home workgroup, never thought it would be playing a role here with WMI!

Hope this helps someone. :)

PortQry and PortQryUI

I was aware of PortQry but didn’t know it has a GUI counterpart too PortQryUI. For a quick read on PortQry check out this link, if you have more time and interest check out this KB article. PortQry/ PortQryUI can be used to check the status of TCP and UDP ports on a remote computer. For TCP ports I usually do a telnet to the port (by habit) but didn’t have any equivalent tool for UDP ports. 

Important PortQry switches (as a reference to myself) are:

  • -n -> specifies the server name/ IP address to target
  • -p -> specifies the protocol to test (options are tcp or udp or both; default is tcp)
  • -e -> specifies the port(s) to test (default is port 80)
    • note: it is possible to specify a single port (e.g. -e 81), ports (e.g. -e 80,81) or a range (e.g. -e 80-1024)
    • note: the ports must be in the range 1-65535
  • instead of -e you can use either of the following too:
    • -r -> specifies a port range (e.g. –r 80:90)
    • -o -> specifies a comma-separated list of ports to check in order (e.g. -o 80,443,139)

Some other switches are:

  • -nr -> stops PortQry from resolving an IP address to a name
  • -sl -> waits longer for replies from UDP systems (sl == slow link)
  • -l -> specifies a log file to output to
    • -y -> will over-write the log file if it exists, without prompting

While writing this post I learnt that PortQry can also enumerate the local ports. Nice!

  • The -local switch will list all active TCP/UDP ports on the local system. (Think of it as netstat -a but without any details of the remote end).
  • The -wport (port number) switch will watch a specified port’s state and report when it changes
    • This didn’t work for me, got an error “Port to process mapping is not supported on this system”.
  • The -wpid (PID) switch will watch a specified process ID (PID) and reports when its state changes
    • This too didn’t work for me, same error as above.

A good thing about PortQry is that it can also query protocols that it’s aware of. Thus, for instance, if you query port 53/ UDP (DNS) and something’s listening at the remote end, PortQry can send an additional DNS query to that port. 

This is useful in AD troubleshooting too. For instance, to check whether port 389 of a DC has an LDAP server listening as it should be:

Similarly, RPC:

In the output above, for instance, I query port 135/ TCP which is where the RPC end-mapper service listens. After querying this port and getting a response, PortQry asks it to enumerate the listening services. Of these UUID 12345887-... is what the netlogon service registers under, which as we can see from the above output is listening on ports 49155 (via TCP), 49158 (via TCP), and 49157 (via HTTP). (Netlogon registers with RPC and uses dynamic ports as we saw above, so querying the RPC end-mapper service is the only way to find what ports Netlogon is listening on). 

In contrast to PortQry, PortQryUI has options to query for the services it is aware of. So, for instance, one can use it to query the “Domains and Trusts” service on a DC and it will do PortQry queries to port 135/TCP, port 389/BOTH, port 445/TCP, port 137/UDP, and a few other AD related ports and emit the output in a window (you can see part of the output in the screenshot below). 

portqryUI

What is RPC?

In computer programming a “procedure” is a set of instructions packaged into one unit. For instance: say you regularly need to open a text file, scan it for certain text, and output a yes or no depending on whether there’s a match. One way is to keep writing these instructions at each place in your program. Another way would be to put these instructions at one place in your program, give that logical grouping a name, and then on every time you need to run these instructions simply invoke that grouping. The latter is an easier approach, not only because it saves your typing (or copy-pasting) the same code all over the place, but also because if you ever need to improve the instructions you just have to do it at one place. This logical grouping is what’s known as a procedure. Other names for a procedure are function, subroutine, and method.

When a procedure is invoked at a point in the code, usually with any inputs that are to be passed on to the procedure, the inputs are saved someplace and the procedure is loaded and executed. The main program is on hold while the procedure runs, its current state is stored at a location and execution moves to the procedure. The procedure reads the inputs from the saved place, does its deeds, stores any outputs in a designated place, and passes execution back to the main program (whose location was previously stored). The main program then reads the output and continues.

To give an analogy: think of it as though you are reading a book and you find a word whose meaning you do not know, so you keep down the book and open a dictionary to find the meaning. “Reading a book” is the main program; the “looking up the meaning” is a procedure that takes as input a word whose meaning you want, scans a dictionary, and gives as output an understanding of the word. The word itself is stored in your memory, and so is the the output (the meaning). That’s pretty much how procedures work.

Procedures can be part of the code of the main program or separate from it. They can even be from a 3rd party, packaged into files (such as DLLs) containing multiple procedures. When writing code the author links to these files and invokes procedures from them. When compiling the code to turn them into machine instructions the compiler puts in the required details on how to find the procedures (their address and such). That’s how the processor, which actually executes the program, knows where to find the procedure and what it expects.

When executing a program the processor only has access to its memory space. And since a procedure is linked to the program and has to be executed by the processor, it too must be in the same memory space as the main program. So procedures have to be local to the main program, i.e. on the same computer as the program. They can’t be on a remote computer. If a procedure is on a different machine (a different memory space) the processor has has no way of connecting to it.

As computers started to be connected over networks though, it because useful to have procedures that were remote to the main program. This way you could distribute the load across multiple computers, and maybe use powerful computers for some tasks and regular computers for others. Thus came about the concept of Remote Procedure Calls (RPC). RPC is a way of running procedures on remote computers by tricking the local processor into thinking the procedure is local to it.

Here’s how RPC works:

  1. Instead of a local procedure as usual, you have a stub procedure. This is a local procedure that takes input parameters meant for the remote procedure as though it were the remote procedure. As far as the main program is concerned the procedure is local.
    • To avoid confusion later on, we will refer to this stub as a client stub henceforth.
  2. The client stub procedure converts the input parameters into a form that can be understood by the remote computer. This is necessary as the parameters are in a location (memory space) that cannot be accessed by the remote computer, and also because the local and remote computers may use different ways of representing data (big-endian vs little-endian, for instance) and so the data may need conversion to a standard format. This process is called marshaling.
    • On Windows the client stub takes the input parameters (as in copies the values of the input parameters) and passes on to a Client Runtime Library (called rpcrt4.dll). This library does the marshaling.
    • This standard format is called a Network Data Representation (NDR) format. On Windows there are two marshaling “engines” that do this conversion. NDR20 is for 32-bit programs. NDR64 is for 64-bit programs (thought 64-bit programs can also use NDR20). Both marshaling engines are part of the Client Runtime Library.
  3. The client stub procedure uses the OS to contact the remote end and sends it messages. The OS in turns uses transport layer network protocols.
    • The client stub knows where the remote end is because it is either configured with the information or it looks up a central location to find the remote end. Usually the network address and port number(s) of the remote end.
      • For instance Windows domain joined computers can use Active Directory. In Windows 2000 domains, RPC services register under System\RPCServices.
      • This remote network address and port(s) is called an Endpoint.
    • The remote end usually has an RPC service listening on a designated port(s) and network address. This service allocates dynamic Endpoints for client stubs to connect to.
      • This service is called an Endpoint Mapper (EPM). In Windows you have service called the RPC Endpoint Mapper. It listens on port 135.
    • During this negotiation phase the EPM protocol messages contain items known as towers and floors. I am not too clear on what they are except that they are ways of representing RPC data and also for identifying the host address and port. When doing a network capture of RPC traffic these can be seen (the last link in the list below has examples of capturing RPC traffic).
    • On Windows the Client Runtime Library has three protocol engines – 1) a Connection RPC protocol engine (used when a connection-oriented protocol is required), 2) a Datagram RPC protocol engine (used when a connection-less protocol is required), and 3) a Local RPC protocol engine (used when the remote end is on the same host as the client). These protocol engines are what do the actual message passing.
    • Connection-oriented protocols used are TCP (dynamically assigned ports), SPX (Sequenced Packet Exchange), named pipes (port 445), or HTTP (ports 80, 443, 593). Connection-less protocols used are UDP (dynamically assigned ports) or CDP (Cluster Datagram Protocol).
  4. The remote end has a server stub that receives data from the local stub and unmarshals it. If required, the server stub converts the data to the remote machine specific format.
    • Server stubs register themselves with the Endpoint Mapper. Server stubs have a UUID (a well known GUID that identifies the application).
    • Once client stubs get an Endpoint from the Endpoint Mapper the client and server stubs talk to each other directly.
  5. The server stub then invokes the remote procedure with the data it has. As far as the remote procedure is concerned the main program is local to it. It is not aware that it is remote to the main program.
  6. Output from the remote procedure is passed on to the server stub which sends it to the client stub following a similar process as above (marshaling, unmarshaling).

The neat thing about RPC is that programmers can write distributed applications and don’t have to worry about the network details. In terms of the Internet Protocol suite, RPC sits in the Application layer.

Here are some good links on RPC: