Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

Exchange 2016 & NSX Edge load balancing

This is going to be a quick and dirty post with not much details. Sorry. Spent some time figuring this out today and I wanted to put it here as a reference for anyone else. 

At work we needed to setup two Exchange 2016 servers behind an NSX Edge load balancer. We wanted to capture the source IP too so this meant we had to use the load balancer in transparent mode. A colleague had set it up already but the Exchange servers weren’t seeing the source IP so I took a look to see what was missing. I had to make two changes primarily to get it working. 

First: the Edge had two interfaces assigned to it. One for HA, another that connected to the backend servers. The VIP for the Edge was also in the second subnet. (This needn’t always be the case. I think a usual scenario is for the VIP to be on a different subnet so YMMV). For the Edge to pass on the source IP to the Exchange servers I knew I had to set it up in a transparent mode. This means the Edge passes on any packets it gets off to the appropriate backend server. It does not change the IP in the packet to be itself, so the backend server see the correct source IP. The problem with this though is that the backend server will then send this packet to its default router and from there to the source IP … which we don’t want. We want the source to only see the Edge VIP always and this means we need the return traffic to come back to the Edge, and the Edge will do a NAT to change the backend server IP to be the VIP. 

(This blog post might be a good starting point to read more on the above).

For an Edge to be setup in this fashion we have to set the Edge as the default gateway for the backend servers. (Which is straight-forward – just change the default gateway in the server OS). But for the Edge to then act also as a router for the backend servers we have to tell it that such and such IP is on an internal network and it is ok to do routing on that. In our case the HA network had been set as of type Internal, but the network that was connecting to the backend servers was incorrectly set as Uplink. I changed its type to Internal. This is important as this is what tells the Edge that it can now route any traffic coming to the IP addresses defined on that interface. 

This done, I went to the two backend servers and changed their default gateway to be the IP defined on the above Internal interface (vNIC0 in my case). I tested connectivity and also did a trace route to confirm it is going via the Edge. Great!

Second: By default an Edge load balancer is in L7 mode. When in L7 mode the Edge doesn’t forward on the packets it receives. It initiates a new connection to the Exchange servers, so the source IP is itself and defeats what we are trying to do. L7 mode is useful if we are doing any L7 manipulation such as SSL termination, cookie based persistence, URL rewriting, header insertion, etc. We are not doing anything like that here so I changed the mode to L4. This is done by enabling acceleration. 

Did the same in the virtual server too (this option is only available after enabling it in the above section).

Lastly, I enabled transparent mode on the pool.

That’s all!

Gaining access to Citrix Studio if you don’t already have access

I am proud of this one. Spent a lot of time working my way through this even though I don’t know much SQL and finally cracked it. Probably not a big deal for any “experts” out there but this pretty much was the highlight of my day. :) 

A colleague of mine setup a new Citrix site and went for holiday, without giving the rest of us admin access to the site. As expected we needed to access it and while we were waiting for him to get in touch to our messages I thought there must be a way to hack into the system. There is a database behind the scene after all, so if I could just get access to that then maybe I can give myself admin access. 

Turns out there is.

We had gone with SQL Express with both delivery controller and SQL server on the same machine, and thanks to this Citrix support article I learnt that in such a case the ‘NT AUTHORITY\NETWORK SERVICE’ account is used to login to the SQL server (that article is a good read for other scenarios too BTW). Cool. I knew I could run something as ‘NT AUTHORITY\NETWORK SERVICE’ using SysInternals PSTools. So I downloaded PSTools to that server, opened a command prompt as admin, and ran the following:

All good so far. Next I downloaded SQL Studio and ran that from the above command prompt. Just type "C:\Program Files (x86)\Microsoft SQL Server Management Studio 18\Common7\IDE\Ssms.exe" into the command prompt window. That will give you the login prompt and you can connect (if it asks for any details the server name is “<your server name>\SQLEXPRESS” and authentication is “Windows Authentication”). This worked and I was in! Yay.

Snooping around the various SQL tables I came across [DAS].[Administrators] which looked like it could contain the administrators. Did a right click > “Select Top 1000 Rows” (remember I am no SQL guru) and that opened a new query which I executed … and sure enough I could see the sole admin account of my colleague who’s on holiday. Nice! Seems to be a list of SIDs followed by a UserIdentityType column of value 0 and Enabled column of value 1. Hmm, maybe I can just add to this table and be done with it? Did a bit of Googling on how to insert into a table, found my SID from psgetsid of the PSTools I had already downloaded, and tried the following:

And … that didn’t work! Got the following error: “The INSERT permission was denied on the object ‘Administrators’, database …”

Oh well, worth a shot. I looked around the user accounts on the SQL server and the roles and permissions for the network service account and from what I could see it has all the rights it needs. There’s no other account. So surely that’s what the Delivery Controller too is using to add new admins etc. Time to read more. 

Back to the Citrix support article I came across earlier, I found the same roles that I had found on the SQL server and also this bit: “Each one of the preceding roles has the minimum permissions granted to it to allow the corresponding service on the controller to function. These permissions are restricted to execute on stored procedures and read on some tables.” Ah ha! So it has permissions to only execute stored procedures and that’s obviously how it is adding admins. Cool!

Obviously I have no idea what a stored procedure is, so time to Google again on how to get to that. Did that, and found a ton of them them under Programmability > Stored Procedures. The table was called “DAS” something so upon a hunch I looked around any procedures starting with “DAS” (not entirely a hunch, I noticed that the procedures seemed to start with similar names as the tables so I made a guess that probably the stored procedures for the “DAS” tables would start with the same name). That paid off and I found “DAS.NewAdministrator”. Cool!

Note to anyone else: to see a stored procedure you right click and do “Modify”. That shows you the code. You can run it via right click > “Execute Stored Procedure” which will give a popup to enter the parameters for the procedure. This part stumped me for a while. I entered the parameters as best as I could figure but it kept throwing various errors. That’s when I spent some time looking at the procedure code and cracked the problem. Once you enter the parameters SQL Studio generates a query which you execute, and that was giving errors. I figured the issue and modified the query. It looks like the below in case anyone else wants to copy-paste and modify:

And that worked! Whoo hoo. Still can’t access via Studio, but I double checked the [DAS].[Administrators] table and my account was there. 

Hmm, maybe the issue is that I have added myself as an admin but I haven’t granted myself any rights. Remember when you do this via the Studio you have to select a scope and also what rights you want to assign? Probably got to do that via SQL! Not a problem, back to Google. :)

I came across another Citrix article (why didn’t I just find this the first time!? it tackles pretty much what I am doing here. anyways, the first few steps of that article are incorrect as that’s what I too had tried and it didn’t work for me … so good I didn’t stumble upon this initially). This one showed me how to give my admin account rights and scope. Here’s the additional SQL you need to run:

No rocket science here. It uses another stored procedure called “DAS.AddRight” to give my SID “Full Administrator” rights to the scope of “All Objects”. That completed without any errors, so I closed and opened Citrix Studio and yay I am now in!

And that, ladies and gentlemen is how you get into Citrix Studio if you don’t already have access! :)

Downloading all episodes of a podcast

Not a biggie but in case it helps anyone.

I wanted to download all episodes of the excellent “My Dad Wrote a Porno” podcast for posterity. I couldn’t find any way of doing this so here’s what I ended up doing.

First I found the RSS feed. I noticed that it contains the actual audio file in enclosure tags.

Cool, so I just need to read these for a start. I can do that via curl.

This gives me all the links thus:

I was able to extract just the URL via a modification to the above snippet to match the beginning double quotes:

Now all I needed to do was download these and also rename the “media.mp3” to be the directory name from the path. The following did that:

I use sed to strip out the domain name and also do the word “media”. What remains is the part of the path I am interested in.