Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

Brief notes on the Citrix STA

Wanted to point out this PDF from Citrix on the XenApp/ XenDesktop architecture – especially pages 21, 22 which are on how authentication works. During my Citrix course the instructor had talked about it but like an idiot I didn’t take notes and now I can’t find much info on what he was explaining. 

The part which is of interest to me is the STA (Secure Ticket Authority). 

There’s a couple of steps that happens when a user logs in to access a Citrix solution. First: the StoreFront authenticates the user against AD. Or if the user is accessing remotely, the NetScaler gateway authenticates the user and passes on details to the StoreFront. Then the StoreFront passes on this information to the Delivery Controller so the latter can give a list of resources the user has access to. The Delivery Controllers in turn authenticate the user AD. The Delivery Controller then sends a list of resources the user has access to, to the StoreFront, which sends this on to the user’s Citrix Receiver or Browser. This is when the user sees what is available to them, and can select what they want.

When the user selects what they want, this is information is passed on to the StoreFront, which then passes the info to the Delivery Controller – who then finds an appropriate host that can fulfill the requirement and sends this information to the StoreFront. 

The next step is where the STA comes in. 

In case the user is accessing Citrix locally, the StoreFront can create an ICA file with details of the host and send it over to the user’s Citrix Receiver or Browser and the latter can then directly talk to the VDA agent installed on the host (note the StoreFront & Delivery Controller have no more role to play). But what if the user is accessing remotely? We don’t want to send these sensitive details over the public Internet. So, as a workaround, Citrix creates a “ticket” (which is a randomly generated sequence of 32 uppercase alphabetic or numeric characters) and associates the ticket with the details of the host that the Citrix Receiver or Browser need to contact to access the requested resources. This ticket is what is sent over to Citrix Receiver or Browser in the ICA file, using which it can contact the NetScaler gateway and the NetScaler gateway can validate this and initiate a connection with the VDA on the host on behalf of the user. 

So, as we can see the STA only comes into play in case of remote access. The STA is a part of the Citrix XML Service (once again linking to this excellent post!), which is installed as part of the Delivery Controller (so the STA is a part of the Delivery Controller). It is written as an ISAPI extension (called CtxSta.dll) for the IIS WebServer and runs the /Scripts/CtxSta.dll URL. The STA has an ID called the STA_ID, and this along with the TICKET and an STA_VERSION field are what is put into the ICA file. I am not sure whether the STA requires IIS, or it can run standalone (as I blogged previously the Citrix XML Service can run standalone so I would assume the STA can do the same). The Citrix STA FAQ says IIS is required, but that could be outdated.

The Citrix StoreFront is configured with the STA details in the NetScaler Gateway section (remember you only need to use the STA in case of remote users, for which you would have to configure a NetScaler Gateway). 

Similarly the NetScaler itself is configured with the STA details. 

It is important to keep in mind that there are thus TWO places where the STA details are input, and that the details in both places must be the same. The StoreFront uses its configured details to generate a ticket and put it in the ICA file. And the StoreFront uses its configured details to validate that ticket with an STA and identify what resources it should connect to. If the two details are not identical then you will not be able to launch any resources! (I had this problem at work today which is why I decided to refresh my knowledge about STAs and thought of writing this blog post. If the two details are not identical you will get a “Cannot start App:” error because the ticket the client has cannot be validated or used by the NetScaler). 

Just as an aside to myself – the port used to talk to the VDA is 1494 or 2598. This is the case if the Citrix Receiver or Browser contacts the VDA, or if the NetScaler gateway does so on behalf of these. I like to remember port numbers. :o) 

Also – there is nothing that ties a particular STA generated ticket to the device where the request was made from. That is, in theory a remote user could make a request from Computer A, get the ICA file and run it on Computer B – and NetScaler + STA will happily let the user access resources. A ticket only has a 100 seconds validity, so they’d have to do this switch-over quickly though. ;o) Also, a ticket can only be used once. (Also this and more info are from the very informative Citrix STA FAQ by the way). 

Citrix XML Service headaches

Was setting up a Citrix XenDesktop environment in my test environment past few days and the Citrix XML service has been irritating me. There was no grand fix for the issue, but I spent quite a bit of time banging my head over it (and learnt some stuff along the way) so thought I’d make a post to put it all down.

Whenever I’d connect to the Storefront I get the following error:

I only get this error if you use the Receiver app by the way. If I try and connect via HTML5 you get no error at all. (So when in doubt, try with the Receiver always!)

I noticed that the “Citrix Delivery Service” event logs on the server had messages like these:

An SSL connection could not be established: You have not chosen to trust the issuer of the server’s security certificate, my-CA.. This message was reported from the Citrix XML Service at address https://mydeliverycontroller.mydomain/scripts/wpnbr.dll. The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services.

I sorted that by changing all my certificates to SHA1. Turns out the default certificate signature algorithm from a Windows CA since 2008R2 is RSASSA-PSS, and Citrix doesn’t support RSASSA-PSS, so switching the CA to use SHA256 or SHA1 by creating a new CA certificate and server certificates is the way to go. In my case since this was a test lab and I didn’t want to encounter any more errors I went with SHA1.

I was mistaken however as I soon got the following error:

An SSL connection could not be established: An unclassified SSL error occurred.. This message was reported from the Citrix XML Service at address https://mydelivercontroller.mydomain/scripts/wpnbr.dll. The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services.

This one had me stumped for a long time. I know all my certificates were proper, and they were bound correctly to IIS, so what was this error about? Moreover it didn’t give much details, and there were not many forum or blog post hits either. Everything looked fine – so what the heck?! If I told the Storefront to communicate with the Delivery Controllers over HTTP instead of HTTPS, things worked. So clearly the problem was with HTTPS.

I was able to visit the XML Service URL too with no certificate errors.

Here’s an excellent post on the Citrix XML Service. The important thing to note is that if the IIS role is already present on the server when the Citrix XML Service is being installed, it integrates with IIS; whereas if the IIS role is not present the Citrix XML Service operates in standalone mode. During my install I didn’t have IIS, but since IIS got installed as part of the install I thought Citrix XML Service must be running integrated – but it does not. In my case Citrix XML Service is running standalone.

Anyways, not a good idea to integrate the Citrix XML Service with IIS, so I am going to leave mine standalone. Here’s a Citrix KB article on how to integrate though. Also, for my own info – the registry keys for the Citrix XML Service are under HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer. Apart from the default ones that are present one can also add two DWORD keys XmlServicesEnableNonSsl and XmlServicesEnableSsl to manipulate whether the Citrix XML Service accepts HTTP and HTTPS traffic. By default both keys are not present and have a value of 1, but changing these to 0 will disable HTTP or HTTPS.

Back to HTTPS and the Citrix XML Service. Since it is not integrated in my case, I should follow the SSL instructions for standalone mode. Roughly:

  1. Install the server certificate as usual.
  2. Note the certificate thumbprint.
  3. Find the Citrix Broker Service GUID. Do this via wmic product list (hat tip to this blog post for the latter idea; alternatively the Citrix article shows how to do this via registry).
  4. Use netsh to bind the two.

In my case the command would be something like this:

I didn’t need to do this as my correct certificate was already bound. It was bound to IIS  I guess (the appid wasn’t that of the Citrix Broker Service) so I double checked by removing binding and creating a new one specifically for the Citrix Broker Service. Still no luck!

If I were running Server 2016 there’s some additional steps to follow. But I am running Server 2012 R2.

My setup was such that I had two Delivery Controllers and one of them had the Storefront. It didn’t make a difference which Delivery Controller I chose to add to the Storefront – it never worked. At the same time, switching to HTTP instead of HTTPS always worked. I had no ideas. I posted to the Citrix forums too but only got one reply. Frustrating!

On a whim I installed Storefront on the second Delivery Controller server to see if that works. And it did. The Storefront on that server was able to talk to either Delivery Controllers with no issue. So the issue wasn’t with the Delivery Controllers. For some reason I had always thought the issue was with the Delivery Controllers (I guess because the error message was from the Citrix XML Server/ Citrix Broker Service and that is a part of the Delivery Controller) but now I realized it was to do with the Storefront. And specifically that particular server. I uninstalled and re-installed the Storefront but that didn’t make a difference.

My next suspect was certificates so I compared the trusted root CAs between the broken server and the working server. I found that the broken server had some of my older root CA certificates (remember I had switched my DC/ CA from SHA256 to SHA1) so maybe that was causing an issue? It also had an extra DigiCert certificate. I removed all these and tried again – and voila! it worked!

I am pretty sure I had manually removed all these older DC/ CA certs, so I am not entirely convinced that is the cause. But it sounds plausible and maybe they came back even though I removed.

Update: I hit upon this error again after I stupidly went and renewed my root CA cert (which is my Domain Controller). Stupid, coz I was doing it just for the heck of it (it’s my test lab after all!) but that broke the certs on the Delivery Controllers/ Store Fronts and I began getting these errors again. As a work around I went have deleted the new certs from the local stores of these (Trusted Root CA and also Intermediate CA) . Am sure it will sync in again, so long term I better regenerate my certs or just turn off SSL internally. Most likely the latter as I am lazy. :p

ODBC DSN creation for vCenter database

Was creating a DSN on my vCenter 6.5 to point to a SQL 2014 database and the following post was useful. Take special note of the path from where you can install the SQL driver for the DSN.

 

Unable to ping Nested VMs (XenServer/ VMware ESXi)

Spent the better part of two days chasing an issue only to find it was no issue at all. So irritated! Wasn’t a total waste of time as I got to read stuff, but it side tracked me from the main issue.

Here’s my setup. I have a Windows Server 2012R2 physical server. This runs VMware Workstation 12.5. Within it I have XenServer and VMware ESXi (the hypervisor isn’t relevant to the story but I mention it anyways). Within the hypervisor I have a Windows 8.1 VM – well two of them actually, but again it doesn’t matter much to the story.

Within VMware Workstation I have a couple of other servers too – a mix of Windows Server 2012 R2, 2016, and FreeBSD.

Let’s call the VMs within VMware Workstation as “VMs” while the VMs within the nested hypervisors as “VVMs”. The issue was that from the VVMs I was able to ping the VMs and get info from them (e.g. IP addresses) but I couldn’t ping the VVMs from the VMs. It didn’t matter which hypervisor the VVM was on. Also, the VVMs couldn’t ping each other.

There’s a lot of forum and blog posts on theis topic but their issue seems to be different. Their issue is that the VVMs are unable to see the outside world (i.e. the VMs). But my issue was that the VVMs could see the outside world; it was the outside world that couldn’t see them. All the forum and blog posts pointed to it being a case of the virtual switch not allowing promiscuous mode or forged MAC addresses, and the fix was to enable these. In my case I couldn’t find any such setting on VMware Workstation so I began suspecting it as the culprit.

Some good links I found while reading on these; putting them here as info for myself:

Oh, and if you are on a Linux host (where VMware Workstation is running) then you need to do some extra stuff to enable Promiscuous mode.

Nowhere could I find anything on what to do for VMware Workstation running on Windows and whether it had promiscuous mode enabled or not.

Finally I resorted to using tcpdump (on XenServer)/ tcpdump-uw (on ESXi) to see if the nested hypervisor is receiving the ICMP packets – it was. The ARP requests had the correct MAC addresses too. Next I installed Wireshark on a VM and VVM to see what was happening, and I could see that the VVM was receiving packets but not replying. So the switch in VMware Workstation was definitely in promiscous mode – the problem was in the VVM. I didn’t suspect a VVM firewall at all as I had disabled the Windows firewall service; but just for the heck of it I enabled the firewall service and simply turned off the firewall. And what do you know – suddenly the VVM is responding to ICMP packets!!

I have no idea why this is so. I had always thought disabling the firewall service is enough to … well, disable the firewall. But looks like actually disabling the firewall for each of the network profiles is the important thing. Weird.

Anyways – after two days of scratching my head I now have connectivity from my VMs to VVMs.

Creating an AD certificate for NetScaler 10.5

This post is based on a post by someone else that I found while I had to do this today. I wanted to configure NetScaler 10.5 with Citrix Storefront 3.9 and found that post useful, but some of the screenshots were different in my case – so thought I’d write it down for my future self. This post is going to be less on writing and more of screenshots as I am feeling very lazy.

So without much further ado –

Login to the NetScaler and create an RSA Key

1-2-3 as below.

Fill in the following fields and click “Create”.

The file name and extension doesn’t matter but we will refer to it later.

Create a Certificate Signing Request (CSR) on the NetScaler

Again, the request file name does not matter. The key filename & password is same as what we used earlier. There’s few more fields to fill – obvious ones like the organization name etc, the mandatory ones have an asterisk – then click “Create”.

Open the CSR

Click the link to view. Then click the link to “save text to a file”.

Login to your AD Certification Authority and submit the request

I am going to use the command line as the CSR doesn’t contain info on what template the CA should use, and that gives an error on the GUI: “0x80094801 – the request contains no certificate template information”.

Using the command line is simple. Open the command prompt and type the following:

This will prompt you for the location of the CSR and also the CA to use etc.

If you get any error about missing templates here, it’s possible you haven’t added the “Web Server” template to your CA templates. You can via this menu –

The command will also prompt for a location to save the generated certificate at. Save it someplace, then go back to the NetScaler.

Login to the NetScaler and install this certificate

Click the Install button as above. Then fill in the details as below. The certificate-key pair name does not matter. The certificate file name is chosen by clicking on “Browse”, then “Local”, and selecting the certificate file that you previously saved. The key file name and password are same as what you typed in the initial screenshot.

Finally, click “Install”.

That’s it! The NetScaler now has a certificate issued by the AD CA.

Reboot a bunch of ESXi hosts one after the other

Not a big deal, I know, but I felt like posting this. :)

Our HP Gen8 ESXi hosts were randomly crashing ever since we applied the latest ESXi 5.5 updates to them in December. Logged a call with HP and turns out until a proper fix is issued by VMware/ HPE we need to change a setting on all our hosts and reboot them. I didn’t want to do it manually, so I used PowerCLI to do it en masse.

Here’s the script I wrote to target Gen8 hosts and make the change:

I could have done the reboot along with this, but I didn’t want to. Instead I copy pasted the list of affected hosts into a text file (called ESXReboot.txt in the script below) and wrote another script to put them into maintenance mode and reboot one by one.

The screenshot output is slightly different from what you would get from the script as I modified it a bit since taking the screenshot. Functionality-wise there’s no change.