Subscribe via Email

Subscribe via RSS


Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan


Refresher to myself StoreFront and Delivery Controller authentication

In a previous post I had written about the flow of communication between Citrix Storefront and Delivery Controllers during user authentication. Here’s some more based on a Citrix blog post I am reading. 

Here’s what I had written in my previous post:

There’s a couple of steps that happens when a user logs in to access a Citrix solution. First: the StoreFront authenticates the user against AD. Or if the user is accessing remotely, the NetScaler gateway authenticates the user and passes on details to the StoreFront. Then the StoreFront passes on this information to the Delivery Controller so the latter can give a list of resources the user has access to. The Delivery Controllers in turn authenticate the user AD. The Delivery Controller then sends a list of resources the user has access to, to the StoreFront, which sends this on to the user’s Citrix Receiver or Browser. This is when the user sees what is available to them, and can select what they want.

When the user selects what they want, this is information is passed on to the StoreFront, which then passes the info to the Delivery Controller – who then finds an appropriate host that can fulfill the requirement and sends this information to the StoreFront. 

Emphasis mine. The Storefront communicates with the Delivery Controller using the XML Service. 

Here’s a list of authentication methods supported by the Storefront. 

When the Storefront communicates the user authentication information to the Delivery Controller, it may or may not include the password too (sent in clear-text) in this communication. If “User name and password” or “Pass-through from NetScaler” is selected, then the password is included. If “Domain pass-through” or “Smart card” is selected, then the password is not. The blog post doesn’t say anything about these, but I think “SAML Authentication” (used for ADFS) will not include the password, while “HTTP Basic” will. 

The StoreFront and Delivery Controller communicates twice (the two times I emphasized above). The first time is when the user authenticates and the StoreFront sends this information to the Delivery Controller to get a list of resources. The second time is when the user makes a selection and this information is passed on to the Delivery Controller so that an appropriate host can be selected. In both instances the password could be sent from the StoreFront to the Delivery Controller.

Notes on MCS disks

Primer 1. Primer 2. MCS Prep overview (good post, I don’t refer to all its points below). 

  • MCS creates a snapshot of the master VM you specify, but if you specify a snapshot it will not create another one. 
  • This snapshot is used to create to create a full clone. A full snapshot, so to say. 
    • This way the image used by the catalog is independent of the master VM. 
    • During the preparation of this full snapshot an “instruction disk” is attached to the VM that is temporarily created using the full snapshot. This disk enables DHCP on all interfaces of the full snapshot; does some KMS related tasks; and runs vDisk inventory collection if required.
  • This full snapshot is stored on each storage repository that is used by Desktop Studio. 
    • This full snapshot is shared by all VMs on that storage repository. 
  • Each storage repository will also have an identity disk (16 MB) per VM.
  • Each storage repository will also have a delta/ difference disk per VM.
    • This is thin provisioned if the storage supports it.
    • Can increase up to the maximum size of the VM.

Remember my previous post on the types:

  • Random.
    • Delta disk is deleted during reboot. 
  • Static + Save changes.
    • Changes are saved to a vDisk. 
    • Delta disk not used?
  • Static + Dedicated VM.
    • Delta disk is not deleted during reboot. 
    • Important to keep in mind: if the master image in the catalog is updated, existing VMs do not automatically start using it upon next reboot. Only newly created dedicated VMs use the new image. 
    • The delta disk is deleted when the master image is updated and existing VMs are made to use the new image (basically, new VMs are created and the delta disk starts from scratch; user customizations are lost). 
    • Better to use desktop management tools (of the OS) to keep dedicated VMs up to date coz of the above issue. 
  • Static + Discard changes.
    • Delta disk is deleted during reboot. 

A post on sealing the vDisk after changes. Didn’t realize there’s so many steps to be done. 

MCS choices (RAM cache & Disk cache)

Just a reminder to myself …

When creating a Desktop based Machine Catalog here are my choices:

If I choose Random then I get the option to allocate some of my RAM towards a cache, and also create a disk cache. RAM cache means all data is written to RAM first and then to disk as RAM fills up. And disk cache is like the Write Cache disk in PVS – you can specify a separate disk (maybe local to the host, or SSD storage) where data is written to.

Important to keep in mind here that the actual VM disk will not have any data written to it. All data writes either goes to the RAM cache or Disk cache. First RAM cache, then Disk cache. Both are optional; best to have both (or at least don’t do RAM cache only unless you have oodles or RAM!).

Read this post – it’s a good one. Also, check out the official post from Citrix introducing this feature in XenDesktop 7.9. MCS (Machine Creation Services) that makes use of RAM or Disk cache is known as MCSIO (Machine Creation Services Storage Optimization (beats me how that acronym works! :p)).

MCS VMs have two disks apart from the OS base disk – an identity disk and a delta disk. MCSIO VMs too have the identity disk and delta disk, but the delta disk is only used for maintenance tasks. Hence my comment above that when using either of these cache options, the size you allocate for these is your write cache/ delta disk. 

If I choose static I have three further options. 

If I go with static + save changes to a personal vDisk, I don’t get the option for cache disk etc. I can only choose my vDisk letter and size. 

 If I go with static + create a dedicated VM, again I don’t get any option for cache disk; I can only choose the copy mode (i.e. a linked clone or a full clone). 

If I go with static + discard all changes, then I get the option for cache disk and RAM allocation towards cache. Basically, static + discard is similar to random. You are not storing any changes, so it makes sense to use cache (RAM and/ or write cache). 

In the case of Server OS, I don’t have any choices (it’s always random) and I get the option for cache disk and RAM allocation.

MCSIO is only for non-persistent experiences. 

Notes to self on XenServer storage

Playing with XenServer in my testlab (basically as a VM in VMware Workstation hah!) I ran into trouble while creating a Machine Catalog via Citrix Desktop Studio. I forget the exact message but it was about lack of resources. I could see that in the create catalog process it was creating a snapshot and making a copy VM, powering it on and off successfully, and then it was failing. I kept an eye on my storage during this and saw that indeed it was exceeding the allocated space. I had thought it would do thin provisioning but in retrospect I realize XenServer never asked me about thick or thin when I added my iSCSI storage. Hmm.

Well turns out that for iSCSI XenServer has only thick provisioning. You get thin provisioning only if you are using the ext3 filesystem or NFS. Since iSCSI uses LVM, bummer! 

Here’s a forum post on how to identify if your SR is thick or not. 

Regarding thin provisioning – it is only for locally attached storage (which can use ext3) or NFS. Block attached storage is thick.

Before I realized all this I had spent some Googling on how to create a thin provisioned SR (Storage Repository). I felt that maybe it’s a GUI restriction and I can workaround by using the CLI. Turns out I was wrong. Here’s an article that explains SRs in XenServer anyway. It’s a good read. Here’s an article just on enabling thin provisioning for ext3 SRs via the CLI. 

While on the topic of storage, this is something I wanted to blog about earlier but never got around to. When using SMB/ CIFS shares, XenServer only supports NTLMv1. Here’s instructions on using NTLMv2

Also, smbclient is a good tool to test SMB connects from a XenServer. Example:

That seems to work, but I get a logon failure. This is because I didn’t put the username in quotes. 

That works!

I have no idea what the three commands below except that they are to do with mounting an SMB/ CIFS share on a XenServer permanently. I had noted these commands as part of my would be blog post, but it’s been a while now and I forget. Sometime when I get around to doing SMB3 or NTLMv2 with XenServer again I hope to refer to these again and better explain. I don’t want to spend too much time on XenServers now and get sidetracked …

After issue the above commands I think the shared folder is mounted only on one host in the pool. But right clicking on it and doing a repair will get it mounted on all hosts in the pool.

XenServer 7.0 and above support SMB for VM disk storage too. Prior versions support SMB only for ISO storage. 

Nested XenServer crashes when scrubbing memory

In case anyone else runs into this. I noticed that both XenServer 6.5 and 7.0 crash at the memory scrubbing stage during boot up when run as a VM within VMware Workstation (and possibly other virtualization products too – I didn’t try it with anything else). 

Am guessing the crash happens because the memory is not really available (this being a nested VM) and so the process crashes. Anyhoo, the workaround is to disable memory scrubbing. Check this blog post for instructions. 

In brief, the instructions are to add the option bootscrub=false to the boot options. This is via the file /boot/extlinux.conf in XenServer 6.5; or via /boot/grub/grub.cfg in XenServer 7.0.

Desktop VDA only listens on port 1494/2598 when the connection comes in

Was troubleshooting a Citrix issue (“Failed with status 1110”) and one of the possibilities was that something is blocking the VDA ports 1494/2598 (two other possibilities seem to be mismatched STAs or issues with the root CA certs – neither seems to be the problem in my case as only one user seems to affected) .

My first response was to fire up telnet and try connecting 1494/ 2598. That gave me mixed results until I realized that the VDA only starts listening on these ports when the user is going to connect to it. From CTX213761:

Windows 7 – Desktop OS will listen on Port 1494 only when request comes in from StoreFront or WebInterface.
netstat -ano on Windows 7 will not show 1494 | 2598 listening up until the time of ICA launch.
netstat -ano  on Windows 2012R2 – Server OS will be listening on Port 1494 | 2598 regardless.

 Worth keeping in mind. Two takeaways for me:

  1. This doesn’t affect Server OS (so XenApp is unaffected)
  2. So if VDA isn’t listening on port 1494/ 2598 that means it hasn’t received a request from StoreFront/ WebInterface – so there could be communication trouble between STF/ WI and VDA. 

For future reference:

Going through an earlier post of mine about the flow during a Citrix session (and also CTX128909 – good one by the way, it has a diagram too) I don’t see any step where the StoreFront/ WebInterface talks to the VDA. All the StoreFront communication is with the Delivery Controller or Receiver, so am guessing the VDA starts listening on ports 1494/2598 when the Delivery Controller selects a machine from its Delivery Group and informs the StoreFront/ WebInterface (so it can put this in the ICA file). At this point either the StoreFront or the Delivery Controller talks to the VDA – not sure which one. The troubleshooting flowchart in CTX136668 mentions that one must check whether the VDA and Controller are both listening on port 80 (as that’s the ports they use for talking to each other) so my bet is on the Delivery Controller. When the Delivery Controllers informs the VDA (via port 80) that it is selected, the VDA starts listening for Receiver connections on port 1494/ 2598.

Before I conclude – port 2598 is used for Session Reliability. If Session Reliability is enabled only port 2598 is used; else only port 1494 is used. It’s either, not both!

Brief notes on the Citrix STA

Wanted to point out this PDF from Citrix on the XenApp/ XenDesktop architecture – especially pages 21, 22 which are on how authentication works. During my Citrix course the instructor had talked about it but like an idiot I didn’t take notes and now I can’t find much info on what he was explaining. 

The part which is of interest to me is the STA (Secure Ticket Authority). 

There’s a couple of steps that happens when a user logs in to access a Citrix solution. First: the StoreFront authenticates the user against AD. Or if the user is accessing remotely, the NetScaler gateway authenticates the user and passes on details to the StoreFront. Then the StoreFront passes on this information to the Delivery Controller so the latter can give a list of resources the user has access to. The Delivery Controllers in turn authenticate the user AD. The Delivery Controller then sends a list of resources the user has access to, to the StoreFront, which sends this on to the user’s Citrix Receiver or Browser. This is when the user sees what is available to them, and can select what they want.

When the user selects what they want, this is information is passed on to the StoreFront, which then passes the info to the Delivery Controller – who then finds an appropriate host that can fulfill the requirement and sends this information to the StoreFront. 

The next step is where the STA comes in. 

In case the user is accessing Citrix locally, the StoreFront can create an ICA file with details of the host and send it over to the user’s Citrix Receiver or Browser and the latter can then directly talk to the VDA agent installed on the host (note the StoreFront & Delivery Controller have no more role to play). But what if the user is accessing remotely? We don’t want to send these sensitive details over the public Internet. So, as a workaround, Citrix creates a “ticket” (which is a randomly generated sequence of 32 uppercase alphabetic or numeric characters) and associates the ticket with the details of the host that the Citrix Receiver or Browser need to contact to access the requested resources. This ticket is what is sent over to Citrix Receiver or Browser in the ICA file, using which it can contact the NetScaler gateway and the NetScaler gateway can validate this and initiate a connection with the VDA on the host on behalf of the user. 

So, as we can see the STA only comes into play in case of remote access. The STA is a part of the Citrix XML Service (once again linking to this excellent post!), which is installed as part of the Delivery Controller (so the STA is a part of the Delivery Controller). It is written as an ISAPI extension (called CtxSta.dll) for the IIS WebServer and runs the /Scripts/CtxSta.dll URL. The STA has an ID called the STA_ID, and this along with the TICKET and an STA_VERSION field are what is put into the ICA file. I am not sure whether the STA requires IIS, or it can run standalone (as I blogged previously the Citrix XML Service can run standalone so I would assume the STA can do the same). The Citrix STA FAQ says IIS is required, but that could be outdated.

The Citrix StoreFront is configured with the STA details in the NetScaler Gateway section (remember you only need to use the STA in case of remote users, for which you would have to configure a NetScaler Gateway). 

Similarly the NetScaler itself is configured with the STA details. 

It is important to keep in mind that there are thus TWO places where the STA details are input, and that the details in both places must be the same. The StoreFront uses its configured details to generate a ticket and put it in the ICA file. And the StoreFront uses its configured details to validate that ticket with an STA and identify what resources it should connect to. If the two details are not identical then you will not be able to launch any resources! (I had this problem at work today which is why I decided to refresh my knowledge about STAs and thought of writing this blog post. If the two details are not identical you will get a “Cannot start App:” error because the ticket the client has cannot be validated or used by the NetScaler). 

Just as an aside to myself – the port used to talk to the VDA is 1494 or 2598. This is the case if the Citrix Receiver or Browser contacts the VDA, or if the NetScaler gateway does so on behalf of these. I like to remember port numbers. :o) 

Also – there is nothing that ties a particular STA generated ticket to the device where the request was made from. That is, in theory a remote user could make a request from Computer A, get the ICA file and run it on Computer B – and NetScaler + STA will happily let the user access resources. A ticket only has a 100 seconds validity, so they’d have to do this switch-over quickly though. ;o) Also, a ticket can only be used once. (Also this and more info are from the very informative Citrix STA FAQ by the way). 

Citrix XML Service headaches

Was setting up a Citrix XenDesktop environment in my test environment past few days and the Citrix XML service has been irritating me. There was no grand fix for the issue, but I spent quite a bit of time banging my head over it (and learnt some stuff along the way) so thought I’d make a post to put it all down.

Whenever I’d connect to the Storefront I get the following error:

I only get this error if you use the Receiver app by the way. If I try and connect via HTML5 you get no error at all. (So when in doubt, try with the Receiver always!)

I noticed that the “Citrix Delivery Service” event logs on the server had messages like these:

An SSL connection could not be established: You have not chosen to trust the issuer of the server’s security certificate, my-CA.. This message was reported from the Citrix XML Service at address https://mydeliverycontroller.mydomain/scripts/wpnbr.dll. The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services.

I sorted that by changing all my certificates to SHA1. Turns out the default certificate signature algorithm from a Windows CA since 2008R2 is RSASSA-PSS, and Citrix doesn’t support RSASSA-PSS, so switching the CA to use SHA256 or SHA1 by creating a new CA certificate and server certificates is the way to go. In my case since this was a test lab and I didn’t want to encounter any more errors I went with SHA1.

I was mistaken however as I soon got the following error:

An SSL connection could not be established: An unclassified SSL error occurred.. This message was reported from the Citrix XML Service at address https://mydelivercontroller.mydomain/scripts/wpnbr.dll. The specified Citrix XML Service could not be contacted and has been temporarily removed from the list of active services.

This one had me stumped for a long time. I know all my certificates were proper, and they were bound correctly to IIS, so what was this error about? Moreover it didn’t give much details, and there were not many forum or blog post hits either. Everything looked fine – so what the heck?! If I told the Storefront to communicate with the Delivery Controllers over HTTP instead of HTTPS, things worked. So clearly the problem was with HTTPS.

I was able to visit the XML Service URL too with no certificate errors.

Here’s an excellent post on the Citrix XML Service. The important thing to note is that if the IIS role is already present on the server when the Citrix XML Service is being installed, it integrates with IIS; whereas if the IIS role is not present the Citrix XML Service operates in standalone mode. During my install I didn’t have IIS, but since IIS got installed as part of the install I thought Citrix XML Service must be running integrated – but it does not. In my case Citrix XML Service is running standalone.

Anyways, not a good idea to integrate the Citrix XML Service with IIS, so I am going to leave mine standalone. Here’s a Citrix KB article on how to integrate though. Also, for my own info – the registry keys for the Citrix XML Service are under HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer. Apart from the default ones that are present one can also add two DWORD keys XmlServicesEnableNonSsl and XmlServicesEnableSsl to manipulate whether the Citrix XML Service accepts HTTP and HTTPS traffic. By default both keys are not present and have a value of 1, but changing these to 0 will disable HTTP or HTTPS.

Back to HTTPS and the Citrix XML Service. Since it is not integrated in my case, I should follow the SSL instructions for standalone mode. Roughly:

  1. Install the server certificate as usual.
  2. Note the certificate thumbprint.
  3. Find the Citrix Broker Service GUID. Do this via wmic product list (hat tip to this blog post for the latter idea; alternatively the Citrix article shows how to do this via registry).
  4. Use netsh to bind the two.

In my case the command would be something like this:

I didn’t need to do this as my correct certificate was already bound. It was bound to IIS  I guess (the appid wasn’t that of the Citrix Broker Service) so I double checked by removing binding and creating a new one specifically for the Citrix Broker Service. Still no luck!

If I were running Server 2016 there’s some additional steps to follow. But I am running Server 2012 R2.

My setup was such that I had two Delivery Controllers and one of them had the Storefront. It didn’t make a difference which Delivery Controller I chose to add to the Storefront – it never worked. At the same time, switching to HTTP instead of HTTPS always worked. I had no ideas. I posted to the Citrix forums too but only got one reply. Frustrating!

On a whim I installed Storefront on the second Delivery Controller server to see if that works. And it did. The Storefront on that server was able to talk to either Delivery Controllers with no issue. So the issue wasn’t with the Delivery Controllers. For some reason I had always thought the issue was with the Delivery Controllers (I guess because the error message was from the Citrix XML Server/ Citrix Broker Service and that is a part of the Delivery Controller) but now I realized it was to do with the Storefront. And specifically that particular server. I uninstalled and re-installed the Storefront but that didn’t make a difference.

My next suspect was certificates so I compared the trusted root CAs between the broken server and the working server. I found that the broken server had some of my older root CA certificates (remember I had switched my DC/ CA from SHA256 to SHA1) so maybe that was causing an issue? It also had an extra DigiCert certificate. I removed all these and tried again – and voila! it worked!

I am pretty sure I had manually removed all these older DC/ CA certs, so I am not entirely convinced that is the cause. But it sounds plausible and maybe they came back even though I removed.

Creating an AD certificate for NetScaler 10.5

This post is based on a post by someone else that I found while I had to do this today. I wanted to configure NetScaler 10.5 with Citrix Storefront 3.9 and found that post useful, but some of the screenshots were different in my case – so thought I’d write it down for my future self. This post is going to be less on writing and more of screenshots as I am feeling very lazy.

So without much further ado –

Login to the NetScaler and create an RSA Key

1-2-3 as below.

Fill in the following fields and click “Create”.

The file name and extension doesn’t matter but we will refer to it later.

Create a Certificate Signing Request (CSR) on the NetScaler

Again, the request file name does not matter. The key filename & password is same as what we used earlier. There’s few more fields to fill – obvious ones like the organization name etc, the mandatory ones have an asterisk – then click “Create”.

Open the CSR

Click the link to view. Then click the link to “save text to a file”.

Login to your AD Certification Authority and submit the request

I am going to use the command line as the CSR doesn’t contain info on what template the CA should use, and that gives an error on the GUI: “0x80094801 – the request contains no certificate template information”.

Using the command line is simple. Open the command prompt and type the following:

This will prompt you for the location of the CSR and also the CA to use etc.

If you get any error about missing templates here, it’s possible you haven’t added the “Web Server” template to your CA templates. You can via this menu –

The command will also prompt for a location to save the generated certificate at. Save it someplace, then go back to the NetScaler.

Login to the NetScaler and install this certificate

Click the Install button as above. Then fill in the details as below. The certificate-key pair name does not matter. The certificate file name is chosen by clicking on “Browse”, then “Local”, and selecting the certificate file that you previously saved. The key file name and password are same as what you typed in the initial screenshot.

Finally, click “Install”.

That’s it! The NetScaler now has a certificate issued by the AD CA.