Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

PVS Steps

I need a central place/ post where I can write down (and keep adding) the steps required for a PVS image. Consider this as a continuation to a previous post. So here goes:

  1. Install the OS.
  2. Install Updates and Device Drivers; including integration tools such as VMware Tools, XenServer Tools, etc. 
  3. Install any applications & VDA. (No shutdown or domain join at this point – but you could domain join if needed, just don’t reuse that name later in the catalog)
  4. Install the Target Device software (this is the Provisioning Services target device software).
  5. Launch the imaging wizard. (No need for snapshots as we are capturing to a new vDisk)
    1. The wizard will ask for a Target Device name. This can be different from the name of the machine/ VM. This is a PVS representation of the device.
    2. Give a vDisk name too.
    3. Upon creation this vDisk will be in private mode – which means only 1 machine can boot off it, and the disk is read/write.
  6. Power off the VM.
    1. If this VM will be used as the Master Target Device going forward, delete or remove the original hard disk. And go to the device object in PVS and change the “Boot from” to vDisk.
    2. Alternatively: If this VM will not be used as the Master Target Device, create a new VM (maybe give it the same name as the Master Target Device in PVS) and put its MAC address to the Master Target Device in PVS. Also change the “Boot from” to vDisk.
    3. Note: If the “Boot from” is not changed to vDisk the VM will try to boot from the local disk and fail if it’s not present.
  7. Power on the VM. It will boot from the vDisk now (which is still in private mode). Make any more changes if required. Example domain join and add more applications etc. Whatever changes we make now will be written to the vDisk.
  8. When everything is finalized power off the VM.
  9. Convert the vDisk to standard mode. This means multiple machines can boot off it and the disk is read only (the writes will be made to a write-cache disk).
    1. Note: the VM must be powered off to be able to change the disk type. There has to be no connections to the vDisk.
  10. Now create a template based on the hardware specs you want for your VMs. Memory, CPU, etc. This template won’t have any storage. It will network boot.
  11. From PVS console launch the XenDesktop Setup wizard. At one point it will ask for the template & vDisk we created above.
    1. Quick note on the disk options.
    2. With MCS we had the option of choosing random or static. Within random we had the option of using RAM as a cache. And within static we had the option of (a) using a personal vDisk or (b) a dedicated VM or (c) discarding changes.
    3. With PVS our options are again random or static. No sub-options within random (i.e. no RAM cache etc). And within static the only options are (a) personal vDisk or (b) discard changes.
  12. That’s all. New VMs will be created that can stream from the above vDisk. And AD accounts will be created for them.

Pool Machine booting up … :)

Migrating VMkernel port from Standard to Distributed Switch fails

I am putting a link to the official VMware documentation on this as I Googled it just to confirm to myself I am not doing anything wrong! What I need to do is migrate the physical NICs and Management/ VM Network VMkernel NIC from a standard switch to a distributed switch. Process is simple and straight-forward, and one that I have done numerous times; yet it fails for me now!

Here’s a copy paste from the documentation:

  1. Navigate to Home > Inventory > Networking.
  2. Right-click the dVswitch.
  3. If the host is already added to the dVswitch, click Manage Hosts, else Click Add Host.
  4. Select the host(s), click Next.
  5. Select the physical adapters ( vmnic) to use for the vmkernel, click Next.
  6. Select the Virtual adapter ( vmk) to migrate and click Destination port group field. For each adapter, select the correct port group from dropdown, Click Next.
  7. Click Next to omit virtual machine networking migration.
  8. Click Finish after reviewing the new vmkernel and Uplink assignment.
  9. The wizard and the job completes moving both the vmk interface and the vmnic to the dVswitch.

Basically add physical NICs to the distributed switch & migrate vmk NICs as part of the process. For good measure I usually migrate only one physical NIC from the standard switch to the distributed switch, and then separately migrate the vmk NICs. 

Here’s what happens when I am doing the above now. (Note: now. I never had an issue with this earlier. Am guessing it must be some bug in a newer 5.5 update, or something’s wrong in the underlying network at my firm. I don’t think it’s the networking coz I got my network admins to take a look, and I tested that all NICs on the host have connectivity to the outside world (did this by making each NIC the active one and disabling the others)). 

First it’s stuck in progress:

And then vCenter cannot see the host any more:

Oddly I can still ping the host on the vmk NIC IP address. However I can’t SSH into it, so the Management bits are what seem to be down. The host has connectivity to the outside world because it passes the Management network tests from DCUI (which I can connect to via iLO). I restarted the Management agents too, but nope – cannot SSH or get vCenter to see the host. Something in the migration step breaks things. Only solution is to reboot and then vCenter can see the host.

Here’s what I did to workaround anyways. 

First I moved one physical NIC to the distributed switch.

Then I created a new management portgroup and VMkernel NIC on that for management traffic. Assigned it a temporary IP.

Next I opened a console to the host. Here’s the current config on the host:

The interface vmk0 (or its IPv4 address rather) is what I wanted to migrate. The interface vmk4 is what I created temporarily. 

I now removed the IPv4 address of the existing vmk NIC and assigned that to the new one. Also, confirmed the changes just to be sure. As soon as I did so vCenter picked up the changes. I then tried to move the remaining physical NIC over to the distributed switch, but that failed. Gave an error that the existing connection was forcibly closed by the host. So I rebooted the host. Post-reboot I found that the host now thought it had no IP, even though it was responding to the old IP via the new vmk. So this approach was a no-go (but still leaving it here as a reminder to myself that this does not work)

I now migrated vmk0 from the standard switch to the distributed switch. As before, this will fail – vCenter will lose connectivity to the ESX host. But that’s why I have a console open. As expected the output of esxcli network ip interface list shows me that vmk0 hasn’t moved to the distributed switch:

So now I go ahead and remove the IPv4 address of vmk0 and assign that to vmk4 (the new one). Also confirmed the changes. 

Next I rebooted the host, and via the CLI I removed vmk0 (for some reason the GUI showed both vmk0 and vmk4 with the same IP I assigned above). 

Reboot again!

Post-reboot I can go back to the GUI and move the remaining physical NIC over to the distributed switch. :) Yay!

Certificates, Subject Alternative Names, etc.

I had encountered this in my testlab but never bothered much coz it was just my testlab after all. But now I am dabbling with certificates at work and hit upon the same issue. 

The issue is that if I create a certificate for mymachine.fqdn but I visit the machine at just mymachine, then I get an error. So how can I tell the certificate that the shorter name (and any other aliases I may have) are also valid? Turns out you need to use the Subject Alternative Name (SAN) field for that!

You can’t add a SAN field to an existing certificate. Got to create a new one. In my case I had simply requested a domain certificate from my IIS server and that doesn’t give any option to specify the SAN.

Instructions for creating a new certificate with SAN field are here and here. The latter has screenshots, so check that out first. In my case, at the step where I select “Web Server” I wasn’t getting “Web Server” as an option. I was only getting “Computer”. Looking into this, I realized it’s coz of the permissions difference. The “Web Server” template only has Domain Admins and Enterprise Admins in its ACLs, while the “Computer” template had Domain Computers too with “Enrol” rights. The fix is simple – go the Manage Templates and change the ACL of “Web Server” accordingly. (You could also use ADSI Edit and edit the ACL in the Configuration section). 

[Aside] NetScaler – CLI Networking

Just putting these two here as a reference to myself (no idea why coz I am sure I’ll just Google and find them later when I need to :p)

As an aside (to this aside):

  • The NetScaler config is stored as ns.conf at /nsconfig
  • Older versions have a .0, .1, .2, etc suffixed to the filename. 
  • Backups are stored in /var/ns_sys_backup.
  • More info on backups etc

[Aside] Useful CA/ Certificates info

Notes on Master Image Preparation (PVS & MCS)

Reading some links on creating a Master Image; here’s notes to myself regarding that. These are the links I refer to:

Note these are very rough/ brief. These really are rough notes to myself as I am trying to make organize my mind. 

In case of PVS: Master Target Device – this is the VM whose image is used  to create a virtual hard disk (vDisk). This vDisk is what PVS uses to stream to its VMs.

Unlike MCS, the Master Target Device does not have to be a VM. PVS works against both VMs and physical machines. It does not care about the compute; all it does is look at the machine and create a vDisk by capturing its contents. You network boot into the machine you want to capture, and PVS creates an image by streaming its contents to the PVS server to create a vDisk. 

The Master Target Device or its disk can be removed after vDisk creation.

Here’s my understanding of the order in which to do stuff:

1. Install the OS

2. Install Updates and Device Drivers; including integration tools such as VMware Tools, XenServer Tools, etc. 

In case of MCS:

3. Install any applications (optional) & VDA & domain join (optional) & shutdown the machine. 

4. Add it to MCS to create a catalog. Recommended that we take a snapshot and point MCS to this snapshot. Else MCS will make its own snapshot (and we can’t change the snapshot name). 

In the case of PVS:

3. Install any applications & VDA. (No shutdown!) (No domain join!)

4. Install the Target Device software (this is the Provisioning Services target device software).

5. Launch the imaging wizard. (No need for snapshots either as we are capturing it to a new vDisk). 

5a. Note: The Target Device name can be different from the name of the machine/ VM. 

6. After the vDisk is created we use a VM (existing or new one) to network boot and use this vDisk. Then we can domain join etc. PVS has a lot more steps. Check out http://www.carlstalhood.com/pvs-master-device-convert-to-vdisk/

Update: I have a post that goes into more detail with PVS.

Components of the Provisioning Services target device software include an imaging wizard to capture the image, a NIC filter driver used for streaming images from the PVS server to the target devices (remember: the target device software is not used only for capturing the image – i.e. the master target device, but also by the target devices after an OS is loaded), a virtual disk to store the OS and applications (again, used by the target devices). There’s also a system tray utility. 

Notes on ADFS Certificates

Was trying to wrap my head around ADFS and Certificates today morning. Before I close all my links etc I thought I should make a note of them here. Whatever I say below is more or less based on these links (plus my understanding):

There are three types of certificates in ADFS. 

The “Service communications” certificate is also referred to as “SSL certification” or “Server Authentication Certificate”. This is the certificate of the ADFS server/ service itself. 

  • If there’s a farm of ADFS servers, each must have the same certificate
  • We have the private key too for this certificate and can export it if this needs to be added to other ADFS servers in the farm. 
  • The Subject Name must contain the federation service name. 
  • This is the certificate that end users will encounter when they are redirected to the ADFS page to sign-on, so this must be a public CA issued certificate. 

The “Token-signing” certificate is the crucial one

  • This is the certificate used by the ADFS server to sign SAML tokens.
  • We have the private key too this certificate too but it cannot be exported. There’s no option in the GUI to export the private key. What we can do is export the public key/ certificate. 
  • The exported public certificate can be loaded to the 3rd party provider who would be using our ADFS server for authentication.
  • The ADFS server signs tokens using this certificate (i.e. uses its private key to encrypt the token or a hash of the token – am not sure). The 3rd party using the ADFS server for authentication can verify the signature via the public certificate (i.e. decrypt the token or its hash using the public key and thus verify that it was signed by the ADFS server). This doesn’t provide any protection against anyone viewing the SAML tokens (as it can be decrypted with the public key) but does provide protection against any tampering (and verifies that the ADFS server has signed it). 
  • This can be a self-signed certificate. 
    • By default this certificate expires 1 year after the ADFS server was setup. A new certificate will be automatically generated 20 days prior to expiration. This new certificate will be promoted to primary 15 days prior to expiration. 
    • This auto-renewal can be disabled. PowerShell cmdlet (Get-AdfsProperties).AutoCertificateRollover tells you current setting. 
    • The lifetime of the certificate can be changed to 10 years to avoid this yearly renewal. 
    • No need to worry about this on the WAP server. 

The “Token-decrypting” certificate.

  • This one’s a bit confusing to me. Mainly coz I haven’t used it in practice I think. 
  • This is the certificate used if a 3rd party wants to send us encrypted SAML tokens. 
    • Take note: 1) Sending us. 2) Not signed; encrypted
  • As with “Token-signing” we export the public part of this certificate, upload it with the 3rd party, and they can use that to encrypt and send us tokens. Since only we have the private key, no one can decrypt en route. 
  • This can be a self-signed certificate. 
    • Same renewal rules etc. as the “Token-signing” certificate. 
  • Important thing to remember (as it confused me until I got my head around it): This is not the certificate used by the 3rd party to send us signed SAML tokens. The certificate for that can be found in the signature tab of that 3rd party’s relaying party trust (see screenshot below). 
  • Another thing to take note of: A different certificate is used if we want to send encrypted info to the 3rd party. This is in the encryption tab of the same screenshot. 

So – just to put it all in a table.

Clients accessing ADFS server Service communications certificate
ADFS server signing data and sending to 3rd party Token-signing certificate
3rd party encrypting data and sending ADFS server Token-decrypting certificate
3rd party signing data and sending ADFS server Certificate in Signature tab of Trust
ADFS server encrypting data and sending to 3rd party Certificate in Encryption tab of Trust

[Aside] NetScaler VPX Express limitations etc.

Reading about NetScaler VPX as we are looking at implementing VPX Express in our site as part of a POC. 

  1. VPX Express is limited to 5  Mbps (as opposed to say 1 Gbps for VPX-1000). 
  2. The license is free but you have to keep renewing annually.
  3. The Edition is NetScaler Standard. This and this are two links I found that explain the difference between various editions. 
    1. tl; dr version: Standard is fine for most uses. 
  4. The Gateway part supports 5 concurrent user connections. 
  5. You cannot vMotion or XenMotion VPX. 
  6. This is an excellent blog post on VPX, MPX, and others. Worth a read. 

[Aside] How to Secure an ARM-based Windows Virtual Machine RDP access in Azure

Just putting this here as a bookmark to myself for later. A good post. 

Changing Delivery Controller addresses on VDA

This weekend the NetScaler gateway on one of our offices failed, rendering the office users unable to connect to their resources. I wanted to repoint all those users to the Delivery Controllers of a different office where the NetScaler was working, and ask users to temporarily connect via that. 

I had to two two things basically. 1) Change a registry key to repoint the VDA to the other Delivery Controllers. 2) Restart the VDA service (these machines were running 5.6(!!) so it’s called the Workstation Agent). 

Brief notes on NetScaler and Citrix StoreFront

I spent the last two days intermittently trying to set up NetScaler and Citrix StoreFront in my home lab. It was a mixed bag partly due to my nature of just jumping into something and figuring it out as I go along :) but compounded by the fact that while there’s a lot of documentation on the Internet they seem to be either outdated or don’t explain the big picture of what we are trying to achieve. (Or maybe I am just slow in picking it up – wouldn’t put that possibility aside!)

Anyways. Here’s a couple of stuff in no particular order. 

This PDF is the official documentation on setting up NetScaler with Citrix StoreFront. It’s a good one – lots of screenshots etc. Page 19 onwards seems to be outdated though with the latest version of NetScaler that I have – 11.1 53.11. 

When I started this exercise though I was on a much older version of NetScaler – 10.5 56.22. I started setting up the NetScaler as a gateway (e.g. http://ctxstorefront.myfqdn) to my internal StoreFront servers (http://data01.myfqdn and http://data02.myfqdn), and my steps were more or less along the lines of the PDF above which I discovered later. That wasn’t a success though coz when I’d connect to the NetScaler gateway it gave some errors (I forget what now). Digging into this I realized that the wizard also creates a load balanced virtual server on the NetScaler and that was showing as down. Dug into that a bit and found out that the underlying monitor probes to the two StoreFronts were failing. If I separately created services representing my StoreFronts and attach the same monitor (basically, a monitor of type STOREFRONT, with the correct StoreName etc) it fails. 

That was good to know coz I learnt a bit about the monitor probes. :) I found the following in my NetScaler logs:

That’s same as what the GUI was telling me but I was additionally able to find what error code was being generated (thanks to this KB article):

So it runs a Perl script basically. Nice. Went through that script and here’s the relevant section:

So it tries to access http(s)://<my delivery controller>/Citrix/<my Store>/discovery and gets a 200 error. Odd that it gets it, because if I try to probe that URL via cURL from NetScaler, I get a 404 error (which is sort of correct; I get a 403 error via IE and that’s the correct one I think):

Anyhoo, that stumped me for a while until I found forum/ blog post where they said upgrading to a newer version of NetScaler supposedly fixes this. So I went down that route, and yes, it helped. 

After upgrading though my UI changed. :)

The new UI is very different. But that’s good I think, coz it forced me think at least on what exactly am I trying to achieve here; and to take a step backward and understand what is going on. So here’s my understanding:

  • We have StoreFronts. :) That’s the thing you actually connect to. 
  • We can have a group of StoreFronts for High Availability. You can configure each of these independently or you can keep them in sync from the UI itself. 
  • For each StoreFront we need to specify a base URL.
    • For a single StoreFront it’s easy – http://data01 or whatever (where data01 is my StoreFront server name in this case).
    • But what about when I have a group of StoreFronts? If I am keeping them in sync via the UI itself, the base URL I define on one of them will be pushed out to all. So all my StoreFronts will have a base URL of http://data01 even though they might be called data01, data02, etc. That’s a problem coz all my clients will only connect to the first one as that’s where DNS will point clients to.
  • To avoid the above situation for multiple StoreFronts we need a common base URL which can be load balanced across all. A regular DNS round-robin situation won’t work coz we also need clients to stick on with whichever StoreFront they connect to, and also some form of monitoring to ignore StoreFronts that are down would be good to. So this is where the NetScaler first comes in!
  • Create a Virtual Server on the NetScaler that will load balance the various StoreFront services we define on it. Simple stuff – just HTTP or HTTPS services that are load balanced. Add the STOREFRONT type monitor. And a Persistence of type COOKIEINSERT. That’s all. (Oh, and add SSL certs etc if you are using HTTPS). 
    • I haven’t gone through this link but am putting it here as a reference to my future self – pretty sure I must have missed something when setting up things now. Also, that link goes into some scenarios such as where we do SSL termination at the NetScalers. 

What I realized as I thought about this is that this Virtual Server I create above is the main thing. Going by my URLs above, I should have http(s)://ctxstorefront.myqdn load balance among http(s)://data01 and http(s)://data02. Forget about the whole external access stuff for now. 

Once I do this and ensure things are working (they do), now I can think about external access. What does the NetScaler need to do there? It doesn’t need to do any load balancing coz that’s already setup; so all it needs to do is provide a VPN gateway sort of service for external clients to connect to! So that means a new Virtual IP on the NetScaler. Create this using the wizard in the “Integrate with Citrix Products” > “XenApp and XenDesktop”. This will basically create a Virtual Server in the NetScaler Gateway section (note: not in the Load Balancing section). As part of configuring, we have to point this Virtual Server to a StoreFront. Here use the Load Balanced StoreFront Virtual Server that we created on the NetScaler above – this is where it all ties in! 

I could use the same internal URL for the external access too I guess and use split DNS – not sure, because I do have to specify this on the StoreFronts and I haven’t tried/ thought of any side effects – but in my case I simply decided to go with a different URL for the external access. Specify that for the certificate and Virtual Server etc, add that to DNS, and now I can access the StoreFronts externally too via the NetScaler. 

If I were to go to the Virtual Server in the Gateway section there’s no obvious mapping from this one to the internal load balance Virtual Server. But there is. It’s in the policies section. That has a Published Application pointing to the load balanced URL (including the Store name (actually, there’s two policies – one for the Web another for the Receiver, to capture the different names)) so that’s how the traffic flow works. Users hit the gateway, the gateway does the authentication etc and checks its policies, finds one that offers uses the StoreFront published application at such and such URL, and it looks that up (it is with itself) and thus hits the load balanced Virtual Server, and so on … 

Finally! I have some idea of how this all ties in together. :)

Scanning for MS17-010

Was reading about the WannaCrypt attacks. If you have the MS17-010 bulletin patches installed in your estate, you are safe. I wanted to quickly scan our estate to see if the servers are patches with this. Not my job really, but I wanted to do it anyways. 

The security bulletin page lists the actual patch numbers for each version of Windows. We only have Server 2008 – 2016 so that’s all I was interested in. 

Here’s a list of the Server name, internal version, and the patch they should have.

  • Server 2008 | 6.0.6002 | KB4012598
  • Server 2008 R2 | 6.1.7600 | KB4012215 or KB4012212
  • Server 2012 | 6.2.9200 | KB4012214 or KB4012217
  • Server 2012 R2 | 6.3.9600 | KB4012213 or KB4012216
  • Server 2016 | 10.0.14393 | KB4013429

One thing to bear in mind is that it’s possible a server doesn’t have the exact patch installed, but is still not at any risk. That is because since October 2016 Windows patches are cumulative. So if you don’t have the particular March 2017 patch installed, but do have the April 2017 one, you are good to go. The numbers above are from March 2017 – so you will have to update them with patch numbers of subsequent months too to be thorough. 

Another thing – I had one server in my entire estate where the patch above was actually installed but turned up as a false positive in my script. Not sure why. I know it isn’t a script issue. For some reason that patch wasn’t being returned as part of the “Win32_QuickFixEngineering” output. Am assuming it wasn’t installed that way on this particular server.

Without further ado, here’s the script I wrote:

That’s all. Nothing fancy. 

PVS Console XenDesktop Setup Wizard crash and timeout

If you get the following error when running the “XenDesktop Setup Wizard” from PVS console (and a catalog is created with no machines):

Or you get the following error (and again a catalog is created but no machines):

Or you get neither of these but the PVS Console simply freezes when you click OK/ Next at one of the steps and it throws an error about timeout and being unable to connect to the remote server (sorry no screenshot, forgot to take it when I got the error and now I can’t replicate it) …

The solution is simple! Install the PVS Console on your Delivery Controller server and run the wizard from there. For some reason that seems to do the trick. Thanks to this forum post

Quickly move DHCP scopes between servers

Wish I had known this earlier. If you want to migrate DHCP servers and have got the two servers setup, run the following on the source server:

And the following on the destination server (the first command is to just connect to the C: drive of the source):

Amazing!

This only does IPv4 scopes, so replace v4 with v6 for IPv6. And this exports all scopes, so replace “all” with the scopes you want if you want a selected few. 

Get-WindowsUpdateLog SymSrv.dll error

Starting with Windows 10 and Server 2016 Microsoft isn’t providing a WindowsUpdate.log file any more. Instead we have to run a cmdlet to generate it manually.

On my Server 2016 the cmdlet gave the following error:

I checked the path “C:\Program Files\Windows Defender” and there’s no “SymSrv.dll” file present there. That’s because I had removed the Windows Defender feature from my Server 2016 install (we use Sophos, so no need really for Windows Defender). Who thought removing Windows Defender would break this cmdlet?

If I check the module “C:\Windows\system32\WindowsPowerShell\v1.0\Modules\WindowsUpdate\WindowsUpdateLog.psm1” sure enough it looks for this DLL and tries to copy it:

Bugger!

Well, workaround is simple. Copy this DLL from any other 2016 machine (or search through the “C:\Windows\WinSxS\” folder on the same machine and copy it from there) to “C:\Program Files\Windows Defender” and the cmdlet will work. 

Or … enable the Windows Defender feature and disable it from Settings.