 © Rakhesh Sasidharan
|
Posted: August 5th, 2017 | Tags: ica, netscaler | Category: Citrix | § Wanted to find out if a certain end-user had connected to our NetScaler gateway. Couldn’t figure out how. (And initially I went the long route of looking at the /tmp/aaadebug.log file – not really needed here!)
It’s easy. Login to the NetScaler device. Click on “NetScaler Gateway” in left pane. On the right you will find “Active user sessions” and “ICA Connections”. The former shows users who have authenticated against the gateway, and the latter is those who have an ICA connection open through the gateway. The lists could be different as a user might have timed out on the gateway but still have an ICA connection open.
Via CLI the former is show aaa session. The latter is show vpn icaConnection. The latter will show connects to the VDA (port 2598 usually).
Posted: August 2nd, 2017 | Tags: dhcp, windows | Category: Windows | § This problem ate my head for the past 2 days and wasted a lot of time. For such a simple issue it drove me quite mad.
Built a bunch of DCs for our branch offices. One of them gave trouble with the DHCP server. I authorized it successfully, but the service kept complaining that it wasn’t authorized. Event ID 1046.
The DHCP/BINL service on the local machine, belonging to the Windows Administrative domain mydomain.dom, has determined that it is not authorized to start. It has stopped servicing clients. The following are some possible reasons for this:
This machine is part of a directory service enterprise and is not authorized in the same domain. (See help on the DHCP Service Management Tool for additional information).
This machine cannot reach its directory service enterprise and it has encountered another DHCP service on the network belonging to a directory service enterprise on which the local machine is not authorized.
Some unexpected network error occurred.
Did the obvious ones like reboot server :p and restart service :) and un-authorize and re-authorize the server (no errors either time). Also went ahead and removed the role itself and added back. Nothing helped!
Found a helpful post finally that pointed me in the right direction.
- I un-authorized the DHCP server.
- Opened up AD Sites and Services.
- Browsed to the Services section (which can be enabled from the View menu if not already visible).
- Browsed to the NetServices section within this.
- On the right pane I had an entry for the IP address for the DHCP server I was trying to authorize. Not an entry by name, but by IP. Dunno why. (All other entries were by name, so I am guessing this is a leftover or a mistake by someone in the past).
- I deleted this entry.
- Waited a while, and then authorized the server.
- No errors now!
Screenshot of the offending entry just for the heck of it (the blacked out part was an IP address):
Alternatively one can open ADSI Edit and go to CN=NetServices,CN=Services,CN=Configuration,DC=myDomain,DC=dom. Then delete the entry (as above) from there.
What’s odd in my case is that the IP that I deleted was assigned to the DHCP server I wanted to authorize. Am guessing the CNF (short for conflict?) following by the GUID indicates some issue.
Posted: August 1st, 2017 | Tags: netscaler, receiver, studio, vpx, xenapp | Category: Citrix | § Busy with a lot of Citrix and NetScaler work recently. Want to put the various links I came across someplace.
Was also looking into why Citrix Receiver wasn’t creating shortcuts for our published apps even though we had ticked it in in Citrix Studio. The trick was to mark the application as a Favorite in Receiver and the shortcut would be created. Here’s some links I had found while reading on this:
Posted: August 1st, 2017 | Tags: bash, esxcli, loop, unmap | Category: Virtualization | § It’s a good idea to periodically run the UNMAP command on all your thin-provisioned LUNs. This allows the storage system to reclaim deleted blocks. (What is SCSI UNMAP?)
The format of the command is:
|
|
esxcli storage vmfs unmap -l <volume_label> |
I wanted to make a script to run this on all attached datastores so here’s what I came up with:
|
|
#!/bin/sh for datastore in `esxcli storage filesystem list | grep "VMDATA" | awk -F " " '{print $2}'`; do echo "Performing UNMAP on $datastore ..." esxcli storage vmfs unmap -l $datastore done |
The esxcli storage filesystem list command outputs a list of datastores attached to the system. The second column is what I am interested in, so that’s what awk takes care for me. I don’t want to target any local datastores, so I use grep to filter out the ones I am interested in.
Next step would be to add this to a cron job. Got to follow the instructions here, it looks like.
Posted: July 23rd, 2017 | Tags: groups, whoami | Category: Windows | § I knew of doing a gpresult /v and finding the group membership. An even better (and faster) way is whoami /groups.
Other useful whoami switches.
Posted: July 20th, 2017 | Tags: ADFS, netscaler | Category: Asides, Infrastructure | §
Update: To test ADFS as an end-user, go to https://<adfsfqdn>/adfs/ls/IdpInitiatedSignon.aspx. Should get a page where you can sign in and select what trusts are present.
Posted: July 15th, 2017 | Tags: bored, purpose | Category: Musings | § Watching “Cosmos: A Space Time Odyssey” nowadays.
Also completed “The Leftovers” Season 1 yesterday. Great show, especially the last few episodes where there’s a lot of talk about purpose and such – which is still in my head.
Now sitting at a restaurant, bored, eating chicken kababs. Thinking: life is so easy for humans now. Ages ago we had to hunt for food. Each day was an unknown – whether we would get something or not. Life itself was unsure. Survival during hunting for instance. But now – here I am, a chubby spectacled nerd who probably wouldn’t have survived at all during the hunter days, sitting here eating a piece of chicken with a fork. How things have changed.
I guess that’s why we have no sense of purpose now. Back then we had a purpose – survival; fight against nature. Now we have troubles but most of it isn’t of the survival kind. So there’s no sense of purpose – there’s an emptiness. Nothing to do. Work is what we try to find purpose in. But that’s not really purpose. Mostly it’s a means to earn money. And it’s filled with politics and whatnot. It’s not purely about “your” purpose – it’s about the company and the people in it etc.
Ok time to go, cutting this short! :)
Posted: July 13th, 2017 | Tags: active directory | Category: Asides, Windows | §
- How Domain Controllers are Located Across Trusts – this is a delightful article. I don’t know why, but I simply loved the way the author presented the information. Very logically written. Wish I could write blog posts with such clarity.
- Praise aside, it is a good article on how subnet and site definitions are used to find a Domain Controller closest to you, and especially how it works across forest trusts.
- Using Catch-All subnets in AD – Wanted to know how catch-all subnets in AD Sites will interact with specific ones. This one explained it. The specific one takes precedence. Which is exactly what you want. :)
Posted: July 10th, 2017 | Tags: netscaler, nstrace | Category: Citrix | § It is not possible to do a telnet from the NetScaler to any server to troubleshoot connectivity issues. The telnet may or may not succeed, but it doesn’t mean anything as the telnet is initiated from the NSIP where all NetScaler communications to its services happen from the SNIP. Only option in such cases is to create a service bound to that port & protocol, and monitor that.
At work, for instance, we had STA issues. So I created an HTTP service, bound to port 80, for each Delivery Controller. Then I created a new HTTP monitor that checks for /Scripts/CtxSta.dll and expects return code 406. This also lets me create an nstrace against this service itself to see what’s happening.
- nstrace reference
- Set the packet size to 0 and file size to 0.
- Expression will be something like
CONNECTION.SVCNAME.EQ("mySTA_svc_name")
- There’s also nstcpdump.sh, which is a lighter version of nstrace. Less details, but quicker to get up and running. I prefer nstrace. :)
- A blog post with examples of both.
Posted: July 10th, 2017 | Category: PowerShell, Windows | § Quick one-liner to find all AD user objects with ACL inheritance disabled:
|
|
Get-ADUser -SearchBase "DC=myDomain,DC=com" -Filter * | ?{ (Get-Acl $_.DistinguishedName).AreAccessRulesProtected -eq "True" } | ft SamAccountName,Name -AutoSize |
Another one:
|
|
Get-ADUser -SearchBase "DC=myDomain,DC=com" -Filter * -Properties nTSecurityDescriptor | ?{ $_.nTSecurityDescriptor.AreAccessRulesProtected -eq "True" } |
Posted: July 2nd, 2017 | Tags: ADFS, WID | Category: Windows | § Spent a bit of time today tracking down an ADFS/ WID issue. Turned out to be a silly one in the end (silly on my part actually, should have spotted the cause right away!) but it was a good learning exercise in the end.
The issue was that ADFS refused to launch after a server reboot. The console gave an error that it couldn’t connect to the configuration database. The ADFS service refused to start and the event logs were filled with errors such as these:
The Federation Service configuration could not be loaded correctly from the AD FS configuration database.
Additional Data
Error:
ADMIN0012: OperationFault
There was an error in enabling endpoints of Federation Service. Fix configuration errors using PowerShell cmdlets and restart the Federation Service.
Additional Data
Exception details:
System.ServiceModel.FaultException`1[Microsoft.IdentityServer.Protocols.PolicyStore.OperationFault]: ADMIN0012: OperationFault (Fault Detail is equal to Microsoft.IdentityServer.Protocols.PolicyStore.OperationFault).
A SQL operation in the AD FS configuration database with connection string Data Source=np:\\.\pipe\microsoft##wid\tsql\query;Initial Catalog=AdfsConfiguration;Integrated Security=True failed.
Additional Data
Exception details:
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 – Could not open a connection to SQL Server)
The last one repeated many times.
I hadn’t installed the ADFS server in our firm so I had no clue how it was setup. Importantly, I didn’t know it used the Windows Internal Database (WID) which you can see in the error messages above. It is possible to have ADFS work with SQL for a larger setup but that wasn’t the case here. Following some blog posts on the Internet (this and this) I downloaded SQL Server Management Studio (SSMS) and tried connecting to the WID at the path given in the error (\\.\pipe\microsoft##wid\tsql\query). That didn’t work for me – it just gave me some errors that the SQL server was unreachable.
BTW, according to one of the blog posts it is better to launch SSMS as the user who has rights to connect to the WID database (the service account under which your ADFS service runs for instance). That didn’t help in my case (not saying the advice is incorrect, my issue was something else). Found a Microsoft blog post too that confirmed I was connecting to the correct server name – \\.\pipe\microsoft##wid\tsql\query for Windows 2012 and above; \\.\pipe\MSSQL$MICROSOFT##SSEE\sql\query for Windows 2003 & 2008 – but no go.
That’s when I realized the WID has its own service. I had missed this initially. Trying to start that gave an error that it couldn’t start due to a login failure. This service runs under an account NT SERVICE\MSSQL$MICROSOFT##WID and looks like it didn’t have logon as service rights. It looks like someone had played around with our GPOs (or moved this server to a different OU) and this account had lost its rights.
The fix is simple – just give this account rights via GPO (or exclude the server from whatever GPO is fiddling with logon as a service rights; or move this server to some other OU). Since the NT SERVICE\MSSQL$MICROSOFT##WID is not a regular account, you can’t add it to GPO from any server (because the account is local and will only exist if WID is installed). So I opened GPMC on my ADFS server and modified the GPO to give this account logon as a service rights.
Posted: July 1st, 2017 | Tags: CA, CN, netscaler, san, ssl | Category: Citrix, Infrastructure | § I want to create a certificate for my NetScaler and get it working in Chrome. Creating a certificate is easy – there are Citrix docs etc for it – but Chrome keeps complaining about missing subjectAlternativeName. This is because Chrome 58 and upwards ignore the Common Name (CN) field in a certificate and only check the Subject Alternative Names (SAN) field. Other browsers too might ignore the CN field if the SAN field is present (they are supposed to at least); so as a best practice it’s a good idea to fill the SAN field in my NetScaler certificate and put all the names (including the CN) in this field.
Problem is the NetScaler web UI doesn’t have an option for specifying the SAN field. Windows CA (which is what I use internally) supports SAN when making requests, but since the CSR is usually created on the NetScaler and that doesn’t have a way of mentioning SAN, I need an alternative approach.
Here’s one approach from a Citrix blog post. Typically the CLI loving geek in me would have taken that route and stopped at that, but today I feel like exploring GUI options. :)
So I came across the DigiCert Certificate Utility and a guide on how to generate a CSR using that. I don’t need to use the guide entirely as my CA is internal, but the tool (download link) is useful. So I downloaded it and created a certificate request.
A bit of background on the above. I have two NetScalers: ns105-01.rockylabs.zero (IP 10.10.1.150) and ns105-02.rockylabs.zero (IP 10.10.1.160) in an HA pair. For management purposes I have a SNIP 10.10.1.170 (DNS name ns105.rockylabs.zero) which I can connect to without bothering which is the current primary. So I want to create a certificate that will be valid for all three DNS names and IP addresses. Hence in the Subject Alternative Names field I fill in all three names and IP address – note: all three names including the one I put in the common name, since Chrome ignores this field (and other browsers are supposed to ignore the CN if SAN is present).
I click Generate and the tool generates a new CSR. I save this someplace.
Now I need to use this CSR to generate a certificate. Typically I would have gone with the WebServer template in my internal CA, but thing is eventually I’ll have to import this CSR, the generated certificate, and the private key of that certificate to the NetScaler – and the default WebServer template does not allow key exporting.
So I make a new template on my CA. This is just a copy of the default “Web Server” template, but I make a change to allow exporting of the private key (see checkbox below).
Then I create a certificate on my CA using this CSR.
|
|
certreq -attrib "CertificateTemplate:WebServer_withKey" |
The template name “WebServer_withKey” is the name of the template. Need to use that with the certreq command instead of the display name.
This will create the certificate and save it at a location I specify.
At this point I have the CSR and the certificate. I can’t import these into the NetScaler as that also requires the private key. The DigiCert tool generates the private key automatically and keeps it with itself, so we need to import this certificate into the tool and export with key from there. This exports the certificate, along with key, into a PFX format.
This Citrix article is a good reference on the various certificate formats. It also gives instructions on how to import a PFX certificate into NetScaler.
Before proceeding however, a quick summary of the certificate formats from the same article for my own reference:
- PFX is a format for storing a server certificate or any intermediate certificate along with private key in one encrypted file.
- PFX == PKCS#12 (i.e. both terms can be used interchangeably).
- PEM is another format. And a very common one actually. It can contain both certificates and keys, or only either separately.
- These are Base64 encoded ASCII files and have extensions such as .pem, .crt, .cer, or .key.
- DER is a binary form of the PEM format. (So while PEM formats can be opened in Notepad, for instance, as a text file, DER format cannot).
- These are binary files. Have extensions such as .cer and .der. (Note: .cer can be a PEM format too).
So I go ahead and import the PFX file.
And then I install a new certificate created from this imported PFX file.
Note: After taking the screenshot I changed the first field (certificate-key pair name) to “ns105_rockylabs_zero_withKey” just to make it clear to my future self that this certificate includes the key with itself and that I won’t find a separate key file as is usually the case. The second field is the name of the PEM file that was previously created and is already on the appliance.
The certificate is successfully installed:
The next step is to go ahead replace the default NetScaler certificate with this one. This can be done via GUI or CLI as in this Citrix article. The GUI is a bit of a chore here, so I went ahead the CLI way.
|
|
bind ssl service nshttps-10.10.1.170-443 -certkeyName ns105_rockylabs_zero_withKey bind ssl service nsrpcs-10.10.1.170-3008 -certkeyName ns105_rockylabs_zero_withKey bind ssl service nskrpcs-127.0.0.1-3009 -certkeyName ns105_rockylabs_zero_withKey bind ssl service nshttps-::1l-443 -certkeyName ns105_rockylabs_zero_withKey bind ssl service nsrpcs-::1l-3008 -certkeyName ns105_rockylabs_zero_withKey bind ssl service nshttps-127.0.0.1-443 -certkeyName ns105_rockylabs_zero_withKey bind ssl service nsrpcs-127.0.0.1-3008 -certkeyName ns105_rockylabs_zero_withKey |
And that’s it! Now I can access my NetScalers over SSL using Chrome, with no issues.
Posted: June 29th, 2017 | Tags: SPN | Category: Asides, Windows | § Trying to get people at work to clean up duplicate SPNs, and came across some links while reading about this topic.
From the official MSDN article: A service principal name (SPN) is a unique identifier of a service instance. SPNs are used by Kerberos authentication to associate a service instance with a service logon account. This allows a client application to request that the service authenticate an account even if the client does not have the account name.
Basically when a client application tries to authenticate with a service instance and the domain controller needs to issues it Kerberos tickets, the domain controller needs to know whose password to use for the service instance – is it that of the server where this instance runs, or any service account responsible for it. This mapping of service -> service account/ computer account is an SPN. It’s of the format service/host:port and is associated with the AD account of the service account or computer account (stored in the servicePrincipalName attribute actually).
That’s all!
Posted: June 29th, 2017 | Tags: dfsr, sysvol | Category: Asides, Windows | § Just putting these here as bookmarks to myself.
One of our DCs at work had the following DFSR warnings in the DFS Replication logs:
The DFS Replication service stopped replication on volume C:. This occurs when a DFSR JET database is not shut down cleanly and Auto Recovery is disabled. To resolve this issue, back up the files in the affected replicated folders, and then use the ResumeReplication WMI method to resume replication.
Additional Information:
Volume: C:
GUID: 56234A2C-C156-11E2-93E8-806E6F6E6111
Recovery Steps
1. Back up the files in all replicated folders on the volume. Failure to do so may result in data loss due to unexpected conflict resolution during the recovery of the replicated folders.
2. To resume the replication for this volume, use the WMI method ResumeReplication of the DfsrVolumeConfig class. For example, from an elevated command prompt, type the following command:
wmic /namespace:\\root\microsoftdfs path dfsrVolumeConfig where volumeGuid=”56234A2C-C156-11E2-93E8-806E6F6E6111″ call ResumeReplication
For more information, see http://support.microsoft.com/kb/2663685.
Sounded like an easy fix, so I went ahead and tried resuming replication as directed. That didn’t work though. Got the following:
The DFS Replication service stopped replication on the folder with the following local path: D:\SYSVOL_DFSR\domain. This server has been disconnected from other partners for 154 days, which is longer than the time allowed by the MaxOfflineTimeInDays parameter (60). DFS Replication considers the data in this folder to be stale, and this server will not replicate the folder until this error is corrected.
To resume replication of this folder, use the DFS Management snap-in to remove this server from the replication group, and then add it back to the group. This causes the server to perform an initial synchronization task, which replaces the stale data with fresh data from other members of the replication group.
Additional Information:
Error: 9061 (The replicated folder has been offline for too long.)
Replicated Folder Name: SYSVOL Share
Replicated Folder ID: xxxxx
Replication Group Name: Domain System Volume
Replication Group ID: xxxxxx
Member ID: xxxxx
Yeah, bummer!
Check out this Microsoft blog post for content freshness and the MaxOfflineTimeInDays parameter. You can’t simple remove SYSVOL from DFSR replication groups via the GUI as it is a special folder, so you have to work around. I found some forum posts and blog posts that suggested simply raising this parameter for the broken server to a number larger than the number of days its currently been offline (154 in the above case) and then resuming replication. I wasn’t too comfortable with that. What if any older changes from this server now replicate to the other servers? That could cause more damage than it’s worth. I don’t think this will happen, but why take a risk. What I really want is to force a replication onto this server from some other server. Do a non-authoritative replication basically. So I followed the steps in this article and that worked.
A non-authoritative sync is like a regular sync, just that it is rigged to let the source win. :p So all the existing files on the destination server are preserved. The event log gets filled with entries like these:
The DFS Replication service detected that a file was changed on multiple servers. A conflict resolution algorithm was used to determine the winning file. The losing file was moved to the Conflict and Deleted folder.
Additional Information:
Original File Path: D:\SYSVOL_DFSR\domain\Policies\{F4A04331-3C62-474A-A1CE-517F17914111}\GPT.INI
New Name in Conflict Folder: GPT-{4B7F4510-3C34-4A8A-A397-BC736AE5D9B6}-v55459.INI
Replicated Folder Root: D:\SYSVOL_DFSR\domain
File ID: {8189DA2F-DCBE-4755-A042-72154B648111}-v949
Replicated Folder Name: SYSVOL Share
Replicated Folder ID: 05ECDBE0-A0E6-4A9A-B5BE-7C404E323600
Replication Group Name: Domain System Volume
Replication Group ID: 46C8DB83-463F-459F-8785-DCB19231C52B
Member ID: 7A0E2DA6-4841-40A1-B02D-F5F341345B98
Partner Member ID: 7851F335-6824-4B0D-9978-5A5520ECD547
If you want to see where these files are now moved to check out this blog post. That post has a lot more useful info.
Posted: June 23rd, 2017 | Tags: audiobooks, reading | Category: Musings | § When I was a kid I used to always finish whatever book I was reading. Once I reached college and was generally lost in life my reading habit took a turn for the worse and many a times I didn’t manage to finish what I began reading. I took this as a negative thing and much later when I became relatively less lost in life I started reading avidly again and tried to finish whatever I started. I didn’t read every book I bought – mostly because I was still sort of lost in life, but also because the books I started reading since college were mostly non-fiction and I just never got to reading them all due to changing interests. However, if I started a book, I did my best to complete it especially if it was fiction.
I don’t know if this is a good decision though. I don’t have an answer either ways – I am just unsure. The reason why I take this habit of “not finishing a book” as bad is coz that’s what gets drilled into your head. If you start something but don’t complete, it’s generally frowned upon. Plus I read this essay as a child where the author said that young people do more coz they don’t have a choice – they are forced to do from school or parents etc and so they do what is told even if they don’t like, and generally manage to do something of it – but as we grow older we have choices and so become spoilt/ pampered and just give up at the first time something doesn’t go our way.
I get these points but nowadays I also feel that maybe wasting time finishing something just coz we have to finish it is probably just a waste of time. Yes it’s an accomplishment that you don’t leave things half way, but maybe it’s better to just restrict this philosophy to stuff that matters? Like say if you are a person who does a half job of everything – then yes, not good! But maybe you try and do a good job of most things, and mostly succeed too, so perhaps it is ok to ignore it when it comes to some areas (such as reading)? I don’t know.
If I am watching a TV show or movie and leave it midway I don’t chide myself. But I do when it comes to reading a book. That’s because reading a book is more effort than watching something, but end of the day both are entertainment after all. If the objective is to be entertained then why must I give more importance and suffering when it comes to reading?
One reason why I am thinking all this now is due to Audible. They have their Great Listen guarantee wherein if you don’t like an audiobook you can return it. That’s amazing coz sometimes I just don’t like an audiobook – not coz of the narrator or narration or quality etc, I just don’t like it. But since Audible is giving me permission to return it back I don’t have any guilt that oh I bought something and will be wasting money not listening to it. If I am not enjoying it, I can return it – period. There’s nothing Bad involved. Wish similar programs existed for eBooks too!
Recently for instance I started reading two books. “The 100 Year Old Man Who Climbed Out Of The Window and Disappeared” and also “The Winter Fortress”.
“The 100 Year” is great in movie form, and sort of interesting in reading form. I guess coz that sort of content translates well to a movie structure with good score and camera work etc; while if you are reading it all the coincidences and luck get trite after a while.
“The Winter Fortress” is a great non-fiction book about the efforts during World Wor II to destroy a factory in Norway that produced heavy water (used in making nuclear bombs). I read about a third of it and it was a great read. I didn’t know most of it. Then I got side tracked with some other stuff (father in law passing away) and I lost the flow. Now I am trying to get back into it and not in the mood coz I simply have lost the flow. I tried to cheat by purchasing the audiobook version but a) I am still not managing to get into the mood and b) the narrator wasn’t that great (I didn’t like his voice). But I was able to return the audiobook thanks to Audible and so felt no guilt, but I had a heavy heart deciding what to do about the eBooks. Finally I decided it was pointless wasting more time with these two books and so decided to move on. And thought I’d write this post too putting my thoughts down. :)
Part of me feels bad at leaving these two books midway. But (a larger) part of me is relieved at moving on coz I would just have been depressed trying to get “entertained” with these books and not getting anywhere.
Recently I also finished hearing James Franco’s narration of Stephen King’s “The Dark Zone”. This book was nothing like I expected – coz it was quite detailed and the overall plot was simple but what mattered was the details and descriptions and thoughts etc – and while I struggled to finish through it, I didn’t let go because it was manageable. I knew it was only a case of me expecting something else, but the book was well written and narrated and I could hold on till the end. Sometimes it’s worth it; sometimes (like now) it isn’t. Just got to make a case by case decision I think rather than some overarching “policy”.
|