Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

ADFS monitoring on NSX

Was looking at setting up monitoring of my ADFS servers on NSX.

I know what to monitor on the ADFS and WAP servers thanks to this article.

http://<Web Application Proxy name>/adfs/probe
http://<ADFS server name>/adfs/probe
http://<Web Application Proxy IP address>/adfs/probe
http://<ADFS IP address>/adfs/probe

Need to get an HTTP 200 response for these.

So I created a service monitor in NSX along these lines:

And I associated it with my pool:

Bear in mind the monitor has to check port 80, even though my pool might be on port 443, so be sure to change the monitor port as above.

The “Show Pool Statistics” link on the “Pools” section quickly tells us whether the member servers are up or not:

The show service loadbalancer pool command can be used to see what the issue is in case the monitor appears down. Here’s an example when things aren’t working:

Here’s an example when all is well:

Thanks to this document for pointing me in the right troubleshooting direction. Quoting from that document, the list of error codes:

UNK: Unknown

INI: Initializing

SOCKERR: Socket error

L4OK: Check passed on layer 4, no upper layers testing enabled

L4TOUT: Layer 1-4 timeout

L4CON: Layer 1-4 connection problem. For example, “Connection refused” (tcp rst) or “No route to host” (icmp)

L6OK: Check passed on layer 6

L6TOUT: Layer 6 (SSL) timeout

L6RSP: Layer 6 invalid response – protocol error. May caused as the:

Backend server only supports “SSLv3” or “TLSv1.0”, or

Certificate of the backend server is invalid, or

The cipher negotiation failed, and so on

L7OK: Check passed on layer 7

L7OKC: Check conditionally passed on layer 7. For example, 404 with disable-on-404

L7TOUT: Layer 7 (HTTP/SMTP) timeout

L7RSP: Layer 7 invalid response – protocol error

L7STS: Layer 7 response error. For example, HTTP 5xx

Nice!

Quick note to self on NSX Load Balancing

Inline mode == Transparent mode (the latter is the terminology in the UI).

In this mode the load balancer is usually the default gateway for the servers it load balances. Traffic comes to the load balancer, it sends to the appropriate server (after changing the destination IP of the packet – hence DNAT), and replies come to it as it is the default gateway for the server. Note that as far as the destination server is concerned the source IP address is not the load balancer but the client who made the request. Thus the destination server knows who is making the request.

When the load balancer replies to the client who made the request it changes the source IP of the reply from the selected server to its own IP (hence SNAT when replying only).

One-Armed mode == Proxy mode

In this mode the load balancer is not the default gateway. The servers it load balance don’t have any changes required to be made to them. The load balancer does a DNAT as before, but also changes the source IP to be itself rather than the client (hence SNAT). When the selected server replies this time, it thinks the source is the load balancer and so replies to it rather than the client. Thus there’s no changes required on the server side. Because of this though, the server doesn’t know who made the request. All requests appear to come from the load balancer (unless you use some headers to capture the info).

As before, when the load balancer replies to the client who made the request it changes the source IP of the reply from the selected server to its own IP (hence SNAT when replying too).

You set the inline/ transparent vs. one-armed/ proxy mode per pool.

To have load balancing in NSX you need to deploy an ESG (Edge Services Gateway). I don’t know why, but I always associated an ESG with just external routing so it took me by surprise (and still does) when I think I need to deploy an ESG for load balancing, DHCP, and other edge- sort of services (VPN, routing, etc). I guess the point to remember is that it’s not just a gateway – it’s an edge services gateway. :)

Anyways, feel free to deploy as many ESGs as you feel like. You can have one huge ESG that takes care of all your load balancing needs, or you can have multiple small ones and hand over control to the responsible teams.

This is a good starting point doc from VMware.

You can have L4 and L7 load balancing. If you need only L4 (i.e. TCP, UDP, port number) the UI calls it acceleration. It’s a global configuration, on the ESG instance itself, so bear that in mind.

If you enable acceleration on an ESG, you have to also enable it per virtual server.

L4 load balancing is packet based (obviously, coz it doesn’t need to worry about the application as such). L7 load balancing is socket based. Quoting from this doc (highlight mine):

Packet-based load balancing is implemented on the TCP and UDP layer. Packet-based load balancing does not stop the connection or buffer the whole request, it sends the packet directly to the selected server after manipulating the packet. TCP and UDP sessions are maintained in the load balancer so that packets for a single session are directed to the same server. You can select Acceleration Enable in both the global configuration and relevant virtual server configuration to enable packet-based load balancing.

Socket-based load balancing is implemented on top of the socket interface. Two connections are established for a single request, a client-facing connection and a server-facing connection. The server-facing connection is established after server selection. For HTTP socket-based implementation, the whole request is received before sending to the selected server with optional L7 manipulation. For HTTPS socket-based implementation, authentication information is exchanged either on the client-facing connection or on the server-facing connection. Socket-based load balancing is the default mode for TCP, HTTP, and HTTPS virtual servers.

Also worth noting:

The L4 VIP (“acceleration enabled” in the VIP configuration and no L7 setting such as AppProfile with cookie persistence or SSL-Offload) is processed before the edge firewall, and no edge firewall rule is required to reach the VIP. However, if the VIP is using a pool in non-transparent mode, the edge firewall must be enabled (to allow the auto-created SNAT rule).

The L7 HTTP/HTTPS VIPs (“acceleration disabled” or L7 setting such as AppProfile with cookie persistence or SSL-Offload) are processed after the edge firewall, and require an edge firewall allow rule to reach the VIP.

Application Profiles define common application behaviors client SSL, server SSL, x-forwarded-for, and persistence. These can be reused across virtual server and is mandatory when defining a virtual server. This is also where you can do HTTP redirects.

Godless poem – To Love What Death Can Touch

Just finished watching “Godless”. I took my time with this show coz even though I liked it I don’t think this is something I could binge watch. Funny though, that’s something I had wondered to myself – how come I like this show but don’t feel like binge watching it; does it mean I don’t really love this show thaat much? – until I came across a comment from the show’s creator and director Scott Frank on how he too feels this is not a show one can binge watch. It’s a slow burn, best enjoyed over a long time. (I think he said so on the NPR Fresh Air podcast).

A lovely poem from the show finale:

‘Tis a fearful thing
to love what death can touch.
A fearful thing
to love, to hope, to dream, to be –
to be,
And oh, to lose.
A thing for fools, this,
And a holy thing,
a holy thing
to love.
For your life has lived in me,
your laugh once lifted me,
your word was gift to me.
To remember this brings painful joy.
‘Tis a human thing, love,
a holy thing, to love
what death has touched.

– Yehuda HaLevi

That poem’s something.

It’s funny how when I read a written poem it doesn’t hit me as much as it does when I see it on a scene or someone recites it. I guess when I read a poem I just read it without really feeling it.

Absentia

I binge watched “Absentia” today and it was amazing. I came across it by chance on Prime Video and since it has Stana Katvic (“Kate” from the “Castle” TV show) I decided to give it a try. And before I know I was hooked onto it. That’s how the shows that really get to me are – they just hook me up and I naturally binge watch it (e.g. “DareDevil”, “The Punisher”, “Luke Cage”, “Fargo”). There are other shows that I might watch to completion but I wouldn’t mind taking breaks in between (e.g. “Godless” which I am watching off and on nowadays).

Schedule Machine Catalog updates with MCS

I wasn’t aware of this until yesterday when I took a leap of faith and tried it out. I had to update my Machine Catalog with a new image but in the last step of the update my two options are to either reboot all my VMs now and update them, or wait for the next shutdown. The latter is what I want but I don’t have the Citrix Connector for SCCM (nor do my VMs have SCCM agent installed on them).

My delivery groups are set to reboot every weekend and since that was a few hours after I was doing the update above, I chose to go with updating on next shutdown. Then I took a look today and sure enough when the machines rebooted they picked up the new image. Nice!

Anyways, turns out this is well known info. Found this forum post where someone confirms this, as well as a blog post on how someone’s scripted this. Looks like this only happens if you reboot the VMs from Citrix. So a reboot from within Windows (or whatever OS you are running) won’t update. It has to be initiated from Citrix.

[Aside] ESXi and NTP (more)

VMware VMs tend to correct the OS time by syncing with the host. You can disable this but it still syncs during vMotion and some other tasks. This was causing trouble with our Exchange environment as there was a few seconds difference between our hosts and one of the Exchange VMs had its time go back by 5seconds after a vMotion. So we decided to totally disable time sync with the host.

Instructions are simple – https://kb.vmware.com/s/article/1189.

I am supposed to add the following to the VM’s config file after shutting it down:

Sounds simple, but it didn’t go smooth. After doing this the VM refused to start and gave errors about an invalid config file. After some trial an error I figured that the first line was causing trouble. Remove that and now the VM starts as expected. Odd that no one else seems to have encountered this issue!

Curious about what these options do? This PDF is a trove of information on timekeeping and VMware. From the PDF I found the following:

What else? This KB helped me with reloading the VM config file after I made changes. Do vim-cmd vmsvc/getallvms to get a list of VMs and note the ID of the one we are interested in. Then do vim-cmd vmsvc/reload <vmid> to reload.

Update: I realized why things broke. My VM was already set to not update time via VMware Tools, so it already had a line like tools.syncTime = "FALSE". That’s why the first line was causing a conflict.

[Aside] ESXi and NTP

Two posts as a reference to myself –

HPE Synergy and eFuse Reset

In the HPE BladeSystem c7000 Enclosures one can do something called an eFuse reset to power cycle any the server blades. I have blogged about it previously here.

Now we are on the HPE Synergy 12000 Frames at work and I wanted to do something similar. One of the compute modules (aka server :p) was complaining that the server profile couldn’t be applied due to some errors. The compute module was off and refusing to power on, so it looked like there was nothing we could do short of removing it from the frame and putting back. I felt an eFuse reset would do the trick here – it does the same after all.

I couldn’t find any way of doing this via an SSH into the frame’s OneView (which is the equivalent of the Onboard Administrator in a c7000 Enclosure) but then found this PowerShell library from HPE. Now that is pretty cool! Here’s a wiki page too with all the cmdlets – a good page to bookmark and keep handy. Using this I was able to power cycle the compute module.

1) Install the library following instructions in the first link.

2) Login.

3) Get a list of the modules in the enclosure (not really required but I did anyways to confirm the PowerShell view matches my expectations).

4) Now assign the enclosure object containing the module I want to reset to a variable. We need this for the next step.

In my case the Synergy 12000 Frame (capital “F”) is made up of two frame enclosures. (The frame enclosure is where you have the compute modules and interconnects and frame link modules etc).  The module I want to reset is in bay 1 of frame 2. So below I assign the frame 2 object to a variable.

5) Now do the actual eFuse reset.

The -Component parameter can take as argument Device (for compute modules), FLM (for Frame Link Modules), ICM (for InterConnect Modules), and Appliance (for the Synergy Composer or Image Streamer). The -DeviceID parameter is the bay number for the type of component we are trying to reset (so -Component Device -DeviceID 1 is not the same as -Component ICM -DeviceID 1).

An eFuse reset is optional. You could do a simple reset too by skipping the -Efuse switch. The Appliance and ICM components only do eFuse reset though. I am not sure what a regular (non eFuse) reset does.

How to remove complex scripts from Word DOCX documents

Recently came across a Word document where some parts of the document seemed to ignore the general rules. The document was in English, and its language was set to English (U.S.) but certain parts were set to Arabic (Saudi) and none of the usual methods of selecting the text and marking it as English (U.S.) was helping. Very weird.

After a lot of fiddling around I also noticed that if I change the style of a paragraph containing such text, the adjoining text changes but this particular one stays as it. I am able to change the font and size directly by applying them, but changes via styles seem to get ignored.

Then I realized that although this text was in English, since it was marked as Arabic (Saudi) they were being treated as “complex scripts” in the style definitions and hence had separate rules. I guess that at some point someone had marked this text as of being Arabic (Saudi) and continued typing in English, or perhaps the original text was Arabic but someone had changed the font to an English one like “Times New Roman” and typed in English, so even though the text was appearing as English in fact Word was treating it as Arabic written in English (I guess). Anyways, point was Word was treating these blocks as complex scripts (as opposed to Latin for other parts) and so the usual formatting rules didn’t apply to them. Moreover I could change the language from Arabic (Saudi) to Arabic (UAE), for instance, so that seemed to support my theory that it was letting me changing the language to other complex scripts – just not from complex to Latin and vice versa.

This being a DOCX file, it is really just a zip file. So I unzipped it using 7-Zip. Went to the word\styles.xml file (which I came across through trial and error actually, I went through pretty much all the XML files there) in the extracted folder and found the  following:

Since I didn’t want the document to have any Arabic at all, I simply changed the “ar-SA” to “en-US”. Saved the XML file, went back to the extracted folder, and zipped all its contents up again. Renamed this from .zip to .docx and opened the document, and bingo! now all that complex stuff weirdness was gone! :)

(A word to note about zipping back the folder. The format is ZIP. And also, don’t zip the top level folder as then your zip file will be the top level folder followed by all the sub-folders. No, what we want is that the zip file is all the sub-folders directly).

[Aside] Web Servers

I came across these recently and wanted to put them here as a bookmark to myself.

  • h5ai – A modern file browsing UI for web server. Looks amazing!
  • HFS – HTTP File Server. It’s a web server and also a way to send and receive files over HTTP. I haven’t used it by my colleagues recently did.
  • Fenix – A web server you can run on your desktop or laptop. Looks nice too!
  • TinyWeb – A very tiny web server you can run on your desktop or laptop.
  • Caddy – an HTTP/2 web server with automatic HTTPS. Got to check it out sometime.

Asus RT-AC68U router, firmware, etc.

Bought an Asus RT-AC68U router today. I didn’t like my existing D-Link much and a colleague bought the Asus and was all praises so I thought why not try that.

Was a bit put off that many of the features (especially the parental control ones) seem to be tied up with a Trend Micro service that’s built into the router. When you enable these you get an EULA agreement from Trend Micro, and while I usually just click EULA agreements this one caught my eye coz it said somewhere that Asus takes no responsibility for any actions of Trend Micro and so they pretty much wash their hands off whatever Trend Micro might do once you sign up for it. That didn’t sound very nice. I mean, yes, I knew the router had some Trend Micro elements in it, and I have used Trend Micro in the past and have no beef with them, but I bought an Asus router and I expect them to take responsibility for whatever they put in the box.

Anyways, Googling about it I found some posts like this, this, and this that echoed similar sentiments and put me off. It was upsetting as a lot of value I was hoping to get out of the router was centered around using Trend Micro, and since I didn’t want to accept the EULA I would never be able to use it.

I briefly thought of flashing some other firmware in the hopes that that will give me more feature. Advanced Tomato looks nice, but then I came across Asus WRT Merlin which seems to be based on the official firmware but with some additional features and bug fixes and a focus on performance and safety rather than new features. (Also, the official Asus firmware and also the Merlin one have hardware NAT acceleration and proprietary NTFS drivers that offer better performance, while other third party firmware don’t have this. The hardware NAT only matters if your WAN connection is > 100Mbps, which wasn’t so in my case). Asus WRT Merlin looks good. The UI is same as the official one, and it appears that the official firmware has slowly embraced many of the newer features of Merlin. Also, this discussion from the creator of the Merlin firmware on the topic of Trend Micro was good too. Wasn’t as doom and gloom like the others (but I still haven’t enabled the Trend Micro stuff nor do I plan on doing so).

The Merlin firmware is amazing. Flashing it is easy, and it gives some nifty new features. For example you can have custom config files that extend the inbuilt DHCP/ DNS server dnsmasq, have other 3rd party software, and so on. This official Wiki page is a good read. I came across this malware blocking script and installed it. I also made some changes to DHCP so that certain machines get different DNS servers (e.g. point my daughter’s machine to use the Yandex.DNS). Here’s a bit from my config file in case it helps –

This dnsmasq manpage was helpful, so was this page of examples. Also this StackOverflow post.

I liked this idea of having separate DHCP options for specific SSIDs, and also this one of having a separate SSID that’s connected to VPN (nice!). I wanted to try these but was feeling lazy so didn’t get around to doing it. I read a lot about it though and liked this post on having separate VLANs within the router. That post also explains the port numbering etc. of the router – its a good read. I also wanted to see if it was possible to have a separate VLAN for an SSID – lets say have all my visitors connect to a different SSID with its own VLAN and IP range etc. I know I can do the IP range and stuff but looks like if I need to do a separate VLAN I’ll have to give up one of the four ports on the back of the router. Basically the way things seem to be setup are that the 5 ports on the back of the router are part of the same switch, just that the WAN port is in its own VLAN 2 while the LAN ports are in their own VLAN 1.  The WLAN (Wireless) are bridged to this VLAN 1. So if you want a separate WLAN SSID with its own VLAN, we must create a new VLAN on one of the four ports and bridge the new SSID to that.

In the above port 0 is the WAN, port 1-4 are the LAN ports, and port 5 is the router itself (the SOC on the router). Since port 5 is part of both VLANs the router can route between them. The port numbers vary per model. Here’s a post showing what the above output might look like in such a case. As a reference to myself this person was trying to do something similar (I didn’t read all the posts so there could be stuff I missed in there).

Lastly these two wiki pages from DD-WRT Wiki are worth referring to at some point – on the various ports, and multiple WLANs.

At some point, when I am feeling less lazy, I must fiddle around with this router a bit more. It’s fun, reminds me of my younger days with Linux. :)

Launching XenApp sessions in the receiver time zone

Gotta do two things for this to happen.

1. In the Delivery Controller policies enable this setting –

2. In the GPOs on the XenApp servers allow time zone redirection –

Using PowerShell to find Computer objects in AD that have inheritance disabled

I needed to find the computer objects in an AD OU that had inheritance disabled. Did the following:

And to extend this to enable inheritance on the affected objects:

Deleting credentials from Credential Manager remotely

The title is a bit misleading, but whatever. I wanted to delete the credentials in Windows Credential Manager on a remote machine. I didn’t want to delete any particular credential – what I suggest below won’t work for that – but simply all the credentials stored for a particular user.

What you need to do is go to the C: drive of that remote machine and then C:\Users\<the user>\AppData\Roaming\Microsoft\Credentials. You may not see anything here as they are System files and hence hidden. But once you do the needful to see them, delete all the files present here and the credentials are removed.

[Aside] XenApp beta testing with Application Groups

Application Groups is a new feature introduced in XenApp and XenDesktop 7.9 (speaking of which: XenApp and XenDesktop are the same thing just that different functionality is exposed based on the license. I kind of knew this, but thanks to proper testing by James Rankin as shown in his YouTube video I can now say this with confidence). I’d thought of writing a blog post on this but (a) I am lazy and (b) this blog post from Citrix explains it much better. Take note of the example they give with beta testers – that’s just what I do in my environment too.

Machine Catalogs contain your machines. Delivery Groups target a subset (or entirety) of the machines in a Machine Catalog. Delivery Groups can contain machines from multiple Machine Catalogs but a single machine can only be a member of one Delivery Group.

Typically you’d create Machine Catalogs and assign machines from these to a Delivery Group. Then you’d define applications in the Delivery Group and assign users who can access them. When you use Application Groups, however, you continue to assign users in Delivery Groups but now you associate the Application Group with one or more Delivery Groups and define applications in the Application Group. You can set priorities for the Delivery Groups within an Application Group, and if an application is present in more than one Delivery Group (and the user launching the application has permissions to these Delivery Groups) then it is launched from the Delivery Group with the higher priority (a lower number has higher priority).

Once we start using Application Groups there’s no need to define applications in Delivery Groups.

Application Groups also help in targeting specific machines in a Delivery Group. As I mentioned above a Delivery Group can contain machines from multiple Catalogs. Using Application Groups its possible that some users are “pinned” to applications from machines in specific Machine Catalogs.

Here are more links on how Application Groups can be used along with tags: