Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

Disabling Exchange 2013 Managed Availability monitors

Posted: July 30th, 2018 | Tags: , | Category: Exchange | §

Check out this blog post from Microsoft first. Mine’s mostly based on that but tailored to my specific situation.

We have a CAS server that’s purely for internal admin ECP functions. Managed Availability was running some ActiveSync tests on it and failing (because they don’t exist) with errors like these in SCOM:

So my mission, which I’ve chosen to accept (I saw “Mission Impossible: Fallout” this weekend!) is to disable this. :)

Managed Availability has the concept of health sets. From this page which lists all the health sets in Exchange 2013:

In Managed Availability, each component in Exchange 2013 monitors itself using probes, monitors and responders. Each Exchange 2013 component that implements Managed Availability is referred to as a health set.

So what are the unhealthy health sets on my server?

The result of this in my case are the following:

So how do I find which monitors in these health sets are failing? The following cmdlet can help:

I’ll just pipe both cmdlets to get a list of monitors across all unhealthy health sets:

In my case I get:

Should probably have filtered to just the unhealthy monitors:

Anyways, the SCOM error referred to an EAS component, but I don’t see anything with that name. ActiveSyncProxy is probably the one it was referring to?

As an aside, if I want to see the components of a health set (i.e. the monitors, probes, responders) I can do the following:

In the case of the ActiveSync.Proxy health set (which has the ActiveSyncProxy component) I can see:

Note that the ActiveSyncProxyTestMonitor monitor is what was showing as unhealthy earlier.

To disable a monitor I need to use the Add-ServerMonitoringOverride cmdlet. This is of the format:

In my case, to disable ActiveSync.Proxy (health set) ActiveSyncProxyTestMonitor (monitoring item – you can see this in the list of unhealthy monitors as well as in the list above) I do:

That’s it. Wait a while and now it will appear as disabled.

Next thing is how do I find out why the ActiveSync health set is unhealthy? Let’s take a  look at the probes in that:

I can invoke the probe manually using the following cmdlet:

Pipe this out as a list to read better. Here’s what I did:

The output gives the errors encountered by the tests. I could see that it was related to EAS so decided to disable it too.

Lastly, if you are curious as to what overrides exist the following cmdlet will help:

Also, if you want to double check that a particular component on the Exchange server is inactive (and that’s why monitors are failing) the following cmdlet will help (I sort it by state for easy reading but that’s optional):

The last section of the article I referred to at the beginning of this post on editing C:\Program Files\Microsoft\Exchange Server\V15\Bin\Monitoring\Config\ClientAccessProxyTest.xml to disable certain probes. Not sure why they suggest that instead of disabling probes via the cmdlet – I think that’s because the cmdlets way is more of a temporary thing (for a certain duration) while modify the config file is a permanent fix. I should probably do the config file in my case.

[Aside] Various Azure links

Posted: May 21st, 2018 | Tags: , , , , , | Category: Asides, Infrastructure | §

My blog posting has taken a turn for the worse. Mainly coz I have been out of country and since returning I am busy reading up on Azure monitoring.

Anyways, some quick links to tabs I want to close now but which will be useful for me later –

  • A funny thing with Azure monitoring (OMS/ Log Analytics) is that it can’t just do simple WMI queries against your VMs to check if a service is running. Crazy, right! So you have to resort to tricks like monitor the event logs to see any status messages. Came across this blog post with a neat idea of using performance counters. I came across that in turn from this blog post that has a different way of using the event logs.
  • We use load balancers in Azure and I was thinking I could tap into their monitoring signals (from the health probes) to know if a particular server/ service is up or down. In a way it doesn’t matter if a particular server/ service is down coz there won’t be a user impact coz of the load balancer, so what I am really interested in knowing is whether a particular monitored entity (from the load balancer point of view) is down or not. But turns out the basic load balancer cannot log monitoring signals if it is for internal use only (i.e. doesn’t have a public IP). You either need to assign it a public IP or use the newer standard load balancer.
  • Using OMS to monitor and send alert for BSOD.
  • Using OMS to track shutdown events.
  • A bit dated, but using OMS to monitor agent health (has some queries in the older query language).
  • A useful list of log analytics query syntax (it’s a translation from old to new style queries actually but I found it a good reference)

Now for some non-Azure stuff which I am too lazy to put in a separate blog post:

  • A blog post on the difference between application consistent and crash consistent backups.
  • At work we noticed that ADFS seemed to break for our Windows 10 machines. I am not too clear on the details as it seemed to break with just one application (ZScaler). By way of fixing it we came across this forum post which detailed the same symptoms as us and the fix suggested there (Set-ADFSProperties -IgnoreTokenBinding $True) did the trick for us. So what is this token binding thing?
    • Token Binding seems to be like cookies for HTTPS. I found this presentation to be a good explanation of it. Basically token binding binds your security token (like cookies or ADFS tokens) to the TLS session you have with a server, such that if anyone were to get hold of your cookie and try to use it in another session it will fail. Your tokens are bound to that TLS session only. I also found this medium post to be a good techie explanation of it (but I didn’t read it properly*). 
    • It seems to be enabled on the client side from Windows 10 1511 and upwards.
    • I saw the same recommendation in these Microsoft Docs on setting up Azure stack.

Some excerpts from the medium post (but please go and read the full one to get a proper understanding). The excerpt is mostly for my reference:

Most of the OAuth 2.0 deployments do rely upon bearer tokens. A bearer token is like ‘cash’. If I steal 10 bucks from you, I can use it at a Starbucks to buy a cup of coffee — no questions asked. I do not want to prove that I own the ten dollar note.

OAuth 2.0 recommends using TLS (Transport Layer Security) for all the interactions between the client, authorization server and resource server. This makes the OAuth 2.0 model quite simple with no complex cryptography involved — but at the same time it carries all the risks associated with a bearer token. There is no second level of defense.

OAuth 2.0 token binding proposal cryptographically binds security tokens to the TLS layer, preventing token export and replay attacks. It relies on TLS — but since it binds the tokens to the TLS connection itself, anyone who steals a token cannot use it over a different channel.

Lastly, I came across this awesome blog post (which too I didn’t read properly* – sorry to myself!) but I liked a lot so here’s a link to my future self – principles of token validation.

 

* I didn’t read these posts properly coz I was in a “troubleshooting mode” trying to find out why ADFS broke with token binding. If I took more time to read them I know I’d get side tracked. I still don’t know why ADFS broke, but I have an idea.

Enabling SNMPv3 on ESXi hosts

Posted: November 6th, 2016 | Tags: , , , | Category: Virtualization | §

A continuation to my earlier post which was to do with SNMPv2.

As before, connect to the vCenter via PowerCLI. And as before the set() method can be used to set SNMP – both v2 and/or v3. The definition of this method is as follows:

That’s confusing so best to copy paste the definition into notepad or something so you can be sure you are passing the correct arguments.

First things first. There doesn’t seem to be a way of turning off something. As in, say you already have SNMPv2 turned on, you can’t turn it off by setting the community strings to blank. Doing so generates an error. So if you want to turn previous things off it’s best to do a reset and start with a clean slate.

This sets things back to their defaults:

Before going ahead with any SNMPv3 configuration we need to decide on what authentication and privacy protocols to use. In my case I want to use SHA1 and AES-128. So I need to set that first:

Once I have done this I can generate the hashes. I will need this later to configure SNMPv3.

In the example above both my passwords are Password1.

With this in hand I configure SNMPv3:

That’s it really. In the above example I will be using an SNMPv3 user called snmpUser1.

Now to do it across my estate I can make a loop. No need to create password hashes for each host. The hash stays the same as long as you are using the same password for each host.

That’s all!

Enabling SNMP on ESXi hosts

Posted: May 22nd, 2016 | Tags: , , , | Category: Virtualization | §

I wanted to enable SNMP on our ESXi hosts for monitoring via Solarwinds. Here’s what I did. (I am doing this kind of generically, using variables etc, so I can script the thing for multiple hosts).

First I connected to the vCenter Server from PowerCLI.

Next I got its ESXCLI object. This will let me run ESXCLI commands against the host.

To view the current status of SNMP you can do can invoke a get() method –

Nothing’s configured currently. To configure something we can use the set() method. From the definition of this method we can see it takes a whole bunch of parameters –

Here’s what I did to configure SNMP. I want a community string of “public”, enable SNMP, and specify two trap destinations.

The result of that will either be a true or false. The get() method can be used again to confirm it is set correctly. And the test() method can be used to test it works –

Now Solarwinds will be able to poll the host via SNMP.

To do this en-masse on all your hosts the following should help –

Shout out to this VMware blog post which helped a lot and has more info.

The above script failed on some of our ESX hosts with the following error –

Turns out these hosts only accept 16 parameters instead of 17 (the one called largestorage is missing). Not sure why. All our hosts are ESXi 5.5 but am thinking the problem ones are perhaps not using the HP customized version of ESXi.

Anyways, so I modified my script above to take care of this –

Also, just for my own info – the $null above means the parameter is not set. If that parameter already has a value on the server it is not over-written. To over-write or blank out the existing value replace $null with "".

Create Solarwinds account limitations based on Custom Properties

Posted: April 29th, 2016 | Tags: , , | Category: Infrastructure | §

I wanted to create account limitations in Solarwinds based on Custom Properties but the web console by default doesn’t give an option to do that.

default limitationsThen I came across this helpful video.

The trick is to logon to your Solarwinds server and find the “Account Limitation Builder”. Then click “Add” and create a new limitation similar to this –

new limitation

Now the limitation you create will come in the list –

new limitation2

Select that, and in the next screen you can choose the value you want to limit to –

new limitation3

Removing a monitored resource from multiple nodes in Solarwinds

Posted: April 28th, 2016 | Tags: , | Category: Infrastructure | §

Had to remove a drive from being monitored from multiple servers in Solarwinds. Rather than go to each node, edit its properties and untick the drive, I figured that if you do a search for the nodes and expand each of them it’s possible to tick the ones you don’t need and click “Delete”.

multiple nodes

Nice!

Using Solarwinds to monitor Windows Services

Posted: April 28th, 2016 | Tags: , | Category: Infrastructure, Windows | §

This is similar to how I monitored performance counters with Solarwinds.

I want to monitor a bunch of AppSense services.

Similar to the performance counters where I created an application monitor template so I could define the threshold values, here I need to create an application monitor so that the service appears in the alert manager. This part is easy (and similar to what I did for the performance counters) so I’ll skip and just put a screenshot like below –

applimonitor

I created a separate application monitor template but you could very well add it to one of the standard templates (if it’s a service that’s to be monitored on all nodes for instance).

Now for the part where you create alerts.

Initially I thought this would be a case of creating triggers when the above application monitor goes down. Something like this – alert1

And create an alert message like this –

alert1a

With this I was hoping to get a one or more alert messages only for the services that actually went down. Instead, what happened is that whenever any one service went down I’d get an alert for the service that went down and also a message for the services that were up. Am guessing since I was triggering on the application monitor, Solarwinds helpfully sent the status for each of its components – up or down.

alert2

The solution is to modify your trigger such that you target each component.

alert3

Now I get alerts the way I want.

Hope this helps!

Mute Solarwinds alerts during reboots/ maintenance windows

Posted: April 25th, 2016 | Tags: , , | Category: Infrastructure | §

I wanted to mute Solarwinds alerts during our patch weekends when all servers are rebooted because they have to be and our mailboxes get flooded with Solarwinds alerts. I decided to use custom properties for this purpose. Here’s what I did.

Login to the Solarwinds web console. Go to the “Settings” page, and then “Manage Custom Properties” under “Node *& Group Management”.

Click “Add Custom Property”, select the default of “Nodes” from the drop down, and create something along the following lines –

customproperties

Select the nodes you’d like to apply this custom property to. I chose to apply it on all my Windows and VMware nodes. Set the value to be “No”.

customvalues

Now login to Orion Alerts Manager and pick an alert you’d like to mute during patch weekends. Go to its “Alert Suppression” tab and add a condition on the custom property we created earlier.

alert custom properties

alert custom properties2

And that’s it, really!

Update: Not sure why, but the above didn’t seem to work for me. So I added the Mute_Alerts check as part of the trigger condition itself.

new trigger

Note: If you don’t get the custom property in Orion, close and restart it as an administrator (i.e. right click and do “Run as Administrator” even if you are already running it with an admin account). Not sure why, but until I did that the custom property didn’t get picked up. You only need to do it one time; after that you can launch Orion normally.

Next time your server estate is being rebooted/ undergoing maintenance, login to Solarwinds webconsole and change the “Mute_Alerts” custom property to “Y” for a node/ nodes that you want to mute alerts for. Below I show how I will mute the alerts for all my Windows nodes.

Go to “Manage Nodes”. Group by “Vendor” and select Windows. Then select all nodes. (The checkbox to select all nodes got blanked out in the screenshot below but it’s easy to find).

apply custom property

Then click on “Custom Property Editor” to get to the screen below.

apply custom property

Here too select all the nodes and click “Edit multiple values”.

From the drop down, change the value for “Mute_Alerts” to “true”. Then save changes and that’s it. :)