 © Rakhesh Sasidharan
|
Posted: July 30th, 2018 | Tags: exchange 2013, monitoring | Category: Exchange | § Check out this blog post from Microsoft first. Mine’s mostly based on that but tailored to my specific situation.
We have a CAS server that’s purely for internal admin ECP functions. Managed Availability was running some ActiveSync tests on it and failing (because they don’t exist) with errors like these in SCOM:
|
|
ActiveSync is failing on ClientAccess server XXXXX. Incident start time: 7/29/2018 1:54:55 PM Last failed result: Failing Component - EAS Failure Reason - Unknown Reason: 503 <snip> |
So my mission, which I’ve chosen to accept (I saw “Mission Impossible: Fallout” this weekend!) is to disable this. :)
Managed Availability has the concept of health sets. From this page which lists all the health sets in Exchange 2013:
In Managed Availability, each component in Exchange 2013 monitors itself using probes, monitors and responders. Each Exchange 2013 component that implements Managed Availability is referred to as a health set.
So what are the unhealthy health sets on my server?
|
|
Get-HealthReport -Identity <server name> | ?{ $_.AlertValue -ne "Healthy" } |
The result of this in my case are the following:
|
|
Server State HealthSet AlertValue LastTransitionTime MonitorCount ------ ----- --------- ---------- ------------------ ------------ <server> NotApplicable ActiveSync Unhealthy 7/29/2018 2:24:5... 2 <server> Offline ActiveSync.Proxy Unhealthy 4/24/2018 8:55:3... 1 <server> Offline EWS.Proxy Unhealthy 4/24/2018 8:55:4... 1 <server> Offline OAB.Proxy Unhealthy 4/24/2018 8:56:2... 1 <server> NotApplicable EWS Unhealthy 4/25/2018 12:55:... 3 |
So how do I find which monitors in these health sets are failing? The following cmdlet can help:
|
|
Get-ServerHealth <server name> |
I’ll just pipe both cmdlets to get a list of monitors across all unhealthy health sets:
|
|
Get-HealthReport -Identity <server name> | ?{ $_.AlertValue -ne "Healthy" } | %{ Get-ServerHealth -Identity <server name> -HealthSet $_.HealthSet } |
In my case I get:
|
|
Server State Name TargetResource HealthSetName AlertValue ServerComponent ------ ----- ---- -------------- ------------- ---------- --------------- <server> NotApplicable ActiveSyncCTPMonitor ActiveSync ActiveSync Unhealthy None <server> NotApplicable RequestsQueuedGt500Monitor MSExchangeSyncAppPool ActiveSync Healthy None <server> Offline ActiveSyncProxyTestMonitor MSExchangeSyncAppPool ActiveSync.Proxy Unhealthy ActiveSyncProxy <server> Offline EWSProxyTestMonitor MSExchangeServicesAppPool EWS.Proxy Unhealthy EwsProxy <server> Offline OABProxyTestMonitor MSExchangeOABAppPool OAB.Proxy Unhealthy OabProxy <server> NotApplicable EWSCtpMonitor <redacted> EWS Unhealthy None <server> NotApplicable MaintenanceFailureMonitor.EWS EWS Healthy None <server> NotApplicable MaintenanceTimeoutMonitor.EWS EWS Healthy None |
Should probably have filtered to just the unhealthy monitors:
|
|
Get-HealthReport -Identity <server name> | ?{ $_.AlertValue -ne "Healthy" } | %{ Get-ServerHealth -Identity <server name> -HealthSet $_.HealthSet } | ?{ $_.AlertValue -ne "Healthy" } |
Anyways, the SCOM error referred to an EAS component, but I don’t see anything with that name. ActiveSyncProxy is probably the one it was referring to?
As an aside, if I want to see the components of a health set (i.e. the monitors, probes, responders) I can do the following:
|
|
Get-MonitoringItemIdentity -Identity <health set name> -Server <server name> | ft Identity,ItemType,TargetResource -AutoSize |
In the case of the ActiveSync.Proxy health set (which has the ActiveSyncProxy component) I can see:
|
|
Identity ItemType TargetResource -------- -------- -------------- ActiveSync.Proxy\ActiveSyncProxyTestProbe\MSExchangeSyncAppPool Probe MSExchangeSyncAppPool ActiveSync.Proxy\ActiveSyncProxyTestMonitor\MSExchangeSyncAppPool Monitor MSExchangeSyncAppPool ActiveSync.Proxy\ActiveSyncProxyTestRecycleAppPool\MSExchangeSyncAppPool Responder MSExchangeSyncAppPool ActiveSync.Proxy\ActiveSyncProxyTestOffline\MSExchangeSyncAppPool Responder MSExchangeSyncAppPool ActiveSync.Proxy\ActiveSyncProxyTestOfflineFailedEscalate\MSExchangeSyncAppPool Responder MSExchangeSyncAppPool ActiveSync.Proxy\ActiveSyncProxyTestEscalate\MSExchangeSyncAppPool Responder MSExchangeSyncAppPool |
Note that the ActiveSyncProxyTestMonitor monitor is what was showing as unhealthy earlier.
To disable a monitor I need to use the Add-ServerMonitoringOverride cmdlet. This is of the format:
|
|
Add-ServerMonitoringOverride -Server <server name> -Identity <HealthSet>\<MonitoringItemName>[\<TargetResource>] -ItemType <Probe | Monitor | Responder | Maintenance> -PropertyName Enabled -PropertyValue 0 |
In my case, to disable ActiveSync.Proxy (health set) ActiveSyncProxyTestMonitor (monitoring item – you can see this in the list of unhealthy monitors as well as in the list above) I do:
|
|
Add-ServerMonitoringOverride -Server <server name> -Identity ActiveSync.Proxy\ActiveSyncProxyTestMonitor -ItemType Monitor -PropertyName Enabled -PropertyValue 0 |
That’s it. Wait a while and now it will appear as disabled.
|
|
PS> Get-HealthReport -Identity <server> | ?{ $_.alertvalue -ne "Healthy" } | %{ Get-ServerHealth -Identity <server> -HealthSet $_.HealthSet } | ?{ $_.AlertValue -ne "Healthy" } | ft -AutoSize Server State Name TargetResource HealthSetName AlertValue ServerComponent ------ ----- ---- -------------- ------------- ---------- --------------- <server> NotApplicable ActiveSyncCTPMonitor ActiveSync ActiveSync Unhealthy None <server> Offline ActiveSyncProxyTestMonitor MSExchangeSyncAppPool ActiveSync.Proxy Disabled ActiveSyncProxy <server> Offline EWSProxyTestMonitor MSExchangeServicesAppPool EWS.Proxy Unhealthy EwsProxy <server> Offline OABProxyTestMonitor MSExchangeOABAppPool OAB.Proxy Unhealthy OabProxy <server> NotApplicable EWSCtpMonitor <redacted> EWS Unhealthy None |
Next thing is how do I find out why the ActiveSync health set is unhealthy? Let’s take a look at the probes in that:
|
|
PS>Get-MonitoringItemIdentity -Identity "ActiveSync" -Server ca1exchangeecp | ft Identity,ItemType,TargetResource -autosize Identity ItemType TargetResource -------- -------- -------------- ActiveSync\ActiveSyncCTPProbe\MSExchangeSyncAppPool Probe MSExchangeSyncAppPool ActiveSync\ActiveSyncCTPMonitor\ActiveSync Monitor ActiveSync ActiveSync\RequestsQueuedGt500Monitor\MSExchangeSyncAppPool Monitor MSExchangeSyncAppPool ActiveSync\ActiveSyncCTPEscalate\ActiveSync Responder ActiveSync ActiveSync\RequestsQueuedGt500Escalate\MSExchangeSyncAppPool Responder MSExchangeSyncAppPool |
I can invoke the probe manually using the following cmdlet:
|
|
Invoke-MonitoringProbe <health set name>\<probe name> -Server <server name> |
Pipe this out as a list to read better. Here’s what I did:
|
|
Invoke-MonitoringProbe ActiveSync\ActiveSyncCTPProbe\MSExchangeSyncAppPool -Server <server name> | fl |
The output gives the errors encountered by the tests. I could see that it was related to EAS so decided to disable it too.
Lastly, if you are curious as to what overrides exist the following cmdlet will help:
|
|
Get-ServerMonitoringOverride -Server <server name> |
Also, if you want to double check that a particular component on the Exchange server is inactive (and that’s why monitors are failing) the following cmdlet will help (I sort it by state for easy reading but that’s optional):
|
|
Get-ServerComponentState -Identity <server name> | Sort-Object State |
The last section of the article I referred to at the beginning of this post on editing C:\Program Files\Microsoft\Exchange Server\V15\Bin\Monitoring\Config\ClientAccessProxyTest.xml to disable certain probes. Not sure why they suggest that instead of disabling probes via the cmdlet – I think that’s because the cmdlets way is more of a temporary thing (for a certain duration) while modify the config file is a permanent fix. I should probably do the config file in my case.
My blog posting has taken a turn for the worse. Mainly coz I have been out of country and since returning I am busy reading up on Azure monitoring.
Anyways, some quick links to tabs I want to close now but which will be useful for me later –
- A funny thing with Azure monitoring (OMS/ Log Analytics) is that it can’t just do simple WMI queries against your VMs to check if a service is running. Crazy, right! So you have to resort to tricks like monitor the event logs to see any status messages. Came across this blog post with a neat idea of using performance counters. I came across that in turn from this blog post that has a different way of using the event logs.
- We use load balancers in Azure and I was thinking I could tap into their monitoring signals (from the health probes) to know if a particular server/ service is up or down. In a way it doesn’t matter if a particular server/ service is down coz there won’t be a user impact coz of the load balancer, so what I am really interested in knowing is whether a particular monitored entity (from the load balancer point of view) is down or not. But turns out the basic load balancer cannot log monitoring signals if it is for internal use only (i.e. doesn’t have a public IP). You either need to assign it a public IP or use the newer standard load balancer.
- Using OMS to monitor and send alert for BSOD.
- Using OMS to track shutdown events.
- A bit dated, but using OMS to monitor agent health (has some queries in the older query language).
- A useful list of log analytics query syntax (it’s a translation from old to new style queries actually but I found it a good reference)
Now for some non-Azure stuff which I am too lazy to put in a separate blog post:
- A blog post on the difference between application consistent and crash consistent backups.
- At work we noticed that ADFS seemed to break for our Windows 10 machines. I am not too clear on the details as it seemed to break with just one application (ZScaler). By way of fixing it we came across this forum post which detailed the same symptoms as us and the fix suggested there (
Set-ADFSProperties -IgnoreTokenBinding $True) did the trick for us. So what is this token binding thing?
- Token Binding seems to be like cookies for HTTPS. I found this presentation to be a good explanation of it. Basically token binding binds your security token (like cookies or ADFS tokens) to the TLS session you have with a server, such that if anyone were to get hold of your cookie and try to use it in another session it will fail. Your tokens are bound to that TLS session only. I also found this medium post to be a good techie explanation of it (but I didn’t read it properly*).Â
- It seems to be enabled on the client side from Windows 10 1511 and upwards.
- I saw the same recommendation in these Microsoft Docs on setting up Azure stack.
Some excerpts from the medium post (but please go and read the full one to get a proper understanding). The excerpt is mostly for my reference:
Most of the OAuth 2.0 deployments do rely upon bearer tokens. A bearer token is like ‘cash’. If I steal 10 bucks from you, I can use it at a Starbucks to buy a cup of coffee — no questions asked. I do not want to prove that I own the ten dollar note.
OAuth 2.0 recommends using TLS (Transport Layer Security) for all the interactions between the client, authorization server and resource server. This makes the OAuth 2.0 model quite simple with no complex cryptography involved — but at the same time it carries all the risks associated with a bearer token. There is no second level of defense.
OAuth 2.0 token binding proposal cryptographically binds security tokens to the TLS layer, preventing token export and replay attacks. It relies on TLS — but since it binds the tokens to the TLS connection itself, anyone who steals a token cannot use it over a different channel.
Lastly, I came across this awesome blog post (which too I didn’t read properly* – sorry to myself!) but I liked a lot so here’s a link to my future self – principles of token validation.
* I didn’t read these posts properly coz I was in a “troubleshooting mode” trying to find out why ADFS broke with token binding. If I took more time to read them I know I’d get side tracked. I still don’t know why ADFS broke, but I have an idea.
Posted: November 6th, 2016 | Tags: monitoring, powercli, snmp, solarwinds | Category: Virtualization | § A continuation to my earlier post which was to do with SNMPv2.
As before, connect to the vCenter via PowerCLI. And as before the set() method can be used to set SNMP – both v2 and/or v3. The definition of this method is as follows:
|
|
set(string authentication, string communities, boolean enable, string engineid, string hwsrc, boolean largestorage, string loglevel, string notraps, long port, string privacy, string remoteusers, boolean reset, string syscontact, string syslocation, string targets, string users, string v3targets) |
That’s confusing so best to copy paste the definition into notepad or something so you can be sure you are passing the correct arguments.
First things first. There doesn’t seem to be a way of turning off something. As in, say you already have SNMPv2 turned on, you can’t turn it off by setting the community strings to blank. Doing so generates an error. So if you want to turn previous things off it’s best to do a reset and start with a clean slate.
|
|
$esxcli.system.snmp.set($null, $null, $null, $null, $null, $null, $null, $null, $null, $null, $null, $true, $null, $null, $null, $null, $null) |
This sets things back to their defaults:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
> $esxcli.system.snmp.get() authentication : communities : enable : false engineid : hwsrc : indications largestorage : true loglevel : info notraps : port : 161 privacy : remoteusers : syscontact : syslocation : targets : users : v3targets : |
Before going ahead with any SNMPv3 configuration we need to decide on what authentication and privacy protocols to use. In my case I want to use SHA1 and AES-128. So I need to set that first:
|
|
$esxcli.system.snmp.set("SHA1", $null, $null, $null, $null, $null, $null, $null, $null, "AES128", $null, $null, $null, $null, $null, $null, $null) |
Once I have done this I can generate the hashes. I will need this later to configure SNMPv3.
|
|
> $esxcli.system.snmp.hash("Password1!","Password1!",$true) authhash privhash -------- -------- 9e79c8406990d1dxxxxxxx975b364b7e90e9c2ab 9e79c8406990d1dxxxxxxx975b364b7e90e9c2ab |
In the example above both my passwords are Password1.
With this in hand I configure SNMPv3:
|
|
$esxcli.system.snmp.set("SHA1", $null, $true, $null, $null, $null, "info", $null, 161, "AES128", $null, $null, "IT@myfirm.com", "My Firm IT Department", $null, "snmpUser1/9e79c8406990d1dxxxxxxx975b364b7e90e9c2ab/9e79c8406990d1dxxxxxxx975b364b7e90e9c2ab/priv", "xxx.xxx.xxx.xxx/user/priv/trap,xxx.xxx.xxx.xxx/user/priv/trap") |
That’s it really. In the above example I will be using an SNMPv3 user called snmpUser1.
Now to do it across my estate I can make a loop. No need to create password hashes for each host. The hash stays the same as long as you are using the same password for each host.
|
|
$Hosts = Get-VMHost | ?{ $_.ConnectionState -eq "Connected" } foreach ($VMHost in $Hosts) { Write-Host "Working on $VMHost" $ESXClI = Get-ESXCli -VMHost $VMHost $ESXCLI.system.snmp.set($null, $null, $null, $null, $null, $null, $null, $null, $null, $null, $null, $true, $null, $null, $null, $null, $null) $ESXCLI.system.snmp.set("SHA1", $null, $true, $null, $null, $null, "info", $null, 161, "AES128", $null, $null, "IT@myfirm.com", "My Firm IT Department", $null, "snmpUser1/9e79c8406990d1dxxxxxxx975b364b7e90e9c2ab/9e79c8406990d1dxxxxxxx975b364b7e90e9c2ab/priv", "xxx.xxx.xxx.xxx/user/priv/trap,xxx.xxx.xxx.xxx/user/priv/trap") } |
That’s all!
Posted: May 22nd, 2016 | Tags: monitoring, powercli, snmp, solarwinds | Category: Virtualization | § I wanted to enable SNMP on our ESXi hosts for monitoring via Solarwinds. Here’s what I did. (I am doing this kind of generically, using variables etc, so I can script the thing for multiple hosts).
First I connected to the vCenter Server from PowerCLI.
|
|
$vcenter = "myvcenter" $vcuser = Get-Credential Connect-VIServer -Server $vcenter -Credential $vcuser |
Next I got its ESXCLI object. This will let me run ESXCLI commands against the host.
|
|
$vmhost = "myHost" $esxcli = Get-EsxCli -VMHost $vmhost |
To view the current status of SNMP you can do can invoke a get() method –
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
> $esxcli.system.snmp.get() authentication : communities : enable : false engineid : 00000063000000a100000000 hwsrc : indications largestorage : true loglevel : info notraps : port : 161 privacy : remoteusers : syscontact : syslocation : targets : users : v3targets : |
Nothing’s configured currently. To configure something we can use the set() method. From the definition of this method we can see it takes a whole bunch of parameters –
|
|
> $esxcli.system.snmp =================== EsxCliElement: snmp Methods: -------- AgentConfig get() SNMPv3Hash hash(string authhash, string privhash, boolean rawsecret) boolean set(string authentication, string communities, boolean enable, string engineid, string hwsrc, boolean largestorage, string loglevel, string notraps, long port, string privacy, string remoteusers, boolean reset, string syscontact, string syslocation, string targets, string users, string v3targets) AgentTest test(string authhash, string privhash, boolean rawsecret, string user) |
Here’s what I did to configure SNMP. I want a community string of “public”, enable SNMP, and specify two trap destinations.
|
|
> $esxcli.system.snmp.set($null, "public", $true, $null, $null, $null, $null, $null, $null, $null, $null, $null, $null, $null, "192.x.x.x/public,192.x.x.x/public", $null, $null) |
The result of that will either be a true or false. The get() method can be used again to confirm it is set correctly. And the test() method can be used to test it works –
|
|
> $esxcli.system.snmp.test() comments -------- {There are 2 targets configured, send warmStart requested, test completed normally.} |
Now Solarwinds will be able to poll the host via SNMP.
To do this en-masse on all your hosts the following should help –
|
|
# modify this with your vCenter server name $vcenter = "myvcenter" $vcuser = Get-Credential Connect-VIServer -Server $vcenter -Credential $vcuser Get-VMHost | %{ $vmhost = $_.Name $esxcli = Get-EsxCli -VMHost $vmhost $esxcli.system.snmp.set($null, "public", $true, $null, $null, $null, $null, $null, $null, $null, $null, $null, $null, $null, "192.y.y.y/public,192.x.x.x/public", $null, $null) } |
Shout out to this VMware blog post which helped a lot and has more info.
The above script failed on some of our ESX hosts with the following error –
|
|
The method 'set' is invoked with '17' parameters, but the expected parameter count is '16'. At line:5 char:3 + $esxcli.system.snmp.set($null, "public", $true, $null, $null, $null, $null, $n ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : OperationStopped: (:) [], InvalidArgument + FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidArgument |
Turns out these hosts only accept 16 parameters instead of 17 (the one called largestorage is missing). Not sure why. All our hosts are ESXi 5.5 but am thinking the problem ones are perhaps not using the HP customized version of ESXi.
Anyways, so I modified my script above to take care of this –
|
|
# modify this with your vCenter server name $vcenter = "myvcenter" $vcuser = Get-Credential Connect-VIServer -Server $vcenter -Credential $vcuser Get-VMHost | %{ $vmhost = $_.Name Write-Host "$vmhost" $esxcli = Get-EsxCli -VMHost $vmhost $esxcli.system.snmp.set($null, "public", $true, $null, $null, $null, $null, $null, $null, $null, $null, $null, $null, $null, "192.x.x.x/public,192.y.y.y/public", $null, $null) if ($? -ne $true) { $esxcli.system.snmp.set($null, "public", $true, $null, $null, $null, $null, $null, $null, $null, $null, $null, $null, "192.x.x.x/public,192.y.y.y/public", $null, $null) } } |
Also, just for my own info – the $null above means the parameter is not set. If that parameter already has a value on the server it is not over-written. To over-write or blank out the existing value replace $null with "".
Posted: April 29th, 2016 | Tags: custom properties, monitoring, solarwinds | Category: Infrastructure | § I wanted to create account limitations in Solarwinds based on Custom Properties but the web console by default doesn’t give an option to do that.
 Then I came across this helpful video.
The trick is to logon to your Solarwinds server and find the “Account Limitation Builder”. Then click “Add” and create a new limitation similar to this –
Now the limitation you create will come in the list –
Select that, and in the next screen you can choose the value you want to limit to –
Posted: April 28th, 2016 | Tags: monitoring, solarwinds | Category: Infrastructure | § Had to remove a drive from being monitored from multiple servers in Solarwinds. Rather than go to each node, edit its properties and untick the drive, I figured that if you do a search for the nodes and expand each of them it’s possible to tick the ones you don’t need and click “Delete”.
Nice!
Posted: April 28th, 2016 | Tags: monitoring, solarwinds | Category: Infrastructure, Windows | § This is similar to how I monitored performance counters with Solarwinds.
I want to monitor a bunch of AppSense services.
Similar to the performance counters where I created an application monitor template so I could define the threshold values, here I need to create an application monitor so that the service appears in the alert manager. This part is easy (and similar to what I did for the performance counters) so I’ll skip and just put a screenshot like below –
I created a separate application monitor template but you could very well add it to one of the standard templates (if it’s a service that’s to be monitored on all nodes for instance).
Now for the part where you create alerts.
Initially I thought this would be a case of creating triggers when the above application monitor goes down. Something like this – 
And create an alert message like this –
With this I was hoping to get a one or more alert messages only for the services that actually went down. Instead, what happened is that whenever any one service went down I’d get an alert for the service that went down and also a message for the services that were up. Am guessing since I was triggering on the application monitor, Solarwinds helpfully sent the status for each of its components – up or down.
The solution is to modify your trigger such that you target each component.
Now I get alerts the way I want.
Hope this helps!
Posted: April 25th, 2016 | Tags: alerts, monitoring, solarwinds | Category: Infrastructure | § I wanted to mute Solarwinds alerts during our patch weekends when all servers are rebooted because they have to be and our mailboxes get flooded with Solarwinds alerts. I decided to use custom properties for this purpose. Here’s what I did.
Login to the Solarwinds web console. Go to the “Settings” page, and then “Manage Custom Properties” under “Node *& Group Management”.
Click “Add Custom Property”, select the default of “Nodes” from the drop down, and create something along the following lines –
Select the nodes you’d like to apply this custom property to. I chose to apply it on all my Windows and VMware nodes. Set the value to be “No”.
Now login to Orion Alerts Manager and pick an alert you’d like to mute during patch weekends. Go to its “Alert Suppression” tab and add a condition on the custom property we created earlier.
And that’s it, really!
Update: Not sure why, but the above didn’t seem to work for me. So I added the Mute_Alerts check as part of the trigger condition itself.
Note: If you don’t get the custom property in Orion, close and restart it as an administrator (i.e. right click and do “Run as Administrator” even if you are already running it with an admin account). Not sure why, but until I did that the custom property didn’t get picked up. You only need to do it one time; after that you can launch Orion normally.
Next time your server estate is being rebooted/ undergoing maintenance, login to Solarwinds webconsole and change the “Mute_Alerts” custom property to “Y” for a node/ nodes that you want to mute alerts for. Below I show how I will mute the alerts for all my Windows nodes.
Go to “Manage Nodes”. Group by “Vendor” and select Windows. Then select all nodes. (The checkbox to select all nodes got blanked out in the screenshot below but it’s easy to find).
Then click on “Custom Property Editor” to get to the screen below.
Here too select all the nodes and click “Edit multiple values”.
From the drop down, change the value for “Mute_Alerts” to “true”. Then save changes and that’s it. :)
|