 © Rakhesh Sasidharan |
Had this issue today (and a few months ago). I open vCenter client, type in the vCenter server name, tick “Use Windows Session Credentials” as usual, and login fails. Says it cannot login with the given credentials. At the same time I can login with the vSphere Web Client and also by un-ticking the box and manually entering the username/ password. Fix for both times was to reset the secure channel by logging in to the vCenter server –
This big question with time travel is what sort of a music player is life. Is life the sort of music player wherein once you start a playlist in shuffle mode the music player shuffles the tracks internally and you can go back and fro among tracks and the same pre-shuffled order is maintained (i.e. the playlist appears random, but it’s not really random because the randomness is introduced just at that starting moment; so if you were to build a time machine and go back in time you can’t really change anything because you follow that set path, everything is kind of pre-determined). Or is life the sort of music player wherein once you start a playlist in shuffle mode the music player always chooses the next track to be played only when the current track reaches an end, and while you can go back to the previously played tracks (i.e. you can build a time machine and go back to past events) but once you go back to a previously played track, the next track in the list is a new randomly chosen track (i.e. once you go back in time you can change the future; things aren’t pre-determined, there is no set path). :) So here’s the situation. We have a bunch of HP rack enclosures. Some blade servers were moved from one rack to another but the person doing the move forgot to note down the new location. I knew the iLO IP address but didn’t know which enclosure it was in. Rather than login to each enclosure OA, expand the device bays and iLO info and find the blade I was interested in, I wrote this batch file that makes use of SSH/ PLINK to quickly find the enclosure the blade was in.
Put this in a batch file in the same folder as PLINK and run it. Note that this does depend on SSH access being allowed to your enclosures. Update: An alternative way if you want to use PowerShell for the looping –
I wanted to enable SNMP on our ESXi hosts for monitoring via Solarwinds. Here’s what I did. (I am doing this kind of generically, using variables etc, so I can script the thing for multiple hosts). First I connected to the vCenter Server from PowerCLI.
Next I got its ESXCLI object. This will let me run ESXCLI commands against the host.
To view the current status of SNMP you can do can invoke a
Nothing’s configured currently. To configure something we can use the
Here’s what I did to configure SNMP. I want a community string of “public”, enable SNMP, and specify two trap destinations.
The result of that will either be a
Now Solarwinds will be able to poll the host via SNMP. To do this en-masse on all your hosts the following should help –
Shout out to this VMware blog post which helped a lot and has more info. The above script failed on some of our ESX hosts with the following error –
Turns out these hosts only accept 16 parameters instead of 17 (the one called Anyways, so I modified my script above to take care of this –
Also, just for my own info – the Following on my previous post I wanted to set NTP servers for my ESX servers and also start the service & allow firewall exceptions. Here’s what I did –
Had an interesting issue at work today. When our Exchange servers (which are in a 2 node DAG) rebooted after patch weekend one of them had trouble starting the Information Store service. The System log had entries such as these (event ID 7024) –
The Application log had entries such as these (event ID 5003) –
So it looked like time synchronization was an issue. Which is odd coz all our servers should be correctly syncing time from the Domain Controllers. Our Exchange team fixed the issue by forcing a time sync from the DC –
I was curious as to why so went through the System logs in detail. What I saw a sequence of entries such as these –
Notice how time jumps ahead 13:21 when the OS starts to 13:27 suddenly, then jumps back to 13:22 when the Windows Time service starts and begins syncing time from my DC. It looked like this jump of 6 mins was confusing the Exchange services (understandably so). But why was this happening? I checked the time configuration of the server –
Seems to be normal. It was set to pick time from the site DC via NTP (the first entry under Only one of the mailbox servers was having this jump though. The other mailbox server had a slight jump but not enough to cause any issues. While the problem server had a jump of 6 mins, the ok server had a jump of a few seconds. I thought to check the ESXi hosts of both VMs anyways. Yes, they are not set to sync time from the host, but let’s double check the host times anyways. And bingo! turns out the ESXi hosts have NTP turned off and hence varying times. The host with the problem server was about 6 mins ahead in terms of time from the DC, while the host with the ok server was about a minute or less ahead – too coincidental to match the time jumps of the VMs! So it looked like the Exchange servers were syncing time from the ESXi hosts even though I thought they were not supposed to. I read a bit more about this and realized my understanding of host-VM time sync was wrong (at least with VMware). When you tick/ untick the option to synchronize VM time with ESX host, all you are controlling is a periodic synchronization from host to VM. This does not control other scenarios where a VM could synchronize time with the host – such as when it moves to a different host via vMotion, has a snapshot taken, is restored from a snapshot, disk is shrinked, or (tada!) when the VMware Tools service is restarted (like when the VM is rebooted, as was the case here). Interesting. So that explains what was happening here. When the problem server was rebooted it synced time with the ESXi host, which was 6 mins ahead of the domain time. This was before the Windows Time service kicked in. Once the Windows Time service started, it noticed the incorrect time and set it correct. This time jump confused Exchange – am thinking it didn’t confuse Exchange directly, rather one of the AD services running on the server most likely, and due to this the Information Store is unable to start. The fix for this is to either disable VMs from synchronizing time from the ESXi host or setup NTP on all the ESXi hosts so they have the correct time going forward. I decided to go ahead with the latter. Update: Found this and this blog post. They have more screenshots and a better explanation, so worth checking out. :) I saw “X-Men: Apocalypse” at the movies yesterday and loved it. Before going for the movie I had a headache; by the time I came out I was a cured man. :) I feel conscious saying I enjoyed “X-Men: Apocalypse” because most reviews I chanced upon seem to hate it. They don’t like its villain, or the themes, the continuity with other X-Men movies, it’s plot, etc – but I liked it, and probably for these same reasons really (except the continuity mess-up). It was such a relief watching “X-Men: Apocalypse” after all these complicated superhero movies like “Batman vs. Superman” or “Captain America: Civil War”. None of this heroes fighting heroes stuff – particularly for not much reason – none of this gray area-ness or questions of right & wrong and morals and whatnot. Just a simple straight-forward good vs bad superhero movie – wow, I missed those! Most superhero movies try to be dark and realistic nowadays, inspired by “The Dark Knight”. I loved “The Dark Knight” – probably my favorite super hero movie of all – but that too was more or less clear in terms of sides. Batman was good, Joker was bad, but Joker’s reasons for being bad was the cool deal as was the way he went about being bad. But there was no question in your head of choosing sides or any other issues. Simple. I wish “X-Men: Apocalypse” didn’t bring in Wolverine to confuse the timeline, but apart from that niggle it was good. I guess the best way to ignore that niggle is to ignore the Wolverine movies altogether – which is easily done as the two Wolverine movies have not much continuity between them anyways! I liked the villain Apocalypse. There have been comments of him focusing too much on dressing up or not being menacing enough – that didn’t bother me much. Yes he was dressed all grandly, but he’s from the Egyptian civilization era and they were grandly dressed back then so that’s just how he is. He was not menacing enough – I don’t know, I enjoyed watching him. The plot made sense to me as did what Apocalypse wanted. The ending was a bit of a we-need-to-destroy-him-somehow-so-the-movie-can-conclude, but that’s fine too – we get to see Phoenix and her introduction makes sense with the ending. I enjoyed the opening sequence that introduced Apocalypse. I enjoyed the title credits. I enjoyed QuickSilver and his amazing sequence. I enjoyed most of the plot. Professor Xavier going bald didn’t make sense :) but then what the heck, he had to go bald eventually and this seemed like a good place. I enjoyed Jean the subtle hint/ reference to Phoenix (without saying the name but just through the imagery). The amount of destruction in the end was a bit over the top and and repetitive (like every other mass destruction movie sequence you have seen), but you can’t call a movie Apocalypse and not have some apocalyptic stuff happening. So yeah – that’s it I guess. I loved “X-Men: Apocalypse”. A simple old-fashioned superhero movie. Good heroes vs Bad villains. No other drama. :) Just some links I found on PowerShell remoting security –
Had a request to use SolarWinds to highlight servers in a pending reboot status. Here’s what I did. Sorry, this is currently broken. After implementing this I realized I need to enable PowerShell remoting on all servers for it to work, else the script just returns the result from the SolarWinds server. Will update this post after I fix it at my workplace. If you come across this post before that, all you need to do is enable PowerShell remoting across all your servers and change the script execution to “Remote Host”.
SolarWinds has a built in application monitor called “Windows Update Monitoring”. It does a lot more than what I want so I disabled all the components I am not interested in. (I could have also just created a new application monitor, I know, just was lazy).
The part I am interested in is the PowerShell Monitor component. By default it checks for the reboot required status by checking a registry key:
Inspired by this blog post which monitors three more registry keys and also queries ConfigMgr, I replaced the default PowerShell script with the following –
Then I added the application monitor to all my Windows servers. The result is that I can see the following information on every node –
Following this I created alerts to send me an email whenever the status of the above component (“Machine restart status …”) went down for any node. And I also created a SolarWinds report to capture all nodes for which the above component was down.
Then I assigned this to a schedule to run once in a month after our patching window to email me a list of nodes that require reboots.
AppInsight from Solarwinds is pretty cool. At least the one for Exchange is. Trying out the one for IIS now. Got it configured on a few of our servers easily but it failed on one. Got the following error –
Bummer! Manual install it is then. (Or maybe not! Read on and you’ll see a hopeful fix that worked for me). First step in that is to install PowerShell (easy) and the IIS PowerShell snap-in. The latter can be downloaded from here. This downloads the Web Platform Installer (a.k.a. “webpi” for short) and that connects to the Internet to download the goods. In theory it should be easy, in practice the server doesn’t have connectivity to the Internet except via a proxy so I have to feed it that information first. Go to
This should be within the
Time to look at the logs. Go to From the logs it looked like the connection was going through –
But the problem was this –
If I go to the link – https://www.microsoft.com/web/webpi/5.0/webproductlist.xml – via IE on that server I get the following –
However, when I visit the same link on a different server there’s no error. Interesting. I viewed the untrusted certificate from IE on the problem server and compared it with the certificate from the non-problem server.  Certificate on the problem server  Certificate on a non-problem server Comparing the two I can see that the non-problem server has a VeriSign certificate in the root of the path, because of which there’s a chain of trust.
If I open Certificate Manager on both servers (open
So here’s what I did. :) I exported the certificate from the server that had it and imported it into the “Trusted Root Certification Authorities” store of the problem server. Then I closed and opened IE and went to the link again, and bingo! the website opens without any issues. Then I tried the Web Platform Installer again and this time it loads. Bam! The problem though is that it can’t find the IIS PowerShell snap-in. Grr!
That sucks! However, at this point I had an idea. The SolarWinds error message was about an invalid signature, and what do we know of that can cause an invalid signature? Certificate issues! So now that I have installed the required CA certificate for the Web Platform Installer, maybe it sorts out SolarWinds too? So I went back and clicked “Configure Server” again and bingo! it worked this time. :) Hope this helps someone. Added the Exchange 2010 Database Availability Group application monitor to couple of our Exchange 2010 servers and got the following error –
Clicking “More” gives the following –
This is because Solarwinds is trying to run a PowerShell script on the remote server and the script is unable to run due to authentication errors. That’s because Solarwinds is trying to connect to the server using its IP address, and so instead of using Kerberos authentication it resorts to Negotiate authentication (which is disabled). The error message too says the same but you can verify it for yourself from the Solarwinds server too. Try the following command –
This is what’s happening behind the scenes and as you will see it fails. Now replace “Negotiate” with “Kerberos” and it succeeds –
So, how to fix this? Logon to the remote server and launch IIS Manager. It’s under “Administrative Tools” and may not be there by default (my server only had “Internet Information Services (IIS) 6.0 Manager”), in which case add it via Server Manager/ PowerShell –
Then open IIS Manager, go to Sites > PowerShell and double click “Authentication”.
Select “Windows Authentication” and click “Enable”.
Now Solarwinds will work. I had to create multiple A DNS records of the format below –
Just 9 records, I could create them manually, but I thought let’s try and create en-mass using PowerShell. If you are on Windows Server 2012 and above, you have PowerShell cmdlets for DNS. So I did the following –
To confirm they are created, the following helps –
Nice! |
