Contact

Subscribe via Email

Subscribe via RSS/JSON

Categories

Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Elsewhere

Exchange DAG fails. Information Store service fails with error 2147221213.

Had an interesting issue at work today. When our Exchange servers (which are in a 2 node DAG) rebooted after patch weekend one of them had trouble starting the Information Store service. The System log had entries such as these (event ID 7024) –

The Microsoft Exchange Information Store service terminated with service-specific error %%-2147221213.

The Application log had entries such as these (event ID 5003) –

Unable to initialize the Information Store service because the clocks on the client and server are skewed. This may be caused by a time change either on the client or on the server, and may require a restart of that computer. Verify that your domain is correctly configured and  is currently online.

So it looked like time synchronization was an issue. Which is odd coz all our servers should be correctly syncing time from the Domain Controllers.

Our Exchange team fixed the issue by forcing a time sync from the DC –

I was curious as to why so went through the System logs in detail. What I saw a sequence of entries such as these –

Notice how time jumps ahead 13:21 when the OS starts to 13:27 suddenly, then jumps back to 13:22 when the Windows Time service starts and begins syncing time from my DC. It looked like this jump of 6 mins was confusing the Exchange services (understandably so). But why was this happening?

I checked the time configuration of the server –

Seems to be normal. It was set to pick time from the site DC via NTP (the first entry under TimeProviders) as well as from the ESXi host the VM is running on (the second entry – VM IC Time Provider). I didn’t think much of the second entry because I know all our VMs have the VMware Tools option to sync time from the host to VM unchecked (and I double checked it anyways).

Only one of the mailbox servers was having this jump though. The other mailbox server had a slight jump but not enough to cause any issues. While the problem server had a jump of 6 mins, the ok server had a jump of a few seconds.

I thought to check the ESXi hosts of both VMs anyways. Yes, they are not set to sync time from the host, but let’s double check the host times anyways. And bingo! turns out the ESXi hosts have NTP turned off and hence varying times. The host with the problem server was about 6 mins ahead in terms of time from the DC, while the host with the ok server was about a minute or less ahead – too coincidental to match the time jumps of the VMs!

So it looked like the Exchange servers were syncing time from the ESXi hosts even though I thought they were not supposed to. I read a bit more about this and realized my understanding of host-VM time sync was wrong (at least with VMware). When you tick/ untick the option to synchronize VM time with ESX host, all you are controlling is a periodic synchronization from host to VM. This does not control other scenarios where a VM could synchronize time with the host – such as when it moves to a different host via vMotion, has a snapshot taken, is restored from a snapshot, disk is shrinked, or (tada!) when the VMware Tools service is restarted (like when the VM is rebooted, as was the case here). Interesting.

So that explains what was happening here. When the problem server was rebooted it synced time with the ESXi host, which was 6 mins ahead of the domain time. This was before the Windows Time service kicked in. Once the Windows Time service started, it noticed the incorrect time and set it correct. This time jump confused Exchange – am thinking it didn’t confuse Exchange directly, rather one of the AD services running on the server most likely, and due to this the Information Store is unable to start.

The fix for this is to either disable VMs from synchronizing time from the ESXi host or setup NTP on all the ESXi hosts so they have the correct time going forward. I decided to go ahead with the latter.

Update: Found this and this blog post. They have more screenshots and a better explanation, so worth checking out. :)

PowerShell – Create a list of all Exchange mailboxes in an OU with mailbox size, Inbox size, etc

As the title says, here’s a one-liner to quickly create a list of all Exchange mailboxes in an OU with mailbox size, Inbox size, Sent Items size, and the number of items in each of these folders.

 

Delays with Send As permissions, Quotas, Mailbox permissions in Exchange

Send As permissions prompted this post so I’ll stick to that. But what I describe below also affects mailbox permissions, quotas, and other Exchange information stored in AD. 

Assigning someone Send As permissions in Exchange is easy. Use the EMC, or if you love PowerShell do something like this:

However, there is a catch in that the permission isn’t always granted immediately. It could be instant, but it could also take up to 2 hours.

This is because when we make a change to a user object – even of a mail related attribute such as Send As – the change isn’t immediately read by Exchange. The change is in fact made in Active Directory (notice the cmdlet above) and Exchange has to query AD to get the changed value. If Exchange were to query AD each time it needs to do something with a mail enabled user object, that would be poor performance for both Exchange and Domain Controllers. Instead, Exchange servers have a component called Directory Service Access (DSAccess) (renamed to Active Directory Access (ADAccess) since Exchange 2007) that periodically queries AD and caches the results for a period of time. When you grant someone Send As permissions as above, if Exchange’s cache isn’t updated with the latest value it will reject users trying to send email on behalf of the other user until such time the cache is updated. 

This cache is commonly referred to as Mail Box Information (MBI) Cache. 

The items in the MBI Cache are controlled by something called the Mailbox Cache Age Limit. There is a registry key HKLM\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem which has two values:

  • Mailbox Cache Age Limit – a value of type DWORD (decimal) that controls the age of the items in the cache.
    • The higher the age limit, the longer the item can stay in cache (and so won’t be refreshed). 
    • The default for this is 120 minutes (i.e. 2 hours). This is the default even if the value does not exist.
    • So this means any item present in the cache is not updated with new information for up to 2 hours – even if DSAccess/ ADAccess queries AD in between
  • Reread Logon Quotas Interval – a value of type DWORD (decimal) that controls how often the Exchange information store reads the mailbox quota information from the cache.
    • This is not relevant for the current topic but I thought to mention as it is present in the same place, and quotas are another area which commonly affect the user / administrator expectation. 
    • The default for this 7200 seconds (i.e. 2 hours). This is the default even if the value does not exist. 
    • The recommended value is 1200 seconds (i.e. 20 mins)?
    • Note that this controls how often information is read from the cache. You could read from the cache every 20 mins, but if you only update the cache every 2 hours (the default for the previous registry value) you will still get stale information! So any changes to this registry value must be done in conjunction with the previous registry value. 

To get changes made to mail objects in AD propagate faster to Exchange one must change the two values above (or at least the first one if you don’t care about quotas). 

Bear in mind though, that the above values only control how often the cache items expire or are referred to. They do not control how often DSAccess/ ADAccess queries AD for new information. This is controlled by a value at another registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchange ADAccess\Instance0:

  • CacheTTLUser – a value of type DWORD (decimal) that controls how often DSAccess/ ADAccess queries a DC for new information. 
    • The default for this is 300 seconds (i.e. 5 mins). This is the default even if the value does not exist. 
    • Lowering this value is not recommended as it will affect Exchange performance. 

Once any of these values are changed the Information Store service (or the Exchange server itself) must be restarted. 

In addition to the registry values I mention above, there’s one more worth mentioning. It isn’t too important because it’s value can never exceed that of the Mailbox Cache Age Limit, but it’s worth knowing about. In the same registry key as Mailbox Cache Age Limit – i.e. at HKLM\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem – is present another value: 

  • Mailbox Cache Idle Limit – a value of type DWORD (decimal) that controls how long an item can be idle in the cache before it is removed from the cache.  
    • The default for this is 900 seconds (i.e. 15 mins). This is the default even if the value does not exist. 
    • The maximum for this value can be up to the value of Mailbox Cache Age Limit (for obvious reasons – that’s when the item is removed anyway!).
    • This is an interesting setting. It seems to be there to introduce some randomness in how cache items are refreshed, and also to stir up the cache a bit?
      • Without it all items in the cache will expire at the same time (every 2 hours by default) and will need to be refreshed together. But with this setting some items will be periodically discarded (every 15 mins by default) if they were not accessed by the Exchange server during that interval and a fresh copy cached the next time DSAccess/ ADAccess contacts a DC. 
      • This way any object that wasn’t accessed recently will have an up-to-date copy in the cache, but objects that were accessed recently continue to be cached (and potentially have stale information). 
      • So at any given point of time the cache will have items of varying expiry time (i.e. not all will expire together in 2 hours). And the first time an item is accessed in a long time, it will possibly have the latest information. 

With this in mind one can appreciate why updates to a mail enabled object may not immediately take effect. There’s so many factors! It will depend on whether the object has past its idle time. If not, it will depend on whether the object has past its cache age. Further, if it’s a quota setting, it will also depend on the quota reread interval. Also, how often ADAccess/ DSAccess contacts a DC matters. And on top of all this, AD replication too plays a role because the change has to replicate from the DC you made it on to the DC that ADAccess/ DSAccess recently contacted to cache information.

You could be lucky in that the object you changed has just past its idle time and so ADAccess/ DSAccess will get an up-to-date copy soon. You could be lucky in that even though the object hasn’t past its idle time it’s about to expire in a few minutes and so will be refreshed. Or you could be unlucky and make a change just after the object was refreshed, and it’s not idle any time soon, so you’ll have to wait a whole 2 hours (by default) before your change takes effect! 

As Yoda would say, patience you must with Exchange and permissions. :)

p.s. While writing this post I learnt that Update Rollup 4 for Exchange 2010 SP 2 introduces the ability to automatically copy emails you send on behalf of someone to that person’s Sent Items folder too. Good to know! That’s a useful feature. 

Outlook search and additional mailboxes

Something to keep in mind regarding Outlook searches and additional mailbox. The comments below are in the context of Outlook 2010 and Exchange 2010 as that’s what we use at work.

There’s two different search providers that come into play when searching via Outlook. They are used depending on how the mailbox is accessed by Outlook.

search optionsIf the mailbox is accessed in cached mode (so the folders are actually in an OST file on your computer) or if you are searching a PSTsearch locations file, Outlook uses Windows Desktop Search (WDS) to do the search. WDS is set to index many locations on your computer and if there are OST and PST files in Outlook it indexes these too. You can tweak WDS’s options by clicking on “Search Tools” and then “Search Options”. Click “Indexing Options” in the window that opens to change the locations and file types that are indexed.

However, if the mailbox is accessed in online mode (so there are no OST files involved) Outlook passes your search query to the Exchange server who then returns with results.

This distinction doesn’t usually matter except when searching attachment contents. In that scenario WDS returns more results as it is able to index more file types. The Exchange server in contrast has a lesser number of default file types it can index and it is up to the Exchange admin to install filters for additional file types.

At work, for instance, all our users access their own mailboxes in cached mode – so search results there are more comprehensive when compared to search results from additional shared mailboxes that are accessed in online mode. The PDF file type is not included in the default file types for Exchange 2010, so shared mailbox searches don’t return emails with PDF attachments even if they contain the search term. Excel and Word file types are returned in the results though.

This is a good link to read for more info.