## [Aside] Exporting Exchange mailboxes to PST (with the ‘ContentFilter’ date format fixed)

Quick shoutout to this blog post by Tony (cached copy as I can’t access the original). The cmdlet has a -ContentFilter switch where you can specify the date and other parameters by which you can filter out what is exported. Very irritatingly the date parameter expects to be in US format. And even if you specify it in US format to begin with, it converts it to US format again switching the date and month and complaining if it’s an incorrect date. Thanks to this blog post which explained this to me and this blog post for a super cool trick to work around this. Thank you all!

## Exchange 2013 Restore and Operation terminated with error -1216 errors

You know how when you are nervous about something and so try to do a perfect job and hope for the best, things will inevitably go wrong! Yeah, they don’t lie about that. :)

I had to do an Exchange 2013 restore. My first time, to be honest, if you don’t count the one time I restored a folder in my home lab. And things didn’t go as smoothly as one would have hoped for. But hey it’s not a bad thing coz I get to learn a lot in the process. So no complaints, but it has been a nervous experience.

Initially I had a lot of DPM troubles restoring. These were solved by attaching my Azure recovery vault to a different server and doing a restore from there. In the process though I didn’t get DPM to clean up and mount the database directly on Exchange; rather I asked it to restore as files to a location and I decided to do the restore from there.

It is pretty straightforward according to the docs

1. Restore the database files.
2. Bring the files of the restored database to a clean shutdown state
3. Create a new recovery database, pointing to the previously cleaned up database files.
4. Mount database.
5. Do a restore request.

That’s all!

Here’s some more useful details I picked up along the way, including an error that caused some delays for me.

### 1 Restore all files to a location

Now, this is important. After restoring *do not* rename the database file. I did that and it an “Operation terminated with error -1216 (JET_errAttachedDatabaseMismatch)” error. You don’t want that! Keep the database name as is. Keep the folder names too as is, it doesn’t really matter.

### 2.1 Check the database state

You run the eseutil /mh “\path\to\edb\file” command for this.

Two things to note. First, the state – “Dirty Shutdown” – so I need to clean up the database. And second, the logs required. If I check the restored folder I should see files of this name there. In my case I did have files along these lines:

### 2.2 Confirm that the logs are in a good state

You run the eseutil /mh “\path\to\log files\EXX” command for this (where XX is the log prefix).

All looks good.

### 2.3 Bring the database files to a clean shutdown state

For this you run eseutil /R EXX /l “\path\to\log\files” /s “\path\to\edb\file” /d “\path\to\edb\file” (where XX is the log prefix). I remember the /lsd switches coz they look like LSD. :) They specify the path to the log files, the system files (this can be skipped in which case the current directory is used; I prefer to point this to the recovered folder), and the database file.

Looks good!

The first few times I ran this command I got the following error –

I went on a tangent due to this. :( Found many outdated blog posts (e.g. this one) that suggest adding a “/i” switch because apparently the error is caused by a mis-match with the STM database. I should have remembered STM databases don’t exist in 2013 (they were discontinued in 2010 I think, but I could be wrong; they used to be the database where you stored attachments). When I added the “/i” switch the operation succeeded but the database remained in a “dirty shutdown” state and subsequent runs of the eseutil command without this switch didn’t make any difference. I had to start from scratch (luckily I was working on a copy of the restored files, so didn’t have to do a fresh restore!).

Later I had the smart idea of checking the event logs to see why this was happening. And there I saw entries like this:

You see I had the “smart” idea of renaming my database file to something else. It was a stupid idea because the database still had references to the old name and that’s why this was failing. None of the blog posts or articles I read did a database rename, but none of them said don’t do it either … so how was I to know? :)

Anyways, if you haven’t renamed your database you should be fine. You could still get this or any other error … but again, don’t be an idiot like me and go Googling. Check your event logs. That’s what I usually do when I am sensible, just that being nervous and all here I skipped my first principles.

### 2.4 Confirm the database state is clean

Run the eseutil /mh command of step 2.1 and now it should say “Clean Shutdown”.

At this point your recovered database files are in a clean state. Now to the next step!

### 3 Create a new recovery database, pointing to the restored database files.

Do this in the Exchange Management Shell.

You will get a warning about restarting the Information Store service. Don’t do it! Restarting the Information Store will cause all Outlook connections to that server to disconnect and reconnect – you definitely don’t want that in a production environment! :) Restarting is only required if you are adding a new production database and Exchange has to allocate memory for this database. Exchange 2013 does this when the Information Store is (re)started. We are dealing with a Recovery Database here and it will be deleted soon – so don’t bother.

Also, remember that Recovery databases don’t appear in EAC so don’t worry if you don’t see it there.

If you want to read more on Recovery databases this is a good article

Again in EMS:

### 5 Do a restore request

Still in EMS:

I am restoring a folder called “Contacts” from “User Mailbox” back to “User Mailbox” under a folder called “RecoveredContacts”.

You can keep an eye on the restore request thus:

And once done you can remove all completed requests:

That’s it!

You can also remove the Recovery database now. Dismount it first.

This doesn’t remove the EDB files or logs, so remove them manually

My first baby step production Exchange 2013 restore. Finally! :)

## [Aside] Exchange Mutual TLS

For incoming connections: a) specify the domain as being secure (i.e. requires TLS) via something like this –

Do the above on the Mailbox server. You can force a sync to edge after that via Start-EdgeSynchronization on the Mailbox server.

Then b) on the Edge server enable domain secured and TLS (they are likely to be already enabled by default).

## Etisalat blocks outgoing port 25

There, I’ve said it. Hopefully anyone else Googling for this will find it now!

I spent a better part of yesterday checking all my firewalls because I couldn’t connect to port 25 of public mail servers from my VM. As a SysAdmin I have an Exchange VM etc. in my home lab that I wanted to setup for outgoing email … and when I couldn’t connect to port 25 I wondered whether there was something wrong with my network configuration or firewalls (being a VM there are layers of networking and firewalls involved). Turns out none of these is the case. My host itself cannot connect to port 25. The ISP is blocking it – simple!

This is Etisalat at home. Things might be different for business connections of course. And, this is nothing against Etisalat – many ISPs apparently do this to block spam, so this is standard practice. First time I noticed that’s all.

## Exchange DAG fails. Information Store service fails with error 2147221213.

Had an interesting issue at work today. When our Exchange servers (which are in a 2 node DAG) rebooted after patch weekend one of them had trouble starting the Information Store service. The System log had entries such as these (event ID 7024) –

The Microsoft Exchange Information Store service terminated with service-specific error %%-2147221213.

The Application log had entries such as these (event ID 5003) –

Unable to initialize the Information Store service because the clocks on the client and server are skewed. This may be caused by a time change either on the client or on the server, and may require a restart of that computer. Verify that your domain is correctly configured and  is currently online.

So it looked like time synchronization was an issue. Which is odd coz all our servers should be correctly syncing time from the Domain Controllers.

Our Exchange team fixed the issue by forcing a time sync from the DC –

I was curious as to why so went through the System logs in detail. What I saw a sequence of entries such as these –

Notice how time jumps ahead 13:21 when the OS starts to 13:27 suddenly, then jumps back to 13:22 when the Windows Time service starts and begins syncing time from my DC. It looked like this jump of 6 mins was confusing the Exchange services (understandably so). But why was this happening?

I checked the time configuration of the server –

Seems to be normal. It was set to pick time from the site DC via NTP (the first entry under TimeProviders) as well as from the ESXi host the VM is running on (the second entry – VM IC Time Provider). I didn’t think much of the second entry because I know all our VMs have the VMware Tools option to sync time from the host to VM unchecked (and I double checked it anyways).

Only one of the mailbox servers was having this jump though. The other mailbox server had a slight jump but not enough to cause any issues. While the problem server had a jump of 6 mins, the ok server had a jump of a few seconds.

I thought to check the ESXi hosts of both VMs anyways. Yes, they are not set to sync time from the host, but let’s double check the host times anyways. And bingo! turns out the ESXi hosts have NTP turned off and hence varying times. The host with the problem server was about 6 mins ahead in terms of time from the DC, while the host with the ok server was about a minute or less ahead – too coincidental to match the time jumps of the VMs!

So it looked like the Exchange servers were syncing time from the ESXi hosts even though I thought they were not supposed to. I read a bit more about this and realized my understanding of host-VM time sync was wrong (at least with VMware). When you tick/ untick the option to synchronize VM time with ESX host, all you are controlling is a periodic synchronization from host to VM. This does not control other scenarios where a VM could synchronize time with the host – such as when it moves to a different host via vMotion, has a snapshot taken, is restored from a snapshot, disk is shrinked, or (tada!) when the VMware Tools service is restarted (like when the VM is rebooted, as was the case here). Interesting.

So that explains what was happening here. When the problem server was rebooted it synced time with the ESXi host, which was 6 mins ahead of the domain time. This was before the Windows Time service kicked in. Once the Windows Time service started, it noticed the incorrect time and set it correct. This time jump confused Exchange – am thinking it didn’t confuse Exchange directly, rather one of the AD services running on the server most likely, and due to this the Information Store is unable to start.

The fix for this is to either disable VMs from synchronizing time from the ESXi host or setup NTP on all the ESXi hosts so they have the correct time going forward. I decided to go ahead with the latter.

Update: Found this and this blog post. They have more screenshots and a better explanation, so worth checking out. :)

## PowerShell – Create a list of all Exchange mailboxes in an OU with mailbox size, Inbox size, etc

As the title says, here’s a one-liner to quickly create a list of all Exchange mailboxes in an OU with mailbox size, Inbox size, Sent Items size, and the number of items in each of these folders.

## Delays with Send As permissions, Quotas, Mailbox permissions in Exchange

Send As permissions prompted this post so I’ll stick to that. But what I describe below also affects mailbox permissions, quotas, and other Exchange information stored in AD.

Assigning someone Send As permissions in Exchange is easy. Use the EMC, or if you love PowerShell do something like this:

However, there is a catch in that the permission isn’t always granted immediately. It could be instant, but it could also take up to 2 hours.

This is because when we make a change to a user object – even of a mail related attribute such as Send As – the change isn’t immediately read by Exchange. The change is in fact made in Active Directory (notice the cmdlet above) and Exchange has to query AD to get the changed value. If Exchange were to query AD each time it needs to do something with a mail enabled user object, that would be poor performance for both Exchange and Domain Controllers. Instead, Exchange servers have a component called Directory Service Access (DSAccess) (renamed to Active Directory Access (ADAccess) since Exchange 2007) that periodically queries AD and caches the results for a period of time. When you grant someone Send As permissions as above, if Exchange’s cache isn’t updated with the latest value it will reject users trying to send email on behalf of the other user until such time the cache is updated.

This cache is commonly referred to as Mail Box Information (MBI) Cache.

The items in the MBI Cache are controlled by something called the Mailbox Cache Age Limit. There is a registry key HKLM\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem which has two values:

• Mailbox Cache Age Limit – a value of type DWORD (decimal) that controls the age of the items in the cache.
• The higher the age limit, the longer the item can stay in cache (and so won’t be refreshed).
• The default for this is 120 minutes (i.e. 2 hours). This is the default even if the value does not exist.
• So this means any item present in the cache is not updated with new information for up to 2 hours – even if DSAccess/ ADAccess queries AD in between
• Reread Logon Quotas Interval – a value of type DWORD (decimal) that controls how often the Exchange information store reads the mailbox quota information from the cache.
• This is not relevant for the current topic but I thought to mention as it is present in the same place, and quotas are another area which commonly affect the user / administrator expectation.
• The default for this 7200 seconds (i.e. 2 hours). This is the default even if the value does not exist.
• The recommended value is 1200 seconds (i.e. 20 mins)?
• Note that this controls how often information is read from the cache. You could read from the cache every 20 mins, but if you only update the cache every 2 hours (the default for the previous registry value) you will still get stale information! So any changes to this registry value must be done in conjunction with the previous registry value.

To get changes made to mail objects in AD propagate faster to Exchange one must change the two values above (or at least the first one if you don’t care about quotas).

Bear in mind though, that the above values only control how often the cache items expire or are referred to. They do not control how often DSAccess/ ADAccess queries AD for new information. This is controlled by a value at another registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchange ADAccess\Instance0:

• CacheTTLUser – a value of type DWORD (decimal) that controls how often DSAccess/ ADAccess queries a DC for new information.
• The default for this is 300 seconds (i.e. 5 mins). This is the default even if the value does not exist.
• Lowering this value is not recommended as it will affect Exchange performance.

Once any of these values are changed the Information Store service (or the Exchange server itself) must be restarted.

In addition to the registry values I mention above, there’s one more worth mentioning. It isn’t too important because it’s value can never exceed that of the Mailbox Cache Age Limit, but it’s worth knowing about. In the same registry key as Mailbox Cache Age Limit – i.e. at HKLM\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem – is present another value:

• Mailbox Cache Idle Limit – a value of type DWORD (decimal) that controls how long an item can be idle in the cache before it is removed from the cache.
• The default for this is 900 seconds (i.e. 15 mins). This is the default even if the value does not exist.
• The maximum for this value can be up to the value of Mailbox Cache Age Limit (for obvious reasons – that’s when the item is removed anyway!).
• This is an interesting setting. It seems to be there to introduce some randomness in how cache items are refreshed, and also to stir up the cache a bit?
• Without it all items in the cache will expire at the same time (every 2 hours by default) and will need to be refreshed together. But with this setting some items will be periodically discarded (every 15 mins by default) if they were not accessed by the Exchange server during that interval and a fresh copy cached the next time DSAccess/ ADAccess contacts a DC.
• This way any object that wasn’t accessed recently will have an up-to-date copy in the cache, but objects that were accessed recently continue to be cached (and potentially have stale information).
• So at any given point of time the cache will have items of varying expiry time (i.e. not all will expire together in 2 hours). And the first time an item is accessed in a long time, it will possibly have the latest information.

With this in mind one can appreciate why updates to a mail enabled object may not immediately take effect. There’s so many factors! It will depend on whether the object has past its idle time. If not, it will depend on whether the object has past its cache age. Further, if it’s a quota setting, it will also depend on the quota reread interval. Also, how often ADAccess/ DSAccess contacts a DC matters. And on top of all this, AD replication too plays a role because the change has to replicate from the DC you made it on to the DC that ADAccess/ DSAccess recently contacted to cache information.

You could be lucky in that the object you changed has just past its idle time and so ADAccess/ DSAccess will get an up-to-date copy soon. You could be lucky in that even though the object hasn’t past its idle time it’s about to expire in a few minutes and so will be refreshed. Or you could be unlucky and make a change just after the object was refreshed, and it’s not idle any time soon, so you’ll have to wait a whole 2 hours (by default) before your change takes effect!

As Yoda would say, patience you must with Exchange and permissions. :)

p.s. While writing this post I learnt that Update Rollup 4 for Exchange 2010 SP 2 introduces the ability to automatically copy emails you send on behalf of someone to that person’s Sent Items folder too. Good to know! That’s a useful feature.

## Outlook search and additional mailboxes

Something to keep in mind regarding Outlook searches and additional mailbox. The comments below are in the context of Outlook 2010 and Exchange 2010 as that’s what we use at work.

There’s two different search providers that come into play when searching via Outlook. They are used depending on how the mailbox is accessed by Outlook.

If the mailbox is accessed in cached mode (so the folders are actually in an OST file on your computer) or if you are searching a PST file, Outlook uses Windows Desktop Search (WDS) to do the search. WDS is set to index many locations on your computer and if there are OST and PST files in Outlook it indexes these too. You can tweak WDS’s options by clicking on “Search Tools” and then “Search Options”. Click “Indexing Options” in the window that opens to change the locations and file types that are indexed.

However, if the mailbox is accessed in online mode (so there are no OST files involved) Outlook passes your search query to the Exchange server who then returns with results.

This distinction doesn’t usually matter except when searching attachment contents. In that scenario WDS returns more results as it is able to index more file types. The Exchange server in contrast has a lesser number of default file types it can index and it is up to the Exchange admin to install filters for additional file types.

At work, for instance, all our users access their own mailboxes in cached mode – so search results there are more comprehensive when compared to search results from additional shared mailboxes that are accessed in online mode. The PDF file type is not included in the default file types for Exchange 2010, so shared mailbox searches don’t return emails with PDF attachments even if they contain the search term. Excel and Word file types are returned in the results though.