## [Aside] Exporting Exchange mailboxes to PST (with the ‘ContentFilter’ date format fixed)

Quick shoutout to this blog post by Tony (cached copy as I can’t access the original). The cmdlet has a -ContentFilter switch where you can specify the date and other parameters by which you can filter out what is exported. Very irritatingly the date parameter expects to be in US format. And even if you specify it in US format to begin with, it converts it to US format again switching the date and month and complaining if it’s an incorrect date. Thanks to this blog post which explained this to me and this blog post for a super cool trick to work around this. Thank you all!

## Exchange 2013 Restore and Operation terminated with error -1216 errors

You know how when you are nervous about something and so try to do a perfect job and hope for the best, things will inevitably go wrong! Yeah, they don’t lie about that. :)

I had to do an Exchange 2013 restore. My first time, to be honest, if you don’t count the one time I restored a folder in my home lab. And things didn’t go as smoothly as one would have hoped for. But hey it’s not a bad thing coz I get to learn a lot in the process. So no complaints, but it has been a nervous experience.

Initially I had a lot of DPM troubles restoring. These were solved by attaching my Azure recovery vault to a different server and doing a restore from there. In the process though I didn’t get DPM to clean up and mount the database directly on Exchange; rather I asked it to restore as files to a location and I decided to do the restore from there.

It is pretty straightforward according to the docs

1. Restore the database files.
2. Bring the files of the restored database to a clean shutdown state
3. Create a new recovery database, pointing to the previously cleaned up database files.
4. Mount database.
5. Do a restore request.

That’s all!

Here’s some more useful details I picked up along the way, including an error that caused some delays for me.

### 1 Restore all files to a location

Now, this is important. After restoring *do not* rename the database file. I did that and it an “Operation terminated with error -1216 (JET_errAttachedDatabaseMismatch)” error. You don’t want that! Keep the database name as is. Keep the folder names too as is, it doesn’t really matter.

### 2.1 Check the database state

You run the eseutil /mh “\path\to\edb\file” command for this.

Two things to note. First, the state – “Dirty Shutdown” – so I need to clean up the database. And second, the logs required. If I check the restored folder I should see files of this name there. In my case I did have files along these lines:

### 2.2 Confirm that the logs are in a good state

You run the eseutil /mh “\path\to\log files\EXX” command for this (where XX is the log prefix).

All looks good.

### 2.3 Bring the database files to a clean shutdown state

For this you run eseutil /R EXX /l “\path\to\log\files” /s “\path\to\edb\file” /d “\path\to\edb\file” (where XX is the log prefix). I remember the /lsd switches coz they look like LSD. :) They specify the path to the log files, the system files (this can be skipped in which case the current directory is used; I prefer to point this to the recovered folder), and the database file.

Looks good!

The first few times I ran this command I got the following error –

I went on a tangent due to this. :( Found many outdated blog posts (e.g. this one) that suggest adding a “/i” switch because apparently the error is caused by a mis-match with the STM database. I should have remembered STM databases don’t exist in 2013 (they were discontinued in 2010 I think, but I could be wrong; they used to be the database where you stored attachments). When I added the “/i” switch the operation succeeded but the database remained in a “dirty shutdown” state and subsequent runs of the eseutil command without this switch didn’t make any difference. I had to start from scratch (luckily I was working on a copy of the restored files, so didn’t have to do a fresh restore!).

Later I had the smart idea of checking the event logs to see why this was happening. And there I saw entries like this:

You see I had the “smart” idea of renaming my database file to something else. It was a stupid idea because the database still had references to the old name and that’s why this was failing. None of the blog posts or articles I read did a database rename, but none of them said don’t do it either … so how was I to know? :)

Anyways, if you haven’t renamed your database you should be fine. You could still get this or any other error … but again, don’t be an idiot like me and go Googling. Check your event logs. That’s what I usually do when I am sensible, just that being nervous and all here I skipped my first principles.

### 2.4 Confirm the database state is clean

Run the eseutil /mh command of step 2.1 and now it should say “Clean Shutdown”.

At this point your recovered database files are in a clean state. Now to the next step!

### 3 Create a new recovery database, pointing to the restored database files.

Do this in the Exchange Management Shell.

You will get a warning about restarting the Information Store service. Don’t do it! Restarting the Information Store will cause all Outlook connections to that server to disconnect and reconnect – you definitely don’t want that in a production environment! :) Restarting is only required if you are adding a new production database and Exchange has to allocate memory for this database. Exchange 2013 does this when the Information Store is (re)started. We are dealing with a Recovery Database here and it will be deleted soon – so don’t bother.

Also, remember that Recovery databases don’t appear in EAC so don’t worry if you don’t see it there.

If you want to read more on Recovery databases this is a good article

Again in EMS:

### 5 Do a restore request

Still in EMS:

I am restoring a folder called “Contacts” from “User Mailbox” back to “User Mailbox” under a folder called “RecoveredContacts”.

You can keep an eye on the restore request thus:

And once done you can remove all completed requests:

That’s it!

You can also remove the Recovery database now. Dismount it first.

This doesn’t remove the EDB files or logs, so remove them manually

My first baby step production Exchange 2013 restore. Finally! :)

## Edge Subscription process (and some extra info)

It’s documented well, I know. This is just for my quick reference.

To create a new edge subscription, type the following on edge –

Copy the file to a Mailbox servers (just one is fine, the rest will sync from this). Then –

Change the file paths & site etc. as required. To remove edge subscriptions, either on Edge or on Mailbox –

To set the FQDN on the outgoing (send) connector in response to EHLO do this on the Mailbox server –

If you want to change the incoming (receive) connector name run this on the Edge server –

You can find the certificates on the Edge server via Get-ExchangeCertificate. To use one of them as the certificate for the SMTP service do –

## Disabling Exchange 2013 Managed Availability monitors

Check out this blog post from Microsoft first. Mine’s mostly based on that but tailored to my specific situation.

We have a CAS server that’s purely for internal admin ECP functions. Managed Availability was running some ActiveSync tests on it and failing (because they don’t exist) with errors like these in SCOM:

So my mission, which I’ve chosen to accept (I saw “Mission Impossible: Fallout” this weekend!) is to disable this. :)

Managed Availability has the concept of health sets. From this page which lists all the health sets in Exchange 2013:

In Managed Availability, each component in Exchange 2013 monitors itself using probes, monitors and responders. Each Exchange 2013 component that implements Managed Availability is referred to as a health set.

So what are the unhealthy health sets on my server?

The result of this in my case are the following:

So how do I find which monitors in these health sets are failing? The following cmdlet can help:

I’ll just pipe both cmdlets to get a list of monitors across all unhealthy health sets:

In my case I get:

Should probably have filtered to just the unhealthy monitors:

Update: Re-reading this later, I realize this is all complicate and wrong. All I really need to do is Get-ServerHealth -Identity <server name>. I don’t need to loop over each health set.

Anyways, the SCOM error referred to an EAS component, but I don’t see anything with that name. ActiveSyncProxy is probably the one it was referring to?

As an aside, if I want to see the components of a health set (i.e. the monitors, probes, responders) I can do the following:

In the case of the ActiveSync.Proxy health set (which has the ActiveSyncProxy component) I can see:

Note that the ActiveSyncProxyTestMonitor monitor is what was showing as unhealthy earlier.

To disable a monitor I need to use the Add-ServerMonitoringOverride cmdlet. This is of the format:

In my case, to disable ActiveSync.Proxy (health set) ActiveSyncProxyTestMonitor (monitoring item – you can see this in the list of unhealthy monitors as well as in the list above) I do:

That’s it. Wait a while and now it will appear as disabled.

Next thing is how do I find out why the ActiveSync health set is unhealthy? Let’s take a  look at the probes in that:

I can invoke the probe manually using the following cmdlet:

Pipe this out as a list to read better. Here’s what I did:

The output gives the errors encountered by the tests. I could see that it was related to EAS so decided to disable it too.

Lastly, if you are curious as to what overrides exist the following cmdlet will help:

Also, if you want to double check that a particular component on the Exchange server is inactive (and that’s why monitors are failing) the following cmdlet will help (I sort it by state for easy reading but that’s optional):

The last section of the article I referred to at the beginning of this post on editing C:\Program Files\Microsoft\Exchange Server\V15\Bin\Monitoring\Config\ClientAccessProxyTest.xml to disable certain probes. Not sure why they suggest that instead of disabling probes via the cmdlet – I think that’s because the cmdlets way is more of a temporary thing (for a certain duration) while modify the config file is a permanent fix. I should probably do the config file in my case.

## [Aside] Various Exchange 2013 links

I am reading up on Exchange 2013 nowadays (yes, I know, bit late in the day to be doing that considering it is going out of support :) and these are some links I want to put here as a bookmark to myself. Some excellent blog posts and videos that detail the changes in Exchange 2013.

(By way of background: I am not an Exchange admin. I am Exchange 2010 certified as I have a huge interest in Exchange and as part of preparing for the certification I had attended a course and setup a lab on my laptop and even begun this blog to start posting about my adventures with it. I never got to work with Exchange 2010 at work – except as a helpdesk administrator one could say – but I am familiar with the concepts even though I have forgotten more than I remember. I have dabbled with Exchange 2000 before that. Going through these links and videos is like a trip down memory line – seeing concepts that I was once familiar with but have since changed for the better. Hopefully this time around I get to do more Exchange 2013 work! Fingers crossed).

### Preferred Architecture

From the preferred architecture link I like to highlight this point about DAG design as I wasn’t aware of it (PA == Preferred Architecture; this is also discussed in the video):

Data resiliency is achieved by deploying multiple database copies. In the PA, database copies are distributed across the site resilient datacenter pair, thereby ensuring that mailbox data is protected from software, hardware and even datacenter failures.

Each database has four copies, with two copies in each datacenter, which means at a minimum, the PA requires four servers. Out of these four copies, three of them are configured as highly available. The fourth copy (the copy with the highest Activation Preference) is configured as a lagged database copy. Due to the server design, each copy of a database is isolated from its other copies, thereby reducing failure domains and increasing the overall availability of the solution as discussed in DAG: Beyond the “A”.

The purpose of the lagged database copy is to provide a recovery mechanism for the rare event of system-wide, catastrophic logical corruption. It is not intended for individual mailbox recovery or mailbox item recovery.

The lagged database copy is configured with a seven day ReplayLagTime. In addition, the Replay Lag Manager is also enabled to provide dynamic log file play down for lagged copies. This feature ensures that the lagged database copy can be automatically played down and made highly available in the following scenarios:

• When a low disk space threshold is reached
• When the lagged copy has physical corruption and needs to be page patched
• When there are fewer than three available healthy copies (active or passive) for more than 24 hours

By using the lagged database copy in this manner, it is important to understand that the lagged database copy is not a guaranteed point-in-time backup. The lagged database copy will have an availability threshold, typically around 90%, due to periods where the disk containing a lagged copy is lost due to disk failure, the lagged copy becoming an HA copy (due to automatic play down), as well as, the periods where the lagged database copy is re-building the replay queue.

With all of these technologies in play, traditional backups are unnecessary; as a result, the PA leverages Exchange Native Data Protection.

The last line made me smile. Never thought I’d read someplace that backups for Exchange are unnecessary! :) If you have a lagged copy database, then you can enable circular logging on the database (this only affects the non-lagged copies) and skip taking backups – or at least not worry about the database dismounting because your backups are failing and logs are filling up disk space!

So what’s a lagged database copy? Basically it’s a copy of the database (in a DAG) that lags behind other members by a specified duration (maximum is 14 days). So if the other servers in your DAG have some issue, rather than restore the database from backup you can simply “play down” the lagged database copy (i.e. tell that copy to process all the transaction logs it already has an thus become up-to-date) and activate it. Neat, huh. I want to delve a bit more into this, so check out this “Lagged copy enhancements” section from the Exchange 2013 HA improvements page.

First there’s Safety Net (it’s not related to lagged copies, but it plays along well with it in a cool way so worth pointing out):

Safety Net is a feature of transport that replaces the Exchange 2010 feature known as transport dumpster. Safety Net is similar to transport dumpster, in that it’s a delivery queue that’s associated with the Transport service on a Mailbox server. This queue stores copies of messages that were successfully delivered to the active mailbox database on the Mailbox server. Each active mailbox database on the Mailbox server has its own queue that stores copies of the delivered messages. You can specify how long Safety Net stores copies of the successfully delivered messages before they expire and are automatically deleted.

Ok – so each mailbox server has a queue for each of its active database (remember lagged copies are active too, just that they have a higher number and hence not preferred). This queue contains messages that were delivered. Even after a message is delivered to a user, Safety Net can keep it around. You get to specify how long a message is kept for. Cool! Next up is this cool integration:

With the introduction of Safety Net, activating a lagged database copy becomes significantly easier. For example, consider a lagged copy that has a 2-day replay lag. In that case, you would configure Safety Net for a period of 2 days. If you encounter a situation in which you need to use your lagged copy, you can suspend replication to it, and copy it twice (to preserve the lagged nature of the database and to create an extra copy in case you need it). Then, take a copy and discard all the log files, except for those in the required range. Mount the copy, which triggers an automatic request to Safety Net to redeliver the last two days of mail. With Safety Net, you don’t need to hunt for where the point of corruption was introduced. You get the last two days mail, minus the data ordinarily lost on a lossy failover.

Whoa! So when a lagged copy is mounted, it asks Safety Net to redeliver all messages in the specified period – so as long as your Safety Net and lagged database copy have the same period, if you mount the lagged copy from the specified period ago, Safety Net will deliver all the messages since then. (It’s cool, but yeah I can imagine users complaining about a whole bunch of unread messages now, and missing Sent Items etc. – but it’s cool, I like it for the geek factor). :)

To re-emphasize something that was mentioned earlier:

Lagged copies can now care for themselves by invoking automatic log replay to play down the log files in certain scenarios:

• When a low disk space threshold is reached
• When the lagged copy has physical corruption and needs to be page patched
• When there are fewer than three available healthy copies (active or passive only; lagged database copies are not counted) for more than 24 hours

Lagged copy play down behavior is disabled by default, and can be enabled by running the following command.

After being enabled, play down occurs when there are fewer than three copies. You can change the default value of 3, by modifying the following DWORD registry value.

HKLM\Software\Microsoft\ExchangeServer\v15\Replay\Parameters\ReplayLagManagerNumAvailableCopies

To enable play down for low disk space thresholds, you must configure the following registry entry.

HKLM\Software\Microsoft\ExchangeServer\v15\Replay\Parameters\ReplayLagLowSpacePlaydownThresholdInMB

After configuring either of these registry settings, restart the Microsoft Exchange DAG Management service for the changes to take effect.

As an example, consider an environment where a given database has 4 copies (3 highly available copies and 1 lagged copy), and the default setting is used for ReplayLagManagerNumAvailableCopies. If a non-lagged copy is out-of-service for any reason (for example, it is suspended, etc.) then the lagged copy will automatically play down its log files in 24 hours.

For future reference this doc has steps on how to mount a lagged database copy – i.e. if you are not doing the automatic play down behavior. You can manually play down via the Move-ActiveMailboxDatabase cmdlet with the -SkipLagChecks switch.

However, it is recommended you first suspend the copy (i.e. make it not “active”) and make a copy of the database and logs just in case.

Optionally, if you want to recover to a specific point in time you’d 1) suspend the database, 2) make a copy just in case, 3) move elsewhere all log files after the time you want to recover, 4) delete the checkpoint file, 5) run eseutil to recover the database – this is what replays the remaining logs and brings the database up to the point in time you want, and 6) move the database elsewhere to use as a recovery database for a restore. After this you move back the logs file previously moved away, and resume the database copy. This blog post has a bit more details but it is more or less same as the Microsoft doc. Note: I’ve never ever done this, so all this more of info for future me. :)

Lastly, that doc also has info on how activate a logged copy using Safety Net. Step 4 of the instructions made no sense to me.

Moving on … (but pointing to this HA TechNet link again coz it has a lot of other info that I skipped here).

### Outlook Anywhere & OWA behind a WAP server

Some links around publishing Exchange namespaces such as OWA and Outlook Anywhere externally via a WAP server:

The easiest thing to do is pass-through everything via the WAP to the internal URL. But if you want, you can setup OWA authentication via ADFS claims. A step-by-step official guide is here, but the two links above cover the same stuff.

### Healthcheck.htm

Exchange 2013 has a new monitoring architecture. When monitoring via a load balancer one can use a “healthcheck.htm” URL to test the health of each virtual directory (corresponding to each of the user consumed services). This URL is per virtual directoy, here’s an example from Citrix on how to add monitors for each service in NetScaler:

If the service is up the URL returns an HTTP 200 OK.

### Virtual Directory cmdlets

Speaking of virtual directories, if any of the PowerShell Get- cmdlets for virtual directories are slow this blog post tells you why and what to do about it. These are the cmdlets, and the workaround is to add a switch -ADPropertiesOnly (this makes the cmdlet query AD Config partition for the same info rather than query each server’s IIS Metabase, which is slower):

• Get-WebServicesVirtualDirectory
• Get-OwaVirtualDirectory
• Get-ActiveSyncVirtualDirectory
• Get-AutodiscoverVirtualDirectory
• Get-EcpVirtualDirectory
• Get-PowerShellVirtualDirectory
• Get-OABvirtualDirectory
• Get-OutlookAnywhere

Update: Thought I’d add more videos and links to this post than make separate posts.

## Transport Architecture

Check out this talk: https://channel9.msdn.com/events/TechEd/2013/OUC-B319. Slides available online here. I wanted to put a screenshot of the transport components as a quick reference to myself in this post:

So the CAS has stateless SMTP service. Affectionately called FET, or Front-End Transport.

The MBX has a stateful and stateless SMTP service. Called Transport and Mailbox Transport respectively. (Transport replaces the Hub Transport role of Exchange 2010).

There’s no longer a Hub Transport role. Previously the Hub Transport role on one server could directly talk to the store of another server – thus there were no clear layers. Everything was messy and tied up using RPC. Now there are clear layers as below and communication between servers happen at the protocol layer. Within a server communication goes up & down the layer; across servers it is using protocols like SMTP, EWS, and MRS proxy. Everything is clean.

Some slides on all three components:

Outbound email from the transport component on a MBX server can go out directly to an external SMTP server, or it can be delivered to the FET on any CAS server in the same site. This delivery happens on port 717 and needs to be specifically enabled.

Transport component listens on port 25 if MBX and CAS are on separate servers. Else it listens on port 2525 as CAS is already listening on 25. These ports are for accepting messages from the FET. For accepting messages from the Mailbox Transport component, it listens on port 465.

Remember that Transport is stateful.

Destination can be a CAS server or another transport component (on another MBX server). The Transport component is what does the lookup of the mailbox database.

Last component: Mailbox Transport. This is the component that actually talks to the next layer in the mailbox server. This talks MAPI and receives emails from the Transport component. This is also the component that does the message conversion (TNF to MIME and vice versa). No extensibility at this component as all that is at the Transport component. Once a message reaches Mailbox Transport there’s no changes happening to it!