It’s 2am and I spent the last 2-3 hours chasing a shitty problem in my home lab to which I haven’t yet found a satisfactory answer. What a waste of time (sort of)!
It all began when I enabled MAPI/HTTP on my home Exchange 2013 and noticed that Outlook (2016 & 2019) was still connecting to it via RPC/HTTP. To troubleshoot that I deleted my Outlook profile to see if a fresh auto-discovery would get it to connect using MAPI/HTTP. That ended up being a long rabbit hole!
Background: all my user accounts are of the form firstname.lastname@raxnet.global. I have autodiscovery setup via SCP (here’s a good blog post to read if you want an overview of auto-discovery) and in the past when I launch Outlook it used to give me a prompt like this –
– followed by this one, where I choose “Exchange” and am done.
Today, however, I was getting prompted for a password after the first screen and when I entered the password it complained that the password was wrong. Moreover, I noticed that it was trying to connect via IMAP and picking up an external IMAP server for this domain (the domain raxnet.global is setup externally too with IMAP auto discovery in the form of SRV records for _imaps._tcp.raxnet.global). That didn’t make sense. All my internal clients were pointing to the internal DC for DNS, and this DC had the raxnet.global zone with itself so there was no reason why queries should be going to the Internet.
(Notice the imap.fastmail.com? That should not be happening.)
Moreover, if I did an nslookup -type=SRV _imaps._tcp.raxnet.global.
from my clients they weren’t resolving the name either. How was Outlook able to resolve this then?! (I still don’t have an answer to this so don’t get your hopes up, dear reader). Clients can query the name only if they query an external DNS server but that wasn’t happening in my case.
Here’s an article on Outlook 2016’s implementation of auto discover. It too tells me that SCP is the default mechanism for auto-discovery when Outlook discovers it’s on a domain joined machine. Starting from Outlook 2016 however, there is a special case in that if Outlook feels the account is an Office 365 enabled one it will try and auto-discover using Office 365 endpoints (typically https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml or https://autodiscover-s.partner.outlook.cn/autodiscover/autodiscover.xml). My domain isn’t on Office 365, but what the heck I disabled it via a registry key but no luck it still goes to IMAP.
Why I don’t understand is why it keeps ignoring SCP. It is as if Outlook thinks I am not domain joined and is magically able to query external DNS. The Windows VM on which Outlook runs can’t query the outside world so how is Outlook doing it?
I verified the SCP end points in Exchange (this is a good article to read) and also came across this script which you can run on the client side to make it query AD for the SCP endpoint via LDAP. (I couldn’t access the blog so had to get a cached copy via Google. So I am going to copy paste the script here for my future reference. All credit to the author of the blog).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
$obj = @() $ADDomain = Get-ADDomain | Select DistinguishedName $DSSearch = New-Object System.DirectoryServices.DirectorySearcher $DSSearch.Filter = '(&(objectClass=serviceConnectionPoint)(|(keywords=67661d7F-8FC4-4fa7-BFAC-E1D7794C1F68)(keywords=77378F46-2C66-4aa9-A6A6-3E7A48B19596)))' $DSSearch.SearchRoot = 'LDAP://CN=Configuration,'+$ADDomain.DistinguishedName $DSSearch.FindAll() | %{ $ADSI = [ADSI]$_.Path $autodiscover = New-Object psobject -Property @{ Server = [string]$ADSI.cn Site = $adsi.keywords[0] DateCreated = $adsi.WhenCreated.ToShortDateString() AutoDiscoverInternalURI = [string]$adsi.ServiceBindingInformation } $obj += $autodiscover } Write-Output $obj | Select Server,Site,DateCreated,AutoDiscoverInternalURI | ft -AutoSize |
This correctly returned my client machines’ site as well as the autodiscover URL.
1 2 3 |
Server Site DateCreated AutoDiscoverInternalURI ------ ---- ----------- ----------------------- RAK1EXCH01 Site=Site-2 19-Oct-18 https://exchange.raxnet.global/Autodiscover/Autodiscover.xml |
I have no idea why auto-discover was failing. My hunch at this point was that Outlook was querying DNS some other way, because of which it was thinking it is not domain joined and so all the usual auto-discovery was failing until it hit up on the Internet face IMAP auto-discovery. So what was my DNS setup like?
All my clients were pointing to the DC via DHCP for DNS. The DNS in turn was forwarding to my OPNSense router which had Unbound DNS running on it. I verified again, from the client and DC, that I am unable to resolve the external records of raxnet.global. I suspected OPNSense at this point so I turned off Unbound DNS to see if that impacts the auto-discovery. Sure enough, it did! Now Outlook was behaving as it used to.
Unbound DNS was set to listen only on the servers subnet. So there was no way a query could come from my clients subnet to Unbound (my bad, I should have checked this by querying Unbound directly – too sleepy now to enable Unbound again and see what happens). I couldn’t see any other settings either so I stopped Unbound and added root hints to the DC so it can talk to the Internet directly. (Here yet another thing wasted my time. I always set DNS servers to listen on “All IP addresses” – including IPv6. With that setting my DC was unable to resolve external names using the root hints. If I disable it to listen only on IPv4, the DC is able to resolve as expected! Something in the IPv6 was causing an issue. Funnily a colleague had mentioned this in the past but I never believed him).
That done, my clients would now talk to the DC for DNS queries (as before), and the DC would go and lookup on its own (unlike before). No more forwarder. I tried Outlook again with this setting and auto-discover correctly works.
The issue isn’t Unbound or OPNSense. The issue is that when my DC is using a forwarder (any forwarder, even 8.8.8.8 for instance) Outlook thinks it is not domain joined and it can view my external DNS records – breaking auto-discovery. I don’t know why this happens, especially considering the OS on which Outlook is running can’t resolve the external records and can in fact resolve the internal LDAP end-points correctly. Something in the way Outlook is sending DNS queries is causing it to work differently. (Or maybe I am just too sleepy and can’t think of a simple explanation now!)