Earlier today I had blogged about nslookup working but ping and other methods not resolving names to IP addresses. That problem started again, later in the evening.<\/p>\n
Today morning though as a precaution I had enabled the DNS Client logs on my computer. (To do that open Event Viewer with an admin account, go down to Applications and Services Logs > Microsoft > Windows > DNS Client Events > Operational – and click “Enable log” in the “Actions” pane on the right).<\/p>\n
That showed me an error along the following lines:<\/p>\n
\nA name not found error was returned for the name vcenter01.rakhesh.local. Check to ensure that the name is correct. The response was sent by the server at 10.50.1.21:53.<\/p>\n<\/blockquote>\n
Interesting. So it looked like a particular DC was the culprit. Most likely when I restarted the DNS Client service it just chose a different DC and the problem temporarily went away. And sure enough nslookup for this record against this DNS server returned no answers.<\/p>\n
I fired up DNS Manager and looked at this server. It seemed quite outdated with many missing records. This is my simulated branch office DC so I don’t always keep it on\/ online. Looks like that was coming back to bite me now.<\/p>\n
The DNS logs in Event Manager on that server had errors like this:<\/p>\n
\nThe DNS server was unable to complete directory service enumeration of zone TrustAnchors.\u00a0 This DNS server is configured to use information obtained from Active Directory for this zone and is unable to load the zone without it.\u00a0 Check that the Active Directory is functioning properly and repeat enumeration of the zone. The extended error debug information (which may be empty) is “”. The event data contains the error.<\/p>\n<\/blockquote>\n
So Active Directory is the culprit (not surprising as these zones are AD integrated so the fact that they weren’t up to date indicated AD issues to me). I ran
repadmin \/showrepl<\/code> and that had many errors:<\/p>\n\nNaming Context: CN=Configuration,DC=rakhesh,DC=local<\/p>\n
Source: COCHIN\\WIN-DC03
******* WARNING: KCC could not add this REPLICA LINK due to error.<\/p>\nNaming Context: DC=DomainDnsZones,DC=rakhesh,DC=local
Source: COCHIN\\WIN-DC03
******* WARNING: KCC could not add this REPLICA LINK due to error.<\/p>\nNaming Context: DC=ForestDnsZones,DC=rakhesh,DC=local
Source: COCHIN\\WIN-DC03
******* WARNING: KCC could not add this REPLICA LINK due to error.<\/p>\nNaming Context: DC=rakhesh,DC=local
Source: COCHIN\\WIN-DC03
******* WARNING: KCC could not add this REPLICA LINK due to error.<\/p>\nNaming Context: CN=Configuration,DC=rakhesh,DC=local
Source: COCHIN\\WIN-DC01
******* WARNING: KCC could not add this REPLICA LINK due to error.<\/p>\nNaming Context: DC=DomainDnsZones,DC=rakhesh,DC=local
Source: COCHIN\\WIN-DC01
******* WARNING: KCC could not add this REPLICA LINK due to error.<\/p>\nNaming Context: DC=ForestDnsZones,DC=rakhesh,DC=local
Source: COCHIN\\WIN-DC01
******* WARNING: KCC could not add this REPLICA LINK due to error.<\/p>\nNaming Context: DC=rakhesh,DC=local
Source: COCHIN\\WIN-DC01
******* WARNING: KCC could not add this REPLICA LINK due to error.<\/p>\n<\/blockquote>\nGreat! I fired up AD Sites and Services and the links seemed ok. Moreover I could ping the DCs from each other. Event Logs on the problem DC (WIN-DC02) had many entries like this though:<\/p>\n
\nThe Kerberos client received a KRB_AP_ERR_MODIFIED error from the server win-dc01$. The target name used was Rpcss\/WIN-DC01. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (RAKHESH.LOCAL) is different from the client domain (RAKHESH.LOCAL), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.<\/p>\n<\/blockquote>\n
Hmm, secure channel issues? I tried resetting it but that too failed:<\/p>\n
C:\\>nltest \/sc_reset:rakhesh.local\r\nI_NetLogonControl failed: Status = 5 0x5 ERROR_ACCESS_DENIED<\/pre>\n(Ignore the above though. Later I realized that this was because I wasn’t running command prompt as an admin. Because of UAC even though I was logged in as admin I should have right clicked and ran command prompt as admin).<\/p>\n
Since I know my environment it looks likely to be a case of this DC losing trust with other DCs. The
KRB_AP_ERR_MODIFIED<\/code> error also seems to be related to Windows Server 2003 and Windows Server 2012 R2<\/a> but mine wasn’t a Windows Server 2003. That blog post confirmed my suspicions that this was password related.<\/p>\nTime to check the last password set attribute for this DC object on all my other DCs. Time to run
repadmin \/showobjmeta<\/code><\/a>.<\/p>\nC:\\Windows\\system32>repadmin \/showobjmeta WIN-DC01 \"CN=WIN-DC02,OU=Domain Controllers,DC=rakhesh,DC=local\"\r\n\r\n32 entries.\r\nLoc.USN Originating DSA Org.USN Org.Time\/Date Ver Attribute\r\n======= =============== ========= ============= === =========\r\n 42577 COCHIN\\WIN-DC01 42577 2014-03-19 14:21:40 1 objectClass\r\n 111364 COCHIN\\WIN-DC01 111364 2014-12-03 19:10:59 2 cn\r\n 42577 COCHIN\\WIN-DC01 42577 2014-03-19 14:21:40 1 instanceType\r\n 42577 COCHIN\\WIN-DC01 42577 2014-03-19 14:21:40 1 whenCreated\r\n 42734 KOTTAYAM\\WIN-DC02 12293 2014-03-19 15:58:03 2 nTSecurityDescriptor\r\n 111364 KOTTAYAM\\WIN-DC02 69783 2014-12-03 19:07:14 6 name\r\n 42648 COCHIN\\WIN-DC01 42648 2014-03-19 15:57:34 2 userAccountControl\r\n 42578 COCHIN\\WIN-DC01 42578 2014-03-19 14:21:40 1 codePage\r\n 42578 COCHIN\\WIN-DC01 42578 2014-03-19 14:21:40 1 countryCode\r\n 233726 COCHIN\\WIN-DC01 233726 2015-07-25 18:47:47 11 dBCSPwd\r\n 42577 COCHIN\\WIN-DC01 42577 2014-03-19 14:21:40 1 localPolicyFlags\r\n 42578 COCHIN\\WIN-DC01 42578 2014-03-19 14:21:40 1 logonHours\r\n 233726 COCHIN\\WIN-DC01 233726 2015-07-25 18:47:47 11 unicodePwd\r\n 233726 COCHIN\\WIN-DC01 233726 2015-07-25 18:47:47 11 ntPwdHistory\r\n 233726 COCHIN\\WIN-DC01 233726 2015-07-25 18:47:47 11 pwdLastSet\r\n 42648 COCHIN\\WIN-DC01 42648 2014-03-19 15:57:34 2 primaryGroupID\r\n 233727 COCHIN\\WIN-DC01 233727 2015-07-25 18:47:47 11 supplementalCredentials\r\n 42577 COCHIN\\WIN-DC01 42577 2014-03-19 14:21:40 1 objectSid\r\n 42578 COCHIN\\WIN-DC01 42578 2014-03-19 14:21:40 1 accountExpires\r\n 233726 COCHIN\\WIN-DC01 233726 2015-07-25 18:47:47 11 lmPwdHistory\r\n 42577 COCHIN\\WIN-DC01 42577 2014-03-19 14:21:40 1 sAMAccountName\r\n 42577 COCHIN\\WIN-DC01 42577 2014-03-19 14:21:40 1 sAMAccountType\r\n 42586 COCHIN\\WIN-DC01 42586 2014-03-19 14:32:22 1 operatingSystem\r\n 42586 COCHIN\\WIN-DC01 42586 2014-03-19 14:32:22 1 operatingSystemVersion\r\n 42586 COCHIN\\WIN-DC01 42586 2014-03-19 14:32:22 1 operatingSystemServicePack\r\n 42577 COCHIN\\WIN-DC01 42577 2014-03-19 14:21:40 1 dNSHostName\r\n 42668 COCHIN\\WIN-DC01 42668 2014-03-19 15:58:34 1 rIDSetReferences\r\n 111582 KOTTAYAM\\WIN-DC02 69785 2014-12-03 19:11:30 10 servicePrincipalName\r\n 42577 COCHIN\\WIN-DC01 42577 2014-03-19 14:21:40 1 objectCategory\r\n 42648 COCHIN\\WIN-DC01 42648 2014-03-19 15:57:34 2 isCriticalSystemObject\r\n 229554 KOTTAYAM\\WIN-DC02 114732 2015-07-25 18:08:14 23 lastLogonTimestamp\r\n 42734 KOTTAYAM\\WIN-DC02 12294 2014-03-19 15:58:17 2 msDS-SupportedEncryptionTypes\r\n0 entries.\r\nType Attribute Last Mod Time Originating DSA Loc.USN Org.USN Ver\r\n======= ============ ============= ================= ======= ======= ===\r\n Distinguished Name\r\n =============================<\/pre>\nThe above output gives metadata of the WIN-DC02 object on WIN-DC01. I am interested in the
pwdLastSet<\/code> attribute and its timestamp.\u00a0 Here’s a comparison of this across my three DCs:<\/p>\nC:\\Windows\\system32>repadmin \/showobjmeta WIN-DC01 \"CN=WIN-DC02,OU=Domain Controllers,DC=rakhesh,DC=local\" | findstr pwdLastSet\r\n 233726 COCHIN\\WIN-DC01 233726 2015-07-25 18:47:47 11 pwdLastSet\r\n\r\nC:\\Windows\\system32>repadmin \/showobjmeta WIN-DC02 \"CN=WIN-DC02,OU=Domain Controllers,DC=rakhesh,DC=local\" | findstr pwdLastSet\r\n 103907 COCHIN\\WIN-DC01 149656 2015-05-09 17:41:18 10 pwdLastSet\r\n\r\nC:\\Windows\\system32>repadmin \/showobjmeta WIN-DC03 \"CN=WIN-DC02,OU=Domain Controllers,DC=rakhesh,DC=local\" | findstr pwdLastSet\r\n 135409 COCHIN\\WIN-DC01 233726 2015-07-25 18:47:47 11 pwdLastSet<\/pre>\nThat confirms the problem. WIN-DC02 thinks its password last changed on 9th May whereas WIN-DC01 changed it on 25th July and replicated it to WIN-DC03.<\/p>\n
Interestingly that date of 25th July is when I first started having problems in my test lab<\/a>. I thought I had sorted them but apparently they were only lurking beneath. The solution here is to reset the WIN-DC02 password on itself and WIN-DC01 and replicate it across. The steps are in this KB article<\/a>, here’s what I did:<\/p>\n
\n
- On WIN-DC02 (the problem DC) I stopped the KDC service and set it to start Manual.<\/li>\n
- Purge the Kerberos ticket cache. You can view the ticket cache by typing the command:
klist<\/code>.\u00a0 To purge, do:klist purge<\/code>.<\/li>\n- Open a command prompt as administrator (i.e. right click and do a “Run as administrator”) then type the following command:
netdom resetpwd \/server WIN-DC01.rakhesh.local \/UserD MyAdminAccount \/PasswordD *<\/code><\/li>\n- Restart WIN-DC02.<\/li>\n
- After logon start the KDC service and set it to Automatic.<\/li>\n<\/ol>\n
Checked the Event Logs to see if there are any errors. None initially but after I forced a sync via
repadmin \/syncall \/e<\/code> I got a few. All of them had the following as an error:<\/p>\n\n2148074274 The target principal name is incorrect.<\/p>\n<\/blockquote>\n
Odd. But at least it was different from the previous errors and we seemed to be making progress. <\/p>\n
After a bit of trial and error I noticed that whenever the KDC service on the DC was stopped it seemed to work fine.<\/p>\n
I could access other servers (file shares), connect to them via DNS Manager, etc. But start KDC and all these would break with errors indicating the target name was wrong or that “a security package specific error occurred”. Eventually I left KDC stay off, let the domain sync via
repadmin \/syncall<\/code>, and waited a fair amount of time (about 15-20 mins) for things to settle. I kept an eye onrepadmin \/replsummary<\/code> to see the deltas between WIN-DC02 and the rest, and also kept an eye on the DNS zones to see if WIN-DC02 was picking up newer entries from the others. Once these two looked positive, I started KDC. And finally things were working!<\/p>\n<\/p>\n","protected":false},"excerpt":{"rendered":"
Earlier today I had blogged about nslookup working but ping and other methods not resolving names to IP addresses. That problem started again, later in the evening. Today morning though as a precaution I had enabled the DNS Client logs on my computer. (To do that open Event Viewer with an admin account, go down … Continue reading Cannot ping an address but nslookup works (contd)<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[333,85],"tags":[4,47,552,465],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/2686"}],"collection":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/comments?post=2686"}],"version-history":[{"count":0,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/posts\/2686\/revisions"}],"wp:attachment":[{"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/media?parent=2686"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/categories?post=2686"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rakhesh.com\/wp-json\/wp\/v2\/tags?post=2686"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}