Subscribe via Email

Subscribe via RSS


Creative Commons Attribution 4.0 International License
© Rakhesh Sasidharan

Useful WMIC filters

I have these tabs open in my browser from last month when I was doing some WMI based GPO targeting. Meant to write a blog post but I keep getting side tracked and now it’s been nearly a month so I have lost the flow. But I want to put these in the blog as a reference to my future self. 

That’s all.

Get a list of users in an OU along with last logged on date

Trivial stuff. Wanted to note it down someplace for future reference –


The “Administrators” group

Note to self: the “Domain Admins” and “Enterprise Admins” groups aren’t the primary groups in a domain. The primary group is the “Administrators” group, present in the “Builtin” folder. The other two groups are members of this group and thus get rights over the domain. The “Enterprise Admins” group is also a member of the “Administrator” group in all other domains/ child-domains of that forest, hence its members get rights over those domains too.

So if you want to create a separate group in your domain and want to give its members domain admin rights over (say) a child domain, all you need to do is create the group (must be Global or Universal) an add this group to the “Administrators” group in the child domain. That’s it!

Cannot ping an address but nslookup works (contd)

Earlier today I had blogged about nslookup working but ping and other methods not resolving names to IP addresses. That problem started again, later in the evening.

Today morning though as a precaution I had enabled the DNS Client logs on my computer. (To do that open Event Viewer with an admin account, go down to Applications and Services Logs > Microsoft > Windows > DNS Client Events > Operational – and click “Enable log” in the “Actions” pane on the right).

That showed me an error along the following lines:

A name not found error was returned for the name vcenter01.rakhesh.local. Check to ensure that the name is correct. The response was sent by the server at

Interesting. So it looked like a particular DC was the culprit. Most likely when I restarted the DNS Client service it just chose a different DC and the problem temporarily went away. And sure enough nslookup for this record against this DNS server returned no answers.

I fired up DNS Manager and looked at this server. It seemed quite outdated with many missing records. This is my simulated branch office DC so I don’t always keep it on/ online. Looks like that was coming back to bite me now.

The DNS logs in Event Manager on that server had errors like this:

The DNS server was unable to complete directory service enumeration of zone TrustAnchors.  This DNS server is configured to use information obtained from Active Directory for this zone and is unable to load the zone without it.  Check that the Active Directory is functioning properly and repeat enumeration of the zone. The extended error debug information (which may be empty) is “”. The event data contains the error.

So Active Directory is the culprit (not surprising as these zones are AD integrated so the fact that they weren’t up to date indicated AD issues to me). I ran repadmin /showrepl and that had many errors:

Naming Context: CN=Configuration,DC=rakhesh,DC=local

******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: DC=DomainDnsZones,DC=rakhesh,DC=local
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: DC=ForestDnsZones,DC=rakhesh,DC=local
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: DC=rakhesh,DC=local
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: CN=Configuration,DC=rakhesh,DC=local
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: DC=DomainDnsZones,DC=rakhesh,DC=local
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: DC=ForestDnsZones,DC=rakhesh,DC=local
******* WARNING: KCC could not add this REPLICA LINK due to error.

Naming Context: DC=rakhesh,DC=local
******* WARNING: KCC could not add this REPLICA LINK due to error.

Great! I fired up AD Sites and Services and the links seemed ok. Moreover I could ping the DCs from each other. Event Logs on the problem DC (WIN-DC02) had many entries like this though:

The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server win-dc01$. The target name used was Rpcss/WIN-DC01. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (RAKHESH.LOCAL) is different from the client domain (RAKHESH.LOCAL), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

Hmm, secure channel issues? I tried resetting it but that too failed:

(Ignore the above though. Later I realized that this was because I wasn’t running command prompt as an admin. Because of UAC even though I was logged in as admin I should have right clicked and ran command prompt as admin).

Since I know my environment it looks likely to be a case of this DC losing trust with other DCs. The KRB_AP_ERR_MODIFIED error also seems to be related to Windows Server 2003 and Windows Server 2012 R2 but mine wasn’t a Windows Server 2003. That blog post confirmed my suspicions that this was password related.

Time to check the last password set attribute for this DC object on all my other DCs. Time to run repadmin /showobjmeta.

The above output gives metadata of the WIN-DC02 object on WIN-DC01. I am interested in the pwdLastSet attribute and its timestamp.  Here’s a comparison of this across my three DCs:

That confirms the problem. WIN-DC02 thinks its password last changed on 9th May whereas WIN-DC01 changed it on 25th July and replicated it to WIN-DC03.

Interestingly that date of 25th July is when I first started having problems in my test lab. I thought I had sorted them but apparently they were only lurking beneath. The solution here is to reset the WIN-DC02 password on itself and WIN-DC01 and replicate it across. The steps are in this KB article, here’s what I did:

  1. On WIN-DC02 (the problem DC) I stopped the KDC service and set it to start Manual.
  2. Purge the Kerberos ticket cache. You can view the ticket cache by typing the command: klist.  To purge, do: klist purge.
  3. Open a command prompt as administrator (i.e. right click and do a “Run as administrator”) then type the following command: netdom resetpwd /server WIN-DC01.rakhesh.local /UserD MyAdminAccount /PasswordD *
  4. Restart WIN-DC02.
  5. After logon start the KDC service and set it to Automatic.

Checked the Event Logs to see if there are any errors. None initially but after I forced a sync via repadmin /syncall /e I got a few. All of them had the following as an error:

2148074274 The target principal name is incorrect.

Odd. But at least it was different from the previous errors and we seemed to be making progress.

After a bit of trial and error I noticed that whenever the KDC service on the DC was stopped it seemed to work fine.

I could access other servers (file shares), connect to them via DNS Manager, etc. But start KDC and all these would break with errors indicating the target name was wrong or that “a security package specific error occurred”. Eventually I left KDC stay off, let the domain sync via repadmin /syncall, and waited a fair amount of time (about 15-20 mins) for things to settle. I kept an eye on repadmin /replsummary to see the deltas between WIN-DC02 and the rest, and also kept an eye on the DNS zones to see if WIN-DC02 was picking up newer entries from the others. Once these two looked positive, I started KDC. And finally things were working!


vCenter – Cannot load the users for the selected domain

I spent the better part of today evening trying to sort this issue. But didn’t get any where. I don’t want to forget the stuff I learnt while troubleshooting so here’s a blog post.

Today evening I added one of my ESXi hosts to my domain. The other two wouldn’t add, until I discovered that the time on those two hosts were out of sync. I spent some time trying to troubleshoot that but didn’t get anywhere. The NTP client on these hosts was running, the ports were open, the DC (which was also the forest PDC and hence the time keeper) was reachable – but time was still out of sync.

Found an informative VMware KB article. The ntpq command (short for “NTP query”) can be used to see the status of NTP daemon on the client side. Like thus:

The command has an interactive mode (which you get into if run without any switches; read the manpage for more info). The -p switch tells ntpq to output a list of peers and their state. The KB article above suggests running this command every 2 seconds using the watch command but you don’t really need to do that.

Important points about the output of this command:

  • If it says “No association ID's returned” it means the ESXi host cannot reach the NTP server. Considering I didn’t get that, it means I have no connectivity issue.
  • If it says “***Request timed out” it means the response from the NTP server didn’t get through. That’s not my problem either.
  • If there’s an asterisk before the remote server name (like so) it means there is a huge gap between the time on the host and the time given by the NTP server. Because of the huge gap NTP is not changing the time (to avoid any issues caused by a sudden jump in the OS time). Manually restarting the NTP daemon (/etc/init.d/ntpd restart) should sort it out.
    • The output above doesn’t show it but one of my problem hosts had an asterisk. Restarting the daemon didn’t help.

The refid field shows the time stream to which the client is syncing. For instance here’s the w3tm output from my domain:

Notice the PDC has a refid of LOCL (indicating it is its own time source) while the rest have a refid of the PDC name. My ESXi host has a refid of .INIT. which means it has not received any response from the NTP server (shouldn’t the error message have been something else!?). So that’s the problem in my case.

Obviously the PDC is working because all my Windows machines are keeping correct time from it. So is vCenter. But some my ESXi hosts aren’t.

I have no idea what’s wrong. After some troubleshooting I left it because that’s when I discovered my domain had some inconsistencies. Fixing those took a while, after which I hit upon a new problem – vCenter clients wouldn’t show me vCenter or any hosts when I login with my domain accounts. Everything appears as expected under the administrator@vsphere.local account but the domain accounts return a blank.

While double-checking that the domain admin accounts still have permissions to vCenter and SSO I came across the following error:

Cannot load the users

Great! (The message is “Cannot load the users for the selected domain“).

I am using the vCenter appliance. Digging through the /var/log/messages on this I found the following entries:

Searched Google a bit but couldn’t find any resolutions. Many blog posts suggested removing vCenter from the domain and re-adding but that didn’t help. Some blog posts (and a VMware KB article) talk about ensuring reverse PTR records exist for the DCs – they do in my case. So I am drawing a blank here.

Odd thing is the appliance is correctly connected to the domain and can read the DCs and get a list of users. The appliance uses Likewise (now called PowerBroker Open) to join itself to the domain and authenticate with it. The /opt/likewise/bin directory has a bunch of commands which I used to verify domain connectivity:

All looks well! In fact, I added a user to my domain and re-ran the lw-enum-users command it correctly picked up the new user. So the appliance can definitely see my domain and get a list of users from it. The problem appears to be in the upper layers.

In /var/log/vmware/sso/ssoAdminServer.log I found the following each time I’d query the domain for users via the SSO section in the web client:

Makes no sense to me but the problem looks to be in Java/ SSO.

I tried removing AD from the list of identity sources in SSO (in the web client) and re-added it. No luck.

Tried re-adding AD but this time I used an SPN account instead of the machine account. No luck!

Finally I tried adding AD as an LDAP Server just to see if I can get it working somehow – and that clicked! :)


So while I didn’t really solve the problem I managed to work around it …

Update: Added the rest of my DCs as time sources to the ESXi hosts and restarted the ntpd service. Maybe that helped, now NTP is working on the hosts.


Fixing “The DNS server was unable to open Active Directory” errors

For no apparent reason my home testlab went wonky today! Not entirely surprising. The DCs in there are not always on/ connected; and I keep hibernating the entire lab as it runs off my laptop so there’s bound to be errors lurking behind the scenes.

Anyways, after a reboot my main DC was acting weird. For one it took a long time to start up – indicating DNS issues, but that shouldn’t be the case as I had another DC/ DNS server running – and after boot up DNS refused to work. Gave the above error message. The Event Logs were filled with two errors:

  • Event ID 4000: The DNS server was unable to open Active Directory.  This DNS server is configured to obtain and use information from the directory for this zone and is unable to load the zone without it. Check that the Active Directory is functioning properly and reload the zone. The event data is the error code.
  • Event id 4007: The DNS server was unable to open zone <zone> in the Active Directory from the application directory partition <partition name>. This DNS server is configured to obtain and use information from the directory for this zone and is unable to load the zone without it. Check that the Active Directory is functioning properly and reload the zone. The event data is the error code.

A quick Google search brought up this Microsoft KB. Looks like the DC has either lost its secure channel with the PDC, or it holds all the FSMO roles and is pointing to itself as a DNS server. Either of these could be the culprit in my case as this DC indeed had all the FSMO roles (and hence was also the PDC), and so maybe it lost trust with itself? Pretty bad state to be in, having no trust in oneself … ;-)

The KB article is worth reading for possible resolutions. In my case since I suspected DNS issues in the first place, and the slow loading usually indicates the server is looking to itself for DNS, I checked that out and sure enough it was pointing to itself as the first nameserver. So I changed the order, gave the DC a reboot, and all was well!

In case the DC had lost trust with itself the solution (according to the KB article) was to reset the DC password. Not sure how that would reset trust, but apparently it does. This involves using the netdom command which is installed on Server 2008 and up (as well as on Windows 8 or if RSAT is installed and can be downloaded for 2003 from the Support Tools package). The command has to be run on the computer whose password you want to reset (so you must login with an account whose initials are cached, or use a local account). Then run the command thus:

Of course the computer must have access to the PDC. And if you are running it on a DC the KDC service must be stopped first.

I have used netdom in the past to reset my testlab computer passwords. Since a lot of the machines are usually offline for many days, and after a while AD changes the computer account password but the machine still has the old password, when I later boot up the machine it usually gives are error like this: “The trust relationship between this workstation and the primary domain failed.”

A common suggestion for such messages is to dis-join the machine from the domain and re-join it, effectively getting it a new password. That’s a PITA though – I just use netdom and reset the password as above. :)


[Aside] Bridgehead Server Selection improvements in Server 2008 R2

Came across this blog post when researching for something. Long time since I read anything AD related (since I am more focused on VMware and HP servers) at work nowadays. Was a good read.

Summary of the post:

  • When you have a domain spread over multiple sites, there is a designated Bridgehead server in each site that replicates changes with/ to the Bridgehead server in other sites.
    • Bridgehead servers talk to each other via IP or SMTP.
    • Bridgehead servers are per partition of the domain. A single Bridgehead server can replicate for multiple partitions and transports.
    • Since Server 2003 there can be multiple Bridgehead servers per partition in the domain and connections can be load-balanced amongst these. The connections will be load-balanced to Server 2000 DCs as well.
  • Bridgehead servers are automatically selected (by default). The selection is made by a DC that holds the Inter-Site Topology Generator (ISTG) role.
    • The DC holding the ISTG role is usually the first DC in the site (the role will failover to another DC if this one fails; also, the role can be manually moved to another DC).
    • It is possible designate certain DCs are preferred Bridgehead servers. In this case the ISTG will choose a Bridgehead server from this list.
  • It is also possible to manually create connections from one DC to another for each site and partition, avoiding ISTG altogether.
  • On each DC there is a process called the Knowledge Consistency Checker (KCC). This process is what actually creates the replication topology for the domain.
  • The KCC process running on the DC holding the ISTG role is what selects the Bridgehead servers.

The above was just background. Now on to the improvements in Server 2008 R2:

  • As mentioned above, in Server 2000 you had one Bridgehead server per partition per site.
  • In Server 2003 you could have multiple Bridgehead servers per partition per site. There was no automatic load-balancing though – you had to use a tool such as Adlb.exe to manually load-balance among the multiple Bridgehead servers.
  • In Server 2008 you had automatic load-balancing. But only for Read-Only Domain Controllers (RODCs).
    • So if Site A had 5 DCs, the RODCs in other sites would load-balance their incoming connections (remember RODCs only have incoming connections) across these 5 DCs. If a 6th DC was added to Site A, the RODCs would automatically load-balance with that new DC.
    • Regular DCs (Read-Write DCs) too would load-balance their incoming connections across these 5 DCs. But if a 6th DC was added they wouldn’t automatically load-balance with that new DC. You would still need to run a tool like Aldb.exe to load-balance (or delete the inbound connection objects on these regular DCs and run KCC again?).
    • Regular DCs would sort of load-balance their outbound connections to Site A. The majority of incoming connections to Site A would still hit a single DC.
  • In Server 2008 R2 you have complete automatic load-balancing. Even for regular DCs.
    • In the above example: not only would the regular DCs automatically load-balance their incoming connections with the new 6th DC, but they would also load-balance their outbound connections with the DCs in Site A (and when the new DC is added automatically load-balance with that too). 

To view Bridgeheads connected to a DC run the following command:

The KCC runs every 15 minutes (can be changed via registry). The following command runs it manually:

Also, the KCC prefers DCs that are more stable / readily available than DCs that are intermittently available. Thus DCs that are offline for an extended period do not get rebalanced automatically when they become online (at least not immediately.

DFS Namespaces missing on some servers / sites

This is something I had sorted out for one of our offices earlier. Came across it for another office today and thought I should make a blog post of it. I don’t remember if I made a a blog post the last time I fixed it (and although it’s far easier to just search my blog and then decide whether to make a blog post or not, I am lazy that way :)).

Here’s the problem. We use AppV to stream some applications to our users. One of our offices started complaining that AppV was no longer working for them. Here’s the error they got:


I logged on to the user machine to find out where the streaming server is:

Checked the Event Logs on that server and there were errors from the “Application Virtualization” source along these lines:

So I checked the DFS path and it was empty. In fact, there was no folder called “Content” under “\\\dfs” – which was odd!

I switched DFS servers to that of another location and all the folders appeared. So the problem definitely was with the DFS server of this site.

From the DFS Management tool I found the namespace server for this site (it was the DC) and the folder target (it was one of the data servers). Checked the folder target share and that was accessible, so no issues there. It was looking to be a DFS Namespace issue.

Hopped on to the DFS and checked “C:\DFSRoots\DFS”. It didn’t have the “Content” folder above – so definitely a DFS Namespace issue!

I ran dfsdiag and it gave no errors:

So I checked the Event Logs on the DC. No errors there. Next I restarted the “DFS Namespace” service and voila! all the namespaces appeared correctly.

Ok, so that fixed the problem but why did it happen in the first place? And what’s there to ensure it doesn’t happen again? The site was restarted yesterday as part of some upgrade work so did that cause the namespaces to fail?

I checked the timestamps of the DFS Namespace entries (these are from source “DfsSvc” in “Custom Views” > “Server Roles” > “File Server”). Once the namespaces were ready there was an entry along the following lines at 08:06:43 (when the DC came back up from the reboot):

No errors there. But where does the DC get its name spaces from? This was a domain DFS so it would be getting it from Active Directory. So let’s look at the AD logs under “Applications and Services Logs” > “Directory Service”. When the domain services are up and running there’s an entry from the “ActiveDirectory_DomainService” source along these lines:

On this server the entry was made at 08:06:47. Notice the timestamp – it occurs after the “DFS Namespace” service has finished building namespace! And therein lies the problem. Because the Directory Services took a while to be ready, when DFS was being initialized it went ahead with stale data. When I restarted the service later, it picked up the correct data and sorted the namespaces.

I rebooted the server a couple more times but couldn’t replicate the problem as the Directory Service always initialized before DFS Namespace. To be on the safe side though I set the startup type of “DFS Namespace” to be “Automatic (Delayed)” so it always starts after the “Directory Service”.


AD Domain Password policies

Password policies are set in the “Computer Configuration” section at Computer Configuration\Windows Settings\Security Settings\Account Policies\Password Policy.

The policy applies to the computer as user objects are stored in the database of the computer. A Domain Controller is a special computer in that its database holds user objects for the entire domain, so it takes its password policy settings from whatever policy wins at the domain level. What this means is that you can have multiple policies in a domain – each containing different password policies – but the only one that matters for domain users is the policy that wins at the domain level. Any policy that applies to OUs will only apply to local user objects that reside in the SAM database of computers in that OU. 

A quick recap on how GPO precedence works: OU linked GPOs override Domain linked GPOs which override Site linked GPOs which override local GPOs (i.e. OU > Domain > Site > local). Within each of these there can be multiple GPOs. The link order matters here. The GPOs with lower link order win over GPOs with higher link order (i.e. link order 1 > link order 2 > …). Of course all these precedence order can be subverted if a GPO is set as enforced. In this case the parent GPO cannot be overridden by a child GPO. 

By default the Default Domain Policy is set at the Domain level and has link order 1. If you want to change the password policy for the domain you can either modify this GPO, or create a new GPO and apply it at the Domain level (but remember to set it at a lower link order than the Default Domain Policy – i.e. link order 1 for instance). 

For a list of the password policy settings check out this TechNet page. 

Since Windows Server 2008 it is possible to have Fine Grained Password Policies (FGPP) that can apply to OUs. These are not GPOs, so you can’t set them via GPMC. For Server 2008 this TechNet page has instructions (you have to use PowerShell or ADSI Edit), for Server 2012 check out this blog post (you can use ADAC; obviously Server 2012 makes it easier). Check out this article too for Server 2008 (it is better than the TechNet page which is … dense on details). 

Replicate with repadmin

The following command replicates the specified partition from the source DC to the destination DC. You can use this command to force a replication. Note that these three arguments are mandatory. 

An optional switch /full will HWMV and UTDV tables to be reset, replicating all changes from the source DC to the destination DC.

The following command synchronizes the specified DC with all its replication partners. You can specify a partition to replicate; if nothing is specified, the Configuration partition is used by default

Instead of specifying a partition the /A switch can be used to sync all partitions held by the DC. By default only partner DCs in the same site are replicated with, but the /e switch will cause replication to happen with all partners across all sites. Also, changes can be pushed from the DC to others rather than pulled (the default) using the /P switch. 

SOA records (and dynamic DNS in Windows)

I am on the DNS section of my notes from the AD WorkshopPLUS I attended a few months back. That’s why the recent posts are about DNS …

The SOA (Start of Authority) record is something DNS administrators are familiar with. It specifies details about the zone such as the serial number (which can be used by secondary name servers to know the zone has changed), the preferred refresh periods for secondary name servers to sync the zone, the time between retries, whom to contact, the primary name server, and so on. Here’s the SOA record for my domain:

(In the example above the results also include all the name server records of the zone, but that needn’t be the case always).

In traditional zones you have one primary name server and many secondaries. So you can set one server as the primary in the record above. But what about AD-integrated zones? Since each DNS server is also a primary in that case, things are a bit different. 

What happens is that the primary name server is set to the name of whichever DNS server you ask. Thus, if you query WIN-DC01 for the SOA record to rakhesh.local, it will give itself as the primary, while if you query WIN-DC02 it will return itself as the primary. 

In Windows the name server returned by the SOA record is also used by clients for dynamic DNS updates. Clients query DNS for the SOA record. Whichever server they get a response from will return an SOA record containing itself as the primary name server. Clients then use that name server to dynamically register their A and PTR records. 

An exception to the above is Read-Only DCs (RODCs). These point to another server as the SOA for the zone. A new server is selected every 20 mins. When clients contact a RODC DNS server, they thus get another server as primary in the SOA record and send their dynamic updates to this other server. 

PortQry and PortQryUI

I was aware of PortQry but didn’t know it has a GUI counterpart too PortQryUI. For a quick read on PortQry check out this link, if you have more time and interest check out this KB article. PortQry/ PortQryUI can be used to check the status of TCP and UDP ports on a remote computer. For TCP ports I usually do a telnet to the port (by habit) but didn’t have any equivalent tool for UDP ports. 

Important PortQry switches (as a reference to myself) are:

  • -n -> specifies the server name/ IP address to target
  • -p -> specifies the protocol to test (options are tcp or udp or both; default is tcp)
  • -e -> specifies the port(s) to test (default is port 80)
    • note: it is possible to specify a single port (e.g. -e 81), ports (e.g. -e 80,81) or a range (e.g. -e 80-1024)
    • note: the ports must be in the range 1-65535
  • instead of -e you can use either of the following too:
    • -r -> specifies a port range (e.g. –r 80:90)
    • -o -> specifies a comma-separated list of ports to check in order (e.g. -o 80,443,139)

Some other switches are:

  • -nr -> stops PortQry from resolving an IP address to a name
  • -sl -> waits longer for replies from UDP systems (sl == slow link)
  • -l -> specifies a log file to output to
    • -y -> will over-write the log file if it exists, without prompting

While writing this post I learnt that PortQry can also enumerate the local ports. Nice!

  • The -local switch will list all active TCP/UDP ports on the local system. (Think of it as netstat -a but without any details of the remote end).
  • The -wport (port number) switch will watch a specified port’s state and report when it changes
    • This didn’t work for me, got an error “Port to process mapping is not supported on this system”.
  • The -wpid (PID) switch will watch a specified process ID (PID) and reports when its state changes
    • This too didn’t work for me, same error as above.

A good thing about PortQry is that it can also query protocols that it’s aware of. Thus, for instance, if you query port 53/ UDP (DNS) and something’s listening at the remote end, PortQry can send an additional DNS query to that port. 

This is useful in AD troubleshooting too. For instance, to check whether port 389 of a DC has an LDAP server listening as it should be:

Similarly, RPC:

In the output above, for instance, I query port 135/ TCP which is where the RPC end-mapper service listens. After querying this port and getting a response, PortQry asks it to enumerate the listening services. Of these UUID 12345887-... is what the netlogon service registers under, which as we can see from the above output is listening on ports 49155 (via TCP), 49158 (via TCP), and 49157 (via HTTP). (Netlogon registers with RPC and uses dynamic ports as we saw above, so querying the RPC end-mapper service is the only way to find what ports Netlogon is listening on). 

In contrast to PortQry, PortQryUI has options to query for the services it is aware of. So, for instance, one can use it to query the “Domains and Trusts” service on a DC and it will do PortQry queries to port 135/TCP, port 389/BOTH, port 445/TCP, port 137/UDP, and a few other AD related ports and emit the output in a window (you can see part of the output in the screenshot below). 


SRV records and AD

Example of an SRV record:

Format is:


  • class is always IN
  • TTL is in seconds 
  • service is the name of the service (example: LDAP, GC, DC, Kerberos). It is preceded by an underscore (prevents collisions with any existing names). 
  • protocol is the protocol over which the service is offered (example: TCP, UDP). It is preceded by an underscore (prevents collisions with any existing names). 
  • domain is the DNS name of the domain for which the service is offered. 
  • host is the machine that will provide this service. 
  • SRV is the text SRV. Indicates that this is an SRV record. 
  • priority is the priority of the host. Similar to SMTP MX record priorities. Host with lower number has higher preference (similar to SMTP records). 
  • weight allows for load balancing between hosts of the same priority. Host with higher number has higher preference (intuitive: higher weight wins).

Similar to MX records, the host of an SRV record must be a name with an A or AAAA record. It cannot point to a CNAME record. 

A machine starting up in a domain can easily query for DCs of that domain via DNS. DCs provide the LDAP service, so a query for _ldap._tcp.dnsdomain will return a list of DCs the machine can use. 

On DCs, the netlogon service registers many records relevant to the services offered by the DC. The A records of the DC are registered by the DNS client service (for Server 2008 and above) but the other records are taken care of by the netlogon service. A copy of the registered records is also stored in the %systemroot%\system32\config\netlogon.dns file in case it needs to be imported manually or compared for troubleshooting. Note that each DC only adds the records for itself. That is, WIN-DC01 for instance, will add a record like this:

It will not add records for WIN-DC02 and others as it does not know of them. The records added by each DC will be combined together in the DNS database as it replicates and gets records from all the DCs, so when a client queries for the above record, for instance, it will get records added by all DCs. 

AD creates a sub-domain called _msdcs.domainname to hold items pertaining to AD for that domain. (MSDCS = Microsoft Domain Controller Services) This sub-domain is created for each domain (including child-domains) of the forest. The _msdcs sub-domain belonging to the forest root domain is special in that it is a separate zone that is replicated to all DNS servers in the forest. The other _msdcs sub-domains are part of the parent zone itself. For instance below are the _msdcs sub-domains for a forest root domain (rakhesh.local) and the _msdcs sub-domain for a child domain (anoushka.rakhesh.local). Notice how the former is a separate zone with a delegation to it, while the latter is a part of the parent zone. 

For forest-root domain

For forest-root domain

For child-domain

For child-domain

Under the _msdcs sub-domain a convention such as the following is used:

Here DcType is one of dc (domain controller), gc (global catalog), pdc (primary domain controller), or domains (GUIDs of the domains).

The various SRV records registered by netlogon can be found at this link. Note that SRV records are created under both the domain/ child-domain and the forest root domain (the table in the link marks these accordingly). Below are some of the entries added by netlogon for DCs – WIN-DC04 and WIN-DC05 – in a site called KOTTAYAM for my domain anoushka.rakhesh.local. Of these WIN-DC05 is also a GC. 

Advertise that the DCs offer the LDAP service over TCP protocol for the anoushka.rakhesh.local domain:

Advertise that the DCs offer the LDAP service over TCP protocol for the anoushka.rakhesh.local domain for both sites of the forest (even though the DCs themselves are only in a single site – this way clients in any site can get these DC names when querying DNS SRV records):

WIN-DC05 has a few additional records compared to WIN-DC04 because it is a GC. 

Notice all of them are specific to its site and are created in the forest root domain zone/ _msdcs sub-domain of the forest root domain. This is because GCs are used forest-wide also. In contrast, similar records advertising the DC service are created for both sites and in the _msdcs sub-domain of the child zone:

To re-register SRV records, either restart the netlogon service of a DC or use nltest as below:


Useful ldp queries

Just dumping useful ldp.exe queries as I come across/ think of them. Better to keep them here in one consolidated post for future reference …

Find all GCs in the forest

If a DC is a GC, its NTDS Setting object has an attribute called options whose value is 0x1. Not sure if a DC is a GC and something else, whether the value changes, but for now I’ll assume it doesn’t and so one can quickly search for all GCs in the forest by connecting to the Configuration partition and filtering by the following:

  • Filter: (&(cn=NTDS Settings)(options=1))
  • Base DN: CN=Configuration,DC=domainname
  • Attributes: distinguishedName
  • Scope: Subtree


Notes on AD Replication (contd.)

This post is a continuation to my previous one

How the AD Replication Model Works


Conflict Resolution

Previously I mentioned that conflict resolution in AD does not depend on timestamp. What is used instead of the “volatility” of changes. Here’s how it works in practice. 

Remember the replication metadata stored on each object? The ones you can view using repadmin /showobjmeta. I mentioned 5 metadata there – the Local USN, the Originating DSA, the Originating USN, the Originating Timestamp, and Version. Three of these metadata are used a conflict resolution stamp for every attribute:

  • Version, which as we know is updated each time the attribute is updated
  • Originating Timestamp, which is the timestamp from the DC where the update originated from
  • Originating DSA, which is not the DSA name or GUID as you’d expect from the repadmin output, but the invocationID of the DSA where the update originated from.

How is this stamp used? If there’s a conflict to an attribute – i.e. a change is made to an attribute on two separate DCs – first the Version is considered. Whichever update has the higher Version wins. Notice how the timestamp of the change doesn’t matter. Say WIN-DC01 had a change to an attribute twice (thus incrementing the Version twice) while WIN-DC02 had a change to the same attribute once, but at a later time, and both these changes reached WIN-DC03 together – the change from WIN-DC01 will win over the later change from WIN-DC02 because the number of changes were more there. 

If two conflicting changes have the same Version then the timestamp is considered. This has a one-second resolution, and so unless the conflict changes happened at the exact same second this is usually enough to resolve the conflict. 

However, if both Version and timestamp are unable to resolve the conflict, then the invocationID is considered. This is guaranteed to be different for each DC, and is random, so whichever change is from a DC with higher invocationID wins. 

Replication Metadata

The Knowledge Consistency Checker (KCC) (will be discussed in a later post) is the component that is responsible for maintaining the replication topology. It is maintains connection objects with the replication partners and stores this information, for each domain partition, in a multivalued attribute called repsFrom in the root of that domain partition. 

For example, here are the replication partners for WIN-DC02. Although not shown here, WIN-DC04 & WIn-DC05 are of a child domain. 


Now consider the repsFrom attribute of the domain partition on WIN-DC02: 

And here’s the repsFrom from the Configuration partition:

Each entry starts from dwVersion and contains information like the number of failures, time of last successful sync, the DSA GUID, the database GUID, USNs, etc. Since only one DC is replicating with WIN-DC02 for the domain partition there’s only one value for that partition; while there are three DCs replicating for the Configuration partition and so there are three values for that partition. 

Each DC polls the DSAs (DCs) in this attribute for changes (that’s for the scheduled changes, not the ones where the source DC sends and update to all its partners and they poll for changes). If a DC is demoted – i.e. its NTDS settings object is deleted (i.e. the DSA is no longer valid) – the KCC will remove this DSA from the attribute. This prevents replication attempts to demoted DCs. (Prior to Windows 2003 though, and even now if this attribute is assigned a value, there used to be an attribute called replTopologyStayOfExecution. This value had a default of 14 days, and a maximum value of half the tombstone lifetime (the period for which deleted objects are retained). In the presence of this attribute – which existed by default in Window Server 2003 and prior, and can be set if required in later versions – if the KCC detects an invalid DSA, instead of removing it from the repsFrom attribute it will let it remain until such time the duration of the object being deleted exceeds replTopologyStayOfExecution). 


Atomicity is a term encountered in databases and operating systems (I first encountered it during my CS classes, specifically the OS course I think). An atomic operation can be thought of as an indivisible operation – meaning all events that take place during an atomic operation either take place together, or they don’t. (It comes from the idea that an atom was thought to be indivisible). With respect to databases, this is a guarantee that if a bunch of values are written in an atomic operation, either all the values are written or none of them. There’s no confusion that perhaps only a few values got committed while the rest were missed out. Even if one value didn’t get written, all others will not be written. That’s the guarantee! 

In the context of AD, updates are written to the AD database. And all attribute updates to a single object will be written atomically in a single transaction. However:

  • If the attributes are linked attributes (remember the previous post where there are attributes with forward and back links, for e.g. member and memberOf) the updates won’t be atomic – not too surprising, they are for different objects after all, and also usually the back link is generated by the system not sent as an update. 
  • Remember: the maximum number of values that can be written in a single transaction is 5000. 
  • To ensure that (nonlinked) attributes to an object are written atomically, updates to nonlinked attributes are prioritized over updates to linked attributes. This happens when a source DC packages all the updates into replication packets. The DC prioritizes nonlinked attributes over linked attributes. When it comes to writing the updates to the destination DC database though:
    • For linked attributes, because of parent-child relationships the objects might be written out of order to how the updates are received. This is to ensure that objects are created before any links are applied to that object.
    • When an object already exists on the destination DC, even though nonlined attributes are replicated first, they are not guaranteed to be written first to the database. Generally they are applied first, but it’s not guaranteed. (Note to self: I am not very clear about this point)
  • Remember: the number of values in a replication packet is approximately 100. If there are more than 100 values, again the nonlinked attributes are tried to put in one packet, while the linked attributes can span multiple packets. In such cases, when they are written on the destination DC database, all updates to a single object can require multiple transactions. (They are still guaranteed to be written in the same replication cycle). 
  • Note: Only originating updates must be applied in the same database transaction. Replicated updates can be applied in more than one database transaction.