Contact

Subscribe via RSS

Categories

Creative Commons Attribution 4.0 International License

SRV records and AD

Example of an SRV record:

Format is:

Where:

  • class is always IN
  • TTL is in seconds 
  • service is the name of the service (example: LDAP, GC, DC, Kerberos). It is preceded by an underscore (prevents collisions with any existing names). 
  • protocol is the protocol over which the service is offered (example: TCP, UDP). It is preceded by an underscore (prevents collisions with any existing names). 
  • domain is the DNS name of the domain for which the service is offered. 
  • host is the machine that will provide this service. 
  • SRV is the text SRV. Indicates that this is an SRV record. 
  • priority is the priority of the host. Similar to SMTP MX record priorities. Host with lower number has higher preference (similar to SMTP records). 
  • weight allows for load balancing between hosts of the same priority. Host with higher number has higher preference (intuitive: higher weight wins).

Similar to MX records, the host of an SRV record must be a name with an A or AAAA record. It cannot point to a CNAME record. 

A machine starting up in a domain can easily query for DCs of that domain via DNS. DCs provide the LDAP service, so a query for _ldap._tcp.dnsdomain will return a list of DCs the machine can use. 

On DCs, the netlogon service registers many records relevant to the services offered by the DC. The A records of the DC are registered by the DNS client service (for Server 2008 and above) but the other records are taken care of by the netlogon service. A copy of the registered records is also stored in the %systemroot%\system32\config\netlogon.dns file in case it needs to be imported manually or compared for troubleshooting. Note that each DC only adds the records for itself. That is, WIN-DC01 for instance, will add a record like this:

It will not add records for WIN-DC02 and others as it does not know of them. The records added by each DC will be combined together in the DNS database as it replicates and gets records from all the DCs, so when a client queries for the above record, for instance, it will get records added by all DCs. 

AD creates a sub-domain called _msdcs.domainname to hold items pertaining to AD for that domain. (MSDCS = Microsoft Domain Controller Services) This sub-domain is created for each domain (including child-domains) of the forest. The _msdcs sub-domain belonging to the forest root domain is special in that it is a separate zone that is replicated to all DNS servers in the forest. The other _msdcs sub-domains are part of the parent zone itself. For instance below are the _msdcs sub-domains for a forest root domain (rakhesh.local) and the _msdcs sub-domain for a child domain (anoushka.rakhesh.local). Notice how the former is a separate zone with a delegation to it, while the latter is a part of the parent zone. 

For forest-root domain

For forest-root domain

For child-domain

For child-domain

Here are some entries added by netlogon for two DCs – WIN-DC04 and WIN-DC05 – in a site called KOTTAYAM for the domain anoushka.rakhesh.local. Of these WIN-DC05 is also a GC. 

Advertise that the DCs offer the LDAP service over TCP protocol for the anoushka.rakhesh.local domain:

Advertise that the DCs offer the LDAP service over TCP protocol for the anoushka.rakhesh.local domain for both sites of the forest (even though the DCs themselves are only in a single site – this way clients in any site can get these DC names when querying DNS SRV records):

WIN-DC05 has a few additional records compared to WIN-DC04 because it is a GC. 

Notice all of them are specific to its site and are created in the parent zone/ _msdcs sub-domain of the parent zone. I am not sure why these are specific to the GC site. Similar records advertising the DC service, for instance, are created for both sites:

 

Useful ldp queries

Just dumping useful ldp.exe queries as I come across/ think of them. Better to keep them here in one consolidated post for future reference …

Find all GCs in the forest

If a DC is a GC, its NTDS Setting object has an attribute called options whose value is 0x1. Not sure if a DC is a GC and something else, whether the value changes, but for now I’ll assume it doesn’t and so one can quickly search for all GCs in the forest by connecting to the Configuration partition and filtering by the following:

  • Filter: (&(cn=NTDS Settings)(options=1))
  • Base DN: CN=Configuration,DC=domainname
  • Attributes: distinguishedName
  • Scope: Subtree

 

Quick ffmpeg tricks (extract audio, convert audio codec of a video)

I use ffmpeg to convert between audio and video formats. Two situations I use it for are: (1) when I have a video whose audio codec is of a format my media players can’t handle (I use WD TV Live and it has trouble with certain formats), or (2) when I want to extract the audio only from a video. 

For converting only the audio here’s the command I use:

The syntax is pretty obvious. The input file is taken, the video codec is copied as-is to the new file, the audio codec is re-encoded to mp3. I could have used -acodec mp3 too. In the past there used to be an in-built mp3 encoder as well as the mp3 encoder provided by the LAME project (ffmpeg must be built with LAME encoder via the --enable-libmp3lame switch for this to work) so you could choose between either via the two switches, but now there’s no in-built encoder so both --acodec mp3 and --acodec libmp3lame do the same. 

To confirm what switches ffmpeg was built with simply run the command. For example, on my machine:

Notice it was built with --enable-libmp3lame

When it comes to extracting just the audio from a video there’s two ways to do it: 1) you can simply extract the audio in the codec as it is, or 2) you can extract & convert to the codec you want. The latter has the disadvantage that if the original video is in a lossy audio codec, converting will result in some degradation of quality. 

To check what audio codec the file is in, do the following: 

Notice it identifies the audio stream as AAC in this case. If I am happy with extracting that as it is I can do the following:

The -vn switch tells ffmpeg to ignore the video. The -acodec copy switch tells it to copy the audio codec as it is. Since this is an AAC file, I assign an extension of .aac to the output file. 

However, if I didn’t want the audio stream as AAC, I would have done the following:

Here I am converting the audio to mp3. Once again I ignore the video via -vn. I specify the audio codec via -acodec libmp3lame. The rest of the switches are as follows:

  • -ac 2 => two channels (stereo) (note this is same as the input, so is an optional switch)
  • -ar 44100 => sammple rate 44100 Hz (CD quality) (note this is same as the input, so is an optional switch)
  • -ab 320k => bit-rate of 320 kb/s (if I don’t specify this the bit-rate will be 128 kb/s for mp3)

Essentially, instead of just copying the audio codec you convert it. Otherwise the idea is the same. Apart from libmp3lame (or mp3) I could have also used the following audio codecs: vorbis (for ogg), aac, flac, and wma

Before concluding, here’s a link to ffmpeg’s documentation (for all the command-line switches etc). Also, this is a good page on audio/ video containers and ffmpeg. The latter is a very thorough and informative page, I am sure I’ll be referring to it in the future. 

[Aside] A great discussion on copyright and patents

Check out this episode of the Exponent podcast by Ben Thompson and James Allworth.

Ben Thompson is the author of the stratechery blog which is an amazing read for Ben’s insights into technology. James Allworth, I am not much aware of but he was terrific in the above episode. I began following this podcast recently and have only heard one episode prior to this (on Xaomi, again a great episode; in fact the one on copyrights and patents continues on a point mentioned in the Xaomi episode).

This episode was great for a couple of reasons. I felt Ben was caught out of element in the episode, which is unlike how I have read/ heard him previously where he is confident and authoritative. In this episode he was against abolishing copyrights – which is what James was advocating for – but he didn’t have convincing arguments. So he resorted to the usual arguing tricks like prop up examples and try to get the argument to be about the example (and when it still failed he would withdraw the example claiming it wasn’t appropriate here). Or he’d just take a firm stand and refuse to budge. Or incite James by insults and such. Or try and win by conflating the argument with something else which had no relation to it. Basically, usual debating tricks I believe, but it was fun to hear and I was surprised to hear him resorting to these.

Eventually when Ben clarified his point it made sense to me. His argument is that patents are harmful when they apply to “ingredients” (parts of an invention, e.g. pull to refresh) but he has no issues when it applies to a whole thing (e.g. medicine). Moreover, the question is whether the presence of the patent is required to spur invention (not required in the case of technology, required/ preferred in case of medicines) and whether society would be better off without the monopoly afforded by patents (again, no in the case of tech as it leads to barriers of enter and unnecessary patent wars and trolling for new inventions). Copyright, for Ben, is neither harmful to society nor will its absence spur more innovation, so he doesn’t see why it must be abolished. He seems to agree that copyright has its negatives and is harmful in some cases, but he still feels it is useful to make supply scarce (by preventing others from copying the work).

James agrees with most of these but his point is that the same effect can be arrived at without copyrights. Maybe by innovation in other areas, or by agreements between the creator and audience. His argument is more about considering a world without as an option, and to look at how things can be done differently. Moreover, such a world will lead to more creativity and he feels that’s better in the long run.

I can’t write more as I have a flight to catch, so I’ll end this post now. And it’s better to hear the arguments than my summary. Go check out the podcast. It’s a great one! Skip the first few minutes as it is some user feedback etc.

[Aside] Improving your PuTTY connections

Wish I had come across this during my PuTTY days!

The TL;DR summary is that by default PuTTY uses an ASCII encoding while most Linux and other OSes uses UTF-8 encoding. It’s because of this mismatch that manpages and such sometimes appear with â characters. Change the PuTTY encoding and find happiness! 

Start Menu and other directory junctions access denied

Here’s a screenshot of my c:\Users\UserName folder. Notice the “Start Menu” folder:

users-folder

It is a directory junction to another folder, left behind for compatibility purposes. If I double click the folder though I get an access denied:

access-denied

 

The actual target to which this points is accessible, but the junction itself isn’t. Every time I encounter this I think I must make a note of “why” this is so, but I forget. Today I intend to note it down once and for all. 

Remember from an earlier post: there are hard links and there are soft links. The former points to the data itself, and so only work for files. The latter points to the folder/ file containing the data. Soft links are an evolution of directory junctions (there are directory junctions and volume junctions). While directory junctions make use of something called reparse points and were introduced in Windows 2000, soft links were introduced in Vista and are baked into the kernel itself. Microsoft uses directory junctions – as we saw above – to redirect some of its special folders.

The important thing with both directory junctions and soft links is that they can have their own ACLs. So while a user might have full permissions to the target folder, the directory junction or soft link itself may not grant the user permissions and so the contents cannot be accessed via the directory junction or soft link. That’s what happening here too. 

First, from the command prompt note that this is a junction and that I can enter the directory junction but cannot see any files:

The files exist, however, as I can directly browse the target:

Compare the ACLs of the target and directory junction and we see the problem:

(I could have used the Explorer GUI here but I prefer icacls. In the GUI we have to dig down a bit more to see the relevant ACEs).

Notice a DENY entry for Everyone on the directory junction for listing the contents (RD). That’s why I can’t list the junction contents (in Explorer double clicking results in trying to list the contents, while in Command Prompt entering a junction and listing are two separate tasks – that’s why entering the junction worked, but listing the contents failed). 

What the above tells us is that only listing the junction contents is prohibited. If we know the names of some folders in there – as older software for whom this directory junction is present would know – we can go to those locations using the directory junction. Thus the following works:

There you go!

Add Readability and Instapaper ‘Read Now’ links to your posts

Yesterday I went back and read one of my older posts from my tablet. That made me realize my blog doesn’t have a mobile friendly view. Sure, it doesn’t have any ads or widgets, and the posts appear clean on a browser as my emphasis is on the text/ code, but that doesn’t translate well to a mobile device as the fonts are small and a bit of zooming and scrolling is required to hide the left sidebar and other bits. 

Eventually I read the post using a Readability bookmarklet I had on the mobile browser so that got me thinking I must add quick links to do this for each post so any visitors can take advantage of the same. 

I use Instapaper for most of my reading. For some posts that Instapaper has difficulty rendering (mostly posts with a lot of code, pictures) I use Pocket. I prefer Readability over Pocket as its iOS app is terrific, but Readability’s Android app sucks (poor UI, syncing issues, doesn’t keep track of my last read location) and so I use Pocket rather than Readability. 

For publishers Readability offers an embed code. This is good in that it allows one to read the page in a Readability view without adding to Readability (similar to what I did yesterday using the bookmarklet). It also lets you add the page to Readability, print it, send to Kindle, or email – useful stuff. What I don’t like about the embed code, though, is that it pulls in JavaScript from their website and adds a block of buttons to my posts. I don’t want a block of buttons – in my case, all I want is to offer users a link they can click to get the page in a Readability view. 

For readers Readability offers the bookmarklets I mentioned earlier. I wrapped the “Read Now” bookmarklet as a link for my purpose (as I’ll show in a bit). 

Instapaper too gives bookmarklets for readers – the “Instapaper Text” bookmarklet one is what I am interested in. For publishers Instapaper gives an URL that will add the page to the reader’s Instapaper queue. Unfortunately, there’s no similar URL to simply show the page in an Instapaper view without adding to queue. 

Pocket too gives bookmarklets and tools for publishers. However, both options only allow the page to be added to the Pocket queue, there’s no way to just get a Pocket view display of the page. 

So Readability and Instapaper are what I can use. I added the following text to each of my posts (I use the Atahualpa theme so it was just a matter of adding this text to the “Byline” section of each post and applying some formatting):

All this does is that it wraps the Javascript code from the Readability Read Now and the Instapaper Text bookmarklets within an a block like thus:

When a user clicks the text the JavaScript is executed as though they had clicked on the bookmarklet. (Hat tip to this page where I picked up the syntax from. I haven’t programmed with JavaScript so had to search around for how to invoke JavaScript from a link). 

Hope this helps someone! I have put links to both Instapaper and Readability because I prefer Instapaper but it is lousy with code blocks while Readability handles code blocks better. I feel Instapaper is better for text – it has more fonts, background colors, etc.

[Aside] What doesn’t seem like work?

From this essay by Paul Graham:

If something that seems like work to other people doesn’t seem like work to you, that’s something you’re well suited for.

That essay links to another longer essay about work and doing what you love. Have added it to my Instapaper queue.

Notes on AD Replication (contd.)

This post is a continuation to my previous one

How the AD Replication Model Works

(source)

Conflict Resolution

Previously I mentioned that conflict resolution in AD does not depend on timestamp. What is used instead of the “volatility” of changes. Here’s how it works in practice. 

Remember the replication metadata stored on each object? The ones you can view using repadmin /showobjmeta. I mentioned 5 metadata there – the Local USN, the Originating DSA, the Originating USN, the Originating Timestamp, and Version. Three of these metadata are used a conflict resolution stamp for every attribute:

  • Version, which as we know is updated each time the attribute is updated
  • Originating Timestamp, which is the timestamp from the DC where the update originated from
  • Originating DSA, which is not the DSA name or GUID as you’d expect from the repadmin output, but the invocationID of the DSA where the update originated from.

How is this stamp used? If there’s a conflict to an attribute – i.e. a change is made to an attribute on two separate DCs – first the Version is considered. Whichever update has the higher Version wins. Notice how the timestamp of the change doesn’t matter. Say WIN-DC01 had a change to an attribute twice (thus incrementing the Version twice) while WIN-DC02 had a change to the same attribute once, but at a later time, and both these changes reached WIN-DC03 together – the change from WIN-DC01 will win over the later change from WIN-DC02 because the number of changes were more there. 

If two conflicting changes have the same Version then the timestamp is considered. This has a one-second resolution, and so unless the conflict changes happened at the exact same second this is usually enough to resolve the conflict. 

However, if both Version and timestamp are unable to resolve the conflict, then the invocationID is considered. This is guaranteed to be different for each DC, and is random, so whichever change is from a DC with higher invocationID wins. 

Replication Metadata

The Knowledge Consistency Checker (KCC) (will be discussed in a later post) is the component that is responsible for maintaining the replication topology. It is maintains connection objects with the replication partners and stores this information, for each domain partition, in a multivalued attribute called repsFrom in the root of that domain partition. 

For example, here are the replication partners for WIN-DC02. Although not shown here, WIN-DC04 & WIn-DC05 are of a child domain. 

win-dc02-replications

Now consider the repsFrom attribute of the domain partition on WIN-DC02: 

And here’s the repsFrom from the Configuration partition:

Each entry starts from dwVersion and contains information like the number of failures, time of last successful sync, the DSA GUID, the database GUID, USNs, etc. Since only one DC is replicating with WIN-DC02 for the domain partition there’s only one value for that partition; while there are three DCs replicating for the Configuration partition and so there are three values for that partition. 

Each DC polls the DSAs (DCs) in this attribute for changes (that’s for the scheduled changes, not the ones where the source DC sends and update to all its partners and they poll for changes). If a DC is demoted – i.e. its NTDS settings object is deleted (i.e. the DSA is no longer valid) – the KCC will remove this DSA from the attribute. This prevents replication attempts to demoted DCs. (Prior to Windows 2003 though, and even now if this attribute is assigned a value, there used to be an attribute called replTopologyStayOfExecution. This value had a default of 14 days, and a maximum value of half the tombstone lifetime (the period for which deleted objects are retained). In the presence of this attribute – which existed by default in Window Server 2003 and prior, and can be set if required in later versions – if the KCC detects an invalid DSA, instead of removing it from the repsFrom attribute it will let it remain until such time the duration of the object being deleted exceeds replTopologyStayOfExecution). 

Atomicity

Atomicity is a term encountered in databases and operating systems (I first encountered it during my CS classes, specifically the OS course I think). An atomic operation can be thought of as an indivisible operation – meaning all events that take place during an atomic operation either take place together, or they don’t. (It comes from the idea that an atom was thought to be indivisible). With respect to databases, this is a guarantee that if a bunch of values are written in an atomic operation, either all the values are written or none of them. There’s no confusion that perhaps only a few values got committed while the rest were missed out. Even if one value didn’t get written, all others will not be written. That’s the guarantee! 

In the context of AD, updates are written to the AD database. And all attribute updates to a single object will be written atomically in a single transaction. However:

  • If the attributes are linked attributes (remember the previous post where there are attributes with forward and back links, for e.g. member and memberOf) the updates won’t be atomic – not too surprising, they are for different objects after all, and also usually the back link is generated by the system not sent as an update. 
  • Remember: the maximum number of values that can be written in a single transaction is 5000. 
  • To ensure that (nonlinked) attributes to an object are written atomically, updates to nonlinked attributes are prioritized over updates to linked attributes. This happens when a source DC packages all the updates into replication packets. The DC prioritizes nonlinked attributes over linked attributes. When it comes to writing the updates to the destination DC database though:
    • For linked attributes, because of parent-child relationships the objects might be written out of order to how the updates are received. This is to ensure that objects are created before any links are applied to that object.
    • When an object already exists on the destination DC, even though nonlined attributes are replicated first, they are not guaranteed to be written first to the database. Generally they are applied first, but it’s not guaranteed. (Note to self: I am not very clear about this point)
  • Remember: the number of values in a replication packet is approximately 100. If there are more than 100 values, again the nonlinked attributes are tried to put in one packet, while the linked attributes can span multiple packets. In such cases, when they are written on the destination DC database, all updates to a single object can require multiple transactions. (They are still guaranteed to be written in the same replication cycle). 
  • Note: Only originating updates must be applied in the same database transaction. Replicated updates can be applied in more than one database transaction.

 

Notes on AD Replication, Updates, Attributes, USN, High-Watermark Vector, Up-to-dateness Vector, Metadata, etc.

Reading a couple of AD Replication posts from TechNet/ MSDN. Below are my notes on them. 

Note

This is a super long post! It began as notes to the TechNet/ MSDN posts but it quickly grew into more. Lots of additional stuff and examples from my side/ other blog posts. Perhaps I should have split these into separate posts but I didn’t feel like it.

What is the AD Replication Model?

(source)

  • AD makes use of pull replication and not push. That is, each DC pulls in changes from other DCs (as opposed to the DC that has changes pushing these to targets). Reason for pulling instead of pushing is that only the destination DC knows what changes it needs. It is possible it has got some changes from another DC – remember AD is multimaster – so no point pushing all changes to a destination DC.
  • Initial replication to a new DC in the domain is via LDAP. Subsequent replications are via RPC.
  • All DCs do not replicate with all other DCs. They only replicate with a subset of DCs, as determined by the replication topology. (I am not sure, but I think this is only in the case of InterSite replication …). This is known as store and forward replication. DCs store updates they get from replication partners ams forward these to other DCs.
  • There are two sorts of replication – state-based and log-based.
    • State-based means DCs apply updates to their replicas (their copies of the partitions) as and when they arrive.
    • In log-based replication each DC maintains a log of the changes and sends this log to all other DCs. Once a destination DC receives this log it applies it to its replica and becomes up-to-date. This also means the destination DC will receive a list of all changes (from the source DC perspective), not just the changes it wants. 
    • Active Directory (actually called Active Directory Domain Services since Server 2008) uses state-based replication. 
    • Each DC maintains a list of objects by last modification. This way it can easily examine the last few objects in the list to identify how much of the replication source’s changes have already been processed. 

Linked & Multivalued attributes

Before the next post it’s worth going into Linked attributes and Multivalued attributes.

  • Linked attributes are pairs of attributes. AD calculates the value of one of the attributes (called the “back link”) based on the value of the other attribute (called the “forward link”). For example: group membership. Every user object has an attribute called memberOf. You can’t see this by default in ADUC in the “Attribute Editor” tab. But if you click “Filter” and select to show “Backlinks” too then you can view this attribute. backlinks The memberOf attribute is an example of a back link. And it’s generated by AD, which is why it’s a read-only attribute. It’s counterpart, the forward link, is the member attribute of a group object. Linked attributes are linked together in the schema. In the schema definitions of those attributes – which are instances of an attributeSchema class – there is a linkID attribute that defines the back and forward links. The attribute with an even linkID is the forward link (the write-able attribute that we can set) while the attribute with a linkID that’s one more than the linkID of the forward link is the back link (the read-only, AD generated attribute). For instance, using ADSI Edit I found the CN=Member,CN=Schema,CN=Configuration,DC=mydomain object in the Schema partition. This is the schema definition of the member attribute. Being a forward link it has an even linkID. linkid Its counterpart can be easily found using ldp.exe. Search the Schema partition for all objects with linkID of 3. linkid-2 The result is CN=Is-Member-Of-DL,CN=Schema,CN=Configuration,DC=mydomain, which is the attributeSchema object defining the memberOf attribute (notice the lDAPDisplayName attribute of this object in the above screenshot).
    • It’s worth taking a moment to understand the above example. When one uses ADUC to change a user/ group’s group membership, all one does is go to the user/ group object, the “Member Of” tab, and add a group. This gives the impression that the actual action happens in the user/ group object. But as we see above that’s not the case. Although ADUC may give such an impression, when we add a user/ group to a group, it is the group that gets modified with the new members and this reflects in the user/ group object that we see. ADUC makes the change on the user/ group, but the DSA makes the real change behind the scenes. The cause and effect is backwards here …
    • Another obvious but not so obvious point is that when you add a user/ group to a group, it is the group’s whenChanged attribute that gets changed and it is the change to the group that is replicated throughout the domain. The user object has no change and nothing regarding it is replicated. Obvious again, because the change really happens on the group and the effect we see on the user/ group is what AD calculates and shows us. 
  • Multivalued attributes, as the name suggests, are attributes that can hold multiple values. Again, a good example would be group membership. The member and memberOf attributes mentioned above are also multivalued attributes. Obvious, because a group can contain multiple members, and a user/ group can be a member of multiple groups. Multivalued attributes can be identified by an attribute isSingleValued in their attributeSchema definition. If this value is TRUE it’s a single valued attribute, else its a multivalued attribute.

How the AD Replication Model Works

(source)

A must read post if you are interested in the details! 

On replication

  • AD objects have attributes. AD replicates data at the attribute level - i.e. only changes to the attributes are replicated, not the entire object itself. 
    • Attributes that cannot be changed (for e.g. back links, administrative attributes) are never replicated. 
  • Updates from the same directory partition are replicated as a unit to the destination DC – i.e. over the same network connection – to optimize network traffic.
  • Updates are always specific a single directory partition. 
  • Replication of data between DCs consist of replication cycles (i.e. the DCs keep replicating until all the data has replicated). 
    • There’s no limit to the number of values that can be transferred in a replication cycle. 
    • The transfer of data happens via replication packets. Approximately 100 values can be transferred per packet. 
      • Replication packets encode the data using ASN.1 (Abstract Syntax Notation One) BER (Basic Encoding Rules). Check out Wikipedia and this post for more information on ASN.1 BER (note: ASN.1 defines the syntax; BER defines the encoding. There are many other encodings such as CER, DER, XER but BER is what LDAP uses). See this blog post for an example LDAP message (scroll down to the end). 
    • Once a replication cycle has begun, all updates on the source DC – including updates that occur during the replication cycle – are send to the destination DC. 
    • If an attribute changes multiple times between replication cycles only the latest value is replicated. 
    • As soon as an attribute is written to the AD database of a DC, it is available for replication. 
  • The number of values that can be written in a single database transaction (i.e. the destination DC has all the updates and needs to commit them to the AD database) is 5000. 
    • All updates to a single object are guaranteed to be written as a single database transaction. This way the state of an object is always consistent with the DC and there are no partially updated objects. 
      • However, for updates to a large number of values to multivalued attributes (e.g. the member attribute from the examples above) the update may not happen in the same transaction. In such cases the values are still guaranteed to be written in the same replication cycle (i.e. all updates from the source DC to that object will be written before the DC performs another replication cycle). (This exception for multivalued attributes is from Server 2003 and above domains, I think).
    • Prior to Windows Server 2003, when a multivalued attribute had a change the entire attribute was replicated (i.e. the smallest change that could be replicated was an attribute). 
      • For example, if a multivalued attribute such as member (referring to the members of a group) had 300 entries and now an additional entry was added, all 301 entries were replicated rather than just the new entry. 
      • So if a group had 5000 members and you added an extra member, 5001 entries were replicated as updates to its member attribute. Since that is larger than the number of values that can be committed in a single transaction, it would fail. Hence the maximum number of members in a group in a Windows 2000 forest functional level domain was 5000. 
    • Starting from Windows Server 2003, when a multivalued attribute has a change only the changed value is replicated (i.e. the smallest change that can be replicated is a value). Thus groups in Windows Server 2003 or Windows Server 2003 interim functional level domains don’t have the limitation of 5000 members. 
      • This feature is known as Linked Value Replication (LVR). I mentioned this in an earlier post of mine in the context of having multiple domains vs a single domain. Not only does LVR remove limitations such as the above, it also leads to efficient replication by reducing the network traffic.
      • When a Windows 2000 forest level domain is raised to Windows 2003/ 2003 interim forest level, existing multivalued attributes are not affected in any way. (If they were, there’d be huge network traffic as these changes propagate through the domain). Only when the multivalued attribute is modified does it convert to a replicate as single values. 
  • Earlier I mentioned how all attributes are actually objects of the class attributeSchema.  The AD schema is what defines attributes and the relations between them.
    • Before replication between two DCs can happen, the replication system (DRS – see below) checks whether the schema of both DCs match. If the schemas do not match, replication is rescheduled until the schemas match (makes sense – if replication happens between DCs with different schemas, the attributes may have different meanings and relationships, and replication could mess things up). 
    • If the forest schema is changed, schema replication takes precedence over all other replication. In fact, all other replication is blocked until the schema replicates. 

Architecture

  • On each DC, replication operates within the Directory System Agent (DSA) component – ntdsa.dll
    • DSA is the interface through which clients and other DCs gain access to the AD database of that DC. 
    • DSA is also the LDAP server. Directory-aware applications (basically, LDAP-aware applications) access DSA via the LDAP protocol or LDAP C API, through the wldap32.dll component  (am not very clear about this bit). LDAPv3 is used. 
  • A sub-component of the DSA is DRS (Directory Replication System) using the DRS RPC protocol (Microsoft’s specification of this protocol can be found at this MSDN link). Client & server components of the DRS interact with each other to transfer and apply updates between DCs. 
    • Updates happen in two ways: via RPC or SMTP (the latter is only for non-domain updates, which I infer to mean Schema or Configuration partition updates) 
    • Domain joined computers have a component called ntdsapi.dll which lets them communicate with DCs over RPC.
      • This is what domain joined computers use to communicate with the DC.
      • This is also what tools such as repadmin.exe or AD Sites and Services (dssites.msc) use to communicate with the DC (even if these tools are running on the DC). 
    • DCs additionally have a private version of ntdsapi.dll which lets them replicate with other DCs over RPC. (I infer the word “private” here to mean it’s meant only for DCs).
      • As mentioned earlier, DC to DC replication can also bypass RPC and use SMTP instead. Then the ntdsapi.dll component is not used. Other components, such as ISMServ.exe and the CDO library are used in that case. Remember: this is only for non-domain updates. 
    • The RPC interface used by DRS is called drsuapi. It provides the functions (also see this link) for replication and management. Check this section of the TechNet post for an architecture diagram and description of the components. 
      • The TechNet post also mentions DRS.idl. This is a file that contains the interface and type library definitions. 
  • The DSA also has a sub-component for performing low-level operations on the AD database. 
  • The DSA also has a sub-component that provides an API for applications to access the AD database. 
    • The AD database is called ntds.dit (DIT == Directory Information Tree; see this link for more info). The database engine is Extensible Storage Engine (ESE; esent.dll)

Characteristics of AD replication

  • Multimaster
    • Changes can be made to any DCs (as long as they are authoritative for the objects).
  • Pull
    • When an update occurs on a DC it notifies its partners. The partners then requests (pulls) changes from the source DC. 
  • Store-and-forward
    • When a DC receives a change from its partners, it stores the change and in-turn forwards on to others (i.e. informs others so they can issue pull requests from itself). Thus, the DC where a change is made does not have to update every other DCs in the domain. 
    • Because of the store-and-forward mechanism replication must be thought of as sequentially moving through the domain. 
  • State-based
    • Each DC has metadata to know the “state” of its replicas. This state is compared with that of its partner to identify the changes required. (This is in contrast to log-based replication where each DC keeps a log of changes it made and sends that log to its partners so they can replay the log and update themselves). 
    • This makes uses of metadata such as Update Sequence Number (USN), Up-to-dateness vector, and High-watermark vector. Synchronized time is not primarily used or required to keep track of this (it is used, but in addition to other mechanisms). 

Updates

LDAP/ AD supports the following four types of update requests:

  • Add an object to the directory.
  • Delete an object from the directory. 
  • Modify (add, delete, remove) attribute values of an existing object in the directory. 
  • Move an object by changing the name or parent of the object. 

Each of the above update requests generates a separate write transaction (because remember, all updates to a single object happen in a single transaction). Updates are an all-or-nothing event. If multiple attributes of an object are updated, and even one of those updates fail when writing to the database, all attributes fail and are not updated. 

Once an update request is committed to the database it is called an originating update. When a DC receives an originating update, writes it to its database, and sends out updates to other DCs these are called replication updates. There is no one-to-one relation between originating updates and replication updates. For instance, a DC could receive multiple originating updates to an object – possible even from different DCs – and then send a replication update with the new state of that object. (Remember: AD is state-based, not log-based. It is the state that matters, not the steps taken to reach that state). 

Adding request:

  • Creates a new object with a unique objectGUID attribute. Values of all replicated attributes are set to Version = 1. (The Version attribute will be discussed later, below). 
  • The add request fails if the parent object does not exist or the DC does not contain a writeable replica of the parent object’s directory partition. 

Modify request:

  • If an attribute is deleted then its value is set with NULL and Version is incremented. 
  • If values are to be added/ removed from an attribute, it is done so and the attribute Version is incremented. The Modify request compares the existing and new values. If there are no changes then nothing happens – the request is ignored, the Version does not change. 

Move request:

  • This is a special case of the Modify request in that only the name attribute is changed. 

Delete request:

  • The isDeleted attribute is set to TRUE. This marks the object as tombstoned (i.e. deleted but not removed from AD). 
  • The DN of the object is set to a value that cannot be set by an LDAP application (i.e. an impossible value). 
  • Except a few important attributes, the rest are stripped from the object (“stripped”, not removed as one would do above). 
  • The object itself is moved to the Deleted Objects container. 

Certain objects are protected from deletion:

  • Cross-references (class crossRef) and RID Object for the DC. Any delete request for these objects are rejected. And any originating update deletion of these objects is rejected, and all attributes of the object are updated (Version incremented) so the object replicates to other DCs and is reanimated wherever it was deleted.  
    • Cross-references are present in the Configuration partition: CN=Partitions,CN=Configuration,DC=(domain)
    • RID Objects are present under each DC at CN=RID Set,CN=(DC),OU=Domain Controllers,DC=(domain)
  • The NTDS Settings (class nTDSDSA) object.
    • This object represents the DSA (Directory System Agent). It is present in the Configuration partition under the site and DC: CN=NTDS Settings,CN=(DC),CN=Servers,CN=(SITE),CN=Sites,CN=Configuration,DC=(domain)
    • Remember, the DSA is the LDAP server within the DC. The DSA’s GUID is what we see when DCs replicate or in their DNS names (DSAGUID._msdcs.domain is a CNAME to the DC name).
    • The objectGUID of the DC object is not the same as the objectGUID of its DSA object. The latter is what matters. When a DC computer object is opened in ADUC, it is possible to view the corresponding DSA object by clicking on the “NTDS Settings” button in the first tab. 
    • Trivia: to find all the DCs in a forest one can query for objects of class nTDSDSA. Below query, for instance, finds all objects of that class and returns their DN and objectGUID

      nTDSDSA

    • When a DC is demoted, its DSA object is deleted but not really deleted. It is protected and disabled from receiving replication requests. Thus, a query such as the above will return DSA objects of DCs that may no longer exist in the forest.

      A better way to structure the above query then would be to also filter out objects whose isDeleted attribute is not TRUE.

      nTDSDSA-2

Conflicts

AD does not depend on timestamps to resolve conflicts. In case of conflicts the “volatility” of changes is what primarily matters. That is, if an attribute was updated twice on one DC and thrice on another DC, even if the timestamps of changes from the first DC are later than the second DC, the change from the second DC will win because the attribute was updated thrice there. More on this in my next post

Database GUID

I mentioned above that each DSA has its own objectGUID. This GUID is created with the server is promoted to a DC and deleted (sort of) when the DC is demoted. Thus the GUID is present for the lifetime of the DC and doesn’t change even if the DC name changes. 

The AD database (ntds.dit) on each DC has its own GUID. This GUID is stored in the invocationId attribute of the DSA. Unlike the DSA GUID, the database GUID can change. This happens when (1) the DC is demoted and re-promoted (so the database changes), (2) when an application NC is added/ removed to the DC, (3) when the DC/ database is restored from a backup, or (4) (only in Server 2012 and above) whenever a DC running in a VM is snapshotted or copied.

The invocationId attribute can be viewed via ldp.exe as above, or in the “NTDS Settings” of the DC computer object in AD Users & Computers, or in the “NTDS Settings” in AD Sites & Services. It can also be viewed in the output of repadmin.exe.

The invocationID is a part of replication requests (more later). When the invocationID changes other know that they have to replicate changes to this DC so it can catch up. The first DC in a domain will have its invocationID and objectGUID as same (until the invocationID changes). All other DCs will have different values for these both. 

Update Sequence Numbers (USNs)

(I was planning on covering this separately as part of my AD Troubleshooting WorkshopPLUS posts, but USNs are mentioned in this TechNet post so I might as well cover them now). 

USNs are 64-bit counters maintained on each DC. The number is different for each DC and is stored in the highestCommittedUsn attribute of the rootDSE object.

rootDSE is an imaginary object. It is the root of the directory namespace. Under it we have the various domain partitions, configuration partition, schema partition, and application partitions. rootDSE can be queried to find the various partitions known to the DC. It can be viewed through ADSI Edit or ldp.exe (the latter makes it easier to view all the attributes together).

rootDSE

As can be seen in the screenshot above, one of the attributes is highestCommittedUSN.

Each time a DC commits an update (originating update/ replication update), it updates the highestCommittedUSN attribute. Thus the USN is associated with each successful transaction to the AD database. Remember: all updates to a single object are written in a single transaction, so a USN is essentially associated with each successful update to an object – be it a single attribute, or multiple updates to the same/ multiple attributes (but all written in the same transaction). 

USNs are update incrementally for each write transaction. So, for example, when I took the above screenshot the USN for WIN-DC02 was 77102. When I am writing this paragraph (the next day) the USN is 77230. This means between yesterday and today WIN-DC02 has written 128 transactions (77230-77102) to its database.  

Every attribute of an object has a USN associated with it. This is a part of the replication metadata for that object, and can be viewed through the repadmin.exe command. For instance, using the /showobjmeta switch (note that here we specify DC first and then object):

Notice how the attribute USNs vary between the two DCs. Also notice the metadata stored – the Local USN, the originating DSA, the Originating DSA's USN, the timestamp of the update, and Version. If you are running the above command it is best to set the command prompt window width to 160, else the output doesn’t make much sense. I will talk about Local USN and Originating USN in a moment. 

Another switch is /showmeta (here we specify the object first and then the DC):

Both switches seem to produce the same output. The important items are the Local USN and Originating USN, and the Version

  • Version starts from 1 and is incremented for each update. Version will be same on all DCs – when an DC commits an update request (originating update or replication update) it will increment the Version. Since all attributes start with the same Version = 1, the current value will be the same on all DCs. 
  • Local USN is the USN value for that attribute on the DC we are looking at. It is the value of the highestCommittedUSN for the transaction that committed the update to this attribute/ set of attributes
  • Originating USN is the USN for that attribute on the DSA where the originating update was sent from.

For instance: the attribute description. WIN-DC01 has local USN 46335, WIN-DC02 has 19340, and WIN-DC03 has 8371. The latter two DCs got their update for this attribute from WIN-DC01 so they show the originating DSA as WIN-DC01 and the Originating USN as 46335.

Every object has an attribute called uSNChanged. This is the highest Local USN among all the attributes of that object. (What this means is that from the USN of an object we can see which of its attributes have the same local USN and so easily determine the attributes changed last). Unlike the attribute USNs which are metadata, uSNChanged is an object attribute and thus can be viewed via ADSI Edit or ADUC. From the command line, repadmin.exe with the /showattr switch can be used to view all attributes of an object (this switch takes the DC name first, then the object). 

Above output tells me that on WIN-DC01 the uSNChanged for this object is 46335.  From the earlier /showobjmeta output I know that only the description attribute has that local USN, so now I know this was the last attribute changed for this object. 

So, to recap: DCs have a USN counter stored in highestCommitedUSN attribute. This counter is updated for each successful write transaction to the database. Since writes happen per object, it means this USN is updated every time an object is updated in the database. Each object attribute has its own USN – this too is per DC, but is not an attribute, it is metadata. Finally, each object has a uSNChanged attribute which is simply the highest attribute USN of that object. This too is per DC as the attribute USN is per DC. The DC’s highestCommittedUSN attribute and an object’s uSNChanged attribute are related thus: 

  • When an attribute update is committed to the database the DC’s highestCommittedUSN is incremented by 1.
  • The Local USN of the attribute/ attributes is set to this new highestCommittedUSN
  • This in turns updates the object’s uSNChanged to be the new highestCommittedUSN (because that is the highest attribute Local USN now)

Thus, the highestCommittedUSN is the highest uSNChanged attribute among all the replicas held by the DC.  

Here’s an example. First I note the DC’s highestCommitedUSN (I have removed the other attributes from the output). 

Then I note an object’s uSNChanged:

Now I connect via ADUC and change the description field. Note the new USN. 

And note the DC’s USN:

It has increased, but due to the other changes happening on the DC it is about 10 higher than the uSNChanged of the object I updated. I searched the other replicas on my DC for changes but couldn’t find anything. (I used a filter like (uSNChanged>=125979) in ldp.exe and searched every replica but couldn’t find any other changes. Maybe I missed some replica – dunno!) This behavior is observed by others too. (From the previous linked forum post I came across this blog post. Good read).  

Finally, I must point out that even though I said the attribute metadata (Local USN, Originating USN, Originating DSA, Originating Time/ Date, Version) are not stored as attributes, that is not entirely correct. Each object has an attribute called replPropertyMetaData. This is an hexadecimal value that contains the metadata stored as a binary value. In fact, if we right click on an object in ldp.exe it is possible to view the replication metadata because ldp.exe will read the above attribute and output its contents in a readable form. 

ldp-metadata

Bear in mind this attribute is not replicated. It is an attribute that cannot be administratively changed, so can never be updated by end-users and doesn’t need replicating. It is only calculated and maintained by each DC. uSNChanged is a similar attribute – not replicated, only administratively changed. 

Note to self: I need to investigate further but it looks like uSNChanged cannot be viewed by ldp.exe for all objects. For instance, ldp.exe shows this attribute for one user object in my domain but not for the other! I think that’s because the attribute is generated by the AD tools when we view it. repadmin.exe and the GUI tools show it always. Similarly, the attribute level metadata attribute too cannot be viewed by ldp.exe for all objects. For some objects it gives an error that the replPropertyMetaData attribute cannot be found and so cannot show the replication metadata. This could also be why there was a gap between the uSNChanged and highestCommittedUSN above. 

High-watermark Vector (HWMV) & Up-to-dateness Vector (UTDV)

To recap once more, what we know so far is:

  • Every attribute has replication metadata that contains its Local USN, the originating DSA's USN, the originating DSA, the timestamp, and Version
  • The highest Local USN of an attribute is stored as the object’s uSNChanged attribute. 
  • The highest uSNChanged attribute among all objects on all replicas held by the DSA is stored as its highestCommittedUSN attribute in the rootDSA
  • All these USN counters are local to the DC (except for the Originating USN which is local to that the originating DSA). 

How can DCs replicate with each other using the above info? Two things are used for this: High-watermark vector and Up-to-dateness vector. 

The HWMV is a table maintained on each DC.

  • There is one table for each directory partition the DC holds replicas for (so at minimum there will be three HWMV tables – for the domain partition, the Schema partition, and the Configuration partition). 
  • The HWMV table contains the highest USN of the updates the DC has received from each of its replication partners for that replica. Note: the table contains the USN that’s local to the replication partner.
  • Thus the HWMV can be thought of as a table contain the highest uSNChanged value for each directory partition of its replication partners when they last replicated.

Got it? Now whenever a DC needs to replicate with another DC, all it needs to do is ask it for each of the changes since the last uSNChanged value the destination is aware of! Neat, right! The process looks roughly like this:

  1. The originating DC will have some changes. These changes will cause uSNChanged values of the affected objects to change. The highestCommittedUSN of the DC too changes. All these are local changes.
  2. Now the DC will inform all its replication partners that there are changes. 
  3. Each of these replication partners will send their HWMV for the partition in question.
  4. From the received HWMV table, the originating DC can work what changes need to be sent to the destination DC. 
    1. The originating DC now knows what USNs it needs to look for. From this it knows what objects, and in turn which of their attributes, were actually changed. So it sends just these changed attributes to the destination DC. 

To make the process of replication faster, all DCs have one more table. That is the Up-to-dateness Vector (UTDV).

  • Like the HWMV, this too is for every replica the DC holds. 
  • Unlike the HWMV this contains the highest USN of the updates the DC has received from every other DC in the domain/ forest for that replica. 
  • The table also contains the timestamp of last successful replication with each of those DCs for that replica.

The UTDV table has a different purpose to the HWMV. This table is sent by the destination DC to the originating DC when it requests for changes – i.e. in step 3 above.

When the originating DC gets the UTDV table for its destination DC, it can look at the table and note the destination DC’s highest received USNs for that partition from other DCs. Maybe the destination DC has asked for changes from USN number x upwards (the USN number being of the originating DC). But these changes were already received by the destination DC from another DC, under USN number y and upwards (the USN number being of that other DC). The destination DC does not know that the changes requested by USN x and upwards are already with it from another DC, but by looking at the UTDV table the originating DC can see that USNs y and above already contain the changes the destination DC is requesting, so it can filter out those updates when sending. (This feature is called “propagation dampening”). 

  • In practice, once the originating DC compiles a list of USNs that need to be sent to the destination DC – at step 4 above – it goes through its replication metadata to check each attribute and the originating DSA and originating USN associated with that attribute.
  • The originating DC then looks at the UTDV table of the destination DC, specifically at the entry for the DC that had sent it an update for the changed attribute. (This DC could be same as the originating DC). From the table it can see what USNs from this DC are already present at the destination DC. If the USN value in the table is higher than the USN value of the attribute, it means the destination DC already has the change with it. So the originating DC can filter out these and similar attributes from sending to the destination DC.  

Thus the UTDV table works along with the HWMV table to speed up replication (and also avoid loops wherein one DC updates another DC who updates the first DC and thus they keep looping). And that is how replications happen behind the scenes! 

Once a destination DC updates itself from an originating DC – i.e. the replication cycle completes – the source DC sends its UTDV table to the destination DC. The destination DC then updates its UTDV table with the info from the received UTDV table. Each entry in the received table is compared with the one it has and one of the following happens:

  • If the received table has an entry that the destination DC’s UTDV table does not have – meaning there’s another DC for this replica that it isn’t aware of, this DC has replicated successfully with the originating DC and so all the info it has is now also present with the destination DC, and so it is as good as saying this new DC has replicated with the destination DC and we are aware of it the same way the originating DC is aware – so a new entry is added to the destination DC’s UTDV table with the name of this unknown DC and the corresponding info from the received UTDV table. 
  • If the received table has an entry that the destination DC’s UTDV table already has, and its USN value is higher than what the destination DC’s table notes – meaning whatever changes this known DC had for this partition has already replicated with the originating DC and thus the destination DC – and so its entry in the UTDV can actually be updated, the UTDV table for that server is updated with the value from the received UTDV table.  

The UTDV table also records timestamps along with the USN value. This way DCs can quickly identify other DCs that are not replicating. These timestamps record the time the DC last replicated with the other DC – either directly or indirectly. 

Both HWMV and UTDV tables also include the invocationID (the database GUID) of the DCs. Thus, if a DC’s database is restored and its invocationID changes, other DCs can take this into account and replicate any changes they might have already replicated in the past.  

From the book I am reading side-by-side (excellent book, highly recommended!) I learnt that apart from the HWMV and UTDV tables and the naming context it wants to replicate, the destination DC also sends two other pieces of info to originating DC. These are: (1) the maximum number of object updates the destination DC wishes to receive during that replication cycle, and (2) the maximum number of values the destination DC wishes to receive during that replication cycle. I am not entirely clear what these two do. Once a replication cycle begins all object updates and values are sent to the destination DC, so the two pieces above seems to be about whether all the updates are sent in one replication packet or whether they are split and sent in multiple packets. The maximum number of values in a single packet is about 100, so I guess these two numbers are useful if you can only accept less than 100 values per packet – possibly due to network constraints. 

More later …

Unfortunately I have to take a break with this post here. I am about halfway down the TechNet post but I am pressed for time at the moment so rather than keep delaying this post I am going to publish it now and continue with the rest in another post (hopefully I get around to writing it!). As a note to myself, I am currently at the Active Directory Data Updates section, in the sub-section called “Multimaster Conflict Resolution Policy”.

[Aside] Some quotes

On passion

I think it’s not exactly true to say that if you do something you are passionate about, the money will come. But, if you do almost anything really diligently then the money will come. And it takes passion to have that kind of diligence. And … if you almost anything really diligently you will figure out what parts of it like and what you don’t and that will help inform your choices (when it comes to choosing a job or doing another thing). 

- Isaac Schlueter (from this podcast)

On money

Money is not the root of all evil. The love for money is the root of all evil. 

- Apparently this is the original version but somewhere along the line became misquoted. I came across this via another podcast (I think Jonathan Strickland said the correct version on the podcast)  

[Aside] Pocket Casts (web) Player

I use (and luuuuuv!) the excellent Pocket Casts podcasts app. I discovered it when I switched to the OnePlus One and it’s probably my number 1 used app on the phone. I have discovered so many new podcasts thanks to it (because it has a good discovery tab and also because I listen to more podcasts now so I try and discover new stuff).

I wouldn’t be too far from the truth in saying that one major reason why I am still with the OnePlus One rather than going back to my iPhone 5S is Pocket Casts. Not coz Pocket Casts isn’t available for iOS, but because the OnePlus One with its 64GB of storage lets me download tons of podcasts while my iPhone 5S is a measly 16GB and so I can’t download much stuff for offline use. Sure, the OnePlus One’s larger screen is good for reading, but I don’t do much reading on that nowadays. The OnePlus One has the advantage of 64GB storage and the ability to just copy movies and music to it via MTP, while the iPhone 5S has the disadvantage of low storage (in my case) and the inability to just copy stuff over it without iTunes. The iPhone 5S keyboard is way better though (I hate typing on the OnePlus One, even with custom keyboards like Flesky) and its camera is miles ahead of the OnePlus One too. 

Anyways, I digress …

Pocket Casts is an amazing podcasts and you must check it out if you are into podcasts. Apart from a ton of great features, it also has sync. This means all my subscriptions on the OnePlus One will easily be in sync on the iPhone 5S too. More importantly, not just the subscriptions, but also my progress with each podcast. Isn’t that cool! 

As if that wasn’t cool enough though, I discovered via the latest episode of All About Android (where one of the Pocket Casts developer was a guest) that they now have a web player version. W00t! You can view it at https://play.pocketcasts.com/ – as with the Android/ iOS apps it’s not free, there’s a one time purchase, but that’s fine in my opinion. Free doesn’t have a long term future so I am happy paying for apps & services as long as they are good and meet my needs. (Apparently the web player was released on their blog in October itself). The web player too syncs with the mobile apps.

A note about the sync: unlike Kindle syncing for books (which is what I am used to) the app/ web player does not prompt you that you are currently on a further location with another device. If you already have the podcast open and you click “play” it will continue from where you are. But if you re-open the episode it will continue from where you are on the other device. 

Update: Some blog posts from Russell Ivanovic (one of the creators of Pocket Casts; he was the guest in the All About Android podcast). The first post was mentioned on the podcast so I checked out his blog:

p.s. I began using the iOS version of Pocket Casts the other day, and boy, is the Android version much better designed! Wow. This is the first app where I feel the Android version is much better designed than iOS. Most other apps have it the other way around. In the podcast Russell too mentioned something similar – that they are giving priority to Android over iOS nowadays. 

[Aside] Elsewhere on the Web

  • GeekTyper – go to the site, click one of the themes shown, and just start typing away random stuff on your keyboard. You may type gibberish but the screen will appear as though you are typing something important – be it in a Word doc, or as if you were to hack into a website like in movies. Impressive stuff. (via)
  • Why aren’t we using SSH for everything? – good question! The author created an SSH based chat room. I haven’t tried it but I liked the post and the details it goes into. (via)
  • Al Weiwei is Living in Our Future – a good read on privacy and surveillance. Starts off with Chinese artist Al Weiwei who is under permanent surveillance (initially in prison but now house arrest) but soon moves on to how Governments and Corporations now use technology to keep us under ubiquitous surveillance. Creepy. Gotta read two books I came across from this post:

Switching to Route 53

Started using Amazon’s Route 53 (cool name!) today to serve DNS queries for this domain (rakhesh.com). Previously I was using DNS services from my excellent email provider FastMail but I thought I must try other options just to know what else is there. I was testing another domain of mine with Route 53 this past month, today I switched this main domain over. 

For my simple DNS requirements the two things that attracted me to Route 53 were the cost-effective pricing and the fact that it uses Anycast (see this CloudFlare post for an intro to Anycast).  It also has two other features that I am interested in exploring – Health Checks and the fact that it has an API. Need to explore these when I get some time. 

A cool thing about the AWS documentation is that it lets you download as a PDF or send to Kindle. Wish the TechNet/ MSDN documentation had a similar feature. Currently it’s a hit and miss when I send pages from TechNet/ MSDN to Kindle (via the Chrome extension). Sometimes bits and pieces are missed out, which I only realize later when reading the article. InstaPaper never manages to get any of the text, while Pocket is generally good at getting it (but I don’t use Pocket as it doesn’t have highlights, while both Kindle and InstaPaper have that). 

[Aside] Couple of GPO notes

Figured something that had been bugging me for long. Got a couple of Win 8.1 machines in my test lab and startup scripts would take a while to launch on them. Initially I thought it must be security related or something, then I noticed that the Win 7 machines in the lab don’t have an issue. This got my checking for any Win 8.1 changes with startup scripts, and found this blog post. Turns out with Win 8.1 startup scripts are delayed for 5 mins! Good news is this setting can be turned off via a GPO – Computer Configuration\Policies\Administrative Templates\System\Group Policy\Configure Logon Script Delay.

Another thing I learnt today was regarding applying language settings via GPO preferences. By default they don’t get applied until you press the F5/F6 key to mark it as enabled. 

Lastly, just as a reminder to myself – to change the default Start Menu power button from Shutdown to anything else, use this GPO setting

Windows Server 2012 licensing

I have been catching up on a lot of MVA courses lately and one of them mentioned a short course on Server 2012 licensing so I checked that out yesterday.

The first half of the course made sense to me but the second half didn’t. Mainly coz I haven’t worked with Server & Cloud Enrollment (SCE) and the course assumes you are aware of it, so it only talks about the improvements with Server 2012 R2. From the first half here are some salient points:

  • Server 2012 R2 has four editions. Two of these are for small business (less than 25 & 15 users). Two of these are for private clouds.
    • The private cloud editions are Windows Server 2012 R2 Datacenter and Windows Server 2012 R2 Standard. 
    • The small business editions are Windows Server 2012 R2 Essentials and Windows Server 2012 R2 Foundation.
  • Unlike Server 2008 where the three private cloud editions – Standard, Enterprise, and Datacenter – had differing features, now Standard and Datacenter have the same features. They only differ in terms of virtual instance licenses (i.e. the number of times you can use the license as a virtual server).
    • Standard has two virtual instance licenses.
      • Worth noting that these virtual licenses can be stacked. That is if you attach two licenses to a server then you get 2+2 = 4 virtual licenses. 
      • If you have multiple licenses, when you step up from Standard to Datacenter via the Software Assurance program you can only exchange one Standard license for a Datacenter license. Meaning, you can’t step up two Standard licenses for one Datacenter licenses. It’s a one-t0-one upgrade. 
    • Datacenter has unlimited virtual instance licenses. (This explains why all the Microsoft training centers use Datacenter! No need to worry about guest OS licensing then.)
  • Also, in Server 2008 Datacenter was licensed per processor, while Enterprise and Standard were licensed per server. But in Server 2012 R2 both Standard and Datacenter are licensed per processor.
    • A single license covers up to two processors on a single server. 
  • In contrast Server 2012 R2 Foundation and Essentials are licensed per server. 
    • Essentials is licensed per server for a maximum of two processors.
    • Foundation is licensed per server for a maximum of one processor. 
  • Unlike Server 2008, both versions of Server 2012 – Data Center and Standard – have the same features. Only difference is in terms of virtualization. The Data Center edition supports unlimited virtual guests, the Standard edition only supports two.
  • In addition to the licensing above, you also need device or user Client Access Licenses (CALs). 

The Disturbances of my Mind

I like classical music. Both Western and Indian. I also like Jazz and most instrumental music. I don’t know why I like them, nor do I understand much about the performances themselves, except that I like them. For instance I hear people talk about how such and such performance was great or how a certain artist added his/ her touch to a particular piece, but none of that usually makes sense to me. I just enjoy the music and some performers, but I have no real reason behind it. Nor do I have any drive to learn a musical instrument or create music etc – I am just an audience who likes to enjoy the performance, not a creator, probably not even much of a technical admirer.

I like learning languages. I know English, Hindi, Malayalam, and can understand Tamil. I spent 4 months in a beginner course of German (but had to give up for other reasons even though I was quite good at it and the instructor was impressed with my interest). Most people in the German class had joined coz they wanted to relocate to Germany. I too had that reason in mind, but it was a secondary reason – I was more into the class coz I liked learning a new language, and I was very influenced by the process of learning a new language or how it got me thinking differently. I want to learn Arabic. I don’t know of any classes near my place, plus time is a constraint I guess, but Arabic is something I definitely want to learn. Back to English, I like it a lot and I love reading and trying to write stuff in it. But I am not very creative. Once in a while I get a spark and I write some stuff – once in a few years I mean – but that’s about it. But I like the language as such, and I love reading to get a feel of how to use the language, and I try to use what I read in whatever little bit I write. And even though I am not very creative I do try and write stuff like this blog – which isn’t very creative, I know, but is nevertheless an outlet to write.

Similarly I love computer languages. I love C, C++, Perl, Bash, PowerShell. I also know Java, HTML, and CSS. I want to learn Python, Ruby, new stuff like node.js (JavaScript). I briefly programmed in LISP and Scheme in college and I think they are amazing languages and would like to learn more about them. When I think of these languages the feeling I have is of passion, of love. I don’t just think of them as tools or as a means to an end. I think of them as an end themselves. I really love these languages, I really love the time I spent learning and programming in them – those are some of my fondest memories. But again, I am not very creative. I am not good at coming up with smart algorithms, but if I am given an algorithm I am good at expressing it. I think I write beautiful code – at least that’s what I always felt compared to my classmates in college. They’d come up with smart algorithms and generally solve the problem way better I ever could, but there was no beauty to their code. My code, on the other hand, was beautiful and I was proud of whatever I had come up with. I looked neater, more elegant, and I always felt that was because I loved the language and looked to expressing what I want beautifully in that language. Sort of like how you’d use certain words that are better suited than other words, to express the same idea. Words matter. The language matters. But the underlying point is I am not creative. I may love the language, I may love the music, but I am not creative enough to come up with my own creations – and that has always hurt. Why am I not creative enough?

On to computers themselves. My biggest and sort of only passion. (I have other passions like reading about evolution, psychology, history, etc. but none of them come near my passion for computers). Again, I have no clue why I love computers so much. I don’t even do much with computers – at work I am a glorified helpdesk person though I know I am much more capable than that. Again, I see others who are doing more work than me – implementing stuff, designing solutions – but here I am. Most of these people don’t even love computers the way I do, I feel. To them computers are a means to an end – of implementing whatever they are doing, of getting paid – but not to me. I really love this stuff, and it really hurts me that I can’t spend as much time doing the stuff I want to do. For instance, I love the BSDs. (I am not much into Linux – they are fine, and I like Debian and Slackware – but I find Linux too superficial, too confused, too much about just doing things for some random reason. BSDs have more “soul” as far as I am concerned). I wish I were doing more stuff with BSDs. Maybe maintaining webservers, email servers, DNS servers etc with them. Not in isolation, but along with Windows – which too I love, and which I feel has really jumped in leaps and bounds since Server 2008. At a previous job I met someone who had implemented a Samba Active Directory environment using Fedora, with Windows XP clients. I wish I were doing that! The closest I have ever come to doing something like that was implementing a Samba Active Directory environment for my Dad’s office, using Debian Squeeze with Windows 7 clients. It was a good experience but I didn’t get to doing much. I learnt a lot of Samba and realized how cumbersome it was to do the simplest of things with it, but I also feel it probably felt cumbersome coz I never used it much. I mean, looking after my Dad’s office wasn’t really my full time work so I’d only do this now and then – which meant the workflow wasn’t ingrained into me and most of the time I’d forget how to do things when I needed to do it again. Plus there were issues which I couldn’t sort out the way I wanted to coz I wasn’t full time there. If it were my full time job I could have experimented with a test PC, got things right, then rolled it out to everyone. But I didn’t have that luxury there so it was a matter of me picking up things as I went along without much time to test or experiment. That was very lousy and eventually when someone else was going to take care of their IT stuff (coz his office had merged with another office) I was happy to let go.

Still, the point remains that I love these things and I wish I were more creative and/ or had more opportunities. I tack these together because sometimes I feel creativity is probably also related to opportunities. You have to put coal through fire (pressure) to get a diamond. Similarly maybe if I had enough opportunities (pressure) I might pick stuff up and get better and better at it and start being creative. It amazes me how some people are able to solve problems wonderfully in PowerShell, or implement superb solutions with the BSDs – just blows my mind! Compared to such people I am just a kid. My gripe isn’t that I am a kid, mind you – that’s OK, I am a kid because of the kind of opportunities presented to me which have only offered me limited growth – my gripe is that I wish I had more learning opportunities so I had a chance to grow, to do things, to learn, to develop myself, to just do stuff I love. Ideally I am doing a bit of server stuff – Windows, BSDs – plus dabbling a bit in coding. Not a full time programmer mind you, but someone who dabbles in it, and knows enough coding to quickly put together stuff and/ or tweak existing stuff. I do a bit of the latter now and then – especially in PowerShell at work – but my output (and the quality of output) has been dwindling because there aren’t enough opportunities and so I slowly forget what I know and because of this the output suffers. A year ago, for instance, most of my PowerShell scripts were much better written – with plenty of switches and some good coding – but over time, due to disuse, I forget most of it, so now when I am have to write some code I know it isn’t as excellent as my previous effort. If I had more opportunities I would be more in touch with the concepts – which I can easily pick up, after which it’s only a matter of retaining them by regular use – so opportunities are what I want. Plus a creative spark to make use of these opportunities and really do amazing stuff with the things I love.

This rant has been all over the place, I know. Off late I have been listening to too many podcasts on things I love – like the BSDs – and today I was listening to a podcast on Perl and that just overwhelmed me. I love Perl, and I still remember picking it up from Larry Wall’s book (and what an amazing book that was! he was someone with passion for languages and that showed itself in the book and Perl) and using it in one of our programming assignments. I was able to solve it way easily than my classmates coz Perl made it easy, and I just loved coding in Perl and writing stuff in it. The podcast brought back all those memories, along with all the regrets, so I finally quit listening to it midway … but by then my mind was already disturbed and I had to let it out somewhere, which is what this blog post is for. The Disturbances of my Mind. 

[Aside] Everything you need to know about cryptography in 1 hour – Colin Percival

Good talk about crypto by Colin Percival. The talk is slightly less than 2 hours long (but that could be due to questions at the end – I don’t know, I am only 50 mins into it so far). 

A good talk. With a lot of humor in between. Only reason I decided to watch this video is because I follow Colin’s blog (or rather I try to follow – there’s a lot of crypto stuff in there that goes over my head). He was the FreeBSD Security officer when I was into FreeBSD and that’s how I started following his blog and became aware of him.  Am glad I am watching it though, it clarifies a lot of things I was vaguely aware of without going into too much details. I don’t think I’ll ever be doing programming that involves crypto, but it’s good to know and is an area of interest. 

[Aside] Multiple domain forests

Was reading about multiple domains/ child domains in a forest and came across these interesting posts. They talk pretty much the same stuff. 

Key points are:

  • In the past a domain was considered to be the security boundary. But since Windows 2000 a domain is no longer considered a security boundary, a forest is.
  • Domain Admins from child domain can gain access to control the forest. The posts don’t explain how but allude that it is possible and widely known.
  • Another reason for multiple domains was password policies. In Windows Server 2000 and 2003 password policies were per domain. But since Windows Server 2008 it is possible to define Fine-Grain Password Policies (FGPPs) that can override the default domain password policy. 
  • Multiple domains were also used when security was a concern. Maybe a remote location had poor security and IT Admins weren’t comfortable with having all the domain usernames and password replicating to DCs in such locations. Solution was the create a separate domain with just the users of that domain. But since Windows Server 2008 we have Read-Only Domain Controllers (RODCs) that do not store any password and can be set to cache passwords of only specified users.
  • Yet another reason for multiple domains was DNS replication. In Windows Server 2000 AD integrated DNS zones replicated to all DCs of the domain – that is, even DCs not holding the DNS role. To avoid such replication traffic multiple domains were created so the DNS replication was limited to only DCs of those domains. Again, starting Windows Server 2003 we have Application Partitions which can be set to replicate to specific DCs. In fact, Server 2003 introduced two Application Partitions specifically for DNS – a Forest DNS Zone partition, and a Domain DNS Zone partition (per domain). These replicate to all DCs that are also DNS servers in the forest and domain respectively, thus reducing DNS replication traffic. 
  • Something I wasn’t aware of until I read these articles was the Linked Value Replication (LVR). In Server 2000 whenever an attribute changed the entire attribute was replicated – for example, if a user is added to a group, the list of all group members is replicated – obviously too much traffic, and yet another reason for multiple domains (to contain the replication traffic). But since Server 2003 we have LVR which only replicates the change – thus, if a user is added to the group, only the addition is replicated. 

One recommendation (itself a matter of debate and recommended against in the above two posts) is to have two domains in the forest with one of them being a placeholder:

  1. A root domain, which will be the forest root domain and will contain the forest FSMO roles as well as Schema Admins and Enterprise Admins; and 
  2. A child domain, which will be the regular domain and will contain everything else (all the OUs, users, Domain Admins)

The root domain will not contain any objects except the Enterprise & Schema admins and the DCs. Check out this article for a nice picture and more details on this model. It’s worth pointing out that such a model is only recommended for medium to large domains, not small domains (because of overhead of maintaining two domains with the additional DCs).

Also check out this post on domain models in general. It is a great post and mentions the “placeholder forest root domain” model of above and how it is often used. From the comments I learnt why it’s better to create child domains rather than peer domains in case of the placeholder forest root domain model. If you create peers there’s no way to indicate a specific domain is the forest root – from the name they all appear the same – while if you create child domains you can easily identify who the forest root is. Also, with child domains you know that the parent forest root domain is important because you can’t remove that domain (without realizing its role) because the child domain namespace depends on it. Note that creating a domain as child to another does not give Domain Admins of the latter administrative rights to it (except of course if these Domain Admins are also Enterprise Admins). The domain is a child only in that its namespace is a child. The two domains have a two way trust relationship – be it peers or parent/ child – so users can logon to each domain using credentials from their domain, but they have no other rights unless explicitly granted. 

The authors of the “Active Directory (5th ed.)” book (a great book!) recommend keeping things simple and avoiding placeholder forest root domains.

[Aside] Systems, Science and FreeBSD

Interesting talk by George Neville-Neil on FreeBSD and research. At Microsoft Cambridge, of all the places! 

I am listening to two BSD podcasts nowadays – bsdtalk and BSD Now – and it’s been both great (coz I just love BSD!) and sad (coz I just love BSD and wish I were using them as actively as I was many years ago … sigh!). Came across the above talk via BSD Now, I listened to the audio version (you don’t really need the video version as it’s mostly talking and the slides are a separate download anyway and you can skip those). Earlier this week I listened to the bsdtalk interview with Mathew Dillon, founder of DragonFlyBSD. That was a great interview. I admire Matthew Dillon a lot and have listened to his previous interview (interviews?) on bsdtalk and elsewhere. Takes passion and vision and knowledge to stand by what you feel and fork an OS such as FreeBSD and actually get it going this far – I totally admire that! Sadly, DragonFlyBSD is the one I have least used – except briefly to check out HAMMER.

Anyways, thought I’d post links to these …