Notes on X.400, X.500, IMCEA, and Exchange

Was reading about X.400, X.500, IMCEA, and Exchange today and yesterday. Here are my notes on these.

SMTP

We all know of SMTP email addresses. These are email addresses of the form user@domain.com and are defined by the SMTP standard. SMTP is the protocol for transferring emails between users with these addresses. It runs on the IP stack and makes uses of DNS to look up domain names and mail servers.

SMTP was defined by the IETF for sending emails over the Internet. SMTP grew out of standards developed in the 1970s and became widely used during the 1980s.

X.400

Before SMTP became popular, during the initial days of email, each email system had its own way of defining email addresses and transferring emails.

To standardize the situation the ITU-T came up with a set of recommendations. These were known as the X.400 recommendations, and although they were designed to be a new standard they didn’t catch on as expected. Mainly because the ITU-T charged for X.400 and systems had to be certified as X.400 compliant by the ITU-T; but also because X.400 was more complex (the X.400 recommendations were published in two large books known as the Red Book and Blue Book, with an additional book added later called the White Book) and its email addresses were complicated. X.400 is still used in many areas though, where some of its advantageous features over SMTP are required. For instance,  in military, financial (EDI), and aviation systems. Two articles that provide a good comparison of X.400 and SMTP are this MSExchange article and this Microsoft KB page.

X.400 email addresses are different and more complicated compared to SMTP email addresses. For instance the equivalent of an SMTP email address such as some.one@somedomain.com in X.400 could be C=US;ADMD=;PRMD=SomeDomain;O=First Organization;G=Some;S=One.

From the X.400 email address above one gets an idea of how the address is defined. An X.400 address uses a hierarchical naming system and consists of a series of elements that are put together to form the address. Elements such as C (country name), ADMD (administrative management domain, usually the ISP), O (organization name), G (given name), and so on. The original X.400 standard only specified the elements but not the way they could be written. Later, RFC 1685 defined one way of writing an X.400 address. The format defined by RFC 1685 is just one way though. Other standards too exist, so there is no one way of writing an X.400 address. This adds to the complexity of X.400.

X.400 was originally designed to run on the OSI stack but has since been adapted to run on the TCP/IP stack.

X.500

The ITU-T, in association with the ISO, designed another standard called the X.500. This was meant to be a global directory service and was developed to support the requirements of X.400 email messaging and name lookup. X.500 defined many protocols running on the OSI stack. One of these protocols was DAP (Directory Access Protocol), specified in X.511, and it defined how clients accessed the X.500 directory. A popular implementation of DAP for the TCP/IP stack is LDAP (Lightweight Directory Access Protocol). Microsoft’s Active Directory is based on LDAP.

(As an aside, the X.509 standard for a Public Key Infrastructure (PKI) began as part of the X.500 standards).

The X.500 directory structure is familiar to those who have worked with Active Directory. This is not surprising as Active Directory is based on X.500 (via LDAP). Like Active Directory and X.400, X.500 defines a hierarchical namespace, with elements having Distinguished Names (DNs) and Relative Distinguished Names (RDNs). As with X.400 though, X.500 does not define how to represent the names (i.e. X.500 does not define a representation like CN=SomeUser,OU=SomeOU,OU=AnotherOU,DC=SomeDomain,DC=Com, it only specifies what the elements are). X.500 does not even require the elements to be strings as Active Directory and LDAP do. The elements are represented using an ASN.1 notation and each X.500 implementation is free to use its own form of representation. The idea being that whist different X.500 implementations can use their own representation – since the elements used are specified by the X.500 standard these implementations can communicate with each other by passing the elements without structure.

Active Directory and LDAP use a string representation.

  • LDAP uses a comma-delimited list arranged from right to left (i.e. the the object referred to appears rightmost (last), the root appears left most (first)). For example: DC=Com,DC=SomeDomain,OU=AnotherOU,OU=SomeOU,CN=SomeUser.
  • Active Directory uses a forward slash (/) and arranges the list from left to right (i.e. the object referred to appears left most (first), the root appears rightmost (last)). For example: CN=SomeUser/OU=SomeOU/OU=AnotherOU/DC=SomeDomain/DC=Com. Unlike LDAP, Active Directory does not support the C (Country name) element and it uses DC (Domain Component) instead of O (Organization).

Some examples can of X.500 elements in non-string representations can be found on this page from IBM and this page from Microsoft.

Exchange

Now on to how all these tie together. Since I am mostly concerned with Exchange that’s the context I will look at these in.

Exchange 4.0, the first version of Exchange, used X.400 (via RPC) internally for email messaging. It had its own X.500/LDAP based directory service (Windows didn’t have Active Directory yet). User objects had an X.400 email address and were referred to in the X.500/LDAP directory via their obj-Dist-Name attribute (Distinguished Name (DN) attribute).

For example, the obj-Dist-Name attribute could be /O=SomeDomain/OU=SomeOU/OU=AnotherOU/CN=SomeUser. This object might have an X.400 email address of C=US;ADMD=;PRMD=SomeDomain;O=First Organization;OU1=SomeOU;G=UserFirstName;S=UserLastName.

The obj-Dist-Name attribute is important because this is what uniquely represents a user. In the previous example, for instance, the user with obj-Dist-Name attribute /O=SomeDomain/OU=SomeOU/OU=AnotherOU/CN=SomeUser could have two X.400 addresses: C=US;ADMD=;PRMD=SomeDomain;O=First Organization;OU1=SomeOU;G=UserFirstName;S=UserLastName and C=US;ADMD=;PRMD=SomeDomain;O=First Organization;OU1=SomeOU1;G=UserFirstName;S=UserNewLastName for instance (the last name is different between these addresses). Emails addressed to both addresses are to be sent to the same user, so Exchange resolves the X.400 address to the obj-Dist-Name attribute and routes emails to the correct user object.

This approach has a catch, however. Suppose a user object is moved to a different OU or renamed, its obj-Dist-Name attribute too changes. The Distinguished Name (DN) is made up of the Common Name (CN) and path to the object after all, so any changes to these will result in a new Distinguished Name and even though the object may have the same X.400 email addresses as before, as far as the X.500/ LDAP directory is concerned it is a new user. Exchange is fine with this because it will resolve an email sent to one of the X.400 addresses to the new object, but the catch is that clients (such as Outlook) cache the From: address of an email with the obj-Dist-Name attribute rather than the X.400 email address (because that’s what Exchange uses after all) and so any replies sent to such emails will bounce with an NDR. As far as Exchange is concerned the object referenced by the older obj-Dist-Name attribute does not exist, so it doesn’t know what to do with the email. This catch also affects users when Outlook caches email addresses. Again, Outlook caches the obj-Dist-Name attribute of object rather than the X.400 address, so if the obj-Dist-Name attribute changes the older one is no longer valid and emails to it bounce.

To work around this, apart from the X.400 address and the obj-Dist-Name attribute of an object, a new address type called the X.500 address is created in case of obj-Dist-Name attribute changes. This X.500 address simply holds the previous obj-Dist-Name attribute value (and that’s why it’s called an X.500 address as its format is like that of the obj-Dist-Name attribute, it holds the previous X.500 location address of the object). Each time the obj-Dist-Name attribute changes, a new X.500 address is created with the older value. When clients send an email to a non-existent obj-Dist-Name attribute, Exchange finds that it does not exist and so searches through the X.500 addresses it knows. If the X.500 address was created correctly, Exchange finds a match and the email is delivered to the user.

Exchange 5.0 was similar to Exchange 4.0 but it had an add-on called Internet Mail Connector which let it communicate directly with SMTP servers on the Internet. This was a big step because Exchange 5.0 users could now email SMTP addresses on the Internet. These users don’t have an SMTP address themselves, so Exchange 5.0 needed some way of providing them with an SMTP address automatically. Thus was born the idea of encapsulating addresses using an Internet Mail Connector Encapsulated Address (IMCEA) method.

IMCEA lets Exchange 5.0 send emails from a user with a X.400 address to an external SMTP address by encapsulating the X.400 address within an IMCEA encapsulated SMTP address. It also let the user receive SMTP emails at this IMCEA encapsulated SMTP address. IMCEA is a straightforward way of converting a non-SMTP email address to an IMCEA encapsulated SMTP address (the string “IMCEA” followed by the type of the non-SMTP address followed by a dash is prefixed to the address; the address is modified to leave alphabets and numbers as they are, slashes are converted to underscores, everything else is converted to its ASCII code with a plus put before the code; lastly the Exchange server’s primary domain is appended with an @ symbol). Thus every non-SMTP email address has a corresponding IMCEA encapsulated address and vice versa.

Consider the following X.400 address: C=US;ADMD=;PRMD=SomeDomain;O=First Organization;G=UserFirstName;S=UserLastName.

It could be encapsulated in IMCEA as IMCEAEX-C+3DUS+3BADMD+3D+3BPRMD+3DSomeDomain+3BO+3DFirst+20Organization+3BG+3DUserFirstName+3BS+3DUserLastName@somedomain.com. As you can see the IMCEA address looks like a regular SMTP email address (albeit a long and weird looking one).

Internally Exchange 5.0 continued using X.400 (via RPC) for email messaging and stored objects in its X.500/LDAP directory.

Exchange 5.5 was similar to Exchange 5.0. It added an X.400 connector to let it communicate with other X.400 email systems. Internally it continued using X.400 (via RPC) for email messaging and stored objects in its X.500/LDAP directory.

Exchange 2000 took another big step in that it no longer had its own directory service. Instead it made use of Active Directory, which had launched with Windows 2000 Server. Exchange 2000 thus required a Windows 2000 Active Directory environment.

Internally, Exchange 2000 switched to SMTP for email messaging. This meant new accounts created on Exchange 2000 would have an SMTP and X.400 email addresses by default.  Exchange 2000 still supported X.400 via its MTA (Message Transfer Agent) and the X.400 connector let it talk to external X.400 systems, but SMTP was what the server’s core engine used.

The switch to Active Directory meant that Exchange 2000 started using the Active Directory attribute distinguishedName instead of obj-Dist-Name to identify objects. The distinguishedName attribute in Active Directory has a different form to the obj-Dist-Name in the X.500/LDAP directory. To give an example, an object with obj-Dist-Name attribute of /O=SomeDomain/OU=SomeOU/OU=AnotherOU/CN=SomeUser could have the equivalent distinguishedName attribute as CN=SomeUser,OU=AnotherOU,OU=SomeOU,O=SomeDomain.

The distinguishedName attribute is dynamically updated by Active Directory based on the location of the object and its name (i.e. moving the object to a different OU or renaming it will change this attribute).

To provide backward compatibility with Exchange 5.5, older clients, and 3rd party software, Exchange 2000 added a new attribute called LegacyExchangeDN to Active Directory. This attribute holds the info that previously used to be in the obj-Dist-Name attribute (in the same format used by that attribute). Thus in the example above, the LegacyExchangeDN attribute would be /O=SomeDomain/OU=SomeOU/OU=AnotherOU/CN=SomeUser. A new service called the Exchange Recipient Update Service (RUS) was introduced to keep the LegacyExchangeDN attribute up to date.

Although Exchange 2000 switched to SMTP for email messaging, it continues using the X.500/LDAP directory addressing scheme (via the LegacyExchangeDN attribute) to route internal emails. This means the LegacyExchangeDN must be kept up-to-date and correctly formatted for internal email delivery to work. When the Exchange server receives an internal email, it looks up the SMTP email address in Active Directory and finds the LegacyExchangeDN to get the object. If the LegacyExchangeDN attribute cannot be found, it searches the X.500 address (as before) to find a match. Thus Exchange 2000 behaves similar to its predecessors except for the fact that it uses the LegacyExchangeDN attribute.

Exchange 2003 was similar to Exchange 2000.

Exchange 2007 dropped support for X.400. This also meant Exchange 5.5 couldn’t be upgraded to Exchange 2007 (as Exchange 5.5 users would have X.400 addresses and support for that is dropped in Exchange 2007). New accounts created on Exchange 2007 have only SMTP addresses. Migrated accounts will have X.400 addresses too.

Exchange 2007 dropped both the X.400 connector and MTA. To connect to an external X.400 system Exchange 2007 thus requires use of a foreign connector that will send the message to an Exchange 2003 server hosting an X.400 connector.

Exchange 2007 also got rid of the concept of administrative groups. This affects the format of the LegacyExchangeDN attribute in the sense that previously a typical LegacyExchangeDN attribute would be of the format /o=MyDomain/ou=First Administrative Group/cn=Recipients/cn=UserA (the administrative group name could change) but now it would be of the format /o=MyDomain/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=UserA (notice the administrative group name is fixed as there’s only one default group; “FYDIBOHF23SPDLT” is code for “EXCHANGE12ROCKS”).

Exchange 2010 was similar to Exchange 2007 in this context.

Exchange 2010 SP1 Rollup 6 made a change to the LegacyExchangeDN attribute to add three random hex characters at the end for uniqueness (previously Exchange would append an 8 digit random number only if there was a clash).

Exchange 2013 is similar to Exchange 2010 to the best of my knowledge (haven’t worked much with it so I could be mistaken).

Implications

It’s worth stepping back a bit to look at the implications of the above.

Say I am in an Exchange 2010 SP2 environment and I have a user account testUser1 with SMTP email address testUser1@mydomain.com. The Active Directory object associated with this account could have a LegacyExchangeDN attribute of the form /o=MyDomain/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=testUser1abc. If I have previously sent emails from this SMTP email address to another internal user, that user’s Outlook will cache the address of testUser1 as /o=MyDomain/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=testUser1abc and NOT testUser1@mydomain.com.

Next, I create a new user testUser2, remove the testUser1@mydomain.com SMTP email address from testUser1, and assign it to testUser2. This new account will have its own LegacyExchangeDN attribute.

Now, suppose Outlook were to reply to one of the emails it previously received, one would expect the reply to go to testUser2 as that’s where the testUser1@mydomain.com SMTP address now points to (and Exchange 2010 uses SMTP internally so nothing else should matter!) but that is not what happens. Instead the reply will go to testUser1 as Outlook has stored the LegacyExchangeDN attribute of this object in its received email, and so even though we have typed an SMTP address that is now assigned to a different user, Exchange only goes by the LegacyExchangeDN attribute and happily routes email to the first user.

On the other hand, suppose another user (who has never received written to the testUser1@mydomain.com address before) were to compose a fresh email to the testUser1@mydomain.com address it will correctly go to the newly created account. This is because Outlook doesn’t have a cached LegacyExchangeDN entry for this address and so it is up to the Exchange server to resolve the SMTP address to LegacyExchangeDN and it correctly picks up the new account.

A variant of this recently affected a colleague of mine. We use Outlook 2010 in cached mode at work. This colleague made a new user account and email address. Later that evening she was asked to delete that account and recreate it afresh. She did so, then noted that the new account has the same SMTP email address as the previous one, and so assumed everything should work fine. They did, sort of.

The user was able to receive external emails fine, and also emails sent to Distribution Lists she was a member of. But if anyone internally were to compose a fresh email to this user, it bounces with an NDR.

Generating server: mail.domain.com

IMCEAEX?_O=EXAMPLE_OU=EXCHANGE+20ADMINISTRATIVE+20GROUP+20+28FYDIBOHF23SPDLT+29_CN=RECIPIENTS_CN=LastName+2C+20FirstName264@domain.com
#550 5.1.1 RESOLVER.ADR.ExRecipNotFound; not found ##

Original message headers:

Received: from …

From the NDR we can see the email was sent to the LegacyExchangeDN attribute address (encapsulate in an IMCEA address as Exchange uses SMTP for delivering messages). Since this is the first time any users are emailing this address, it should have worked as there’s no cached LegacyExchangeDN attribute value to mess things up. And it would have worked, but for two things:

  1. Because we are on Exchange 2010 SP1 Rollup 6 or above (notice the three hex digits in the LegacyExchangeDN address encapsulated in the IMCEA address above) the LegacyExchangeDN value has changed even though the user account details are exactly the same as before. If we were on an earlier version of Exchange the LegacyExchangeDN attribute would have stayed the same and the email will be happily be delivered. But the random digits change the LegacyExchangeDN attribute to be a different one, thus bouncing the email. The email is being sent to a LegacyExchangeDN attribute value ending with the digits “264”, while the actual digits are “013” (found via the Get-Mailbox userName | fl LegacyExch* cmdlet).
  2. That still doesn’t explain why Outlook has outdated info. Since no one has emailed this new user before, Outlook shouldn’t be having cached info.This is where the fact that Outlook was in cached mode comes into the picture. Cached mode implies Outlook has an Offline Address Book. The Exchange server generates the OAB periodically, clients download the OAB periodically. So it’s possible one of these stages still has the cached info. The best fix here is to manually regenerate the OAB, then get Outlook to download the OAB. We did that at work here, and users were now able to email the new user correctly (of course, if they have already tried emailing the user before and got an NDR, be sure to remove the name from the auto complete list and re-select from GAL).

Recap

  • X.400 addresses are of the form: C=US;ADMD=;PRMD=SomeDomain;O=First Organization;G=Some;S=One. They are hierarchical and consist of elements. There is no true standard for the representation of an X.400 address.
  • X.500 is a directory service specification. It is not an email address (just clarifying!). The representation of an X.500 object is up to the implementation. 
    • An LDAP example would be: DC=Com,DC=SomeDomain,OU=AnotherOU,OU=SomeOU,CN=SomeUser.
    • An AD example would be: CN=SomeUser/OU=SomeOU/OU=AnotherOU/DC=SomeDomain/DC=Com.
  • Exchange 4.0, 5.0, 5.5 used X.400 internally. Exchange 2000 onwards use SMTP internally.
    • All versions of Exchange identify objects via their obj-Dist-Name attribute (Exchange 4.0 – 5.5) or distinguishedName attribute (Exchange 2000 and upwards) in the directory.
    • Example obj-Dist-Name attribute: /O=SomeDomain/OU=SomeOU/OU=AnotherOU/CN=SomeUser.
    • Example distinguishedName attribute: CN=SomeUser,OU=AnotherOU,OU=SomeOU,O=SomeDomain.
    • Notice the format is different!
  • For backwards compatibility Exchange uses the LegacyExchangeDN attribute. It is same as the obj-Dist-Name attribute.
    • This attribute is kept up-to-date by Exchange.
    • For all intents and purposes it is fair to say Exchange uses this attribute to identify objects. Users may think they are emailing SMTP addresses internally, but they are really emailing LegacyExchangeDN attributes.
  • Since the LegacyExchangeDN attribute changes when the location or name of an object changes, and this could break mail-flow even though the X.400 or SMTP address may not have changed, Exchange adds and X.500 “address” for each previous LegacyExchangeDN attributes an object may have. That is, when the LegacyExchangeDN attribute changes, the previous value is stored as an X.500 address. This way Exchange can fall back on X.500 addresses if it is unable to find an object by LegacyExchangeDN.
  • IMCEA is a way of encapsulating a non-SMTP address as an SMTP address. Exchange uses this internally when sending email to an X.500 address or LegacyExchangeDN attribute. Example IMCEA address: IMCEAEX?_O=EXAMPLE_OU=EXCHANGE+20ADMINISTRATIVE+20GROUP+20+28FYDIBOHF23SPDLT+29_CN=RECIPIENTS_CN=LastName+2C+20FirstName264@domain.com

That’s all for now!