Pure Exchange 2000 Admin Groups and Group Membership

[The_Exchange_Team]

I just worked on one of these again today, so time to post about this issue.  If you create pure Exchange 2000 or 2003 Administrative Groups, you’ll want to read on to find out how these appear in 5.5 and about an interesting issue if you don’t have your ADC connection agreements set up just right.

First, a background – If you are running in a mixed mode organization (Exchange 5.5 + Exchange 2000 and/or Exchange 2003), you can use Exchange System Manager to create a new Administrative Group, then install new Exchange 2000/2003 servers into that Administrative Group.  This is known as a pure Exchange 2000 / 2003 Administrative Group (Referred to hereafter as a “Pure AG”).  When you do this, some special magic has to occur to get this new Pure AG to appear in Exchange 5.5 as a new Site.  This is a pretty intricate process, but I’ll try to be succinct:

1. The Site Replication Service has a process called “SKCC” which runs in it to keep track of things like new sites in 5.5 or AD.  The SKCC (Super Knowledge Consistency Checker, or Site SCC, depending on who you ask) sees that a new administrative group is created in the AD, and the msExchAdminGroupMode on that AG is set to “0” which means this is a pure AG.  (1 means Pure 5.5 and 2 means Mixed).  Every Pure AG must have a “Responsible SRS” which will own that site so that it will appear in 5.5.
2. SKCC sees that no SRS owns this new AG.  How, you ask?  By checking the msExchServer1ExportContainers attribute on each Config_CA to see if the Distinguished Name (DN) of that AG appears as an export container.
3. Now that SKCC knows that there is a new pure AG with no owning SRS, it goes through a process called Arbitration to determine which SRS should take ownership.  If the server is at least E2K SP2, the first step is to check to see if PreferredSRS is set on the AG.  See this KB for more info on PreferredSRS:  http://support.microsoft.com/?id=315408.  If PreferredSRS is not set, arbitration uses an algorithm to figure out which SRS should take it.  Basically, it takes a hash of the site name to get a magic number.  Then it takes a hash of each Config_CA, and whichever Config_CA’s hash value is closest to the magic number wins.  The reason this works is each SRS’s SKCC uses the exact same algorithm, so only the SRS that wins arbitration actually does any work to take ownership of a site
4. If the local SRS wins arbitration, it does a few things to take ownership of the site.  It adds that AG’s DN to its msExchServer1ExportContainers on the Config CA.  It also adds that site as an inbound site on the local ADNAutoDRC directory replication connector.  Note that the ADNAutoDRC is a disabled DRC, it doesn’t actually replicate anything – but it does have a special purpose which we will get to
5. Once the Config CA replicates to the SRS, it will see the new export container, and it will create some new naming contexts in the local SRS database for the pure AG.  It creates these as writeable replicas so that it can actually write changes for that site directly in the local database – much different than usual dirrep of remote sites, where the local replica is NOT marked as writeable.  So, now, the Config CA can write stuff like the Pure AGs servers and connectors and other config information directly into the SRS database
6. But, there is still a problem as 5.5 does not understand writeable replicas for remote sites – the only way 5.5 can learn about a remote site is by sites listed as Inbound on a Directory Replication Connector.  This is where the ADNAutoDRC comes in – the change made to add the Pure AG to the ADNAutoDRC is replicated to the 5.5 servers in the site, and when KCC runs on those servers, they realize there is a new site, and that the SRS server is the dirrep bridgehead.  So they add a local non-writeable replica for that site and then send directory update requests to the SRS to get updates about the new site.  The SRS can then replicate information about this new Pure AG to the 5.5 servers in the site.
7. Then, by normal 5.5 directory replication, this new AG can replicate to any connected sites in 5.5, and everyone will eventually gain knowledge of the new AG

So, this is fine and dandy, but at this point we have a site with no users in it in 5.5.  The arbitration process and Config CA replication only handle configuration objects – not objects in the Site naming context such as users, groups, or contacts created in the new Pure AG.  In order to get those down into 5.5, we have to get them replicated via the ADC into our only writeable copy of that Pure Site in the 5.5 world – the SRS database that owns that Pure AG.

At this point, you may be tempted to create a one-way CA to replicate the users with mailboxes in the Pure AG down to the SRS, which will then replicate the users all around 5.5 and be done with it.  While that may seem to work just fine, there is actually one big problem with using one-way CAs to replicate the Pure AG mailboxes into 5.5.  The problem is group membership updates.  You will find that if a Pure AG mailbox is added to a DL in 5.5, and that DL is in a different site than the Pure AG mailbox, the user won’t be added to the group in the AD.  And conversely, if you add a pure E2K user to a group in the AD, the user may not show up on the group once it replicates to 5.5.

The reason for this is as follows:  When the ADC replicates a group from 5.5 to AD (or vise-versa but we’ll talk about this direction), it must resolve each Member of the DL (which is a 5.5 style DN) to a user account in the Active Directory.  The way that it does this is by doing an LDAP query in the AD.  The query that it does is based on the msExchADCGlobalNames attribute -- this is the attribute used by the ADC to match objects together.  So, for instance, if a user on a DL in 5.5 has a DN of “/o=org/ou=site/cn=Recipients/cn=user” then the ADC will do a query against the AD of (msExchADCGlobalNames=EX5: /o=org/ou=site/cn=Recipients/cn=user*).  If this query doesn’t return a user, then the ADC assumes there must not be a corresponding user in the AD to match the mailbox, and it sticks the user’s DN into an attribute called unmergedAtts in the AD.  Note that this is a binary encoded attribute which contains the Hexadecimal representation of the Unicode characters of the DN.  So, it’s not too easy to see what is in that attribute.  But, it does this so that if a user later shows up in the AD which has the matching msExchADCGlobalNames value, the ADC can later clean up those unmerged attributes and put the member on the DL in the AD.

You may still be wondering why all of this matters for Pure AGs and DL membership.  So, here’s the kicker:  If you use a one-way CA from AD to the SRS to create the Pure AG mailboxes, the users in the AD will never get msExchADCGlobalNames stamped on them.  This is because msExchADCGlobalNames is only stamped on an object when that object is a TARGET of ADC replication.  If you are only replicating one-way from AD to the SRS, that means the objects will never replicate from the SRS back to the AD, and the AD user will never be the target of ADC replication, and it will never get msExchADCGlobalNames stamped on it.  Add to this that 5.5 inter-site replication does not replicate ADC Global Names between sites.  So, when the group in Site A replicates down from AD to 5.5 with a Pure AG user as a member, the 5.5 directory in Site A will not have msExchADCGlobalNames in it for the Pure AG user since that user is from a remote site.  So, in order to resolve the DL membership, the ADC must search in the AD for a user with msExchADCGlobalNames in order to find the 5.5 DN of that user, so they can be added to the DL.  In this case the search is like (msExchADCGlobalNames=NT5:ABC123…*) where the “ABC123…” is the objectGUID of the AD user.  If this query fails, the user is not added to the group in Exchange 5.5 and instead unmergedAtts is set on the 5.5 DL.

And what is the fix?  The fix is to change your one-way CA to a two-way CA, and to add the Pure AG an export container of the From Exchange tab of the CA, then force a full replication of the CA.  Now, since the Pure AG is an export container, the ADC will replicate all those mailboxes from the SRS back to the AD and stamp msExchADCGlobalNames on the AD objects.  Then, when unmerged Attribute cleanup happens on the other CAs (every 12 hours or if you restart the ADC service) it will start taking users out of unmergedAtts and putting them in the membership of the group.

Whew!  So, if you are having a problem with group membership replication, check the group with missing users to see if unmergedAtts is set on it.  If so, check the users that are missing from the group to make sure msExchADCGlobalNames is set on them in the AD and in 5.5 (using raw admin, the attribute is called ADC-Global-Names).  If it is not set, then check out your CAs to make sure they are all two-way and configured properly, and try making a change to the object in the remote directory.  Once you get msExchADCGlobalNames populated, the user will appear on the DL after some time.

- Graham McIntyre