ADAMSync 101

Ryan Ries · ‎Apr 04 2019

First published on TechNet on Nov 12, 2012

Hi Everyone, Kim Nichols here again, and this time I have an introduction to ADAMSync. I take a lot of cases on ADAM and AD LDS and have seen a number of problems arise from less than optimally configured ADAMSync XML files. There are many sources of information on ADAM/AD LDS and ADAMSync (I'll include links at the end), but I still receive lots of questions and cases on configuring ADAM/AD LDS for ADAMSync.

We'll start at the beginning and talk about what ADAM/AD LDS is, what ADAMSync is and then finally how you can get AD LDS and ADAMSync working in your environment.

What is ADAM/AD LDS?

ADAM (Active Directory Application Mode) is the 2003 name for AD LDS (Active Directory Lightweight Directory Services). AD LDS is, as the name describes, a lightweight version of Active Directory. It gives you the capabilities of a multi-master LDAP directory that supports replication without some of the extraneous features of an Active Directory domain controller (domains and forests, Kerberos, trusts, etc.). AD LDS is used in situations where you need an LDAP directory but don't want the administration overhead of AD. Usually it's used with web applications or SQL databases for authentication. Its schema can also be fully customized without impacting the AD schema.

AD LDS uses the concept of instances, similar to that of instances in SQL. What this means is one AD LDS server can run multiple AD LDS instances (databases). This is another differentiator from Active Directory: a domain controller can only be a domain controller for one domain. In AD LDS, each instance runs on a different set of ports. The default instance of AD LDS listens on 389 (similar to AD).

Here's some more information on AD LDS if you're new to it:

AD LDS Installed Help

Active Directory Lightweight Directory Services (AD LDS) Getting Started Step-by-Step Guide ...

Active Directory Lightweight Directory Services Overview

AD LDS Backup and Restore Step-by-Step Guide

Active Directory Lightweight Directory Services Operations Guide

What is ADAMSync?

In many scenarios, you may want to store user data in AD LDS that you can't or don't want to store in AD. Your application will point to the AD LDS instance for this data, but you probably don't want to manually create all of these users in AD LDS when they already exist in AD. If you have Forefront Identity Manager (FIM), you can use it to synchronize the users from AD into AD LDS and then manually populate the AD LDS specific attributes through LDP, ADSIEdit, or a custom or 3rd party application. If you don't have FIM, however, you can use ADAMSync to synchronize data from your Active Directory to AD LDS.

It is important to remember that ADAMSync DOES NOT synchronize user passwords! If you want the AD LDS user account to use the same password as the AD user, then userproxy transformation is what you need. (That's a topic for another day, though. I'll include links at the end for userproxy.)

ADAMSync uses an XML file that defines which data will synchronize from AD to AD LDS. The XML file includes the AD partition from which to synchronize, the object types (classes or categories), and attributes to synchronize. This file is loaded into the AD LDS database and used during ADAMSync synchronization. Every time you make changes to the XML file, you must reload the XML file into the database.

In order for ADAMSync to work:

The MS-AdamSyncMetadata.LDF file must be imported into the schema of the AD LDS instance prior to attempting to install the XML file. This LDF creates the classes and attributes for storing the ADAMSync.xml file.

The schema of the AD LDS instance must already contain all of the object classes and attributes that you will be syncing from AD to AD LDS. In other words, you can't sync a user object from AD to AD LDS unless the AD LDS schema contains the User class and all of the attributes that you specify in the ADAMSync XML (we'll talk more about this next). There is a blog post on using ADSchemaAnalyzer to compare the AD schema to the AD LDS schema and export the differences to an LDF file that can be imported into AD LDS.

Unless you plan on modifying the schema of the AD LDS instance, your instance should be named DC=<partition name>, DC=<com or local or whatever> and not CN=<partition name>. Unfortunately, the example in the AD LDS setup wizard uses CN= for the partition name. If you are going to be using ADAMSync, you should disregard that example and use DC= instead. The reason behind this change is that the default schema does not allow an organizationalUnit (OU) object to have a parent object of the Container (CN) class. Since you will be synchronizing OUs from AD to AD LDS and they will need to be child objects of your application partition head, you will run into problems if your application partition is named CN=.

Obviously, this limitation is something you can change in the AD LDS schema, but simply naming your partition with DC= name component will eliminate the need to make such a change. In addition, you won't have to remember that you made a change to the schema in the future.

The best advice I can give regarding ADAMSync is to keep it as simple as possible to start off with. The goal should be to get a basic XML file that you know will work, gradually add attributes to it, and troubleshoot issues one at a time. If you try to do too much (too wide of object filter or too many attributes) in the XML from the beginning, you will likely run into multiple issues and not know where to begin in troubleshooting.

KEEP IT SIMPLE!!!

MS-AdamSyncConf.xml

Let's take a look at the default XML file that Microsoft provides and go through some recommendations to make it more efficient and less prone to issues. The file is named MS-AdamSyncConf.XML and is typically located in the %windir%\ADAM directory.

<?xml version="1.0"?>
<doc>
<configuration>
<description>sample Adamsync configuration file</description>
<security-mode>object</security-mode>
<source-ad-name>fabrikam.com</source-ad-name> <------ 1
<source-ad-partition>dc=fabrikam,dc=com</source-ad-partition> <------ 2
<source-ad-account></source-ad-account> <------ 3
<account-domain></account-domain> <------ 4
<target-dn>dc=fabrikam,dc=com</target-dn> <------ 5
<query>
<base-dn>dc=fabrikam,dc=com</base-dn> <------ 6
<object-filter>(objectClass=*)</object-filter> <------ 7
<attributes> <------ 8
<include></include>
<exclude>extensionName</exclude>
<exclude>displayNamePrintable</exclude>
<exclude>flags</exclude
<exclude>isPrivelegeHolder</exclude>
<exclude>msCom-UserLink</exclude>
<exclude>msCom-PartitionSetLink</exclude>
<exclude>reports</exclude>
<exclude>serviceprincipalname</exclude>
<exclude>accountExpires</exclude>
<exclude>adminCount</exclude>
<exclude>primarygroupid</exclude>
<exclude>userAccountControl</exclude>
<exclude>codePage</exclude>
<exclude>countryCode</exclude>
<exclude>logonhours</exclude>
<exclude>lockoutTime</exclude>
</attributes>
</query>
<schedule>
<aging>
<frequency>0</frequency>
<num-objects>0</num-objects>
</aging>
<schtasks-cmd></schtasks-cmd>
</schedule> <------ 9
</configuration>
<synchronizer-state>
<dirsync-cookie></dirsync-cookie>
<status></status>
<authoritative-adam-instance></authoritative-adam-instance>
<configuration-file-guid></configuration-file-guid>
<last-sync-attempt-time></last-sync-attempt-time>
<last-sync-success-time></last-sync-success-time>
<last-sync-error-time></last-sync-error-time>
<last-sync-error-string></last-sync-error-string>
<consecutive-sync-failures></consecutive-sync-failures>
<user-credentials></user-credentials>
<runs-since-last-object-update></runs-since-last-object-update>
<runs-since-last-full-sync></runs-since-last-full-sync>
</synchronizer-state>
</doc>

Let's go through the default XML file by number and talk about what each section does, why the defaults are what they are, and what I typically recommend when working with customers.

<source-ad-name>fabrikam.com</source-ad-name>

Replace fabrikam.com with the FQDN of the domain/forest that will be your synchronization source

<source-ad-partition>dc=fabrikam,dc=com</source-ad-partition>

Replace dc=fabrikam,dc=com with the DN of the AD partition that will be the source for the synchronization

<source-ad-account></source-ad-account>

Contains the account that will be used to authenticate to the source forest/domain. If left empty, the credentials of the logged on user will be used

<account-domain></account-domain>

Contains the domain name to use for authentication to the source domain/forest. This element combined with <source-ad-account> make up the domain\username that will be used to authenticate to the source domain/forest. If left empty, the domain of the logged on user will be used.

<target-dn>dc=fabrikam,dc=com</target-dn>

Replace dc=fabrikam,dc=com with the DN of the AD LDS partition you will be synchronizing to.

NOTE: In 2003 ADAM, you were able to specify a sub-ou or container of the of the ADAM partition, for instance OU=accounts,dc=fabrikam,dc=com. This is not possible in 2008+ AD LDS. You must specify the head of the partition, dc=fabrikam,dc=com. This is publicly documented here .

<base-dn>dc=fabrikam,dc=com</base-dn>

Replace dc=fabrikam,dc=com with the base DN of the container in AD that you want to synchronize objects from.

NOTE: You can specify multiple base DNs in the XML file, but it is important to note that due to the way the dirsync engine works the entire directory will still be scanned during synchronization. This can lead to unexpectedly long synchronization times and output in the adamsync.log file that is confusing. The short of this this is that even though you are limiting where to synchronize objects from, it doesn't reduce your synchronization time and you will see entries in the adamsync.log file that indicate objects being processed but not written. This can make it appear as though ADAMSync is not working correctly if your directory is large but you are syncing is a small percentage of the directory. Also, the log will grow and grow, but it may take a long time for objects to begin to appear in AD LDS. This is because the entire directory is being enumerated, but only a portion is being synchronized.

<object-filter>(objectClass=*)</object-filter>

The object filter determines which objects will be synchronized from AD to AD LDS. While objectClass=* will get you everything, do you really want or need EVERYTHING? Consider the amount of data you will be syncing and the security implications of having everything duplicated in AD LDS. If you only care about user objects, then don't sync computers and groups.

The filter that I generally recommend as a starting point is:

(|(objectCategory=Person)(objectCategory=OrganizationalUnit))

Rather than objectClass=User , I recommend objectCategory=Person . But, why, you ask? I'll tell you :) If you've ever looked that the class of a computer object, you'll notice that it contains an objectClass of user.

What this means to ADAMSync is that if I specify an object filter of objectClass=user , ADAMSync will synchronize users and computers (and contact objects and anything else that inherits from the User class). However, if I use objectCategory=Person , I only get actual user objects. Pretty neat, eh?

So, what does this | mean and why include objectCategory=OrganizationalUnit ? The literal | is the XML representation of the | (pipe) character which represents a logical OR. True, I've seen customers just use the | character in the XML file and not have issues, but I always use the XML rather than the | just to be certain that it gets translated properly when loaded into the AD LDS instance. If you need to use an AND rather than an OR, the XML for & is & .

You need objectCategory=OrganizationalUnit so that objects that are moved within AD get synchronized properly to AD LDS. If you don't specify this, the OUs that contain objects within scope of the object filter will be created on the initial creation of the object in AD LDS. But, if that object is ever MOVED in the source AD, ADAMSync won't be able to synchronize that object to the new location. Moving an object changes the full DN of the object. Since we aren't syncing the OUs the object just "disappears" from an ADAMSync perspective and never gets updated/moved.

If you need groups to be synchronized as well you can add (objectclass=group) inside the outer parentheses and groups will also be synced.

(|(objectCategory=Person)(objectCategory=OrganizationalUnit)(objectClass=Group))

<attributes>

The attributes section is where you define which attributes to synchronize for the object types defined in the <object-filter>.

You can either use the <include></include> or <exclude></exclude> tabs, but you cannot use both.

The default XML file provided by Microsoft takes the high ground and uses the <exclude></exclude> tags which really means include all attributes except the ones that are explicitly defined within the <exclude></exclude> element. While this approach guarantees that you don't miss anything important, it can also lead to a lot of headaches in troubleshooting.

If you've ever looked at an AD user account in ADSIEdit (especially in an environment with Exchange), you'll notice there are hundreds of attributes defined. Keeping to my earlier advice of "keep it simple", every attribute you sync adds to the complexity.

When you use the <exclude></exclude> tags you don't know what you are syncing; you only know what you are not syncing. If your application isn't going to use the attribute then there is no reason to copy that data to AD LDS. Additionally, there are some attributes and classes that just won't sync due to how the dirsync engine works. I'll include the list as I know it at the end of the article. Every environment is different in terms of which schema updates have been made and which attributes are being used. Also, as I mentioned earlier, if your AD LDS schema does not contain the object classes and attributes that you have defined in your ADAMSync XML file you're your synchronization will die in a big blazing ball of flame.

Whoosh!!

A typical attributes section to start out with is something like this:

<include>objectSID</include> <----- only needed for userproxy
<include>userPrincipalName</include> <----- must be unique in AD LDS instance
<include>displayName</include>
<include>givenName</include>
<include>sn</include>
<include>physicalDeliveryOfficeName</include>
<include>telephoneNumber</include>
<include>mail</include>
<include>title</include>
<include>department</include>
<include>manager</include>
<include>mobile</include>
<include>ipPhone</include>
<exclude></exclude>

Initially, you may even want to remove userPrincipalName, just to verify that you can get a sync to complete successfully. Synchronization issues caused by the userPrincipalName attribute are among the most common ADAMSync issues I see. Active Directory allows multiple accounts to have the same userPrincipalName, but ADAMSync will not sync an object if it has the same userPrincipalName of an object that already exists in the AD LDS database.

If you want to be a superhero and find duplicate UPNs in your AD before you attempt ADAMSync, here's a nifty csvde command that will generate a comma-delimited file that you can run through Excel's "Highlight duplicates" formatting options (or a script if you are a SUPER-SUPERHERO) to find the duplicates.

csvde -f upn.csv -s localhost:389 -p subtree -d "DC=fabrikam,DC=com" -r "(objectClass=user)" -l sAMAccountName,userPrincipalName

Remember, you are targeting your AD with this command, so the localhost:389 implies that the command is being run on the DC. You'll need to replace "DC=fabrikam, DC=com" with your AD domain's DN.

</schedule>

After </schedule> is where you would insert the elements to do user proxy transformation. In the References section, I've included links that explain the purpose and configuration of userproxy. The short version is that you can use this section of code to create userproxy objects rather than AD LDS user class objects. Userproxy objects are a special class of user that links back to an Active Directory domain account to allow the AD LDS user to utilize the password of their corresponding user account in AD. It is NOT a way to logon on to AD from an external network. It is a way to allow an application that utilizes AD LDS as its LDAP directory to authenticate a user via the same password they have in AD. Communication between AD and AD LDS is required for this to work and the application that is requesting the authentication does not receive a Kerberos ticket for the user.

Here is an example of what you would put after </schedule> and before </configuration>

<user-proxy>
<source-object-class>user</source-object-class>
<target-object-class>userProxyFull</target-object-class>
</user-proxy>

Installing the XML file

OK! That was fun, wasn't it? Now that we have an XML file, how do we use it? This is covered in a lot of different materials, but the short version is we have to install it into the AD LDS instance. To install the file, run the following command from the ADAM installation directory (%windir%\ADAM):

Adamsync /install localhost:389 CustomAdamsync.xml

The command above assumes you are running it on the AD LDS server, that the instance is running on port 389 and that the XML file is located in the path of the adamsync command.

What does this do exactly, you ask? The adamsync install command copies the XML file contents into the configurationFile attribute on the AD LDS application partition head. You can view the attribute by connecting to the application partition via LDP or through ADSIEdit. This is a handy thing to know. You can use this to verify for certain exactly what is configured in the instance. Often there are several versions of the XML file in the ADAM directory and it can be difficult to know which one is being used. Checking the configurationFile attribute will tell you exactly what is configured. It won't tell you which XML file was used, but at least you will know the configuration.

The implication of this is that anytime you update the XML file you must reinstall it using the adamsync /install command otherwise the version in the instance is not updated. I've made this mistake a number of times during troubleshooting!

Synchronizing with AD

Finally, we are ready to synchronize! Running the synchronization is the "easy" part assuming we've created a valid XML file, our AD LDS schema has all the necessary classes and attributes, and the source AD data is without issue (duplicate UPN is an example of a known issue).

From the ADAM directory (typically %windir%\ADAM), run the following command:

Adamsync /sync localhost:389 "DC=fabrikam,DC=com" /log adamsync.log

Again, we're assuming you are running the command on the AD LDS server and that the instance is running on port 389. The DN referenced in the command is the DN of your AD LDS application partition. /log is very important (you can name the log anything you want). You will need this log if there are any issues during the synchronization. The log will tell you which object failed and give you a cryptic "detailed" reason as to why. Below is an example of an error due to a duplicate UPN. This is one of the easier ones to understand.

====================================================
Processing Entry: Page 67, Frame 1, Entry 64, Count 1, USN 0
Processing source entry <guid=fe36238b9dd27a45b96304ea820c82d8>
Processing in-scope entry fe36238b9dd27a45b96304ea820c82d8.

Adding target object CN=BillyJoeBob,OU=User Accounts,dc=fabrikam,dc=com. Adding attributes: sourceobjectguid, objectClass, sn, description, givenName, instanceType, displayName, department, sAMAccountName, userPrincipalName, Ldap error occurred. ldap_add_sW: Attribute Or Value Exists. Extended Info: 0000217B: AtrErr: DSID-03050758, #1:
0: 0000217B: DSID-03050758, problem 1006 (ATT_OR_VALUE_EXISTS), data 0, Att 90290 (userPrincipalName)

. Ldap error occurred. ldap_add_sW: Attribute Or Value Exists. Extended Info: 0000217B: AtrErr: DSID-03050758, #1:
0: 0000217B: DSID-03050758, problem 1006 (ATT_OR_VALUE_EXISTS), data 0, Att 90290 (userPrincipalName)
===============================================

During the sync, if you are syncing from the Active Directory domain head rather than an OU or container, your objects should begin showing up in the AD LDS instance almost immediately. The objects don't synchronize in any order that makes sense to the human brain, so don't worry if objects are appearing in a random order. There is no progress bar or indication of how the sync is going other than fact that the log file is growing. When the sync completes you will be returned to the command prompt and your log file will stop growing.

Did it work?

As you can see there is nothing on the command line nor are there any events in any Windows event log that indicate that the synchronization was successful. In this context, successful means completed without errors and all objects in scope, as defined in the XML file, were synchronized. The only way to determine if the synchronization was successful is to check the log file. This highlights the importance of generating the log. Additionally, it's a good idea to keep a reasonable number of past logs so if the sync starts failing at some point you can determine approximately when it started occurring. Management likes to know things like this.

Since you'll probably be automating the synchronization (easy to do with a scheduled task) and not running it manually, it's a good idea to set up a reminder to periodically check the logs for issues. If you've never looked at a log before, it can be a little intimidating if there are a lot of objects being synchronized. The important thing to know is that if the sync was successful, the bottom of the log will contain a section similar to the one below:

Updating the configuration file DirSync cookie with a new value.

Beginning processing of deferred dn references.
Finished processing of deferred dn references.

Finished (successful) synchronization run.
Number of entries processed via dirSync: 16
Number of entries processed via ldap: 0
Processing took 4 seconds (0, 0).
Number of object additions: 3
Number of object modifications: 13
Number of object deletions: 0
Number of object renames: 2
Number of references processed / dropped: 0, 0
Maximum number of attributes seen on a single object: 9
Maximum number of values retrieved via range syntax: 0

Beginning aging run.
Aging requested every 0 runs. We last aged 2 runs ago.
Saving Configuration File on DC=instance1,DC=local
Saved configuration file.

If your log just stops without a section similar to the one above, then the last entry will indicate an error similar to the one above for the duplicate UPN.

Conclusion and other References

That covers the basics of setting up ADAMSync! I hope this information makes the process more straight forward and gives you some tips for getting it to work the first time! The most important point I can make is to start very simple with the XML file and get something to work. You can always add more attributes to the file later, but if you start from broken it can be difficult to troubleshoot. Also, I highly recommend using <include> over <exclude> when specifying attributes to synchronize. This may be more work for your application team since they will have to know what their application requires, but it will make setting up the XML file and getting a successful synchronization much easier!

ADAMSync excluded objects

As I mentioned earlier, there are some attributes, classes and object types that ADAMSync will not synchronize. The items listed below are hard-coded not to sync. There is no way around this using ADAMSync. If you need any of these items to sync, then you will need to use LDIFDE exports, FIM, or some other method to synchronize them from AD to AD LDS. The scenarios where you would require any of these items are very limited and some of them are dealt with within ADAMSync by converting the attribute to a new attribute name (objectGUID to sourceObjectGUID).

Attributes

cn, currentValue, dBCSPwd, fSMORoleOwner, initialAuthIncoming, initialAuthOutgoing, isCriticalSystemObject, isDeleted, lastLogonTimeStamp, lmPwdHistory, msDS-ExecuteScriptPassword, ntPwdHistory, nTSecurityDescriptor, objectCategory, objectSid (except when being converted to proxy), parentGUID, priorValue, pwdLastSet, sAMAccountType, sIDHistory, supplementalCredentials, supplementalCredentials, systemFlags, trustAuthIncoming, trustAuthOutgoing, unicodePwd, whenChanged

Classes

crossRef, secret, trustedDomain, foreignSecurityPrincipal, rIDSet, rIDManager

Other

Naming Context heads, deleted objects, empty attributes, attributes we do not have permissions to read, objectGUIDs (gets transferred to sourceObjectGUID), objects with del-mangeled distinguished names (DEL:\)