(For Part 1 of the series, please go here)
Question 2 when troubleshooting the RUS - How to determine if a RUS Rebuild was run?
One way to check for a Rebuild is to use the repadmin tool from the Windows 2000 Support Tools. Use ADSI Edit or LDP to get the distinguishedName of the RUS you are troubleshooting. Then, from a command line, run:
repadmin /showmeta "<distinguishedName of RUS>" >rusmeta.txt
The resulting text file will contain a list of properties on the RUS object. Find the line corresponding to the msExchDoFullReplication property:
298589 Default-First-Site-Name\BILONGEXCH1 298589 2004-06-29 17:10:59 2 msExchDoFullReplication
When an admin chooses Rebuild, msExchDoFullReplication is set to TRUE, and the RUS later sets it to FALSE after the Rebuild kicks off. By looking at the timestamp shown in the repadmin output, you can see when this attribute was last changed, and thus when a Rebuild was last run.
Another way to check for a Rebuild is to turn down the diagnostics logging on everything but Address List Sychronization, and leave Address List Synchronization at Medium. Then, watch the application log for the following events. When a Rebuild is initiated, event ID 8329 is logged:
Event Type: Information
Event Source: MSExchangeAL
Event Category: Address List Synchronization
Event ID: 8329
Description:
The Recipient Update Service is starting a rebuild of DC=bilong,DC=test
Then, about every 10% of the way through the entire range of USNs, the RUS will report the progress of the Rebuild in event ID 8332:
Event Type: Information
Event Source: MSExchangeAL
Event Category: Address List Synchronization
Event ID: 8332
Description:
The Recipient Update Service has started to export a block of entries from DC=bilong,DC=test, beginning at USN 1. It will finish processing the directory when it reaches USN 298599
When the Rebuild completes, the RUS will log an 8330:
Event Type: Information
Event Source: MSExchangeAL
Event Category: Address List Synchronization
Event ID: 8330
Description:
The Recipient Update Service has completed the rebuild of DC=bilong,DC=test
Note that running a Rebuild usually is not useful for troubleshooting. The only difference between a Rebuild and an Update Now is that Rebuild causes the RUS to start over from a USN of 1. Update Now starts from the highest USN last recorded by the RUS, which is stored in the msExchServer1HighestUSN property on the RUS object in the Active Directory. So if the RUS is not working as expected against new or modified objects (the objects that would be touched by an Update Now), running a Rebuild will not help. Because of the time it can take for a Rebuild to complete in a large environment, carefully consider how much time it will take to resume normal RUS operation before you choose to do a Rebuild. Once a Rebuild has started, you must wait for the RUS to catch up before you can do any useful troubleshooting against new and modified objects. For more information on how the RUS queries for changes, see KB:328738 .
Question 3 when troubleshooting the RUS - Did the query return results?
If you found an 8011 that shows a search for a range of USNs including your test user, the next question is did the search return any results. Just after the 8011 you should find an 8012:
Event Type: Information
Event Source: MSExchangeAL
Event Category: LDAP Operations
Event ID: 8012
Description:
Search of directory bilongexch1.bilong.test at base 'DC=bilong,DC=test' returned 16 objects.
If you do not find an 8012 event corresponding to the 8011, then Exchange did not see a response to that search. Typically this would indicate a network problem and cause the RUS to hang at that point. After that you usually do not see any more 8011 queries against the root of the domain, because the RUS continues to wait for a response to this search. If you are seeing this behavior repeatedly, it's best to get a netmon trace capturing the behavior so the network problem can be identified.
If the search returned 0 objects, then the Exchange server computer account did not have permissions to see that user object. These permissions come from the Exchange Enterprise Servers group, which is granted permissions at the root of the domain when setup /domainprep is run. If these permissions are changed, or if inheritance on a subcontainer is removed, this can prevent Exchange from seeing the user. Also, the Exchange Enterprise Servers group for that domain should contain the Exchange Domain Servers groups for all the other domains, and one of the Exchange Domain Servers groups should contain the Exchange server responsible for this RUS. If this chain of membership has been broken, that can also keep the Exchange server from seeing the user.
If the search returns more than 20 objects, you will see more than one 8012 event. The RUS uses a page size of 20 for this search, so the results are returned in batches of 20. Expect to see an 8012 for every 20 objects returned.
If the search did return some objects, the events following the 8012 should list the objects that are being queued for processing:
Event Type: Information
Event Source: MSExchangeAL
Event Category: Address List Synchronization
Event ID: 8175
Description:
Processing change to 'CN=e2kuser7,CN=Users,DC=bilong,DC=test'.
Event Type: Information
Event Source: MSExchangeAL
Event Category: Address List Synchronization
Event ID: 8134
Description:
Queuing request to process 'CN=e2kuser7,CN=Users,DC=bilong,DC=test'.
By examining the 8175 and 8134 events following the 8012, you can determine if the user you're interested in was returned in the search. If the user in question was not returned, this would indicate a permissions problem as noted above. When the RUS is done queuing changes to process, you'll see:
Event Type: Information
Event Source: MSExchangeAL
Event Category: Address List Synchronization
Event ID: 8169
Description:
Retrieved all directory changes under: 'DC=bilong,DC=test'.
Stay tuned for more...
- Bill Long
