Scenario
DC is Virtualized in VMware, I got it restored from Veeam backup, meaning it is not in the current state, that caused Active Directory Broken, how could I get it fixed? I forced replication between 2 DCs it failed. Here and there we got several PCs that have the error:
“The trust relationship between this Workstation and the primary Domain failed”
Based on above use case, identified certain errors.
Investigation
So first, a piece of advice that, you should never restore a domain controller in a multi-domain controller environment. Instead, you should stand up a new DC and start replication, it will take time but will replicate from a fully healthy DC.
Then we ran the below command-lets and collected the logs for review.
Dcdiag /v /c /d /e /s:%computername% >C:\dcdiag.log
repadmin /showrepl >C:\repl.txt
ipconfig /all > C:\dc1.txt
ipconfig /all > C:\dc2.txt
ipconfig /all > C:\problemworkstation.txt
Errors Observed in DC Diagnostic Report & Replication Summary
We found following two errors in DC diagnostic report and Replication summary :
- Active Directory Replication Error 8606: Insufficient attributes were given to create an object.
- Active Directory Replication Error 1988: The local domain controller has attempted to replicate the following object from the following source domain controller. This object is not present on the local domain controller because it may have been deleted and already garbage collected.
Logging Conditions for Error 8606
Upon further research, we found out that Error 8606 is logged when the following conditions are true:
- A source domain controller sends an update to an object (instead of an originating object create) that has already been created, deleted, and then reclaimed by garbage collection from a destination domain controller's copy of Active Directory.
- The destination domain controller was configured to run in strict replication consistency.
Cause of Error 8606
The error is caused by one of the following:
- A permanently lingering object whose removal will require admin intervention.
- A transient lingering object that will correct itself when the source domain controller performs its next garbage-collection cleanup. Introduction of the first domain controller in an existing forest and updates to the partial attribute set are known causes of this condition.
- An object that was undeleted or restored at the cusp of tombstone lifetime expiration.
Key Points to Remember for Troubleshooting Error 8606
When you troubleshoot 8606 errors, think about the following points:
- Although error 8606 is logged on the destination domain controller, the problem object that is blocking replication resides on the source domain controller. Additionally, the source domain controller or a transitive replication partner of the source domain controller potentially did not inbound-replicate knowledge of a deleted tombstone lifetime number of days in the past.
- Remember to search for potentially lingering objects by object GUID versus DN path so that objects can be found regardless of their host partition and parent container. Searching by objectguid will also locate objects that are in the deleted objects container without using the deleted objects LDAP control.
- The NTDS Replication 1988 event identifies only the current object on the source domain controller that is blocking incoming replication by a strict mode destination domain controller. There are likely additional objects "behind" the object that is referenced in the 1988 event that is also lingering.
- The presence of lingering objects on a source domain controller prevents or blocks strict mode destination domain controllers from inbound replicating "good" changes that exist behind the lingering object in the replication queue.
- Because of the way that domain controllers individually delete objects from their deleted object containers (the garbage-collection daemon runs every 12 hours from the last time each domain controller last started), the objects that are causing 8606 errors on destination domain controllers could be subject to removal in the next garbage-collection cleanup execution. Lingering objects in this class are transient and should remove themselves in no more than 12 hours from problem start.
- The lingering object in question is likely one that was intentionally deleted by an administrator or application. Factor this into your resolution plan, and beware of reanimating objects, especially security principals that were intentionally deleted. Resolution
Resolution
For our need, to check the replication status in between only 2 DCs (The affected one and a healthy one), we have also tried disabling “Strict Replication Consistency” that prevents destination domain controllers from replicating in lingering objects, but it is highly recommended not to disable “Strict Replication Consistency”, there can be a risk that lingering objects could be replicated to a domain controller or many where this setting is not enabled.
Reference Microsoft Documentation for enabling this setting:
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc...
As an actual fix, we must have to remove the lingering objects from the recovered DC for the smooth replication. While many methods exist to remove lingering objects, there are two primary tools commonly used: Lingering Object Liquidator (LoL) and repadmin.exe.
- Lingering Object Liquidator (LoL)
The easiest method to clean up Lingering Objects is to use the LoL. The LoL tool was developed to help automate the cleanup process against an Active Directory Forest. The tool is GUI-based and can scan the current Active Directory Forest and detect and cleanup lingering objects.
The tool is available on Microsoft Download Center.
The following command in REPADMIN.EXE can remove lingering objects from directory partitions:
Repadmin.Exe /RemoveLingeringObjects
Repadmin / RemoveLingeringObjects can be used to remove lingering objects from writable and read-only directory partitions on source domain controllers. The syntax is as follows:
c:\>repadmin /removelingeringobjects <Dest_DSA_LIST> <Source DSA GUID> <NC> [/ADVISORY_MODE]
Where:
- <Dest_DSA_LIST> is the name of a domain controller that contains lingering objects (such as the source domain controller that is cited in the NTDS Replication 1988 event).
- <Source DSA GUID> is the name of a domain controller that hosts a writable copy of the directory partition that contains lingering objects to which the domain controller in <Dest_DSA_LIST> has network connectivity. The DC to be cleaned up (first DC specified in the command) must be able to connect directly to port 389 on the DC that hosts a writable copy of the directory partition (specified second in the command).
- <NC> is the DN path of the directory partition that is suspected of containing lingering objects, such as the partition that is specified in a 1988 event.
Monitoring Active Directory Replication Health Daily
If error 8606 / Event 1988 was caused by the domain controller's failing to replicate Active Directory changes in the last tombstone lifetime number of days, make sure that Active Directory replication health is being monitored on a day-to-day basis going forward. Replication health may be monitored by using a dedicated monitoring application or by viewing the output from the one inexpensive but effective option to run "repadmin /showrepl * /csv" command in a spreadsheet application such as Microsoft Excel.
Thus, keeping tabs on Active Directory Health overall is significant. In order to do that, its important for an IT Professional to have an understanding of How to define Active Directory Health?
