First published on TechNet on Feb 26, 2018
Hello Everyone, my name is Zoheb Shaikh and I'm a Premier Field Engineer out of Malaysia. Today for my first post on AskPFEPlat, I wanted to share something interesting with you that I came across recently caused by a KRBTGT_RODC account deletion. Before I talk more about the issue, I would like to share a bit of background about KRBTGT account and its use briefly. I could try to explain what the krbtgt account is, but here is a short article on the KDC and the krbtgt to take a look at: Key Distribution Center - Win32 apps | Microsoft Learn
"All instances of the KDC within a domain use the domain account for the security principal "krbtgt". Clients address messages to a domain's KDC by including both the service's principal name, "krbtgt", and the name of the domain. Both items of information are also used in tickets to identify the issuing authority. For information about name forms and addressing conventions, see RFC 4120." Likewise, a snip for the RODC krbtgt_##### account: http://technet.microsoft.com/en-us/library/cc753223(v=WS.10).aspx "The RODC is advertised as the Key Distribution Center (KDC) for the branch office. The RODC uses a different krbtgt account and password than the KDC on a writable domain controller uses when it signs or encrypts ticket-granting ticket (TGT) requests. This provides cryptographic isolation between KDCs in different branches, which prevents a compromised RODC from issuing service tickets to resources in other branches or a hub site."
The krbtgt_##### account is unique to each RODC and minimizes impact if the RODC is compromised. The RODC does not have the krbtgt secret. It only has its own krbtgt_##### secret (and other accounts you have allowed). Thus, when removing a compromised RODC, the domain krbtgt account is not lost. Getting back to the scenario, the customer had multiple DC's running 2012 R2 and 3 Read Only Domain Controllers (RODC). We observed that the writable DC's were flooded with the Event IDs 1168 stating "Internal error: An Active Directory Domain Services error has occurred". They were not experiencing any functional loss because of this, but were worried about the h`ealth of the Domain Controllers. Log Name: Directory Service Source: Microsoft-Windows-ActiveDirectory_DomainService Date: 6/2/2017 3:18:01 AM Event ID: 1168 Task Category: Internal Processing Level: Error Keywords: Classic User: Contoso\contosoRODC$ Computer: ContosoDC.contoso.local Description: Internal error: An Active Directory Domain Services error has occurred. Additional Data Error value (decimal): 8995 Error value (hex): 2323 Internal ID: 124013b So we asked, what changes have been made recently? In this case, the customer was unsure about what exactly happened, and these events seem to have started out of nowhere. They reported no major changes done for AD in the past 2 months and suspected that this might be an underlying problem for a long time. So, we investigated the events and when we looked at it granularly we found that the event 1168 was coming from a RODC: Keywords: Classic User: Contoso\contosoRODC$ Computer: ContosoDC.contoso.local Then we checked one of the RODC's and could not see any reference to these. So, we turned up the Active Directory Diagnostics to 5 and saw an event Id Event 1084 (refer blog for enabling Active Directory Diagnostic logging AD and LDS diagnostic event logging - Windows Server | Microsoft Learn). Event ID: 1084 Internal event: Active Directory Domain Services could not update the following object with changes received from the following source directory service. This is because an error occurred during the application of the changes to Active Directory Domain Services on the directory service. Object: CN=krbtgt_37540\0ADEL:1gc5th4-88yy-4194-th65-avf12a8621324,CN=Deleted Objects,DC=contoso,DC=local Object GUID: 0e8478c5-3605-4e8c-8497-1e730c959516 Source directory service: b137e78d-e45f-4e88-aaee-379dd9b7e66f._msdcs.contoso.local From this error, it was clear that this was caused by krbtgt_RODC account deletion and the customer said that they may have run a script to delete Disabled accounts. So, we proposed below options to resolve this issue
- Restore the KRBTGT_RODC account from Active Directory Recycle Bin if it was enabled.
- Restore the KRBTGT_RODC account from a System State backup
- Demote and repromoted RODC as KRBTGT_RODC account is unique for each RODC
To reproduce this error in lab we followed the below steps: -
1. Promoted a RODC in the environment
2. Changed the attribute ms-ds-krbTGT-Link Removed the user (cn=krbtgt_37540,dc=contoso,dc=com) "key distribution center for service account > Put the value to <not Set>
3. Once done checked events and it gave us the same event (1168)
4. Added the ms-ds-krbTGT-Link value back to KRBTGT_RODC account and the event stopped coming.
If you have a RODC in your environment, do keep this in mind. Thanks for reading, and hope this helps! Zoheb