Hey there! Josh Mora here, with a brief post on an issue I recently had and wanted to make public, in hopes this will help those that run into this issue, and in addition, provide some helpful logging information that can be useful for any ADWS issues you might come across.
Scenario:
Let's delve into the recurring issue at hand: Your AD LDS server, running ADWS, is consistently generating Event 1202 in the ADWS events, repeatedly, minute after minute. Here's the information it presents:
Log Name: Active Directory Web Services
Source: ADWS
Date: 5/05/2020 1:30:00 PM
Event ID: 1202
Task Category: ADWS Instance Events
Level: Error
Keywords: Classic
User: N/A
Computer: LDS01.Contoso.com
Description:
This computer is now hosting the specified directory instance, but Active Directory Web Services could not service it. Active Directory Web Services will retry this operation periodically.
Directory instance: ADAM_INSTANCE
Directory instance LDAP port: 389
Directory instance SSL port: 636
While this excessive logging of 1202 events may not be disrupting services and everything may continue working properly, it can become a nuisance. Moreover, it could indicate underlying issues that you aren’t aware of. So, let’s dive straight into how we can identify the cause and resolve it.
ADWS Debug Logging:
In this scenario, we're utilizing the built-in functionality of ADWS Debug Logging. Enabling debug logging involves modifying the “Microsoft.ActiveDirectory.WebServices.exe.config” file. This file allows you to adjust various configuration parameters to enhance ADWS functionality, as outlined in this Microsoft Documentation. However, this documentation does not cover the parameters for enabling Debug Logging, precisely why I've decided to provide detailed information on this matter below.
Checking ADWS Configuration Information:
Special thanks to Jason Bender, who put these two commands together that conveniently provide the configuration information from the ADWS Config file.
- In a PowerShell window, run the following: [xml]$ADWSConfiguration = get-content -path c:\windows\adws\microsoft.activedirectory.webservices.exe.config
- Then, run: $ADWSConfiguration.configuration.appsettings.add
- You should get an output like this:
Enabling ADWS Debug Logging:
- Navigate to ‘C:\Windows\ADWS’. The file we are looking to modify is “Microsoft.ActiveDirectory.WebServices.Exe.Config”.
- Now, before making any changes, I strongly suggest to take a backup of the “Microsoft.ActiveDirectory.WebServices.Exe.Config” file. You can never be too safe!
- Right-click the file “Microsoft.ActiveDirectory.WebServices.Exe.Config”, then Open with, and select Notepad, or any other text editor. Right under <appSettings>, enter the following two lines:
<add key="DebugLevel" value="Info"/>
<add key="DebugLogFile" value="C:\Windows\Debug\ADWSlog.txt"/>
- Save the file, then restart the ADWS service for the changes to take effect.
- To verify the changes, run the PowerShell commands from before. You should now be able to see the DebugLevel and DebugLogFile set.
Information to keep in mind:
- Typos or extra spaces in the Config file can cause the ADWS service to fail to start with the following error: “Windows could not start the Active Directory Web Services service on Local Computer. Error 1053: The service did not respond to the start or control request in a timely fashion.”
- There are other debug levels for the DebugLevel parameter, including “None”, “Warn” and “Error”. However, the most helpful and informative is “Info”.
- The DebugLogFile location can be specified per your needs, it’s not a fixed location for the log file.
- This ADWS Debug Logging can log a lot of information when set to “Info”, so it’s recommended to only have this running while you are reproducing your issue. You should disable the logging afterwards, do this by deleting the lines that were added to the Config file.
Analyzing the ADWS Debug Log file:
To clarify, this is not a guide on overall analysis of the ADWS Debug Log file, but more focused on the issue at hand, the excessive 1202 events.
The first line we will see is the ScavengerThread waking up and beginning to look for Instances:
LdapSessionPoolImplementation: [05/05/2020 1:29:40 PM] [8] ScavengerThread: woke up
LdapSessionPoolImplementation: [05/05/2020 1:29:40 PM] [8] ScavengerThread: processing next pool
ConnectionPool: [05/05/2020 1:29:40 PM] [8] GetReservationEnumerator: entering, instance=NTDS
LdapSessionPoolImplementation: [05/05/2020 1:29:40 PM] [8] ScavengerThread: processing next pool
ConnectionPool: [05/05/2020 1:29:40 PM] [8] GetReservationEnumerator: entering, instance=ADAM_INSTANCE
LdapSessionPoolImplementation: [05/05/2020 1:29:40 PM] [8] Scavenger: waking up at 00:00:40 interval
EnumerationContextCache: [05/05/2020 1:30:00 PM] [b] OnTimedEvent: got an event
EnumerationContextCache: [05/05/2020 1:30:00 PM] [b] RemoveExpiredEntries called
EnumerationContextCache: [05/05/2020 1:30:00 PM] [b] RemoveExpiredEntries -- found 0 entries to remove
EnumerationContextCache: [05/05/2020 1:30:00 PM] [b] RemoveExpiredEntries done
Next, we see ADWS checking registry keys for NTDS, in order to determine if this Instance is servicing or not:
InstanceMap: [05/05/2020 1:31:00 PM] [d] CheckAndLoadNTDSInstance: entered
InstanceMap: [05/05/2020 1:31:00 PM] [d] CheckAndLoadNTDSInstance: found NTDS Parameters key
ADWS has found an NTDS Parameter registry key (which would contain all the NTDS settings), and due to the presence of this key, ADWS believes this is a Domain Controller providing ADDS services.
Next, ADWS needs to determine if we are indeed meeting basic requirements for providing ADDS services, by verifying if the server is providing Global Catalog services:
InstanceMap: [05/05/2020 1:31:00 PM] [d] CheckAndLoadGCInstance: entered
InstanceMap: [05/05/2020 1:31:00 PM] [d] CheckForGlobalCatalog: entered
DirectoryUtilities: [05/05/2020 1:31:00 PM] [d] GetTimeRemaining: remaining time is 00:02:00
InstanceMap: [05/05/2020 1:31:01 PM] [d] CheckForGlobalCatalog: isGlobalCatalogReady:
InstanceMap: [05/05/2020 1:31:01 PM] [d] GlobalCatalog is not ready to service the requests.
InstanceMap: [05/05/2020 1:31:01 PM] [d] CheckAndLoadGCInstance: CheckForGlobalCatalog=False
After this, ADWS moves on to checking if ADAM Instances are ready for servicing as well, however we won't be discussing further, as we’ve found our problem.
Interpretation of the Data:
The data above tells us the following:
- An NTDS Parameters registry key was found by the ADWS service.
- ADWS now believes that this server is providing ADDS services (due to the presence of NTDS Parameters).
- Because ADWS believes this is a Domain Controller, it checked if Global Catalog is ready and/or if the ports are opened and servicing. This check failed, therefor ADWS is unable to start servicing.
Cause:
Simply put, ADWS was fooled into believing the server was a Domain Controller. Since it’s not an actual Domain Controller, the isGlobalCatalogReady/CheckForGlobalCatalog checks failed, naturally.
Failure to pass ADWS checks will trigger the Event 1202 to get logged, being logged every minute, as this is the default interval in which these checks are performed.
Solution:
An AD LDS server is not supposed to have a Parameters key under NTDS, as it’s not a Domain Controller and will not require any of the values specified under that key.
Navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NTDS\, and delete the Parameters key.
A Similar Scenario:
In other scenarios, the same 1202 event may be logged, but the server is not an AD LDS server; rather, it's an actual Domain Controller. In these cases, the common solution is to delay the startup type for the ADWS service.
This is because, in such scenarios, the issue arises due to a 'race condition,' where ADWS begins performing its checks before ADDS services have started. Consequently, it fails the check and logs the event. I've encountered this scenario primarily with Domain Controllers running 2012 R2 and below.
Thank you, and that's all for now!
- Josh Mora