Hi, Ajay.
A few random thoughts come to mind.
Firstly, if you want an authoritative answer on this, you're going to need to run a network capture on the AAD Connect host you're seeing the error on. But I'll put my first impressions here as well.
Domain controller selection
Next, AAD Connect and MIM have different starting points for the management agent configuration:
- AAD Connect limits you to completing everything it thinks it needs in the setup wizard;
- The Active Directory connector in MIM gives you complete control as-you-go.
Where this may be important is that the AAD Connect wizard does not let you specify a specific domain controller to connect to where the MIM management agents does. This wouldn't matter if AAD Connect could talk to any remote domain controller, however, given you're running a DMZ configuration, it very much matters. More on how this relates to your DNS request routing below.
As a side note, you do get full control over the management agent within AAD Connect (noting the "preferred domain controllers" option in the picture below) but only after you've finished the setup, leading to a chicken-and-egg scenario:
Figure 1: Preferred domain controller option - same for MIM and AAD Connect.
There's a number of ways you can detect a domain controller programmatically, including letting the underlying Windows system do it for you. I haven't had the need to crack open the MIM management agent and AAD Connect setup wizard to compare them, so I'll simply point out that you could easily get different results for both even if they were to be running on the exact same host.
I'd recommend running the more granular commandlets - almost all of them are useful - listed here as I can't help but think that while your DNS is indeed being directed to the remote DMZ domain controllers, the AAD Connect wizard is receiving an unordered list of domain controllers as the DNS response meaning it's then trying to establish subsequent connections to any random domain controller in that remote forest:
While you've run Confirm-DnsConnectivity, that's not nearly enough. A better indicator using this PowerShell module would be from Confirm-ForestExists and Confirm-TargetsAreReachable.
Figure 2: Using the AAD Connect diagnostic commandlets to see which domain controllers are returned and if they're reachable.
In contrast, perhaps a preferred domain controller has been entered into the MIM management agent (as above) though that isn't the only option, which brings us to your point on DNS.
DNS request routing != domain controller selection
As noted above, having DNS queries go to a specific domain controller doesn't guarantee that the same domain controller will be returned first in the DNS response.
Other factors such as your Active Directory topology (i.e. the defined sites and subnets) and whatever entries reside in your hosts file (in the case of your MIM host) will play a significant role in selection.
My educated guess is that the wizard does rely on the underlying Windows system to perform the check though, as the relevant .NET class throws that same error word-for-word:
Figure 3: Reproducing the error outside of the AAD Connect wizard.
Chances are that because AAD Connect is using a conditional forwarder and likely has no subnet assignment to the DMZ site (if one even exists) within each remote forest, it's sending a DNS query to the DMZ domain controller and coming back with a different domain controller to talk to. Again, this would show up very quickly in a network capture but you may also have some luck with the diagnostic commandlets referenced above.
Of course, it pays to remember that the error message covers two separate causes:
- Can't find the forest, which implies a name resolution issue;
- Can't contact the forest, which implies a firewall issue, which itself may be because the "wrong" domain controller was returned in the DNS response.
If it's the first cause, the above .NET-based check will fail very quickly. If it's the second cause, it will take longer (~ 9 seconds in my testing). That's a very unscientific test but a good indicator and quick to run.
Here's the command in text form for ease of reference:
[System.DirectoryServices.ActiveDirectory.Forest]::GetForest([System.DirectoryServices.ActiveDirectory.DirectoryContext]::new([System.DirectoryServices.ActiveDirectory.DirectoryContextType]::Forest, "forestFqdn.somedomain.com"))
As I mentioned in the first section, I anticipate you're running into the second issue of not being able to reach the returned domain controller.
ICMP
This section is speculation on my part, as - again - I haven't done a network capture of my own to confirm any of this.
You've already listed the ports from the AAD Connect troubleshooting guide:
However, from the diagnostic commandlets documentation, we can see there's an ICMP test in the Confirm-TargetsAreReachable commandlet, implying that the wizard may also check for the ICMP protocol where the MIM Active Directory agent does not.
It might be worth allowing ICMP through from the AAD Connect hosts to the remote DMZs.
Other protocols
Other protocols I've seen used by other facets of the AAD Connect tool are:
- RPC: TCP 135
- WinRM: TCP 5985 (or TCP 5986 for WinRM/TLS)
I'd also strongly recommend enabling the following for better-secured AAD Connect-to-domain controller communications, although there is a bit of post configuration work to be done to enable this:
- LDAP/TLS: TCP 636, TCP 3269
Reference:
These won't assist with resolving your error, but I figured since I've written this much, I'd include it anyway.
Summary
The AAD Connect wizard and the MIM (and AAD Connect) Active Directory management agent are different beasts.
What has worked for MIM in the past may not be enough for the AAD Connect wizard, and a network capture will provide the most authoritative answer to your question.
It's common for hosts files to provide a very small subset of responses that a conditional forwarder will provide. Be sure to compare the differences and not simply assume they provide the same results to the DNS client.
Cheers,
Lain
@LainRobertson Thanks so much for your time on this post and the detailed analysis. I further tested as you suggested and have very strange results. Seems like ICMP is blocked for the very DMZ DC which is supposed to have more connectivity with the ADC forest as compared to the local network of the country forest. We actually have onboarded all countries on ADC successfully except last 3.
Sorry need to hide the IPs as they are of our production environment, is ICMP mandatorily required for ADC?
The below command took well more than 9secs, around 30 actually to fail and give the exact error. However, my observation till now was that the GetForest succeeds and GetDomain fails.
[System.DirectoryServices.ActiveDirectory.Forest]::GetForest([System.DirectoryServices.ActiveDirectory.DirectoryContext]::new([System.DirectoryServices.ActiveDirectory.DirectoryContextType]::Forest, "forestFqdn.somedomain.com"))
Also worth mentioning is that since that local network of countries is not directly pingable from ADC forest and we rely on the DMZ DC for that. We ensure that all the DCs IP are listed in the output for NSLookup and they further resolve to the machine FQDN by setting the Server to the DMZ DC IP.
We also had some countries where there are syncing users in both Parent & Child domains, so we ensure both atleast 1 child & root DCs are included in the DMZ. The local network again will not have direct connectivity.
Last but not the least, all ports mentioned are open except 5986, need to check if WinRM/TLS is required and will get it opened if yes. Here is the exact error for your ready reference:-
PS C:\Program Files\Microsoft Azure Active Directory Connect\Tools> Start-ConnectivityValidation -Forest xxxxx -AutoCreateConnectorAccount:$false -UserName "xxxxx\yyyyyy"
Please provide the credentials of the account you entered on AADConnect Wizard.
DOMAIN\Username: xxxxx\yyyyyy (previously obtained)
Diagnosis is starting...
Attempting to obtain a domainFQDN
Attempting to retrieve DomainFQDN object...
Exception calling "GetDomain" with "1" argument(s): "The specified domain does not exist or cannot be contacted."
There has been a problem while validating connectivity between AADConnect and the Active Directory.
An attempt to diagnose the problem will be performed by running a set of network connectivity tests
Press ENTER to continue:
Starting NetworkConnectivityDiagnosisTools
Verifying that 'xxxxxxx' exists
xxxxxxxxx exists
Verifying if the provided credentials are correct
Attempting to obtain a domainFQDN
Attempting to retrieve DomainFQDN object...
There was an error during the validation of the credentials you have entered. Details: Exception calling "GetDomain" with "1" argument(s): "The specified domain does not exist or cannot be contacted."
This is what you want to focus on (at least, if you want to achieve the best outcome), and what I have come back to in the previous two posts now:
The following is not sound advice and the final paragraph is factually incorrect:
The error from the AAD Connect installation wizard doesn't strictly relate to speaking directly with a FSMO role holder, either.
As I outlined earlier, this error can be resolved by doing the following in the remote forest:
- Create a new site for the DMZ;
- Create a new subnet matching the IP range of the DMZ and attach it to the site;
- Create a new subnet matching the IP range of your central subnet hosting your AAD Connect hosts and attach that to the DMZ site.
You'll now find that the netlogon.log entries stop and the DMZ domain controller is reliably returned as the appropriate domain controller.
Separately, don't edit domain controller-owned DNS SRV records directly. This can break your environment if you're not completely aware of the effects of doing this. You'll likely also lose any changes you made within 24 hours when the NETLOGON service from the owning host refreshes the record (unless automatic registration has been disabled, which also should not be done unless there's a compelling reason.)
Cheers,
Lain
The short answer is I don't know, as I'm not privy to how the AAD Connect installation process works to that kind of depth.
You may not have a choice except to allow the AAD Connect host to have full internal access to all 40 forest/domains. The references on the AAD Connect requirements pages are vague and don't make it clear if what you're trying to do with a DMZ approach will or will not work.
I just finished some testing against the .NET "System.DirectoryServices.ActiveDirectory.Domain" class - which I expect is used in the installation wizard, in conjunction with some high-level Wireshark tracing and the takeaway is that the .NET class produces numerous domain controller references with no way to limit or order them.
The AAD Connect wizard could be utilising any of the properties - from the multi-valued "DomainControllers" to the single-valued xxxRoleOwner (and the Forest class is more of the same.)
If my assumption is correct then DNS is not going to help you solve this problem at all.
Again, the MIM Active Directory agent does not work this way which is why it can be directed to use a specific domain controller, and because the AAD Connect sync engine is indeed just MIM, it could work under this DMZ model. But if you can't get past the AAD Connect installation wizard for the above reasons then it counts for nothing.
You might be tempted to "hack" around the problem by temporarily granting the AAD Connect host access to all 40 remote locations' domain controllers but, of course, you'll only potentially run into the same issue when you have to re-run the wizard, making it not worth it.
DNS isn't going to solve your requirements on its own. Either give your central AAD connect host access to all remote domain controllers or stick with MIM. Or petition Microsoft via a design change request to alter the AAD Connect wizard to allow for the specification of a specific domain controller - if you're an influential-enough client to take that path.
Cheers,
Lain
@LainRobertson Thanks you so much again for pointing us in the right direction. We have finally been able to resolve this issue. In the end, it was not a DNS or Network issue. Here are the steps that resolved it:-
1) By narrowing down the error in PowerShell by running just the Confirm-ValidDomains or Get-ForestFQDN commands and simultaneously running a live network capture on Netmon, we got a Netlogon error however, the service was always running in the Yellow DC and restarting it as well didn’t change anything.
2) Then finally the DS Engineers from MS team asks to collect the Netlogon & DC Diag logs. It becomes clearly visible that the Yellow DC isn’t advertising itself, hence we are asked to enable the SysvolReady flag is the Registry Editor by setting its value to 1. Upon rebooting, the AD Connect validation tests are successful.