Forum Discussion
AD Connect Start-ConnectivityValidation - GetDomain failing error while running adding directories
Hi, Ajay.
A few random thoughts come to mind.
Firstly, if you want an authoritative answer on this, you're going to need to run a network capture on the AAD Connect host you're seeing the error on. But I'll put my first impressions here as well.
Domain controller selection
Next, AAD Connect and MIM have different starting points for the management agent configuration:
- AAD Connect limits you to completing everything it thinks it needs in the setup wizard;
- The Active Directory connector in MIM gives you complete control as-you-go.
Where this may be important is that the AAD Connect wizard does not let you specify a specific domain controller to connect to where the MIM management agents does. This wouldn't matter if AAD Connect could talk to any remote domain controller, however, given you're running a DMZ configuration, it very much matters. More on how this relates to your DNS request routing below.
As a side note, you do get full control over the management agent within AAD Connect (noting the "preferred domain controllers" option in the picture below) but only after you've finished the setup, leading to a chicken-and-egg scenario:
Figure 1: Preferred domain controller option - same for MIM and AAD Connect.
There's a number of ways you can detect a domain controller programmatically, including letting the underlying Windows system do it for you. I haven't had the need to crack open the MIM management agent and AAD Connect setup wizard to compare them, so I'll simply point out that you could easily get different results for both even if they were to be running on the exact same host.
I'd recommend running the more granular commandlets - almost all of them are useful - listed here as I can't help but think that while your DNS is indeed being directed to the remote DMZ domain controllers, the AAD Connect wizard is receiving an unordered list of domain controllers as the DNS response meaning it's then trying to establish subsequent connections to any random domain controller in that remote forest:
While you've run Confirm-DnsConnectivity, that's not nearly enough. A better indicator using this PowerShell module would be from Confirm-ForestExists and Confirm-TargetsAreReachable.
Figure 2: Using the AAD Connect diagnostic commandlets to see which domain controllers are returned and if they're reachable.
In contrast, perhaps a preferred domain controller has been entered into the MIM management agent (as above) though that isn't the only option, which brings us to your point on DNS.
DNS request routing != domain controller selection
As noted above, having DNS queries go to a specific domain controller doesn't guarantee that the same domain controller will be returned first in the DNS response.
Other factors such as your Active Directory topology (i.e. the defined sites and subnets) and whatever entries reside in your hosts file (in the case of your MIM host) will play a significant role in selection.
My educated guess is that the wizard does rely on the underlying Windows system to perform the check though, as the relevant .NET class throws that same error word-for-word:
Figure 3: Reproducing the error outside of the AAD Connect wizard.
Chances are that because AAD Connect is using a conditional forwarder and likely has no subnet assignment to the DMZ site (if one even exists) within each remote forest, it's sending a DNS query to the DMZ domain controller and coming back with a different domain controller to talk to. Again, this would show up very quickly in a network capture but you may also have some luck with the diagnostic commandlets referenced above.
Of course, it pays to remember that the error message covers two separate causes:
- Can't find the forest, which implies a name resolution issue;
- Can't contact the forest, which implies a firewall issue, which itself may be because the "wrong" domain controller was returned in the DNS response.
If it's the first cause, the above .NET-based check will fail very quickly. If it's the second cause, it will take longer (~ 9 seconds in my testing). That's a very unscientific test but a good indicator and quick to run.
Here's the command in text form for ease of reference:
[System.DirectoryServices.ActiveDirectory.Forest]::GetForest([System.DirectoryServices.ActiveDirectory.DirectoryContext]::new([System.DirectoryServices.ActiveDirectory.DirectoryContextType]::Forest, "forestFqdn.somedomain.com"))
As I mentioned in the first section, I anticipate you're running into the second issue of not being able to reach the returned domain controller.
ICMP
This section is speculation on my part, as - again - I haven't done a network capture of my own to confirm any of this.
You've already listed the ports from the AAD Connect troubleshooting guide:
However, from the diagnostic commandlets documentation, we can see there's an ICMP test in the Confirm-TargetsAreReachable commandlet, implying that the wizard may also check for the ICMP protocol where the MIM Active Directory agent does not.
It might be worth allowing ICMP through from the AAD Connect hosts to the remote DMZs.
Other protocols
Other protocols I've seen used by other facets of the AAD Connect tool are:
- RPC: TCP 135
- WinRM: TCP 5985 (or TCP 5986 for WinRM/TLS)
I'd also strongly recommend enabling the following for better-secured AAD Connect-to-domain controller communications, although there is a bit of post configuration work to be done to enable this:
- LDAP/TLS: TCP 636, TCP 3269
Reference:
These won't assist with resolving your error, but I figured since I've written this much, I'd include it anyway.
Summary
The AAD Connect wizard and the MIM (and AAD Connect) Active Directory management agent are different beasts.
What has worked for MIM in the past may not be enough for the AAD Connect wizard, and a network capture will provide the most authoritative answer to your question.
It's common for hosts files to provide a very small subset of responses that a conditional forwarder will provide. Be sure to compare the differences and not simply assume they provide the same results to the DNS client.
Cheers,
Lain
- Ajay_JoshiMar 19, 2023Brass Contributor
LainRobertson Thanks so much for your time on this post and the detailed analysis. I further tested as you suggested and have very strange results. Seems like ICMP is blocked for the very DMZ DC which is supposed to have more connectivity with the ADC forest as compared to the local network of the country forest. We actually have onboarded all countries on ADC successfully except last 3.
Sorry need to hide the IPs as they are of our production environment, is ICMP mandatorily required for ADC?
The below command took well more than 9secs, around 30 actually to fail and give the exact error. However, my observation till now was that the GetForest succeeds and GetDomain fails.
[System.DirectoryServices.ActiveDirectory.Forest]::GetForest([System.DirectoryServices.ActiveDirectory.DirectoryContext]::new([System.DirectoryServices.ActiveDirectory.DirectoryContextType]::Forest, "forestFqdn.somedomain.com"))
Also worth mentioning is that since that local network of countries is not directly pingable from ADC forest and we rely on the DMZ DC for that. We ensure that all the DCs IP are listed in the output for NSLookup and they further resolve to the machine FQDN by setting the Server to the DMZ DC IP.
We also had some countries where there are syncing users in both Parent & Child domains, so we ensure both atleast 1 child & root DCs are included in the DMZ. The local network again will not have direct connectivity.
Last but not the least, all ports mentioned are open except 5986, need to check if WinRM/TLS is required and will get it opened if yes. Here is the exact error for your ready reference:-
PS C:\Program Files\Microsoft Azure Active Directory Connect\Tools> Start-ConnectivityValidation -Forest xxxxx -AutoCreateConnectorAccount:$false -UserName "xxxxx\yyyyyy"
Please provide the credentials of the account you entered on AADConnect Wizard.
DOMAIN\Username: xxxxx\yyyyyy (previously obtained)
Diagnosis is starting...Attempting to obtain a domainFQDN
Attempting to retrieve DomainFQDN object...
Exception calling "GetDomain" with "1" argument(s): "The specified domain does not exist or cannot be contacted."
There has been a problem while validating connectivity between AADConnect and the Active Directory.
An attempt to diagnose the problem will be performed by running a set of network connectivity testsPress ENTER to continue:
Starting NetworkConnectivityDiagnosisTools
Verifying that 'xxxxxxx' existsxxxxxxxxx exists
Verifying if the provided credentials are correct
Attempting to obtain a domainFQDN
Attempting to retrieve DomainFQDN object...
There was an error during the validation of the credentials you have entered. Details: Exception calling "GetDomain" with "1" argument(s): "The specified domain does not exist or cannot be contacted."- LainRobertsonMar 20, 2023Silver Contributor
Here's another AAD Connect-specific article that outlines in greater detail the port requirements:
It's considerably more useful than the previous troubleshooting article and provides a per component breakdown (including the wizard.)
As I said above, I feel your issue is with domain controller selection but this is still a very useful reference.
Cheers,
Lain
- Ajay_JoshiMar 20, 2023Brass Contributor
There are 4 DCs in the forest and 1 of them is in the DMZ, conditional forwarder in the ADC forward points to it.
Although I'm yet to check the Netlogon logs in the target forest, I noticed something interesting with the network connectivity tests. While the Confirm-TargetsAreReachable & Confirm-DNSConnectivity results are all green & successful, Confirm-NetworkConnectivity provides something to think about.
If I run this and DC1 doesn't have port 53 open, its gives an error and doesn't even check the next DC.
Confirm-NetworkConnectivity -DCs DC1.forest.com,DC2.forest.com,DC3.forest.com
TCP connection to DC1.forest.com on port 53 failed.
But if we mention DC3 first which has the connectivity, then it succeeds for it and then does check the next and then if it fails, stops checking further.
Confirm-NetworkConnectivity -DCs DC3.forest.com,DC2.forest.com,DC1.forest.comTCP connection to DC3.forest.com on port 53 succeeded.
TCP connection to DC3.forest.com on port 88 succeeded.
TCP connection to DC3.forest.com on port 389 succeeded.
TCP connection toDC2.forest.com on port 53 failed.
Could this be the issue? Probably the lookup list has the internal DC as the first entry which doesn't have connectivity. Although an assumption, I'm more suspicious after seeing this result:-PS C:\Program Files\Microsoft Azure Active Directory Connect\Tools> Confirm-FunctionalLevel -Forest forest.com
Verifying that the AD forest functional level is >= Windows2003Forest
Obtaining ForestFQDN
Attempting to retrieve ForestFQDN...
Exception calling "GetForest" with "1" argument(s): "The specified forest does not exist or cannot be contacted."CurrentForestLevel is
The binary operator GreaterThanOrEqual is not defined for the types 'System.DirectoryServices.ActiveDirectory.ForestMode' and
'System.DirectoryServices.ActiveDirectory.ForestMode'.
At C:\Program Files\Microsoft Azure Active Directory Connect\Tools\ADConnectivityTool.psm1:1127 char:8
+ If($CurrentForestLevel -lt $MinAdForestVersion)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], InvalidOperationException
+ FullyQualifiedErrorId : System.InvalidOperationException
The Active Directory forest functional level is correct
TrueIt fails in the beginning probably with the internal DC and then succeeds with the DMZ DC. This output is actually similar to the forest which were successfully onboarded to ADC. Just that the initial error statement is "The user name or password is incorrect." instead of "The specified forest does not exist or cannot be contacted". Rest all output is same including the "The binary operator GreaterThanOrEqual is not defined for the types" error.
- LainRobertsonMar 20, 2023Silver Contributor
The only way I could provide authoritative answers is install a new instance of AAD Connect and track what it's doing with a network capture utility - which is a bit more than I can easily accommodate in my small business environment.
It's important to keep in mind that the requirements of different components can be different, meaning something like the setup wizard could easily be different to the synchronisation service, hence why I'd have to trace the whole process from end-to-end.
So, that means I'm still making assumptions based on the error plus what's documented.
So, jumping to your first screenshot, what I would infer from that is that while Confirm-TargetsAreReachable tests using ICMP, the warning message in yellow indicates that it's simply a convenience test and not required for the operation of the wizard or synchronisation service. Of course, that assumption could also be wrong but since they've gone to the trouble of putting in such a message, that's my starting assumption.
As an aside, the synchronisation service within AAD Connect is just MIM with some customised agents. It doesn't need ICMP - I know that for sure - but I can't be so sure for the setup wizard, which of course is where you're currently stuck.
But for now I'd set the ICMP failure aside.
The more interesting part of that screenshot is that five domain controllers were detected for that forest.
Would I be correct in assuming that all five do not reside within that forest's DMZ? I'm guessing one or maybe two reside in the DMZ with the remaining three or four being on their internal network.
The fact that AAD Connect is seeing five domain controllers means that it is likely trying to contact one of the domain controllers on the internal network rather than the one in the DMZ.
If any of the IP addresses you've blacked out relate to internally-located domain controllers then I'm certain this will be the cause of the AAD Connect wizard's error.
There's multiple ways in which you can deal with this scenario - more than I have the time to comprehensively cover here at the moment.
My initial preference if I were faced with this situation would be to get the remote forest administrators to create an Active Directory site for their DMZ, then subnets matching the DMZ and your AAD Connect subnets which would then be attached to the DMZ site so that for any traffic coming from either subnet range, domain controller referrals (which is separate to the DNS resolution architecture) will prefer the domain controllers in the DMZ.
With respect to being able to obtain the forest object but not the domain, that could be for a couple of reasons. The ones that come to mind immediately are:
- Differing results for the forest DNS query versus the domain DNS query (since there are separate zones for ForestDnsZone and DomainDnsZone which will reflect the architecture and FSMO role allocation; and DNS round robin-ing can play a role in this scenario, too, if it's enabled);
- No sites matching the client IP address (which is that of your AAD Connect host) leading to a random domain controller being provided as the referral - which you can track via the netlogon.log file under C:\Windows\debug\netlogon.log on the remote forest's domain controller in their DMZ.
Ultimately though, however you choose to do it, you need to ensure that the centralised hosts running AAD Connect reliably locate the DMZ domain controllers and not those located internally in the remote forests, which requires more than just DNS sending queries to the DMZ domain controllers.
I may come back and re-read this and add more again tonight, but this is probably enough for now until I know whether those five domain controllers are positioned in the DMZ, or spread between the DMZ and internal networks.
Cheers,
Lain