During my day to day work as a part of support organization, I work with and help troubleshoot Hybrid Configuration Wizard (HCW) failures. One of the more common causes of HCW failures is the Federation Trust step for the Exchange on-premises organizations in Full hybrid configurations (Classic or Modern topologies).
Federation trust is a mandatory step in the on-premises Exchange organizations when configuring Full hybrid deployments with Exchange Server 2010 coexistence, as this allows us to create organization relationships (for features like hybrid free/busy or OWA/EAS redirection) and sharing policies (1:1 hybrid calendar sharing). In Exchange Online multi-tenant organizations, federation trust is already in place.
Below is an illustration of an Exchange hybrid deployment where both the Exchange on-premises organization and the Exchange Online organization have a trust with Azure Authentication System (formerly called Microsoft Federation Gateway):
More info on federation trust can be found here.
Before getting to our subject, let’s quickly go over different hybrid configurations and Hybrid Configuration Wizard (HCW) - as this is the supported tool to configure hybrid deployments.
There are 2 flavors of hybrid configurations:
At this time, each of those supports the following hybrid modes:
- Full
- Minimal (which further breaks down into…)
- Express (a one-time sync)
- “Actual minimal”
A quick overview of Full / Minimal / Express options, can be found here. More info on HCW is here.
As mentioned earlier, a federation trust is created by HCW only in Full Hybrid.
HCW logs are located at %appdata%\Microsoft\Exchange Hybrid Configuration on the machine from where HCW was ran. The easiest way to get to them is to press F12 in the HCW window to open the Diagnostic tools and from there you can Open Folder Logging or Open Log File directly.
When you have issues with federation trust, the log will usually show errors when one of the following cmdlets are executed: Set-FederationOrganizationIdentifier or Add-FederatedDomain (but can be other cmdlets as well).
Once you identified the exact cmdlet failing and where (Session=OnPremises – means Exchange Management Shell and Session=Tenant means Exchange Online PowerShell), you should copy-paste the failing command and try to execute it manually and see if that is failing as well (most likely it will). You can also open the shells from F12 Diagnostic tools windows in HCW.
In order to get more details on the error and to rule out this is not an issue with HCW itself, you will need to separately run the same command that threw exception in HCW log and add Verbose switch to get verbose details of the error and the serialized remote exception.
For example, if the Exchange server version is Exchange 2010, you will run the failing command with Verbose switch in Exchange Management Shell (EMS), see if that fails and then get the serialized remote exception.
Example:
start-transcript
Set-FederatedOrganizationIdentifier -AccountNamespace <contoso.com> -DelegationFederationTrust 'Microsoft Federation Gateway' -Enabled:$true -VERBOSE
$Error[0].Exception |fl -f
$Error[0].Exception.SerializedRemoteException |fl –f
Get-FederatedOrganizationIdentifier |FL
Get-FederationTrust |FL
stop-transcript
If the Exchange Server version is Exchange 2013/2016 and the above commands didn’t show more details on the error, we can also try the following:
- Open regular Windows PowerShell (blue background) on the Exchange Server 2013/2016
- Run command: add-pssnapin *exchange*
- Run command that gave error in HCW and add a Verbose switch
Example:
start-transcript
Set-FederatedOrganizationIdentifier -AccountNamespace <contoso.com> -DelegationFederationTrust 'Microsoft Federation Gateway' -Enabled:$true -DefaultDomain $null -VERBOSE
$Error[0].Exception |fl -f
$Error[0].Exception.SerializedRemoteException |fl –f
Get-FederatedOrganizationIdentifier |FL
Get-FederationTrust |FL
stop-transcript
Once you've gathered the verbose error / serialized exception, try to understand where it is failing (or provide it to Microsoft Support together with the HCW log).
We have gathered some common federation trust errors and some tips to fix them:
1. Federation trust fails with "Object reference not set to an instance of an object"
This is a known old issue on Exchange 2016 CU7 servers, make sure your Exchange servers are updated to the latest CU.
Full error in the HCW log:
2017.10.06 01:45:56.562 *ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=21] FINISH Time=398.4ms Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Object reference not set to an instance of an object. An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object.
2017.10.06 01:45:56.563 *ERROR* 10224 [Client=UX, Page=DomainProof, Thread=21] Microsoft.Online.CSE.Hybrid.PowerShell.PowerShellInvokeException: PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Object reference not set to an instance of an object. An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object. ---> System.Management.Automation.RemoteException: Object reference not set to an instance of an object.
Resolution: Install the latest CU for Exchange 2016
2. Federation fails with "Proof of domain ownership has failed"
Full error in the HCW log:
2019.07.16 17:53:14.750 10276 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Add-FederatedDomain, Thread=19] START Add-FederatedDomain -DomainName 'contoso.com'
2019.07.06 17:53:15.375 10177 [Client=UX, Activity=Domain Ownership, Provider=OnPremises, Thread=19] PowerShell Error Record: {CategoryInfo={Activity=Add-FederatedDomain,Category=InvalidResult,Reason=DomainProofOwnershipException,TargetName=,TargetType=},ErrorDetails=,Exception=Proof of domain ownership has failed. Make sure that the TXT record for the specified domain is available in DNS. The format of the TXT record should be "example.com IN TXT hash-value" where "example.com" is the domain you want to configure for Federation and "hash-value" is the proof value generated with "Get-FederatedDomainProof -DomainName example.com".,FullyQualifiedErrorId=367408EF,Microsoft.Exchange.Management.SystemConfigurationTasks.AddFederatedDomain}
Resolution:
- Check the TXT record for your domain(s) in HCW log or in Exchange Management Shell with command Get-FederatedDomainProof -DomainName <CONTOSO.COM>
- See if it matches your published TXT record with either nslookup utility or by checking internet websites like https://digwebinterface.com/ put your domain in hostnames, type=txt, Nameservers - Authoritative
You would look for errors, missing records or unusual formatting (characters, spaces, quotes, TXT record split in half).
3. Federation fails with "An unexpected error occurred on a receive" or "An unexpected error occurred on a send."
Error in the HCW log:
2018.10.10 17:03:31.277 *ERROR* [Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier] FINISH Time=64.3s Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': An error occurred while attempting to provision Exchange to the Partner STS. Detailed Information "An error occurred accessing Windows Live. Detailed information: "The underlying connection was closed: An unexpected error occurred on a receive.".".
Verbose log shows something like this:
Set-FederatedOrganizationIdentifier -AccountNamespace 'domain.com' -DelegationFederationTrust 'Microsoft Federation Gateway' -Enabled: $true -DefaultDomain $null -Verbose VERBOSE: [12:29:07.754 GMT] Set-FederatedOrganizationIdentifier : Calling 'CreateAppId(uri='FYDIBOHF25SPDLT.domain.com',properties=[0])' at the domain services endpoint https://domains.live.com/service/managedelegation2.asmx . VERBOSE: [12:29:08.535 GMT] Set-FederatedOrganizationIdentifier : The request to Windows Live Domain Services failed with the following exception:
[0]: Microsoft.Exchange.Management.FederationProvisioning.LiveDomainServicesException An error occurred accessing Windows Live. Detailed information: "The underlying connection was closed: An unexpected error occurred on a send.".
[1]: System.Net.WebException The underlying connection was closed: An unexpected error occurred on a send.
[2]: System.IO.IOException Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
[3]: System.Net.Sockets.SocketException An existing connection was forcibly closed by the remote host
Resolution:
Check outbound access from all your Exchange Servers to Microsoft Federation Gateway by browsing using Internet Explorer with PSEXEC tool (with -s and -i switches) from the Exchange Server (this will use Internet Explorer under System Account / Exchange Server Account). Example of cmdlet:
…\Downloads\PSTools> PsExec.exe -i -s "c:\Program Files\Internet Explorer\iexplore.exe"
In this example, “Windows Live” is actually this exact URL: https://domains.live.com/service/managedelegation2.asmx
From on-premises Exchange to Office 365, the Exchange 2010 MBX & CAS or 2013 MBX (backend) or 2016 / 2019 would need outbound Internet access to the Microsoft Federation Gateway in addition to https://outlook.office365.com/ews/exchange.asmx
Verify the machine/system account can access these Microsoft Federation Gateway URLs:
- https://nexus.microsoftonline-p.com/federationmetadata/2006-12/federationmetadata.xml [<-- You should see an xml page.]
- https://login.microsoftonline.com/extSTS.srf [<-- You should see “Sorry, but we’re having trouble signing you in”.]
- https://domains.live.com/service/managedelegation2.asmx [<-- You should see the operations supported by ManageDelegation2.]
For a complete list of O365 URL & IP addresses, see these articles:
- Office 365 URLs and IP address ranges – Exchange Online
- Office 365 URLs and IP address ranges – Common
- Additional endpoints
Note: If the Exchange requires a proxy server to access the Internet, specify the proxy server using "Set-ExchangeServer myExchange01 -InternetWebProxy http://myproxy:80". Notice such proxy can't require any user authentication for outbound Internet access, and the proxy must start with HTTP: and not HTTPS: (secure SSL).
Another common cause for this error is TLS 1.2. Please make sure your on-premises environment supports TLS 1.2, Office 365 has deprecated TLS protocols 1.0 and 1.1, references below on how to implement TLS 1.2.
- Exchange Server TLS guidance, part 1: Getting Ready for TLS 1.2
- Exchange Server TLS guidance Part 2: Enabling TLS 1.2 and Identifying Clients Not Using It
- Exchange Server TLS guidance Part 3: Turning Off TLS 1.0/1.1
- Preparing for TLS 1.2 in Office 365 and Office 365 GCC
In rare instances, you can use the machine/system account to access the URLs from the browser, but Exchange cmdlets still failed with "Could not establish trust relationship for the SSL/TLS secure channel." If that happens, make sure the certificate authorities for the urls are installed at the Third-Party Root Certification Authorities of the machine local certificate location.
Reference:
Netsh Commands for Windows Hypertext Transfer Protocol (WINHTTP)
Firewall Considerations for Federated Delegation Federated delegation features require that the Mailbox and Client Access servers in your organization have outbound access to the Internet by using HTTPS. You must allow outbound HTTPS access (port 443 for TCP) from all Exchange 2010 Mailbox and Client Access servers in the organization.
4. There is no specific error / exception, in HCW log you would see it stops without any specific error.
From the HCW log:
2019.02.14 12:56:21.658 [Activity=Domain Ownership, Session=OnPremises, Cmdlet=Get-FederatedOrganizationIdentifier] FINISH Time=133.0ms Results=1 FederatedOrganizationIdWithDomainStatus {AccountNamespace='FYDIBOHF25SPDLT.contoso.com' DelegationTrustLink='contoso.local/Configuration/Deleted Objects/Microsoft Federation Gateway DEL:8e834abf-5154-4540-a3c6-5a5c614c6a06'Enabled=1 ExchangeVersion='0.10 (14.0.100.0)' Guid=2e1da884-9686-4221-8098-d34ced5a2f85 Id='Federation' Identity='Federation' IsValid=1 Name='Federation' ObjectState='Unchanged' WhenChanged='8/11/2015 5:35:58 PM' WhenChangedUTC='8/11/2015 2:35:58 PM' WhenCreated='10/18/2009 10:30:09 AM' WhenCreatedUTC='10/18/2009 6:30:09 AM'}
2019.02.14 12:56:21.677 [Client=UX, Page=DomainProof] Unproven Domains: 1
Resolution:
Look for orphaned federation trust in Get-FederatedOrganizationIdentifier | FL or in HCW log if you see something with "DEL": "contoso.com/Configuration/Deleted Objects/Microsoft Federation Gateway/DEL: <xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx>". Solution is to remove the orphaned federation trust and re-run HCW.
Reference here.
Note: as a first step, you can try to run the command remove-federateddomain with the switch -Force. Also, you don't need to recreate federation trust manually, just re-run HCW (this will recreate federation trust for us)
5. Federation Trust fails with "InternalError InternalError: Internal error.".".""
Error from the HCW log:
2019.08.23 07:45:22.914 10276 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=20] START Set-FederatedOrganizationIdentifier -AccountNamespace 'contoso.com' -DelegationFederationTrust 'Microsoft Federation Gateway' -Enabled: $true -DefaultDomain $null
2019.08.23 07:45:23.239 *ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=20] FINISH Time=325.0ms Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': An error occurred while attempting to provision Exchange to the Partner STS. Detailed Information "An unexpected result was received from Windows Live. Detailed information: "InternalError InternalError: Internal error.".". {CategoryInfo={Activity=[System.String] Set-FederatedOrganizationIdentifier,Category=[System.Management.Automation.ErrorCategory] InvalidResult,Reason=[System.String]
Resolution:
Open request with Microsoft Support or check if any Service Incident is published. Please see this.
6. Federation trust fails with "1007 Access Denied"
Error from the HCW log:
Set-FederatedOrganizationIdentifier,Category=[System.Management.Automation.ErrorCategory] InvalidResult,Reason=[System.String] ProvisioningFederatedExchangeException,TargetName=[System.String] ,TargetType=[System.String] },ErrorDetails=,Exception=[System.Management.Automation.RemoteException] An error occurred while attempting to provision Exchange to the Partner STS. Detailed Information "An unexpected result was received from Windows Live. Detailed information: "1007 AccessDenied: Access Denied."."
Resolution:
"1007 Access Denied" error is usually when we have issues with:
- Windows Time on the Exchange Server. See this article or this article.
- Outdated federation trust (for example, federation trust certificate expired) and in this case you would remove federation trust by following these steps.
If the federation trust certificate is not found on any of the servers, then proceed with resolution from the next error.
As an example, from one HCW log, there seems to be this federation trust certificate expired on 13/05/2019:
OrgCertificate=[Subject] CN=Federation [Issuer] CN=Federation [Serial Number] 4E91XXXXXXXXXXXXXXXXXXXXXXXXXXXX [Not Before] 5/13/2014 11:21:36 AM [Not After] 5/13/2019 11:21:36 AM
7. Federation trust fails with “Federation Certificate cannot be found”
Error from HCW log:
2019.11.22 14:40:32.569 *ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Get-FederatedDomainProof, Thread=6] FINISH Time=125.1ms Results=PowerShell failed to invoke Get-FederatedDomainProof: Federation certificate with the thumbprint ‘03650FFAF05E83E3B007DF3473CB5753F5C4459’cannot be found.
Resolution:
Follow the procedure here to manually cleanup the federation trust from AD. Once this is done, re-run HCW to re-create it automatically.
Hopefully, this helps with troubleshooting those errors! I wanted to thank Raymond Fong and Nino Bilic for their review of this post.