%3CLINGO-SUB%20id%3D%22lingo-sub-1144285%22%20slang%3D%22en-US%22%3EHow%20to%20address%20Federation%20Trust%20issues%20in%20Hybrid%20Configuration%20Wizard%20(HCW)%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1144285%22%20slang%3D%22en-US%22%3E%3CP%3EDuring%20my%20day%20to%20day%20work%20as%20a%20part%20of%20support%20organization%2C%20I%20work%20with%20and%20help%20troubleshoot%20Hybrid%20Configuration%20Wizard%20(HCW)%20failures.%20One%20of%20the%20more%20common%20causes%20of%20HCW%20failures%20is%20the%20Federation%20Trust%20step%20for%20the%20Exchange%20on-premises%20organizations%20in%20Full%20hybrid%20configurations%20(Classic%20or%20Modern%20topologies).%3C%2FP%3E%0A%3CP%3EFederation%20trust%20is%20a%20mandatory%20step%20in%20the%20on-premises%20Exchange%20organizations%20when%20configuring%20Full%20hybrid%20deployments%2C%20as%20this%20allows%20us%20to%20create%20organization%20relationships%20(for%20features%20like%20hybrid%20free%2Fbusy%20or%20OWA%2FEAS%20redirection)%20and%20sharing%20policies%20(1%3A1%20hybrid%20calendar%20sharing).%20In%20Exchange%20Online%20multi-tenant%20organizations%2C%20federation%20trust%20is%20already%20in%20place.%3C%2FP%3E%0A%3CP%3EBelow%20is%20an%20illustration%20of%20an%20Exchange%20hybrid%20deployment%20where%20both%20the%20Exchange%20on-premises%20organization%20and%20the%20Exchange%20Online%20organization%20have%20a%20trust%20with%20Azure%20Authentication%20System%20(formerly%20called%20Microsoft%20Federation%20Gateway)%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F168340i6498300C7C2FA3EE%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22HFB1%22%20title%3D%22HFB1%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EMore%20info%20on%20federation%20trust%20can%20be%20found%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fconfigure-a-federation-trust-exchange-2013-help%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehere%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3EBefore%20getting%20to%20our%20subject%2C%20let%E2%80%99s%20quickly%20go%20over%20different%20hybrid%20configurations%20and%20Hybrid%20Configuration%20Wizard%20(HCW)%20-%20as%20this%20is%20the%20supported%20tool%20to%20configure%20hybrid%20deployments.%3C%2FP%3E%0A%3CP%3EThere%20are%202%20flavors%20of%20hybrid%20configurations%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fexchange-hybrid%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EClassic%20hybrid%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment%2Fhybrid-agent%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EModern%20hybrid%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EAt%20this%20time%2C%20each%20of%20those%20supports%20the%20following%20hybrid%20modes%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EFull%3C%2FLI%3E%0A%3CLI%3EMinimal%20(which%20further%20breaks%20down%20into%E2%80%A6)%3COL%3E%0A%3CLI%3EExpress%20(a%20one-time%20sync)%3C%2FLI%3E%0A%3CLI%3E%E2%80%9CActual%20minimal%E2%80%9D%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EA%20quick%20overview%20of%20Full%20%2F%20Minimal%20%2F%20Express%20options%2C%20can%20be%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FExchange-Team-Blog%2FNew-Exchange-Online-migration-options%2Fba-p%2F606109%22%20target%3D%22_blank%22%3Efound%20here%3C%2FA%3E.%20More%20info%20on%20HCW%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-configuration-wizard-faqs%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eis%20here%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3EAs%20mentioned%20earlier%2C%20a%20federation%20trust%20is%20created%20by%20HCW%20only%20in%20%3CSTRONG%3EFull%20Hybrid%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3EHCW%20logs%20are%20located%20at%20%3CSTRONG%3E%25appdata%25%5CMicrosoft%5CExchange%20Hybrid%20Configuration%3C%2FSTRONG%3E%20on%20the%20machine%20from%20where%20HCW%20was%20ran.%20The%20easiest%20way%20to%20get%20to%20them%20is%20to%20press%20F12%20in%20the%20HCW%20window%20to%20open%20the%20Diagnostic%20tools%20and%20from%20there%20you%20can%20%3CSTRONG%3EOpen%20Folder%20Logging%3C%2FSTRONG%3E%20or%20%3CSTRONG%3EOpen%20Log%20File%3C%2FSTRONG%3E%20directly.%3C%2FP%3E%0A%3CP%3EWhen%20you%20have%20issues%20with%20federation%20trust%2C%20the%20log%20will%20usually%20show%20errors%20when%20one%20of%20the%20following%20cmdlets%20are%20executed%3A%20Set-FederationOrganizationIdentifier%20or%20Add-FederatedDomain%20(but%20can%20be%20other%20cmdlets%20as%20well).%3C%2FP%3E%0A%3CP%3EOnce%20you%20identified%20the%20exact%20cmdlet%20failing%20and%20where%20(Session%3DOnPremises%20%E2%80%93%20means%20Exchange%20Management%20Shell%20and%20Session%3DTenant%20means%20Exchange%20Online%20PowerShell)%2C%20you%20should%20copy-paste%20the%20failing%20command%20and%20try%20to%20execute%20it%20manually%20and%20see%20if%20that%20is%20failing%20as%20well%20(most%20likely%20it%20will).%20You%20can%20also%20open%20the%20shells%20from%20F12%20Diagnostic%20tools%20windows%20in%20HCW.%3C%2FP%3E%0A%3CP%3EIn%20order%20to%20get%20more%20details%20on%20the%20error%20and%20to%20rule%20out%20this%20is%20not%20an%20issue%20with%20HCW%20itself%2C%20you%20will%20need%20to%20separately%20run%20the%20same%20command%20that%20threw%20exception%20in%20HCW%20log%20and%20add%20Verbose%20switch%20to%20get%20verbose%20details%20of%20the%20error%20and%20the%20serialized%20remote%20exception.%3C%2FP%3E%0A%3CP%3EFor%20example%2C%20if%20the%20Exchange%20server%20version%20is%20Exchange%202010%2C%20you%20will%20run%20the%20failing%20command%20with%20Verbose%20switch%20in%20Exchange%20Management%20Shell%20(EMS)%2C%20see%20if%20that%20fails%20and%20then%20get%20the%20serialized%20remote%20exception.%3C%2FP%3E%0A%3CP%3EExample%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Estart-transcript%0ASet-FederatedOrganizationIdentifier%20-AccountNamespace%20%3CCONTOSO.COM%3E%20-DelegationFederationTrust%20'Microsoft%20Federation%20Gateway'%20-Enabled%3A%24true%20-VERBOSE%0A%24Error%5B0%5D.Exception%20%7Cfl%20-f%0A%24Error%5B0%5D.Exception.SerializedRemoteException%20%7Cfl%20%E2%80%93f%0AGet-FederatedOrganizationIdentifier%20%7CFL%0AGet-FederationTrust%20%7CFL%0Astop-transcript%3C%2FCONTOSO.COM%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20the%20Exchange%20Server%20version%20is%20Exchange%202013%2F2016%20and%20the%20above%20commands%20didn%E2%80%99t%20show%20more%20details%20on%20the%20error%2C%20we%20can%20also%20try%20the%20following%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EOpen%20regular%20Windows%20PowerShell%20(blue%20background)%20on%20the%20Exchange%20Server%202013%2F2016%3C%2FLI%3E%0A%3CLI%3ERun%20command%3A%20add-pssnapin%20*exchange*%3C%2FLI%3E%0A%3CLI%3ERun%20command%20that%20gave%20error%20in%20HCW%20and%20add%20a%20Verbose%20switch%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EExample%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Estart-transcript%0ASet-FederatedOrganizationIdentifier%20-AccountNamespace%20%3CCONTOSO.COM%3E%20-DelegationFederationTrust%20'Microsoft%20Federation%20Gateway'%20-Enabled%3A%24true%20-DefaultDomain%20%24null%20-VERBOSE%20%0A%24Error%5B0%5D.Exception%20%7Cfl%20-f%0A%24Error%5B0%5D.Exception.SerializedRemoteException%20%7Cfl%20%E2%80%93f%0AGet-FederatedOrganizationIdentifier%20%7CFL%0AGet-FederationTrust%20%7CFL%0Astop-transcript%3C%2FCONTOSO.COM%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20you've%20gathered%20the%20verbose%20error%20%2F%20serialized%20exception%2C%20try%20to%20understand%20where%20it%20is%20failing%20(or%20provide%20it%20to%20Microsoft%20Support%20together%20with%20the%20HCW%20log).%3C%2FP%3E%0A%3CH3%20id%3D%22toc-hId-1085364976%22%20id%3D%22toc-hId-1114860798%22%20id%3D%22toc-hId-1114860798%22%3E%3CFONT%20size%3D%224%22%3EWe%20have%20gathered%20some%20common%20federation%20trust%20errors%20and%20some%20tips%20to%20fix%20them%3A%3C%2FFONT%3E%3C%2FH3%3E%0A%3CH4%20id%3D%22toc-hId-1775926450%22%20id%3D%22toc-hId-1805422272%22%20id%3D%22toc-hId-1805422272%22%3E%3CFONT%20size%3D%225%22%3E%3CSPAN%3E1.%20Federation%20trust%20fails%20with%20%22Object%20reference%20not%20set%20to%20an%20instance%20of%20an%20object%22%3C%2FSPAN%3E%3C%2FFONT%3E%3C%2FH4%3E%0A%3CP%3EThis%20is%20a%20known%20old%20issue%20on%20Exchange%202016%20CU7%20servers%2C%20make%20sure%20your%20Exchange%20servers%20are%20updated%20to%20the%20latest%20CU.%3C%2FP%3E%0A%3CP%3EFull%20error%20in%20the%20HCW%20log%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3E2017.10.06%2001%3A45%3A56.562%20*ERROR*%2010277%20%5BClient%3DUX%2C%20Activity%3DDomain%20Ownership%2C%20Session%3DOnPremises%2C%20Cmdlet%3DSet-FederatedOrganizationIdentifier%2C%20Thread%3D21%5D%20FINISH%20Time%3D398.4ms%20Results%3DPowerShell%20failed%20to%20invoke%20'Set-FederatedOrganizationIdentifier'%3A%20Object%20reference%20not%20set%20to%20an%20instance%20of%20an%20object.%20An%20unexpected%20error%20has%20occurred%20and%20a%20Watson%20dump%20is%20being%20generated%3A%20Object%20reference%20not%20set%20to%20an%20instance%20of%20an%20object.%0A2017.10.06%2001%3A45%3A56.563%20*ERROR*%2010224%20%5BClient%3DUX%2C%20Page%3DDomainProof%2C%20Thread%3D21%5D%20Microsoft.Online.CSE.Hybrid.PowerShell.PowerShellInvokeException%3A%20PowerShell%20failed%20to%20invoke%20'Set-FederatedOrganizationIdentifier'%3A%20Object%20reference%20not%20set%20to%20an%20instance%20of%20an%20object.%20An%20unexpected%20error%20has%20occurred%20and%20a%20Watson%20dump%20is%20being%20generated%3A%20Object%20reference%20not%20set%20to%20an%20instance%20of%20an%20object.%20---%26gt%3B%20System.Management.Automation.RemoteException%3A%20Object%20reference%20not%20set%20to%20an%20instance%20of%20an%20object.%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EResolution%3A%20%3C%2FSTRONG%3E%3CA%20href%3D%22https%3A%2F%2Fsupport.microsoft.com%2Fen-gb%2Fhelp%2F4051381%2Foffice-365-hcw-fails-to-run-on-exchange-2016-cu7%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EInstall%20the%20latest%20CU%20for%20Exchange%202016%3C%2FA%3E%3C%2FP%3E%0A%3CH4%20id%3D%22toc-hId--31528013%22%20id%3D%22toc-hId--2032191%22%20id%3D%22toc-hId--2032191%22%3E%3CFONT%20size%3D%225%22%3E%3CSPAN%3E2.%20%3C%2FSPAN%3E%3CSPAN%3EFederation%20fails%20with%20%22Proof%20of%20domain%20ownership%20has%20failed%22%3C%2FSPAN%3E%3C%2FFONT%3E%3C%2FH4%3E%0A%3CP%3EFull%20error%20in%20the%20HCW%20log%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3E2019.07.16%2017%3A53%3A14.750%2010276%20%5BClient%3DUX%2C%20Activity%3DDomain%20Ownership%2C%20Session%3DOnPremises%2C%20Cmdlet%3DAdd-FederatedDomain%2C%20Thread%3D19%5D%20START%20Add-FederatedDomain%20-DomainName%20'contoso.com'%0A2019.07.06%2017%3A53%3A15.375%2010177%20%5BClient%3DUX%2C%20Activity%3DDomain%20Ownership%2C%20Provider%3DOnPremises%2C%20Thread%3D19%5D%20PowerShell%20Error%20Record%3A%20%7BCategoryInfo%3D%7BActivity%3DAdd-FederatedDomain%2CCategory%3DInvalidResult%2CReason%3DDomainProofOwnershipException%2CTargetName%3D%2CTargetType%3D%7D%2CErrorDetails%3D%2CException%3DProof%20of%20domain%20ownership%20has%20failed.%20Make%20sure%20that%20the%20TXT%20record%20for%20the%20specified%20domain%20is%20available%20in%20DNS.%20The%20format%20of%20the%20TXT%20record%20should%20be%20%22example.com%20IN%20TXT%20hash-value%22%20where%20%22example.com%22%20is%20the%20domain%20you%20want%20to%20configure%20for%20Federation%20and%20%22hash-value%22%20is%20the%20proof%20value%20generated%20with%20%22Get-FederatedDomainProof%20-DomainName%20example.com%22.%2CFullyQualifiedErrorId%3D367408EF%2CMicrosoft.Exchange.Management.SystemConfigurationTasks.AddFederatedDomain%7D%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EResolution%3A%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3ECheck%20the%20TXT%20record%20for%20your%20domain(s)%20in%20HCW%20log%20or%20in%20Exchange%20Management%20Shell%20with%20command%20Get-FederatedDomainProof%20-DomainName%20%3CCONTOSO.COM%3E%3C%2FCONTOSO.COM%3E%3C%2FLI%3E%0A%3CLI%3ESee%20if%20it%20matches%20your%20published%20TXT%20record%20with%20either%20nslookup%20utility%20or%20by%20checking%20internet%20websites%20like%20%3CA%20href%3D%22https%3A%2F%2Fdigwebinterface.com%2F%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdigwebinterface.com%2F%3C%2FA%3E%20put%20your%20domain%20in%20hostnames%2C%20type%3Dtxt%2C%20Nameservers%20-%20Authoritative%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EYou%20would%20look%20for%20errors%2C%20missing%20records%20or%20unusual%20formatting%20(characters%2C%20spaces%2C%20quotes%2C%20TXT%20record%20split%20in%20half).%3C%2FP%3E%0A%3CH4%20id%3D%22toc-hId--1838982476%22%20id%3D%22toc-hId--1809486654%22%20id%3D%22toc-hId--1809486654%22%3E%3CFONT%20size%3D%225%22%3E%3CSPAN%3E3.%20Federation%20fails%20with%20%22An%20unexpected%20error%20occurred%20on%20a%20receive%22%20or%20%22An%20unexpected%20error%20occurred%20on%20a%20send.%22%3C%2FSPAN%3E%3C%2FFONT%3E%3C%2FH4%3E%0A%3CP%3EError%20in%20the%20HCW%20log%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3E2018.10.10%2017%3A03%3A31.277%20*ERROR*%20%5BActivity%3DDomain%20Ownership%2C%20Session%3DOnPremises%2C%20Cmdlet%3DSet-FederatedOrganizationIdentifier%5D%20FINISH%20Time%3D64.3s%20Results%3DPowerShell%20failed%20to%20invoke%20'Set-FederatedOrganizationIdentifier'%3A%20An%20error%20occurred%20while%20attempting%20to%20provision%20Exchange%20to%20the%20Partner%20STS.%20Detailed%20Information%20%22An%20error%20occurred%20accessing%20Windows%20Live.%20Detailed%20information%3A%20%22The%20underlying%20connection%20was%20closed%3A%20An%20unexpected%20error%20occurred%20on%20a%20receive.%22.%22.%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EVerbose%20log%20shows%20something%20like%20this%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3ESet-FederatedOrganizationIdentifier%20-AccountNamespace%20'domain.com'%20-DelegationFederationTrust%20'Microsoft%20Federation%20Gateway'%20-Enabled%3A%20%24true%20-DefaultDomain%20%24null%20-Verbose%20VERBOSE%3A%20%5B12%3A29%3A07.754%20GMT%5D%20Set-FederatedOrganizationIdentifier%20%3A%20Calling%20'CreateAppId(uri%3D'FYDIBOHF25SPDLT.domain.com'%2Cproperties%3D%5B0%5D)'%20at%20the%20domain%20services%20endpoint%20%3CA%20href%3D%22https%3A%2F%2Fdomains.live.com%2Fservice%2Fmanagedelegation2.asmx%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdomains.live.com%2Fservice%2Fmanagedelegation2.asmx%3C%2FA%3E%20.%20VERBOSE%3A%20%5B12%3A29%3A08.535%20GMT%5D%20Set-FederatedOrganizationIdentifier%20%3A%20The%20request%20to%20Windows%20Live%20Domain%20Services%20failed%20with%20the%20following%20exception%3A%20%0A%5B0%5D%3A%20Microsoft.Exchange.Management.FederationProvisioning.LiveDomainServicesException%20An%20error%20occurred%20accessing%20Windows%20Live.%20Detailed%20information%3A%20%22The%20underlying%20connection%20was%20closed%3A%20An%20unexpected%20error%20occurred%20on%20a%20send.%22.%0A%5B1%5D%3A%20System.Net.WebException%20The%20underlying%20connection%20was%20closed%3A%20An%20unexpected%20error%20occurred%20on%20a%20send.%0A%5B2%5D%3A%20System.IO.IOException%20Unable%20to%20read%20data%20from%20the%20transport%20connection%3A%20An%20existing%20connection%20was%20forcibly%20closed%20by%20the%20remote%20host.%0A%5B3%5D%3A%20System.Net.Sockets.SocketException%20An%20existing%20connection%20was%20forcibly%20closed%20by%20the%20remote%20host%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EResolution%3A%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3ECheck%20outbound%20access%20from%20all%20your%20Exchange%20Servers%20to%20Microsoft%20Federation%20Gateway%20by%20browsing%20using%20Internet%20Explorer%20with%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fsysinternals%2Fdownloads%2Fpstools%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EPSEXEC%20tool%3C%2FA%3E%20(with%20-s%20and%20-i%20switches)%20from%20the%20Exchange%20Server%20(this%20will%20use%20Internet%20Explorer%20under%20System%20Account%20%2F%20Exchange%20Server%20Account).%3C%2FP%3E%0A%3CP%3EIn%20this%20example%2C%20%E2%80%9CWindows%20Live%E2%80%9D%20is%20actually%20this%20exact%20URL%3A%20%3CA%20href%3D%22https%3A%2F%2Fdomains.live.com%2Fservice%2Fmanagedelegation2.asmx%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdomains.live.com%2Fservice%2Fmanagedelegation2.asmx%3C%2FA%3E%3C%2FP%3E%0A%3CP%3EFrom%20on-premises%20Exchange%20to%20Office%20365%2C%20the%20Exchange%202010%20MBX%20%26amp%3B%20CAS%20or%202013%20MBX%20(backend)%20or%202016%20%2F%202019%20would%20need%20outbound%20Internet%20access%20to%20the%20Microsoft%20Federation%20Gateway%20in%20addition%20to%20%3CA%20href%3D%22https%3A%2F%2Foutlook.office365.com%2Fews%2Fexchange.asmx%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Foutlook.office365.com%2Fews%2Fexchange.asmx%3C%2FA%3E%3C%2FP%3E%0A%3CP%3EVerify%20the%20machine%2Fsystem%20account%20can%20access%20these%20%3CSTRONG%3EMicrosoft%20Federation%20Gateway%20%3C%2FSTRONG%3EURLs%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fnexus.microsoftonline-p.com%2Ffederationmetadata%2F2006-12%2Ffederationmetadata.xml%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fnexus.microsoftonline-p.com%2Ffederationmetadata%2F2006-12%2Ffederationmetadata.xml%3C%2FA%3E%20%5B%26lt%3B--%20You%20should%20see%20an%20xml%20page.%5D%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Flogin.microsoftonline.com%2FextSTS.srf%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Flogin.microsoftonline.com%2FextSTS.srf%3C%2FA%3E%20%5B%26lt%3B--%20You%20should%20see%20%E2%80%9CSorry%2C%20but%20we%E2%80%99re%20having%20trouble%20signing%20you%20in%E2%80%9D.%5D%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fdomains.live.com%2Fservice%2Fmanagedelegation2.asmx%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdomains.live.com%2Fservice%2Fmanagedelegation2.asmx%3C%2FA%3E%20%5B%26lt%3B--%20You%20should%20see%20the%20operations%20supported%20by%20ManageDelegation2.%5D%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EFor%20a%20complete%20list%20of%20O365%20URL%20%26amp%3B%20IP%20addresses%2C%20see%20these%20articles%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Foffice365%2Fenterprise%2Furls-and-ip-address-ranges%23exchange-online%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EOffice%20365%20URLs%20and%20IP%20address%20ranges%20%E2%80%93%20Exchange%20Online%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Foffice365%2Fenterprise%2Furls-and-ip-address-ranges%23microsoft-365-common-and-office-online%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EOffice%20365%20URLs%20and%20IP%20address%20ranges%20%E2%80%93%20Common%3C%2FA%3E%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Foffice365%2Fenterprise%2Fadditional-office365-ip-addresses-and-urls%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAdditional%20endpoints%3C%2FA%3E%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3E%3CSTRONG%3ENote%3A%3C%2FSTRONG%3E%20If%20the%20Exchange%20requires%20a%20proxy%20server%20to%20access%20the%20Internet%2C%20specify%20the%20proxy%20server%20using%20%22Set-ExchangeServer%20myExchange01%20-InternetWebProxy%20http%3A%2F%2Fmyproxy%3A80%22.%20Notice%20such%20proxy%20can't%20require%20any%20user%20authentication%20for%20outbound%20Internet%20access%2C%20and%20the%20proxy%20must%20start%20with%20HTTP%3A%20and%20not%20HTTPS%3A%20(secure%20SSL).%3C%2FP%3E%0A%3CP%3EYou%20can%20also%20set%20the%20proxy%20using%20netsh%20as%20well.%3C%2FP%3E%0A%3CP%3Eset%20proxy%20proxy-server%3D%22http%3Dmyproxy%3Bhttps%3Dsproxy%3A88%22%20bypass-list%3D%22*.contoso.com%22%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20rare%20instances%2C%20you%20can%20use%20the%20machine%2Fsystem%20account%20to%20access%20the%20URLs%20from%20the%20browser%2C%20but%20Exchange%20cmdlets%20still%20failed%20with%20%22Could%20not%20establish%20trust%20relationship%20for%20the%20SSL%2FTLS%20secure%20channel.%22%20If%20that%20happens%2C%20make%20sure%20the%20certificate%20authorities%20for%20the%20urls%20are%20installed%20at%20the%20%3CSTRONG%3EThird-Party%20Root%20Certification%20Authorities%3C%2FSTRONG%3E%20of%20the%20machine%20local%20certificate%20location.%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EReference%3A%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fprevious-versions%2Fwindows%2Fit-pro%2Fwindows-server-2008-R2-and-2008%2Fcc731131(v%3Dws.10)%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3ENetsh%20Commands%20for%20Windows%20Hypertext%20Transfer%20Protocol%20(WINHTTP)%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fprevious-versions%2Foffice%2Fexchange-server-2010%2Fdd638083(v%3Dexchg.141)%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EFirewall%20Considerations%20for%20Federated%20Delegation%3C%2FA%3E%20Federated%20delegation%20features%20require%20that%20the%20Mailbox%20and%20Client%20Access%20servers%20in%20your%20organization%20have%20outbound%20access%20to%20the%20Internet%20by%20using%20HTTPS.%20You%20must%20allow%20outbound%20HTTPS%20access%20(port%20443%20for%20TCP)%20from%20all%20Exchange%202010%20Mailbox%20and%20Client%20Access%20servers%20in%20the%20organization.%3C%2FP%3E%0A%3CH4%20id%3D%22toc-hId-648530357%22%20id%3D%22toc-hId-678026179%22%20id%3D%22toc-hId-678026179%22%3E%3CFONT%20size%3D%225%22%3E%3CSPAN%3E4.%20There%20is%20no%20specific%20error%20%2F%20exception%2C%20in%20HCW%20log%20you%20would%20see%20it%20stops%20without%20any%20specific%20error.%3C%2FSPAN%3E%3C%2FFONT%3E%3C%2FH4%3E%0A%3CP%3EFrom%20the%20HCW%20log%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3E2019.02.14%2012%3A56%3A21.658%20%5BActivity%3DDomain%20Ownership%2C%20Session%3DOnPremises%2C%20Cmdlet%3DGet-FederatedOrganizationIdentifier%5D%20FINISH%20Time%3D133.0ms%20Results%3D1%20FederatedOrganizationIdWithDomainStatus%20%7BAccountNamespace%3D'FYDIBOHF25SPDLT.contoso.com'%20DelegationTrustLink%3D'contoso.local%2FConfiguration%2FDeleted%20Objects%2FMicrosoft%20Federation%20Gateway%20DEL%3A8e834abf-5154-4540-a3c6-5a5c614c6a06'Enabled%3D1%20ExchangeVersion%3D'0.10%20(14.0.100.0)'%20Guid%3D2e1da884-9686-4221-8098-d34ced5a2f85%20Id%3D'Federation'%20Identity%3D'Federation'%20IsValid%3D1%20Name%3D'Federation'%20ObjectState%3D'Unchanged'%20WhenChanged%3D'8%2F11%2F2015%205%3A35%3A58%20PM'%20WhenChangedUTC%3D'8%2F11%2F2015%202%3A35%3A58%20PM'%20WhenCreated%3D'10%2F18%2F2009%2010%3A30%3A09%20AM'%20WhenCreatedUTC%3D'10%2F18%2F2009%206%3A30%3A09%20AM'%7D%0A2019.02.14%2012%3A56%3A21.677%20%5BClient%3DUX%2C%20Page%3DDomainProof%5D%20Unproven%20Domains%3A%201%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EResolution%3A%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3ELook%20for%20orphaned%20federation%20trust%20in%20Get-FederatedOrganizationIdentifier%20%7C%20FL%20or%20in%20HCW%20log%20if%20you%20see%20something%20with%20%22%3CSTRONG%3EDEL%3C%2FSTRONG%3E%22%3A%20%22contoso.com%2FConfiguration%2FDeleted%20Objects%2FMicrosoft%20Federation%20Gateway%2FDEL%3A%20%3CXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX%3E%22.%20Solution%20is%20to%20remove%20the%20orphaned%20federation%20trust%20and%20re-run%20HCW.%3C%2FXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX%3E%3C%2FP%3E%0A%3CP%3EReference%20here.%3C%2FP%3E%0A%3CP%3ENote%3A%20as%20a%20first%20step%2C%20you%20can%20try%20to%20run%20the%20command%20%3CEM%3Eremove-federateddomain%3C%2FEM%3E%20with%20the%20switch%20%3CSTRONG%3E-Force%3C%2FSTRONG%3E.%20Also%2C%20you%20don't%20need%20to%20recreate%20federation%20trust%20manually%2C%20just%20re-run%20HCW%20(this%20will%20recreate%20federation%20trust%20for%20us)%3C%2FP%3E%0A%3CH4%20id%3D%22toc-hId--1158924106%22%20id%3D%22toc-hId--1129428284%22%20id%3D%22toc-hId--1129428284%22%3E%3CFONT%20size%3D%225%22%3E%3CSPAN%3E5.%20Federation%20Trust%26nbsp%3Bfails%20with%20%22InternalError%20InternalError%3A%20Internal%20error.%22.%22.%22%22%3C%2FSPAN%3E%3C%2FFONT%3E%3C%2FH4%3E%0A%3CP%3EError%20from%20the%20HCW%20log%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3E2019.08.23%2007%3A45%3A22.914%20%20%20%20%20%20%20%20%2010276%20%5BClient%3DUX%2C%20Activity%3DDomain%20Ownership%2C%20Session%3DOnPremises%2C%20Cmdlet%3DSet-FederatedOrganizationIdentifier%2C%20Thread%3D20%5D%20START%20Set-FederatedOrganizationIdentifier%20-AccountNamespace%20'contoso.com'%20-DelegationFederationTrust%20'Microsoft%20Federation%20Gateway'%20-Enabled%3A%20%24true%20-DefaultDomain%20%24null%0A2019.08.23%2007%3A45%3A23.239%20*ERROR*%2010277%20%5BClient%3DUX%2C%20Activity%3DDomain%20Ownership%2C%20Session%3DOnPremises%2C%20Cmdlet%3DSet-FederatedOrganizationIdentifier%2C%20Thread%3D20%5D%20FINISH%20Time%3D325.0ms%20Results%3DPowerShell%20failed%20to%20invoke%20'Set-FederatedOrganizationIdentifier'%3A%20An%20error%20occurred%20while%20attempting%20to%20provision%20Exchange%20to%20the%20Partner%20STS.%20%20Detailed%20Information%20%22An%20unexpected%20result%20was%20received%20from%20Windows%20Live.%20%20Detailed%20information%3A%20%22InternalError%20InternalError%3A%20Internal%20error.%22.%22.%20%7BCategoryInfo%3D%7BActivity%3D%5BSystem.String%5D%20Set-FederatedOrganizationIdentifier%2CCategory%3D%5BSystem.Management.Automation.ErrorCategory%5D%20InvalidResult%2CReason%3D%5BSystem.String%5D%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EResolution%3A%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EOpen%20request%20with%20Microsoft%20Support%20or%20check%20if%20any%20Service%20Incident%20is%20published.%20Please%20%3CA%20href%3D%22https%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Fhelp%2F3063582%2F-internalerror-internalerror-internal-error-when-you-run-set-federated%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Esee%20this%3C%2FA%3E.%3C%2FP%3E%0A%3CH4%20id%3D%22toc-hId-1328588727%22%20id%3D%22toc-hId-1358084549%22%20id%3D%22toc-hId-1358084549%22%3E%3CFONT%20size%3D%225%22%3E%3CSPAN%3E6.%20Federation%20trust%20fails%20with%20%221007%20Access%20Denied%22%3C%2FSPAN%3E%3C%2FFONT%3E%3C%2FH4%3E%0A%3CP%3EError%20from%20the%20HCW%20log%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3ESet-FederatedOrganizationIdentifier%2CCategory%3D%5BSystem.Management.Automation.ErrorCategory%5D%20InvalidResult%2CReason%3D%5BSystem.String%5D%20ProvisioningFederatedExchangeException%2CTargetName%3D%5BSystem.String%5D%20%2CTargetType%3D%5BSystem.String%5D%20%7D%2CErrorDetails%3D%2CException%3D%5BSystem.Management.Automation.RemoteException%5D%20An%20error%20occurred%20while%20attempting%20to%20provision%20Exchange%20to%20the%20Partner%20STS.%20%20Detailed%20Information%20%22An%20unexpected%20result%20was%20received%20from%20Windows%20Live.%20%20Detailed%20information%3A%20%221007%20AccessDenied%3A%20Access%20Denied.%22.%22%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EResolution%3A%20%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%221007%20Access%20Denied%22%20error%20is%20usually%20when%20we%20have%20issues%20with%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EWindows%20Time%20on%20the%20Exchange%20Server.%20See%20%3CA%20href%3D%22https%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Fhelp%2F3107293%2F1007-accessdenied-or-ensure-your-system-time-is-correct-error-when-you%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ethis%20article%3C%2FA%3E%20or%20%3CA%20href%3D%22https%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Fhelp%2F816042%2Fhow-to-configure-an-authoritative-time-server-in-windows-server%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ethis%20article%3C%2FA%3E.%3C%2FLI%3E%0A%3CLI%3EOutdated%20federation%20trust%20(for%20example%2C%20federation%20trust%20certificate%20expired)%20and%20in%20this%20case%20you%20would%20remove%20federation%20trust%20by%20following%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fprevious-versions%2Fexchange-server%2Fexchange-160%2Fmt779252(v%3Dexchg.160)%3Fredirectedfrom%3DMSDN%23ReplaceExpired%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ethese%20steps%3C%2FA%3E.%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EIf%20the%20federation%20trust%20certificate%20is%20not%20found%20on%20any%20of%20the%20servers%2C%20then%20proceed%20with%20resolution%20from%20the%20next%20error.%3C%2FP%3E%0A%3CP%3EAs%20an%20example%2C%20from%20one%20HCW%20log%2C%20there%20seems%20to%20be%20this%20federation%20trust%20certificate%20expired%20on%2013%2F05%2F2019%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EOrgCertificate%3D%5BSubject%5D%20CN%3DFederation%20%5BIssuer%5D%20CN%3DFederation%20%5BSerial%20Number%5D%204E91XXXXXXXXXXXXXXXXXXXXXXXXXXXX%20%5BNot%20Before%5D%205%2F13%2F2014%2011%3A21%3A36%20AM%20%5BNot%20After%5D%205%2F13%2F2019%2011%3A21%3A36%20AM%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CFONT%20size%3D%225%22%3E%3CSPAN%3E7.%20Federation%20trust%20fails%20with%20%E2%80%9CFederation%20Certificate%20cannot%20be%20found%E2%80%9D%3C%2FSPAN%3E%3C%2FFONT%3E%3CBR%20%2F%3E%3CSPAN%3EError%20from%20HCW%20log%3A%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3E2019.11.22%2014%3A40%3A32.569%20*ERROR*%2010277%20%5BClient%3DUX%2C%20Activity%3DDomain%20Ownership%2C%20Session%3DOnPremises%2C%20Cmdlet%3DGet-FederatedDomainProof%2C%20Thread%3D6%5D%20FINISH%20Time%3D125.1ms%20Results%3DPowerShell%20failed%20to%20invoke%20Get-FederatedDomainProof%3A%20Federation%20certificate%20with%20the%20thumbprint%20%E2%80%9803650FFAF05E83E3B007DF3473CB5753F5C4459%E2%80%99cannot%20be%20found.%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%3CSTRONG%3EResolution%3A%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFollow%20the%20procedure%20%3CA%20href%3D%22https%3A%2F%2Fsupport.microsoft.com%2Fhelp%2F3215261%2Ffederation-certificate-with-the-thumbprint-cannot-be-found-error-when%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehere%3C%2FA%3E%20to%20manually%20cleanup%20the%20federation%20trust%20from%20AD.%20Once%20this%20is%20done%2C%20re-run%20HCW%20to%20re-create%20it%20automatically.%3C%2FP%3E%0A%3CP%3EHopefully%2C%20this%20helps%20with%20troubleshooting%20those%20errors!%20I%20wanted%20to%20thank%20Raymond%20Fong%20and%20Nino%20Bilic%20for%20their%20review%20of%20this%20post.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22author%22%3EMirela%20Buruiana%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1144285%22%20slang%3D%22en-US%22%3E%3CP%3EFederation%20trust%20is%20a%20mandatory%20step%20in%20the%20on-premises%20Exchange%20organizations%20when%20configuring%20Full%20hybrid%20deployments%2C%20as%20this%20allows%20us%20to%20create%20organization%20relationships...%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1144285%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EHybrid%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EOn%20Premises%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ETips%20'n%20Tricks%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ETroubleshooting%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1472427%22%20slang%3D%22en-US%22%3ERe%3A%20How%20to%20address%20Federation%20Trust%20issues%20in%20Hybrid%20Configuration%20Wizard%20(HCW)%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1472427%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20is%20great%2C%20but%20the%20struggle%20I'm%20having%20is%20that%20may%20exchange%202010%20server%20wants%20to%20use%20an%20old%20proxy%20server%20to%20access%20the%20internet.%26nbsp%3B%20Here's%20what%20I've%20tried%3A%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3E1.%20I%20made%20sure%20it%20was%20removed%20from%20internet%20options%3C%2FP%3E%3CP%3E2.%20Run%20the%20netsh%20proxy%20reset%20command%26nbsp%3B%20(It%20shows%20direct%20connection)%3C%2FP%3E%3CP%3E3.%20Removed%20it%20from%20the%20HKLU%20registry%20for%20all%20users%20on%20the%20box%3C%2FP%3E%3CP%3E4.%20Searched%20all%20exchange%2FIIS%20.config%20files%20for%20any%20trace%20of%20the%20proxy%20server%20name.%3C%2FP%3E%3CP%3E5.%20Browsed%20to%20the%20nexus%20federation%20metata%20file%20without%20an%20issue.%3C%2FP%3E%3CP%3E6.%20Rebooted%20the%20server%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAny%20other%20thoughts%3F%26nbsp%3B%20When%20I%20run%20the%20command%3A%3C%2FP%3E%3CP%3ENew-FederationTrust%20-Name%20'Microsoft%20Federation%20Gateway'%20-Thumbprint%201E60F7D21795D75F0CC51CA22644251BFD4D1CDA%20-Verbose%2C%20here's%20what%20I%20get%3A%3C%2FP%3E%3CP%3EVERBOSE%3A%20%5B21%3A08%3A01.167%20GMT%5D%20New-FederationTrust%20%3A%20Active%20Directory%20session%20settings%20for%20'New-FederationTrust'%20are%3A%20View%20Entire%20Forest%3A%20'False'%2C%20Default%20Scope%3A%20'mydomain.local'%2C%20Configuration%20Domain%20Controller%3A%20'mydc.mydomain.local'%2C%20Preferred%20Global%3CBR%20%2F%3ECatalog%3A%20'mydc.mydomain.local'%2C%20Preferred%20Domain%20Controllers%3A%20'%7B%20mydc.mydomain.local%7D'%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A01.167%20GMT%5D%20New-FederationTrust%20%3A%20Runspace%20context%3A%20Executing%20user%3A%20mydomain.local%2FUsers%20and%20Groups%2FNetwork%20Admins%2FAdmin%2C%20Executing%20user%20organization%3A%20%2C%20Current%20organization%3A%20%2C%20RBAC-enabled%3A%20Enabled.%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A01.167%20GMT%5D%20New-FederationTrust%20%3A%20Beginning%20processing%20%26amp%3B%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A01.167%20GMT%5D%20New-FederationTrust%20%3A%20Instantiating%20handler%20with%20index%200%20for%20cmdlet%20extension%20agent%20%22Admin%20Audit%20Log%20Agent%22.%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A01.183%20GMT%5D%20New-FederationTrust%20%3A%20Current%20ScopeSet%20is%3A%20%7B%20Recipient%20Read%20Scope%3A%20%7B%7B%2C%20%7D%7D%2C%20Recipient%20Write%20Scopes%3A%20%7B%7B%2C%20%7D%7D%2C%20Configuration%20Read%20Scope%3A%20%7B%7B%2C%20%7D%7D%2C%20Configuration%20Write%20Scope(s)%3A%20%7B%7B%2C%20%7D%2C%20%7D%2C%20Exclusive%20Recipient%20Scope(s)%3A%3CBR%20%2F%3E%7B%7D%2C%20Exclusive%20Configuration%20Scope(s)%3A%20%7B%7D%20%7D%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A01.183%20GMT%5D%20New-FederationTrust%20%3A%20Processing%20object%20%22Microsoft%20Federation%20Gateway%22.%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A01.183%20GMT%5D%20New-FederationTrust%20%3A%20Searching%20the%20local%20certificate%20store%20for%20a%20certificate%20with%20thumbprint%20%221E60F7D21795D75F0CC51CA22644251BFD4D1CDA%22.%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A01.198%20GMT%5D%20New-FederationTrust%20%3A%20Admin%20Audit%20Log%3A%20Entered%20Handler%3AValidate.%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A01.198%20GMT%5D%20New-FederationTrust%20%3A%20Admin%20Audit%20Log%3A%20Exited%20Handler%3AValidate.%3CBR%20%2F%3EVERBOSE%3A%20Creating%20new%20Federation%20Trust%20%22Microsoft%20Federation%20Gateway%22%20for%20federation%20partner%20%22LiveId%22.%20Federation%20certificate%20has%20thumbprint%20%221E60F7D21795D75F0CC51CA22644251BFD4D1CDA%22.%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A01.198%20GMT%5D%20New-FederationTrust%20%3A%20Resolved%20current%20organization%3A%20.%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A01.198%20GMT%5D%20New-FederationTrust%20%3A%20Requesting%20Federation%20Metadata%20from%20%3CA%20href%3D%22https%3A%2F%2Fnexus.passport.com%2FFederationMetadata%2F2006-12%2FFederationMetadata.xml%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fnexus.passport.com%2FFederationMetadata%2F2006-12%2FFederationMetadata.xml%3C%2FA%3E.%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A22.212%20GMT%5D%20New-FederationTrust%20%3A%20Failed%20to%20retrieve%20Federation%20Metadata%20from%20the%20Microsoft%20Federation%20Gateway.%20This%20operation%20will%20be%20retried%20in%20a%20few%20seconds.%20Last%20error%3A%20System.Net.WebException%3A%20Unable%20to%20connect%20to%20the%3CBR%20%2F%3Eremote%20server%20---%26gt%3B%20System.Net.Sockets.SocketException%3A%20A%20connection%20attempt%20failed%20because%20the%20connected%20party%20did%20not%20properly%20respond%20after%20a%20period%20of%20time%2C%20or%20established%20connection%20failed%20because%20connected%20host%20has%20failed%20to%20respond%3CBR%20%2F%3E10.50.10.50%3A1050%3CBR%20%2F%3Eat%20System.Net.Sockets.Socket.DoConnect(EndPoint%20endPointSnapshot%2C%20SocketAddress%20socketAddress)%3CBR%20%2F%3Eat%20System.Net.ServicePoint.ConnectSocketInternal(Boolean%20connectFailure%2C%20Socket%20s4%2C%20Socket%20s6%2C%20Socket%26amp%3B%20socket%2C%20IPAddress%26amp%3B%20address%2C%20ConnectSocketState%20state%2C%20IAsyncResult%20asyncResult%2C%20Int32%20timeout%2C%20Exception%26amp%3B%20exception)%3CBR%20%2F%3E---%20End%20of%20inner%20exception%20stack%20trace%20---%3CBR%20%2F%3Eat%20System.Net.HttpWebRequest.GetResponse()%3CBR%20%2F%3Eat%20Microsoft.Exchange.Management.FederationProvisioning.PartnerFederationMetadata.GetFederationMetadataXPathDocument(Uri%20partnerFederationMetadataEpr).%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A08%3A48.248%20GMT%5D%20New-FederationTrust%20%3A%20Failed%20to%20retrieve%20Federation%20Metadata%20from%20the%20Microsoft%20Federation%20Gateway.%20This%20operation%20will%20be%20retried%20in%20a%20few%20seconds.%20Last%20error%3A%20System.Net.WebException%3A%20Unable%20to%20connect%20to%20the%3CBR%20%2F%3Eremote%20server%20---%26gt%3B%20System.Net.Sockets.SocketException%3A%20A%20connection%20attempt%20failed%20because%20the%20connected%20party%20did%20not%20properly%20respond%20after%20a%20period%20of%20time%2C%20or%20established%20connection%20failed%20because%20connected%20host%20has%20failed%20to%20respond%3CBR%20%2F%3E10.50.10.50%3A1050%3CBR%20%2F%3Eat%20System.Net.Sockets.Socket.DoConnect(EndPoint%20endPointSnapshot%2C%20SocketAddress%20socketAddress)%3CBR%20%2F%3Eat%20System.Net.ServicePoint.ConnectSocketInternal(Boolean%20connectFailure%2C%20Socket%20s4%2C%20Socket%20s6%2C%20Socket%26amp%3B%20socket%2C%20IPAddress%26amp%3B%20address%2C%20ConnectSocketState%20state%2C%20IAsyncResult%20asyncResult%2C%20Int32%20timeout%2C%20Exception%26amp%3B%20exception)%3CBR%20%2F%3E---%20End%20of%20inner%20exception%20stack%20trace%20---%3CBR%20%2F%3Eat%20System.Net.HttpWebRequest.GetResponse()%3CBR%20%2F%3Eat%20Microsoft.Exchange.Management.FederationProvisioning.PartnerFederationMetadata.GetFederationMetadataXPathDocument(Uri%20partnerFederationMetadataEpr).%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A09%3A14.269%20GMT%5D%20New-FederationTrust%20%3A%20Admin%20Audit%20Log%3A%20Entered%20Handler%3AOnComplete.%3CBR%20%2F%3EVERBOSE%3A%20%5B21%3A09%3A14.285%20GMT%5D%20New-FederationTrust%20%3A%20Admin%20Audit%20Log%3A%20Exited%20Handler%3AOnComplete.%3CBR%20%2F%3EUnable%20to%20access%20the%20Federation%20Metadata%20document%20from%20the%20federation%20partner.%20Detailed%20information%3A%20%22Unable%20to%20connect%20to%20the%20remote%20server%22.%3CBR%20%2F%3E%2B%20CategoryInfo%20%3A%20MetadataError%3A%20(%3A)%20%5BNew-FederationTrust%5D%2C%20FederationMetadataException%3CBR%20%2F%3E%2B%20FullyQualifiedErrorId%20%3A%20B77AC03F%2CMicrosoft.Exchange.Management.SystemConfigurationTasks.NewFederationTrust%3C%2FP%3E%3CP%3EVERBOSE%3A%20%5B21%3A09%3A14.285%20GMT%5D%20New-FederationTrust%20%3A%20Ending%20processing%20%26amp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EPlease%20note%3A%20We%20do%20not%20have%20a%20proxy%20server.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWhat%20service%20actually%20handles%20the%20call%20to%20the%20federation%3F%26nbsp%3B%20(Hoping%20to%20dive%20deeper%20with%20process%20explorer).%26nbsp%3B%20Any%20other%20thoughts%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E

During my day to day work as a part of support organization, I work with and help troubleshoot Hybrid Configuration Wizard (HCW) failures. One of the more common causes of HCW failures is the Federation Trust step for the Exchange on-premises organizations in Full hybrid configurations (Classic or Modern topologies).

Federation trust is a mandatory step in the on-premises Exchange organizations when configuring Full hybrid deployments, as this allows us to create organization relationships (for features like hybrid free/busy or OWA/EAS redirection) and sharing policies (1:1 hybrid calendar sharing). In Exchange Online multi-tenant organizations, federation trust is already in place.

Below is an illustration of an Exchange hybrid deployment where both the Exchange on-premises organization and the Exchange Online organization have a trust with Azure Authentication System (formerly called Microsoft Federation Gateway):

HFB1

More info on federation trust can be found here.

Before getting to our subject, let’s quickly go over different hybrid configurations and Hybrid Configuration Wizard (HCW) - as this is the supported tool to configure hybrid deployments.

There are 2 flavors of hybrid configurations:

At this time, each of those supports the following hybrid modes:

  1. Full
  2. Minimal (which further breaks down into…)
    1. Express (a one-time sync)
    2. “Actual minimal”

A quick overview of Full / Minimal / Express options, can be found here. More info on HCW is here.

As mentioned earlier, a federation trust is created by HCW only in Full Hybrid.

HCW logs are located at %appdata%\Microsoft\Exchange Hybrid Configuration on the machine from where HCW was ran. The easiest way to get to them is to press F12 in the HCW window to open the Diagnostic tools and from there you can Open Folder Logging or Open Log File directly.

When you have issues with federation trust, the log will usually show errors when one of the following cmdlets are executed: Set-FederationOrganizationIdentifier or Add-FederatedDomain (but can be other cmdlets as well).

Once you identified the exact cmdlet failing and where (Session=OnPremises – means Exchange Management Shell and Session=Tenant means Exchange Online PowerShell), you should copy-paste the failing command and try to execute it manually and see if that is failing as well (most likely it will). You can also open the shells from F12 Diagnostic tools windows in HCW.

In order to get more details on the error and to rule out this is not an issue with HCW itself, you will need to separately run the same command that threw exception in HCW log and add Verbose switch to get verbose details of the error and the serialized remote exception.

For example, if the Exchange server version is Exchange 2010, you will run the failing command with Verbose switch in Exchange Management Shell (EMS), see if that fails and then get the serialized remote exception.

Example:

 

start-transcript
Set-FederatedOrganizationIdentifier -AccountNamespace <contoso.com> -DelegationFederationTrust 'Microsoft Federation Gateway' -Enabled:$true -VERBOSE
$Error[0].Exception |fl -f
$Error[0].Exception.SerializedRemoteException |fl –f
Get-FederatedOrganizationIdentifier |FL
Get-FederationTrust |FL
stop-transcript

 

If the Exchange Server version is Exchange 2013/2016 and the above commands didn’t show more details on the error, we can also try the following:

  1. Open regular Windows PowerShell (blue background) on the Exchange Server 2013/2016
  2. Run command: add-pssnapin *exchange*
  3. Run command that gave error in HCW and add a Verbose switch

Example:

 

start-transcript
Set-FederatedOrganizationIdentifier -AccountNamespace <contoso.com> -DelegationFederationTrust 'Microsoft Federation Gateway' -Enabled:$true -DefaultDomain $null -VERBOSE 
$Error[0].Exception |fl -f
$Error[0].Exception.SerializedRemoteException |fl –f
Get-FederatedOrganizationIdentifier |FL
Get-FederationTrust |FL
stop-transcript

 

Once you've gathered the verbose error / serialized exception, try to understand where it is failing (or provide it to Microsoft Support together with the HCW log).

We have gathered some common federation trust errors and some tips to fix them:

1. Federation trust fails with "Object reference not set to an instance of an object"

This is a known old issue on Exchange 2016 CU7 servers, make sure your Exchange servers are updated to the latest CU.

Full error in the HCW log:

 

2017.10.06 01:45:56.562 *ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=21] FINISH Time=398.4ms Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Object reference not set to an instance of an object. An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object.
2017.10.06 01:45:56.563 *ERROR* 10224 [Client=UX, Page=DomainProof, Thread=21] Microsoft.Online.CSE.Hybrid.PowerShell.PowerShellInvokeException: PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Object reference not set to an instance of an object. An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object. ---> System.Management.Automation.RemoteException: Object reference not set to an instance of an object.

 

Resolution: Install the latest CU for Exchange 2016

2. Federation fails with "Proof of domain ownership has failed"

Full error in the HCW log:

 

2019.07.16 17:53:14.750 10276 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Add-FederatedDomain, Thread=19] START Add-FederatedDomain -DomainName 'contoso.com'
2019.07.06 17:53:15.375 10177 [Client=UX, Activity=Domain Ownership, Provider=OnPremises, Thread=19] PowerShell Error Record: {CategoryInfo={Activity=Add-FederatedDomain,Category=InvalidResult,Reason=DomainProofOwnershipException,TargetName=,TargetType=},ErrorDetails=,Exception=Proof of domain ownership has failed. Make sure that the TXT record for the specified domain is available in DNS. The format of the TXT record should be "example.com IN TXT hash-value" where "example.com" is the domain you want to configure for Federation and "hash-value" is the proof value generated with "Get-FederatedDomainProof -DomainName example.com".,FullyQualifiedErrorId=367408EF,Microsoft.Exchange.Management.SystemConfigurationTasks.AddFederatedDomain}

 

Resolution:

  • Check the TXT record for your domain(s) in HCW log or in Exchange Management Shell with command Get-FederatedDomainProof -DomainName <CONTOSO.COM>
  • See if it matches your published TXT record with either nslookup utility or by checking internet websites like https://digwebinterface.com/ put your domain in hostnames, type=txt, Nameservers - Authoritative

You would look for errors, missing records or unusual formatting (characters, spaces, quotes, TXT record split in half).

3. Federation fails with "An unexpected error occurred on a receive" or "An unexpected error occurred on a send."

Error in the HCW log:

 

2018.10.10 17:03:31.277 *ERROR* [Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier] FINISH Time=64.3s Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': An error occurred while attempting to provision Exchange to the Partner STS. Detailed Information "An error occurred accessing Windows Live. Detailed information: "The underlying connection was closed: An unexpected error occurred on a receive.".".

 

Verbose log shows something like this:

 

Set-FederatedOrganizationIdentifier -AccountNamespace 'domain.com' -DelegationFederationTrust 'Microsoft Federation Gateway' -Enabled: $true -DefaultDomain $null -Verbose VERBOSE: [12:29:07.754 GMT] Set-FederatedOrganizationIdentifier : Calling 'CreateAppId(uri='FYDIBOHF25SPDLT.domain.com',properties=[0])' at the domain services endpoint https://domains.live.com/service/managedelegation2.asmx . VERBOSE: [12:29:08.535 GMT] Set-FederatedOrganizationIdentifier : The request to Windows Live Domain Services failed with the following exception: 
[0]: Microsoft.Exchange.Management.FederationProvisioning.LiveDomainServicesException An error occurred accessing Windows Live. Detailed information: "The underlying connection was closed: An unexpected error occurred on a send.".
[1]: System.Net.WebException The underlying connection was closed: An unexpected error occurred on a send.
[2]: System.IO.IOException Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
[3]: System.Net.Sockets.SocketException An existing connection was forcibly closed by the remote host

 

Resolution:

Check outbound access from all your Exchange Servers to Microsoft Federation Gateway by browsing using Internet Explorer with PSEXEC tool (with -s and -i switches) from the Exchange Server (this will use Internet Explorer under System Account / Exchange Server Account). Example of cmdlet:

…\Downloads\PSTools> PsExec.exe -i -s "c:\Program Files\Internet Explorer\iexplore.exe"

In this example, “Windows Live” is actually this exact URL: https://domains.live.com/service/managedelegation2.asmx

From on-premises Exchange to Office 365, the Exchange 2010 MBX & CAS or 2013 MBX (backend) or 2016 / 2019 would need outbound Internet access to the Microsoft Federation Gateway in addition to https://outlook.office365.com/ews/exchange.asmx

Verify the machine/system account can access these Microsoft Federation Gateway URLs:

For a complete list of O365 URL & IP addresses, see these articles:

Note: If the Exchange requires a proxy server to access the Internet, specify the proxy server using "Set-ExchangeServer myExchange01 -InternetWebProxy http://myproxy:80". Notice such proxy can't require any user authentication for outbound Internet access, and the proxy must start with HTTP: and not HTTPS: (secure SSL).

You can also set the proxy using netsh as well.

set proxy proxy-server="http=myproxy;https=sproxy:88" bypass-list="*.contoso.com" 

In rare instances, you can use the machine/system account to access the URLs from the browser, but Exchange cmdlets still failed with "Could not establish trust relationship for the SSL/TLS secure channel." If that happens, make sure the certificate authorities for the urls are installed at the Third-Party Root Certification Authorities of the machine local certificate location.

Reference:

Netsh Commands for Windows Hypertext Transfer Protocol (WINHTTP)

Firewall Considerations for Federated Delegation Federated delegation features require that the Mailbox and Client Access servers in your organization have outbound access to the Internet by using HTTPS. You must allow outbound HTTPS access (port 443 for TCP) from all Exchange 2010 Mailbox and Client Access servers in the organization.

4. There is no specific error / exception, in HCW log you would see it stops without any specific error.

From the HCW log:

 

2019.02.14 12:56:21.658 [Activity=Domain Ownership, Session=OnPremises, Cmdlet=Get-FederatedOrganizationIdentifier] FINISH Time=133.0ms Results=1 FederatedOrganizationIdWithDomainStatus {AccountNamespace='FYDIBOHF25SPDLT.contoso.com' DelegationTrustLink='contoso.local/Configuration/Deleted Objects/Microsoft Federation Gateway DEL:8e834abf-5154-4540-a3c6-5a5c614c6a06'Enabled=1 ExchangeVersion='0.10 (14.0.100.0)' Guid=2e1da884-9686-4221-8098-d34ced5a2f85 Id='Federation' Identity='Federation' IsValid=1 Name='Federation' ObjectState='Unchanged' WhenChanged='8/11/2015 5:35:58 PM' WhenChangedUTC='8/11/2015 2:35:58 PM' WhenCreated='10/18/2009 10:30:09 AM' WhenCreatedUTC='10/18/2009 6:30:09 AM'}
2019.02.14 12:56:21.677 [Client=UX, Page=DomainProof] Unproven Domains: 1

 

Resolution:

Look for orphaned federation trust in Get-FederatedOrganizationIdentifier | FL or in HCW log if you see something with "DEL": "contoso.com/Configuration/Deleted Objects/Microsoft Federation Gateway/DEL: <xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx>". Solution is to remove the orphaned federation trust and re-run HCW.

Reference here.

Note: as a first step, you can try to run the command remove-federateddomain with the switch -Force. Also, you don't need to recreate federation trust manually, just re-run HCW (this will recreate federation trust for us)

5. Federation Trust fails with "InternalError InternalError: Internal error.".".""

Error from the HCW log:

 

2019.08.23 07:45:22.914         10276 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=20] START Set-FederatedOrganizationIdentifier -AccountNamespace 'contoso.com' -DelegationFederationTrust 'Microsoft Federation Gateway' -Enabled: $true -DefaultDomain $null
2019.08.23 07:45:23.239 *ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=20] FINISH Time=325.0ms Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': An error occurred while attempting to provision Exchange to the Partner STS.  Detailed Information "An unexpected result was received from Windows Live.  Detailed information: "InternalError InternalError: Internal error.".". {CategoryInfo={Activity=[System.String] Set-FederatedOrganizationIdentifier,Category=[System.Management.Automation.ErrorCategory] InvalidResult,Reason=[System.String]

 

Resolution:

Open request with Microsoft Support or check if any Service Incident is published. Please see this.

6. Federation trust fails with "1007 Access Denied"

Error from the HCW log:

 

Set-FederatedOrganizationIdentifier,Category=[System.Management.Automation.ErrorCategory] InvalidResult,Reason=[System.String] ProvisioningFederatedExchangeException,TargetName=[System.String] ,TargetType=[System.String] },ErrorDetails=,Exception=[System.Management.Automation.RemoteException] An error occurred while attempting to provision Exchange to the Partner STS.  Detailed Information "An unexpected result was received from Windows Live.  Detailed information: "1007 AccessDenied: Access Denied."."

 

Resolution:

"1007 Access Denied" error is usually when we have issues with:

  1. Windows Time on the Exchange Server. See this article or this article.
  2. Outdated federation trust (for example, federation trust certificate expired) and in this case you would remove federation trust by following these steps.

If the federation trust certificate is not found on any of the servers, then proceed with resolution from the next error.

As an example, from one HCW log, there seems to be this federation trust certificate expired on 13/05/2019:

 

OrgCertificate=[Subject] CN=Federation [Issuer] CN=Federation [Serial Number] 4E91XXXXXXXXXXXXXXXXXXXXXXXXXXXX [Not Before] 5/13/2014 11:21:36 AM [Not After] 5/13/2019 11:21:36 AM

 

7. Federation trust fails with “Federation Certificate cannot be found”
Error from HCW log:

 

2019.11.22 14:40:32.569 *ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Get-FederatedDomainProof, Thread=6] FINISH Time=125.1ms Results=PowerShell failed to invoke Get-FederatedDomainProof: Federation certificate with the thumbprint ‘03650FFAF05E83E3B007DF3473CB5753F5C4459’cannot be found.

 

Resolution:

Follow the procedure here to manually cleanup the federation trust from AD. Once this is done, re-run HCW to re-create it automatically.

Hopefully, this helps with troubleshooting those errors! I wanted to thank Raymond Fong and Nino Bilic for their review of this post.

Mirela Buruiana

2 Comments
Occasional Visitor

This is great, but the struggle I'm having is that may exchange 2010 server wants to use an old proxy server to access the internet.  Here's what I've tried:  

1. I made sure it was removed from internet options

2. Run the netsh proxy reset command  (It shows direct connection)

3. Removed it from the HKLU registry for all users on the box

4. Searched all exchange/IIS .config files for any trace of the proxy server name.

5. Browsed to the nexus federation metata file without an issue.

6. Rebooted the server

 

Any other thoughts?  When I run the command:

New-FederationTrust -Name 'Microsoft Federation Gateway' -Thumbprint 1E60F7D21795D75F0CC51CA22644251BFD4D1CDA -Verbose, here's what I get:

VERBOSE: [21:08:01.167 GMT] New-FederationTrust : Active Directory session settings for 'New-FederationTrust' are: View Entire Forest: 'False', Default Scope: 'mydomain.local', Configuration Domain Controller: 'mydc.mydomain.local', Preferred Global
Catalog: 'mydc.mydomain.local', Preferred Domain Controllers: '{ mydc.mydomain.local}'
VERBOSE: [21:08:01.167 GMT] New-FederationTrust : Runspace context: Executing user: mydomain.local/Users and Groups/Network Admins/Admin, Executing user organization: , Current organization: , RBAC-enabled: Enabled.
VERBOSE: [21:08:01.167 GMT] New-FederationTrust : Beginning processing &
VERBOSE: [21:08:01.167 GMT] New-FederationTrust : Instantiating handler with index 0 for cmdlet extension agent "Admin Audit Log Agent".
VERBOSE: [21:08:01.183 GMT] New-FederationTrust : Current ScopeSet is: { Recipient Read Scope: {{, }}, Recipient Write Scopes: {{, }}, Configuration Read Scope: {{, }}, Configuration Write Scope(s): {{, }, }, Exclusive Recipient Scope(s):
{}, Exclusive Configuration Scope(s): {} }
VERBOSE: [21:08:01.183 GMT] New-FederationTrust : Processing object "Microsoft Federation Gateway".
VERBOSE: [21:08:01.183 GMT] New-FederationTrust : Searching the local certificate store for a certificate with thumbprint "1E60F7D21795D75F0CC51CA22644251BFD4D1CDA".
VERBOSE: [21:08:01.198 GMT] New-FederationTrust : Admin Audit Log: Entered Handler:Validate.
VERBOSE: [21:08:01.198 GMT] New-FederationTrust : Admin Audit Log: Exited Handler:Validate.
VERBOSE: Creating new Federation Trust "Microsoft Federation Gateway" for federation partner "LiveId". Federation certificate has thumbprint "1E60F7D21795D75F0CC51CA22644251BFD4D1CDA".
VERBOSE: [21:08:01.198 GMT] New-FederationTrust : Resolved current organization: .
VERBOSE: [21:08:01.198 GMT] New-FederationTrust : Requesting Federation Metadata from https://nexus.passport.com/FederationMetadata/2006-12/FederationMetadata.xml.
VERBOSE: [21:08:22.212 GMT] New-FederationTrust : Failed to retrieve Federation Metadata from the Microsoft Federation Gateway. This operation will be retried in a few seconds. Last error: System.Net.WebException: Unable to connect to the
remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
10.50.10.50:1050
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.Exchange.Management.FederationProvisioning.PartnerFederationMetadata.GetFederationMetadataXPathDocument(Uri partnerFederationMetadataEpr).
VERBOSE: [21:08:48.248 GMT] New-FederationTrust : Failed to retrieve Federation Metadata from the Microsoft Federation Gateway. This operation will be retried in a few seconds. Last error: System.Net.WebException: Unable to connect to the
remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
10.50.10.50:1050
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.Exchange.Management.FederationProvisioning.PartnerFederationMetadata.GetFederationMetadataXPathDocument(Uri partnerFederationMetadataEpr).
VERBOSE: [21:09:14.269 GMT] New-FederationTrust : Admin Audit Log: Entered Handler:OnComplete.
VERBOSE: [21:09:14.285 GMT] New-FederationTrust : Admin Audit Log: Exited Handler:OnComplete.
Unable to access the Federation Metadata document from the federation partner. Detailed information: "Unable to connect to the remote server".
+ CategoryInfo : MetadataError: (:) [New-FederationTrust], FederationMetadataException
+ FullyQualifiedErrorId : B77AC03F,Microsoft.Exchange.Management.SystemConfigurationTasks.NewFederationTrust

VERBOSE: [21:09:14.285 GMT] New-FederationTrust : Ending processing &

 

Please note: We do not have a proxy server.

 

What service actually handles the call to the federation?  (Hoping to dive deeper with process explorer).  Any other thoughts?

 

 

Microsoft

@johndennis250 , I am sorry, I didn't see the comment and I wasn't notified about it. I really hope you managed to find the issue / old proxy.

These would have been my suggestions:

  1. Run this Exchange Management Shell on-premises: Get-ExchangeServer | FL identity, admindisplayversion, serverrole, *proxy*
  2. Run in CMD netsh winhttp show proxy
  3. Download PSExec tool on the Exchange server(s) – start with the server that threw the error  and run the following command PsExec.exe -i -s "c:\Program Files\Internet Explorer\iexplore.exe" to launch IE under System Account.

Now, in IE settings, check if any proxy is set there or if you have “Automatically Detect Settings” Enabled for the Local System Account combined with PAC file.
From on-premises Exchange to Office 365, the Exchange 2010 MBX & CAS or 2013 MBX (backend) or 2016 / 2019 would need outbound Internet access to the Microsoft Federation Gateway in addition to https://outlook.office365.com/ews/exchange.asmx

 

Verify the machine/system account can access these Microsoft Federation Gateway URLs:

https://nexus.microsoftonline-p.com/federationmetadata/2006-12/federationmetadata.xml  [<-- You should see an xml page.]

https://login.microsoftonline.com/extSTS.srf   [<-- You should see “Sorry, but we’re having trouble signing you in”.]

https://domains.live.com/service/managedelegation2.asmx   [<-- You should see the operations supported by ManageDelegation2.]


4. On the Exchange server(s) open regedit and see if a proxy is set there: HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ by looking at ProxyEnable if 1 and a ProxyServer is set.