Home
%3CLINGO-SUB%20id%3D%22lingo-sub-953945%22%20slang%3D%22en-US%22%3ERe%3A%20Troubleshooting%20Hybrid%20Migration%20Endpoints%20in%20Classic%20and%20Modern%20Hybrid%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-953945%22%20slang%3D%22en-US%22%3E%3CP%3EAwesome%20article%2C%20as%20usual.%20Love%20the%20attention%20to%20detail.%20Someone%20give%20this%20gal%20a%20cookie!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-953988%22%20slang%3D%22en-US%22%3ERe%3A%20Troubleshooting%20Hybrid%20Migration%20Endpoints%20in%20Classic%20and%20Modern%20Hybrid%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-953988%22%20slang%3D%22en-US%22%3E%3CP%3EThanks%2C%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F58%22%20target%3D%22_blank%22%3E%40Vasil%20Michev%3C%2FA%3E%20%2C%20cookie%20sounds%20good.%20I%20like%20beer%20%2Fwine%20too%26nbsp%3B%3CIMG%20class%3D%22lia-deferred-image%20lia-image-emoji%22%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Fhtml%2Fimages%2Femoticons%2Fsmile_40x40.gif%22%20alt%3D%22%3Asmile%3A%22%20title%3D%22%3Asmile%3A%22%20%2F%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-958202%22%20slang%3D%22en-US%22%3ERe%3A%20Troubleshooting%20Hybrid%20Migration%20Endpoints%20in%20Classic%20and%20Modern%20Hybrid%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-958202%22%20slang%3D%22en-US%22%3E%3CP%3EHello%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EEverything%20is%20working%20fine%20on%20my%20hybrid%20environment%20except%20the%26nbsp%3B%3CSTRONG%3EHybrid%20remote%20move%20endpoint%20with%20Autodiscover%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3EDo%20you%20know%20what%20to%20verify%20and%20how%20the%20best%20way%20to%20debug%20%3F%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EMessage%20%3A%20AutoDiscover%20failed%20with%20a%20configuration%20error%3A%20The%20migration%20service%20failed%20to%20detect%20the%20migration%20endpoint%20using%20the%20Autodiscover%3CBR%20%2F%3Eservice.%20Consider%20using%20the%20Exchange%20Remote%20Connectivity%20Analyzer%20(%3CA%20href%3D%22https%3A%2F%2Ftestexchangeconnectivity.com%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Ftestexchangeconnectivity.com%3C%2FA%3E)%20to%20diagnose%20the%20connectivity%20issues.%3CBR%20%2F%3ESupportsCutover%20%3A%20False%3CBR%20%2F%3EErrorDetail%20%3A%20internal%20error%3AMicrosoft.Exchange.Migration.AutoDiscoverFailedConfigurationErrorException%3A%20AutoDiscover%20failed%20with%20a%20configuration%20error%3A%3CBR%20%2F%3EThe%20migration%20service%20failed%20to%20detect%20the%20migration%20endpoint%20using%20the%20Autodiscover%20service.%20Consider%20using%20the%20Exchange%20Remote%3CBR%20%2F%3EConnectivity%20Analyzer%20(%3CA%20href%3D%22https%3A%2F%2Ftestexchangeconnectivity.com%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Ftestexchangeconnectivity.com%3C%2FA%3E)%20to%20diagnose%20the%20connectivity%20issues.%3CBR%20%2F%3Eat%20Microsoft.Exchange.Migration.DataAccessLayer.MigrationEndpointBase.InitializeFromAutoDiscover(MigrationEndpoint%20presentationObject%2C%3CBR%20%2F%3ESmtpAddress%20emailAddress%2C%20Boolean%20acceptUntrustedCertificates)%3CBR%20%2F%3Eat%20Microsoft.Exchange.Management.Migration.MigrationService.Endpoint.TestMigrationServerAvailability.InternalProcessExchangeRemoteMoveAuto%3CBR%20%2F%3EDiscover()%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-958207%22%20slang%3D%22en-US%22%3ERe%3A%20Troubleshooting%20Hybrid%20Migration%20Endpoints%20in%20Classic%20and%20Modern%20Hybrid%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-958207%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F346709%22%20target%3D%22_blank%22%3E%40benoitrando%3C%2FA%3E%20%2C%20as%20indicated%20in%20the%20error%20message%2C%20you%20have%20an%20Autodiscover%20issue%2C%20you%20need%20to%20use%20exrca.com%20(%3CA%20style%3D%22background-color%3A%20transparent%3B%20box-sizing%3A%20border-box%3B%20color%3A%20%23146cac%3B%20font-family%3A%20%26amp%3Bquot%3B%20segoeui%26amp%3Bquot%3B%2C%26amp%3Bquot%3Blato%26amp%3Bquot%3B%2C%26amp%3Bquot%3Bhelvetica%20neue%26amp%3Bquot%3B%2Chelvetica%2Carial%2Csans-serif%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20underline%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%20href%3D%22https%3A%2F%2Ftestexchangeconnectivity.com%2F%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Ftestexchangeconnectivity.com%3C%2FA%3E%3CSPAN%20style%3D%22display%3A%20inline%20!important%3B%20float%3A%20none%3B%20background-color%3A%20%23f8f8f8%3B%20color%3A%20%23333333%3B%20font-family%3A%20'SegoeUI'%2C'Lato'%2C'Helvetica%20Neue'%2CHelvetica%2CArial%2Csans-serif%3B%20font-size%3A%2016px%3B%20font-style%3A%20normal%3B%20font-variant%3A%20normal%3B%20font-weight%3A%20300%3B%20letter-spacing%3A%20normal%3B%20orphans%3A%202%3B%20text-align%3A%20left%3B%20text-decoration%3A%20none%3B%20text-indent%3A%200px%3B%20text-transform%3A%20none%3B%20-webkit-text-stroke-width%3A%200px%3B%20white-space%3A%20normal%3B%20word-spacing%3A%200px%3B%22%3E)%3C%2FSPAN%3E%20website%20and%20perform%20Outlook%20Autodiscover%20test%20for%20the%20Email%20Address%20used%20in%20Test-MigrationServerAvailability%20to%20understand%20what%20is%20the%20cause%20of%20the%20Autodiscover%20(maybe%20your%20Exchange%20certificate%20is%20not%20trusted).%20%3CBR%20%2F%3EAlso%2C%20check%20if%20you%20have%20an%20error%20for%20the%20second%20option%3A%26nbsp%3B%20%3CSTRONG%3E2.%20Hybrid%20remote%20move%20endpoint%20without%20Autodiscover%20(testing%20EWS%20directly%2C%20bypassing%20AutoD)%3C%2FSTRONG%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-953006%22%20slang%3D%22en-US%22%3ETroubleshooting%20Hybrid%20Migration%20Endpoints%20in%20Classic%20and%20Modern%20Hybrid%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-953006%22%20slang%3D%22en-US%22%3E%3CP%3EIn%20our%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FExchange-Team-Blog%2FUnderstanding-Hybrid-Migration-Endpoints-in-Classic-and-Modern%2Fba-p%2F916993%22%20target%3D%22_self%22%3Eprevious%20blog%20post%3C%2FA%3E%20we%20covered%20an%20overview%20of%20what%20migration%20endpoints%20are%2C%20how%20to%20find%20them%20and%20what%20makes%20them%20tick.%20In%20this%20post%2C%20we%20will%20cover%20related%20troubleshooting.%20Note%20that%20this%20post%20has%20some%20in-depth%20troubleshooting%20steps%2C%20so%20it%20is%20not%20necessarily%20something%20that%20you%E2%80%99ll%20read%20for%20fun%2C%20but%20we%20wanted%20to%20make%20it%20available%20for%20those%20times%20when%20you%20run%20into%20trouble!%3C%2FP%3E%0A%3CP%3ENow%20let%E2%80%99s%20cover%20some%20troubleshooting!%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId-2046205722%22%20id%3D%22toc-hId-2046205722%22%3E%3CFONT%20size%3D%226%22%3EThings%20that%20are%20commonly%20mis-configured%3C%2FFONT%3E%3C%2FH1%3E%0A%3CP%3EBefore%20getting%20to%20the%20part%20where%20we%20troubleshoot%20specific%20migration%20endpoint%20issues%2C%20be%20aware%20of%20the%20following%20top%20reasons%20why%20a%20migration%20endpoint%20cannot%20be%20created%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Foffice365%2Ftroubleshoot%2Fsecurity%2Fprepare-tls-1.2-in-office-365%23references%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3ETLS1.2%20should%20be%20enabled%20in%20the%20on-premises%20infrastructure%3C%2FA%3E.%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fsupport.microsoft.com%2Fhelp%2F3063913%2Fthe-remote-server-returned-an-error-403-forbidden-error-when-you-try-t%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMRSproxy%20needs%20to%20be%20enabled%20and%20running%20in%20Exchange%20on-premises%3C%2FA%3E.%3C%2FLI%3E%0A%3CLI%3EExchange%20Online%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fexchange%2Fhybrid-deployment-prerequisites%23hybrid-deployment-protocols-ports-and-endpoints%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Erequires%20Negotiate%20(NTLM)%20authentication%20for%20MRSProxy%3C%2FA%3E.%3C%2FLI%3E%0A%3CLI%3EOn-premises%20migration%20Admin%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2FExchange%2Fpermissions%2Ffeature-permissions%2Frecipient-permissions%3Fview%3Dexchserver-2019%23mailbox-move-and-migration-permissions%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eneeds%20to%20have%20the%20minimum%20required%20permissions%3C%2FA%3E%20and%20valid%20credentials.%3C%2FLI%3E%0A%3CLI%3ELoad-balancing%20Exchange%202010%20MRSproxy%20servers%20requires%20%3CA%20href%3D%22https%3A%2F%2Fblogs.msdn.microsoft.com%2Fbrad_hughes%2F2017%2F02%2F21%2Fits-always-the-load-balancer%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EIP%20persistence%3C%2FA%3E%3CSPAN%3E%20(affinity)%3C%2FSPAN%3E.%3C%2FLI%3E%0A%3CLI%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fexchange%2Fconfiguring-ssl-offloading-in-exchange-2013-exchange-2013-help%23configuring-ssl-offloading-for-the-mailbox-replication-proxy-service-mrsproxy%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3ESSL%20offloading%20is%20not%20supported%20for%20MRSProxy%3C%2FA%3E.%3C%2FLI%3E%0A%3CLI%3EFor%20Classic%20hybrid%20%E2%80%93%20where%20we%20require%20inbound%20connectivity%20from%20Exchange%20Online%20to%20on-premises%20Exchange%2C%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Foffice365%2Fenterprise%2Furls-and-ip-address-ranges%23exchange-online%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eallow%20all%20Exchange%20Online%20IP%20addresses%20to%20connect%20to%20on-premises%20EWS%20%2F%20Autodiscover%3C%2FA%3E.%3C%2FLI%3E%0A%3CLI%3EFor%20Classic%20hybrid%2C%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fexchange%2Fhybrid-deployment-prerequisites%23hybrid-deployment-protocols-ports-and-endpoints%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Epre-authentication%20for%20EWS%20%2F%20Autodiscover%20virtual%20directories%20is%20not%20supported%3C%2FA%3E.%3C%2FLI%3E%0A%3CLI%3EFor%20Classic%20hybrid%2C%20a%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fexchange%2Fhybrid-deployment-prerequisites%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Evalid%203%3CSUP%3Erd%3C%2FSUP%3E%20party%20certificate%20is%20required%20for%20EWS%20%2F%20IIS%3C%2FA%3E.%20Also%20see%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fexchange%2Fcertificate-requirements%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ethis%3C%2FA%3E.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CH1%20id%3D%22toc-hId--505951239%22%20id%3D%22toc-hId--505951239%22%3E%3CFONT%20size%3D%226%22%3ETroubleshooting%3C%2FFONT%3E%3C%2FH1%3E%0A%3CP%3EWhether%20you%20are%20having%20trouble%20creating%20new%20migration%20endpoints%20in%20Office%20365%20Exchange%20Online%20or%20are%20not%20able%20to%20migrate%20anymore%20to%20or%20from%20Exchange%20Online%20using%20an%20existing%20migration%20endpoint%2C%20the%20cmdlet%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fpowershell%2Fmodule%2Fexchange%2Fmove-and-migration%2Ftest-migrationserveravailability%3Fview%3Dexchange-ps%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3ETest-MigrationServerAvailability%3C%2FA%3E%20is%20your%20dear%20friend.%20%3CEM%3EAlways%3C%2FEM%3E%20run%20this%20cmdlet%20in%20Exchange%20Online%20PowerShell%2C%20%3CEM%3Enot%3C%2FEM%3E%20from%20on-premises%20Exchange%20Management%20Shell.%3C%2FP%3E%0A%3CP%3EWe%20will%20focus%20on%20hybrid%20migration%20endpoint%20issues%20and%20the%20cmdlet%20syntax%20for%20this%20endpoint%20type.%20Below%20we%20will%20show%20you%203%20commands%20that%20can%20help%20you%20check%20for%20underlying%20issues%20or%20error%20messages%3A%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E1.%20Hybrid%20remote%20move%20endpoint%20with%20Autodiscover%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3ETest-MigrationServerAvailability%20-ExchangeRemoteMove%20-Autodiscover%20-EmailAddress%20user%40contoso.com%20-Credentials%20(get-credential%20contoso%5Cadministrator)%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F149675iF7AA3C064F32E251%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22troublhybmig02.jpg%22%20title%3D%22troublhybmig02.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3ENote%3A%20The%20option%20with%20Autodiscover%20is%20not%20used%20in%20Modern%20hybrid%20as%20we%20go%20directly%20to%20EWS%20server(s)%20for%20both%20Migration%20Endpoints%20and%20Free%2FBusy%20configuration%20(Cloud%20Intra-Organization%20Connectors%20and%20Organization%20Relationships%20have%20TargetSharingEpr%20set%20to%20the%20EWS%20namespace.)%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E2.%20%3C%2FSTRONG%3E%3CSTRONG%3EHybrid%20remote%20move%20endpoint%20without%20Autodiscover%20(testing%20EWS%20directly)%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3ETest-MigrationServerAvailability%20-ExchangeRemoteMove%20%E2%80%93RemoteServer%20mail.contoso.com%20-Credentials(get-credential%20contoso%5Cadministrator)%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F149680i38AF67EB48040D94%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22troublhybmig03.jpg%22%20title%3D%22troublhybmig03.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3ENote%3A%20In%20Modern%20hybrid%2C%20the%20RemoteServer%20is%20in%20the%20format%20of%20'%3CGUID%3E.resource.mailboxmigration.his.msappproxy.net'%2C%20where%20%3CGUID%3E%20is%20unique%20for%20each%20organization.%20This%20is%20randomly%20generated%20and%20stored%20encoded%20base%2064%20in%20the%20OnPremisesOrganization%20object%E2%80%99s%20Comment%20in%20Exchange%20Online%20when%20you%20will%20run%20Modern%20HCW%20and%20it%20gets%20to%20that%20configuration%20part.%20%26nbsp%3BThis%20GUID%20will%20be%20stamped%20on%20the%20Migration%20Endpoint%20in%20the%20RemoteServer%20value%20for%20both%20Full%20and%20Minimal%20Modern%20Hybrid%20Topologies%20and%20also%20on%20the%20TargetSharingEpr%20values%20for%20Cloud%20Intra-Organization%20Connector%20%2F%20Organization%20Relationship%20.%20You%20can%20check%20the%20GUID%20in%20the%20HCW%20log%20and%20on%20Get-MigrationEndpoint%20%2F%20Get-IntraOrganizationConnector%20%2F%20Get-OrganizationRelationship%20EXO%20cmdlets.%3C%2FGUID%3E%3C%2FGUID%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F149691i29A775843F7301D6%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22troublhybmig04.jpg%22%20title%3D%22troublhybmig04.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3ECmdlets%20ran%20above%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EGet-IntraOrganizationConnector%20%7C%20fl%20targetsharingepr%0AGet-OrganizationRelationship%20%7C%20fl%20targetsharingepr%0AGet-MigrationEndpoint%0A%24strdata%20%3D%20(Get-OnPremisesOrganization).comment%0A%24bytes%20%3D%20%5BConvert%5D%3A%3AFromBase64String(%24strdata)%0A%24ms%20%3D%20New-Object%20System.IO.MemoryStream(%40(%2C%24bytes))%0A%24deflate%20%3D%20New-Object%20System.IO.Compression.DeflateStream(%24ms%2C%20%5BSystem.IO.Compression.CompressionMode%5D%3A%3ADecompress)%0A%24reader%20%3D%20New-Object%20System.IO.StreamReader(%24deflate)%0A%24text%20%3D%20%24reader.ReadToEnd()%0A%24text%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E3.%20Testing%20an%20existing%20hybrid%20remote%20move%20endpoint%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3ETest-MigrationServerAvailability%20-Endpoint%20%3CIDENTITY%20of%3D%22%22%20the%3D%22%22%20endpoint%3D%22%22%3E%3C%2FIDENTITY%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F149700i976D9FC10141F024%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22troublhybmig05.jpg%22%20title%3D%22troublhybmig05.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CH1%20id%3D%22toc-hId-1236859096%22%20id%3D%22toc-hId-1236859096%22%3E%3CFONT%20size%3D%226%22%3EOK%2C%20I%20ran%20these%20and%20found%20errors%3B%20now%20what%3F%3C%2FFONT%3E%3C%2FH1%3E%0A%3CP%3ELet%E2%80%99s%20walk%20through%20some%20common%20Test-MigrationServerAvailability%20errors%20and%20how%20to%20troubleshoot%20these%20in%20Classic%20%2F%20Modern%20hybrid.%3CBR%20%2F%3ETip%3A%20Whenever%20a%20command%20returns%20an%20error%20in%20PowerShell%2C%20you%20should%20run%20the%20command%20%24Error%5B0%5D.Exception%20%7Cfl%20-f%20to%20get%20more%20details%20on%20the%20exception%20thrown.%3C%2FP%3E%0A%3CP%3EHowever%2C%20test-migrationserveravailability%20failures%20are%20not%20actually%20those%20red%20errors%20that%20you%20get%20when%20a%20command%20does%20not%20work%20in%20PowerShell.%20So%2C%20in%20this%20situation%2C%20we%20can%20use%20New-MoveRequest%20command%20to%20throw%20the%20same%20error%20and%20get%20the%20serialized%20exception%20from%20this%20one.%3CBR%20%2F%3ETest-MigrationServerAvailability%20which%20returns%20the%20result%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F149705i1DFA1D442253F0C9%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22troublhybmig06.jpg%22%20title%3D%22troublhybmig06.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3ERunning%20the%20New-MoveRequest%20command%20to%20test%20migration%20of%20a%20synced%20user%20will%20give%20me%20same%20error%20message%20but%20in%20%E2%80%9Cred%E2%80%9D%20and%20we%20can%20then%20get%20the%20serialized%20exception%20from%20it.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F149709iDFF0ED35CCE3BF0A%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22troublhybmig07.jpg%22%20title%3D%22troublhybmig07.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EBased%20on%20these%20error%20messages%2C%20we%20gathered%20most%20common%20scenarios.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--1511811370%22%20id%3D%22toc-hId--1511811370%22%3EScenario%201%3A%20Test-MigrationServerAvailability%20fails%20because%20of%20connectivity%20%2F%20timeout%20errors%3C%2FH2%3E%0A%3CP%3EExamples%20of%20some%20of%20those%20errors%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EThe%20call%20to%20'https%3A%2F%2F%3CGUID%3E.resource.mailboxmigration.his.msappproxy.net%2FEWS%2Fmrsproxy.svc'%20timed%20out.%20Error%20details%3A%20The%20request%20channel%20timed%20out%20attempting%20to%20send%20after%2000%3A00%3A00.%20The%20time%20allotted%20to%20this%20operation%20may%20have%20been%20a%20portion%20of%20a%20longer%20timeout.%0A%0AThe%20call%20to%20'%3CA%20href%3D%22https%3A%2F%2Fmail.contoso.com%2FEWS%2Fmrsproxy.svc%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fmail.contoso.com%2FEWS%2Fmrsproxy.svc%3C%2FA%3E'%20timed%20out.%20Error%20details%3A%20The%20operation%20did%20not%20complete%20within%20the%20allotted%20timeout%20of%2000%3A00%3A50.%20The%20time%20allotted%20to%20this%20operation%20may%20have%20been%20a%20portion%20of%20a%20longer%20timeout.%20%0A%0AThe%20request%20channel%20timed%20out%20while%20waiting%20for%20a%20reply%20after%2000%3A00%3A09.9996191.%20The%20time%20allotted%20to%20this%20operation%20may%20have%20been%20a%20portion%20of%20a%20longer%20timeout.%20The%20remote%20server%20returned%20an%20error%3A%20(504)%20Gateway%20Timeout.%3C%2FGUID%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CU%3ETroubleshooting%20these%20timeout%20errors%20in%20Modern%20hybrid%3A%3C%2FU%3E%3C%2FP%3E%0A%3CP%3E%3CBR%20%2F%3EDuring%20the%20Modern%20hybrid%20configuration%2C%20you%20will%20be%20asked%20to%20input%20the%20credentials%20for%20the%20on-premises%20migration%20admin%20%E2%80%93%20these%20can%20be%20the%20same%20credentials%20inserted%20in%20the%20beginning%20of%20HCW%20or%20new%20ones.%20The%20on-premises%20Migration%20admin%20credentials%20are%20needed%20by%20HCW%20in%20order%20to%20run%20the%20Test-MigrationServerAvailability%20cmdlet%20and%20this%20account%20can%20have%20lesser%20privileges%20than%20the%20admin%20account%20running%20HCW%20(which%20is%20Organization%20Management%20role).%20Once%20Modern%20HCW%20has%20tested%20the%20on-premises%20migration%20server%20availability%2C%20we%20will%20know%20if%20we%20are%20able%20to%20create%20the%20migration%20endpoint%20in%20Exchange%20Online%20or%20not%20and%20also%20in%20this%20step%20we%20would%20see%20the%20dynamically%20generated%20GUID%20for%20your%20on-premises%20MRSProxy%20server(s)%3A%20%3CEM%3Ehttps%3A%2F%2F%3CGUID%3E.resource.mailboxmigration.his.msappproxy.net%2FEWS%2Fmrsproxy.svc.%20%3CBR%20%2F%3E%3C%2FGUID%3E%3C%2FEM%3ESupposing%20that%20you%20get%20a%20timeout%20error%20in%20HCW%20when%20testing%20migration%20server%20availability%2C%20you%20would%20first%20search%20in%20the%20HCW%20log%20for%20the%20Test-MigrationServerAvailability%20cmdlet%20executed%20by%20HCW%2C%20copy-paste%20the%20exact%20command%20to%20a%20Notepad%20file%20(or%20at%20least%20make%20a%20note%20of%20the%20RemoteServer%20value)%20and%20then%20connect%20to%20Exchange%20Online%20PowerShell%20and%20run%20the%20same%20command%20to%20confirm%20the%20error%20message%20thrown%20by%20HCW.%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fpowershell%2Fexchange%2Fexchange-online%2Fconnect-to-exchange-online-powershell%2Fconnect-to-exchange-online-powershell%3Fview%3Dexchange-ps%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EConnect%20to%20Exchange%20Online%20PowerShell%3C%2FA%3E%20using%20an%20O365%20Global%20Admin%20account%2C%20you%20can%20use%20for%20example%20this%20one%20line%20command%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EImport-PSSession%20%24(New-PSSession%20-ConfigurationName%20Microsoft.Exchange%20-ConnectionUri%20%3CA%20href%3D%22https%3A%2F%2Foutlook.office365.com%2Fpowershell-liveid%2F%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Foutlook.office365.com%2Fpowershell-liveid%2F%3C%2FA%3E%20-Authentication%20Basic%20%E2%80%93AllowRedirection%20-Credential%20%24(Get-Credential))%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOnce%20you%20are%20successfully%20connected%20to%20EXO%20PS%2C%20run%20the%20same%20command%20that%20HCW%20ran%20in%20the%20log%20file%20to%20see%20if%20you%20get%20same%20error%20or%20different%20one%2C%20providing%20the%20credentials%20for%20the%20on-premises%20migration%20admin%20and%20your%20RemoteServer%20value%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3ETest-MigrationServerAvailability%20-ExchangeRemoteMove%3A%20%24true%20-RemoteServer%20'%3CGUID%3E.resource.mailboxmigration.his.msappproxy.net'%20-Credentials%20(Get-Credential%20domain%5Cadmin)%3C%2FGUID%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EMost%20likely%20you%20will%20get%20same%20result%20as%20HCW%20did%20when%20you%20will%20run%20the%20same%20command%20in%20PowerShell.%20Although%20this%20step%20may%20seem%20redundant%2C%20it%20is%20always%20a%20good%20idea%20to%20analyze%20the%20command%20ran%20by%20HCW%20(if%20you%20selected%20Classic%20Hybrid%20option%20during%20HCW%2C%20then%20HCW%20should%20not%20try%20to%20create%20the%20migration%20endpoint%20to%20%3CGUID%3E.resource.mailboxmigration.his.msappproxy.net%20which%20is%20for%20Modern%20Hybrid%20topology)%20and%20confirm%20at%20the%20same%20time%20that%20the%20error%20was%20not%20for%20example%20a%20transient%20one.%3C%2FGUID%3E%3C%2FP%3E%0A%3CP%3EOnce%20you%20confirm%20that%20you%20still%20get%20the%20error%20that%20HCW%20was%20complaining%20about%20and%20you%20are%20running%20Modern%20Hybrid%20Mode%2C%20you%20would%20need%20to%20investigate%20it.%3C%2FP%3E%0A%3CP%3EOn%20the%20server%20where%20you%20ran%20the%20Modern%20HCW%2C%20you%20need%20to%20check%20the%20Hybrid%20Agent%20Status%20(ACTIVE%20or%20INACTIVE).%20There%20are%20two%20ways%20to%20do%20this%3A%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3ERe-run%20Modern%20HCW%20and%20check%20it%20in%20the%20GUI%2C%20reference%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment%2Fhybrid-agent%23checking-the-status-of-your-hybrid-agents%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ehere%3C%2FA%3E.%3C%2FLI%3E%0A%3CLI%3EThrough%20PowerShell%20as%20described%20below%2C%20reference%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment%2Fhybrid-agent%23option-2-get-status-via-the-hybrid-management-powershell-module%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ehere%3C%2FA%3E.%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3E%3CSTRONG%3ENOTE%3A%20Make%20sure%20you%20have%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fpowershell%2Fazure%2Fservicemanagement%2Finstall-azure-ps%3Fview%3Dazuresmps-4.0.0%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Einstalled%20Azure%20PowerShell%20Module%3C%2FA%3E%20before%20proceeding%20further.%3C%2FSTRONG%3E%3CBR%20%2F%3EImport%20the%20Hybrid%20Management%20Module%20in%20Windows%20PowerShell%20(ran%20as%20Administrator)%20from%20%5CProgram%20Files%5CMicrosoft%20Hybrid%20Service%5C%20directory%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EImport-Module%20.%5CHybridManagement.psm1%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ERun%20the%20Get-HybridAgent%20command%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EGet-HybridAgent%20-credential%20(get-credential)%20%20%20%20%23%23%20cloud%20admin%20credentials%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECheck%20that%20the%20Hybrid%20Agent(s)%20is%20%3CSTRONG%3EACTIVE%3C%2FSTRONG%3E.%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20%3CSTRONG%3EINACTIVE%3C%2FSTRONG%3E%2C%20you%20would%20check%3A%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EIf%20you%20switched%20from%20Modern%20hybrid%20to%20Classic%20hybrid%20(as%20this%20would%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment%2Fhybrid-agent%23uninstall-the-hybrid-agent%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Euninstall%20the%20agent%3C%2FA%3E).%20If%20that%20is%20the%20case%20%E2%80%93%20you%20proceeded%20with%20Classic%20hybrid%20topology%20and%20this%20successfully%20uninstalled%20the%20Hybrid%20Agent%2C%20then%20you%20can%20ignore%20this%20warning%20thrown%20by%20HCW%20related%20to%20the%20migration%20endpoint%20for%20%3CGUID%3E.resource.mailboxmigration.his.msappproxy.net%20and%20you%20should%20create%20the%20migration%20endpoint%20in%20EAC%20using%20Autodiscover%20or%20your%20published%20EWS%20URL.%20At%20the%20moment%20of%20the%20writing%20of%20this%20blog%20post%2C%20if%20you%20switch%20from%20Modern%20to%20Classic%20successfully%20%E2%80%93%20HCW%20still%20tries%20to%20create%20the%20migration%20endpoint%20going%20to%20the%20Hybrid%20Agent%20proxy%20instead%20of%20using%20your%20published%20EWS%20URL.%3C%2FGUID%3E%3C%2FLI%3E%0A%3CLI%3EIf%20Hybrid%20Service%20is%20installed%20on%20the%20machine%20and%20is%20up%20and%20running%20and%20the%20Hybrid%20Agent%20machine%20itself%20is%20running%3C%2FLI%3E%0A%3CLI%3ECheck%20all%20things%20from%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment%2Fhybrid-agent%23additional-information%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAdditional%20Information%26nbsp%3Bhere%3C%2FA%3E%20to%20see%20if%20the%20Agent%20is%20installed%20properly%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EMost%20important%2C%20if%20Hybrid%20Agent%20is%3CSTRONG%3E%20ACTIVE%3C%2FSTRONG%3E%20but%20you%20get%20the%20error%20%E2%80%98unable%20to%20connect%20to%20the%20server%E2%80%99%20in%20Test-MigrationServerAvailability%2C%20check%20and%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment%2Fhybrid-agent%23testing-and-validation-of-the-hybrid-agent%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Econfirm%20with%20Performance%20Monitor%3C%2FA%3E%20that%20you%20see%20the%20requests.%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment%2Fhybrid-agent%23testing-and-validation-of-the-hybrid-agent%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3E%3CBR%20%2F%3E%3C%2FA%3EIf%20the%20request%20counters%20(for%20%23of%20requests)%20go%20up%20on%20the%20Agent%20machine%20when%20you%20do%20Test-MigrationServerAvailability%20to%20the%20Hybrid%20Agent%2C%20the%20problem%20is%20likely%20on%20the%20on-premises%20server%3B%20if%20they%20don't%2C%20the%20problem%20is%20probably%20with%20either%20the%20connector%20or%20the%20cloud%20configuration.%20If%20we%20suspect%20on-premises%20(most%20likely%20the%20case)%2C%20you%20need%20to%20check%20again%20the%20on-premises%20infrastructure%2C%20especially%20proxy%20and%20firewall%20settings.%26nbsp%3BSeveral%20things%20that%20might%20be%20helpful%3A%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment%2Fhybrid-agent%23agent-install-location--requirements%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Einstall%20requirements%3C%2FA%3E%2C%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment%2Fhybrid-agent%23system-requirements%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Esystem%20requirements%3C%2FA%3E%2C%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment%2Fhybrid-agent%23port-and-protocol-requirements%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eport%20and%20protocol%20requirements%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CU%3ETroubleshooting%20these%20errors%20in%20Classic%20hybrid%3A%3C%2FU%3E%3C%2FP%3E%0A%3CP%3E%3CEM%3E%26nbsp%3B%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3ECheck%20your%20network%20devices%20logs%20and%20IIS%20logs%20%2F%20HTTPProxy%20logs%20at%20the%20time%20your%20run%20Test-MigrationServerAvailability%20command%2C%20usually%20if%20the%20timeout%20happens%20very%20quick%20(under%2050%20sec)%20it%20could%20probably%20be%20a%20network%20device%20that%20is%20blocking%20%2F%20closing%20the%20connection.%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CP%3ELocation%20of%20IIS%20and%20HTTPProxy%20logs%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EIIS%20logs%20Default%20Web%20Site%20(DWS)%3A%20%25SystemDrive%25%5Cinetpub%5Clogs%5CLogFiles%5CW3SVC1%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EExample%3A%20C%3A%5Cinetpub%5Clogs%5CLogFiles%5CW3SVC1%3C%2FP%3E%0A%3CP%3EThe%20name%20of%20the%20IIS%20logs%20contains%20the%20date%20of%20the%20log%2C%20for%20example%20u_ex%3CSTRONG%3E1909%3C%2FSTRONG%3E%3CSTRONG%3E30%3C%2FSTRONG%3E.log%20is%20from%20Sept%2030%2C%202019.%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EHTTPProxy%20logs%20for%20EWS%3A%20%26nbsp%3B%25ExchangeInstallPath%25Logging%5CHttpProxy%5CEws%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EExample%3A%20C%3A%5CProgram%20Files%5CMicrosoft%5CExchange%20Server%5CV15%5CLogging%5CHttpProxy%5CEws%3C%2FP%3E%0A%3CP%3EThe%20name%20of%20the%20HTTPProxy%20logs%20contains%20the%20date%20and%20hour%20starting%20to%20log%2C%20for%20example%20HttpProxy_%3CSTRONG%3E2019093014-10%3C%2FSTRONG%3E.LOG%20(10%3CSUP%3Eth%3C%2FSUP%3E%20log%20from%20Sept%2030%2C%202019%2C%20starting%20hour%2014%3A00%20UTC)%3C%2FP%3E%0A%3CP%3EBelow%20we%20have%20some%20examples%20of%20entries%20from%20IIS%20logs%20when%20we%20get%20a%20successful%20request%20(200%20OK)%20and%20a%20failed%20request%20(500)%20that%20could%20correspond%20with%20a%20timeout%20error%20in%20Test-MigrationServerAvailability%20if%20the%20request%20reached%20IIS%20%2F%20Exchange%20Server.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EIIS%20logs%20-%20200%20OK%0A2019-08-28%2006%3A57%3A38%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20-%204.4.0.1%20-%20401%200%200%200%0A2019-08-28%2006%3A57%3A42%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20-%204.4.0.1%20-%20401%201%202148074254%2015%0A2019-08-28%2006%3A57%3A42%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20contoso%5Cadministrator%204.4.0.1%20-%20200%200%200%20125%0A2019-08-28%2006%3A57%3A42%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20contoso%5Cadministrator%204.4.0.1%20-%20200%200%200%20296%0A%0AIIS%20logs%20-%20500%20error%0A2019-08-28%2007%3A15%3A48%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20-%204.4.0.1%20-%20401%202%205%204890%0A2019-08-28%2007%3A15%3A52%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20-%204.4.0.1%20-%20401%201%202148074254%200%0A2019-08-28%2007%3A15%3A55%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20contoso%5Cadministrator%204.4.0.1%20-%20500%200%200%202562%0A2019-08-28%2007%3A15%3A55%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20contoso%5Cadministrator%204.4.0.1%20-%20500%200%200%2093%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20you%20don%E2%80%99t%20see%20the%20failed%20requests%20in%20IIS%20logs%2C%20make%20sure%20you%20allow%20all%20Exchange%20Online%20IP%20addresses%20to%20connect%20to%20your%20on-premises%20and%20check%20firewall%20logs%20to%20see%20if%20connections%20were%20blocked.%3C%2FP%3E%0A%3CP%3EThis%20is%20an%20extract%20from%20HTTP%20proxy%20logs%20with%20a%20500%20error%20code%20and%20a%20timeout%20when%20proxying%20to%20another%20Exchange%20server%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3E2019-09-30T12%3A02%3A55.930Z%2Ca0ff365c-741b-4e59-b2e3-760991d3a27a%2C15%2C1%2C1713%2C5%2C%2CEws%2Cexch01.contoso.com%2C%2FEWS%2Fmrsproxy.svc%2C%2CNegotiate%2Ctrue%2Ccontoso%5Cadmin%2C%2COrganizationId~OrganizationAnchor%40%2C%2C40.100.175.55%2Cexch01%2C500%2C%2CServerLocatorError%2CPOST%2C%2C%2C%2C%2CForestWideOrganization%2C%2C%2C2807%2C664%2C%2C%2C%2C1%2C2819%2C0%3B%2C0%2C%2C0%2C8%3B2%3B%2C10%2C10%2C%2C0%2C2821%2C0%2C%2C%2C%2C%2C%2C%2C%2C%2C0%2C2819%2C0%2C%2C2819%2C%2C2820%2C2820%2C%2C%2C%2CBeginRequest%3D2019-09-30T12%3A02%3A53.109Z%3BCorrelationID%3D%3CEMPTY%3E%3BProxyState-Run%3DNone%3BServerLocatorCall%3DDM%3Aa10ad628-e020-409e-9f1e-22a700182ac1~~contoso.structure%3BProxyState-Complete%3DCalculateBackEnd%3BSharedCacheGuard%3D0%3BEndRequest%3D2019-09-30T12%3A02%3A55.930Z%3BS%3AServiceCommonMetadata.Cookie%3Dee4af79a0a144bcaa9a5bc0af6eec215%3BI32%3AADR.C%5BDC04%5D%3D1%3BF%3AADR.AL%5BDC04%5D%3D1.554922%3BI32%3AATE.C%5BDC04.contoso.local%5D%3D2%3BF%3AATE.AL%5BDC04.contoso.local%5D%3D1%3BI32%3AADS.C%5BDC04%5D%3D1%3BF%3AADS.AL%5BDC04%5D%3D2.897117%2CHttpProxyException%3DMicrosoft.Exchange.HttpProxy.HttpProxyException%3A%20Server%20Locator%20Service%20call%20had%20a%20communication%20error.%20---%26gt%3B%20Microsoft.Exchange.Data.ServerLocator.ServerLocatorClientTransientException%3A%20Server%20Locator%20Service%20call%20had%20a%20communication%20error.%20---%26gt%3B%20System.ServiceModel.EndpointNotFoundException%3A%20Could%20not%20connect%20to%20net.tcp%3A%2F%2Fexch02.contoso.local%3A64337%2FExchange.HighAvailability%2FServerLocator.%20The%20connection%20attempt%20lasted%20for%20a%20time%20span%20of%2000%3A00%3A02.8010812.%20TCP%20error%20code%2010060%3A%20A%20connection%20attempt%20failed%20because%20the%20connected%20party%20did%20not%20properly%20respond%20after%20a%20period%20of%20time%20%20or%20established%20connection%20failed%20because%20connected%20host%20has%20failed%20to%20respond%2010.2.2.1%3A%3A64337.%20%20---%26gt%3B%20System.Net.Sockets.SocketException%3A%20A%20connection%20attempt%20failed%20because%20the%20connected%20party%20did%20not%20properly%20respond%20after%20a%20period%20of%20time%20%20or%20established%20connection%20failed%20because%20connected%20host%20has%20failed%20to%20respond%2010.2.2.1%3A%3A64337%20%20%20%20at%20System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult%20asyncResult)%20%20%20%20at%20System.Net.Sockets.Socket.EndConnect(IAsyncResult%20asyncResult)%20%20%20%20at%20System.ServiceModel.Channels.SocketConnectionInitiator.ConnectAsyncResult.OnConnect(IAsyncResult%20result)%20%20%20%20---%20End%20of%20inner%20exception%20stack%20trace%20---%3C%2FEMPTY%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-230998965%22%20id%3D%22toc-hId-230998965%22%3EScenario%202%3A%20Test-MigrationServerAvailability%20fails%20with%20403%20Forbidden%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EThe%20connection%20to%20the%20server%20'%3CGUID%3E.resource.mailboxmigration.his.msappproxy.net'%20could%20not%20be%20completed.%2C%20The%20call%20to%20'https%3A%2F%2F%3CGUID%3E.resource.mailboxmigration.his.msappproxy.net%2FEWS%2Fmrsproxy.svc'%20failed.%20Error%20details%3A%20The%20HTTP%20request%20was%20forbidden%20with%20client%20authentication%20scheme%20'Negotiate'.%20--%26gt%3B%20The%20remote%20server%20returned%20an%20error%3A%20(403)%20Forbidden..%2C%20The%20HTTP%20request%20was%20forbidden%20with%20client%20authentication%20scheme%20'Negotiate'.%2C%20The%20remote%20server%20returned%20an%20error%3A%20(403)%20Forbidden.%3C%2FGUID%3E%3C%2FGUID%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFollow%20%3CA%20href%3D%22https%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Fhelp%2F3063913%2Fthe-remote-server-returned-an-error-403-forbidden-error-when-you-try-t%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ethis%3C%2FA%3E%3CSPAN%3E%20article%3C%2FSPAN%3E%2C%20as%20it%20is%20applicable%20for%20both%20Modern%20and%20Classic%20hybrid%20topologies.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1973809300%22%20id%3D%22toc-hId-1973809300%22%3E%3CFONT%20size%3D%225%22%3EScenario%203%3A%20Test-MigrationServerAvailability%20fails%20with%20401%20Unauthorized%2C%20Access%20denied%20or%20Invalid%20credentials%3CBR%20%2F%3E%3C%2FFONT%3E%3C%2FH2%3E%0A%3CP%3ECheck%20%3CA%20href%3D%22https%3A%2F%2Fsupport.microsoft.com%2Fen-us%2Fhelp%2F2975731%2Faccess-is-denied-error-when-you-try-to-move-mailboxes-to-exchange-onli%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ethis%20support%20article%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3ECheck%20authentication%20methods%20on%20all%20EWS%20virtual%20directories%20in%20IIS%20and%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fclients%2Fdefault-virtual-directory-settings%3Fview%3Dexchserver-2019%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Emake%20sure%20Negotiate%20provider%20under%20Windows%20Authentication%20is%20enabled%20for%20all%20EWS.%3C%2FA%3E%3C%2FP%3E%0A%3CP%3EMake%20sure%20on-premises%20migration%20Admin%20has%20at%20least%20Exchange%20Recipient%20Admin%20permissions%20(or%20Recipient%20Management%2C%20depending%20on%20the%20Exchange%20version).%20If%20you%20are%20running%20Modern%20HCW%2C%20usually%20this%20is%20the%20same%20on-premises%20account%20that%20has%20Organization%20Management%20rights%2C%20but%20if%20you%20change%20the%20account%2C%20you%20would%20see%20here%20if%20the%20permissions%20are%20right%20or%20the%20credentials%20are%20correct%20when%20we%20would%20input%20account%20credentials.%20We%20recommend%20that%20you%20create%20(do%20not%20copy%20account)%20another%20on-premises%20account%20that%20purely%20has%20Exchange%20Recipient%20Admin%20permissions%2C%20then%20test%20with%20that%20account%20(Test-MigrationServerAvailability)%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F149727i5614A11E239CB3FF%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20alt%3D%22troublhybmig08.jpg%22%20title%3D%22troublhybmig08.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EDo%20you%20have%20devices%20that%20pre-authenticate%20the%20requests%20coming%20from%20Exchange%20Online%20to%20Exchange%20on-premises%20servers%20(EWS%20and%20Autodiscover%20paths)%3F%20If%20yes%2C%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fhybrid-deployment-prerequisites%23hybrid-deployment-protocols-ports-and-endpoints%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ethis%20is%20not%20supported%3C%2FA%3E.%20If%20you%20are%20not%20sure%20of%20it%2C%20we%20recommend%20you%20temporarily%20bypass%20network%20devices%20in%20front%20of%20Exchange%20Servers%2C%20allow%20direct%20access%20to%20Exchange%20Servers%20and%20see%20if%20same%20error%20when%20doing%20Test-MigrationServerAvailability%3C%2FP%3E%0A%3CP%3EWhen%20you%20run%20Test-MigrationServerAvailability%2C%20make%20a%20note%20of%20the%20timestamp%20when%20you%20get%20the%20error%2C%20then%20check%20IIS%20logs%20on%20each%20Exchange%20Client%20Access%20Server%20(logs%20are%20UTC%20timezone)%20around%20the%20exact%20time%20when%20Test-MigrationServerAvailability%20has%20been%20ran%20(HH%3AMM%3ASS)%20and%20check%20entries%20for%20mrsproxy.svc%20and%20see%20the%20statuses%20mentioned.%20Normally%2C%20there%20following%20are%20the%20first%203%20IIS%20entries%20for%201%20successful%20request%20(Test-MigrationServerAvailability)%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3ESuccessful%0A2019-08-28%2006%3A57%3A38%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20-%204.4.0.1%20-%20401%200%200%200%0A2019-08-28%2006%3A57%3A42%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20-%204.4.0.1%20-%20401%201%202148074254%2015%0A2019-08-28%2006%3A57%3A42%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20miry%5Cadministrator%204.4.0.1%20-%20200%200%200%20125%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESome%20other%20error%20examples%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EIssue%3A%20wrong%20credentials%20of%20migration%20admin%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EMessage%20%20%20%20%20%20%20%20%20%20%20%20%3A%20The%20connection%20to%20the%20server%20'mail.contoso.com'%20could%20not%20be%20completed.%0AErrorDetail%20%20%20%20%20%20%20%20%3A%20Microsoft.Exchange.MailboxReplicationService.RemotePermanentException%3A%20The%20Mailbox%20Replication%20Service%20was%20unable%20to%20connect%20to%20the%20remote%20server%20using%20the%20credentials%20provided.%20Please%20check%20the%20credentials%20and%20try%20again.%20The%20call%20to%20'%3CA%20href%3D%22https%3A%2F%2Fmail.contoso.com%2FEWS%2Fmrsproxy.svc%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fmail.contoso.com%2FEWS%2Fmrsproxy.svc%3C%2FA%3E'%20failed.%20Error%20details%3A%20The%20HTTP%20request%20is%20unauthorized%20with%20client%20authentication%20scheme%20'Negotiate'.%20The%20authentication%20header%20received%20from%20the%20server%20was%20'Negotiate%2CNTLM'.%20--%26gt%3B%20The%20remote%20server%20returned%20an%20error%3A%20(401)%20Unauthorized.%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EIIS%20Logs%0A---------%0A2019-08-28%2007%3A19%3A05%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20-%204.4.0.1%20-%20401%200%200%202390%0A2019-08-28%2007%3A19%3A10%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20-%204.4.0.1%20-%20401%201%202148074254%200%0A2019-08-28%2007%3A19%3A10%20192.168.2.50%20POST%20%2FEWS%2Fmrsproxy.svc%20-%20443%20-%204.4.0.1%20-%20401%201%202148074252%2046%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EIssue%3A%20authentication%20scheme%20mismatch%20(EXO%20requires%20Negotiate%20%2F%20NTLM%2C%20on-premises%20gives%20us%20Basic%20only)%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EMessage%20%20%20%20%20%20%20%20%20%3A%20The%20connection%20to%20the%20server%20'mail.contoso.com'%20could%20not%20be%20completed%0AErrorDetail%20%20%20%20%20%20%3A%20Microsoft.Exchange.MailboxReplicationService.RemotePermanentException%0AThe%20Mailbox%20Replication%20Service%20was%20unable%20to%20connect%20to%20the%20remote%20server%20using%20the%20credentials%20provided.%20Please%20check%20the%20credentials%20and%20try%20again.%20The%20call%20to%20'%3CA%20href%3D%22https%3A%2F%2Fmail.contoso.com%2FEWS%2Fmrsproxy.svc%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fmail.contoso.com%2FEWS%2Fmrsproxy.svc%3C%2FA%3E%20failed.%20Error%20details%3A%20The%20HTTP%20request%20is%20unauthorized%20with%20client%20authentication%20scheme%20'Negotiate'.%20The%20authentication%20header%20received%20from%20the%20server%20was%20'Basic%20Realm%3D%22mail.contoso.com%22'.%20--%26gt%3B%20The%20remote%20server%20returned%20an%20error%3A%20(401)%20Unauthorized.%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFirst%2C%20make%20sure%20that%20NTLM%20is%20enabled%20on%20the%20EWS%20virtual%20directory.%20You%20would%20check%20with%20Get-WebServicesVirtualDirectory%20%7CFL%20cmdlet%20if%20NTLM%20is%20present%20in%20the%20Authentication%20Methods.%20You%20should%20also%20double%20check%20in%20IIS%20Manager%2C%20to%20make%20sure%20that%20the%20Negotiate%20provider%20is%20present%20as%20well%20under%20Windows%20Authentication.%20Negotiate%20provider%20is%20falling%20back%20to%20NTLM%20if%20Kerberos%20is%20not%20possible%20and%20with%20external%20clients%20that%20is%20always%20the%20case.%20So%2C%20Exchange%20Online%20MRS%20requires%20NTLM%20inside%20the%20Negotiate%20Provider%20on%20the%20EWS%20on-premises%20virtual%20directory.%3C%2FP%3E%0A%3CP%3EIf%20you%20are%20using%20Azure%20AD%20App%20Proxy%20as%20Reverse%20Proxy%20for%20your%20MRSProxy%20servers%2C%20then%20you%20should%20be%20aware%20that%20there%20is%20a%20limitation%20of%20AADAP%20that%20cannot%20present%20both%20Negotiate%20and%20NTLM%20providers%20in%20the%20WWW-Authenticate%20header%2C%20no%20matter%20the%20order%20of%20the%20providers%20in%20IIS.%20If%20you%20have%20this%20setup%2C%20you%20can%20remove%20NTLM%20provider%20from%20Windows%20Authentication%20on%20EWS%20in%20IIS%20Manager%20(leave%20only%20Negotiate%20provider%20that%20does%20also%20NTLM)%20or%20bypass%20AADAP.%3CBR%20%2F%3EIf%20that%20is%20fine%20(configuration%20on%20Exchange%20and%20IIS%20side)%2C%20then%20you%20would%20check%20the%20publishing%20rules%20for%20EWS%20on%20your%20firewall%2Freverse%20proxy.%3C%2FP%3E%0A%3CP%3EYou%20can%20also%20use%20this%20mini%20PowerShell%20script%20to%20check%20the%20Authentication%20Methods%20advertised%20by%20your%20on-premises%20(replace%20the%20URL%20with%20your%20on-premises%20MRSProxy%20%2FEWS%20namespace).%26nbsp%3B%20It%20is%20also%20a%20good%20practice%20to%20check%20the%20WWW-Authenticate%20headers%20both%20from%20external%20and%20internal%20PC%20and%20notice%20if%20there%20are%20differences%20in%20the%20output%20(for%20example%20you%20see%20Basic%2C%20Negotiate%2C%20NTLM%20from%20internal%20%E2%80%93%20all%20ok%20but%20you%20see%20only%20Basic%20from%20external%20%E2%80%93%20not%20ok).%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3E%24req%20%3D%20%5BSystem.Net.HttpWebRequest%5D%3A%3ACreate(%22%3CA%20href%3D%22https%3A%2F%2Fmail.contoso.com%2Fews%2FMRSProxy.svc%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fmail.contoso.com%2Fews%2FMRSProxy.svc%3C%2FA%3E%22)%0A%24req.UseDefaultCredentials%20%3D%20%24false%0A%24req.GetResponse()%0A%23%20Expected%20error%3A%20Exception%20calling%20%22GetResponse%22%20with%20%220%22%20argument(s)%3A%0A%23%20%22The%20remote%20server%20returned%20an%20error%3A%20(401)%20Unauthorized.%22%0A%24ex%20%3D%20%24error%5B0%5D.Exception%0A%24resp%20%3D%20%24ex.InnerException.Response%0A%24resp.Headers%5B%22WWW-Authenticate%22%5D%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId--578347661%22%20id%3D%22toc-hId--578347661%22%3EScenario%204%3A%20Test-MigrationServerAvailability%20fails%20with%20SSL%20%2F%20TLS%20error%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EMicrosoft.Exchange.Migration.MigrationServerConnectionFailedException%3A%20The%20connection%20to%20the%20server'hybrid.contoso.com'%20could%20not%20be%20completed.%20---%26gt%3B%0AMicrosoft.Exchange.MailboxReplicationService.MRSRemotePermanentException%3A%20The%20Mailbox%20Replication%20Service%20could%20not%20connect%20to%20the%20remote%20server%20because%20the%20certificate%20is%20invalid.%20The%20call%20to%20'%3CA%20href%3D%22https%3A%2F%2Fhybrid.contoso.com%2FEWS%2Fmrsproxy.svc%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fhybrid.contoso.com%2FEWS%2Fmrsproxy.svc%3C%2FA%3E'%20failed.%20Error%20details%3A%20Could%20not%20establish%20trust%20relationship%20for%20the%20SSL%2FTLS%20secure%20channel%20with%20authority%20'hybrid.contoso.com'.%20--%26gt%3BThe%20underlying%20connection%20was%20closed%3A%20Could%20not%20establish%20trust%20relationship%20for%20the%20SSL%2FTLS%20secure%20channel.%20--%26gt%3B%20The%20remote%20certificate%20is%20invalid%20according%20to%20the%20validation%20procedure.%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhenever%20you%20see%20SSL%2FTLS%20errors%2C%20you%20would%20check%20the%20following%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Foffice365%2Ftroubleshoot%2Fsecurity%2Fprepare-tls-1.2-in-office-365%23references%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3ETLS1.2%20should%20be%20enabled%20in%20the%20on-premises%20infrastructure%3C%2FA%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CP%3EFor%20Classic%20hybrid%2C%20you%20can%20use%20this%20PowerShell%20command%20while%20logged%20to%20your%20Office%20365%20Exchange%20Online%20tenant%20to%20test%20the%20network%20request%20on%20TLS1.2%20protocol%20and%20see%20if%20you%20also%20get%20a%20SSL%2FTLS%20error%20here%3A%3CBR%20%2F%3E%5BNet.ServicePointManager%5D%3A%3ASecurityProtocol%20%3D%20%5BNet.SecurityProtocolType%5D%3A%3ATls12%3BInvoke-WebRequest%26nbsp%3Bhttps%3A%2F%2F%3CENDPOINT%3E%2Fews%2Fmrsproxy.svc%20-Verbose%3C%2FENDPOINT%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20Modern%20Hybrid%2C%20you%20can%20check%20Test-HybridConnectivity%20which%20also%20checks%20for%20TLS1.2%20and%20this%20is%20mandatory%20step%20when%20configuring%20%2F%20installing%20the%20Hybrid%20Agent.%20See%20Verifying%20Connectivity%20section%20from%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FExchange-Team-Blog%2FThe-Microsoft-Hybrid-Agent-Public-Preview%2Fba-p%2F608939%22%20target%3D%22_blank%22%20rel%3D%22noopener%22%3Ehere%3C%2FA%3E.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EMake%20sure%20you%20are%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fconfiguring-ssl-offloading-in-exchange-2013-exchange-2013-help%23configuring-ssl-offloading-for-the-mailbox-replication-proxy-service-mrsproxy%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3Enot%20doing%20SSL%20offloading%20for%20MRSProxy.svc%3C%2FA%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CP%3EFor%20Classic%20hybrid%2C%20a%20valid%203%3CSUP%3Erd%3C%2FSUP%3E%20party%20valid%20certificate%20is%20required%20for%20EWS.%20See%20this%20and%20this.%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CP%3EYou%20can%20use%20the%20following%20command%20in%20Exchange%20Management%20Shell%20to%20quickly%20check%20the%20Exchange%20certificates%20and%20some%20properties%20on%20them%20from%20all%20your%20Exchange%20servers%20in%20the%20organization%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EForeach%20(%24i%20in%20(Get-ExchangeServer))%20%7BWrite-Host%20%24i.FQDN%3B%20Get-ExchangeCertificate%20-Server%20%24i.Identity%20%7C%20FT%20Thumbprint%2C%20Status%2C%20RootCAtype%2C%20Services%2C%20Subject%7D%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAlso%2C%20you%20should%20be%20able%20to%20access%20the%20CRLs%20for%20the%20certificate.%3C%2FP%3E%0A%3CH2%20id%3D%22toc-hId-1164462674%22%20id%3D%22toc-hId-1164462674%22%3EScenario%205%3A%20Test-MigrationServerAvailability%20fails%20with%20503%20Service%20Unavailable%3C%2FH2%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EThe%20call%20to%20'https%3A%2F%2F%3CGUID%3E%20resource.mailboxmigration.his.msappproxy.net%2FEWS%2Fmrsproxy.svc'%20failed.%20Error%20details%3A%20The%20HTTP%20service%20located%20at%20https%3A%2F%2F%3CGUID%3E.resource.mailboxmigration.his.msappproxy.net%2FEWS%2Fmrsproxy.svc%20is%20unavailable.%20%20This%20could%20be%20because%20the%20service%20is%20too%20busy%20or%20because%20no%20endpoint%20was%20found%20listening%20at%20the%20specified%20address.%20Please%20ensure%20that%20the%20address%20is%20correct%20and%20try%20accessing%20the%20service%20again%20later.%20--%26gt%3B%20The%20remote%20server%20returned%20an%20error%3A%20(503)%20Server%20Unavailable%3C%2FGUID%3E%3C%2FGUID%3E%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20this%20error%2C%20best%20is%20to%20check%20HTTPProxy%20logs%20for%20EWS%20and%20see%20if%20it%20tried%20to%20proxy%20to%20a%20server%20that%20is%20unavailable%2C%20for%20example%20Server%20Wide%20Offline%20in%20Get-ServerComponentState%20or%20maybe%20EWS%20application%20pool%20is%20not%20started%20or%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fpowershell%2Fmodule%2Fexchange%2Fmailbox-databases-and-servers%2Ftest-mrshealth%3Fview%3Dexchange-ps%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%22%3EMRS%20service%20is%20stopped%3C%2FA%3E.%20One%20thing%20to%20add%20here%20is%20that%2C%20if%20you%20have%20Exchange%202013%2F2016%20in%20coexistence%20with%20Exchange%202010%2C%20the%20Exchange%202013%2F%202016%20servers%20will%20proxy%20to%20the%20same%20version%20server%20(2013%2F2016)%20and%20will%20not%20proxy%20down%20to%20Exchange%202010.%3C%2FP%3E%0A%3CP%3EThis%20brings%20us%20to%20the%20end%20of%20this%20post!%20Hope%20you%20find%20this%20useful%20if%20you%20want%20to%20learn%20a%20bit%20more%20about%20how%20migration%20endpoints%20work%2C%20or%20you%20need%20to%20troubleshoot%20this%20area.%3C%2FP%3E%0A%3CP%3EI%20wanted%20to%20thank%20Brad%20Hughes%2C%20Jason%20Nelson%2C%20Nino%20Bilic%20and%20Greg%20Taylor%20for%20their%20review%20of%20this%20post.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22author%22%3EMirela%20Buruiana%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-953006%22%20slang%3D%22en-US%22%3E%3CP%3EIn%20our%20previous%20blog%20post%20we%20covered%20an%20overview%20of%20what%20migration%20endpoints%20are%2C%20how%20to%20find%20them%20and%20what%20makes%20them%20tick.%20In%20this%20post%2C%20we%20will%20cover%20related%20troubleshooting.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-953006%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EHybrid%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMigration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ETips%20'n%20Tricks%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ETroubleshooting%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E

In our previous blog post we covered an overview of what migration endpoints are, how to find them and what makes them tick. In this post, we will cover related troubleshooting. Note that this post has some in-depth troubleshooting steps, so it is not necessarily something that you’ll read for fun, but we wanted to make it available for those times when you run into trouble!

Now let’s cover some troubleshooting!

Things that are commonly mis-configured

Before getting to the part where we troubleshoot specific migration endpoint issues, be aware of the following top reasons why a migration endpoint cannot be created:

 

Troubleshooting

Whether you are having trouble creating new migration endpoints in Office 365 Exchange Online or are not able to migrate anymore to or from Exchange Online using an existing migration endpoint, the cmdlet Test-MigrationServerAvailability is your dear friend. Always run this cmdlet in Exchange Online PowerShell, not from on-premises Exchange Management Shell.

We will focus on hybrid migration endpoint issues and the cmdlet syntax for this endpoint type. Below we will show you 3 commands that can help you check for underlying issues or error messages:

1. Hybrid remote move endpoint with Autodiscover

 

Test-MigrationServerAvailability -ExchangeRemoteMove -Autodiscover -EmailAddress user@contoso.com -Credentials (get-credential contoso\administrator)

 

troublhybmig02.jpg

Note: The option with Autodiscover is not used in Modern hybrid as we go directly to EWS server(s) for both Migration Endpoints and Free/Busy configuration (Cloud Intra-Organization Connectors and Organization Relationships have TargetSharingEpr set to the EWS namespace.)

2. Hybrid remote move endpoint without Autodiscover (testing EWS directly)

 

Test-MigrationServerAvailability -ExchangeRemoteMove –RemoteServer mail.contoso.com -Credentials(get-credential contoso\administrator)

 

troublhybmig03.jpg

Note: In Modern hybrid, the RemoteServer is in the format of '<GUID>.resource.mailboxmigration.his.msappproxy.net', where <GUID> is unique for each organization. This is randomly generated and stored encoded base 64 in the OnPremisesOrganization object’s Comment in Exchange Online when you will run Modern HCW and it gets to that configuration part.  This GUID will be stamped on the Migration Endpoint in the RemoteServer value for both Full and Minimal Modern Hybrid Topologies and also on the TargetSharingEpr values for Cloud Intra-Organization Connector / Organization Relationship . You can check the GUID in the HCW log and on Get-MigrationEndpoint / Get-IntraOrganizationConnector / Get-OrganizationRelationship EXO cmdlets.

troublhybmig04.jpg

Cmdlets ran above:

 

Get-IntraOrganizationConnector | fl targetsharingepr
Get-OrganizationRelationship | fl targetsharingepr
Get-MigrationEndpoint
$strdata = (Get-OnPremisesOrganization).comment
$bytes = [Convert]::FromBase64String($strdata)
$ms = New-Object System.IO.MemoryStream(@(,$bytes))
$deflate = New-Object System.IO.Compression.DeflateStream($ms, [System.IO.Compression.CompressionMode]::Decompress)
$reader = New-Object System.IO.StreamReader($deflate)
$text = $reader.ReadToEnd()
$text

 

 

3. Testing an existing hybrid remote move endpoint

 

Test-MigrationServerAvailability -Endpoint <Identity of the Endpoint>

 

troublhybmig05.jpg

OK, I ran these and found errors; now what?

Let’s walk through some common Test-MigrationServerAvailability errors and how to troubleshoot these in Classic / Modern hybrid.
Tip: Whenever a command returns an error in PowerShell, you should run the command $Error[0].Exception |fl -f to get more details on the exception thrown.

However, test-migrationserveravailability failures are not actually those red errors that you get when a command does not work in PowerShell. So, in this situation, we can use New-MoveRequest command to throw the same error and get the serialized exception from this one.
Test-MigrationServerAvailability which returns the result:

troublhybmig06.jpg

Running the New-MoveRequest command to test migration of a synced user will give me same error message but in “red” and we can then get the serialized exception from it.

troublhybmig07.jpg

Based on these error messages, we gathered most common scenarios.

Scenario 1: Test-MigrationServerAvailability fails because of connectivity / timeout errors

Examples of some of those errors:

 

The call to 'https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' timed out. Error details: The request channel timed out attempting to send after 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.

The call to 'https://mail.contoso.com/EWS/mrsproxy.svc' timed out. Error details: The operation did not complete within the allotted timeout of 00:00:50. The time allotted to this operation may have been a portion of a longer timeout. 

The request channel timed out while waiting for a reply after 00:00:09.9996191. The time allotted to this operation may have been a portion of a longer timeout. The remote server returned an error: (504) Gateway Timeout.

 

Troubleshooting these timeout errors in Modern hybrid:


During the Modern hybrid configuration, you will be asked to input the credentials for the on-premises migration admin – these can be the same credentials inserted in the beginning of HCW or new ones. The on-premises Migration admin credentials are needed by HCW in order to run the Test-MigrationServerAvailability cmdlet and this account can have lesser privileges than the admin account running HCW (which is Organization Management role). Once Modern HCW has tested the on-premises migration server availability, we will know if we are able to create the migration endpoint in Exchange Online or not and also in this step we would see the dynamically generated GUID for your on-premises MRSProxy server(s): https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc.
Supposing that you get a timeout error in HCW when testing migration server availability, you would first search in the HCW log for the Test-MigrationServerAvailability cmdlet executed by HCW, copy-paste the exact command to a Notepad file (or at least make a note of the RemoteServer value) and then connect to Exchange Online PowerShell and run the same command to confirm the error message thrown by HCW.

Connect to Exchange Online PowerShell using an O365 Global Admin account, you can use for example this one line command:

 

Import-PSSession $(New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Authentication Basic –AllowRedirection -Credential $(Get-Credential))

 

Once you are successfully connected to EXO PS, run the same command that HCW ran in the log file to see if you get same error or different one, providing the credentials for the on-premises migration admin and your RemoteServer value:

 

Test-MigrationServerAvailability -ExchangeRemoteMove: $true -RemoteServer '<GUID>.resource.mailboxmigration.his.msappproxy.net' -Credentials (Get-Credential domain\admin)

 

Most likely you will get same result as HCW did when you will run the same command in PowerShell. Although this step may seem redundant, it is always a good idea to analyze the command ran by HCW (if you selected Classic Hybrid option during HCW, then HCW should not try to create the migration endpoint to <GUID>.resource.mailboxmigration.his.msappproxy.net which is for Modern Hybrid topology) and confirm at the same time that the error was not for example a transient one.

Once you confirm that you still get the error that HCW was complaining about and you are running Modern Hybrid Mode, you would need to investigate it.

On the server where you ran the Modern HCW, you need to check the Hybrid Agent Status (ACTIVE or INACTIVE). There are two ways to do this:

  1. Re-run Modern HCW and check it in the GUI, reference here.
  2. Through PowerShell as described below, reference here.

NOTE: Make sure you have installed Azure PowerShell Module before proceeding further.
Import the Hybrid Management Module in Windows PowerShell (ran as Administrator) from \Program Files\Microsoft Hybrid Service\ directory:

 

Import-Module .\HybridManagement.psm1

 

Run the Get-HybridAgent command:

 

Get-HybridAgent -credential (get-credential)    ## cloud admin credentials

 

Check that the Hybrid Agent(s) is ACTIVE

If INACTIVE, you would check:

  • If you switched from Modern hybrid to Classic hybrid (as this would uninstall the agent). If that is the case – you proceeded with Classic hybrid topology and this successfully uninstalled the Hybrid Agent, then you can ignore this warning thrown by HCW related to the migration endpoint for <GUID>.resource.mailboxmigration.his.msappproxy.net and you should create the migration endpoint in EAC using Autodiscover or your published EWS URL. At the moment of the writing of this blog post, if you switch from Modern to Classic successfully – HCW still tries to create the migration endpoint going to the Hybrid Agent proxy instead of using your published EWS URL.
  • If Hybrid Service is installed on the machine and is up and running and the Hybrid Agent machine itself is running
  • Check all things from Additional Information here to see if the Agent is installed properly

Most important, if Hybrid Agent is ACTIVE but you get the error ‘unable to connect to the server’ in Test-MigrationServerAvailability, check and confirm with Performance Monitor that you see the requests.
If the request counters (for #of requests) go up on the Agent machine when you do Test-MigrationServerAvailability to the Hybrid Agent, the problem is likely on the on-premises server; if they don't, the problem is probably with either the connector or the cloud configuration. If we suspect on-premises (most likely the case), you need to check again the on-premises infrastructure, especially proxy and firewall settings. Several things that might be helpful: install requirements, system requirements, port and protocol requirements.

 

Troubleshooting these errors in Classic hybrid:

 

Check your network devices logs and IIS logs / HTTPProxy logs at the time your run Test-MigrationServerAvailability command, usually if the timeout happens very quick (under 50 sec) it could probably be a network device that is blocking / closing the connection.

Location of IIS and HTTPProxy logs:

  • IIS logs Default Web Site (DWS): %SystemDrive%\inetpub\logs\LogFiles\W3SVC1

Example: C:\inetpub\logs\LogFiles\W3SVC1

The name of the IIS logs contains the date of the log, for example u_ex190930.log is from Sept 30, 2019.

  • HTTPProxy logs for EWS:  %ExchangeInstallPath%Logging\HttpProxy\Ews

Example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews

The name of the HTTPProxy logs contains the date and hour starting to log, for example HttpProxy_2019093014-10.LOG (10th log from Sept 30, 2019, starting hour 14:00 UTC)

Below we have some examples of entries from IIS logs when we get a successful request (200 OK) and a failed request (500) that could correspond with a timeout error in Test-MigrationServerAvailability if the request reached IIS / Exchange Server.

 

IIS logs - 200 OK
2019-08-28 06:57:38 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 0
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 15
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contoso\administrator 4.4.0.1 - 200 0 0 125
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contoso\administrator 4.4.0.1 - 200 0 0 296

IIS logs - 500 error
2019-08-28 07:15:48 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 2 5 4890
2019-08-28 07:15:52 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 0
2019-08-28 07:15:55 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contoso\administrator 4.4.0.1 - 500 0 0 2562
2019-08-28 07:15:55 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contoso\administrator 4.4.0.1 - 500 0 0 93

 

If you don’t see the failed requests in IIS logs, make sure you allow all Exchange Online IP addresses to connect to your on-premises and check firewall logs to see if connections were blocked.

This is an extract from HTTP proxy logs with a 500 error code and a timeout when proxying to another Exchange server:

 

2019-09-30T12:02:55.930Z,a0ff365c-741b-4e59-b2e3-760991d3a27a,15,1,1713,5,,Ews,exch01.contoso.com,/EWS/mrsproxy.svc,,Negotiate,true,contoso\admin,,OrganizationId~OrganizationAnchor@,,40.100.175.55,exch01,500,,ServerLocatorError,POST,,,,,ForestWideOrganization,,,2807,664,,,,1,2819,0;,0,,0,8;2;,10,10,,0,2821,0,,,,,,,,,0,2819,0,,2819,,2820,2820,,,,BeginRequest=2019-09-30T12:02:53.109Z;CorrelationID=<empty>;ProxyState-Run=None;ServerLocatorCall=DM:a10ad628-e020-409e-9f1e-22a700182ac1~~contoso.structure;ProxyState-Complete=CalculateBackEnd;SharedCacheGuard=0;EndRequest=2019-09-30T12:02:55.930Z;S:ServiceCommonMetadata.Cookie=ee4af79a0a144bcaa9a5bc0af6eec215;I32:ADR.C[DC04]=1;F:ADR.AL[DC04]=1.554922;I32:ATE.C[DC04.contoso.local]=2;F:ATE.AL[DC04.contoso.local]=1;I32:ADS.C[DC04]=1;F:ADS.AL[DC04]=2.897117,HttpProxyException=Microsoft.Exchange.HttpProxy.HttpProxyException: Server Locator Service call had a communication error. ---> Microsoft.Exchange.Data.ServerLocator.ServerLocatorClientTransientException: Server Locator Service call had a communication error. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://exch02.contoso.local:64337/Exchange.HighAvailability/ServerLocator. The connection attempt lasted for a time span of 00:00:02.8010812. TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time  or established connection failed because connected host has failed to respond 10.2.2.1::64337.  ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time  or established connection failed because connected host has failed to respond 10.2.2.1::64337    at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)    at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)    at System.ServiceModel.Channels.SocketConnectionInitiator.ConnectAsyncResult.OnConnect(IAsyncResult result)    --- End of inner exception stack trace ---

 

Scenario 2: Test-MigrationServerAvailability fails with 403 Forbidden

 

The connection to the server '<GUID>.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to 'https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP request was forbidden with client authentication scheme 'Negotiate'. --> The remote server returned an error: (403) Forbidden.., The HTTP request was forbidden with client authentication scheme 'Negotiate'., The remote server returned an error: (403) Forbidden.

 

Follow this article, as it is applicable for both Modern and Classic hybrid topologies.

Scenario 3: Test-MigrationServerAvailability fails with 401 Unauthorized, Access denied or Invalid credentials

Check this support article.

Check authentication methods on all EWS virtual directories in IIS and make sure Negotiate provider under Windows Authentication is enabled for all EWS.

Make sure on-premises migration Admin has at least Exchange Recipient Admin permissions (or Recipient Management, depending on the Exchange version). If you are running Modern HCW, usually this is the same on-premises account that has Organization Management rights, but if you change the account, you would see here if the permissions are right or the credentials are correct when we would input account credentials. We recommend that you create (do not copy account) another on-premises account that purely has Exchange Recipient Admin permissions, then test with that account (Test-MigrationServerAvailability):

troublhybmig08.jpg

Do you have devices that pre-authenticate the requests coming from Exchange Online to Exchange on-premises servers (EWS and Autodiscover paths)? If yes, this is not supported. If you are not sure of it, we recommend you temporarily bypass network devices in front of Exchange Servers, allow direct access to Exchange Servers and see if same error when doing Test-MigrationServerAvailability

When you run Test-MigrationServerAvailability, make a note of the timestamp when you get the error, then check IIS logs on each Exchange Client Access Server (logs are UTC timezone) around the exact time when Test-MigrationServerAvailability has been ran (HH:MM:SS) and check entries for mrsproxy.svc and see the statuses mentioned. Normally, there following are the first 3 IIS entries for 1 successful request (Test-MigrationServerAvailability):

 

Successful
2019-08-28 06:57:38 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 0
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 15
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 miry\administrator 4.4.0.1 - 200 0 0 125

 

Some other error examples:

 

Issue: wrong credentials of migration admin

 

Message            : The connection to the server 'mail.contoso.com' could not be completed.
ErrorDetail        : Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The Mailbox Replication Service was unable to connect to the remote server using the credentials provided. Please check the credentials and try again. The call to 'https://mail.contoso.com/EWS/mrsproxy.svc' failed. Error details: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate,NTLM'. --> The remote server returned an error: (401) Unauthorized.

 

 

IIS Logs
---------
2019-08-28 07:19:05 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 2390
2019-08-28 07:19:10 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 0
2019-08-28 07:19:10 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074252 46

 

Issue: authentication scheme mismatch (EXO requires Negotiate / NTLM, on-premises gives us Basic only)

 

Message         : The connection to the server 'mail.contoso.com' could not be completed
ErrorDetail      : Microsoft.Exchange.MailboxReplicationService.RemotePermanentException
The Mailbox Replication Service was unable to connect to the remote server using the credentials provided. Please check the credentials and try again. The call to 'https://mail.contoso.com/EWS/mrsproxy.svc failed. Error details: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Basic Realm="mail.contoso.com"'. --> The remote server returned an error: (401) Unauthorized.

 

First, make sure that NTLM is enabled on the EWS virtual directory. You would check with Get-WebServicesVirtualDirectory |FL cmdlet if NTLM is present in the Authentication Methods. You should also double check in IIS Manager, to make sure that the Negotiate provider is present as well under Windows Authentication. Negotiate provider is falling back to NTLM if Kerberos is not possible and with external clients that is always the case. So, Exchange Online MRS requires NTLM inside the Negotiate Provider on the EWS on-premises virtual directory.

If you are using Azure AD App Proxy as Reverse Proxy for your MRSProxy servers, then you should be aware that there is a limitation of AADAP that cannot present both Negotiate and NTLM providers in the WWW-Authenticate header, no matter the order of the providers in IIS. If you have this setup, you can remove NTLM provider from Windows Authentication on EWS in IIS Manager (leave only Negotiate provider that does also NTLM) or bypass AADAP.
If that is fine (configuration on Exchange and IIS side), then you would check the publishing rules for EWS on your firewall/reverse proxy.

You can also use this mini PowerShell script to check the Authentication Methods advertised by your on-premises (replace the URL with your on-premises MRSProxy /EWS namespace).  It is also a good practice to check the WWW-Authenticate headers both from external and internal PC and notice if there are differences in the output (for example you see Basic, Negotiate, NTLM from internal – all ok but you see only Basic from external – not ok).

 

$req = [System.Net.HttpWebRequest]::Create("https://mail.contoso.com/ews/MRSProxy.svc")
$req.UseDefaultCredentials = $false
$req.GetResponse()
# Expected error: Exception calling "GetResponse" with "0" argument(s):
# "The remote server returned an error: (401) Unauthorized."
$ex = $error[0].Exception
$resp = $ex.InnerException.Response
$resp.Headers["WWW-Authenticate"]

 

Scenario 4: Test-MigrationServerAvailability fails with SSL / TLS error

 

Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server'hybrid.contoso.com' could not be completed. --->
Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: The Mailbox Replication Service could not connect to the remote server because the certificate is invalid. The call to 'https://hybrid.contoso.com/EWS/mrsproxy.svc' failed. Error details: Could not establish trust relationship for the SSL/TLS secure channel with authority 'hybrid.contoso.com'. -->The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. --> The remote certificate is invalid according to the validation procedure.

 

Whenever you see SSL/TLS errors, you would check the following:

 

TLS1.2 should be enabled in the on-premises infrastructure

For Classic hybrid, you can use this PowerShell command while logged to your Office 365 Exchange Online tenant to test the network request on TLS1.2 protocol and see if you also get a SSL/TLS error here:
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12;Invoke-WebRequest https://<endpoint FQDN>/ews/mrsproxy.svc -Verbose

 

For Modern Hybrid, you can check Test-HybridConnectivity which also checks for TLS1.2 and this is mandatory step when configuring / installing the Hybrid Agent. See Verifying Connectivity section from here.

 

Make sure you are not doing SSL offloading for MRSProxy.svc

For Classic hybrid, a valid 3rd party valid certificate is required for EWS. See this and this.

You can use the following command in Exchange Management Shell to quickly check the Exchange certificates and some properties on them from all your Exchange servers in the organization:

 

Foreach ($i in (Get-ExchangeServer)) {Write-Host $i.FQDN; Get-ExchangeCertificate -Server $i.Identity | FT Thumbprint, Status, RootCAtype, Services, Subject}

 

Also, you should be able to access the CRLs for the certificate.

Scenario 5: Test-MigrationServerAvailability fails with 503 Service Unavailable

 

The call to 'https://<GUID> resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP service located at https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. --> The remote server returned an error: (503) Server Unavailable

 

For this error, best is to check HTTPProxy logs for EWS and see if it tried to proxy to a server that is unavailable, for example Server Wide Offline in Get-ServerComponentState or maybe EWS application pool is not started or MRS service is stopped. One thing to add here is that, if you have Exchange 2013/2016 in coexistence with Exchange 2010, the Exchange 2013/ 2016 servers will proxy to the same version server (2013/2016) and will not proxy down to Exchange 2010.

This brings us to the end of this post! Hope you find this useful if you want to learn a bit more about how migration endpoints work, or you need to troubleshoot this area.

I wanted to thank Brad Hughes, Jason Nelson, Nino Bilic and Greg Taylor for their review of this post.

Mirela Buruiana

4 Comments

Awesome article, as usual. Love the attention to detail. Someone give this gal a cookie!

Microsoft

Thanks, @Vasil Michev , cookie sounds good. I like beer /wine too :smile:

New Contributor

Hello,

 

Everything is working fine on my hybrid environment except the Hybrid remote move endpoint with Autodiscover

 

Do you know what to verify and how the best way to debug ?

 

Message : AutoDiscover failed with a configuration error: The migration service failed to detect the migration endpoint using the Autodiscover
service. Consider using the Exchange Remote Connectivity Analyzer (https://testexchangeconnectivity.com) to diagnose the connectivity issues.
SupportsCutover : False
ErrorDetail : internal error:Microsoft.Exchange.Migration.AutoDiscoverFailedConfigurationErrorException: AutoDiscover failed with a configuration error:
The migration service failed to detect the migration endpoint using the Autodiscover service. Consider using the Exchange Remote
Connectivity Analyzer (https://testexchangeconnectivity.com) to diagnose the connectivity issues.
at Microsoft.Exchange.Migration.DataAccessLayer.MigrationEndpointBase.InitializeFromAutoDiscover(MigrationEndpoint presentationObject,
SmtpAddress emailAddress, Boolean acceptUntrustedCertificates)
at Microsoft.Exchange.Management.Migration.MigrationService.Endpoint.TestMigrationServerAvailability.InternalProcessExchangeRemoteMoveAuto
Discover()

Microsoft

@benoitrando , as indicated in the error message, you have an Autodiscover issue, you need to use exrca.com (https://testexchangeconnectivity.com) website and perform Outlook Autodiscover test for the Email Address used in Test-MigrationServerAvailability to understand what is the cause of the Autodiscover (maybe your Exchange certificate is not trusted).
Also, check if you have an error for the second option:  2. Hybrid remote move endpoint without Autodiscover (testing EWS directly, bypassing AutoD)