Troubleshooting Hybrid Migration Endpoints in Classic and Modern Hybrid
Published Oct 25 2019 09:04 AM 160K Views

In our previous blog post we covered an overview of what migration endpoints are, how to find them and what makes them tick. In this post, we will cover related troubleshooting. Note that this post has some in-depth troubleshooting steps, so it is not necessarily something that you’ll read for fun, but we wanted to make it available for those times when you run into trouble!

Now let’s cover some troubleshooting!

Things that are commonly mis-configured

Before getting to the part where we troubleshoot specific migration endpoint issues, be aware of the following top reasons why a migration endpoint cannot be created:

 

Troubleshooting

Whether you are having trouble creating new migration endpoints in Office 365 Exchange Online or are not able to migrate anymore to or from Exchange Online using an existing migration endpoint, the cmdlet Test-MigrationServerAvailability is your dear friend. Always run this cmdlet in Exchange Online PowerShell, not from on-premises Exchange Management Shell.

We will focus on hybrid migration endpoint issues and the cmdlet syntax for this endpoint type. Below we will show you 3 commands that can help you check for underlying issues or error messages:

1. Hybrid remote move endpoint with Autodiscover

 

Test-MigrationServerAvailability -ExchangeRemoteMove -Autodiscover -EmailAddress user@contoso.com -Credentials (get-credential contoso\administrator)

 

troublhybmig02.jpg

Note: The option with Autodiscover is not used in Modern hybrid as we go directly to EWS server(s) for both Migration Endpoints and Free/Busy configuration (Cloud Intra-Organization Connectors and Organization Relationships have TargetSharingEpr set to the EWS namespace.)

2. Hybrid remote move endpoint without Autodiscover (testing EWS directly)

 

Test-MigrationServerAvailability -ExchangeRemoteMove –RemoteServer mail.contoso.com -Credentials(get-credential contoso\administrator)

 

troublhybmig03.jpg

Note: In Modern hybrid, the RemoteServer is in the format of '<GUID>.resource.mailboxmigration.his.msappproxy.net', where <GUID> is unique for each organization. This is randomly generated and stored encoded base 64 in the OnPremisesOrganization object’s Comment in Exchange Online when you will run Modern HCW and it gets to that configuration part.  This GUID will be stamped on the Migration Endpoint in the RemoteServer value for both Full and Minimal Modern Hybrid Topologies and also on the TargetSharingEpr values for Cloud Intra-Organization Connector / Organization Relationship . You can check the GUID in the HCW log and on Get-MigrationEndpoint / Get-IntraOrganizationConnector / Get-OrganizationRelationship EXO cmdlets.

troublhybmig04.jpg

Cmdlets ran above:

 

Get-IntraOrganizationConnector | fl targetsharingepr
Get-OrganizationRelationship | fl targetsharingepr
Get-MigrationEndpoint
$strdata = (Get-OnPremisesOrganization).comment
$bytes = [Convert]::FromBase64String($strdata)
$ms = New-Object System.IO.MemoryStream(@(,$bytes))
$deflate = New-Object System.IO.Compression.DeflateStream($ms, [System.IO.Compression.CompressionMode]::Decompress)
$reader = New-Object System.IO.StreamReader($deflate)
$text = $reader.ReadToEnd()
$text

 

3. Testing an existing hybrid remote move endpoint

 

Test-MigrationServerAvailability -Endpoint <Identity of the Endpoint>

 

troublhybmig05.jpg

OK, I ran these and found errors; now what?

Let’s walk through some common Test-MigrationServerAvailability errors and how to troubleshoot these in Classic / Modern hybrid.
Tip: Whenever a command returns an error in PowerShell, you should run the command $Error[0].Exception |fl -f to get more details on the exception thrown.

However, test-migrationserveravailability failures are not actually those red errors that you get when a command does not work in PowerShell. So, in this situation, we can use New-MoveRequest command to throw the same error and get the serialized exception from this one.
Test-MigrationServerAvailability which returns the result:

troublhybmig06.jpg

Running the New-MoveRequest command to test migration of a synced user will give me same error message but in “red” and we can then get the serialized exception from it.

troublhybmig07.jpg

Based on these error messages, we gathered most common scenarios.

Scenario 1: Test-MigrationServerAvailability fails because of connectivity / timeout errors

Examples of some of those errors:

 

The call to 'https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' timed out. Error details: The request channel timed out attempting to send after 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.

The call to 'https://mail.contoso.com/EWS/mrsproxy.svc' timed out. Error details: The operation did not complete within the allotted timeout of 00:00:50. The time allotted to this operation may have been a portion of a longer timeout. 

The request channel timed out while waiting for a reply after 00:00:09.9996191. The time allotted to this operation may have been a portion of a longer timeout. The remote server returned an error: (504) Gateway Timeout.

 

Troubleshooting these timeout errors in Modern hybrid:


During the Modern hybrid configuration, you will be asked to input the credentials for the on-premises migration admin – these can be the same credentials inserted in the beginning of HCW or new ones. The on-premises Migration admin credentials are needed by HCW in order to run the Test-MigrationServerAvailability cmdlet and this account can have lesser privileges than the admin account running HCW (which is Organization Management role). Once Modern HCW has tested the on-premises migration server availability, we will know if we are able to create the migration endpoint in Exchange Online or not and also in this step we would see the dynamically generated GUID for your on-premises MRSProxy server(s): https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc.
Supposing that you get a timeout error in HCW when testing migration server availability, you would first search in the HCW log for the Test-MigrationServerAvailability cmdlet executed by HCW, copy-paste the exact command to a Notepad file (or at least make a note of the RemoteServer value) and then connect to Exchange Online PowerShell and run the same command to confirm the error message thrown by HCW.

Connect to Exchange Online PowerShell using an O365 Global Admin account, you can use for example this one line command:

 

Import-PSSession $(New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Authentication Basic –AllowRedirection -Credential $(Get-Credential))

 

Once you are successfully connected to EXO PS, run the same command that HCW ran in the log file to see if you get same error or different one, providing the credentials for the on-premises migration admin and your RemoteServer value:

 

Test-MigrationServerAvailability -ExchangeRemoteMove: $true -RemoteServer '<GUID>.resource.mailboxmigration.his.msappproxy.net' -Credentials (Get-Credential domain\admin)

 

Most likely you will get same result as HCW did when you will run the same command in PowerShell. Although this step may seem redundant, it is always a good idea to analyze the command ran by HCW (if you selected Classic Hybrid option during HCW, then HCW should not try to create the migration endpoint to <GUID>.resource.mailboxmigration.his.msappproxy.net which is for Modern Hybrid topology) and confirm at the same time that the error was not for example a transient one.

Once you confirm that you still get the error that HCW was complaining about and you are running Modern Hybrid Mode, you would need to investigate it.

On the server where you ran the Modern HCW, you need to check the Hybrid Agent Status (ACTIVE or INACTIVE). There are two ways to do this:

  1. Re-run Modern HCW and check it in the GUI, reference here.
  2. Through PowerShell as described below, reference here.

NOTE: Make sure you have installed Azure PowerShell Module before proceeding further.
Import the Hybrid Management Module in Windows PowerShell (ran as Administrator) from \Program Files\Microsoft Hybrid Service\ directory:

 

Import-Module .\HybridManagement.psm1

 

Run the Get-HybridAgent command:

 

Get-HybridAgent -credential (get-credential)    ## cloud admin credentials

 

Check that the Hybrid Agent(s) is ACTIVE

If INACTIVE, you would check:

  • If you switched from Modern hybrid to Classic hybrid (as this would uninstall the agent). If that is the case – you proceeded with Classic hybrid topology and this successfully uninstalled the Hybrid Agent, then you can ignore this warning thrown by HCW related to the migration endpoint for <GUID>.resource.mailboxmigration.his.msappproxy.net and you should create the migration endpoint in EAC using Autodiscover or your published EWS URL. At the moment of the writing of this blog post, if you switch from Modern to Classic successfully – HCW still tries to create the migration endpoint going to the Hybrid Agent proxy instead of using your published EWS URL.
  • If Hybrid Service is installed on the machine and is up and running and the Hybrid Agent machine itself is running
  • Check all things from Additional Information here to see if the Agent is installed properly

Most important, if Hybrid Agent is ACTIVE but you get the error ‘unable to connect to the server’ in Test-MigrationServerAvailability, check and confirm with Performance Monitor that you see the requests.
If the request counters (for #of requests) go up on the Agent machine when you do Test-MigrationServerAvailability to the Hybrid Agent, the problem is likely on the on-premises server; if they don't, the problem is probably with either the connector or the cloud configuration. If we suspect on-premises (most likely the case), you need to check again the on-premises infrastructure, especially proxy and firewall settings. Several things that might be helpful: install requirements, system requirements, port and protocol requirements.

 

Troubleshooting these errors in Classic hybrid:

 

Check your network devices logs and IIS logs / HTTPProxy logs at the time your run Test-MigrationServerAvailability command, usually if the timeout happens very quick (under 50 sec) it could probably be a network device that is blocking / closing the connection.

Location of IIS and HTTPProxy logs:

  • IIS logs Default Web Site (DWS): %SystemDrive%\inetpub\logs\LogFiles\W3SVC1

Example: C:\inetpub\logs\LogFiles\W3SVC1

The name of the IIS logs contains the date of the log, for example u_ex190930.log is from Sept 30, 2019.

  • HTTPProxy logs for EWS:  %ExchangeInstallPath%Logging\HttpProxy\Ews

Example: C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews

The name of the HTTPProxy logs contains the date and hour starting to log, for example HttpProxy_2019093014-10.LOG (10th log from Sept 30, 2019, starting hour 14:00 UTC)

Below we have some examples of entries from IIS logs when we get a successful request (200 OK) and a failed request (500) that could correspond with a timeout error in Test-MigrationServerAvailability if the request reached IIS / Exchange Server.

 

IIS logs - 200 OK
2019-08-28 06:57:38 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 0
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 15
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contoso\administrator 4.4.0.1 - 200 0 0 125
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contoso\administrator 4.4.0.1 - 200 0 0 296

IIS logs - 500 error
2019-08-28 07:15:48 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 2 5 4890
2019-08-28 07:15:52 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 0
2019-08-28 07:15:55 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contoso\administrator 4.4.0.1 - 500 0 0 2562
2019-08-28 07:15:55 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contoso\administrator 4.4.0.1 - 500 0 0 93

 

If you don’t see the failed requests in IIS logs, make sure you allow all Exchange Online IP addresses to connect to your on-premises and check firewall logs to see if connections were blocked.

This is an extract from HTTP proxy logs with a 500 error code and a timeout when proxying to another Exchange server:

 

2019-09-30T12:02:55.930Z,a0ff365c-741b-4e59-b2e3-760991d3a27a,15,1,1713,5,,Ews,exch01.contoso.com,/EWS/mrsproxy.svc,,Negotiate,true,contoso\admin,,OrganizationId~OrganizationAnchor@,,40.100.175.55,exch01,500,,ServerLocatorError,POST,,,,,ForestWideOrganization,,,2807,664,,,,1,2819,0;,0,,0,8;2;,10,10,,0,2821,0,,,,,,,,,0,2819,0,,2819,,2820,2820,,,,BeginRequest=2019-09-30T12:02:53.109Z;CorrelationID=<empty>;ProxyState-Run=None;ServerLocatorCall=DM:a10ad628-e020-409e-9f1e-22a700182ac1~~contoso.structure;ProxyState-Complete=CalculateBackEnd;SharedCacheGuard=0;EndRequest=2019-09-30T12:02:55.930Z;S:ServiceCommonMetadata.Cookie=ee4af79a0a144bcaa9a5bc0af6eec215;I32:ADR.C[DC04]=1;F:ADR.AL[DC04]=1.554922;I32:ATE.C[DC04.contoso.local]=2;F:ATE.AL[DC04.contoso.local]=1;I32:ADS.C[DC04]=1;F:ADS.AL[DC04]=2.897117,HttpProxyException=Microsoft.Exchange.HttpProxy.HttpProxyException: Server Locator Service call had a communication error. ---> Microsoft.Exchange.Data.ServerLocator.ServerLocatorClientTransientException: Server Locator Service call had a communication error. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://exch02.contoso.local:64337/Exchange.HighAvailability/ServerLocator. The connection attempt lasted for a time span of 00:00:02.8010812. TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time  or established connection failed because connected host has failed to respond 10.2.2.1::64337.  ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time  or established connection failed because connected host has failed to respond 10.2.2.1::64337    at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)    at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)    at System.ServiceModel.Channels.SocketConnectionInitiator.ConnectAsyncResult.OnConnect(IAsyncResult result)    --- End of inner exception stack trace ---

 

Scenario 2: Test-MigrationServerAvailability fails with 403 Forbidden

 

The connection to the server '<GUID>.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to 'https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP request was forbidden with client authentication scheme 'Negotiate'. --> The remote server returned an error: (403) Forbidden.., The HTTP request was forbidden with client authentication scheme 'Negotiate'., The remote server returned an error: (403) Forbidden.

 

Follow this article, as it is applicable for both Modern and Classic hybrid topologies.

Scenario 3: Test-MigrationServerAvailability fails with 401 Unauthorized, Access denied or Invalid credentials

Check this support article.

Check authentication methods on all EWS virtual directories in IIS and make sure Negotiate provider under Windows Authentication is enabled for all EWS.

Make sure on-premises migration Admin has at least Exchange Recipient Admin permissions (or Recipient Management, depending on the Exchange version). If you are running Modern HCW, usually this is the same on-premises account that has Organization Management rights, but if you change the account, you would see here if the permissions are right or the credentials are correct when we would input account credentials. We recommend that you create (do not copy account) another on-premises account that purely has Exchange Recipient Admin permissions, then test with that account (Test-MigrationServerAvailability):

troublhybmig08.jpg

Do you have devices that pre-authenticate the requests coming from Exchange Online to Exchange on-premises servers (EWS and Autodiscover paths)? If yes, this is not supported. If you are not sure of it, we recommend you temporarily bypass network devices in front of Exchange Servers, allow direct access to Exchange Servers and see if same error when doing Test-MigrationServerAvailability

When you run Test-MigrationServerAvailability, make a note of the timestamp when you get the error, then check IIS logs on each Exchange Client Access Server (logs are UTC timezone) around the exact time when Test-MigrationServerAvailability has been ran (HH:MM:SS) and check entries for mrsproxy.svc and see the statuses mentioned. Normally, there following are the first 3 IIS entries for 1 successful request (Test-MigrationServerAvailability):

 

Successful
2019-08-28 06:57:38 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 0
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 15
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 miry\administrator 4.4.0.1 - 200 0 0 125

 

Some other error examples:

Issue: wrong credentials of migration admin

 

Message            : The connection to the server 'mail.contoso.com' could not be completed.
ErrorDetail        : Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The Mailbox Replication Service was unable to connect to the remote server using the credentials provided. Please check the credentials and try again. The call to 'https://mail.contoso.com/EWS/mrsproxy.svc' failed. Error details: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate,NTLM'. --> The remote server returned an error: (401) Unauthorized.

 

 

 

IIS Logs
---------
2019-08-28 07:19:05 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 2390
2019-08-28 07:19:10 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 0
2019-08-28 07:19:10 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074252 46

 

Issue: authentication scheme mismatch (EXO requires Negotiate / NTLM, on-premises gives us Basic only)

 

Message         : The connection to the server 'mail.contoso.com' could not be completed
ErrorDetail      : Microsoft.Exchange.MailboxReplicationService.RemotePermanentException
The Mailbox Replication Service was unable to connect to the remote server using the credentials provided. Please check the credentials and try again. The call to 'https://mail.contoso.com/EWS/mrsproxy.svc failed. Error details: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Basic Realm="mail.contoso.com"'. --> The remote server returned an error: (401) Unauthorized.

 

First, make sure that NTLM is enabled on the EWS virtual directory. You would check with Get-WebServicesVirtualDirectory |FL cmdlet if NTLM is present in the Authentication Methods. You should also double check in IIS Manager, to make sure that the Negotiate provider is present as well under Windows Authentication. Negotiate provider is falling back to NTLM if Kerberos is not possible and with external clients that is always the case. So, Exchange Online MRS requires NTLM inside the Negotiate Provider on the EWS on-premises virtual directory.

If you are using Azure AD App Proxy as Reverse Proxy for your MRSProxy servers, then you should be aware that there is a limitation of AADAP that cannot present both Negotiate and NTLM providers in the WWW-Authenticate header, no matter the order of the providers in IIS. If you have this setup, you can remove NTLM provider from Windows Authentication on EWS in IIS Manager (leave only Negotiate provider that does also NTLM) or bypass AADAP.
If that is fine (configuration on Exchange and IIS side), then you would check the publishing rules for EWS on your firewall/reverse proxy.

You can also use this mini PowerShell script to check the Authentication Methods advertised by your on-premises (replace the URL with your on-premises MRSProxy /EWS namespace).  It is also a good practice to check the WWW-Authenticate headers both from external and internal PC and notice if there are differences in the output (for example you see Basic, Negotiate, NTLM from internal – all ok but you see only Basic from external – not ok).

 

$req = [System.Net.HttpWebRequest]::Create("https://mail.contoso.com/ews/MRSProxy.svc")
$req.UseDefaultCredentials = $false
$req.GetResponse()
# Expected error: Exception calling "GetResponse" with "0" argument(s):
# "The remote server returned an error: (401) Unauthorized."
$ex = $error[0].Exception
$resp = $ex.InnerException.Response
$resp.Headers["WWW-Authenticate"]

 

Scenario 4: Test-MigrationServerAvailability fails with SSL / TLS error

 

Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server'hybrid.contoso.com' could not be completed. --->
Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: The Mailbox Replication Service could not connect to the remote server because the certificate is invalid. The call to 'https://hybrid.contoso.com/EWS/mrsproxy.svc' failed. Error details: Could not establish trust relationship for the SSL/TLS secure channel with authority 'hybrid.contoso.com'. -->The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. --> The remote certificate is invalid according to the validation procedure.

 

Whenever you see SSL/TLS errors, you would check the following:

 

TLS1.2 should be enabled in the on-premises infrastructure

For Classic hybrid, you can use this PowerShell command while logged to your Office 365 Exchange Online tenant to test the network request on TLS1.2 protocol and see if you also get a SSL/TLS error here:
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12;Invoke-WebRequest https://<endpoint FQDN>/ews/mrsproxy.svc -Verbose

 

For Modern Hybrid, you can check Test-HybridConnectivity which also checks for TLS1.2 and this is mandatory step when configuring / installing the Hybrid Agent. See Verifying Connectivity section from here.

 

Make sure you are not doing SSL offloading for MRSProxy.svc

For Classic hybrid, a valid 3rd party valid certificate is required for EWS. See this and this.

You can use the following command in Exchange Management Shell to quickly check the Exchange certificates and some properties on them from all your Exchange servers in the organization:

 

 

 

Foreach ($i in (Get-ExchangeServer)) {Write-Host $i.FQDN; Get-ExchangeCertificate -Server $i.Identity | FT Thumbprint, Status, RootCAtype, Services, Subject}

 

 

 

Also, you should be able to access the CRLs for the certificate.

Scenario 5: Test-MigrationServerAvailability fails with 503 Service Unavailable

 

 

 

The call to 'https://<GUID> resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP service located at https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. --> The remote server returned an error: (503) Server Unavailable

 

 

 

For this error, best is to check HTTPProxy logs for EWS and see if it tried to proxy to a server that is unavailable, for example Server Wide Offline in Get-ServerComponentState or maybe EWS application pool is not started or MRS service is stopped. One thing to add here is that, if you have Exchange 2013/2016 in coexistence with Exchange 2010, the Exchange 2013/ 2016 servers will proxy to the same version server (2013/2016) and will not proxy down to Exchange 2010.

Scenario 6: Test-MigrationServerAvailability fails with 400 Bad Request

There was no endpoint listening at https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. --> The remote server returned an error: (400) Bad Request.

For both Modern and Classic Hybrid Configurations, check if you see the corresponding 400 entries both in IIS logs and HTTPProxy logs (for ews/mrsproxy.svc) when running Test-MigrationServerAvailability command for ExchangeRemoteMove.

Example of a 400 entry in IIS logs:

2020-02-20 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 miry\administrator 4.4.0.1 - 400 0 0 125

If you see it in IIS logs, check HTTPProxy logs (if Exchange 2013 or higher) and Failed Request Traces in IIS to get more info on the error. See here how to enable the former ones. You can narrow down the statuses between 400-499 as an example for this situation.

If you don’t see the 400 entries in IIS logs, this means the request from Exchange Online didn’t reach the IIS Server / Exchange CAS. You would then check HTTPerr logs (HTTP.sys would be before IIS from a routing point of view), these are found in %SystemRoot%\System32\LogFiles\HTTPERR.

Example of a 400 entry in HTTPerr log:

2020-02-20 12:06:34 192.168.2.50 41743 192.168.2.50 443 HTTP/1.1 POST /EWS/mrsproxy.svc 400 - Hostname -

For this particular error, where we have the keyword “hostname”, you would check the IIS bindings, if there are hostnames populated for HTTPS and HTTP (screenshot below) , if we have HTTP redirection or URL Rewrite in place that would alter the host headers.

hybscenario6_1.jpgIf you don’t see the 400 entries in the HTTPerr logs either, then likely the request didn’t reach the Exchange Server at all. You would check the network devices that are in front of the Exchange Servers (firewall, reverse proxy, load balancer).

In case of Modern Hybrid Configuration, you would check if the Hybrid Connector is up and running. You can check its status in HCW GUI or using the PowerShell method.

If the Hybrid Agent is inactive, then check if the Hybrid Service is installed and running, reference here. You would install the agent automatically via HCW when choosing Modern Hybrid Topology or manually.

The Agent would be uninstalled if for example you switched from Modern to Classic Hybrid Topology or manually uninstall of the Microsoft Hybrid Service in Programs and Feature.

If you didn’t uninstall the Microsoft Hybrid Service and the service is started and running, then you would need to check the Hybrid Service logs. Describing below on how to enable this type of logging and where to get the logs from.

To enable the logging, you would have to navigate to Hybrid Service installation path, for example C:\Program Files\Microsoft Hybrid Service. In this folder, there is a config file of the Hybrid Service called Microsoft.Online.EME.Hybrid.Agent.Service.exe.config. You would run the Notepad.exe program as Administrator and then open this config file to edit it. Remove the <!—and --> characters (uncomment the XML comments) from the config file and save it. The file should look like this:

hybscenario6_2.jpg

Restart the Microsoft Hybrid Services in services.msc. Follow the procedure from here to attempt to reproduce the problem. Navigate to these 2 folders and check the HybridService logs:

  • C:\programdata\Microsoft Hybrid Service\Logging will have logs related to connector registration and startup.
  • C:\programdata\Microsoft\Microsoft Hybrid Service\Trace will have logs regarding application requests.

If the Hybrid Agent is active, then you would check with performance monitor that the #requests go up, as mentioned here. If they go up, this means the Hybrid Agent is fine and that the Exchange Online Request reached the Hybrid Connector. You would further need to check the connector route to make sure it is able to reach Exchange /IIS server.  The Hybrid Connector would be pointing to your Exchange CAS or your load balancer. You can see this in Get-HybridApplication cmdlet where appId is your Remote Server in Test-MigrationServerAvailability cmdlet and the credentials are the Office 365 Global Admin. See below example of the cmdlet:

hybscenario6_3.jpg

 

From the Hybrid Connector machine, you would check if you are able to resolve the internalURL FQDN to the internal IP of your Exchange Server. For testing purposes, try running Update-HybridApplication and point it to a single Exchange Server instead of a load balancer and check if it still fails. If, after pointing to the Exchange Server, you still get the 400 requests, go check HTTPerr, IIS, HTTPProxy and Event viewer logs on that Exchange server and see if you can track the request.

This brings us to the end of this post! Hope you find this useful if you want to learn a bit more about how migration endpoints work, or you need to troubleshoot this area.

I wanted to thank Brad Hughes, Jason Nelson, Nino Bilic and Greg Taylor for their review of this post.

Mirela Buruiana

42 Comments

Awesome article, as usual. Love the attention to detail. Someone give this gal a cookie!

Microsoft

Thanks, @Vasil Michev , cookie sounds good. I like beer /wine too :smile:

Brass Contributor

Hello,

 

Everything is working fine on my hybrid environment except the Hybrid remote move endpoint with Autodiscover

 

Do you know what to verify and how the best way to debug ?

 

Message : AutoDiscover failed with a configuration error: The migration service failed to detect the migration endpoint using the Autodiscover
service. Consider using the Exchange Remote Connectivity Analyzer (https://testexchangeconnectivity.com) to diagnose the connectivity issues.
SupportsCutover : False
ErrorDetail : internal error:Microsoft.Exchange.Migration.AutoDiscoverFailedConfigurationErrorException: AutoDiscover failed with a configuration error:
The migration service failed to detect the migration endpoint using the Autodiscover service. Consider using the Exchange Remote
Connectivity Analyzer (https://testexchangeconnectivity.com) to diagnose the connectivity issues.
at Microsoft.Exchange.Migration.DataAccessLayer.MigrationEndpointBase.InitializeFromAutoDiscover(MigrationEndpoint presentationObject,
SmtpAddress emailAddress, Boolean acceptUntrustedCertificates)
at Microsoft.Exchange.Management.Migration.MigrationService.Endpoint.TestMigrationServerAvailability.InternalProcessExchangeRemoteMoveAuto
Discover()

Microsoft

@benoitrando , as indicated in the error message, you have an Autodiscover issue, you need to use exrca.com (https://testexchangeconnectivity.com) website and perform Outlook Autodiscover test for the Email Address used in Test-MigrationServerAvailability to understand what is the cause of the Autodiscover (maybe your Exchange certificate is not trusted).
Also, check if you have an error for the second option:  2. Hybrid remote move endpoint without Autodiscover (testing EWS directly, bypassing AutoD)

Copper Contributor

We are using Exchange Hybrid Agent and when we run the Wizard it fails with the below error and I ran the Test-migrationserveravailability command from EoL PowerShell and it brought back the same error.

 

Any help to fix this error is appreciated.

 

PS C:\Users\murali> Test-MigrationServerAvailability -ExchangeRemoteMove -RemoteServer GUID.resource.mailboxmigration.his.msappproxy.net -Credentials(get-credential)
cmdlet Get-Credential at command pipeline position 1
Supply values for the following parameters:
Credential

RunspaceId      : ID
Result          : Failed
Message         : The connection to the server 'GUID.resource.mailboxmigration.his.msappproxy.net' could not be completed.
SupportsCutover : False
ErrorDetail     : Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server
                  'GUID.resource.mailboxmigration.his.msappproxy.net' could not be completed. --->
                  Microsoft.Exchange.MailboxReplicationService.MRSRemoteTransientException: The call to
                  'https://GUID.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed because no service was listening on the specified
                  endpoint. Error details: There was no endpoint listening at https://GUID.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc
                  that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. --> The remote name could
                  not be resolved: 'GUID.resource.mailboxmigration.his.msappproxy.net' --->
                  Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: There was no endpoint listening at
                  https://GUID.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc that could accept the message. This is often caused by an
                  incorrect address or SOAP action. See InnerException, if present, for more details. ---> Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: The
                  remote name could not be resolved: 'GUID.resource.mailboxmigration.his.msappproxy.net'
                     --- End of inner exception stack trace ---
                     --- End of inner exception stack trace ---
                     at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.<>c__DisplayClass106_0.<ReconstructAndThrow>b__0()
                     at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(Action operation)
                     at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.ReconstructAndThrow(String serverName, VersionInformation serverVersion)
                     at Microsoft.Exchange.Connections.Common.WcfClientWithFaultHandling`2.<>c__DisplayClass4_0.<CallService>b__0()
                     at Microsoft.Exchange.Net.WcfClientBase`1.CallService(Action serviceCall, String context)
                     at Microsoft.Exchange.Connections.Common.WcfClientWithFaultHandling`2.CallService(Action serviceCall, String context)
                     at Microsoft.Exchange.MailboxReplicationService.WcfClientWithVersion`2.CallService(Action serviceCall, String context)
                     at Microsoft.Exchange.Migration.MigrationExchangeProxyRpcClient.CanConnectToMrsProxy(Fqdn serverName, Guid mbxGuid, NetworkCredential credentials, LocalizedException&
                  error)
                     --- End of inner exception stack trace ---
                     at Microsoft.Exchange.Migration.DataAccessLayer.ExchangeRemoteMoveEndpoint.VerifyConnectivity()
                     at Microsoft.Exchange.Management.Migration.MigrationService.Endpoint.TestMigrationServerAvailability.InternalProcessEndpoint(Boolean fromAutoDiscover)
TestedEndpoint  :
IsValid         : True
Identity        :
ObjectState     : New
Microsoft

Hi @muraly005, first I assume that the Guid is an actual number and you replaced it for privacy concerns?

Second, it could be a DNS propagation issue (about 15 min) or DNS caching issue in your org. If issue persists, open a support case with us so that we can look into it. Provide HCW logs (entire logging folder).

Copper Contributor

Hi @Mirela_Buru, thanks for the quick reply.

 

Yes, I have removed the actual number with GUID.

 

I have already logged the support case as well, but when I saw this article, I thought of mentioning the issue I am facing as well.

 

Is that the DNS cache on the Hybrid Agent server?

 

Is the connection to GUID.resource.mailboxmigration.his.msappproxy.net is initiated from EoL to Azure or EoL to on-prem Exchange?

 

 

Microsoft

Yes, could be DNS caching issue on the on-premises side, in the on-premises org. You can check nslookup or resolve-dnsname for GUID.resource.mailboxmigration.his.msappproxy.net in both external and internal and see if any issues with the name resolution.

For the second, test-migrationserveravailability is from EXO to on-premises Exchange/hybrid agent.

Microsoft

@muraly005 , just following-up on this issue, has this been identified as a problem on our side? If yes, was it fixed by our Engineering team?

Copper Contributor

@Mirela_Buru, not yet, last update to me was "we have engaged our backend team to work on the same".

 

Also when I try to resolve using the DNS name both internally and externally, I couldn't resolve it. 

Microsoft

@muraly005 , you can maybe share the ticket number and I will take a look into it. 

Copper Contributor

@Mirela_Buru, sure, the ticket is Ticket #:17476789

Microsoft

@muraly005 , thanks! As per our records, this DNS issue was related to a service incident that was restored recently. Can you please confirm if this has been solved for you as well?

Copper Contributor

@Mirela_Buru, I was able to resolve the GUID now and was able to complete the HCW now. Are we able to know what the DNS issue was? Thanks.

 

I do have issues now of email not receiving from on-prem mailbox account to Office 365 mailbox account and migrating mailbox from on-prem to office 365, doing some troubleshooting at the moment.

Microsoft

@muraly005 , glad to hear initial issue is fixed. Regarding the root cause, this was published on the Service Incident:
Root cause: The authentication method associated with tenant instance routing for migrating mailboxes in this scenario was configured incorrectly.
Next steps: We're reviewing our Domain Name System routing configurations to gain a better understanding of what caused the instance routing to become initially misconfigured.
For the mailflow issue, this should be treated separately as SMTP traffic doesn't traverse Hybrid Agent. If issue is on-prem to cloud, you would need to enable protocol logging on SMTP send connector and check SMTP Send logs. Additionally, checking message tracking or event viewer logs might help. As common causes: network devices in between, on-premises remote mailbox object missing user@tenant.mail.onmicrosoft.com target address or this missing as a proxy address on the cloud mailbox, certificate or TLS issues etc.
If you are having issues onboarding the mailbox to O365 but Migration Endpoint was created successfully (as original issue has been fixed), you would need to check the Migration Reports (Get-MigrationUserStatistics <user> -IncludeReport -DiagnosticInfo Verbose |FL and Get-MoveRequestStatistics <user> -IncludeReport -DiagnosticInfo Verbose |FL see where exactly it is failing. If a move request is not created, then most likely it fails migration user validation. Again, could be user@tenant.mail.onmicrosoft.com proxy address missing or maybe another smtp domain on the user that is not added as accepted domain in exo, etc.

Copper Contributor

@Mirela_Buru, Thanks for you help in troubleshooting the issues. Mail flow and migration are working.

 

At the moment, troubleshooting the Outlook client prompting for password and not connecting to Exchange for the mailbox migrated to O365

Microsoft

@muraly005 , you're welcome. For OLK prompts, check the suggestions from https://social.technet.microsoft.com/wiki/contents/articles/37418.office-365-credential-prompts-afte... , especially #3. If this doesn't fix the issue, you would need to provide Outlook logs when reproducing issue: https://support.office.com/en-us/article/What-is-the-Enable-logging-troubleshooting-option-0fdc446d-...

Also, I think SARA tool has a scenario for Outlook credentials prompts / keeps disconnecting, feel free to use it: https://support.office.com/en-us/article/About-the-Microsoft-Support-and-Recovery-Assistant-e90bb691... 

Microsoft

Hi all, my name is Mirela Buruiana and I am the main author of this blog. Reply to this comment if you want me to get notified about your question /comment.

Brass Contributor

Mirela, I've seen Exchange 2010 environments throw:

 

The remote server returned an error: (401) Unauthorized.. --> The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate,NTLM,Basic realm="mail.contoso.com"'.

when in fact the credentials and auth headers are both correct (fiddler, powershell). Some moves complete, others don't. Are there other circumstances, such as Exchange 2010 performance issues that would cause this? I've rules out load balancer affinity by pointing all migration traffic at a single server, but still, some complete and others seem to refuse to complete.

Microsoft

@Mike Crowley , 401 is typically pre-authentication or bad credentials. If it happens intermittently, could be the credentials which expired or the admin account getting locked out.  Would need to check IIS logs (the entries with 401) + FREB logs (for 401 status code) when this happens. You have FailureTimestamp and FailureType on the Get-MoveRequestStatistics, if you are able to start the move but it fails at some point with 401 Unauthorized.

You mentioned some complete, some don't. Look for differences: are those batches using the same SourceEndpoint? does Test-MigrationServerAvailability -Endpoint "" work when 401 is given in the migration batch? Or maybe the functional ones were created directly with new-moverequest from powershell, without using a batch/endpoint and different migration admin credentials.

You can open a support case with us and provide here the case number when you have a repro.

Microsoft

For example, in the extract, we see these 3 entries when Migration Admin couldn't log in, whereas in successful one third is for miry\administrator (successful login). Do you see such entries in IIS logs WHEN 401 is given? Where exactly is the 401 given (endpoint/ move request)?

IIS Logs
---------
2019-08-28 07:19:05 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 2390
2019-08-28 07:19:10 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 0
2019-08-28 07:19:10 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074252 46

 

Successful
2019-08-28 06:57:38 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 0
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 15
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 miry\administrator 4.4.0.1 - 200 0 0 125

Brass Contributor

Mirela, thanks for your replies. Some additional detail:

 

  • Yes, same endpoint, same credential, same target server.
  • Sometimes in the same move request, in the same 1hr window, it will flip between unauthorized and not. The AD account is not expiring or getting locked out. Sometimes resubmitting the job seems to help, but not always.
  • Most often, these errors stack up on move requests at 95%, claiming no 401/auth errors until that point. This is why I've started to think the auth issues are really more of a performance issue, and that auth is just being thrown generically.
  • Also, yes in the IIS log, there is a high volume of successful /EWS/mrsproxy.svc entries with the migration account, with no apparent string of repeated 401's or anything like that. When I look at the 401s, they seem to be relate to a new Microsoft IP (perhaps a different move request), but then subsequent entries look healthy.

e.g.

 

$log  Get-Content .\u_ex200429.log
$mrsproxy = $log -match '/EWS/mrsproxy.svc' 
$mrsproxy | select -Last 10

// many more of these above ...
2020-04-29 04:15:24 172.16.16.100 POST /EWS/mrsproxy.svc - 443 contoso\mcrowley 52.96.82.221 - 200 0 0 78
2020-04-29 04:15:24 172.16.16.100 POST /EWS/mrsproxy.svc - 443 contoso\mcrowley 52.96.82.221 - 200 0 0 78
2020-04-29 04:15:24 172.16.16.100 POST /EWS/mrsproxy.svc - 443 contoso\mcrowley 52.96.82.221 - 200 0 0 78
2020-04-29 04:15:24 172.16.16.100 POST /EWS/mrsproxy.svc - 443 contoso\mcrowley 52.96.82.221 - 200 0 0 78
2020-04-29 04:15:25 172.16.16.100 POST /EWS/mrsproxy.svc - 443 contoso\mcrowley 52.96.82.221 - 200 0 0 78
2020-04-29 04:15:25 172.16.16.100 POST /EWS/mrsproxy.svc - 443 contoso\mcrowley 52.96.82.221 - 200 0 0 62
2020-04-29 04:15:25 172.16.16.100 POST /EWS/mrsproxy.svc - 443 contoso\mcrowley 52.96.82.221 - 200 0 0 62
2020-04-29 13:40:55 172.16.16.100 POST /EWS/mrsproxy.svc - 443 - 174.218.149.150 - 401 0 0 0
2020-04-29 13:41:00 172.16.16.100 POST /EWS/mrsproxy.svc - 443 - 174.218.21.141 - 401 1 2148074254 31
2020-04-29 13:41:00 172.16.16.100 POST /EWS/mrsproxy.svc - 443 - 52.96.18.221 - 401 1 2148074248 15
//more 200's below

 

 

Microsoft
@Mike, just try to focus on one 401 failure in the move report,  the most recent FailureTimestamp in $stats.report.failures |FT failuretimestamp,failuretype where $stats = get-moverequeststatistics <user> -includereport.
These should be timezone of the EXO admin, at least I think so.  Then focus on all mrsproxy.svc entries around that HH:MM:SS but UTC timezone in IIS logs.  If you only have 200s at that exact failure time, the request didn't reach Exchange. If the request did reach Exchange at that time, then enable FREB with the rule for 401 status code, resume the failed move request, wait to fail with 401 and get again the IIS logs + FREB logs around that exact time.  
You can also check $stats.report.Entries | % {[string] $_} > entries.txt and let me know if something odd there.
P.S. The IPs with 174.x are not EXO, at least from my internal tool.
Copper Contributor

Hi Mirela,

Following your recommendation, I'm getting this error:

Invoke-WebRequest : The request was aborted: Could not create SSL/TLS secure channel.

 

The original problem is that the migration from on-prem to ExO fails.

No changes to the firewall or the cert, or anything else, to my knowledge.

 

652adbf8-8e94-4336-a08f-9c60bd4ce40a
Result : Failed
Message : The connection to the server '<server address>' could not be completed.
SupportsCutover : False
ErrorDetail : Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server '<server address>' could not be completed. --->
Microsoft.Exchange.MailboxReplicationService.MRSRemoteTransientException: The call to 'https://<server address>/EWS/mrsproxy.svc' failed. Error
details: Could not establish secure channel for SSL/TLS with authority '<server address>'. --> The request was aborted: Could not create SSL/TLS
secure channel.. ---> Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: Could not establish secure channel for SSL/TLS
with authority '<server address>'. ---> Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: The request was aborted: Could not create SSL/TLS secure channel.

Microsoft

@YuriyGK - it looks like you've found your problem. As mentioned above, your server isn't able to be contacted using TLS 1.2.  You really need to update/fix whatever network device or computer is terminating your SSL/TLS sessions.  This will prepare you for the future when TLS 1.0 and TLS 1.1 are completely turned off in Exchange Online.  All that being said, they still should currently be enabled.  We're investigating some similar reports right now and are making a change to hopefully mitigate those reports.  However, officially Microsoft hasn't supported TLS 1.0 or TLS 1.1 since October 31 2018, so you need to get your environment supporting TLS 1.2 as soon as possible.  See this link for more information: https://docs.microsoft.com/en-us/microsoft-365/compliance/technical-reference-details-about-encrypti...

Copper Contributor

@bradhugh thank you. The peculiar thing is it’s only happening to me, and I’m the Exch admin. Other admin, who is also a global admin, has no problems. Thoughts?

Copper Contributor

Alright. Turns out it was a freaky transient kind-of crap.  The global admin took my Exchange admin right away for 30 minutes, and after giving them back, everything is working again. I have no explanation for why or how - just a matter of fact. Thanks @bradhugh for the reply.

Copper Contributor

We eventually figured this out with enough internet searches and log-mining:

 

Issue Description:

Unable to complete the Hybrid Configuration Agent with error below:

The call to 'https://<someGUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP request was forbidden with client authentication scheme 'Negotiate'. 

 

Case resolution:

You resolved the issue after ran the HCW using an account with a limited amount of group memberships. **************[This made the difference - group membership of 2, in our case)

 

Disabling and re-enabling MRSProxy.

IIS RESET in all virtual directories of CAS servers.

Copper Contributor

FYI - When attempting Test-HybridConnectivity, the cmdlet fails with an SSL error. The certificate used for the endpoint (https://aadap-portcheck-seaus.connectorporttest.msappproxy.net:8080) is expired since 12/03/2020.

Test-HybridConnectivity -testO365Endpoints
Testing connection to mscrl.microsoft.com on port 80
Testing connection to crl.microsoft.com on port 80
Testing connection to ocsp.msocsp.com on port 80
Testing connection to www.microsoft.com on port 80
Testing connection to login.windows.net on port 443
Testing connection to login.microsoftonline.com on port 443
Testing connection to aadap-portcheck-seaus.connectorporttest.msappproxy.net on port 8080
Performing GET on https://aadap-portcheck-seaus.connectorporttest.msappproxy.net:8080
Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS
secure channel.

 

Screen Shot 2020-08-13 at 4.59.31 pm.png

Brass Contributor

After months of calls to Microsoft, disabling, Remove-HybridConfiguration, rerunning, getting sidetracked, coming back to it, everything seems to work except we still get this HCW error message, researching more online, etc. etc. etc. (we all know how it goes), I finally have the answer to my problem: Ignore it, it's not a problem, it's a bug.

 

"If you switched from Modern hybrid to Classic hybrid (as this would uninstall the agent). If that is the case – you proceeded with Classic hybrid topology and this successfully uninstalled the Hybrid Agent, then you can ignore this warning thrown by HCW related to the migration endpoint for <GUID>.resource.mailboxmigration.his.msappproxy.net and you should create the migration endpoint in EAC using Autodiscover or your published EWS URL. At the moment of the writing of this blog post, if you switch from Modern to Classic successfully – HCW still tries to create the migration endpoint going to the Hybrid Agent proxy instead of using your published EWS URL."

 

Thanks @Mirela, I will sleep better tonight : )

Microsoft

@Kevin_Davis , I was actually expecting this to have been fixed since when I wrote this.
2 things here:
1) Make sure you are running the latest HCW version (17.x), you get it from aka.ms/hybridwizard

2) Make sure we are indeed selecting Classic Hybrid Topology in GUI. 

If you still see this issue in the above conditions, please send the PSR.exe Capture (screenshots to see the GUI part) and HCW log in a support case.

Provide me with the case number so that I can follow up on that. 

Brass Contributor

Hi @Mirela_Buru ,

Thanks for the reply. I can confirm the HCW was v17.0.4544.0, and I was indeed selecting the classic topology. Another support tech had previously selected modern topology, but was instructed by MS support to change that as a result of a different error message we were getting. It is therefore true that we did previously use modern topology, then changed to classic, as you describe in the article. At that time we were also instructed to run Remove-HybridConfiguration on the on-prem server, and we did not select modern topology again since doing that.

The server in question is a client production system so I'm not keen to make changes for the sake of it, and if I do need to make changes then I need to plan a server maintenance window for a weekend. If you would like to see the results of the process recorder then I can plan to run the HCW in a couple of weeks time, and come back to you. Otherwise, I'm happy to let it lie (until it causes a problem). I did take a screenshot of the final screen, see below.

Another observation is the HCW reports Exchange 2016 CU11 when we are actually running CU16

Kevin

 

HCW Migration Endpoint Error - change to classic after modern.jpg

 

 

Microsoft

@Kevin_Davis, I understand. If you will need to run HCW again in Classic Hybrid Topology and you still notice the issue, then yes, please send us the recording and the HCW log. 

Meanwhile, what I can suggest is to double check the latest HCW .log when you reproduced the issue and look at the entries under HybridConnectorChoice, especially if HybridConnectorSelected is TRUE or FALSE.
If that is TRUE, Test-MigrationServerAvailability on that Hybrid Agent is expected, if that is FALSE, then we are still talking about a bug / regression that needs to be addressed by us. 

Microsoft

Regarding the other mention, I think it is somehow normal, HCW is not always accurate with the Exchange CU release dates vs HCW releases.

Brass Contributor

Thanks @Mirela_Buru .

This looks like the section of the log file you are referring to, suggesting the bug still exists. The 'Connector previously experienced' date of 23 March is about the time we were setting up the hybrid (to allow Exchange mailbox calendars in MS Teams).

 

2020.08.23 09:12:45.650 10302 [Client=UX, Page=Mode, Thread=1]
=========================================
Mode
-----------------------------------------
Mode | Full Hybrid
HybridConnector.IsInstalled | False
NeedToUnInstall | False
=========================================
2020.08.23 09:12:45.665 10443 [Client=UX, Thread=1] Hybrid Connector Availablity: True, Reason: Connector previously experienced on 23/03/2020 18:37:02 +00:00
2020.08.23 09:12:45.681 10265 [Client=UX, Page=HybridConnectorChoice, Thread=1] START via Next
2020.08.23 09:12:47.853 10266 [Client=UX, Page=HybridConnectorChoice, Thread=1] FINISH Time=2171.9ms
2020.08.23 09:12:47.853 10302 [Client=UX, Page=HybridConnectorChoice, Thread=1]
===================================
HybridConnectorChoice
-----------------------------------
HybridConnectorSelected | False
HybridConnector.IsInstalled | False
NeedToUnInstall | False
===================================

 

KR,

Kevin

Microsoft

@Kevin_Davis ,  just to make sure, the HybridConnectorSelected | False is the LAST entry before test-migrationserveravailability -remoteserver <GUID>.resource.mailboxmigration.his.msappproxy.net? If so, the issue seems to persist but would be good to be able to repro on demand when opening a case with us (be able to run HCW and repro the issue in real time).

Brass Contributor

Hi @Mirela_Buru ,

Yes - the log I extracted from was the session that produced the screen grab I included in my previous post. During the work I was doing to try and fix this and rerun the HCW after making adjustments, the migration endpoint error was produced every time.

Just before this last run, I found the adminCount attribute of the Exchange server computer account was set to 1, where it should be set to 0. This server was previously a member of a DAG and a few other high availability options, before changing tack and running a single on-prem Exchange server as a VM on a high availability failover cluster. I checked server account memberships and other suggestions, but was satisfied that it just remained to set that value to 0, reboot the server, and try the HCW again, which I did, but with no change.

I am more than happy to provide after hours remote access for your troubleshooting, or send you the entire contents of the %appdata%\Microsoft\Exchange Hybrid Configuration folder which dates back to 23 March. Please send me a DM if there is anything I can assist with offline from this blog.

Kevin

Microsoft

This is an awesome blog :)

Copper Contributor

Hi @Mirela_Buru

In Classic Hybrid Mode everything works fine. But when we try to change to modern Agent, running the Hybrid Configuration Wizard (with modern Agent), on the Agent Installation part, in the end of it the test (Test-MigrationServerAvailability -ExchangeRemoteMove: $true -RemoteServer 'xxxxxxx-xxxx-xx-xxxx-xxxxxxx.resource.mailboxmigration.his.msappproxy.net' -Credentials (Get-Credential -UserName xxxxxxx)) always fails. We checked all the prerequisites, and everything is as it must be. We have that with several customers.

 

Any ideas?

 

Thanks
Joerg

Copper Contributor

I appreciate this informative post!  Is NTLM still the only method of authenticating the MRSProxy?  Microsoft has recommended disabling legacy auth in favor of modern auth for some time now, and it's strange that this feature still only seems to support a legacy auth type.  

Thank you! 

Brass Contributor

Hi,

 I have an new classic full hybrid Deployment with Exchange2016.

Test-migrationserveravailability shows no errors. HCW configuration works with no errors.

But when I try to migrate a mailbox from Onpremises to O365 through GUI I get the error:

Error: TimeoutErrorTransientException: The call to 'https://<servername>/EWS/mrsproxy.svc' timed out. Error details: The request channel timed out attempting to send after 00:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.

 

This error came on different mailbox migrations.

 

Any ideas?

Thank you

Copper Contributor

For anyone reading the comments.
I had Authentication errors 401 even though everything was set up correct.

I had errors in the event viewer system log every time I did iisreset.
This lead me to look into LSA authentication errors. and I fixed all my authentication issues by adding the FQDN of my EWS domain to this registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0 -> BackConnectionHostNames

Steps:
1. Open REGEDIT

2: go to key: -> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
3. Right click MSV1_0 -> New -> Multi-String Value
4. Enter the name BackConnectionHostNames
5. Right click and select Modify
6. Enter the host you are using for MRSProxy e.g. hybrid.contoso.com
7. Restart IIS


This was my error message for info:

RunspaceId : 189ea21b-b327-40c2-975b-e00cda035dcc
Result : Failed
Message : The connection to the server 'hybrid.contso.com' could not be completed.
ConnectionSettings :
SupportsCutover : False
ErrorDetail : Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the
server 'hybrid.contoso.com' could not be completed. --->
Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The Mailbox Replication
Service was unable to connect to the remote server using the credentials provided. Please check
the credentials and try again. The call to 'https://hybrid.contoso.com/EWS/mrsproxy.svc'
failed. Error details: The HTTP request is unauthorized with client authentication scheme
'Negotiate'. The authentication header received from the server was 'Negotiate,Basic
realm="hybrid.contoso.com"'. --> The remote server returned an error: (401) Unauthorized.. -->
The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The
authentication header received from the server was 'Negotiate,Basic
realm="hybrid.contoso.com"'. --> The remote server returned an error: (401) Unauthorized. --->
Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The call to
'https://hybrid.contoso.com/EWS/mrsproxy.svc' failed. Error details: The HTTP request is
unauthorized with client authentication scheme 'Negotiate'. The authentication header received
from the server was 'Negotiate,Basic realm="hybrid.contoso.com"'. --> The remote server
returned an error: (401) Unauthorized.. --->
Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The HTTP request is
unauthorized with client authentication scheme 'Negotiate'. The authentication header received
from the server was 'Negotiate,Basic realm="hybrid.contoso.com"'. --->
Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned
an error: (401) Unauthorized.
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.<>c__DisplayClas
s97_0.<ReconstructAndThrow>b__0()
at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(Action operation)
at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.ReconstructAndTh
row(String serverName, VersionInformation serverVersion)
at Microsoft.Exchange.MailboxReplicationService.WcfClientWithFaultHandling`2.<>c__DisplayClass7
_0.<CallService>b__0()
at Microsoft.Exchange.Net.WcfClientBase`1.CallService(Action serviceCall, String context)
at
Microsoft.Exchange.MailboxReplicationService.WcfClientWithFaultHandling`2.CallService(Action
serviceCall, String context)
at Microsoft.Exchange.Migration.MigrationExchangeProxyRpcClient.CanConnectToMrsProxy(Fqdn
serverName, Guid mbxGuid, NetworkCredential credentials, LocalizedException& error)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Migration.DataAccessLayer.ExchangeRemoteMoveEndpoint.VerifyConnectivity()
at Microsoft.Exchange.Management.Migration.MigrationService.Endpoint.TestMigrationServerAvailab
ility.InternalProcessEndpoint(Boolean fromAutoDiscover)
IsValid : True
Identity :
ObjectState : New

Version history
Last update:
‎Mar 30 2020 09:00 AM
Updated by: