The Microsoft Hybrid Agent Public Preview
Published Feb 05 2019 08:00 AM 42K Views

Update 4/9/2019: to read about additional features released in the Hybrid Agent, please go here.

The moment you’ve all been waiting for has arrived! We are pleased to announce the Microsoft Hybrid Agent for Exchange Server is now available for preview! We talked in some depth about the Hybrid Agent back at Ignite 2018 in Orlando, if you want a refresher on what we covered there you can see that recording here. The Hybrid Agent was designed to remove some of the existing challenges customers face today when establishing a Hybrid Exchange environment. This includes, but is not limited to, adding external DNS entries, updating certificates, and allowing inbound network connections through the firewall. From today, when running the Hybrid Configuration Wizard, you are presented with a new option for establishing hybrid; “Modern Hybrid”. Modern Hybrid is offered for both Minimal and Full Hybrid Configurations. This new option will only be presented if you have never run the Hybrid Configuration Wizard. If you have successfully established Hybrid in either Minimal or Full config before this release, this new option will not be available. HybridAgent1 This feature will install an agent, built on the same technology as Azure Application Proxy, which will publish the Exchange on-premises environment to Exchange Online to support Free/busy and mailbox migrations without many of the challenges customers previously faced with external DNS, publishing of EWS and inbound connections ports having to be opened. The secret sauce here to explain how this works is that the Hybrid Agent registers a custom URL for only your tenant in the following format:

<guid>.resource.mailboxmigration.his.msappproxy.net

This URL is then used by the Organization Relationship or the Intra Organization Connector and the Mailbox Replication Service to route requests from your tenant to on premises. This URL is only accessible from Exchange Online. Free/busy requests from cloud users to on premises and mailbox migrations to/from the cloud are the two flows currently supported through the Hybrid Agent. HybridAgent2 Yes, this new feature offers some great new capabilities and we’re very proud of it, but the Hybrid Agent preview still has a few constraints:
  • MailTips, Message Tracking and Multi-mailbox search do not traverse the Hybrid Agent. These Hybrid features would require the classic connectivity model where EWS and Autodiscover are published on-premises and externally available to Office 365.
  • The public preview only supports a single Hybrid Agent install for the Exchange Organization. We are working to support multiple agent installs for redundancy, but this is not available yet. If the server running the Hybrid Agent goes offline, free/busy look ups from your tenant to on-premises and mailbox migrations to/from your tenant will no longer work. If the server hosting the agent is permanently offline, was rebuilt, or the agent was uninstalled, you can recover the original configuration by re-running the Hybrid Configuration Wizard to reinstall the Hybrid Agent directly on the new server. Do not attempt to install multiple active Hybrid Agents in your environment with this preview build, this could cause unexpected issues.
  • The Hybrid Agent registers the internal FQDN of the Client Access Server (CAS) selected when running Hybrid Configuration Wizard in Azure Application Proxy. If the registered CAS is offline, free/busy look ups from your tenant to on-premises and mailbox migrations to/from your tenant will no longer work. If the selected CAS is permanently offline, a new CAS must be registered. This can be done by re-running the Hybrid Configuration Wizard.
  • You can’t use the Hybrid Agent if you plan on enabling Hybrid Modern Auth, which you also need to get the most out of Outlook mobile. You need to publish AutoDiscover, EWS, MAPI and OAB the Classic way if you want to use HMA externally.
  • The Hybrid Agent preview comes with some support limitations which are called out in the Terms document that you must agree to before installing the feature.
We also want to point out that SMTP does not traverse the Hybrid Agent and will still require a public certificate for mail flow between Office 365 and on-premises. SMTP traffic is out of scope for the Hybrid Agent, both now and through General Availability. There are a few limitations with this preview build, but that’s why it’s a preview! We still think there’s a lot the agent can do today, and that’s why we’re making it available to you now! Once you are ready to move forward with Hybrid Agent for your deployment, there are a few deployment decisions to make. Please read this next section all the way through before downloading and installing the agent. Choosing the right location for installation of the Hybrid Agent is important. The agent install and subsequent run of configuring Hybrid via the Hybrid Configuration Wizard is supported on either a standalone machine designed as your “agent server”, or an Exchange 2010, 2013, 2016 or 2019 server with the Client Access role. The easiest way to deploy is accomplished by installing the Hybrid Agent directly on an Exchange CAS as it simplifies connectivity, but it is not required.

Here are the agent server requirements

  • The machine hosting the Hybrid Agent install must be able to establish outbound HTTPS connections to the internet, and HTTPS and Remote PowerShell (RPS) connections to the CAS chosen for hybrid configuration.
  • The machine hosting the Hybrid Agent should be running Windows Server 2012 R2 or 2016, with .NET Framework 4.6.2 (or later, as supported by the Exchange version you are installing on) installed.
  • The machine where the Hybrid Agent is installed must have either Edge or Internet Explorer installed and must support ClickOnce.
  • The machine where the Hybrid Agent is installed must be able to communicate with a Domain Controller to authenticate your on-premises Exchange Org admin credentials. This means that the machine must be domain joined.
  • Installation must be done using a local machine administrator account and will require tenant global administrator credentials for registering the connector.
  • TLS 1.2 must be enabled on the machine where the Hybrid Agent is installed.
Now, the smart people with a good memory reading this post might spot one interesting wrinkle in this list above: we support you installing the agent on an Exchange 2010 server, but we require you use Windows Server 2012 R2 or later... Hang on, that’s not possible. That’s not supported. You didn’t spot that? Shame on you. Well, for those that did, firstly, have a gold star and a pat on the back – and secondly, we’re announcing here that we will support Exchange Server 2010 installed on Windows Server 2012 R2 with the upcoming release of Update Rollup 26 for Exchange Server 2010 SP3. We’re doing that so if you really want to add another Exchange 2010 server to your org, on Windows Server 2012 R2, you can. You’re welcome.

Port and Protocol requirements for the agent server

  • Ports to be opened outbound are HTTPS (TCP) 443 and 80, as shown here.
  • The agent machine must be able to connect HTTPS (TCP) 443, 80, 5985 and 5986 to the target CAS selected in the Hybrid Configuration Wizard.

Important notes

  • All Client Access Servers must be able to reach outbound to Office 365 endpoints via HTTPS (TCP) 443 as free/busy request from on-premises users to Office 365 users do not traverse the Hybrid Agent. These requests still require your Exchange servers have outbound connectivity to Office 365 end points. Office 365 URLs and IP address ranges describes the required (and hybrid) ports and IPs outbound from on-premises to the service.
  • The specific Client Access Server selected in the Hybrid Configuration Wizard must be able to make a Remote PowerShell connection to Office 365.
  • The agent does support using an outbound proxy but doing so requires modifications to the configuration file after installation. Also, a proxy which prevents registration will result in the connector failing to install. It is recommended to install allowing the connectors to bypass the proxy until app config changes can be made.
“Wow”, you say, “I wish it was easier to confirm the required connectivity before installing…”. Well you are in luck! We have built a port and endpoint verification script helper just for you! We recommend you install this script on your designated agent machine, run it, and confirm all the port requirements have been met prior to installing. In our experience, it is likely your environment is thoroughly locked down and ensuring the required ports and endpoint are working as required will make your installation and configuration experience much better!

Verifying connectivity

  1. On the server where you will be running the Hybrid Configuration Wizard (Hybrid Agent install and subsequent hybrid configuration steps), download the following sample script and save it to any directory: http://aka.ms/hybridconnectivity.
  2. Open PowerShell and change directory to the location of the script.
  3. Import the cmdlets: Import-Module .\HybridManagement.psm1
  4. Next run Test-HybridConnectivity with the testO365Endpoints option to verify the machine you are installing on can reach out to all required endpoints for the Hybrid Agent installation and Hybrid Configuration Wizard setup.
  5. Sample run below:
HybridAgent3

Uninstalling the Hybrid Agent

To uninstall the Hybrid Agent, re-run Hybrid Configuration Wizard from the same machine you performed the installation against and select Classic Connectivity. This will uninstall and de-register the Hybrid Agent from the machine and Azure, and you can resume setup and configure hybrid in the Classic mode.

More information

Additional details on the installation requirements and steps can be found here. Now you are ready to run the Hybrid Configuration Wizard and install the new Hybrid Agent! Happy Hybriding and we look forward to reading your feedback, please do leave us comments below. The Hybrid Team
33 Comments
Not applicable
I assume that should someone want to move to Classic Hybrid the wizard can be re-run over the top of the existing configuration?
Not applicable
Yes, you can re-run the Hybrid Configuration Wizard from the same machine, select Classic, and it will uninstall and unregister the Hybrid Agent and you can proceed with configuring traditional hybrid.
Not applicable
Hello,

Do the new Hybrid Agent requests a Valid Public Certificate?

TNX

Not applicable
SMTP does not traverse the Hybrid Agent and will still require a public certificate for mail flow between Office 365 and on-premises.
Not applicable
We're in the midst of a Server 2008R2 Exchange 2010(UR 25) migration of a single server to Office 365 and are trying to leverage hybrid configuration, but are running into an issue because it's now requiring .NET 4.6.2 to install the HCW. This version of .NET isn't supported for Exchange 2010 according to the supportability matrix. We've contacted Office 365 support and they have not been able to provide any concrete answers...some are recommending we install 4.6.2 (or even 4.7.x) on the Exchange server. We're hesitant to do so for obvious reasons. Can .NET 3.5 and 4.6.x operate side-by-side without impacting Exchange functionality? We've done 2010 hybrids before without issue, any suggestions on how we should proceed? Also, here is an article from MVP Jaap Wesselius on his workaround: https://jaapwesselius.com/2018/12/04/the-version-of-the-client-access-server-selected-is-not-supported/

I really could use some official answers on this one, because I want to ensure a safe transition to O365!

Cheers,

Trey

Not applicable
Hi Trey,

No need to run the Hybrid Configuration Wizard on your Exchange server, can you run it on a different server in your environment? It will discover the Exchange server during the installation.

Regards,

Not applicable
Will it ever be available to those who have already run the wizard? We ran it but are having issues with firewall/network requirements form our security team. This would be a perfect fit if we could undo that and run this again.
Not applicable
Same situation here. Being able to switch to a proxy service would be nice.
Not applicable
What about replacing a current full hybrid with the agent ? Will this become available soon ?
Not applicable
Is this safe to use Hybrid Agent in the Production environment? If not, when are you planning to make in GA?

Thank you

Not applicable
1) Is this connection required during actual configuration? aadap-portcheck-seaus.connectorporttest.msappproxy.net on port 8080, test is failing since we only open port 80/443 to the O365/Azure IPs. Port 8080 does not show up on the official Azure AD/O365 ports and URLs requirements docs.

2) How do we install Hybrid Agent on a separate machine outside the Exchange Servers? e.g. such as a dedicated machine joined in the domain but does not run Exchange. Should we invoke the HCW from the Exchange Management console and when prompted to install, save the installer and run it on a separate machine? Or is there a direct URL to invoke installing Hybrid Agent on a separate machine other than the Exchange Servers. Running Exchange 2010 SP3.

Not applicable
HCW URL https://aka.ms/HybridWizard
Not applicable
We have already run the HCW on a number of customer sites , but would like to install the Agent to get full features as described. When will you make the agent a separate install or an option in the wizard to install just the Agent?
Not applicable
Question: What should the organization relationship endpoint URLs be after setting up Modern Hybrid with the agent? (for "O365 to On-premises" and "On-premises to O365" respectively)

I.e.

TargetSharingEpr

TargetOwaURL

TargetAutodiscoverEpr

Having the online TargetAutodiscoverEpr pointing to on-prem URL (https://autodiscover.domain.com/autodiscover/autodiscover.svc/WSSecurity) seems incorrect since this URL isn't available externally. Bug?

Same with online TargetOwaURL pointing to on-prem (https://mail.domain.com/owa).

This was the result with Exchange 2010 after running the new HCW with Modern Hybrid.

Not applicable
Same question regarding the external autodiscover DNS CNAME record, where should it point to? autodiscover.outlook.com I suppose since with Hybrid Agent autodiscover.domain.com is not available externally.
Not applicable
I wanted to post this question at blog https://blogs.technet.microsoft.com/exchange/2017/12/06/announcing-hybrid-modern-authentication-for-exchange-on-premises , since comments are closed there asking it here.

I have a mailbox in on prem exchange server (which is in hybrid mode) abc@onprem.com and i am trying to access this via graph api (/messages).

This works perfectly if i do this in graph explorer, but fails when i do via implementation.

Required application permission is given in Azure app registration portal.

Implementation uses grant_type as client_credentials with certificate and this works perfectly for cloud users.

Response of API

{ ‘error’: {

‘innerError’: {

‘date’: ‘2019-02-28T14:17:45’,

‘request-id’: ‘6a85f8c3-4e13-4cf0-84b2-ddc934241afd’

},

‘message’: ”,

‘code’: ‘UnknownError’

}}

Form you previous comment i found that calls are reaching on prem exchange server through IIS Logs

For call came from graph explorer

2019-02-28 15:02:31 172.31.10.98GET /api/V2.0/Users(‘abc@onprem.com’)/Messages/$count &CorrelationID=;&cafeReqId=bc8e8aef-de46-4c72-bcf4-b4f567bc45dd; 443 S-1-5-21-1392771109-4043059535-3934338706-1147 20.190.145.177Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_3)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/71.0.3578.80+Safari/537.36 – 200 0 0 287

For call from implemented app

2019-02-28 15:00:05 172.31.10.98GET /api/V2.0/Users(‘abc@onprem.com’)/Messages/$count &CorrelationID=;&cafeReqId=c504b658-b9df-43b5-9dbb-8e83050c3d2f; 443 – 20.190.128.103- – 401 0 0 102

How to debug why Authentication is failing for on prem mailboxes and reason for this ? Is there any logs which i can refer ?

Also what would be reason for this authentication failure , could it be because that token is provided by azure AD which is authenticated against onprem ?

Should id do Set-MSOLServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 for app id registered in in Azure as well ?

Also curious about which app “00000002-0000-0ff1-ce00-000000000000” belong to ?

Not applicable
Anyone having 403 issues like: "The HTTP request was forbidden with client authentication scheme 'Negotiate'."

TestMigrationServerAvailabilityOutcome

{ErrorDetail='Microsoft.Exchange.Migration.MigrationServerConnectionFailedException:

The connection to the server 'XXX.resource.mailboxmigration.his.msappproxy.net' could not be completed.

---> Microsoft.Exchange.MailboxReplicationService.RemoteTransientException:

The call to 'https://XXX.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed.

Error details: The HTTP request was forbidden with client authentication scheme 'Negotiate'. --> The remote server returned an error: (403) Forbidden..

---> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The HTTP request was forbidden with client authentication scheme 'Negotiate'.

---> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned an error: (403) Forbidden.

Not applicable
Try this for 403 forbidden issue

https://support.microsoft.com/en-us/help/3063913/the-remote-server-returned-an-error-403-forbidden-error-when-you-try-t

Not applicable
Proceeded ahead and configured this. Our EWS and Autodiscover is not externally published which is why we went for the Exchange Hybrid Agent model. After completion of the wizard, we get this warning on HCW.

"Office 365 was unable to communicate with your on-premises Autodiscover endpoint. This is typically due to incorrect DNS or firewall configuration. The Office 365 tenant is currently configured to use the following URL for Autodiscover queries from the Office 365 tenant to the on-premises organization"

What is the impact of this? If we move a mailbox from on prem to the cloud, will it require a reconfiguration of profile? When a cloud user wants to query free busy for on prem user, will it cause an issue? I thought this is what the Hybrid agent should cater for. Don't get why the warning is still there though,

Not applicable
@Yoyong, the warning for autodiscover is probably still there because of Test-OrganizationRelationship or Get-FederationInformation. But Free/Busy should work because of TargetSharingEpr set by Modern where we bypass Autodiscover. If you have free/busy issues after Modern Hybrid, check the f/b error and fix suggestions from here: https://blogs.technet.microsoft.com/exchange/2018/03/02/demystifying-hybrid-freebusy-finding-errors-and-troubleshooting/

Regarding the outlook profile reconfiguration, this should not be the case in MRS migrations because we keep the ExchangeGUID. So if you are able to successfully resolve autodiscover for your Primary SMTP domain from your internal network (SCP), then outlook will try autodiscover for target address of migrated user which is autodiscover.tenant.mail.onmicrosoft.com (based on remote routing address user@tenant.mail.onmicrosoft.com) that points to O365 in DNS and redirects to autodiscover-s.outlook.com on HTTPS (valid certificate too).

Not applicable
It appears that, based on some of the questions below, this page will never go out of style, regardless of the new agent.

https://blogs.technet.microsoft.com/exovoice/2016/09/19/troubleshooting-issues-where-the-migration-endpoint-cannot-be-created-in-hybrid-scenarios/

Not applicable
I have the newest HCW but i havent the choice to select modern or classic configuration. Maybe it's because I want to install it on an SBS2011?
Not applicable
In the Ignite demo they mentioned 2 way mailbox delegation sync. Is that working?
Not applicable
Is Multi-Geo working for the agent?
Not applicable
Managed to get some users onboard to O365 Exchange Online with this model and all is going well. Migration to Exchange Online is quite seamless and calendar free busy is also working fine. However am testing an offboarding scenario moving a mailbox on prem and am getting the following error.

Error: The job encountered too many transient failures (61) and is quitting. The most common failure is CommunicationErrorTransientException/ProtocolException/WebException

with the hit count 60. --> The call to 'https://guid.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc exchangeserver.domain.com (14.3.399.0 ServerCaps:01FFFFFF, ProxyCaps:, MailboxCaps:, legacyCaps:05FFFF)' failed. Error details: The remote server returned an unexpected response: (413) Request Entity too Large

Copper Contributor

I’m having an issue with uninstalling the agent. I reran the wizard as mentioned, but when it gets to the step of actually uninstalling it, it just shows the progress bar infinitely. It sat like that for over 4 hours, which ultimately caused me to have to cancel the wizard. Please help. I need to go back to classic connectivity. 

Microsoft

@Chris Varner - You're probably going to have to reboot to "unstick" the installer, but once you do, do the following:

1. Stop the mshybridsvc.

2. When it's stopped, uninstall from Control Pannel.

3. Run the wizard and switch modes. If you have the log from the uninstall, go ahead and send it to jason (at) microsoft (dot) com - it's the actual uninstall log that's needed, not the SHCW one.

Copper Contributor

@xC0000005 Thank you so much for the reply. I kept tinkering with it after I posted this and I ended up doing what you recommended and it worked. So your recommendation worked in my situation. Once the HCW finished the last time I ran it, the only notification it gave me was the OAuth configuration couldn't be completed. Is that a normal thing now? I used to get the Congratulations with the green check (I think that's what it was) that showed everything finished fine. 

Deleted
Not applicable

For exchange 2010 + server 2008 r2, they have option to install hybrid server with at least server 2012 r2 and take advantage of the hybrid agent for the modern migration?

 

Or exchange 2010 + server 2008 r2 can use the classic migration only?

Copper Contributor

Cannot cross-reference calendar of O365 → Exchange
Do you need any special settings?

 

Is the restricted team a shared mailbox?
----------------
Team's Calendaring features that require access to on premises mailboxes are not supported with the Hybrid Agent.
----------------

Copper Contributor

Hi, 

When running verification script from this page http://aka.ms/hybridconnectivity

it fails.

Probably because certificate at https://aadap-portcheck-seaus.connectorporttest.msappproxy.net/ is expired.

Anyone know how to notice those responsible?

Is this sentence still valid for latest HCW version?

 

"This new option will only be presented if you have never run the Hybrid Configuration Wizard. If you have successfully established Hybrid in either Minimal or Full config before this release, this new option will not be available."

 

Thanks

Copper Contributor

Is the HCW capable of moving the Hybrid Agent server to a different server, i.e., will it remove it from the old server and install it on the new server if I change the target in the wizard (if possible, I haven't tried it yet)?

Version history
Last update:
‎Jul 01 2019 04:35 PM
Updated by: