When we need to deploy a hybrid scenario, the first thing that comes to mind is running the Hybrid Configuration Wizard. Indeed, HCW is enough to make the magic happen between Exchange Online (EXO) and Exchange on-premises. But, if you want to pull back the curtain on the mail flow and connectors, this post is for you! It is not unusual to see connector settings be changed manually, which might result in unexpected behavior. Additionally, some customers want to push the limits of their configuration by introducing additional hops into the mail flow path. This post attempts to provide technical details helpful for troubleshooting and deploying hybrid mail flow securely.
Before starting with deep dive stuff, we must be familiar with some definitions related to Tenant Attribution:
You can check the X-MS-Exchange-Organization-MessageDirectionality header, which tells if the decision is Originating or Incoming.
From a hybrid mail flow perspective, there is an important header which we often check in security assessment situation or any spam, spoof, or phish analysis called: X-MS-Exchange-Organization-AuthAs. We should focus on two possible values of that header: Internal or Anonymous.
Messages are considered Internal if they authenticated in some manner:
Messages are considered External if they are received through an Anonymous source:
Why is this header so important? Because the value of this header is used in many important decisions:
Now that we understand the Incoming and Originating definition and the differences between AuthAs: Internal and AuthAs: Anonymous, the question that we hear often is: Is originating message always marked as Internal ? Not at all. Also, a non-authenticated message received from external sources may be marked as Internal. If you want to know why, let’s figure it out!
There are many layers involving the decision to mark the message as Internal or Anonymous and specially keep the precedent header’s value or not. Usually what happens behind the scenes is:
From on-prem to EXO:
From EXO to on-prem:
Look easy, hmm? All those steps are expected in normal circumstances, in other words, if you did your homework and let HCW do its job, you should be fine.
If you see a different behavior such as non-authenticated message marked as Internal or authenticated message marked as Anonymous, you can start to troubleshoot using the following flowcharts (you can also download accessible version of those flowcharts as the attached ZIP file).
From EXO to on-prem:
From on-prem to EXO:
MX record pointing to on-prem – or even to EXO but Centralized Mail Transport is enabled; all messages to EXO recipients are being marked as Internal and are bypassing EOP spam filters, spoof verdict, phish controls and anti-impersonation controls.
This usually happens when there is an on-prem receive connector with Externally Secured option enabled. An evidence that you probably hit this scenario is when you find the header’s value X-MS-Exchange-Organization-AuthMechanism: 10.
Internal message sent from on-prem mailbox to EXO mailbox is being marked as Anonymous and I’m sure that it meets all requirements described in the flowchart. Besides, the message’s return-path value is rewritten by SRS (e.g.: bounces+SRSfirstname.lastname@example.org).
Probably you hit what we call ‘attribution to the wrong tenant’. If another tenant has Inbound Connector with same value of TlsSenderCertificateName or SenderIPAddresses as yours, the email could attribute to the other tenant instead of yours. An evidence that you hit this scenario can be found on the X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp header. The value of this header is the tenant id where the message has been attributed. The “contoso.com” domain mentioned on the return-path is the default Accepted Domain of the tenant that the message was attributed. If the tenants are owned by you, be sure that both tenants have different TlsSenderCertificateName or SenderIPAddresses values. Otherwise, you should open a case with us.
Legitimate internal messages originating to EXO are being marked as Anonymous and you can find the header: X-CrossPremisesHeadersFilteredBySendConnector.
Usually this scenario involves Exchange 2010 as Hub Transport and Exchange 2013 as Edge Transport. Refer to KB3212872 in order to fix this issue.
When Edge Transport Server is deployed, you should keep the “Inbound to Site” Send Connectors settings as default.
Changing the default Address Space from “--" to any other value may result in authentication issue for email coming from Office 365. In fact, our recommendation is to keep all connectors settings as default.
Why do we not recommend putting firewalls/hosts between EXO & Exchange on-prem?
An appliance or application between EXO and on-premises which modify or perform content inspection can break the cross-premises message authentication. As previous explained, EXO and on-prem rely on a set of headers to “trust” that the message is Internal.
In addition to headers, default hybrid configuration also grants your Exchange Online tenant the ability to relay from the cloud via your on-premises Exchange server. This can only be accomplished via an ESMTP protocol communication that happens at the application layer. Introducing filtering here can interfere. That said, if you do not wish to grant the ability to relay (e.g. outbound mail path is via Office 365), then this is not a big deal. However, if your outbound SMTP traffic needs to go via on-premises, then there is not a better way to grant relay to your tenant – and your tenant only. You may use the Edge Transport role to accomplish this determination. Refer to this article to find all supported hybrid scenarios.
What is required to preserve the headers if I don’t have Exchange and what are the risks of doing so?
Note that only connector type of on-premises supports CloudServicesMailEnabled. For considerations on how to setup on-premises mail flow with a 3rd party service, see Integrate Office 365 mail flow with an email add-on service. If the 3rd party service cannot meet the requirements set in this guidance, you will be unable to use on-premises connectors. If you cannot use on-premises connectors with CloudServicesMailEnabled, then the mail flow path will not be trusted. This means that all mail will be treated as Anonymous and will be re-scanned.
Typically we see this when someone wants to add a 3rd party service into their organization’s inbound or outbound traffic. For outbound, the on-premises connector ensures the ability to relay. For inbound, preservation of the spam verdicts is critical. If your put a 3rd party solution in front of Office 365, pay close attention to the next question below.
Adding an on-premises or customer-managed SMTP host into a mail routing path is a little less strict assuming you can use a specific IPv4 address that you own. However, certificates and tenant checks are still required to make certain that someone does not introduce untrusted mail into your path and compromise your security.
My MX record points to a 3rd party solution, but I want to take advantage of EOP and Office 365 ATP.
There was a time where we used to recommend bypassing EOP checks for 3rd party solution and/or hybrid Exchange on-prem via transport rule (SCL -1) for customers who have the MX record pointing to something other than Office 365. The reason? To avoid false positives due to failed SPF checks and so on. Please, don’t do that anymore. It’s OK if you did in the past but we built a better solution for you, called Enhanced Filtering for Connectors. We still encourage you to switch your MX record to Office 365, but if you are not prepared to do so yet, Enhanced Filtering can help.
Do I need to add on-premises public IP on SPF for hybrid mail flow from on-premises to EXO?
Yes. Even if Internal email bypasses spoof verdict for hybrid mail flow, we encourage you to add your on-premises public IP on SPF. The reason is you might have some application which rely on your Exchange anonymously to send to EXO or even sending directly to the Internet. Without on-prem public IP on SPF, those kinds of messages should land in junk folder – or follow the anti-spoof action.
Do I need to include EOP on my SPF if I’m sending email to internet from my on-premises server?
You should. The connector attribution process looks at the TLS certificate sent by your on-premises. If the recipient’s tenant is in the same forest as yours, the message will be attributed to your tenant before sending to the recipient’s tenant. The SPF check in the recipient’s tenant will be against an EOP IP address. That is why you should include EOP SPF in your SPF Record. If you’re seeing outbound traffic from on-premises being routed via your Office 365 tenant, or is causing mail loops, please review FAQ 6-b from this earlier post.
There are two possible exceptions for this scenario:
How can I force X-MS-Exchange-Organization-AuthAs: Internal for non-authenticated messages?
For Exchange on-premises, the only supported scenario for this would be using a receive connector assigned as Externally Secured permission (which grants other permissions). For Exchange Online, you can use an inbound connector with TreatMessagesAsInternal. Any other method such as Transport Rule is not supported.
I have a lot of custom settings on connectors, my concern is if I run HCW and break mail flow, what should I do?
If you don’t have Centralized Mail Transport enabled, technically you have no reason to be afraid. If you have Centralized Mail Transport enabled and use many other custom settings, we encourage you to assess the environment before running HCW. Avoiding HCW is not really a solution; sooner or later you should run it to change a certificate that will expire, right?
You can get and save all attribute values of Receive Connectors, Send Connectors, Inbound Connectors, Outbound Connectors, accepted domains, and remote domains. Once you assess all this information, even if HCW changes some parameter that breaks the mail flow, you will be able to compare before and after state and fix it.
Let me give you a common example: for Exchange 2010, HCW creates an ad hoc receive connector to receive emails from EXO, called “Inbound From Office 365” containing the whole EOP IP range on RemoteIPRange attribute. If you have a NAT which translates the public IP to an internal IP, you must add only this internal NAT IP to RemoteIPRange, otherwise Exchange 2010 will refuse EXO messages. When you run HCW, it will overwrite your NAT IP and add the whole EOP IP range again. So, you need to know the NAT IP before running HCW to re-add as soon as HCW finishes.
Well, that was a lot! Hope you could find relevant information to perform your next SMTP troubleshooting and please feel free to leave comments and feedback below!
Thanks to Arindam Thokder, Nino Bilic and Scott Landry for their review and contribution of this post.
Denis Vilaça Signorelli
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.