When we need to deploy a hybrid scenario, the first thing that comes to mind is running the Hybrid Configuration Wizard. Indeed, HCW is enough to make the magic happen between Exchange Online (EXO) and Exchange on-premises. But, if you want to pull back the curtain on the mail flow and connectors, this post is for you! It is not unusual to see connector settings be changed manually, which might result in unexpected behavior. Additionally, some customers want to push the limits of their configuration by introducing additional hops into the mail flow path. This post attempts to provide technical details helpful for troubleshooting and deploying hybrid mail flow securely.
Before starting with deep dive stuff, we must be familiar with some definitions related Org-Headers and Tenant Attribution.
What is an Exchange Org-Header?
An email header is a critical section of data at the start of an email that details the message's origin, destination, internet trajectory, and other technical specifics. Exchange organization headers, particularly those prefixed with X-MS-Exchange-Organization, mark a deeper dive into the mechanics of message handling within an Exchange environment. These specialized headers play a key role across various Exchange components, influencing decisions in areas such as rules processing, calendar functionalities, and spam detection, among others. This integration underscores the complexity and sophistication of email management within an Exchange organization.
Header Preservation: The Header Firewall
For security reasons, these headers are typically stripped off when emails leave one Exchange Organization or enter another Exchange organization, forming a 'header firewall.' This mechanism is crucial for maintaining the integrity and confidentiality of internal header information.
- Originating: A message that originates from a tenant’s on-premises server (it will match the tenant’s inbound connector that is of type OnPremises), or from an Exchange Online mailbox. If Office 365 is unable to identify the message as originating and the recipient domain is an accepted domain in an Office 365 organization, the service will identify the message as incoming to the recipient organization.
- Incoming: Messages sent to accepted domains that are in Office 365 that do not match a tenant connector that is of type OnPremises.
You can check the X-MS-Exchange-Organization-MessageDirectionality header, which tells if the decision is Originating or Incoming.
From a hybrid mail flow perspective, there is an important header which we often check in security assessment situation or any spam, spoof, or phish analysis called: X-MS-Exchange-Organization-AuthAs. We should focus on two possible values of that header: Internal or Anonymous.
Messages are considered Internal if they authenticated in some manner:
- Authenticated mailbox on-prem or Exchange Online (StoreDriver)
- Authenticated mailbox using SMTP client submission (SMTP AUTH)
- Received through an on-prem receive connector with ExternalAuthoritative (Externally Secured) permission enabled
- Came into Exchange Online via an inbound connector with TreatMessagesAsInternal set to “true” and the sender is an accepted domain.
Messages are considered External if they are received through an Anonymous source:
- Internet
- SMTP relay (receive connector without ExternalAuthoritative)
- Submitted by Pickup directory
Why is this header so important? Because the value of this header is used in many important decisions:
- Internal
- Bypass spam filters, spoof verdict, phish controls and anti-impersonation controls for inbound email. Internal email going outbound to On-Prem is still scanned by EOP.
- Bypass ATP SafeLink Policy if EnableForInternalSenders is set to $False
- Allowed to send to distribution lists where RequireSenderAuthenticationEnabled is set to $True by default
- By default, Internal messages are processed by resource mailboxes and the Resource Booking Attendant
- In certain circumstances, may trigger transport rule condition “Is received from: Inside the organization”
- Anonymous
- Will be rejected to distribution lists that require authentication
- Will not be processed by the Resource Booking Attendant (ProcessExternalMeetingMessages must be set to $True)
- Display name of the sender doesn’t resolve to an object in the Global Address List
- Images won’t automatically downloaded in an email
- Office documents open in “Protected View” mode
- Scanned by spam filters
Now that we understand the Incoming and Originating definition and the differences between AuthAs: Internal and AuthAs: Anonymous, the question that we hear often is: Is originating message always marked as Internal ? Not at all. Also, a non-authenticated message received from external sources may be marked as Internal. If you want to know why, let’s figure it out!
There are many layers involving the decision to mark the message as Internal or Anonymous and specially keep the precedent header’s value or not. Usually what happens behind the scenes is:
From on-prem to EXO:
- Exchange on-prem user sends an authenticated message to an EXO recipient
- [IF EX2010] Exchange on-prem converts X-MS-Exchange-Organization-* headers to X-MS-Exchange-CrossPremises-* if there is a Remote Domain for the recipient TargetAddress (mail.onmicrosoft.com) AND if the TrustedMailOutboundEnabled attribute is set to $True
- [IF EX2013+] Exchange on-prem converts X-MS-Exchange-Organization-* headers to X-MS-Exchange-CrossPremises-* if there is an Accepted Domain for the recipient TargetAddress (mail.onmicrosoft.com) AND if the Send Connector attribute CloudServicesMailEnabled is $True
- On MAIL FROM command, Exchange Online Protection (EOP) will search for an OnPremises type inbound connector that matches the Exchange on-prem IP or TLS cert used to STARTTLS. If there is a match, message is classified as Originating
- OnEndOfHeader, the connector attribution process will be concluded and on BDAT (DATA) command, Exchange on-prem will send the X-MS-Exchange-CrossPremises-* headers
- If CloudServicesMailEnabled on inbound connector is $True, EXO converts the X-MS-Exchange-CrossPremises-* headers to X-MS-Exchange-Organization-*
From EXO to on-prem:
- EXO user sends an authenticated message to on-prem recipient
- EXO outbound connector converts X-MS-Exchange-Organization-* to X-MS-Exchange-CrossPremises-* if the CloudServicesMailEnabled attribute is $True and the recipient domain is trusted (Accepted Domain or Internal Relay) or the recipient domain is listed as Remote Domain with TrustedMailOutboundEnabled $True
- Exchange on-premises might have a receive connector with TlsDomainCapabilities as:
- Exchange 2010: {mail.protection.outlook.com:AcceptOorgProtocol}
- Exchange 2013 or later: {mail.protection.outlook.com:AcceptCloudServicesMail}
- In any case MAIL FROM command should contains the XOORG value followed by the default AcceptedDomain or sender’s primary SMTP domain (for more details see this)
- OnEndOfHeaders event, EXO sends X-MS-Exchange-CrossPremises-AuthAs: Internal and all others CrossPremises headers
- [IF EX2010] Receive connector checks if there is a remote domain that matches the XOORG value and checks if the parameter TrustedMailInboundEnabled is $True.
- Exchange on-prem converts the X-MS-Exchange-CrossPremises-* headers to X-MS-Exchange-Organization-*. The server performing this action stamps X-OrganizationHeadersPreserved header, although there might be some case where this isn’t logged.
Look easy, hmm? All those steps are expected in normal circumstances, in other words, if you did your homework and let HCW do its job, you should be fine.
If you see a different behavior such as non-authenticated message marked as Internal or authenticated message marked as Anonymous, you can start to troubleshoot using the following flowcharts (you can also download accessible version of those flowcharts as the attached ZIP file).
From EXO to on-prem:
EXO to External via CMT flow:
From on-prem to EXO:
Logs that can help in troubleshooting
- [On-prem] Message tracking: the first step to figure out all servers and connectors that the message passed through.
- [EXO] Message Trace Extended Report: provides useful information, such as:
- Directionality: Originating or Incoming
- InboundConnectorData=Name: Inbound connector name which message was attributed
- ConnectorType: OnPremises or Partner
- Remember: A successful connector attribution (type OnPrem) is required to promote AuthAs: Internal.
- [On-prem] Pipeline Tracing: the best way to see all header modifications performed by transport agents across the transport pipeline. The decision to convert X-MS-Exchange-CrossPremises-* to X-MS-Exchange-Organization-* and vice versa is performed OnEndOfHeaders (Inbound Trust Agent) for Inbound messages or OnCategorizedMessage (Outbound Trust Agent) for outbound messages
- [On-prem] Enable Connectors Verbose Protocol Logs: very useful to see the envelope commands, specially to be sure that Exchange Online performed the XOORG. By default, XOORG is advertised only after STARTTLS. Make sure you have your certificate assigned to SMTP service and if you perform a telnet and send EHLO you can see STARTTLS available
Commonly mis-configured scenarios
MX record pointing to on-prem – or even to EXO but Centralized Mail Transport is enabled; all messages to EXO recipients are being marked as Internal and are bypassing EOP spam filters, spoof verdict, phish controls and anti-impersonation controls.
This usually happens when there is an on-prem receive connector with Externally Secured option enabled. An evidence that you probably hit this scenario is when you find the header’s value X-MS-Exchange-Organization-AuthMechanism: 10.
Internal message sent from on-prem mailbox to EXO mailbox is being marked as Anonymous and I’m sure that it meets all requirements described in the flowchart. Besides, the message’s return-path value is rewritten by SRS (e.g.: bounces+SRS=@contoso.com).
Probably you hit what we call ‘attribution to the wrong tenant’. If another tenant has Inbound Connector with same value of TlsSenderCertificateName or SenderIPAddresses as yours, the email could attribute to the other tenant instead of yours. An evidence that you hit this scenario can be found on the X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp header. The value of this header is the tenant id where the message has been attributed. The “contoso.com” domain mentioned on the return-path is the default Accepted Domain of the tenant that the message was attributed. If the tenants are owned by you, be sure that both tenants have different TlsSenderCertificateName or SenderIPAddresses values. Otherwise, you should open a case with us.
Legitimate internal messages originating to EXO are being marked as Anonymous and you can find the header: X-CrossPremisesHeadersFilteredBySendConnector.
Usually this scenario involves Exchange 2010 as Hub Transport and Exchange 2013 as Edge Transport. Refer to KB3212872 in order to fix this issue.
When Edge Transport Server is deployed, you should keep the “Inbound to Site” Send Connectors settings as default.
Changing the default Address Space from “--" to any other value may result in authentication issue for email coming from Office 365. In fact, our recommendation is to keep all connectors settings as default.
Hybrid mail flow FAQ
Why do we not recommend putting firewalls/hosts between EXO & Exchange on-prem?
An appliance or application between EXO and on-premises which modify or perform content inspection can break the cross-premises message authentication. As previous explained, EXO and on-prem rely on a set of headers to “trust” that the message is Internal.
In addition to headers, default hybrid configuration also grants your Exchange Online tenant the ability to relay from the cloud via your on-premises Exchange server. This can only be accomplished via an ESMTP protocol communication that happens at the application layer. Introducing filtering here can interfere. That said, if you do not wish to grant the ability to relay (e.g. outbound mail path is via Office 365), then this is not a big deal. However, if your outbound SMTP traffic needs to go via on-premises, then there is not a better way to grant relay to your tenant – and your tenant only. You may use the Edge Transport role to accomplish this determination. Refer to this article to find all supported hybrid scenarios.
What is required to preserve the headers if I don’t have Exchange and what are the risks of doing so?
- From on-prem to EXO: If you use an inbound connector in Office 365 with CloudServicesMailEnabled, Office 365 will look for CrossPremises mail flow headers and will promote them to Intra-organizational headers upon entry into Office 365. In this way, you can preserve most intra-organizational headers. Alternative: Setting TreatMessagesAsInternal as $True forces EOP to accept messages from your 3rd party solution as Internal only if the sender matches an accepted domain on EXO. The risk is that EOP can’t really validate if the message is indeed internal or spoofed by someone through your 3rd party solution. Use with extreme caution!
- From EXO to on-prem: With CloudServicesMailEnabled Office 365 will convert Intra-organizational headers to CrossPremises headers. The problem is that since other Office 365 customers could do the same, so you would need to harden the 3rd party solution to accept only Office 365 AND validate the tenant used to send the mail against a unique header that can’t be spoofed by no other tenant such as X-OriginatorOrg: contoso.com where contoso.com is an Accepted Domain. For this reason, we only support using Exchange or a trusted 3rd party service that can distinguish between Office 365 tenants.
Note that only connector type of on-premises supports CloudServicesMailEnabled. For considerations on how to setup on-premises mail flow with a 3rd party service, see Integrate Office 365 mail flow with an email add-on service. If the 3rd party service cannot meet the requirements set in this guidance, you will be unable to use on-premises connectors. If you cannot use on-premises connectors with CloudServicesMailEnabled, then the mail flow path will not be trusted. This means that all mail will be treated as Anonymous and will be re-scanned.
Typically we see this when someone wants to add a 3rd party service into their organization’s inbound or outbound traffic. For outbound, the on-premises connector ensures the ability to relay. For inbound, preservation of the spam verdicts is critical. If your put a 3rd party solution in front of Office 365, pay close attention to the next question below.
Adding an on-premises or customer-managed SMTP host into a mail routing path is a little less strict assuming you can use a specific IPv4 address that you own. However, certificates and tenant checks are still required to make certain that someone does not introduce untrusted mail into your path and compromise your security.
My MX record points to a 3rd party solution, but I want to take advantage of EOP and Office 365 ATP.
There was a time where we used to recommend bypassing EOP checks for 3rd party solution and/or hybrid Exchange on-prem via transport rule (SCL -1) for customers who have the MX record pointing to something other than Office 365. The reason? To avoid false positives due to failed SPF checks and so on. Please, don’t do that anymore. It’s OK if you did in the past but we built a better solution for you, called Enhanced Filtering for Connectors. We still encourage you to switch your MX record to Office 365, but if you are not prepared to do so yet, Enhanced Filtering can help.
Do I need to add on-premises public IP on SPF for hybrid mail flow from on-premises to EXO?
Yes. Even if Internal email bypasses spoof verdict for hybrid mail flow, we encourage you to add your on-premises public IP on SPF. The reason is you might have some application which rely on your Exchange anonymously to send to EXO or even sending directly to the Internet. Without on-prem public IP on SPF, those kinds of messages should land in junk folder – or follow the anti-spoof action.
Do I need to include EOP on my SPF if I’m sending email to internet from my on-premises server?
You should. The connector attribution process looks at the TLS certificate sent by your on-premises. If the recipient’s tenant is in the same forest as yours, the message will be attributed to your tenant before sending to the recipient’s tenant. The SPF check in the recipient’s tenant will be against an EOP IP address. That is why you should include EOP SPF in your SPF Record. If you’re seeing outbound traffic from on-premises being routed via your Office 365 tenant, or is causing mail loops, please review FAQ 6-b from this earlier post.
There are two possible exceptions for this scenario:
- The last on-prem hop is a 3rd party gateway using a different TLS cert CN or SAN which doesn’t match at all the Inbound Connector TlsSenderCertificateName. If your Inbound Connector uses IP instead of TLS, the message won’t be attributed to your tenant only if the 3rd party public IP doesn’t match the Inbound Connector SenderIPAddresses.
- Exchange 2013 or later using two Send Connectors, one just for Office 365 (aka Outbound to Office 365) and the other to send to internet AND they have different TlsCertificateName assigned to them. “Outbound to Office 365” Send Connector will have the certificate which matches TlsSenderCertificateName Inbound Connector and “Outbound to Internet” Send Connector will have a different TLS cert CN or SAN which doesn’t match at all the Inbound Connector TlsSenderCertificateName. Example:
How can I force X-MS-Exchange-Organization-AuthAs: Internal for non-authenticated messages?
For Exchange on-premises, the only supported scenario for this would be using a receive connector assigned as Externally Secured permission (which grants other permissions). For Exchange Online, you can use an inbound connector with TreatMessagesAsInternal. Any other method such as Transport Rule is not supported.
I have a lot of custom settings on connectors, my concern is if I run HCW and break mail flow, what should I do?
If you don’t have Centralized Mail Transport enabled, technically you have no reason to be afraid. If you have Centralized Mail Transport enabled and use many other custom settings, we encourage you to assess the environment before running HCW. Avoiding HCW is not really a solution; sooner or later you should run it to change a certificate that will expire, right?
You can get and save all attribute values of Receive Connectors, Send Connectors, Inbound Connectors, Outbound Connectors, accepted domains, and remote domains. Once you assess all this information, even if HCW changes some parameter that breaks the mail flow, you will be able to compare before and after state and fix it.
Let me give you a common example: for Exchange 2010, HCW creates an ad hoc receive connector to receive emails from EXO, called “Inbound From Office 365” containing the whole EOP IP range on RemoteIPRange attribute. If you have a NAT which translates the public IP to an internal IP, you must add only this internal NAT IP to RemoteIPRange, otherwise Exchange 2010 will refuse EXO messages. When you run HCW, it will overwrite your NAT IP and add the whole EOP IP range again. So, you need to know the NAT IP before running HCW to re-add as soon as HCW finishes.
Well, that was a lot! Hope you could find relevant information to perform your next SMTP troubleshooting and please feel free to leave comments and feedback below!
Thanks to Hui Gao, Arindam Thokder, Nino Bilic and Scott Landry for their review and contribution of this post.