TLS Certificates in Exchange Hybrid - Common Issues & How to Fix them

Overview of Hybrid Mail Flow

The Hybrid deployment between Exchange Server (on-premises) and Exchange Online provides secure mail routing between on-premises and Exchange Online organization with the same domain namespace (both can use Contoso.com SMTP domain). While a self-signed certificate is enough for internal mail flow between servers in on-premises organization, when configuring a hybrid deployment, you must use and configure certificates that you have purchased from a trusted third-party CA for hybrid mail flow.  Typically, running the Hybrid Configuration Wizard (HCW) configures everything seamlessly, and the certificate is assigned to the on-premises connectors sending and receiving email to/from Exchange online. However, issues can arise when the on-premises SMTP-assigned trusted third-party certificate is renewed, potentially disrupting email flow.

This blog explores common Exchange certificate-related email flow issues in hybrid environments and provides effective solutions to resolve them.

Before we start, we highly recommend that you go through the following articles to understand the basics of Hybrid mail flow:

• Certificate requirements for hybrid deployments | Microsoft Learn
• Set up connectors to route mail between Office 365 and your own email servers
• Demystifying and troubleshooting hybrid mail flow: when is a message internal? | Microsoft Community Hub
• Also please watch Establishing Exchange Hybrid Mail flow which explains the basics and some of the scenarios discussed in this blog

The main components for Hybrid mail flow are connectors. There are two sets of connectors involved. For mail flow from Exchange Online to Exchange on-premises, the Outbound Connector from Exchange Online will typically point to your on-premises Exchange server(s). On the on-premises side, the email will be accepted by the Default Frontend Receive Connector. In the other direction, for routing email from on-premises to Exchange Online, a Send Connector is created called “Outbound to Office 365 <GUID>” on the on-premises side which will typically point to your own Exchange Online Tenant. An Inbound Connector on Exchange Online side will accept the email from on-premises servers. All these connectors are either created newly, or existing connectors are modified when you run the HCW.

Connectors most often involved in Hybrid mail flow

Important considerations when renewing Exchange certificates

1. When generating Certificate Signing Request (CSR), make sure to use ‘PrivateKeyExportable’ to TRUE and submit the request to the 3^rd party certificate provider. Renew an Exchange Server certificate | Microsoft Learn

2. Use Exchange Management Shell to complete the Pending Request. When using Import-ExchangeCertificate, set ‘PrivateKeyExportable’ to TRUE, especially if you have multiple Exchange servers. Alternatively, you can use certificate MMC to import a new certificate if PrivateKeyExportable was set to TRUE in the CSR. Complete a pending Exchange Server certificate request | Microsoft Learn. Certificate with Privatekey will show a key icon like below:

3. Ensure that the leaf, intermediate, and root certificates are correctly installed on the server.

To test the health of the certificate chain, you may use the following command:

Get-ChildItem -Path “Cert:\LocalMachine\My\<thumbprint>” | Test-Certificate | FL

If there is an issue with a cert chain, you will see a warning / error 

4. When assigning SMTP service to a renewed certificate, Exchange prompts you to overwrite the existing certificate set for transport services. Some customers may not understand the purpose of this warning and the implications of choosing Yes or No; so, let’s dig more into it:

By default, Exchange servers bind self-signed certificates with transport services. You may check current cert binding using the following command:

Get-TransportService <Server Name> | FL Name, InternalTransportCertificateThumbprint

Exchange uses this certificate for internal communication with other Transport Services. Since this certificate is not going to be utilized outside of your own Exchange environment, it is ok to continue using a self-signed certificate; it works even when the certificate is expired.

When you assign SMTP service to a new certificate, Exchange asks you to overwrite the transport certificate with the new certificate. Note: We encourage customers to continue using the self-signed cert for internal transport communication.

Replacing this self-signed certificate with a third-party certificate for internal transport communication is generally fine. However, if you have an Edge Transport Server with an Edge Subscription, it becomes crucial. A self-signed certificate lasts 5 years, while a third-party certificate lasts only 1 year. If Transport Services use a third-party certificate when completing Edge Subscription, you'll need to resubscribe to your Edge Servers annually. With a self-signed certificate, this task is required every 5 years. This is why we recommend customers not overwrite the default SMTP certificate unless you are removing the self-signed certificate for some reason.

5. It is crucial to comprehend how Exchange selects SMTP certificates for communication. As detailed in the article Selection of Inbound STARTTLS Certificates | Microsoft Learn, Exchange initially examines the `TlsCertificateName` set on the connector and searches the certificate store for a certificate with matching issuer and subject. If multiple certificates with identical values are found, Exchange chooses the most recent certificate based on the expiry date. Notably, Exchange does not verify if the matched certificate has SMTP services enabled. Therefore, it is essential to import the certificate only when you are prepared to assign SMTP services.

Steps to Take After Renewing an Exchange Certificate

Using Hybrid Configuration Wizard [HCW]

As we mentioned before, running HCW is recommended when the certificate is renewed in a hybrid environment. In the latest HCW version, it is possible to update the mail certificate for connectors only, ensuring that HCW does not alter any custom configuration. Please use the Update Secure Mail Certificate for Connectors option during the Choose Exchange Hybrid Configuration run of HCW.

When we run HCW for mail flow configuration in hybrid environment, the following parameters are modified or set on the existing or newly created connectors:

Exchange on-premises:

• Send Connector {Default Hybrid connector [Outbound to 365]
  • AddressSpaces = smtp:<domain>.mail.onmicrosoft.com;1
  • CloudServicesMailEnabled = TRUE
  • DNSRoutingEnabled = TRUE
  • RequireTLS = TRUE
  • SmartHosts = 0
  • SourceTransportServers = All the selected Ttransport server
  • TlsAuthLevel = DomainValidation
  • TlsCertificateName = As per the selected 3^rd party certificate
  • TlsDomain = mail.protection.outlook.com

• Receive Connector {Default Frontend <ServerName>}
  • AuthMechanism = Tls, Integrated, BasicAuth, BasicAuthRequireTLS, ExchangeServer
  • TlsCertificateName = As per the selected 3^rd party certificate
  • PermissionGroups = AnonymousUsers, ExchangeServers, ExchangeLegacyServers
  • TlsDomainCapabilities = mail.protection.outlook.com:AcceptCloudServicesMail

Exchange Online

• Standard Inbound Connector {Inbound from <GUID>}
  • CloudServicesMailEnabled = TRUE
  • ConnectorSource = HybridWizard
  • ConnectorType = OnPremises
  • RequireTLS = TRUE
  • SenderDomains = smtp:*;1
  • TlsSenderCertificateName = <as per the 3^rd party certificate subject>

• Standard Outbound Connector {Outbound to <GUID>}
  • CloudServieMailEnabled = TRUE
  • ConnectorSource = HybridWizard
  • ConnectorType = OnPremises
  • RecipientDomains = <PrimarySMTPDomain>
  • RouteAllMessagesViaOnPremises = False [By default CMT is disabled]
  • SmartHosts = <On-Prem server SMTP address>
  • TLSDomain = “As per 3^rd party certificate subject”
  • TlsSettings = DomainValidation
  • IsTransportRuleSoped = False
  • UseMSRecord = False

When using HCW to update connector configurations, you can compare the pre- and post-HCW configurations on the connectors to check for changes in custom settings.

Option 1 – before re-running HCW back up current configuration

• Login to one of the Exchange on-premises servers
• Open Exchange Management Shell
• Run the following commands:

  • Get-SendConnector | Export-Clixml c:\temp\OnPremSendConBkp.xml

  • Get-ReceiveConnector | Export-Clixml c:\temp\OnPremRcptConBkp.xml

• Connect to Exchange Online Remote PowerShell
• Run the following commands:

  • Get-InboundConnector | Export-Clixml c:\temp\EXOInConBkp.xml

  • Get-OutboundConnector | Export-Clixml c:\temp\EXOOutConBkp.xml

• To review previous configurations when facing email flow issues, import collected XMLs for comparison using the following command:

  • $VariableName = Import-Clixml c:\temp\<filename>.xml

Option 2 – after running HCW, review the HCW log

• Log file location: C:\Users\<Admin Profile>\AppData\Roaming\Microsoft\Exchange Hybrid Configuration
• Open the latest .log file
• Search for “actual”, you will see 4 tables [Send connector / Receive Connector / Inbound Connector / Outbound Connector] with the information as follows:

• Check the 'Equals' column for any False values. If found, it means HCW changed it. To revert, set it back to the Actual value.

Manually Updating Exchange Certificate Configuration

If you still prefer to avoid running HCW for certificate-related changes after renewing a third-party certificate, here are some configurations that need to be set manually on Hybrid SMTP connectors:

Email flow from Exchange Online to Exchange on-premises

• Outbound Connector on Exchange online
  • TlsDomain should match with Exchange on-prem 3^rd party certificate SAN
  • TlsSettings should be set to ‘DomainValidation’

• • • Get-OutboundConnector -Identity "Outbound to <GUID>" | FL Name,TLSDomain,TLSSettings

    • Name : Outbound to <GUID>

    • TlsDomain : *.contoso.com

    • TlsSettings : DomainValidation

  • CloudServicesMailEnabled should be set to $true

• Exchange on-premises Receive Connector
  • By default, ‘Default Frontend’ receive connector is responsible to accept SMTP traffic from Exchange Online. TlsCertificateName should set to the renewed 3^rd part certificate. Use the following command to update TlsCertificateName:

    • $cert = Get-ExchangeCertificate -Thumbprint "<new certificate thumbprint>"

    • $TLSCertificateName = "<i>$($Cert.Issuer)<s>$($Cert.Subject)"

    • Set-ReceiveConnector -Identity "Receive Connector Identity" -TlsCertificateName $TLSCertificateName

Email flow from Exchange on-premises to Exchange Online

• Exchange on-premises Send Connector
  • By default, Outbound to Office 365 <GUID> is the connector responsible to route emails to Exchange Online for EXO routing domain “<domain>.mail.onmicrosoft.com”
  • TlsCertificateName should be set to a valid 3^rd party certificate. Use the following command to update TlsCertificateName:

    • $Cert = Get-ExchangeCertificate -Thumbprint "<new certificate thumbprint>"

    • $TLSCertificateName = "<i>$($Cert.Issuer)<s>$($Cert.Subject)"

    • Set-SendConnector -Identity "Send Connector Identity" -TlsCertificateName $TLSCertificateName

• Inbound Connector on Exchange Online
  • TlsSenderCertificateName matches with Exchange on-premises 3^rd party SMTP certificate SAN. Run the following command to check the configuration:

    • Get-InboundConnector -Identity "Inbound from <GUID>" | FL Name,TLSSender

    • Name: Inbound from <GUID>

    • TlsSenderCertificateName : *.contoso.com

Before diving into some specific troubleshooting of errors that we see most often, we wanted to mention That you can always check the TLS certificate configuration in a hybrid environment using Exchange Health Checker:

HealthChecker - Microsoft - CSS-Exchange

Common issues and solutions after renewing an Exchange SMTP certificate in a Hybrid environment

Possible cause:

Exchange on-premises receive connector does not have correct TlsCertificateName or FQDN set.

What to do:

Make sure Exchange on-prem is using correct Receive Connector when responding to Exchange Online request.

• Make sure ProtocolLogging is enabled on all the receive connectors
• Frontend SMTP Receive logs are located at “%ExchangeInstallPath%\ TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive”
• Open the latest log file and search for the thumbprint mentioned in the error. Check if the correct Receive Connector is mentioned in the log. By default, Default Frontend is responsible for receiving traffic from EXO.
• If receive connector is valid, check TlsCertificateName value on that receive connector and make sure TlsCertificateName matches the value you have set on the EXO Outbound Connector TLSDomain value.
• Set correct TlsCertificateName value by running the following command:

• $cert = Get-ExchangeCertificate -Thumbprint "<new certificate thumbprint>"

• $tlscertificatename = "<i>$($cert.Issuer)<s>$($cert.Subject)"

• Set-ReceiveConnector -Identity "Receive Connector Identity" -TlsCertificateName $tlscertificatename

Possible cause:

An issue with the certificate chain.

What to do:

Check Frontend Receive Connector logs, and you will notice that StartTLS was not called out for Exchange Online connection:

2025-01-29T20:14:51.470Z,E19\Default Frontend E19,08DD34FA3086278D,3,10.0.0.1:25,104.47.58.43:57811,>,250  E19.Contoso.lab Hello [104.47.58.43] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES X-ANONYMOUSTLS AUTH NTLM X-EXPS GSSAPI NTLM 8BITMIME BINARYMIME CHUNKING SMTPUTF8 XRDST,

Note down the TlsCertificateName value set on the Receive Connector responsible to receive tariff from EXO. Run the following command to check cert chain. If you see the error, either the cert chain is missing or added under untrustedroot. Check and make sure leaf, intermediate and root certs are present on the correct certificate container.

[PS] C:\>Get-ChildItem -Path "Cert:\LocalMachine\My\886843E0A76558D7209FD3945B62A951FD08ECD8" | Test-Certificate |fl
WARNING: Chain status:
    CERT_TRUST_IS_REVOKED
    CERT_TRUST_IS_UNTRUSTED_ROOT
    CERT_TRUST_REVOCATION_STATUS_UNKNOWN
    CERT_TRUST_IS_OFFLINE_REVOCATION
Test-Certificate : The certificate is revoked. 0x80092010 (-2146885616 CRYPT_E_REVOKED)

Review Application Event Logs and if there is any issue with certificate or certificate chain, you will see event 12014.

Possible cause:

Exchange on-premises send connector does not have correct TlsCertificateName or FQDN set.

What to do:

This error means that Tenant attribution happened against recipient domain not against ServerIP or Server Certificate. If you are using certificate based inbound connector in Exchange Online, check TlsCertificateName set on the Exchange on-premises send connector and make sure it matches with the TlsSenderCertificateName set on the Exchange Online Inbound connector. If it is incorrect, change the setting by using the following commands:

• $cert = Get-ExchangeCertificate -Thumbprint "<new certificate thumbprint>"

• $tlscertificatename = "<i>$($cert.Issuer)<s>$($cert.Subject)"

• Set-SendConnector -Identity "Send Connector Identity" -TlsCertificateName $tlscertificatename

In Exchange Online, if TlsSenderCertificateName is NOT set as wild card value like “*.contoso.com”, you need to make sure the TlsSenderCertificateName value is added in the AcceptedDomain. Otherwise, you will see the same error when sending email from Exchange on-premises to Exchange Online.

Possible cause:

Invalid TlsCeritificateName set on the Send Connector

What to do:

• Check Exchange on-prem message tracking and see that the email was received by exchange server but was not processed further.
• Check the queue on exchange on-prem server and see the error.

This error occurs if TlsCertificateName value set on the Send connector either does not match with any available certificate or matched cert is invalid.

Change TlsCertificateName value on the respective send connector to match with the correct Exchange certificate.

Shashank Agarwal, Arindam Thokder, Jason Lockridge and Josh Hagen