‘Last Exchange Server’ Scenario Feedback

[The_Exchange_Team]

In April 2022, we released an update to Exchange Server 2019 Management Tools that enables organizations that use Azure AD Connect and sync their Active Directory to manage Exchange recipients without the need for a running Exchange Server on-premises. 

If you have one or more Exchange servers that are used only for recipient management (often referred to as Last Exchange Server - LES), you can install the updated tools on a domain-joined machine and shut down your last Exchange Server. For more information, see Manage recipients in Exchange Server 2019 Hybrid environments. 

We want to hear from you if the above has been helpful to you and further understand the scope for improvement. We're particularly interested in learning what obstacles are still preventing the shutdown of your last Exchange Server if your organization does not need full Exchange on-premises. We’d also like to understand your Exchange management scenarios better if you have a hybrid organization (and plan to stay in hybrid). 

Please take a moment to fill out the following survey: 

https://forms.office.com/r/hMnjGagbGw

Exchange Online Management Team 

[PeterForster]

Simply put, the SMTP relay question remains unresolved. I am aware that we can directly route apps and devices to Exchange Online, but this involves a significant effort. Moreover, many devices lack internet access. The direct relay on port 587 necessitates open firewall ports, and ideally, a static IP address should be configured on the outbound NAT.

While this approach might work for fewer than 10 devices, many of our customers have a much larger number of devices. Consequently, they often choose to keep the last Exchange server operational. Unfortunately, the current options for correctly authenticating emails as ‘Internal’ within the Auth-Headers pose too many security risks.

Thanks for the feedback form: Done!

[DustinDortch]

@PeterForster You can stand up another platform for mail relay and have it treated as "internal" mail, still.  It takes some effort, but I have done this for customers with Postfix (I even built an Ansible role to make it easy).  Use certificates and authenticate the traffic that way.

[DustinDortch]

I think the most significant feature that is lost is Email Address Policies.  This is crucial for larger environments to ensure that addresses are set according to a standard and appropriately remediated when a conflict arises.  If there is a service that could be run, or a well-designed PowerShell script that accounts for the same considerations (perhaps even relies on the actual Email Address Policy data stored in Active Directory for its rules), that would be awesome.  Maybe that has been addressed in some other way that I have missed, though.

[PeterForster]

@DustinDortch thanks for your idea. We thought about this too, but: Exchange did this all out of the box. Not every customer is Linux ready and having a very important function (for many customers SMTP relay is one the business critial functions) on a system they cannot manage is not an option for them.

We have customers still running two Exchange servers based on the relay requirement.

But how did you set the correct headers there? The option to mark the inbound connector as "internal" is not one we would like to use as regardless what system will later send with that public IP is marked as internal. This is the security issue we see here.

[DustinDortch]

@PeterForster That is so often the case.  In fact, the customer I first set this up for basically took away all of the options... all of them.  They were previous on Lotus Notes/Domino and we migrated them to Exchange Online.  The Domino servers were going to be kept around until all of the custom apps they ran were mitigated.  When that happened, they came back to us to ask for assistance with mail relay.  I suggested installing six (6) Exchange server (their applications send a lot of messages, many business critical and customer-facing).  They said that it was fine but insisted on using "Split Permissions"; I work with a lot of folks that have a ton of experience with Exchange and we even spoke with Microsoft... the consensus was, "run for the hills."  So, I proposed Linux with Postfix and they were reluctant.  As far as "enterprise grade" solutions, the options were really, Exchange, Linux with Postfix, or some appliances; they didn't like any of them, but with my Ansible role, I made it to where they could manage it all with a playbook.

In terms of marking it as internal, we rely on certificates to identify that valid traffic.  So, the connector can safely be marked as internal because only the servers with the valid certificates will match to that connector.

[PeterForster]

@DustinDortch That is interesting that messages routed the way you mentioned will have the Auth-As Header as "Internal" as mentioned here: Demystifying and troubleshooting hybrid mail flow: when is a message internal? 

Fully unclear how this should work as the Header Firewall from Exchange comes here into place. This is not related to a TLS session for SMTP.

[DustinDortch]

@PeterForster You can make an "OnPremises" connector and change the configuration for how it identifies the traffic.

New-InboundConnector `
-CloudServicesMailEnabled $true`
-ConnectorType OnPremises`
-Enabled $true`
-Name "Inbound from Postfix On-Premises"`
-RequireTls $true`
-RestrictDomainsToCertificate $true`
-SenderDomains *.domain.com`
-TlsSenderCertificateName *.domain.com`
-TreatMessagesAsInternal $true

Something like that should do the trick.  It requires setting the certificate and restricting only traffic with the matching certificate and setting it for OnPremises and as CloudServicesMailEnabled.

I haven't run it in a while, but it worked the last time I checked it.

Back before cross-tenant migrations were possible, I used this same sort of thing to rewrite email addresses by sending traffic to Postfix, letting it rewrite, and then send it along to the appropriate tenant.

[blog-it-koehler]

As already mentioned, SMTP Gateway to Exchange Online is one issue, especially in smaller environments. Not every SMTP client (printers etc.) is ready for TLS1.2, so these customers need not to replace that systems as soon they are uninstalling the last exchange server. (I know you should replace them, but customers do not recognize and some don't want to hear it).  Another point is, in systems with only one admin, nobody understands why and how you should manage AD attributes. There are no PowerShell skills available and these admins hardly can handle the Exchange GUI / Exchange Online GUI. A lot of things can be done inside Exchange Online, so why not changing or adding SMTP Addresses.  I'm consultant for this, I know how to manage that but people who need to administer many different systems will probably not understand these (little) complex symbiosis of local AD and Exchange Online.  Thanks for your work and maybe there will be a smooth way to make this great E-Mail system a little handsome. 

Thanks for this great blog and your work Exchange Team. 

[MatthiasMichl]

while the already mentioned issues with SMTP relay (we have WinServer default SMTP (deprectated) in place), and software/hardware not capable to use TLS(1.2) etc apply for us as well, we use mailboxes on-prem for our Admin-Accounts.

While yes, we know an Admin should not have a mailbox etc... We were unable to figure out a 'good' solution. Any solution has its drawbacks of any sort and you go in circles and won't find an exit.

So we are using a regular non-permission account for the 'B*S* stuff' and 'elevated accounts' for the 'Admin stuff'. Both on-prem AD and synced up to EntraID. The Admin accounts are also used for the admin stuff in M365 of all sorts. with that - basically everything in M365/Azure sends a mail... but to your admin account. Due to the sync, you can not tell your admin account to use the same email address as your regular account as well as not set your admin mail address as alias to your regular account. you get a conflict. Having shared mailboxes (or even licenced ones) in EXO for admins - not good practice and i think in the long run you get in trouble. Having admin accounts for on-prem and online stuff separated... extremely clunky. Having the admin account being a mail-user -> sync conflict again.

So our solution here is to use on-prem mailboxes for the admins and let the mails forward to the regular account. We found this the least intrusive and best compromise.

My wish to MS: let us somehow define the same emailaddress for the admin accounts (maybe an attribute in AD on the admin account for a fallback-mailaddress or something like that, that then is used in the M365/Azure world to send mails to, so they would appear in the regular mailbox).

Other than that - any idea to solve that kind of 'hen-egg problem' is welcome. I know, if you have a standlone online admin account, you simply fill the mailfield with your mail address and it works without running into a conflict (at least it was in the past). why - no idea.

[zivrivkis]

@MatthiasMichl I may be misunderstanding your issue, but wouldn't a mailbox rule (fwd to second email address?) or similar would work? Create a rule in the admin mailbox to automatically forward all email to your non-admin mailbox? Don't mean to highjack the thread.

[Alexmags1337]

Solving smtp relay could unblock data centre to azure migrations. Azure prohibits outbound SMTP ports so you can't host your own SMTP relay server for internal apps/document scanners. Sendgrid with subdomain of companyname.com could be workaround?

Email address policy to setup new joiner address without collisions (upn not same as mail)

[PeterForster]

@DustinDortch and that is the problem. Now every device/app which is sending is treated as internal. That is not something we would like to allow. Departments (when they know the configuration) can just configure it. Malware that tries to send out messages can to this to internal recipients and the messages are marked as Internal. That is a no go for security reasons to configure it on the connector for us.

Now - it only works when we allow the IP-address on the Exchange or Loadbalancer in front of Exchange. No one else can just send messages without proper configuration and allowance.

[wroot]

I guess we were brave (or stupid) and removed last Exchange server right after migration to EO on my previous work around 2017 😄 We have stood up Windows Server acting as SMTP replay based on IIS. It was enough for all internal systems like on prem SharePoint 2013, various custom built apps and a dozen of printers/scanners. Just pointed them to smtp.contoso.com without auth, but with IP filter and message size limit. And between this server and connector on EO side we used cert auth.

[DustinDortch]

@PeterForster Well, it is starting to seem like you're just looking to shoot this down.  From what you said before, that is surely the behavior you wanted... to treat the messages as internal.  Here's the deal, you can accomplish any of this.  Just like in Exchange, you're able to identify sources by different means and treat them differently in Postfix.  So, if you have some systems that need to be treated as internal, you can do so... using authentication (usernames/passwords, certificates, etc.) or address spaces; then you can send it out to a specific destination with certain properties, like using a certificate to match the traffic to appropriate InboundConnector for treating it as internal.  Then, if you have other systems that you want to treat as external, you can identify it otherwise and have it sent to a different (or the same) destination with different properties that would match the traffic to the default connector to treat it as external.

The external bit is easy because that is the default behavior.  It seemed quite like we were discussing the possibilities of the situation that presents challenges (because that is what you were citing) so we could treat the traffic as internal.

Rest assured, we can accomplish about anything with Exchange and/or Postfix.  They're both highly extensible and mature.

[Thestutes]

I agree with Dustin Dortch. The lack of the Email Address Policy feature in EXO is in my top 3 reasons of why we must retain Exchange On Prem

[Daoudi93]

other than SMTP relay issues, even relying on Exchange online has it is own limitations by default the SMTP controls the rate limiting for SMTP per minute which causing issues for large scale organizations, 10000 message per day in some sceniros is not enough for a legit non-bulk emails, for example notifications from ERP system to be sent across all organization users is a problem, relying on 3rd party tools like SendGrid as example is not always easy to especially when security requirements requires OAuth since the traffic will be over internet, while Exchange on Premises provides till date the best option for these scenarios, and securly accessible within the internal network without exposing it over internet.

[Daoudi93]

Another big issue is dealing with email-enabled security group management, for example organisations which uses the same group as distribution list and granting application or file shares access using the same group will never be able to rely on EXO management, unfortunately even after reaching out to Microsoft team to find a proper solution for this scenario, LES won't work in this case.

I have evaluated Tim McMichael solution to automate the creation of the same group to exchange online and rename the on premise group and remove the email address so we end up with 2 groups, on premises as just security group where the SID maintained the same so the assigned permissions won't be impacted and then the distribution list will be running from EXO., but with +15000 existing groups in scope it won't be practial for managing and maintaining +30000 after splitting the groups and link the permissions manually twice when any membership requires modification for each group.

Below is a reference for the great tool which Tim has worked on 

https://timmcmic.wordpress.com/2023/02/21/office-365-distribution-list-migrations-2-0-part-33/

[QuantumQuest]

It's great to have an option to remove the last Exchange Server, but there are a few things that prevent some of my customers of doing this:

- Exchange uninstallation: It would be great to have an option to uninstall the last Exchange Server without removing the Attributes, that are necessary to run Hybrid. It just looks bad, if you have a dead server there, it feels like missing one step.

- SMTP relay: Often we use third party Software for this. If you have MFPs that you don't want to connect with an anonymous connector, there is no other option to it. The third party Software deploys a SMTP server which takes anonymous e-mails and sends them to the GraphAPI, A tool like this, directly from Microsoft, would be awesome

- Powershell Management: A lot of customers don't feel comfortable to create new remote users within Powershell. If there was an option to do this with a small GUI (reduced ECP) it would help so much.

Last One:
Why are you not just changing Entra Connect, to make the AAD the Primary source of users? User, Group, Mailboxcreation, all should move to M365 and synced back to the local AD. Like this you would have your GUI and don't need to care about doing onPrem stuff if you only need it for a few services. The only thing thats complicate would be to move them to the correct Forest and OU, but you could sync or mirror the whole forest to display it within the AAD.

Any of these updates would be high appreciated by my customers 🙂

[DustinDortch]

@QuantumQuest - I think a lot of these items don't really show an appreciation for the challenges.

If you're looking to remove the last Exchange Server and you're still maintaining Hybrid Identity, you're not the "average" customer, in that situation.  So, the expectation that you would need to prune AD of the related objects for the server in order to clean it up doesn't seem like a significant concern.  At some point, there has to be a limit to what sort of things require some skills to manage something.  This includes PowerShell or otherwise, as well.  That doesn't mean tools can't be made, but it starts to get to the point of "If you see a need, fill the need".

If you want an ECP and still also need SMTP relay, then not eliminating your last Exchange Server is really the way to go.

The synchronization issue is far more difficult than you're giving credit.  If you're still looking for AD to exist, AD should be the source of truth.  If you notice how synchronization works in the other direction, Entra ID/Azure AD has always been a flat structure for customers.  This is because synchronizing into another structure would require a lot of manual care and feeding... mapping OUs from one side to another, or mapping users specifically.  If that is a concern, manually creating users on the other side might be less concern.  If there are only a few things, changing the permissions to work with Entra ID/Azure AD is also a possibility which allows for the elimination of AD (many times easier said than done).  Universal Print is one of those services available that could assist in that effort.

[Gareth Gudger]

100% SMTP Relay. Relaying to Office 365 is (1) Just too cumbersome because every device and app might have different capabilities for relay, and (2) why would I allow and continue to manage hundreds of outbound IPs on my firewall to lock down port 25 when I can use a known set of Exchange Server IPs?

[QuantumQuest]

@DustinDortch 

"If you're looking to remove the last Exchange Server and you're still maintaining Hybrid Identity, you're not the "average" customer, in that situation."
I don't know if thats true. At least from what i've seen, the majority of the customers want's to either get proper access to Teams/SharePoint, or they want to get rid of their Exchange Server. Most of them are very disappointed when all mailboxes are migrated and i say: "Yes we can remove the Exchange server but...". Most of them use 3rd party Software which needs a local AD. Yes, in a perfect world the software would be able to use the Azure AD but thats not always possible. I've migrated around 40 customers to Exchange Online, 2 of them are Azure only, 2 of them have the Exchange Management Tools, around 6 or 7 had no Exchange before, but most of them still have one even if they don't want it.

"If you want an ECP and still also need SMTP relay, then not eliminating your last Exchange Server is really the way to go."
Exchange is way too huge for these two things. There are 3rd party GUI solutions for the Exchange Management Tools, even the 3rd party integration into the AD directly (Easy265Manager). You could even adjust it with ADSI Edit only, but no we should get a full server for this task? I think Microsoft could write a tool for that in 2 weeks which replaces the local ECP standalone. Same thing for Mail relay, the GraphAPI is so good, creating an SMTP Server which accepts unauthenticated SMTP locally from whitelisted IPs and pushing the mails through the GraphAPI should be possible for one of the biggest companys in the world.

I accept your answer regarding the sync back to the AD since this is definitly not so easy. But there is device, group and password writeback, i see no reason why it's not possible to write back a user aswell. This would eliminate the need of a local ECP aswell. When you need to move the user from some sync OU to the destination OU, that would be a manual task which is way faster for some customers.

To finish this: I have no problem on how it is, but a lot of my customers don't like it, and there is 3rd party software which solves all problems. I think Microsoft could copy/buy these within a few months instead of forcing customers to Azure only while not everything is possible there.

[GustavAndersson950]

You really need to make the whole process for de-comissioning the last exchange server much easier. You have to go through a thousand steps 

[MateKovacs]

SMTP relay is a big one for us.

[Dexter_Hoban]

We are currently utilizing the Exchange Admin Center on-premises for our desktop team to create, modify and delete users.  It is nice to have a clean, simple web interface for their whole team to use.  As mentioned in one of the previous comments, we are also utilizing an Email Address Policy to ensure standardized email addresses.  Ideally - there could be an installable version of ONLY the Recipient portion of the Exchange Admin Center to still provide that inferface on-premises.

[StephanGee]

We solved the SMTP Relay with 1 IIS Server with SMTP Role installed and scoped on IPs

Works great (for printers that send PDF scans)

All other tasks are done by scripting and automation. We will get rid of it this year.

[PeterForster]

@StephanGee SMTP in IIS is deprecated since 2012 and the feature is removed in Windows Server 2022 completely - this is not a solution at all for secure environments.

[Jpanski]

Are the results of this survey going to be made available to the public?  Thank you.

[David_Patterson]

Question #3 (how many mailboxes in your on-premises environment) does not have 0 as a possible answer, although question 5 seems to acknowledge the case of a server for doing recipient-management only.  It has been unclear to me if we need to remain in hybrid mode, now that all of our mailboxes are moved to the cloud, when we are still using on-prem AD as the identity source.  I would like to eliminate the hybrid connectors, as the certificate used on those is creating issues for us because the same certificate is also used on other email-sending devices, but I have been afraid undoing it would break something.  I would appreciate an article that clarifies whether the hybrid configuration and connectors can be removed as an initial step toward removing the on-prem server altogether, or if that cleanup can only happen more-or-less simultaneously with removal of the on-prem server.

[Mr1080p]

For me, the documentation has been pretty good. There were a couple of commands I had to adapt for our environment related to oAuth.

Using the EMT has been fine for us, so I am going through the permanent shutdown process - there's an error in the PowerShell commands in the documentation related to finding the KeyId (Step 5b) for oAuth.

The $true I don't think is supposed to be there, and I suspect $.Value no longer exists as Get-member doesn't show that attribute and there's also a syntax error related to brackets.