Released: March 2021 Exchange Server Security Updates
Published Mar 02 2021 01:08 PM 1.1M Views

Note: this post is getting frequent updates; please keep checking back. Last update: 3/19/2021

Microsoft has released a set of out of band security updates for vulnerabilities for the following versions of Exchange Server:

  • Exchange Server 2013
  • Exchange Server 2016
  • Exchange Server 2019

Security updates are available for the following specific versions of Exchange:

IMPORTANT: If manually installing security updates, you must install .msp from elevated command prompt (see Known Issues in update KB articles)

Because we are aware of active exploits of related vulnerabilities in the wild (limited targeted attacks), our recommendation is to install these updates immediately to protect against these attacks.

The vulnerabilities affect Microsoft Exchange Server. Exchange Online is not affected.

For more information, please see the Microsoft Security Response Center (MSRC) blog.

For technical details of these exploits and how to help with detection, please see HAFNIUM Targeting Exchange Servers. There is a scripted version of this check available on GitHub here.

 

Mitigations, investigation and remediation:

Are there any mitigations I can implement right now?

MSRC team has released a One-Click Microsoft Exchange On-Premises Mitigation Tool (EOMT). The MSTIC blog post called Microsoft Exchange Server Vulnerabilities Mitigations – March 2021 can help understand individual mitigation actions. A stand-alone ExchangeMitigations.ps1 script is also available.

How can I tell if my servers have already been compromised?

Information on Indicators of Compromise (IOCs) – such as what to search for, and how to find evidence of successful exploitation (if it happened), can be found in HAFNIUM Targeting Exchange Servers. There is a scripted version of this available on GitHub here.

More information about investigations

To aid defenders in investigating these attacks where Microsoft security products and tooling may not be deployed, we are releasing a feed of observed indicators of compromise (IOCs). The feed of malware hashes and known malicious file paths observed in related attacks is available in both JSON and CSV formats at the below GitHub links. This information is being shared as TLP:WHITE. CSV format and JSON format are available. 

What about remediation?

MSTIC team has (on March 6th) updated their blog post Microsoft Exchange Server Vulnerabilities Mitigations – March 2021 to include information about Microsoft Support Emergency Response Tool (MSERT) having been updated to scan Microsoft Exchange Server. Please download a new copy of MSERT often, as updates are made in the tool regularly! Please also see MSRC Guidance for responders: Investigating and remediating on-premises Exchange Server vulnerabilities.

 

Installing and troubleshooting updates:

Does installing the March Security Updates require my servers to be up to date?

Today we shipped Security Update (SU) fixes. These fixes can be installed only on servers that are running the specific versions listed previously, which are considered up to date. If your servers are running older Exchange Server cumulative or rollup update, we recommend to install a currently supported RU/CU before you install the security updates. If you are unable to get updated quickly, please see March 2021 Exchange Server Security Updates for older Cumulative Updates of Exchange Server.

How can I get an inventory of the update-level status of my on-premises Exchange servers?

You can use the Exchange Server Health Checker script, which can be downloaded from GitHub (use the latest release). Running this script will tell you if you are behind on your on-premises Exchange Server updates (note that the script does not support Exchange Server 2010).

Which of my servers should I update first?

Exploitation of the security vulnerabilities addressed in these fixes requires HTTPS access over the Internet. Therefore, our recommendation is to install the security updates first on Exchange servers exposed/published to the Internet (e.g., servers publishing Outlook on the web/OWA and ECP) and then update the rest of your environment.

Will the installation of the Security Updates take as long as installing an RU/CU?

Installation of Security Updates does not take as long as installing a CU or RU, but you will need to plan for some downtime.

My organization needs to 'get current' first... we need to apply a Cumulative Update. Any tips for us?

Please see the Upgrade Exchange to the latest Cumulative Update article for best practices when installing Exchange Cumulative Updates. To ensure the easiest upgrade experience (and because in many organizations Exchange and Active Directory roles are separate) you might wish to run /PrepareAD (in the Active Directory site that Exchange is a member of) before running the actual CU Setup. You can use this document as a guide to understand what you might have to do.

Errors during or after Security Update installation! Help!

It is extremely important to read the Known Issues section in the Security Update KB article (here and here depending on the version). If installing the update manually, you must run the update from the elevated command prompt. If you are seeing unexpected behavior, check the article addressing troubleshooting failed installations of Exchange security updates (we will keep updating this article).

 

Additional Q&A:

Are there any other resources that you can recommend?

Microsoft Defender Security Research Team has published a related blog post called Defending Exchange servers under attack which can help you understand some general practices around detection of malicious activity on your Exchange servers and help improve your security posture.

My organization is in Hybrid with Exchange Online. Do I need to do anything?

While those security updates do not apply to Exchange Online / Office 365, you need to apply those Security Updates to your on-premises Exchange Server, even if it is used for management purposes only. You do not need to re-run HCW if you are using it.

Do we need to install those updates on Management Tools only workstations or servers?

Machines with Management Tools only are not impacted (there are no Exchange services installed) and do not require installation of March SUs. Please note that a 'management server' which many of our Hybrid customers have (which is an Exchange server kept on premises to be able to run Exchange management tasks) is different. For Hybrid, please see the Hybrid question above.

The last Exchange 2016 and Exchange 2019 CU’s were released in December of 2020. Are new CU’s releasing in March 2021?

EDIT: Exchange Server 2016 CU 20 and Exchange Server 2019 CU 9 are now released and those CUs contain the Security Updates mentioned here (along with other fixes). Customers who have installed SUs for older E2016/2019 CUs can simply update to new CUs and will stay protected.

Are Exchange Server 2003 and Exchange Server 2007 vulnerable to March 2021 Exchange server security vulnerabilities?

No. After performing code reviews, we can state that the code involved in the attack chain to begin (CVE-2021-26855) was not in the product before Exchange Server 2013. Exchange 2007 includes the UM service, but it doesn’t include the code that made Exchange Server 2010 vulnerable. Exchange 2003 does not include the UM service.

 

Major updates to this post:

The Exchange Team

293 Comments
Copper Contributor

All,

 

I am  getting while running the update for Exchange server 2016 CU18

 

"Windows Installer looking for Insert the 'Microsoft Exchange Server' disk and click OK". It's looking for MSEXCHANGSERVER.MSI file

 

Any idea?

 

Copper Contributor

My environment is in Hybrid mode, 3 node DAG, Exchange 2013 CU23 on Windows Server 2012. Before attempting to apply the patch I ran Windows Update and installed all available updates, rebooted, waited until .Net optimization was done (upgraded from .Net Framework 4.7.2 > 4.8). Took a snapshot of the server (thank goodness), UAC is disabled, ran the update from an elevated command prompt, failed miserably. Update program would only stop the following service: Microsoft Exchange Search Host Controller. Update rolled back automatically and stated that the update ended "prematurely." All other services remained running. Tried again and this time I manually stopped all Exchange services as well as the WWW service before running the update from an elevated command prompt...same thing. Failed. Reverted snapshot, tried second way again...fail. Reverted snapshot again and ran the update through via Windows Update...success! These security updates for Exchange Server always seem to be a crap shoot whether they install gracefully or not.

Copper Contributor

Single Exchange 2019 - CU3 here. Stuck on Configuring Prerequisites at 85%. Did a windows update. .NET update, C++ update. Put into Maintenance mode. Prepped Schema/AD. Tried going through powershell and GUI. Just gets stuck at 85% Performing Microsoft Exchange Server Prerequisite Check. Where to start?

@MI_IS1 .NET Framework installs and updates can peg your CPU and make installation go really slow for up to an hour. See URGENT: Patch your Exchange Servers NOW! | The EXPTA {blog} for tips.

Copper Contributor

All,

Had  EMS session opened while trying to update,

Closed the EMS and the update went  smooth.

 

Copper Contributor

I manage several exchange servers and got most done. I have two that will not install due to the error below. Both were behind in CU's so we updated them to the latest and rebooted. I ran the msp from an elevated admin cmd. One is exchange 2013 and the other 2016 at different companies.

This was using the domain admin account.

 

err.JPG

 

Thanks,

Paul

 

Copper Contributor

Dear Techs, Is it applicable for the Hybrid setup of 2016 exchange servers? 
When I try to update CU18/CU19 not allowed and error shows that upgrade path is missing. any ideas please?

Copper Contributor

Never mind my post. It was CarbonBlack causing it.

Copper Contributor

I had to update 6 Exchange servers today, from 2010 to 2019. Here's my feedback (also based on comments I've found here)

 

@EddieRowe - from an elevated Powershell, run "C:\Program Files\MicrosoftExchange ServerV14\Bin\UpdateCas.ps1" (or whatever drive letter Exchange is installed on), then restart IIS (IISReset). I also had to set the MS Exchange services back to Auto or manual on the Exchange 2010 deployment I manage. I went by this doc - https://docs.microsoft.com/en-us/previous-versions/office/exchange-server-2010/ee423542(v=exchg.141)...

 

@SteveInReno  - I had this issue on the first 2013 Exchange server I updated today as a result of not running the patch from an elevated command prompt. To Fix it, I ran C:\Program Files\Microsoft\Exchange Server\V15\Bin\UpdateCas.ps1 (this fixed OWA), and then had to correct the BinSearchFolders in IIS Application settings (to fix ECP):

1. Open IIS Manager. Expand Sites > Exchange Back End.
2. Click ecp. Open Application Settings in /ecp Home.
3. Please check whether the value for “BinSearchFolders” is changed to not absolute paths. If so, please change it to (adjust for the correct path/drive to Exchange Server):
C:\Program Files\Microsoft\Exchange Server\V15\bin;C:\Program Files\Microsoft\Exchange Server\V15\bin\CmdletExtensionAgents;C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\Owa\bin

4. Run IISReset

 

Copper Contributor

We finally had success. After closing all windows before running the setup, we were able to get past being stuck on pre-reqs. I then ran across:

https://practical365.com/exchange-server/service-wmsvc-failed-to-reach-status-running-on-this-server...

 

The above fixed that problem. Finally everything patched, except now mail isn't flowing because the hard drive got too full. I had 47gb used out of 256gb and I was down to 27gb free after the update.

 

I'm cleaning up as fast as I can. There's a bunch of log files in C:\inetpub\logs\LogFiles\ that just need to go.

Copper Contributor

@Jeff GuilletCurrently Exchange 2013 CU22 is installed in our Exchange environment. Can you please guide that current threat is affecting this version and where can i find the updates for Exchange 2013 CU22 if there are any.

 

Copper Contributor

Hi

 

After installing the Security update for CU19 we are getting an error in our Admin portal, after logging in we see the following ECP error in the webportal; “Could not load file or assembly ‘Microsoft.Exchange.Common, Version=15.0.0.0 …”ecp error_ex16.png

 

As suggested in https://www.msnoob.com/exchange-ecp-and-owa-error-after-update.html

We tried the following things

- Change the value in the BinSearchFolders 

- Checking the Web.Config files for OWA and ECP

 

When we change the value in he BinSearchFolders it does show some content after logging in but the format of the Admin portal is lost.

 

As far as i can see our installation is default apart of the location of the Exchange installation (on D:\)

 

Is this a known issue?

 

Regards Bas

 

 

 

Copper Contributor

Hi Guys

I have a question regarding the security patch, it could be important for others too. We have applied the Security patch on our Exchange 2016 CU18 servers. We are planning to update the Servers to CU19 in few weeks. Because CU19 install is basically a new Exchange install and CU19 does not contains the Security patch, should we install again the Security patch CU19 version after the CU19 update?

Thank you for your answer!

 

Copper Contributor

Is there any need to apply the fix to Exchange 2016 Edge Servers and Severs where only the Exchange 2016 Management Tools are installed as well?

Copper Contributor

@baskleian - run D:\Program Files\Microsoft\Exchange Server\V15\Bin\UpdateCas.ps1 (Since your Exchange is installed on the D: drive) and then Change the value in BinSearchFolders to D:\Program Files\Microsoft\Exchange Server\V15\bin;D:\Program Files\Microsoft\Exchange Server\V15\bin\CmdletExtensionAgents;D:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\Owa\bin

 

It is a known issue when you double click the patch to launch instead of launching it from an elevated command prompt. (I did the same thing on the first exchange server I patched yesterday)

Copper Contributor

Our current server is Exchange server 2016 CU 15 , is this vulnerbale to the zero day attack ? what will be the best option upadte all CU 's to the latest?

Copper Contributor

@RaviVamadeva: Yes, its vulnerable but the patch is not available. You will need to update to CU18 or CU19 before you can patch. I would recommend upgrading to CU19 and then patching.

Copper Contributor

thanks a lot @GeekSpaz 

Copper Contributor

NickH15_0-1614860791566.png

Hi - we have been following the instructions above, put the server in maintenance mode and we keep getting the following error we are on CU18

Copper Contributor

Hello @The_Exchange_Team,

Thank you your work and helping the community with this vulnerabilities. 

I see in the comments a lot of people got port 444 detection in the results from running PowerShell command for CVE-2021-26855:

Import-Csv -Path ...

In the Hafnium blog post there is no mentioning of such cases. 

Can you please clarify what does it mean if port 444 is detected in the logs? Does it mean that system was compromised? Or maybe any additional actions should be done except installing the updates?

Thank you.

Copper Contributor

Hello,

 

We are applying the cumulative patch for Exchange Server 2013 and getting the following error:

 

This upgrade patch cannot be installed by the Windows Installer service because the program to be upgrade may be missing, or the upgrade patch may update a different version of the program. Verify that the program to be upgraded eists on your computer and that your have the correct upgrade path.

 

Our exchange servers are down right now and we need urgent help. Can someone help here?

 

 

Copper Contributor

@to_vaib: It sounds like you are not on the most recent CU for your Exchange server. For Exchange 2013 you need to be on CU23, so you probably need to update it to CU23 first. To see what version you are on you can run the HealthChecker.ps1 that is linked to in the blog post that all these comments are on, in the section:

How can I get an inventory of the update-level status of my on-premises Exchange servers?

You can use the Exchange Server Health Checker script, which can be downloaded from GitHub (use the latest release). Running this script will tell you if you are behind on your on-premises Exchange Server updates (note that the script does not support Exchange Server 2010).

 

Copper Contributor

Thanks @GeekSpaz ...CU23 is what I'm trying to install when I'm getting this error.

Copper Contributor

Many Thanks @ajc196 copying those missing files, then restarting the msp file fixed the issue

Copper Contributor

Our Exchange 2013 are on CU21 and I understand I need to have CU23. do I need schema and domain update to move to cu23?

Copper Contributor

We updated our servers, without any noticeable issues, but none of them show any change in build number. All of them still show the build number of Exchange Server 2016 CU19. Is this expected behavior?

Copper Contributor

@to_vaib: My apologies - I do see that you said that. We did our CU23 updates back in November so I don't remember it very clearly. However, I did find the steps we went by:

Install procedure

Copper Contributor

@Bhavesh Shah You need to run /PrepareAD, no need to run/PrepareSchema for an update from Exchange 2013 CU21 to CU23

Copper Contributor

Hello,

 

I habe an EX2016 CU14 - do i have to install the latest cu bevor installing the patch against hafnium?

 

Thanks,

 

John

Copper Contributor

@ All who faced OWA/ECP broke issue post the installation. 

 

I had the same issue on one of the nodes in the DAG. The installation did not say if it failed, however I could find an error in the event says the OWA could not load because the default theme file is missing in the directory "C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\Owa\prem\15.1.2106.13"

 

The target directory did not contain any files except a directory and hence I decided to re-run the patch as it seems the copy process did not go through well. A re-apply fixed it for me, but I had to enable and start the services post the reboot.

 

You may also try running the UpdateCas.PS1 as mentioned above. 

-Praveen

Copper Contributor

@Admin0815 Yes you have to. Ensure you have .NET 4.8 framework installed. If not, install it before CU Update. Exchange 2016 CU14 supports it, latest Exchange 2016 CU requires it.

Copper Contributor

@GeekSpaz actually we ran the healthchecker and weird thing is that it shows that the server is already on CU23 (build 15.0.1497.2) however it also shows this:

 

MAPI/HTTP Enabled: False
MAPI Front End App Pool GC Mode: Workstation --- Error
To Fix this issue go into the file MSExchangeMapiFrontEndAppPool_CLRConfig.config in the Exchange Bin directory and change the GCServer to true and recycle the MAPI Front End App Pool
Exchange Server Maintenance
Component: 'ServerWideOffline' is in Maintenance Mode
Component: 'HubTransport' is in Maintenance Mode
Component: 'FrontendTransport' is in Maintenance Mode
Component: 'Monitoring' is in Maintenance Mode
Component: 'RecoveryActionsEnabled' is in Maintenance Mode
Component: 'ImapProxy' is in Maintenance Mode
Component: 'PopProxy' is in Maintenance Mode
Component: 'PushNotificationsProxy' is in Maintenance Mode
Component: 'XropProxy' is in Maintenance Mode
Component: 'HttpProxyAvailabilityGroup' is in Maintenance Mode
Component: 'EdgeTransport' is in Maintenance Mode
Component: 'HighAvailability' is in Maintenance Mode
Component: 'SharedCache' is in Maintenance Mode

 

I guess they shouldn't be in Maintenance mode? How do take them out from Maint mode?

Brass Contributor

@LeeMEI , in our case on Exchange 2010 the admin didn't elevate the command prompt on the first attempt and we had all sorts of issues with OWA.  After uninstalling the rollup we had to reinstall the prior rollup to get everything back to working the way it was before.

Copper Contributor

Hi Trying to install patch in Ex2016CU19, It runs for a while but eventually fails when trying to create images (sorry didn’t catch the exact .dll). Not sure if the patch has an install log or something to help troubleshoot the issue but any help would be appreciated.

Copper Contributor

Hi for All

We have applied what the Microsoft specialists recommend to install this patch ASAP and "immediately". That's OK.

We have applied the patch on servers 2012 exchange 2013 DAGs and CAS CU23 successfully without issues. "ofcourse we found the exchange services were disabled" we enabled them, all is ok except OWA and ECP. unfortunately, untill writing this post and only 500 error and we cannot find the right solution to retrieve these service back. 

 

To the specialists who strongly  recommend to install this patch immediately, Please, support us by clear steps how can we fix it!!!

Copper Contributor

@Jeff Guillet

 

When running the Check Compromise script, we get an error:

 

WARNING: One or more headers were not specified. Default names starting with "H" have been used in place of any missing headers.

 

Is this something that can be ignored?  Thanks in advance for your help.

Copper Contributor

I came across another failed patch scenario last night and thought I'd post the issue and fix just in case it can help somebody else out.

 

This particular environment is a standalone Exchange 2013 CU23 server on Windows Server 2012. Windows is up-to-date, UAC disabled, PS execution policy set to unrestricted for LocalMachine and CurrentUser scopes, snapshot of VM taken. After an initial reboot, attempted to install the security update from the Windows Update client, but it failed. Reverted snapshot, tried installing the update manually from an elevated command prompt…fail. After trying all sorts of combinations and they all failed, I looked at the file "C:\ExchangeSetupLogs\ServiceControl.log" and noticed that the servicecontrol.ps1 file was failing to stop the “WinMgmt” service because it couldn’t stop its dependancies. Manually stopped the service and tried again and this time it failed on stopping “MSExchangeADTopology” because of its dependancies. Fixed that and then after the patch was just about to finish, it would fail again with an error in the log stating that “IgnoreTimeout” was an unknown parameter. I reverted the snapshot again, manually stopped WinMgmt and all Exchange services, modified line 477 from the file "C:\Program Files\Microsoft\Exchange Server\V15\Bin\ServiceControl.ps1"

 

OLD line: start-setupservice -serviceName $serviceName -ev script:serviceControlError -IgnoreTimeout:$IgnoreTimeout

NEW line: start-setupservice -serviceName $serviceName -ev script:serviceControlError

 

After that, ran Windows Update and the security update applied successfully. Rebooted and all is well.

 

So, in summary: Reboot your server, take a snapshot of it if you can, manually stop WinMgmt and all Exchange Services, edit the ServiceControl.ps1 file, install update via Windows Update (or manually from an Elevated command prompt if you wish), reboot, enjoy.

Copper Contributor

err.png

Has anyone ever had this error?

The directory has permissions as does the file mentioned.

Thanks in advance.

Copper Contributor

This patch broke IMAP shared mailbox access on EX 2013 CU23.

Copper Contributor

@The_Exchange_Team Hi, if i'm currently at Exchange 2013 CU9 what kind of problem could i run into while upgrading to U23 ? Do i need to update to another CU before ?

 

Same question but for Exchange 2016 CU4 to CU19 ?

 

Thanks

Steel Contributor

@Zhymmer Someone else may have to speak on whether jumping that far is still supported in a single go (we always stayed n-1 or n-2, so no experience there).  However, I can say that you'll be looking at a .NET update prior to installing the new CU of both of those 2013/2016 scenarios you mentioned. Check the support matrix for each version.

 

https://docs.microsoft.com/en-us/exchange/plan-and-deploy/supportability-matrix?view=exchserver-2019...

Steel Contributor

@pepe89 If you open up services.msc, can you see if all Exchange services actually stopped?   One of eight servers that I updated did this, turns out the update package didn't succeed at stopping services but continued anyway. Once I stopped the remaining services, the install continued upon hitting "retry".

Copper Contributor

I got it by running the KB from cmd with elevated permissions.

Thanks @ajc196

Microsoft

I got some catching up to do; thanks to everyone who has been answering questions! :)

 

@HYper83 Yes, if you have E2016 CU18 and then install the security patch and then install E2016 CU19, you will have to install the CU19 security patch after CU19 is installed.

@Marc4056 Yes on Edge. No on Management tools

@Dominik Mönks Run the Health Checker script, it will tell you; but if you ran the install (from elevated CMD prompt, or from Microsoft Update) and it installed with no errors, you are good!

@marceloi750 Please reboot the server before trying to install; possibly something has a handle on that file. Do you use a file-level AV software that might be doing that? Then, when you are installing the update, make sure you are running it from elevated CMD line

Brass Contributor

@JBCCNZ , we have struggled with the RU32 patch on Exchange 2010 SP3 on 2008R2, but our issue is that it looks okay but breaks OWA for mailboxes on that Exchange server. (Webmail runs on another server that has not been patched yet.)  We have been able to uninstall RU32 and reinstall RU30 to return everything to working status.  RU31 which does not correct the issue, but is newer than what we have causes similar issues for us. Luckily our firewall vendor has blocked the attack service on the network perimeter while we are working on getting this patch installed without breaking email for the entire company.

Copper Contributor

Anyone notice log replay is slower than before the patch was applied (Exchange 2019/CU7)?  We also saw where the exchange services did not start after the reboot even though they were set to Automatic.  Manually restarting them was fine and the servers were functional afterwards.  Installs were all done from an elevated command prompt.

Copper Contributor

@EddieRowe thanks for sharing your experience :)

Iron Contributor

When running Hafnium check script we get the results like

 

"DateTime","AnchorMailbox"
"2021-03-04T16:08:27.196Z","ServerInfo~a]@SERVERNAME.company.com:444/autodiscover/autodiscover.xml?#"

 

Anything to worry about?

Copper Contributor

Hi there,

 

I have the same question as Mika Melonen, csummers93, bgg_26 and Arūnas Malūkas.

 

We also found "DATETIME","ServerInfo~a]@SERVERNAME.company.com:444/autodiscover/autodiscover.xml?#"

Does anyone know what this means?


Only some non-invasive test of system? or real penetration into system?

 

There is information that the attempt took approximately 50ms.

BeginRequest=2021-03-03T07:42:07.769Z
EndRequest=2021-03-03T07:42:07.816Z

 

Some ideas?

Copper Contributor

@EddieRowe: I got around the OWA issue by running the UpdateCas.ps1 script on the 2010 server I patched last night.

Co-Authors
Version history
Last update:
‎Mar 19 2021 01:44 PM
Updated by: