Released: March 2021 Quarterly Exchange Updates

[The_Exchange_Team]

Today we are announcing the availability of quarterly servicing cumulative updates (CUs) for Exchange Server 2016 and Exchange Server 2019. These CUs include fixes for customer reported issues as well as all previously released security updates. Although we mentioned in a previous announcement that this release would be the final cumulative update for Exchange 2016, we do expect to release one more CU for Exchange 2016 next quarter which will includes all fixes made for customer reported and accepted issues received before the end of mainstream support.

A full list of fixes is contained in the KB article for each CU, but we wanted to highlight that these latest CUs contain the fixes that were previously released as Exchange Server Security Updates on March 2, 2021. This means you don’t have to install the March 2021 Security Updates after installing the March 2021 CUs.

Release Details

The KB articles that describe the fixes in each release and product downloads are as follows:

• Exchange Server 2019 Cumulative Update 9 (KB4602570), VLSC Download, Download
• Exchange Server 2016 Cumulative Update 20 (KB4602569), Download, UM Lang Packs

Additional Information

Microsoft recommends all customers test the deployment of any update in their lab environment to determine the proper installation process for your production environment. These updates may contain schema and directory changes and so require you prepare Active Directory (AD) and all domains. You can find more information on that process here. Schema changes can be tracked here. For best practices for successful installation, please see this document.

Also, to prevent installation issues you should ensure that the Windows PowerShell Script Execution Policy is set to Unrestricted on the server being upgraded or installed. To verify the policy settings, run the Get-ExecutionPolicy cmdlet from PowerShell on the machine being upgraded. If the policies are NOT set to Unrestricted you should use these resolution steps to adjust the settings.

Additionally, a reminder that if you plan to install any Cumulative Update using the unattended option with either PowerShell or Command Prompt, make sure you specify either the full path to the setup.exe file or use a “.” in front of the command if you are running it directly from directory containing the update. If you do not the Exchange Server Setup program may indicate that it completed successfully when it did not. Read more here.

Reminder: Customers in hybrid deployments where Exchange is deployed on-premises and in the cloud, and those using Exchange Online Archiving with their on-premises Exchange deployment are required to deploy the currently supported CU for the product version in use.

For the latest information on the Exchange Server and product announcements please see What's New in Exchange Server and Exchange Server Release Notes.

Note: Documentation may not be fully available at the time this post is published.

The Exchange Server team

[Martin_Aigner]

Mostly welcome the site aware Autodiscover V2 now, thanks for getting this in place! Is this covering also EAS or only EWS while Teams calendaring has been mentioned as an example.

[jordanl17]

Is it safe just to download the ISO file, mount it and run the setup.exe? (As Administrator).  I always hear horror stories about the setup not finishing because it was launched the wrong way.  I just went from CU15 to CU19 by running the CU19 setup.exe. flawless. (Exchange 2016 and I had already installed .net 4.8) But wondering if that's not installation method is not recommended.

[Lukas Sassl]

@jordanl17 please check this article: 

https://docs.microsoft.com/en-us/exchange/plan-and-deploy/install-cumulative-updates?view=exchserver...

It describes the upgrade process altogether with our best practices.

[RalfLe]

@Martin_Aigner the change made to AutodiscoverV2 doesn't just apply to the EWS protocol.

[Martin_Aigner]

@RalfLeThanks for the clarification Ralf!

[Vincent VALENTIN]

Hello,

Recently I discovered that Exchange 2016 CU10 introduce a new argument (-Shared) within New-remotemailbox commandlet to create a shared mailbox
https://docs.microsoft.com/en-us/powershell/module/exchange/new-remotemailbox?view=exchange-ps
Is there somewhere a documment describing all changes introduced by all CU. I want to known if there is others change than the one for shared mailbox that can be usefull for Powersehll scripting.

Thanks in advance.

Regards

[Lukas Sassl]

@Vincent VALENTIN we have the "Updates for Exchange Server" Docs article. It contains links to the Exchange Team blog posts ("What's New" information) for each Exchange CU release.

Please have a look: https://docs.microsoft.com/en-us/exchange/new-features/updates?view=exchserver-2016 

[eklatant]

Do we need do a prepare schema / prepare domain before installing the CU for EX 2019 or 2016 ? 

Thank you ! 

[Lukas Sassl]

@eklatant it depends on the version you are upgrading from. PrepareAD and PrepareDomain are required even if you come from a December 2020 released version.

Schema information:

Exchange 2016: https://docs.microsoft.com/en-us/exchange/plan-and-deploy/active-directory/ad-schema-changes?view=ex...

Exchange 2019: https://docs.microsoft.com/en-us/exchange/plan-and-deploy/active-directory/ad-schema-changes?view=ex... 

[benoitrando]

Based on the documentation  Exchange 2019 CU9 is requiring a Schema Update 

But Exchange 2016 CU20, no schema update 

Why these differences ?

Could you confirm ?

https://docs.microsoft.com/en-us/Exchange/plan-and-deploy/active-directory/ad-schema-changes?view=ex...
https://docs.microsoft.com/en-us/Exchange/plan-and-deploy/active-directory/ad-schema-changes?view=ex...

Thanks

[Lukas Sassl]

Exchange 2019 CU9 doesn't require a Schema Update if you come from Exchange 2019 CU8 (which requires a Schema Update).

If you come from Exchange 2019 CU7 and you install Exchange 2019 CU9 (which is possible because the updates are cumulative) you must do a Schema update (because it contains the required Schema changes from Exchange 2019 CU8). The same goes for Exchange 2016 CU20. CU19 requires to update the Schema.

[McWolf82]

Today I have installed Exchange 2016 CU20 and from what I read the security update (KB5000871) is already part of CU20, so no need to install it afterwards as with CU19.

After installing CU20 and running the EOMT.ps1 script it showed that my server was vulnerable and automatically installed the IIS URL Rewrite Module amd configured it for the Default Web Site in IIS.

Is the EOMT script not compatible with CU20? Because with CU19 and KB5000871 my server wasn't vulnerable, but with CU20 it was?

I  found the same issue for someone else on a German website:

https://administrator.de/forum/schl%C3%A4gt-eomt-ps1-fehl-exchange-2016-cu20-663767.html

[Lukas Sassl]

@McWolf82 this is a know issue. A fix do detect the latest Exchange release has already been checked in. 

[Lukas Sassl]

@McWolf82 I’ve checked it again. Please try the latest release. It should already contain the fix: https://github.com/microsoft/CSS-Exchange/tree/main/Security 

[McWolf82]

@Lukas Sassl Correct, new release EOMT shows 'not vulnerable, mitigation not needed' with CU20 now. Thanks!

[TimJ808]

Can you confirm that these CUs contain the emergency security fixes included in KB5000871 or do we need to reinstall that patch after installation?

[Lukas Sassl]

@TimJ808 

A full list of fixes is contained in the KB article for each CU, but we wanted to highlight that these latest CUs contain the fixes that were previously released as Exchange Server Security Updates on March 2, 2021. This means you don’t have to install the March 2021 Security Updates after installing the March 2021 CUs.

[benoitrando]

For info , if you are updating from Exchange 2016 CU19 to CU20 , you will see the following change for the object

EXCHANGE         FOREST (RANGEUPPER) FOREST (OBJECTVERSION)           DOMAIN (OBJECTVERSION)

2016 CU19          15333                                    16219                                                  13239

2016 CU20          15333                                    16220                                                  13240

If you are not domain admin , your windows AD admin must execute the following command using the exchange binaries from a server where DOT NET 4.8 is installed

Prepare Ad Organization

Setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms

Prepare All Domains

Setup.exe /PrepareAllDomains /IAcceptExchangeServerLicenseTerms

[Martijn_Westera]

Hi,

I have upgraded an Exchange 2016 CU19 server (with March 2021 security updates) to CU20:

• Installation takes much longer (105 minutes) than expected. CU upgrade takes normally 30-40 minutes in our env
• Eventually: The Exchange Server setup operation completed successfully.
• Event Viewer Application log shows 800 warnings:
  • Source: ASP.NET 4.0.30319.0
  • Event ID: 1310
  • Level: Warning
  • Event code: 3008
  • Event message: A configuration error has occurred

Any ideas?

Martijn

[Lukas Sassl]

@Martijn_Westera sounds like a configuration file is missing. Does OWA / ECP work after the installation? If they don't, I'd recommend checking for missing SharedWebConfig.config files (see: Event ID 1309 and you can't access OWA and ECP after you install Exchange Server 2016 or Exchange Se...).

Please make sure to follow the Exchange CU best practices to avoid issues like long running installations or missing files: Upgrade Exchange to the latest Cumulative Update | Microsoft Docs 

[m4shafeeque]

Is there any change in admin role permission after upgrading to 2016 new CU (20). because few of my admins are not able to create new mailboxes after upgrading to new CU. they are part of 'Recipient Management' role group.

[Pepad]

The same issue with Recipient Management like m4shafeeque . Some changes had to be.
Quick workaround is add View-Only Organization management for them but it's not what we want.

[Lukas Sassl]

Thanks for reporting @m4shafeeque @Pepad. We'll have a look at it.

[Martijn_Westera]

Hi Lukas,

OWA & ECP work after installation.

I've upgraded a second server Exchange 2016 CU19 server (with March 2021 security updates) to CU20:

1. Check: all web.config files are present; \FrontEnd\HttpProxy\<dir>\web.config

2. Upgrade to CU 20 takes a very long time (105 minutes)

3. Event Viewer Application log shows 800 warnings during time of installation:

• Source: ASP.NET 4.0.30319.0
• Event ID: 1310
• Level: Warning
• Event code: 3008
• Event message: A configuration error has occurred
• Exception message: Could not load file or assembly 'Microsoft.Exchange.Clients.Strings, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified. (D:\Exchange\FrontEnd\HttpProxy\owa\web.config line 68)

4. Eventually: The Exchange Server setup operation completed successfully.

5. OWA & ECP work after installation

6. Check: all web.config files are present; \FrontEnd\HttpProxy\<dir>\web.config

[eklatant]

Hi Microsoft, 

do we need run the script  EOMT after we have installed at the same day the security patches on 03.03.2021 and than installed the monthly security updates on 10.03.2021 ? We are running 2019 CU8 ( now going to update to CU9 ) 

We found out some other staff like : https://us-cert.cisa.gov/ncas/alerts/aa21-062a

Can you confirm the risk from USA government ? 

Thanks !

[eklatant]

Hi Microsoft !

We are running 2019 CU 8 

what we have done:

On the  03.03.2021 installed KB5000871 installed , than the monthly march updates for windows server 2019. 

We ran the script Test-Hafnium.ps1 and nothing has been found.  

We found an article on US government  : Mitigate Microsoft Exchange Server Vulnerabilities | CISA

Should we take any action ? 

Should we run one of the script from: 

   https://msrc-blog.microsoft.com/2021/03/16/guidance-for-responders-investigating-and-remediating-on-... additionally ?  

Or it is now enough to install the newest CU 9 for 2019 ? 

Thank you ! 

[vigna840]

My current server running with Exchange 2016 CU18.

Shall I apply CU20  without applying CU19?

[Lukas Sassl]

@vigna840 yes, you can go directly to CU20. Please mind that this requires a schema update (the one which was included in CU19). So, you should take care that you have the required permissions in place to perform the schema update.

[damonyang]

I have m4shafeeque same problem

[AWC_OH]

Having the same issue with Recipient Management RBAC no longer working, leading to a 500 error when trying to add mailboxes.

That was a fun conversation with our user access team yesterday..

Confirmed in both our test and prod Exchange environment.

Was actually going to contact Microsoft support today until I found this.

I'll give @Pepad suggestion a shot today as a temp solution until Microsoft releases a fix.

[damir_]

@AWC_OH same issue occurs on my Exchange 2019 CU9.

500 error when trying add mailbox. Works fine from powershell...

[AWC_OH]

@damir_If only our user access team would learn to use powershell...

[Kuyalord]

Upon upgrading exchange server 2013 CU13 to CU23. Install and upgrade went well l. No error issue encounter upon upgrade. Unfortunately all outlook client unable connect to server asking for a password endless loop.. user can only access there email on owa. 

WinServer running windows 2012 in exchange 2013. AD running 2008. Outlook 2013 and higher. Config as ntlm. I Appreciate your help. 

[Martijn_Westera]

I've upgraded server 3 & 4 (Exchange 2016 CU19 server with March 2021 security updates) to CU20.

Installation time now took 30 minutes with +/- 100 entries in Event Viewer Application log (warning 1310; ASP.NET 4.0.30319.0) during installation time.

OWA & ECP work correctly after CU20 installation.

[Nicolas_Debrouwer]

Hello,

I have just finished to deploy the CU20 in our acceptance (2 DAGs, 10 mailbox servers, 8 as VMs, 2 as physical with a big config. In prod, full of physical with Bitlock enabled).

The installation took on average 8 hours (the slowest one took a bit more than 12 hours (!!!)) while usually in acceptance it takes around 3 hours.

When checking the exchange setup logs, I see that each time it blocks on Set-LocalPermissions for a long time. As other comments, I had a huge number of ASP.NET errors in the event viewer during the install but event viewer is clean after setup process.

AV Disabled, CRL Checks disabled and servers freshly rebooted.

Any idea on why the Set-LocalPermissions took so much time (each time around 45 mins).? Any link with all those asp.net errors? Maybe to stop IIS? Any other idea is more than welcome, I can't afford 12 hours of installation time in prod. Coming from CU19 with latest security fixes.

[VincentBurle]

Hi @Lukas Sassl 

Same problem as  @m4shafeeque and @Pepad here. Error 500 when a Service Desk member try to create a new mailbox from ECP. It seems that the problem is coming from the Get-Remotedomain cmdlet that required "View-Only Configuration" role. It was not the case on CU19 before the upgrade for sure.

Even if this role just give read-only permission, I'm a bit reluctant to give it to people with Help-Desk management role.

[Lukas Sassl]

Thanks @VincentBurle,

engineering is aware of this and it's under investigation. I'll keep you posted as soon as we have something new on this. Stay tuned...

[Lukas Sassl]

@VincentBurle @damir_ @AWC_OH 

We’ve publish a KB regarding the “500 Unexpected Error” issue. Please find it here:

https://support.microsoft.com/en-us/topic/-500-unexpected-error-when-trying-to-create-a-user-mailbox...

[damir_]

@Lukas Sassl it works, thanks.

Used Method 2

[VincentBurle]

Thanks @Lukas Sassl ,

I have already applied the method 1 but I will try method 2. It sounds better :)

Regards

-Vincent

[KrystianWieczorek]

@Lukas Sassl thanks! Method number 2 works and also solves another problem (related to creating remote mailboxes in Office 356 in Hybrid configuration). Now Helpdesk can create a mailbox in Exchange OnPrem and also in Exchange Online

[JJuergen]

I know, that patching Exchange is really hard work and most of the time it works as expected, but the last 3 CUs (18-20) for Exchange 2016 opened one bug by another, leading to the misery we saw in March. Admins had to install buggy CUs to get protected against Hafnium attacks. ExchangeAdmins are waiting for "clean" CUs... QualityControl should be alarmed, you are loosing customers, not everyone is able / allowed to use Microsoft365 solutions...

[clindell]

Hey Lukas Sassl Microsoft.....

Would you or anyone on the team that publishes info for the Mitigation tools please include if the ps1 scripts for exchange servers can be run in production yes/no? I am sure that there are some admin who cannot take a server down during production hours. If these scripts can be run on a production server while in use it sure would be helpful to know!

https://msrc-blog.microsoft.com/2021/03/16/guidance-for-responders-investigating-and-remediating-on-...

Immediate temporary mitigations

The following mitigation options can help protect your Exchange Server until the necessary Security Updates can be installed. These solutions should be considered temporary, but can help enhance safety while additional mitigation and investigation steps are being completed.

• Run EOMT.ps1 (Recommended) – The Exchange On-premises Mitigation Tool (EOMT.ps1) mitigates CVE-2021-26855 and attempts to discover and remediate malicious files. When run, it will first check if the system is vulnerable to CVE-2021-26855 and, if so, installs a mitigation for it. It then automatically downloads and runs Microsoft Safety Scanner (MSERT). This is the preferred approach when your Exchange Server has internet access.
• Run ExchangeMitigations.ps1 – The ExchangeMitigations.ps1 script applies mitigations but doesn’t perform additional scanning. This is an option for Exchange Servers without internet access or for customers who do not want Microsoft Safety Scanner to attempt removing malicious activity it finds.

[Lukas Sassl]

@clindell please find more information regarding EOMT.ps1 here: CSS-Exchange/Security at main · microsoft/CSS-Exchange (github.com)

[clindell]

@Lukas Sassl 

We have not observed any impact to Exchange Server functionality via these mitigation methods nor do these mitigation methods make any direct changes that disable features of Exchange.

Yes I understand, however, out of an abundance of caution it would be very helpful if the Exchange team explicitly stated which methods may be run against an Exchange production server. For your team that lives and breathes Exchange every day this is very comfortable to you, however, for those who are responsible for more than Exchange this is only one of our responsibilities and we need to collect as much info as possible to support any action on a production server that may have an undesired effect.

In the future would your team please publish and specifically state actions may be used on a "production server" safely?

[Sam_T]

@Nino Bilic Hello Nino. KB5001779, just like KB5000871 is not visible by running get-hotfix or by looking for updates in WAC when installed on a Windows 2019 Core server.   This is not a problem with other Windows Server security hotfixes - only with Exchange hotfixes.   This is an issue for administrators, installers, etc.  When will this be addressed?   Do you see the same thing on your Windows 2019 Core servers?

[Lukas Sassl]

@Sam_T Get-HotFix commandlet leverages the Win32_QuickFixEngineering WMI class to list Windows Updates, but only returns updates supplied by Component Based Servicing (CBS). Updates supplied by the Microsoft Windows Installer (MSI) or the Windows update site are not returned by Get-Hotfix/Win32_QuickFixEngineering.

You should check the build number (via Exchange Health Checker or by running  Get-Command Exsetup.exe | ForEach {$_.FileVersionInfo}) to validate if your system is vulnerable (SU not installed) or not (SU). 

[PS_Exchange_Adm]

I see that Exchange security patch KB5000871 is included with Exchange 2016 CU20.  I know CU20 was released prior to KB5001779 so it does not currently contain the security fixes included in that patch.  Any chance that CU20 will be rereleased to include KB5001779 as well, or will we need to reapply KB5001779 after upgrading to CU20?

[JoeUSA44]

#exchange2013

Is the CU23 available for Exchange 2013? I ran the current EMOT and checked log with no problem but failed on following the URL indicated on the log. Thanks for any help. 

[Lukas Sassl]

@JoeUSA44 I'll have a look at it.