Released: July 2021 Exchange Server Security Updates

[The_Exchange_Team]

Microsoft has released security updates for vulnerabilities found in:

• Exchange Server 2013
• Exchange Server 2016
• Exchange Server 2019

All versions (Cumulative Update levels) are impacted. Updates are available for the following specific builds of Exchange Server:

IMPORTANT: If manually installing security updates, you must install .msp from elevated command prompt (see Known Issues in update KB article).

• Exchange Server 2013 CU23
• Exchange Server 2016 CU20 and CU21
• Exchange Server 2019 CU9 and CU10

The July 2021 security updates for Exchange Server address vulnerabilities responsibly reported by security partners and found through Microsoft’s internal processes. Although we are not aware of any active exploits in the wild, our recommendation is to install these updates immediately to protect your environment.

These vulnerabilities affect on-premises Microsoft Exchange Server, including servers used by customers in Exchange Hybrid mode. Exchange Online customers are already protected and do not need to take any action.

More details about specific CVEs can be found in Security Update Guide (filter on Exchange Server under Product Family).

Latest /PrepareSchema needed for full effect

Because of additional security hardening work for CVE-2021-34470, the following actions should be taken in addition to application of July 2021 security updates:

The latest version of Exchange installed	Additional steps needed to extend AD schema
Exchange 2016 CU21 or Exchange 2019 CU10	Nothing; schema was extended during installation of June 2021 CUs.
Exchange 2016 CU20 or Exchange 2019 CU9	Extend the schema using June 2021 CUs.
Exchange 2013 CU23	- Install July 2021 Security Update for Exchange 2013 - Extend the Active Directory schema using the elevated Command prompt. Command will be similar to the following: “Setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms” using the setup.exe from location “c:\Program Files\Microsoft\Exchange Server\V15\Bin\setup.exe” (use the folder for the installation location of your Exchange server) NOTES: - For Exchange 2013 only, schema version will not change after this. - In case of Schema Master existing in an empty root domain, consider installing Exchange CU23 Management Tools on Windows 2012 R2 in the same domain, installing July SU and then running \prepareschema from that workstation.
Older versions of Exchange (earlier than 2013) Or Exchange no longer installed in the forest	How to update AD schema to address CVE-2021-34470 if Exchange is very old or no longer installed

Known issues in July 2021 security updates

During the release of April 2021 SUs, we received some reports of issues after installation. The following issues reported for April 2021 SUs also apply to July SUs and have the following workarounds:

• Administrator/Service accounts ending in ‘$’ cannot use the Exchange Management Shell or access ECP. The only workaround at this time is to rename Admin accounts or use accounts with no ‘$’ at the end of the name.
• Some cross-forest Free/Busy relationships based on Availability address space can stop working (depending on how authentication was configured) with the error: “The remote server returned an error: (400) Bad Request.” Please see this KB article for how to solve this problem.
• Cmdlets executed against the Exchange Management Console using an invoked runspace might fail with the following error message: The syntax is not supported by this runspace. This can occur if the runspace is in no-language mode. Please see this KB article for more information.
• Installing June 2021 Cumulative Updates for Exchange 2016 or 2019 might fail with the error: 

  System.NullReferenceException: Object reference not set to an instance of an object. Please see this KB article for resolution.

• Starting with July 2021 updates, users might be redirected back to the login page when using OWA/ECP if organization uses Load Balancing. You should avoid running mixed pools (servers with the latest SU applied together with servers which have not yet received the update). Please see this KB article for more information.
• Prior to installing the Security Update (SU), we recommend you check if a valid Microsoft Exchange Server Auth Certificate is present on every Exchange server (except Edge Transport servers). The easiest way to do this is to run the Exchange Health Checker and check for the Auth Certificate output:

You can also run the following PowerShell command to check if the Auth Certificate is available on your system:

Get-ExchangeCertificate (Get-AuthConfig).CurrentCertificateThumbprint

If there is no Auth Certificate or it has expired, then follow the steps outlined here to configure it correctly.

Please note: In some environments, it may take an hour for the OAuth certificate to be published. If you have a hybrid setup, you have to run the Hybrid Configuration Wizard again to update the changes to Azure Active Directory (Azure AD). If this certificate is missing or is expired, users may face issues logging in to OWA/ECP with HTTP 500 error after application of July updates. KB article is here.

Update installation

Because of the recommended schema update requiring the latest set of June 2021 CUs, there are several scenarios that you might need to follow:

Inventory your Exchange Servers / determine which updates are needed

Use the Exchange Server Health Checker script (use the latest release) to inventory your servers. Running this script will tell you if any of your Exchange Servers are behind on updates (CUs and SUs).

Update to the latest Cumulative Update

Go to https://aka.ms/ExchangeUpdateWizard and choose your currently running CU and your target CU. Then click the “Tell me the steps” button, to get directions for your environment.

If you encounter errors during or after installation of Exchange Server updates

If you encounter errors during installation, see the SetupAssist script. If something does not work properly after updates, see Repair failed installations of Exchange Cumulative and Security updates.

FAQs

My organization is in Hybrid mode with Exchange Online. Do I need to do anything?
While Exchange Online customers are already protected, the July 2021 security updates do need to be applied to your on-premises Exchange Servers, even if they are used only for management purposes. You do not need to re-run the Hybrid Configuration Wizard (HCW) after applying updates.

Do I need to install the updates on ‘Exchange Management Tools only’ workstations?
Servers or workstations running only Microsoft Exchange Management Tools (no Exchange services) do not need to apply these updates.

Instructions seem to indicate that for Exchange 2013, we should extend the schema after July 2021 SU is installed; is that correct?
Yes. Because we did not release an Exchange 2013 Cumulative Update (CU) that contains the new schema updates, the July 2021 SU package updates the schema files in Exchange server folders when July 2021 SU is installed. That is why once those files are updated (SU is installed) – we ask you to go and explicitly update the schema using setup from \v15\Bin folder.

We have Exchange 2016 CU20 and 2019 CU9 servers and have installed July 2021 security updates but did not run /PrepareSchema using June 2021 CUs first. Is this a problem?
No. Extension of AD schema using June 2021 CU is really a separate step that should be taken to address a specific CVE. There is no dependency in July 2021 SUs on this schema change, or vice versa. Just make sure that both of those actions are taken; order is not important.

Updates to this post:

• 8/5: Added a link to How to update AD schema to address CVE-2021-34470 if Exchange is very old or no longer installed
• 7/20: Merged "Installation tips" section into "Known issues" section and provided additional detail and links
• 7/19: Added a note about updating servers in a Load Balancing (LB) pool
• 7/15: Added a clarification that all CU levels of Exchange are impacted; we only release security updates for latest CUs only. Please see this for more information on update cadence.
• 7/15: Added a note about how to extend schema in a root domain with no Exchange servers.
• 7/15: Added a note that schema version does not change after schema extension if Exchange 2013 Server is the latest version in the org.
• 7/15: Added the installation tips section and moved the info about OWA/ECP errors there.
• 7/14: Added a note about what to do if OWA/ECP with HTTP 500 error is seen after application of SUs.
• 7/13: Clarified the graphics to illustrate that Exchange Server 2016 CU20 and Exchange Server 2019 CU9 with July SUs are not 'fully' updated (because we released June CUs for both versions).

The Exchange Team

[NormC20]

Does this update address the AMSI issue that affects Outlook client connectivity?

[Nino_Bilic]

@NormC20 Ummm... something that you can link to on this? I am not aware of this issue. That being said - this is a security update, not a cumulative update...

[Tom B]

@Nino_Bilic what NormC20 is referring to, is the issue with AMSI not compatible with several Antivirus solutions implemented. I gave @ScottSchnoll already a heads up that there is more collaboration with antivirus solutions needed. Clients getting massive Outlook client performance decrease when this is not mitigated.

[NormC20]

Hi Nino,

I have a case open with support, I can email you the case number privately if you'd like.  We've experienced extreme Outlook slowness/poor connectivity since installing Exchange 2016 CU21.  Disabling AMSI has resolved the symptoms, but I'd love to see the problem resolved rather than disable an important feature.

I use Sophos, but I've seen mentions on forums of users with other AV products having the same issue.

First mention I found, sorry it's in German but easy to translate: Exchange 2016/2019: AMSI integration causes problems with Outlook - Frankys Web

Note that remarking out the AMSI lines in the web.config did not solve the problem for me, only disabling AMSI in the Sophos product.

There are a couple of mentions in this reddit thread: June Updates for Exchange 2016 and 2019 Released : exchangeserver (reddit.com)

[Nino_Bilic]

@NormC20 Got it, thanks; I'll check with Scott on this.

Please don't cross the streams... 🙂 Issues related to a previous CU and 3rd party interop are not related to July SU releases. I fear people seeing mention of issues and assuming "issues with July SUs". It has happened before.

[NormC20]

Understood... Apologies for posting on the wrong thread.  

For the record I appreciate these security updates and intend on installing this evening.  Too many threats out there these days to go without.

[culmor]

If the schema master is in a different site - hybrid scenario; what is the process to update the schema?  Is there a separate update? 

[JMaletzky]

@The_Exchange_Team 

From 

https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-34473

"

Jul 13, 2021

Information published. This CVE was addressed by updates that were released in April 2021, but the CVE was inadvertently omitted from the April 2021 Security Updates. This is an informational change only. Customers who have already installed the April 2021 update do not need to take any further action."

Do you have you any further explanation on this?

Thank you

Jörg Maletzky

[Günter Born]

A summary of the German blog post from Frank Zöchling linked above may be found in in English at:

https://borncity.com/win/2021/07/13/exchange-2016-2019-outlook-probleme-durch-amsi-integration/

[Sreejith]

Needed some clarification here is it mandatory to install the Exchange 2019 CU10 to address July Security Update and https://support.microsoft.com/en-us/topic/description-of-the-security-update-for-microsoft-exchange-... in this article it says "Important: Please install the May 2021 security update. That update supersedes this security fix. For more information, see the following Exchange Team Blog article: https://techcommunity.microsoft.com/t5/exchange-team-blog/released-may-2021-exchange-server-security... . In this case do we still need to install (KB5001779)....? Even if we have installed (KB5003435) which was released in May 2021...?

[LukasSMSFT]

@Sreejith just install Exchange 2019 CU10 + the July 2021 Security Update (SU) and you're done. 

You can also use Exchange 2019 CU9 + July 2021 Security Update (SU) + /PrepareSchema (using June 2021 CU).

Then run the latest version of the Exchange Health Checker script (https://aka.ms/ExchangeHealthChecker) after the installation process for validation.

[Sreejith]

@LukasSMSFT Can you please confirm what is July 2021 SU...? https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-34470 If this is what you are referring to then it is directing to install Exchange 2019 CU10 https://support.microsoft.com/en-us/topic/cumulative-update-10-for-exchange-server-2019-kb5003612-b1... 

Please correct me if am wrong.

Thank you for the help.

[LukasSMSFT]

@Sreejith here are the information about the Exchange 2019 July 2021 SU: 

Description of the security update for Microsoft Exchange Server 2019: July 13, 2021 (KB5004780)

You can also use the table on the Exchange Server build number and release dates. We document the Security Updates there as well: Exchange Server build numbers and release dates | Microsoft Docs

[niels haaijer]

When installing the security patch on Exchange2016 cu20 on a Edge server it gives me a long list of applications that are having files in use. (I started the installation running cmd as admin.)

The patch installed correctly on our mailbox server.
Does anyone else having issues installing the patch on a edge server? 

[LukasSMSFT]

@niels haaijer It's always a good idea to reboot the server prior installing CU/SU. 

[BitwinTheSheets]

@The_Exchange_Team  It is a bit unclear to me what is the path to be taken in a mixed 2013 CU23 / 2016 CU2X environment concerning the AD schema update. My understanding is that we should patch 2013 and the top 2 options of the image above. Is this correct?

[Test-RRR]

@niels haaijer : I have had this (about 15 Processes in use) on all of our Mailboxservers (we dont have an Edge server) on every Exchange 2016 Security Update in the last years (Despite booting the server before updating). I usually look for the processids, kill the processes and click "ignore".Thats very annoying. This Problem is only with the security updates the CUs dont have this problem.

[87612378162351623765]

Can you clarify if the patches for 2016 CU20 and CU21 are identical or not? They have the exact same KB number but the files seem to be different.

Can you also clarify if we have exchange 2016 CU20, then install this security patch (CU20 version). 
Then after a while we update to CU21. Will we then have to install this security patch again (CU21 version)

[LukasSMSFT]

@87612378162351623765 

Can you clarify if the patches for 2016 CU20 and CU21 are identical or not? They have the exact same KB number but the files seem to be different.

No, there are different update packages for CU20 and CU21. You can check the file hash information section at the bottom of the KB: Description of the security update for Microsoft Exchange Server 2016: July 13, 2021 (KB5004779)

Exchange 2016 + CU20 + July 2021 SU + /PrepareSchema  (using June 2021 CU)  --> Vulnerabilities addressed

Exchange 2016 + CU21 + July 2021 SU --> Vulnerabilities addressed

Can you also clarify if we have exchange 2016 CU20, then install this security patch (CU20 version). 
Then after a while we update to CU21. Will we then have to install this security patch again (CU21 version)

Yes, you must install the latest July 2021 SU on top of CU21 (even if you've installed it for CU20) to address all vulnerabilities.

[sasger]

After installing the update on several Exchange 2016 CU 21 we see a problem with OWA/ECP.

Immediately after login it says: Session expired. please log in.

So no access is possible.

If I  deactivate the patched servers in our load balancer (just for owa/ecp is enough) everythink works fine.

What can I do?

[mstoffa]

This morning I patched one of our two Exchange Servers and ran into some issues that appear to have occurred after installing the Exchange 2016 CU21 July Security Patches KB5004779.  Both of my Exchange servers was running Exchange 2016 CU 19 with All Security patches.  I installed Exchange 2016 CU21 and this appears to have installed without any problems.  I rebooted the server and let Exchange Start.  I waited for services to start, verify Windows Event Logs to see that nothing appeared out of the ordinary.  I then logged into ECP and verified that the server was online and it was at version 15.1 build 2308.8 and my other server is 15.1 build 2176.2. 

I didn't notice any issues so I wanted to apply the security update for Exchange 2016 along with the other patches that was released yesterday for July 2021 patch Tuesday (Windows 2016 OS patch, .Net Patch, Etc).  We have WSUS running in our setup so I have always let these updates pull from WSUS and install and have never ran into any issues described in previous posts with services not starting or failed installation.  The installation of the updates completed and I rebooted the server.  

Once the server rebooted I logged into it and verified that services started correctly and looked at the Windows system logs to verify that Exchange started as normal and no unusual errors.  Again I didn't see anything and then tried to login to ECP.  I put in my username and password and it just went back to a login screen.  This happened several times and I decided to clear the browser (using Chrome version 91.0.4472.124 which is current).  I was able to get into ECP and then looked at the server.  The databases was mounted and everything looked correct.  I then tried to login to Webmail (OWA, Outlook on the web) and ran into the same problem I had with logging into ECP.  I would type in my username and password and it would just go back to a login screen or appeared to do nothing.  I tested this from multiple devices that was on my network and from a device that was remote and had the same issue.  My Outlook client appears to work both Internal and External.  

My Exchange Servers are configured behind a BigIP Load Balancer and Exchange was setup F5s iApps for Exchange 2016.  This has been in place since we migrated to Exchange 2016 and the only thing I have ever had to add was an iRule for SameSiteCookies to get Chrome to work after one of the CU 2016 for Exchange.  Other browsers didn't have this issue. 

I verified all of the Virtual Directory permissions and configuration between the two Exchange Servers and they are the same.  My next thought was to remove the Exchange 2016 CU21 Security Update and see if the problem still exists because I didn't notice this issue after I had just applied Exchange 2016 CU21. 

I removed the July 2021 Security update for Exchange 2016 and rebooted the server.  I verified that the Exchange Services started and there was nothing unusual in the event logs.  I then tested logging into ECP and didn't have any problems.  I then tested logging into OWA with multiple accounts, multiple browsers, and multiple devices.  I didn't have any issues with this either.

I currently went into our WSUS server that supplies the updates to our services and declined KB5004779 for Exchange 2016 CU21 and CU20 until I can find out what the issue is and we do more testing with just going from Exchange 2016 CU19 to Exchange 2016 CU21.  I would like to see if anyone else has seen issues like this after applying the July 2021 Security Patch for Exchange 2016 CU21. 

[87612378162351623765]

For Exchange 2013,

If we want to do the schema update from a different server, an AD controller, how can we do this? (Schema master in another site)

The instructions:

- Extend the Active Directory schema using the elevated Command prompt. Command will be similar to the following:

“Setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms” using the setup.exe from location “c:\Program Files\Microsoft\Exchange Server\V15\Bin\setup.exe” (use the folder for the installation location of your Exchange server)

[justsomeadmin]

I can confirm OWA/ECP Login Issues

Patched 1 Server 2019 CU10 with Update and immediately started having login issues. 3 Node Cluster behind kemp loadbalancers.

Disabled the server from loadbalancer service for now.

Another cluster (same 3node setup behind kemps) does not have any issues.

Not sure why so far

[Shmeker]

@BitwinTheSheets - my understanding is the same - if 2016 is on CU21 you have to just patch 2013 & 2016. If 2016 is on CU20 then patch 2013 servers -> update schema with CU21 binaries -> patch 2016 servers.

[Marcel111]

Hello,
I can confirm the login issues. After installing the Security Update For Exchange Server 2016 CU21 (KB5004779) on an Exchange Server in the DAG, I get the following error:
After successful login, I am immediately thrown back to the OWA login page.
If I deactivate the server on the loadmaster, OWA works as usual!

Greetings
Marcel

[Nino_Bilic]

@culmor and @87612378162351623765 I assume this is the scenario where Exchange 2013 is the highest version of Exchange server and schema master is in a different site? A preferred way forward is to move the schema master to the same site, run the schema update and then move it back; it just just a bit of AD replication; the impact should be minimal. There is a different way to handle this also and I almost hesitate to suggest it but will mention it: you could use the June CU for Exchange 2016 / 2019 installation media to do the schema extension. Note that this will do two things: you will have to pass all E2016/2019 setup prerequisites (so let's say if there is an Exchange 2007 server in the org, this will not work) - and, more importantly - from that point on, you will always have to extend the schema using the version that you use for this. This might be an option if you are looking to move to the later version of Exchange anyway. Seeing that we are talking Exchange 2013 here, this might be OK.

EDIT 7/15: In case of "empty root domain" scenario, the following should work:

1) Introduce a VM/Server with Windows 2012 R2 (or any OS that is supported by Exchange Server 2013 Management tools) in same AD site/domain as that of root domain.
2) Install only Exchange 2013 management tools on this machine, using CU23 media
3) Install July 2021 security update
4) Perform PrepareSchema from this machine from \bin\setup.exe

[Nino_Bilic]

@JMaletzky  - the tangled web of CVEs for this release is unfortunate; here is the bottom line:

There are 7 Exchange CVEs in the July Release.

• Three of the CVEs were fixed with the July updates (CVE-2021-31196, CVE-2021-31206, CVE-2021-33768). These CVEs have July Packages.
• Three of the CVEs (CVE-2021-34523, CVE-2021-34473, CVE-2021-33766) were fixed in April but CVEs were not released until July. These CVEs have April packages (fixes were released then and people who installed them were protected). The 1.0 revision note on each of the CVEs explains this.
• CVE-2021-34470 was fixed in the June 29 Cumulative Update release for Exchange Server 2019 and 2016. Exchange Server 2013 was fixed in the July Update and has the July Package.

Now, while CVEs are a bit tangled up because of April omission - the update path is simple: July updates + schema update (as appropriate for the version) = done.

[Neill]

Exchange 2013 Schema Version

My understanding is that the rangeUpper value for the ms-Exch-Schema-Version-Pt attribute should increment from 15312 to 15313.

https://eightwone.com/references/schema-versions/ 

I've installed the security update in my lab and run the schema update but the rangeUpper value is staying at 15312.

The release notes file versions entries say that the Schemaversion.ldf file should be dated 7/7/21 with a size of 1,905 bytes.

https://support.microsoft.com/en-us/topic/description-of-the-security-update-for-microsoft-exchange-... 

On both servers I have run the update on the Schemaversion.ldf file is staying at 05/29/19 with a size of 1,905 bytes.

Similarly Schemaadam.ldf is not getting updated. 

The following LDF's ARE getting updated although they are dated 07/08/21 rather than the 07/07/21 listed in the notes, that might just be a time zone thing though.

PostExchange2003_schema99.ldf
PostWindows2003_schema99.ldf

PostExchange2000_schema99.ldf

schema99.ldf

Anyone else seeing this?

Obviously it makes it awkward for auditing and change control purposes if the schema version is not being updated.

Thanks

Neill

[Nino_Bilic]

@BitwinTheSheets In the mixed 2013 CU23 / 2016 CU2X environment and schema updates: all that you have to do is update the schema using 2016 CU21. You do not need to worry about Exchange 2013 schema scenario in that case; June CUs already have the schema change. Then, also - of course - install July SUs and that's that.

[LukasSMSFT]

@Marcel111 can you please run the Exchange Health Checker (https://aka.ms/ExchangeHealthChecker) against the server where you see the login issue? 

What's the output regarding the Auth Certificate?

[Nino_Bilic]

@justsomeadmin @mstoffa @sasger Please check the workaround here: You can't access OWA or ECP after you install Exchange Server 2016 CU6 (microsoft.com) 

[Nino_Bilic]

@Neill Please run Exchange Health Checker (https://aka.ms/ExchangeHealthChecker) - it will check for the schema change.

[Neill]

@Nino_Bilic I don't see the point in that. I can see that the schema version has not updated within AD via adsiedit or script and that the SchemaVersion.ldf file that would actually update the rangeUpper value has not been updated by the SU.

I can see from the Exchange setup log that the changes have been applied, or at least that the LDF's have been run against AD and closed cleanly but if I'm handing this to AD guys to run for me then a normal part of the change is to validate the rangeUpper version after running in the schema update.

I'm not saying this is a general problem, just that it is what I am seeing.

[GeraldSchwab]

I'm running Exchange Server 2013 CU23 and whatever update went out last night is causing my OWA and ECP to 500 error

exception: 

ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1 
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1

Source Error: 

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.  

Stack Trace: 


[ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1]
   Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters) +241
   Microsoft.Exchange.Clients.Common.HmacProvider.GetCertificates() +478
   Microsoft.Exchange.Clients.Common.HmacProvider.GetHmacProvider() +143
   Microsoft.Exchange.Clients.Common.HmacProvider.ComputeHmac(Byte[][] messageArrays) +16
   Microsoft.Exchange.HttpProxy.FbaModule.SetCadataCookies(HttpApplication httpApplication) +826
   Microsoft.Exchange.HttpProxy.FbaFormPostProxyRequestHandler.HandleFbaFormPost(BackEndServer backEndServer) +2778
   Microsoft.Exchange.HttpProxy.FbaFormPostProxyRequestHandler.ShouldContinueProxy() +20
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.BeginProxyRequestOrRecalculate() +229
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalOnCalculateTargetBackEndCompleted(TargetCalculationCallbackBeacon beacon) +1379
   Microsoft.Exchange.HttpProxy.<>c__DisplayClass3f.<OnCalculateTargetBackEndCompleted>b__3e() +311
   Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(TryDelegate tryDelegate, FilterDelegate filterDelegate, CatchDelegate catchDelegate) +35
   Microsoft.Exchange.HttpProxy.Diagnostics.SendWatsonReportOnUnhandledException(MethodDelegate methodDelegate, LastChanceExceptionHandler exceptionHandler) +121
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CallThreadEntranceMethod(MethodDelegate method) +69

[AggregateException: One or more errors occurred.]
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.EndProcessRequest(IAsyncResult result) +416
   System.Web.CallHandlerExecutionStep.InvokeEndHandler(IAsyncResult ar) +231
   System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +172
 


--------------------------------------------------------------------------------
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.8.4330.0 

[GeraldSchwab]

I posted this before and it was deleted ?!!??!?! WHY?

I  am running Exchange Server 2013 CU23 and I am getting the following exception after last nights updates:

ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1 
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1

Source Error: 

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.  

Stack Trace: 


[ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1]
   Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters) +241
   Microsoft.Exchange.Clients.Common.HmacProvider.GetCertificates() +478
   Microsoft.Exchange.Clients.Common.HmacProvider.GetHmacProvider() +143
   Microsoft.Exchange.Clients.Common.HmacProvider.ComputeHmac(Byte[][] messageArrays) +16
   Microsoft.Exchange.HttpProxy.FbaModule.SetCadataCookies(HttpApplication httpApplication) +826
   Microsoft.Exchange.HttpProxy.FbaFormPostProxyRequestHandler.HandleFbaFormPost(BackEndServer backEndServer) +2778
   Microsoft.Exchange.HttpProxy.FbaFormPostProxyRequestHandler.ShouldContinueProxy() +20
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.BeginProxyRequestOrRecalculate() +229
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalOnCalculateTargetBackEndCompleted(TargetCalculationCallbackBeacon beacon) +1379
   Microsoft.Exchange.HttpProxy.<>c__DisplayClass3f.<OnCalculateTargetBackEndCompleted>b__3e() +311
   Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(TryDelegate tryDelegate, FilterDelegate filterDelegate, CatchDelegate catchDelegate) +35
   Microsoft.Exchange.HttpProxy.Diagnostics.SendWatsonReportOnUnhandledException(MethodDelegate methodDelegate, LastChanceExceptionHandler exceptionHandler) +121
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CallThreadEntranceMethod(MethodDelegate method) +69

[AggregateException: One or more errors occurred.]
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.EndProcessRequest(IAsyncResult result) +416
   System.Web.CallHandlerExecutionStep.InvokeEndHandler(IAsyncResult ar) +231
   System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +172
 


--------------------------------------------------------------------------------
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.8.4330.0 

[justsomeadmin]

auth cert is fine for my environment.

no errors in event log.

Direct login through the server itself works, but not anymore when its accessed through loadmaster.

[georgedaly]

We observed the issue with users getting kicked out of OWA after installing this update.  The fix was to change the persistence profile to source-ip on our load-balancer.  It would be good to understand the change in behaviour introduced in this update which has broken session persistence.

[justsomeadmin]

@georgedaly 

Bravo sir, looks like thats it. Thank you very much!

Changed persistence for owa subdir to source-ip and problem is gone. Id like to avoid source ip persistence though. 

obviously there was some change with session persistence in this update.

The second environment without this issue uses simple l4 loadbalancers. The environment with the issue uses l7 profile.

[pputros]

@Neill Exact same scenario as you. We're also not seeing the value get updated.

[georgedaly]

@mstoffa @sasger in case you missed it, try switching persistence profile to source-ip on the load-balancer to work around session persistence issues introduced in this update.  Root cause as yet unclear.

[EricStarker]

@GeraldSchwab We have an automated approval queue (spam filter) that your first message was filtered into - you would have received a notification when you tried to post the first time, but sorry if that was unclear. We review this queue manually daily and as your message was not spam, your message was moved back into publication. Sorry for any hassle. 

[Joshua Davis]

Exchange 2013 CU23. We are experiencing the OWA/ECP login issue as well post KB5004778 install. Our F5 LB persistence profile is not configured.

[georgedaly]

@Joshua Davis We didn't have a persistence profile configured either, as previously session persistence was taken care of by Exchange without the need for enabling sticky sessions on the LB.  We applied the source-ip persistence profile on our F5 LB as a workaround once we suspected the problem related to session persistence.

[a7n8x]

Helo, We have problem with standalone Exchange 2013 on 2012R2 - geetting error:

HMACProvider.GetCertificates:protectionCertificates.Length<1 

Even after applying OAuth "fix"

Is there some way to get more information for troubleshooting? 

Thanks 

[Mildur]

Exchange 2016 CU21 same error, no load balancer:

HMACProvider.GetCertificates:protectionCertificates.Length<1 

[PerplexMM]

Exchange 2019 CU9 also same error:

Exception type: ExAssertException
Exception message: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1

Not possible to login OWA and ECP. After filling in username, password and push submit they run in the error

Tried different things as suggested above (OAuth "fix" as example) and from other websites.

Currently trying to uninstall the update.

Update:  uninstalling the patch solves the problem. OWA and ECP works again.  

[LukasSMSFT]

@Mildur please follow the steps described here: https://docs.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oa...

Please run the Health Checker script (https://aka.ms/ExchangeHealthChecker) after doing these steps and check if there are any errors reported regarding the Auth certificate.

[Nino_Bilic]

@PerplexMM @Mildur @a7n8x - do we know that OAuth cert is not expired? Can't access OWA/EAC with expired OAuth certificate - Exchange | Microsoft Docs 

[Mildur]

@LukasSMSFT @Nino_Bilic 

Thanks guys.

My Oauth Cert was invalid. I have many entries in the event logs about that. I have already created a new Oauth Cert and now I am waiting until it‘s active.

I will give use health check soon and give a feedback here 🙂

[PerplexMM]

@Nino_Bilic Yes I know it wasn't expired.  
I also deployed a new OAuth certificate. As mentioned above.

And if it was expired the problem should exist before the update and wouldn't be fixed by uninstalling the update; the update does not change the certificates.

The one strange thing I had after installing the update that the script Get-ExchangeCertificate (Get-AuthConfig).CurrentCertificateThumbprint gives an error result with a thumbprint of a certificate which did not exist (and never has) on the server.  This thumbprint problem was solved after deploying a new OAuth certificate. But the ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1 was still there.