Released: March 2021 Exchange Server Security Updates
Published Mar 02 2021 01:08 PM 1.1M Views

Note: this post is getting frequent updates; please keep checking back. Last update: 3/19/2021

Microsoft has released a set of out of band security updates for vulnerabilities for the following versions of Exchange Server:

  • Exchange Server 2013
  • Exchange Server 2016
  • Exchange Server 2019

Security updates are available for the following specific versions of Exchange:

IMPORTANT: If manually installing security updates, you must install .msp from elevated command prompt (see Known Issues in update KB articles)

Because we are aware of active exploits of related vulnerabilities in the wild (limited targeted attacks), our recommendation is to install these updates immediately to protect against these attacks.

The vulnerabilities affect Microsoft Exchange Server. Exchange Online is not affected.

For more information, please see the Microsoft Security Response Center (MSRC) blog.

For technical details of these exploits and how to help with detection, please see HAFNIUM Targeting Exchange Servers. There is a scripted version of this check available on GitHub here.

 

Mitigations, investigation and remediation:

Are there any mitigations I can implement right now?

MSRC team has released a One-Click Microsoft Exchange On-Premises Mitigation Tool (EOMT). The MSTIC blog post called Microsoft Exchange Server Vulnerabilities Mitigations – March 2021 can help understand individual mitigation actions. A stand-alone ExchangeMitigations.ps1 script is also available.

How can I tell if my servers have already been compromised?

Information on Indicators of Compromise (IOCs) – such as what to search for, and how to find evidence of successful exploitation (if it happened), can be found in HAFNIUM Targeting Exchange Servers. There is a scripted version of this available on GitHub here.

More information about investigations

To aid defenders in investigating these attacks where Microsoft security products and tooling may not be deployed, we are releasing a feed of observed indicators of compromise (IOCs). The feed of malware hashes and known malicious file paths observed in related attacks is available in both JSON and CSV formats at the below GitHub links. This information is being shared as TLP:WHITE. CSV format and JSON format are available. 

What about remediation?

MSTIC team has (on March 6th) updated their blog post Microsoft Exchange Server Vulnerabilities Mitigations – March 2021 to include information about Microsoft Support Emergency Response Tool (MSERT) having been updated to scan Microsoft Exchange Server. Please download a new copy of MSERT often, as updates are made in the tool regularly! Please also see MSRC Guidance for responders: Investigating and remediating on-premises Exchange Server vulnerabilities.

 

Installing and troubleshooting updates:

Does installing the March Security Updates require my servers to be up to date?

Today we shipped Security Update (SU) fixes. These fixes can be installed only on servers that are running the specific versions listed previously, which are considered up to date. If your servers are running older Exchange Server cumulative or rollup update, we recommend to install a currently supported RU/CU before you install the security updates. If you are unable to get updated quickly, please see March 2021 Exchange Server Security Updates for older Cumulative Updates of Exchange Server.

How can I get an inventory of the update-level status of my on-premises Exchange servers?

You can use the Exchange Server Health Checker script, which can be downloaded from GitHub (use the latest release). Running this script will tell you if you are behind on your on-premises Exchange Server updates (note that the script does not support Exchange Server 2010).

Which of my servers should I update first?

Exploitation of the security vulnerabilities addressed in these fixes requires HTTPS access over the Internet. Therefore, our recommendation is to install the security updates first on Exchange servers exposed/published to the Internet (e.g., servers publishing Outlook on the web/OWA and ECP) and then update the rest of your environment.

Will the installation of the Security Updates take as long as installing an RU/CU?

Installation of Security Updates does not take as long as installing a CU or RU, but you will need to plan for some downtime.

My organization needs to 'get current' first... we need to apply a Cumulative Update. Any tips for us?

Please see the Upgrade Exchange to the latest Cumulative Update article for best practices when installing Exchange Cumulative Updates. To ensure the easiest upgrade experience (and because in many organizations Exchange and Active Directory roles are separate) you might wish to run /PrepareAD (in the Active Directory site that Exchange is a member of) before running the actual CU Setup. You can use this document as a guide to understand what you might have to do.

Errors during or after Security Update installation! Help!

It is extremely important to read the Known Issues section in the Security Update KB article (here and here depending on the version). If installing the update manually, you must run the update from the elevated command prompt. If you are seeing unexpected behavior, check the article addressing troubleshooting failed installations of Exchange security updates (we will keep updating this article).

 

Additional Q&A:

Are there any other resources that you can recommend?

Microsoft Defender Security Research Team has published a related blog post called Defending Exchange servers under attack which can help you understand some general practices around detection of malicious activity on your Exchange servers and help improve your security posture.

My organization is in Hybrid with Exchange Online. Do I need to do anything?

While those security updates do not apply to Exchange Online / Office 365, you need to apply those Security Updates to your on-premises Exchange Server, even if it is used for management purposes only. You do not need to re-run HCW if you are using it.

Do we need to install those updates on Management Tools only workstations or servers?

Machines with Management Tools only are not impacted (there are no Exchange services installed) and do not require installation of March SUs. Please note that a 'management server' which many of our Hybrid customers have (which is an Exchange server kept on premises to be able to run Exchange management tasks) is different. For Hybrid, please see the Hybrid question above.

The last Exchange 2016 and Exchange 2019 CU’s were released in December of 2020. Are new CU’s releasing in March 2021?

EDIT: Exchange Server 2016 CU 20 and Exchange Server 2019 CU 9 are now released and those CUs contain the Security Updates mentioned here (along with other fixes). Customers who have installed SUs for older E2016/2019 CUs can simply update to new CUs and will stay protected.

Are Exchange Server 2003 and Exchange Server 2007 vulnerable to March 2021 Exchange server security vulnerabilities?

No. After performing code reviews, we can state that the code involved in the attack chain to begin (CVE-2021-26855) was not in the product before Exchange Server 2013. Exchange 2007 includes the UM service, but it doesn’t include the code that made Exchange Server 2010 vulnerable. Exchange 2003 does not include the UM service.

 

Major updates to this post:

The Exchange Team

293 Comments
Microsoft

@Brent Berwick Those particular Autodiscover entries mean that the machine was 'probed', yes. I hesitate to make a statement of the state of the machine because all of the logs should be reviewed, but we have seen quite a few cases where this was the only thing on the machine and no other evidence of compromise was found. Please make sure the rest of results are clean and that the machine is updated!

Copper Contributor

For those who installed KB5000871, after reboot server, the Exchange Management Shell and ECP not working anymore. In eventlog,

Event ID 15021, HTTPEvent “error occurred while using SSL configuration for endpoint 0.0.0.0:444″

 

Can refer this link to fix:

 

http://www.stormbreaker.tech/?p=173 

 

 

Copper Contributor

 

For help and update the post by @Atout76  OWA and ECP ERRORS after the update

 

https://docs.microsoft.com/en-us/exchange/troubleshoot/client-connectivity/owa-stops-working-after-u...

 

 

Thank you

 

Copper Contributor
 
 
 
 

 

Regarding your reply to @EddieRowe  "Exchange 2010 is not vulnerable to the same attack chain as Exchange 2013/2016/2019, but there is a vulnerability that we have addressed for Exchange 2010 and our recommendation is to install the update."

 

What exactly does this mean?

 

I discovered that, like the Exchange Health script mentioned in the blog post, the scripts linked to below are also incompatible with Exchange 2010:

Update [03/04/2020]: The Exchange Server team released a script for checking HAFNIUM indicators of compromise (IOCs). See Scan Exchange log files for indicators of compromise.

 

Not only are those scripts incompatible with Exchange 2010, but both the code of the scripts and the the manual methods described seem to be pointing to a completely different logging filestructure (and, presumably architecture) meant to scan exchange 2013 through 2019. I can't even modify the code to point to different relative paths or registry entries to query for said paths. Nothing lines up.

 

My question is,

 

Given your reply to Eddie, does that mean that CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, CVE-2021-27065 do not apply to exchange 2010?

  • IF any of them DO apply, In order to locate IOCs, should I be trying to figure out the equivalent logs and perhaps even logging formats etc on my exchange server on my own to look for the same activity? With the assumption that we might have been compromised with the same end results, but with another "attack chain"? 
  • IF NONE of those CVEs apply, what CVEs DO apply? What is the vulnerability? Is it active and in the wild already? And what exactly should I be looking for and in what logs, etc?

 

It is my responsibility to my company and co-workers to do whatever due diligence must be done to discover if we have been compromised after a zero-day notification like this applies to our environment. As it stands I have no idea what I am looking for.

 

Thanks!

 

 

 

 

 

 

 

Microsoft

@ChrisJButler If you look through all of the CVEs (reference here) - the only CVE that applies to Exchange 2010 is CVE-2021-26857. It is a "Remote Code Execution" but it is not a 'start' of the attach chain that Exchange 2013, 2016, 2019 are vulnerable to (as they all basically begin with CVE-2021-26855).

As MSTIC blog post says here, the exploitation of CVE-2021-26857 requires "administrator permission or another vulnerability to exploit". But again, as CVE-2021-26855 does not apply to Exchange 2010, that can not be the "other known vulnerability" on Exchange 2010.

It is, however, still an issue that was discovered as a part of the big picture here, but because it is not exploited in the wild as other versions (because CVE-2021-26855 does not apply) - it is a 'Defense in depth' update. We deemed it important enough to ship a fix for such old code, but it is not actively exploited because there is not an 'entry point' as with other versions.

Hope that helps?

Copper Contributor

I’ve no idea why the patch would not deploy even though I was 100% on cu19 exchange 2016.

 

I ended up deploying via windows update and it worked with no issues.

 

image.jpg

Brass Contributor

My servers have started to log a new event in the Application log.  I wonder if this is the attack failing due to the security update being in place.

 

[Ecp] An internal server error occurred. The unhandled exception was: System.ArgumentException: Invalid input value
Parameter name: input
at Microsoft.Exchange.Data.ApplicationLogic.Cafe.BackEndServer.FromString(String input)
at Microsoft.Exchange.HttpProxy.BEResourceRequestHandler.ResolveAnchorMailbox()
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalBeginCalculateTargetBackEnd(AnchorMailbox& anchorMailbox)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<BeginCalculateTargetBackEnd>b__3b()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(TryDelegate tryDelegate, FilterDelegate filterDelegate, CatchDelegate catchDelegate)

Copper Contributor

@Nino Bilic 

YES, my good man, it certainly does!

 

You just lowered my blood pressure by a few points :)

 

I just joined techcommunity.microsoft.com and since I am already logged in as my live.com account in another tab for Skype, I was logged in using those credentials, and was taken to a page where I needed to create a profile, choose a profile name, etc.

 

My microsoft (live, what have you) account has my gmail account as my primary login name, and my profile in this community shows that gmail address correctly as my email address for the profile and for the microsoft account. I filled out the Personal information page next and put in my work information, including my non-office365 work email address.

 

Email notifications are turned on but I received not a single email from the blog, neither at setup of my profile nor when you replied with a mention of me here. Any idea where I should look ? I just happened to be refreshing this blog just after posting my comment or I would have missed your very helpful reply.

 

Thanks again!

 

Microsoft

@ChrisJButler Umm... I'd like to be able to say that I fully understand how this notification thing works but alas, I do not. :( I can tell you that I do get email notifications if I get @ mentioned in posts. I know you mentioned notifications are on, but I wonder if you mean the "Email me when someone replies" thing that I see on individual comments (like this one) or when you click on your account avatar, then My Subscriptions > Notification settings > Post's I'm @ mentioned in?

Copper Contributor

@The_Exchange_Team 

Awesome Job by Microsoft. We completed patching our environment with no issue.

To all makes sure you put server in maintenance mode before you start patching and follow proper direction.

Copper Contributor

Hi,

 

Can someone help verify the steps to put exchange to maintenance mode in this article is valid?  https://ehloexchange.com/exchange-maintenance-mode/

 

so after put the exchange in maintenance mode > run ELEVATED command prompt to install the update

 

Once the update complete and server is restarted, is the maintenance mode will be removed by itself due to server restart?

 

Thank you for your advice in advance.

Copper Contributor

Hey everyone,

 

Regarding folks who only saw autodiscover attempts for the administrator email (myself included), I made a poll on Reddit to try to gather more info to see if everyone who had further signs of compromise such as webshells being dropped actually had an active administrator account. If you wouldn't mind sharing your experiences so far the thread is here: https://www.reddit.com/r/exchangeserver/comments/lyr2lr/if_you_were_compromised_by_the_latest_0day_h...

 

What's interesting is the theory has been that if you did not have an active admin email, the attackers seemed to give up and move on. But there were people who responded to the poll saying they were compromised even though they did NOT have an active admin email (obviously small sample size so far, but still).

 

So I continue to wonder what the administrator autodiscover probing means and why the attackers didn't go further (at least that we know of so far) as happened with others.

 

If anyone else can share some information on these autodiscover indicators and can confirm they were probes only, I'd love to hear it.

 

Thanks!

Microsoft

@JamesTechnet - FYI, there is a new development that can help here (with remediation): the MSTIC team has updated their post about March vulnerabilities (scroll all the way down) to mention that the Microsoft Safety Scanner - MSERT tool has been updated to scan Exchange server.

Copper Contributor

@Nino Bilic Thanks Nino. I saw that and just ran it this morning. Came back clean. So I feel based on the information we have up to this point we were very lucky. Do you think it can be said with reasonable confidence that if all you found were Autodiscover log entries and MSERT did not find anything, that there is a high likelihood you were just probed?

 

I also have a question I hope someone can answer: the autodiscover entries correspond to a POST in the IIS logs to /ecp/y.js. This file doesn't exist on the server and I haven't been able to find a good explanation of what is going on here in the attack. Is the POST request sent to this file what contains the commands to perform authentication bypass? Or is it what is performing the autodiscover request? Is it even a real file? Does it matter if it doesn't actually exist?

 

Sorry I don't understand how this works. If anyone could shed some light please!

Copper Contributor

We are seeing HTTP status code 241 in IIS logs taken from compromised Exchange servers - is this something that anyone else has noticed or is there someone at Microsoft who can collaborate this unusual code appearing in logs?

Copper Contributor

Do you prepare to release an update for CU 16 in the future? because we could not update CU easily. Many jobs to be done...

Iron Contributor

@Nino Bilic On a Windows 2019 Core installation, outside of running Healthchecker, how do you confirm the installation of KB5000871 ?  It does not show up in the WAC or via the get-hotfix command.

Copper Contributor

@JamesTechnet Same her, luckly some customer only have these autodiscover entries but no suspicous file's where found on these exchange Servers with the Microsoft Safety Scanner Tool.

 

@Nino Bilic It would nice to know if these systems should be clean if only such "probe" entries where logged but are these Servers now clean or should we start to restore any of these exchange servers to be 100% sure ?

I'mean how can we give our customers an  answer that their system is clean and not comprimised ? Is the Microsoft Safety Scanner, the Health Checker Script and the Test-ProxyLogon (https://github.com/microsoft/CSS-Exchange/tree/main/Security) enough to verify ?

Copper Contributor

Hi,

 

we have a few customers were we get an error while running the script. It can not access the "%exchangeinstallpath%\OABGeneratorLog" because the directory "OABGeneratorLog" does not exist. Is this suspicous or is it possible that there has been no "OABGeneratorLog"?

 

Running the MSERT.EXE tool does not find any threats on those systems.

 

Thank you!

Jan

Copper Contributor

Hi team, I'm late to this party, but is there any risk for machines with just the Exchange 2010 admin tools installed? I've patched anyway but am trying to advise others. Obviously not a concern with newer versions as they are web based.

Brass Contributor

Since installing the security update I've been seeing an "Invalid input value" periodically logged in the Application log.  I wonder if this is coming from the attacker still trying to get into the system.

 

[Ecp] An internal server error occurred. The unhandled exception was: System.ArgumentException: Invalid input value
Parameter name: input
at Microsoft.Exchange.Data.ApplicationLogic.Cafe.BackEndServer.FromString(String input)
at Microsoft.Exchange.HttpProxy.BEResourceRequestHandler.ResolveAnchorMailbox()
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalBeginCalculateTargetBackEnd(AnchorMailbox& anchorMailbox)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<BeginCalculateTargetBackEnd>b__3b()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(TryDelegate tryDelegate, FilterDelegate filterDelegate, CatchDelegate catchDelegate)

Steel Contributor

Hi

 

Security updates are available for the following specific versions of Exchange:

Exchange Server 2010 (update requires SP 3 or any SP 3 RU – this is a Defense in Depth update)
Exchange Server 2013 (update requires CU 23)
Exchange Server 2016 (update requires CU 19 or CU 18)
Exchange Server 2019 (update requires CU 8 or CU 7)

 

I am running Exchange Server 2016 CU 16......am I affected by this?

Copper Contributor

@Navishkar Sadheo"I am running Exchange Server 2016 CU 16......am I affected by this?"

 

Yes! You need to go current first (CU19 (or 18)) and than apply the security fix. Or at least use the mitigation mentioned in this article: Microsoft Exchange Server Vulnerabilities Mitigations – updated March 6, 2021 – Microsoft Security R...

Bronze Contributor

Hi @The_Exchange_Team ,

Are you able to share how EXO was not affected by this? Or have you had a fix longer time on EXO already?

Microsoft

For the Test-Hafnium scripts (found here: CSS-Exchange/Security at main · microsoft/CSS-Exchange · GitHub) is there a minimum Exchange CU version the servers need to be at for the script to run successfully? Current version is Exchange 2016, CU17. Thanks. Would like to run the script before installing the CUs/Security Updates.

Copper Contributor

We finished installing the zero day patch in our environment, but noticed that for a few installs, it said the Microsoft Exchange Active Directory Topology was running (after disabling the services).  Looking at the service, it appears the order of disabling / stopping the services in the Zero day patch is opposite of the process in a rollup where the service is disabled, then stopped.  In the zero day patch, all of the services are stopped completely before disabling the services.  Watching this happen, I saw the topology service stopped, then more services stopped down the line, then the services were disabled.  The patch then proceeded and mentioned the topology service was running.  Viewing the services, you could see the service disabled, but running.  On the next install, I saw the topology service stopped, then automatically restart before all the services were stopped down the line.  I suggest keeping the services mmc open during the install to keep an eye on the services because of this and stop the topology service if it starts after it is stopped initially. 

Brass Contributor
@GeekSpaz , thanks for the suggestions to run C:\Program Files\MicrosoftExchange ServerV14\Bin\UpdateCas.ps1 and the IISReset, but sadly that made no difference.  I also see in the troubleshooting a suggestion to run UpdateConfigFiles.ps1 after UpdateCas.ps1.
 
 
 
 

 

We remain unable to patch our Exchange 2010 SP3 servers with RU31 or Ru32 so it would appear to be me that RU31 is broken and the defect is also in RU32.  After patching the server with the CAS role and holding the test mailbox, I see errors on the OTHER webmail server that has NOT been patched.  Not patched because applying to one server takes it out.  On the webmail server I found an event 136 and Source MSExchange OWA at the same time the OWA login says " The mailbox you're trying to access isn't currently available. If the problem continues, contact your helpdesk."  Are these Exchange 2010 SP3 servers looking to have the same RU level in order to have basic functionality working?  Is it possible that this is just an issue known to an experienced Exchange admin (which we do not have...why I am trying get us to O365)?


"The sign-in to Outlook Web App failed. User /O=xxxxx/OU=yyyyy/cn=Recipients/cn=test2 has a mailbox on 14.3.123.0 server version. However, no Client Access server or front-end server with a matching version was found to handle the request.
 
For users on Exchange 2007 and above, you can configure the Outlook Web Access URL for redirection using the externalURL parameter on the Exchange /owa virtual directory."
Copper Contributor

@Marc K 

Similar here.

 

Source: MSExchange Front End HTTP Proxy

[Owa] An internal server error occurred. The unhandled exception was: System.ArgumentException: Invalid input value
Parameter name: input
at Microsoft.Exchange.Data.ApplicationLogic.Cafe.BackEndServer.FromString(String input)
at Microsoft.Exchange.HttpProxy.OwaResourceProxyRequestHandler.ResolveAnchorMailbox()
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalBeginCalculateTargetBackEnd(AnchorMailbox& anchorMailbox)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<BeginCalculateTargetBackEnd>b__280_0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)

 

Source: ASP.NET 4.0.30319.0

Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 3/8/2021 5:33:57 AM
Event time (UTC): 3/8/2021 1:33:57 PM
Event ID: 049c535e9be849829a634bccfc74e4ea
Event sequence: 5
Event occurrence: 4
Event detail code: 0

Application information:
Application domain: /LM/W3SVC/1/ROOT/owa-1-132593003067932026
Trust level: Full
Application Virtual Path: /owa
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\
Machine name: EXCH

Process information:
Process ID: 12956
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM

Exception information:
Exception type: ArgumentException
Exception message: Invalid input value
Parameter name: input
at Microsoft.Exchange.Data.ApplicationLogic.Cafe.BackEndServer.FromString(String input)
at Microsoft.Exchange.HttpProxy.OwaResourceProxyRequestHandler.ResolveAnchorMailbox()
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalBeginCalculateTargetBackEnd(AnchorMailbox& anchorMailbox)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<BeginCalculateTargetBackEnd>b__280_0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CallThreadEntranceMethod(Action method)



Request information:
Request URL: https://public_ip:443/owa/auth/x.js
Request path: /owa/auth/x.js
User host address: 35.244.82.13
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\SYSTEM

Thread information:
Thread ID: 7
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace: at Microsoft.Exchange.Data.ApplicationLogic.Cafe.BackEndServer.FromString(String input)
at Microsoft.Exchange.HttpProxy.OwaResourceProxyRequestHandler.ResolveAnchorMailbox()
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalBeginCalculateTargetBackEnd(AnchorMailbox& anchorMailbox)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<BeginCalculateTargetBackEnd>b__280_0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CallThreadEntranceMethod(Action method)


Custom event details:

 

Matching event in \Logging\HttpProxy\Owa HttpProxy_2021030813-1.LOG

2021-03-08T13:33:57.477Z,8b72ab0b-1b16-46cf-b84e-48d6cbfa7b45,15,1,2176,9,,Owa,public_ip,/owa/auth/x.js,,FBA,false,,,,Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/88.0.4324.182 Safari/537.36 Edg/88.0.705.81,35.244.82.13,EXCH,302,,,GET,,,,,X-AnonResource-Backend-Cookie,,,,0,,,,0,,,0,,0,,0,0,,0,106,0,,,,,,,,,0,104,2,,106,,106,106,,,,BeginRequest=2021-03-08T13:33:57.371Z;CorrelationID=<empty>;ProxyState-Run=None;ProxyState-Complete=CalculateBackEnd;SharedCacheGuard=0;EndRequest=2021-03-08T13:33:57.477Z;,UnexpectedException=System.ArgumentException: Invalid input value Parameter name: input at Microsoft.Exchange.Data.ApplicationLogic.Cafe.BackEndServer.FromString(String input) at Microsoft.Exchange.HttpProxy.OwaResourceProxyRequestHandler.ResolveAnchorMailbox() at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalBeginCalculateTargetBackEnd(AnchorMailbox& anchorMailbox) at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<BeginCalculateTargetBackEnd>b__280_0() at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate Func`2 filterDelegate Action`1 catchDelegate) at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CallThreadEntranceMethod(Action method);,,,,,

 

and

 

2021-03-06T15:31:05.660Z,16a4dee4-37b2-430f-8df4-3bc228d55faf,15,1,2176,9,,Owa,mail.example.com,/owa/auth/x.js,,FBA,false,,,,Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html),104.225.219.16,EXCH,302,,,GET,,,,,X-AnonResource-Backend-Cookie,,,,0...: Invalid input value Parameter name: input at Microsoft.Exchange.Data.ApplicationLogic.Cafe.BackEndServer.FromString(String input) at Microsoft.Exchange.HttpProxy.OwaResourceProxyRequestHandler.ResolveAnchorMailbox() at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalBeginCalculateTargetBackEnd(AnchorMailbox& anchorMailbox) at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<BeginCalculateTargetBackEnd>b__280_0() at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate Func`2 filterDelegate Action`1 catchDelegate) at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CallThreadEntranceMethod(Action method);,,,,,

 

Is this some kind of new exploit?

 

Copper Contributor

All,

 

We have  already installed the March 2021 Security Update on our Exchange 2013 servers.

We haven't installed  the December 2020 Security patch (Security Update For Exchange Server 2013 CU23 (KB4593466)).

We are planning to install this now. Can  any one please advise if the December Security Update will not overwrite the March Security Update?

Do we need to reinstall the March 2021 update after the December update installation?

Cheers,

Imrul

 

Microsoft

@DVickery - No, machines with management tools only are not impacted (there are no Exchange services). I want to be super clear, though that we are not talking about a 'management server' which many of our Hybrid customers have - a server that you need to keep on premises to be able to run Exchange management tools etc. Because that is vulnerable.

@Petri X - Exchange Online simply does not run using the off the shelf Exchange Server anymore; the code has been diverging for a while.

@MarkMyers - No, it should run. BTW new version of the script is called Test-ProxyLogon

Copper Contributor

@Nino Bilic This bring up an interesting talking point.  MS made the decision to fork the code for Exchange 2019/O365 long ago and one of the selling points for Exchange 2019 was more mature code base with less updates and better stability.  If MS fixed the issue is O365 at some point before this patch was released that says a lot about their priorities and doesn't portend well for on-prem customers and future critical updates.

 

Hopefully this will be fleshed more so that confidence doesn't wane for on-prem customers.

Microsoft

@Netronin I'm just here to say that I never said that "MS fixed the issue is O365 at some point before this patch was released". I have no idea where that came from?

Copper Contributor

@Nino Bilic No but you did mention that O365 no longer runs the off the shelf Exchange version any longer (pretty much since the code was forked) so the assumption is either they fixed it but did not port that fix to Exchange or the design has changed so much from the off the shelf version of Exchange that a fix was not plausible.  Either way, I think there needs to be more discussion about how this will effect on-prem customers with code that exists in Exchange 20xx but not in O365.

 

No way am I blaming you of course.  You just got stuck trying to help out in this mess (which we appreciate).

Copper Contributor

Auto updates and the remediation CU's error out, cant update.

 

"the upgrade patch cannot be installed by the windows installer service....."

Iron Contributor

I realize that a lot of people are still mitigating these vulnerabilities and dealing with the fallout, and that the Exchange team is probably putting in some serious overtime right now, which I appreciate, but a very important question needs to be asked. 

In light of these vulnerabilities, will Microsoft be speeding up the development of the solution to give customers the ability to remove the last Exchange servers in a hybrid environment?

 

Whether the issue is Dirsync, Exchange, AD, Azure AD, or any other technology, 7 years is long enough to wait for a solution. Hybrid customers should not have been put into this precarious situation due to whatever the underlying reasons Microsoft had for not developing a solution for removing the last Exchange servers. 

 

 

Copper Contributor

Hi all, just want to share my experience with upgrading 2013 EXCH from CU4 to CU23 followed by patching with KB5000871.  whole process takes about 6 hours without a glitch!  upgrade steps will be create backup VM (if applicable) > prep schema, AD, domain > restart server > put in maintenance mode > upgrade NET framework 4.8 with elevated command prompt > restart > upgrade CU 23 with elevated command prompt > restart > install KB5000871 with elevated command prompt > restart. 

 

PATCHING DEFINITELY NEEDED.  yesterday I started seeing below log.  after I did patching, I run check again using Zero Day Check Script.ps1 and Test-Hafnium.ps1 and I did not get that result anymore.  perhaps the log get overwritten or CU23 make some changes to Exchange to block that.

 

@{DateTime=2021-03-08T08:31:34.145Z; AnchorMailbox=ServerInfo~a]@SERVER.COM:444/autodiscover/autodiscover.xml?#}
@{DateTime=2021-03-08T08:31:48.208Z; AnchorMailbox=ServerInfo~a]@SERVER.COM:444/mapi/emsmdb/?#}
@{DateTime=2021-03-08T08:31:49.349Z; AnchorMailbox=ServerInfo~a]@SERVER.COM:444/mapi/emsmdb/?#}
@{DateTime=2021-03-08T08:31:54.302Z; AnchorMailbox=ServerInfo~a]@SERVER.COM:444/ecp/proxyLogon.ecp?#}
@{DateTime=2021-03-08T08:32:11.068Z; AnchorMailbox=ServerInfo~a]@SERVER.COM:444/ecp/DDI/DDIService.svc/GetList?msExchEcpCanary=rKkQwilkO0u2gqHxHVTXyVac4PWe49gITJwCW8vyR22f8mjJ7WCvCMHaWiqLfc7R9WF_8XlMAiw.&schema=VirtualDirectory#}
@{DateTime=2021-03-08T08:32:15.084Z; AnchorMailbox=ServerInfo~a]@SERVER.COM:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=rKkQwilkO0u2gqHxHVTXyVac4PWe49gITJwCW8vyR22f8mjJ7WCvCMHaWiqLfc7R9WF_8XlMAiw.&schema=OABVirtualDirectory#}
@{DateTime=2021-03-08T08:32:18.396Z; AnchorMailbox=ServerInfo~a]@SERVER.COM:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=rKkQwilkO0u2gqHxHVTXyVac4PWe49gITJwCW8vyR22f8mjJ7WCvCMHaWiqLfc7R9WF_8XlMAiw.&schema=ResetOABVirtualDirectory#}
@{DateTime=2021-03-08T08:32:21.412Z; AnchorMailbox=ServerInfo~a]@SERVER.COM:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=rKkQwilkO0u2gqHxHVTXyVac4PWe49gITJwCW8vyR22f8mjJ7WCvCMHaWiqLfc7R9WF_8XlMAiw.&schema=OABVirtualDirectory#}
@{DateTime=2021-03-08T08:32:24.568Z; AnchorMailbox=ServerInfo~a]@SERVER.COM:444/autodiscover/autodiscover.xml?#}
@{DateTime=2021-03-08T08:32:28.303Z; AnchorMailbox=ServerInfo~a]@SERVER.COM/OAB/6b6a1886-0f8c-4bab-8ad0-4fa11a04d3b5/oab.xml#}
@{DateTime=2021-03-08T08:32:59.819Z; AnchorMailbox=ServerInfo~a]@SERVER.COM/OAB/6b6a1886-0f8c-4bab-8ad0-4fa11a04d3b5/oab.xml#}
@{DateTime=2021-03-08T08:32:59.850Z; AnchorMailbox=ServerInfo~a]@SERVER.COM/OAB/6b6a1886-0f8c-4bab-8ad0-4fa11a04d3b5/oab.xml#}
@{DateTime=2021-03-08T08:33:00.069Z; AnchorMailbox=ServerInfo~a]@SERVER.COM/OAB/6b6a1886-0f8c-4bab-8ad0-4fa11a04d3b5/oab.xml#}
@{DateTime=2021-03-08T08:33:02.663Z; AnchorMailbox=ServerInfo~a]@SERVER.COM/OAB/6b6a1886-0f8c-4bab-8ad0-4fa11a04d3b5/3f06030f-67ac-4324-956a-65412f64f52d-data-716.lzx#}
@{DateTime=2021-03-08T23:53:23.861Z; AnchorMailbox=ServerInfo~burpcollaborator.net/ecp/default.flt?}

Copper Contributor

I have used the MSERT tool to scan my onsite exchange 2013 server  there were no web shells found by the tools.  However when I ran the Test-ProxyLogon.ps1 it found "[CVE-2021-26855] Suspicious activity found in Http Proxy log! " I have applied KB5000871 that patches this vulnerability and ran the ExchangeMitigations.ps1 scripts  but I am unsure if there are any recommended actions besides this. On Reddit I found this suggestion: "Dust off and nuke the site from orbit; it's the only way to be sure. Seriously though, you can't trust that machine anymore. It could well be rootkitted"

what do you recommend?

Copper Contributor

@Nino Bilic Nino, I don't know if this is something you can help with or if there is a way to ask those updating the GitHub scripts, but I wanted to run CompareExchangeHashes.ps1 on our Exchange server, however it seems the script requires an internet connection and connectivity to AD. I've already pulled our Exchange server offline and replaced it with another, so I'm wondering if it's possible to make the script run offline?

 

Thanks for your help!

Copper Contributor

@The_Exchange_Team 
Just to confirm, the 3/2/2021 security update is cumulative to all previously released security updates for Exchange 2013 CU23?  Typically it is noted in the "security update replacement information" section of the update (here).  The latest article does state that it is a replacement for the previously released security updates for Exchange 2016/2019. However, it does not indicate that it is a replacement for the 12/8/2020 security update for Exchange 2013 (here).  Please verify and/or update the KB description with the appropriate information.

Microsoft

@JamesTechnet I pinged the folks who worked on this script and heard back that yes, the script has been modified to allow for this:

They need to download the baseline hashes separately from https://github.com/microsoft/CSS-Exchange/releases/latest and put it in the same folder as the script and run it. The script should output the exchange versions to compare against.

Copper Contributor

@Nino Bilic You are awesome! Thanks!

Copper Contributor

@Nino Bilic I can't see anywhere an update for Exchange 2010 SP1. Is that Exchange version impacted by this?

Copper Contributor

Hi,

we are running Exchange Server 2016 CU12. I can't find anything about it in any docs. Are we not affected???

rg

Copper Contributor

@RadiologieLuebeck and @adenluong I guess you are also affected, but before you can install the critical fixes you have to move / install the latest CU or SP for Exchange 2010. It may be possible to install the patch directly, but i'm not sure if that will work, we had to update our internal DAG to the latest CU and afterwards we installed the security patch.

On the top of this page you find this additional Info: https://techcommunity.microsoft.com/t5/exchange-team-blog/march-2021-exchange-server-security-update...

 

But i guess your to late, check first with the scripts from Microsoft if your compromised, if your already comprimiesed i would rather turn of or disconnect your exchange server from network and restore it before the attack has startet, update the exchange server with all patches, change all passwords, verify ad accounts etc.

Copper Contributor

@The_Exchange_Team 

Post updating Exchange2016 CU19 Security patch the build version showing as CU19 only.

How can I check its installed or not.

 

 

REgards

Anand S

Copper Contributor

 

Is the build version updated for Security patch which u updated.

In my case I was upgraded Exchange2016CU19 with security patch and its showing CU19 version only.

Not showing security patch build version.

 

 

Regards

Anand S

 

 

Copper Contributor

@reaper111 thank you! Highly appreciate. I checked it out. We will perform update to 2010 SP3 and then apply the pack.

 

 

Copper Contributor

@The_Exchange_Team 

 

Hi, Please can I have clarity on what is the AnchorMailbox that is used in the attack surface?

 

Thank you

Copper Contributor

All, 

Security Update For Exchange Server 2013 CU23 (KB4593466) (Dec 2020) is included in the March 2, 2021 (KB5000871) update.

If you have missed the Dec 2020 SU, and updated your servers with March 2, 2021 (KB5000871) SU,  you don't need to worry.

Brass Contributor

OAB URL changed to server FQDN after installing the security update on Exchange Server 2019 CU19 and users can't download the address book since then with the following message:

 

Task '<email address>' reported error (0x80072F06): 'Unknown Error 0x80072F06'

So I need to manually change it back using Get-OabVirtualDirectory/Set-OabVirtualDirectory.

Co-Authors
Version history
Last update:
‎Mar 19 2021 01:44 PM
Updated by: