Addressing Your Feedback on the Exchange Emergency Mitigation Service
Published Oct 01 2021 10:20 AM 132K Views

Now that the September 2021 Cumulative Updates (CUs) for Exchange Server 2019 and Exchange Server 2016 have been released, customers around the world now have access to a new security feature in Exchange Server called the Microsoft Exchange Emergency Mitigation service (EEMS). We want to address some of the initial feedback we’ve received about EEMS, which we’re thrilled to say has been very positive.

Customers and partners alike have said that EEMS is a great step forward for on-premises Exchange customers, and there is widespread support for Microsoft’s efforts in this ever-changing security landscape. But there is always room for improvement, and we very much appreciate the constructive feedback we’ve received.

Manually removing mitigations

Most of the constructive feedback has centered around the need for admins to manually remove mitigations after an applicable Exchange Server Security Update (SU) has been installed. Installing an SU that fixes a mitigated issue does not currently remove the mitigation from the server. Instead, an admin must first apply the SU and then remove any mitigations which no longer apply.

We understand that the need for manual removal of something that was added to the server automatically creates a burden for the admin. We also know that figuring out which mitigations can be removed, and which ones cannot, poses a problem. We want to provide added protections to customers, but we also want to do that in a frictionless way. So, we are looking into ways in which we can address this, including the possibility of the SU itself automatically removing any mitigations that no longer apply.

Notifying admins when a mitigation is applied

Other feedback we received is about letting the admin know when a mitigation is available or has been applied. As we detailed in our original blog post and in the documentation for EEMS, mitigation activity is logged in the Windows Event Log. For example:

  • Events 1005 and 1006 with a source of MSExchange Mitigation Service will be logged for successful actions such as when a mitigation is applied;
  • Event 1001 with the same source will be logged when the EM service is started;
  • Event 1002 with the same source will be logged when the EM service is stopped; and
  • Event 1008 with the same source will be logged for any errors that are encountered, such as when the EM service cannot reach the OCS.

You can use Search-AdminAuditLog to review mitigation-related events. For example, to see when a mitigation was applied, you can run:

Search-AdminAuditLog -Cmdlets Set-ExchangeServer -Parameters MitigationsApplied

As shown below, the EM service also creates log files named Mitigation-Service_<Date> in the in the “V15\Logging\MitigationService” folder under the Exchange Server installation directory.

The_Exchange_Team_0-1632938362946.png

The log rolls over daily, and each daily log should contain 24 entries (one per hour every day). The logs contain details of the tasks performed by the EM service, including fetching mitigations, parsing mitigations, and applying mitigations, as well as details about the information sent to the OCS if sending diagnostic data is enabled. A sample log file is shown below.

The_Exchange_Team_1-1632938387002.png

This level of logging exists because we want to be completely transparent about the activity performed by the EM service. But we also understand the need for admins to know, in real time, when a mitigation has been applied or not. So, we are looking into ways in which we can address this, including the possibility of alerting an admin when a significant action occurs or fails to occur.

Mitigations may be disabled org-wide by default

As detailed in our original blog post, an admin can enable and disable mitigations at an organizational level or at the Exchange server level. The combination of the two available settings determines the behavior of the EM service on each Exchange server in the organization.

A couple of customers have reported that, after installing the September 2021 CU on their Exchange servers, the MitigationsEnabled property at the organizational level (which is configured using Set-OrganizationConfig) had a value of False, instead of the expected default value of True. When set to False, the EM service still checks for mitigations hourly, but it won’t automatically apply mitigations to any Exchange server in the organization.

We’re currently investigating this issue, but in the meantime, if you see this happen in your environment and you want automatic mitigation to take place, you can easily update the value to True by running the following command:

Set-OrganizationConfig -MitigationsEnabled $True

This re-enables automatic mitigation at the organization level.

Thanks again for all the feedback so far on EEMS. We hope you keep it coming!

The Exchange Team

16 Comments
Copper Contributor

Any updates on the issues with Windows Server Core + Proxy Servers?

Copper Contributor

error for Mitigation Service : 

 

id : 1008

 

An unexpected exception occurred. Diagnostic information:

Exception encountered while fetching mitigations : System.Exception: This XML is not deemed safe to consume since Response xml's signing cert is invalid or not from microsoft
bei Microsoft.Exchange.Mitigation.Service.Common.SignatureVerifierUtils.ThrowIfIntegrityChecksFail(SafeXmlDocument xmlDoc)
bei Microsoft.Exchange.Mitigation.Service.Common.SignatureVerifierUtils.GetValidatedDocumentWithoutSignature(SafeXmlDocument xmlDoc)
bei Microsoft.Exchange.Mitigation.Service.Common.Utils.FetchDataFromXmlStream[T](Stream stream)
bei Microsoft.Exchange.Mitigation.Service.Common.Utils.FetchMitigationsFromUrl[T](String url, RemoteCertificateValidationCallback certValidationCallback, X509Certificate clientAuthCert, Boolean isResponseJson)
bei Microsoft.Exchange.Mitigation.Service.MitigationCloudServiceV2.FetchMitigations()
bei Microsoft.Exchange.Mitigation.Service.Mitigations.MitigationEngine.FetchAndApplyMitigation()

 

id 4999

Watson report about to be sent for process id: 5824, with parameters: E12IIS, c-RTL-AMD64, 15.01.2375.007, M.E.Mitigation.Service, M.E.Mitigation.Service, M.E.M.S.C.SignatureVerifierUtils.GetValidatedDocumentWithoutSignature, System.Exception, 9784-dumptidset, 15.01.2375.007.
ErrorReportingEnabled: False

 

 

The mitigation Test is ok : 

[PS] C:\Program Files\Microsoft\Exchange Server\V15\scripts>.\Test-MitigationServiceConnectivity.ps1
Result: Success.
Message: The Mitigation Service endpoint is accessible from this computer.

 

what's the Problem ?

 

Microsoft

@raphG67 Is there some sort of packet / content inspection device at the edge of the network by any chance? Something that could account for the XML arriving not matching the signature (as the XML is signed)? I have to admit I have not heard of this as of yet...

Copper Contributor

no packet or content inspection. We tried to whitelist the URL https://officeclient.microsoft.com/ on the webproxy. But same issue.  

 

another idee ?

Copper Contributor

We understand that the need for manual removal of something that was added to the server automatically creates a burden for the admin. We also know that figuring out which mitigations can be removed, and which ones cannot, poses a problem. We want to provide added protections to customers, but we also want to do that in a frictionless way.


I don't understand how anyone could think requiring manual removal of automatically added temporary mitigations was a good idea in the first place.  If the system is smart enough to install make changes on its own, it should be smart enough to know when to back them out, too, and not rely on admins having to scour logs to find all the myriad places those mitigations can put themselves, determine whether they should be removed, and unwind them.

 

Sorry, but until this can actually be "frictionless," Set-OrganizationConfig -MitigationsEnabled $False.

Brass Contributor

@Nino BilicWe had a similar predicament mentioned by raphG67 (RemoteCertificateValidationCallback certValidationCallback, X509Certificate clientAuthCert,...

Cert chain validation was failing because of an old gpo (Exch servers were not allowed to reach out...“Allow issuer certificate retrieval during path validation.“  - ...because of reasons...)

 

Incomplete certificate chain
Cannot find certificate:
CN=Microsoft RSA TLS CA 02, O=Microsoft Corporation, C=US

 

PS: the url that Exch srv should contact is signed by an X509 cert so thats why it failed for us...which is a good thing.

 

Maybe that will help others.

 

Curious about the result of feedback you have received and mentioned since it has been almost a month since then....

Thank you.

X

 

 

 

 

Copper Contributor

@Nino Bilic None of our Exchange servers have internet connectivity, so what is the recommended way of disabling EEMS, setting the service to disabled in Services or letting the service run and running "Set-OrganizationConfig -MitigationsEnabled $False"?

Microsoft

@Mwrightson - setting the org setting is definitely something you can do. That, however, will not stop the services from running on all Exchange servers. But you can stop the services (set to disabled) and that'll be that.

Copper Contributor

I have been finding the mitigation service returning OutOfMemoryExceptions when it is running:

Watson report about to be sent for process id: 5892, with parameters: E12IIS, c-RTL-AMD64, 15.01.2375.024, M.E.Mitigation.Service, mscorlib, M.W.RegistryKey.InternalGetValue, System.OutOfMemoryException, cc7f-dumptidset, 04.08.4480.000.
ErrorReportingEnabled: False

 

I believe this may be related to our various web services crashing as well, as this error is frequently followed by an outofmemoryexception for activesync/owa/ews/etc.

 

Any thoughts on what may be causing this?

Copper Contributor

We have been happily using the mitigation service for some time, and all of a sudden it stopped working for us with the following error:

 

Message=Exception encountered while fetching mitigations : This XML is not deemed safe to consume since Response xml''s signing cert is invalid or not from microsoft'

 

I decided to inspect the X509 certificates being sent in the request on the https://officeclient.microsoft.com/getexchangemitigations URL, and I noticed that the "Microsoft Exchange XML Signing" certificate expired on the 9th June:

 

2022-06-13 09_17_05-Certificate.png

 

So I think someone needs to update that at Microsoft! :lol:

Copper Contributor

Thank you, HFlet500!

Same issue here!

 

Copper Contributor

Hi

We experience the same issue that started June, 9.

Best regards,

Dmitry

Brass Contributor

@Nino Bilic we are having the same issue as @HFlet500 and @gghiq1040.  This started 2022-06-09T19:15:08.905Z for us.  The only difference is I can't seem to find that signing certificate anywhere.  There isn't really anything useful in the logs.  It was working at 6:15 PM when it checked in then an hour later when it checked in again it stopped and hasn't been working since.  I have restarted the service.  Any ideas on how to fix this?  Thank you.

 

:LogLevel=Error;'S:Message=Exception encountered while fetching mitigations : This XML is not deemed safe to consume since  Response xml''s signing cert is invalid or not from microsoft';S:Source=Microsoft.Exchange.Mitigation.Service.Mitigations.MitigationEngine

Microsoft

@HFlet500 @gghiq1040 @dgk62 @Jpanski Yes we are aware of this problem and are working to resolve it; it will get resolved service-side with no need for you to do anything. There is no actual impact right now (as we have not released any new mitigations) but yes - it is a known problem right now.

Copper Contributor

Thank you!
The issue is resolved.

King regards,

Dmitry

Copper Contributor

I`m getting this error, was working previously. I suspect a certificate revocation check is failing due to limited internet access. 

Any suggestions how we can get round this, or what I would need to permit to get round this issue?

Co-Authors
Version history
Last update:
‎Oct 01 2021 10:20 AM
Updated by: