Microsoft Defender for Endpoint freeze Windows Server 2012 R2

[LucaCavana]

Hello, We onboarded several Windows Server 2012 R2 VM and physical servers on to Microsoft Defender for Endpoint using the new onboarding package by following this doc "https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/configure-server-endpoints...".

Since then we are experiencing random freezes of several Windows Server 2012 R2 servers. The freezes only happens on Windows Server 2012 R2, our other Windows servers, including 2016 enrolled with the same package are fine.

We already applied the latest cumulative updates for the OS, .Net, Antimalware platform and Defender for Endpoint platform. Even after we have applied the latest version of updates the freezes keep happening.

As of now, we are running on antimalware platform 1.1.1800.4 and product platform 4.18.2111.5 / 4.18.2201.6

It looks like that by disabling the Antimalware by using the Group Policy "Turn off Microsoft Defender Antivirus" the freezes cease to happen.

We already investigated using the Windows logs but they are not written after the VM freeze so we did not find any traces. We collected a complete memory dump from the VMware ESXi hypervisor and we converted it into memory.dmp file and opened it with WinDbg. We found no evidence also in the dump file.

Do anyone have the same problem?

[#2]

Hi, this is not a known issue - I suggest opening a support case as soon as possible, especially if you have a reliable reproduction of the issue.

If you'd like to troubleshoot further yourself, https://docs.microsoft.com/en-us/security/defender-endpoint/tune-performance-defender-antivirus - the analyzer could shed some light on potential conflicts.

[#3]

Hello Paul,
thank you for the reply and acknowledgment that this isn't a known issue as I was unable to find any hint on the internet.

We are already working with the support, I'll keep this post updated.

[#4]

Hi Luca,
we are experiencing the same issue on our virtual environment. We have "3 minutes freezes" on Windows 2012 R2 servers, both while working via RDP on there servers or using applications installed on them. Freezes are random and there's no "standard" procedure to reproduce them. Disabling MDE Real Time Protection on the servers it's of great help, freezes issue disappears. We also opened a ticket to Microsoft and we are replying to their questions. We did many MDE Client Analyzer tool runs and we sent the data collected to them. I would like to share with you our knowledge. I'm looking forward for your reply. BR. Paolo

[#5]

Hi Paolo,
we opened a ticket for our customer and the cause of out freeze was due to the SecureWorks Red Cloak agent. The agent was installed some time ago and never manifested this behavior until MDE was installed on the servers.

If you uninstall the Red Cloak or stop the real time protection of Defender for Endpoint the freezes stops.

This has been determined after sending the VM RAM to Microsoft, actually the first thing we did when our customer notified us.
I suggest you do not reboot a frozen VM but instead collect it's RAM and pass it through the WinDBG or hand it over to Microsoft. It contains valuable information.

[#6]

Hi Luca,
thank you very much for your suggestion ! We are in the middle of a transition phase between two antivirus products, Cylance Protect (former one) and MDE (new one). They both are running on many servers. Maybe there a correlation/interference and maybe the storage behaviour comes from this. We will do the RAM collection and I'll let you know.

[#7]

Hi Luca and Paolo, I think it's very important here to point out that running any security solution alongside another requires some consideration around interaction and identify then configure required exclusions or running mode.

The recommendation is firstly to avoid 2 active AV solutions (like Defender AV+Cylance) as they would both be in the real-time blocking path. Recommend running in Defender Antivirus passive mode until such time Cylance is uninstalled, unless the intent is to maintain Cylance as the AV (but we recommend running our full stack, see https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/microsoft-defender-antivir... to learn about affected functionality).

https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/switch-to-mde-phase-3?view... has a lot of good information on how to switch over, including using passive mode. Paolo, I suggest investigating this approach for your scenario before opening a support case.

Then, if the other security solution is not in the blocking path (like AV), please consult the tool's documentation for suggested AV exclusions. If there are none, the performance analyzer tool (note this is not the connectivity analyzer tool) can help with identification: https://docs.microsoft.com/en-us/security/defender-endpoint/tune-performance-defender-antivirus

Turning off Defender Antivirus altogether in the context of (being onboarded to) Microsoft Defender for Endpoint is not recommended for production; either apply the right exclusions in case of interaction with non-AV, else consider passive mode to coexist with non-Microsoft antimalware solutions.

[#8]

Hello PaulHb,

it's a little different in our case.
The former antivirus solution is of course uninstalled (it was Trend Micro) before installing the Defender AV on 2012 R2 server, as well all the other supported OS.

The problem was the third party EDR (SecureWorks). We did not foresee it as a possible source of problems and the customer does not want to decommission it.

It looks like we resolved by excluding the MDE paths and processes from SecureWorks (on which we have no control on).

[#9]

Thanks Luca - whilst my reply was targeted at multiple scenarios, in your specific case the paragraph around setting exclusions for MDE processes in the non-Microsoft solution applies; as well as the general recommendation to proceed with caution in any such scenario.

Appreciate your work with our support team to get to a resolution path!

[#10]

Hello @LucaCavana,

did you ever come to root cause of the freeze? We are having similar issue, on the case with MS Premier Support and their analysis points to Kernel Extended Attributes (Kernel Extended Attributes - Windows drivers | Microsoft Docs), but the only recommendation is to upgrade to newer OS.

Thanks, Vojtech

[#11]

@Vojtech_Fiurasek Hello,

we removed the old EDR solution, this stopped the freezes.

[#12]

HI, how did you remove the old EDR solution? I'm having the same scenario.@LucaCavana 

[#13]

I like to join

[#14]

We're having the same issue. During MS patch night we randomly have several Windows 2012R2 just completely freeze where we have to hard reboot them. And the patches rolls back. 

All this happened after moving from Cylance to Defender ATP (EDR) and Defender AV. It's been a nightmare!

Anyone else having similar issues?