On November 29, 2018, MITRE published the results of their evaluation of several endpoint detection and response (EDR) solutions, testing them against a chain of attack techniques commonly associated with the APT3 activity group. MITRE avoided direct vendor comparisons, but this has not prevented participating vendors from claiming victory and leveraging the results in aggressive marketing campaigns.
Cutting through the marketing hype, the evaluation highlighted Windows Defender Advanced Threat Protection’s (Windows Defender ATP) distinct, superior capabilities when compared with other participating vendors:
We are proud to have been one of the first vendors to jump all-in and join this first MITRE evaluation—we strongly believe that it is a good first step in effectively identifying the most relevant EDR solutions.
To run the evaluation, MITRE asked EDR vendors to prepare test environments so that detection sensors are in place while ensuring that blocking and other preventive functionality are turned off. MITRE adversary emulators—their red team—performed a series of activities in two separate end-to-end scenarios to simulate an APT3 attack using techniques in the MITRE ATT&CK framework.
While MITRE used detailed detection types to evaluate results, we’ve mapped their detection types to three simple levels of coverage:
We provide the following comparisons noting that all vendors that participated in this first MITRE evaluation should be commended for their willingness to be part of an open exercise that benefits our customers.
Some vendors elected to include their human-assisted managed hunting service in the evaluation. We believe alerts or detections generated manually should be separated from automated alerts, and have done so in our comparisons below. Furthermore, we have retrospectively included our newly announced Microsoft Threat Experts managed hunting service for comparison.
Windows Defender ATP is among the solutions with the highest number of automated alerts. Built on machine learning and behavioral detections, our alerts identify a broad range of attack techniques. And while the threat landscape evolves, powerful automation provides detection capabilities that are scalable, reliable, and adaptive.
Compared to solutions that rely heavily on manual detections and are difficult to scale, Windows Defender ATP clearly offers superior detection and alerting capabilities.
Windows Defender ATP had the fewest number of misses (i.e., undetected red team activity) among all solutions evaluated.
In this MITRE evaluation, all attack techniques appear to be equal in impact and importance. Security analysts, however, will naturally give more importance to a detection of Mimikatz attempting credential theft over a whoami command for enumeration.
Windows Defender ATP is among the few vendors that successfully detected what are widely considered to be high-impact attack techniques—specific methods that can lead to further compromise or do greater damage—like credential dumping (T1003), process injection (T1055), and input capture (T1056). The table below shows how Windows Defender ATP provided the best coverage for these critical techniques.
Coverage of critical techniques as evaluated by MITRE
Windows Defender ATP alert for process injection (image from MITRE)
During the evaluation, Windows Defender ATP and Azure ATP were both enabled as part of the Microsoft Threat Protection solution built on the Microsoft Intelligent Security Graph. As a result, Azure ATP generated additional detections from domain controller signals.
Azure ATP showing attack RDP activity and creation of a remote service (image from MITRE)
This advantage is even more pronounced when signals from other Microsoft Threat Protection solutions, such as Office 365 ATP and Azure Security Center, are available. This integration exponentially increases our ability to find malicious activities and enforce restrictions that prevent malware implantation and exfiltration of sensitive data.
Windows Defender ATP delivered amazing results with its automated detection capabilities only. However, to make the comparison complete, we took another step and simulated the involvement of Microsoft Threat Experts—our recently announced managed hunting service.
We involved security professionals that had no knowledge of the MITRE evaluation parameters or the characteristics of the evaluation network. They used only data collected by Windows Defender ATP during the MITRE evaluation.
With Microsoft Threat Experts, Windows Defender ATP was able to provide full coverage of the entire attack chain. For example, Microsoft Threat Experts raised alerts for the Exfiltration step—a very common miss shared across the most competitive solutions, including the ones that relied on human-assisted services during the MITRE evaluation.
Coverage of the attack chain in scenario 1 with and without Microsoft Threat Experts
Microsoft Threat Experts provided comprehensive coverage of the attack chain in both scenarios.
Coverage of the attack chain in scenario 2 with and without Microsoft Threat Experts
The following screenshot shows an alert for the Data Staged (T1074) technique generated during our tests with Microsoft Threat Experts.
Microsoft Threat Experts alert on Windows Defender ATP for data staging and exfiltration
While attacks constantly evolve and are becoming more and more sophisticated, we believe MITRE provided a comprehensive evaluation that objectively assessed EDR effectiveness against real-world attacks. We particularly like the following aspects of the evaluation:
Of course, there are a few considerations that can help guide future evaluations:
While providing class-leading detection capabilities as evidenced by the evaluation results, Windows Defender ATP has many other powerful capabilities:
Windows Defender ATP Team
(NOTE: MITRE updated their results on February 21, 2019 to incorporate evaluation results for two additional solutions not originally evaluated and to make a few minor changes to existing results. This article is based on the latest available information from MITRE at the time of publication.)
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.