On November 29, 2018, MITRE published the results of their evaluation of several endpoint detection and response (EDR) solutions, testing them against a chain of attack techniques commonly associated with the APT3 activity group. MITRE avoided direct vendor comparisons, but this has not prevented participating vendors from claiming victory and leveraging the results in aggressive marketing campaigns.
Cutting through the marketing hype, the evaluation highlighted Windows Defender Advanced Threat Protection’s (Windows Defender ATP) distinct, superior capabilities when compared with other participating vendors:
- Powerful automated alerting built on machine learning and behavioral detections, vs. vendor solutions that relied heavily on human analysts
- The fewest misses among all participating vendors
- Excellent coverage of the most critical, high-impact attack techniques
- Unique Microsoft Threat Protection suite integration, enabling signal sharing and diverse optics with Azure Advanced Threat Protection (Azure ATP) and other Microsoft security solutions connected to the Intelligent Security Graph
We are proud to have been one of the first vendors to jump all-in and join this first MITRE evaluation—we strongly believe that it is a good first step in effectively identifying the most relevant EDR solutions.
Detection types and coverage levels
To run the evaluation, MITRE asked EDR vendors to prepare test environments so that detection sensors are in place while ensuring that blocking and other preventive functionality are turned off. MITRE adversary emulators—their red team—performed a series of activities in two separate end-to-end scenarios to simulate an APT3 attack using techniques in the MITRE ATT&CK framework.
While MITRE used detailed detection types to evaluate results, we’ve mapped their detection types to three simple levels of coverage:
- Alert—an alert has been generated for a specific behavior, a general behavior, or an indicator of compromise (IoC); telemetry data is available
- Telemetry—no alert has been generated, but telemetry data for the activity is available (correlated, enriched, etc.)
- None—no alert has been generated, and no telemetry data for the activity is available
Evaluation results
We provide the following comparisons noting that all vendors that participated in this first MITRE evaluation should be commended for their willingness to be part of an open exercise that benefits our customers.
Some vendors elected to include their human-assisted managed hunting service in the evaluation. We believe alerts or detections generated manually should be separated from automated alerts, and have done so in our comparisons below. Furthermore, we have retrospectively included our newly announced Microsoft Threat Experts managed hunting service for comparison.
Automated alerting capabilities
Windows Defender ATP is among the solutions with the highest number of automated alerts. Built on machine learning and behavioral detections, our alerts identify a broad range of attack techniques. And while the threat landscape evolves, powerful automation provides detection capabilities that are scalable, reliable, and adaptive.
Compared to solutions that rely heavily on manual detections and are difficult to scale, Windows Defender ATP clearly offers superior detection and alerting capabilities.
Number of misses
Windows Defender ATP had the fewest number of misses (i.e., undetected red team activity) among all solutions evaluated.
Coverage of critical attack techniques
In this MITRE evaluation, all attack techniques appear to be equal in impact and importance. Security analysts, however, will naturally give more importance to a detection of Mimikatz attempting credential theft over a whoami command for enumeration.
Windows Defender ATP is among the few vendors that successfully detected what are widely considered to be high-impact attack techniques—specific methods that can lead to further compromise or do greater damage—like credential dumping (T1003), process injection (T1055), and input capture (T1056). The table below shows how Windows Defender ATP provided the best coverage for these critical techniques.
Coverage of critical techniques as evaluated by MITRE
Windows Defender ATP alert for process injection (image from MITRE)
Extended visibility with Microsoft Threat Protection
During the evaluation, Windows Defender ATP and Azure ATP were both enabled as part of the Microsoft Threat Protection solution built on the Microsoft Intelligent Security Graph. As a result, Azure ATP generated additional detections from domain controller signals.
Azure ATP showing attack RDP activity and creation of a remote service (image from MITRE)
This advantage is even more pronounced when signals from other Microsoft Threat Protection solutions, such as Office 365 ATP and Azure Security Center, are available. This integration exponentially increases our ability to find malicious activities and enforce restrictions that prevent malware implantation and exfiltration of sensitive data.
Significant lead with Microsoft Threat Experts
Windows Defender ATP delivered amazing results with its automated detection capabilities only. However, to make the comparison complete, we took another step and simulated the involvement of Microsoft Threat Experts—our recently announced managed hunting service.
We involved security professionals that had no knowledge of the MITRE evaluation parameters or the characteristics of the evaluation network. They used only data collected by Windows Defender ATP during the MITRE evaluation.
With Microsoft Threat Experts, Windows Defender ATP was able to provide full coverage of the entire attack chain. For example, Microsoft Threat Experts raised alerts for the Exfiltration step—a very common miss shared across the most competitive solutions, including the ones that relied on human-assisted services during the MITRE evaluation.
Coverage of the attack chain in scenario 1 with and without Microsoft Threat Experts
Microsoft Threat Experts provided comprehensive coverage of the attack chain in both scenarios.
Coverage of the attack chain in scenario 2 with and without Microsoft Threat Experts
The following screenshot shows an alert for the Data Staged (T1074) technique generated during our tests with Microsoft Threat Experts.
Microsoft Threat Experts alert on Windows Defender ATP for data staging and exfiltration
Thoughts on MITRE’s evaluation methodology
While attacks constantly evolve and are becoming more and more sophisticated, we believe MITRE provided a comprehensive evaluation that objectively assessed EDR effectiveness against real-world attacks. We particularly like the following aspects of the evaluation:
- Transparency—the details about the evaluation, including the testing protocol and the coverage of the tests, were publicly disclosed.
- Detailed detection assessment—instead of providing simple binary detection results, MITRE provided granular grading based on detection methods and how solutions surfaced the detections.
- Multiple attack techniques—the evaluation covered a broad range of techniques that are documented in the MITRE ATT&CK framework and can come into play during actual, sophisticated attacks, as opposed to focusing on specific artifacts such as malware components.
Of course, there are a few considerations that can help guide future evaluations:
- No false positive evaluation and use of clean environments—the use of fresh virtual machines with no historical and benign background activity could benefit overly aggressive solutions, whereas other solutions generate minimal noise by utilizing anomaly detection, machine learning, and other advanced AI technologies.
- Evaluation of human-generated alerts—two of the participating solutions included a managed, human-assisted service that generated a significant number of detections, while all other solutions relied on automated product-generated alerts. The evaluation, though excellent for testing automated alerting capabilities, is inadequate for assessing the effectiveness of a human-assisted service often affected by red-blue team transparency, network size, and noise levels.
- Emphasis on alerts over correlated telemetry—our conversations with analysts and security practitioners show that relevant data in the form of associated telemetry can be more effective than numerous distinct alerts. This way, SecOps personnel can focus on a few alerts without losing sight of other breach activities.
- No ranking or weighting of attack techniques—by weighting attack techniques, testers can identify solutions that stop the most critical and impactful breach activities.
Additional Windows Defender ATP capabilities
While providing class-leading detection capabilities as evidenced by the evaluation results, Windows Defender ATP has many other powerful capabilities:
- We empower customer SecOps with powerful tools, enabling them to leverage their own expertise during response and hunting. Just as we released a new incident-based experience within the portal, we are continually enhancing existing advanced hunting capabilities as well as providing more direct response channels that allow SecOps to reach specific devices and perform forensics without leaving the operations center.
- We are expanding our threat response capabilities with threat analytics and several other new features that provide proactive entry points for threat and exposure identification.
- While the MITRE evaluation didn’t cover preventive functionality, Microsoft continues to strengthen preventive components in the Windows security stack. We are continuously improving strategic mitigation of threats through powerful exploit protection technologies as well as new attack surface reduction rules.
- With the new Microsoft 365 security center, we are providing deeper integration of security capabilities in Azure, Office, and Windows, delivering even more powerful optics and centralized security management and response capabilities.
Windows Defender ATP Team
(NOTE: MITRE updated their results on February 21, 2019 to incorporate evaluation results for two additional solutions not originally evaluated and to make a few minor changes to existing results. This article is based on the latest available information from MITRE at the time of publication.)