The exponential growth of threat actors, coupled with the proliferation of cybersecurity solutions has inundated security operation centers (SOCs) with a flood of alerts. SOC teams receive an average of 4,484 alerts per day and spend up to 3 hours manually triaging to separate genuine threats from noise. In response, alert correlation has become an indispensable tool in the defender's arsenal, allowing SOCs to consolidate disparate alerts into cohesive incidents, dramatically reducing the number of analyst investigations.
Earlier this year, we announced the general availability of Microsoft’s unified security operations platform that brought together the full capabilities of an industry-leading cloud-native security information and event management (SIEM), comprehensive extended detection and response (XDR), and generative AI built specifically for cybersecurity.
As part of the unified platform, we also evolved our leading correlation engine, which is projected to save 7.2M analyst hours annually, or $241M across our customers per year.
In this blog post we will share deep insights into the innovative research that infuses powerful data science and threat intelligence to correlate detections across first and third-party data via Microsoft Defender XDR & Microsoft Sentinel with 99% accuracy.
The Challenges of Incident Correlation
Cybersecurity incident correlation is critical for any SOC – the correlation helps connect individual security alerts and events to spot patterns and uncover hidden threats that might be missed if looked at individually. It enables organizations to detect and respond to sophisticated cyberattacks more quickly and holistically, but challenges with traditional technologies remain:
- Mitigating false correlations. False correlations pose a significant risk and can lead to unwarranted actions on benign devices or users, disrupting vital company operations. Additionally, over-correlation can result in “black hole”' incidents where all alerts within an enterprise begin to correlate indiscriminately
- Minimizing missed correlations. Avoiding false negatives is equally important, as a missed correlation could be the difference between the key context required to disrupt a cyberattack, preventing the loss of valuable data and intellectual property
- Scalability and timeliness. Ingesting billions of alerts with varying degrees of fidelity across a multitude of security products presents a monumental correlation challenge. Therefore, requiring a robust infrastructure and an efficient methodology Furthermore, these correlations need to happen in near real-time to keep SOCs up to date
- TI and Domain Knowledge. Correlation across diverse entity types such as IP addresses and files often requires customers to rely on specialized threat intelligence (TI) and domain knowledge to mitigate false positive and false negative correlations
Microsoft’s Unified Security Operations Provides Unique Correlation Technology
Microsoft’s XDR and SIEM solutions have long provided effective incident correlation to customers, saving millions of analyst hours and delivering an effective response to attacks.
In the unified security operations platform, we brought together Microsoft Defender XDR and Microsoft Sentinel, which allowed us to evolve and reshape how traditional correlation technologies work. Security analysts now benefit from a scale framework designed to correlate billions of security alerts even more effectively. Unlike traditional methods that rely on predefined conditions and fixed logic to identify relationships and patterns—and struggle to adapt and scale to the evolving and intricate nature of enterprise security landscapes—the correlation engine in the unified security operations platform employs a geo-distributed, graph-based approach that continuously integrates fresh threat intelligence and security domain knowledge to adapt to the evolving security landscape. This allows us to seamlessly handle the vast complexities of alert correlation across numerous enterprises by leveraging data from Defender workloads and third-party sources ingested via Microsoft Sentinel.
This framework infuses expert domain knowledge and real-time threat intelligence, ensuring accurate, context-driven correlations that significantly reduce false positive and false negative correlations. Additionally, the correlation engine dynamically adapts using a self-learning model, continuously refining its processes by mining incident patterns and incorporating feedback from security experts to offer a scalable and precise solution to modern cybersecurity challenges.
Key Innovations
We introduced multiple key innovations tailored to ensure accurate and scalable incident correlation (see Figure 1):
- Geo-distributed architecture. Enhances data handling efficiency by distributing processing across multiple geographic locations and PySpark clusters
- Graph-based approach. Utilizes graph mining algorithms to optimize the correlation process, making the system scalable to billions of alerts
- Breaking the boundary between 1st and 3rd party alerts. Every hour, we profile first and third-party detectors to ensure they meet key correlation safety checks before allowing cross-detector correlation (outlined below)
- Domain knowledge and Threat Intelligence integration. We are now combining real-time threat intelligence with expert security insight to create highly contextualized and accurate incidents
- Continuous adaptation. Features a human-in-the-loop feedback system that mines incident patterns and refines the correlation process, ensuring the framework evolves to tackle emerging threats
- High accuracy. Extensive analysis shows that our correlations are over 99% accurate, significantly up-leveling the incident formation process
Ensuring High Fidelity Correlations for any Data Source
A majority of organizations have detections from multiple data sources and consume data in various ways whether if that’s through an XDR or a data connector. For data consumed through an XDR, because it’s native to the vendor, is normalized and has higher fidelity compared to data that comes through a connector which can produce a ton of noise and at lower fidelity. This is where correlation becomes extremely important, because alerts with varying degrees of fidelity are difficult to analyze and slow down the response time if a pattern is missed or mis-identified.
To ensure alerts can be correlated across any data source, we introduced three safety checks to activate cross-detector correlation:
- Low volume detector. We examine the historical alert volume for each detector to ensure it is below a set threshold
- Low evidence detector. The average historical number of distinct values per entity type in a detector should not exceed predetermined values
- Low evidence alert. Similarly, the number of distinct entities associated with an individual alert are constrained to the same thresholds as the generating detector
Together, these checks ensure incident quality by correlating high-fidelity third-party alerts with first-party ones and creating separate incidents for low-fidelity third-party alerts that do not pass all three safety checks. By filtering out low-fidelity alerts from key incidents, the SOC can focus on quality detections for their threat hunting needs across any data source.
Looking ahead
Defending against cyberattacks hinges on the ability to accurately and correlate alerts at scale across numerous sources and alert types. By leveraging a unified platform that consolidates alerts across multiple workloads, organizations benefit not only from streamlining their security operations but also gain deeper insights into potential threats and vulnerabilities. This integrated approach enhances response times, reduces false positives, and allows for more proactive threat mitigation strategies. Ultimately, the unified platform optimizes the efficiency and efficacy of security measures, enabling organizations to stay ahead of evolving cyber threats and safeguard their critical assets more effectively.
Learn More
Check out our resources to learn more about the new incident correlation engine and our recent security announcements:
- Read the unified security operations platform GA announcement
- Read the full paper on the correlation engine that was accepted into CIKM 2024 here
