Detect more, spend less: the future of threat intelligence correlation

With more data and intelligence than ever, it’s often a challenge to manage it all while making sure you’re maximizing its value for security investigations. We’ve made it easier for customers leveraging Microsoft’s SIEM and XDR. Now, customers can create custom detections that correlate threat intelligence from feeds brought in through the SIEM with their XDR data, without the need to ingest their XDR data as well.

Why this matters

Traditionally, correlating threat intelligence with endpoint and identity data required ingesting large volumes of XDR data into the SIEM. While effective, this approach often drove up ingestion and retention costs. The new capability eliminates that dependency, allowing security teams to:

• Reduce costs – Avoid unnecessary data ingestion charges while still leveraging XDR insights for detection.
• Accelerate detection – Query XDR and SIEM data seamlessly in near real time, enabling faster identification of threats.
• Maintain flexibility – Use custom detection rules to tailor alerts to your organization’s unique threat landscape.

How it works

• Threat intelligence integration – Use curated threat indicators from Microsoft or your own threat intelligence platform (TIP) to power detections.
• Build custom detection rules that query both Sentinel and Defender XDR tables directly. This means you can match threat intelligence indicators—such as malicious IPs, domains, or file hashes (including from third party/non-Microsoft IOCs)—against Defender XDR telemetry without duplicating data in Sentinel. These rules can run on a schedule or in near real time, ensuring timely detection of suspicious activity.

Examples of KQL Queries that can be used with Custom Detections, providing the following capabilities:

• Query the new TI tables (ThreatIntelIndicators and ThreatIntelObjects).
• Correlate with Defender XDR data without ingestion into Sentinel using Custom Detection rules.
• Enrich detections with threat actor context for better triage.

Example 1: This query identifies any Domain indicators of compromise (IOCs) from threat intelligence (TI) by searching for matches in DeviceNetworkEvents.

let dt_lookBack = 1h; // device events time window (how far back to look at traffic)
let ioc_lookBack = 14d; // TI time window (how far back to read indicators)
let DeviceNetworkEvents_ = DeviceNetworkEvents
  | where isnotempty(RemoteUrl)
  | where Timestamp >= ago(dt_lookBack)
  | where ActionType !has "ConnectionFailed"
  | extend Domain = tostring(parse_url(tolower(RemoteUrl)).Host)
  | where isnotempty(Domain)
  | project-rename DeviceNetworkEvents_TimeGenerated = Timestamp;
let DeviceNetworkEventDomains = DeviceNetworkEvents_
  | distinct Domain
  | summarize make_list(Domain);
ThreatIntelIndicators
| extend IndicatorType = replace(@"\[|\]|\""", "", tostring(split(ObservableKey, ":", 0)))
| where IndicatorType == "domain-name"
| extend DomainName = tolower(ObservableValue)
| where TimeGenerated >= ago(ioc_lookBack)
| extend IndicatorId = tostring(split(Id, "--")[2])
| where DomainName in (DeviceNetworkEventDomains)
| summarize LatestIndicatorTime = arg_max(TimeGenerated, *) by Id, ObservableValue
| where IsActive and (ValidUntil > now() or isempty(ValidUntil))
| extend Description = tostring(parse_json(Data).description)
| extend TrafficLightProtocolLevel = tostring(parse_json(AdditionalFields).TLPLevel)
| where Description !contains_cs "State: inactive;" and Description !contains_cs "State: falsepos;"
| project-reorder *, IsActive, Tags, TrafficLightProtocolLevel, DomainName, Type
| join kind=innerunique (DeviceNetworkEvents_) on $left.DomainName == $right.Domain
| where DeviceNetworkEvents_TimeGenerated < ValidUntil
| summarize DeviceNetworkEvents_TimeGenerated = arg_max(DeviceNetworkEvents_TimeGenerated, *) by IndicatorId
| project DeviceNetworkEvents_TimeGenerated, IndicatorId, Url = RemoteUrl, Confidence, Description, Tags, TrafficLightProtocolLevel, ActionType, DeviceId, DeviceName, InitiatingProcessAccountUpn, InitiatingProcessCommandLine, RemoteIP, RemotePort, ReportId
| extend Name = tostring(split(InitiatingProcessAccountUpn, '@', 0)[0]), UPNSuffix = tostring(split(InitiatingProcessAccountUpn, '@', 1)[0])
| extend Timestamp = DeviceNetworkEvents_TimeGenerated, UserPrincipalName = InitiatingProcessAccountUpn

Example 2: Detect Malicious File Hashes Using ThreatIntelIndicators. Identifies a match in DeviceFileEvents event data from any FileHash IOC from TI

let dt_lookBack = 1h; // device events time window (how far back to look at traffic)
let ioc_lookBack = 14d; // TI time window (how far back to read indicators)
let DeviceFileEvents_ = (union
(DeviceFileEvents | where Timestamp > ago(dt_lookBack) | where isnotempty(SHA1) | extend FileHashValue = SHA1),
(DeviceFileEvents | where Timestamp > ago(dt_lookBack) | where isnotempty(SHA256) | extend FileHashValue = SHA256));
let Hashes = DeviceFileEvents_ | distinct FileHashValue;
ThreatIntelIndicators
//extract key part of kv pair
      | extend IndicatorType = replace(@"\[|\]|\""", "", tostring(split(ObservableKey, ":", 0)))
      | where IndicatorType == "file"
      | extend FileHashType = replace("'", "", substring(ObservableKey, indexof(ObservableKey, "hashes.") + 7, strlen(ObservableKey) - indexof(ObservableKey, "hashes.") - 7))
      | extend FileHashValue = ObservableValue
      | extend IndicatorId = tostring(split(Id, "--")[2])
| where isnotempty(FileHashValue)
| where TimeGenerated > ago(ioc_lookBack)
//  | where FileHashValue in (Hashes)
| summarize LatestIndicatorTime = arg_max(TimeGenerated, *) by Id, ObservableValue
| where IsActive and (ValidUntil > now() or isempty(ValidUntil))
| extend Description = tostring(parse_json(Data).description)
| where Description !contains_cs "State: inactive;" and Description !contains_cs "State: falsepos;"
  | project-reorder *, FileHashType, FileHashValue, Type
| join kind=innerunique (DeviceFileEvents_) on $left.FileHashValue == $right.FileHashValue
| where TimeGenerated < ValidUntil
| summarize TimeGenerated = arg_max(Timestamp, *) by  IndicatorId, DeviceId
| extend TrafficLightProtocolLevel = tostring(parse_json(AdditionalFields).TLPLevel)
| extend Description = tostring(parse_json(Data).description)
| extend ActivityGroupNames = extract(@"ActivityGroup:(\S+)", 1, tostring(parse_json(Data).labels))
| project TimeGenerated, TrafficLightProtocolLevel, Description, ActivityGroupNames, IndicatorId, FileHashValue, FileHashType, ValidUntil, Confidence, ActionType, DeviceId, DeviceName, FolderPath, RequestAccountDomain, RequestAccountName, RequestAccountSid, MachineGroup, ReportId
| extend Timestamp = TimeGenerated

Important notes

• To create these types of Custom Detections, some columns like Timestamp and ReportId are required in the query results, for more information: Create custom detection rules in Microsoft Defender XDR - Microsoft Defender XDR | Microsoft Learn
• Use asset mappings and entity mappings. Prioritise mapping for high-value entities:
  • User accounts (for credential-based attacks)
  • Devices/hosts (for lateral movement)
  • IP addresses and domains (for network-based threats)
  • Files and processes (for malware execution)

Key benefits

• Cost optimization:
• No need to ingest Defender XDR data into Sentinel to correlate with threat intelligence.

  • Reduces data ingestion and retention costs significantly while maintaining full detection capability.

• Extended lookback: Analyse historical data up to 30 days without additional storage costs.
• Enhanced threat context: Leverage ThreatIntelIndicators and ThreatIntelObject tables to enrich alerts with threat actor details, confidence scores, and campaign context.
• Flexible and customizable detection logic:
  • Build custom KQL-based rules tailored to your organization’s threat landscape.
  • Combine multiple data sources (including third party/non-Microsoft sources) and enrich alerts with contextual threat intelligence.
    
    
• Faster, proactive threat detection:
  • Detect threats without waiting for data ingestion pipelines.
  • Supports scheduled or near real-time queries, improving response times.

Key takeaway

Security teams can maximize the value of threat intelligence while optimizing costs. By reducing data duplication and enabling advanced correlation, organizations can strengthen their security posture without compromising efficiency.

Useful links

• Custom detection rules get a boost—explore what’s new in Microsoft Defender | Microsoft Community Hub
• Create custom detection rules in Microsoft Defender XDR - Microsoft Defender XDR | Microsoft Learn
• Threat intelligence - Microsoft Sentinel | Microsoft Learn