Data lake tier Ingestion for Microsoft Defender Advanced Hunting Tables is Now Generally Available
Today, we’re excited to announce the general availability (GA) of data lake tier ingestion for Microsoft XDR Advanced Hunting tables into Microsoft Sentinel data lake. Security teams continue to generate unprecedented volumes of high‑fidelity telemetry across endpoints, identities, cloud apps, and email. While this data is essential for detection, investigation, and threat hunting, it also creates new challenges around scale, cost, and long‑term retention. With this release, users can now ingest Advanced Hunting data from: Microsoft Defender for Endpoint (MDE) Microsoft Defender for Office 365 (MDO) Microsoft Defender for Cloud Apps (MDA) directly into Sentinel data lake, without requiring ingestion into the Microsoft Sentinel Analytics tier. Support for Microsoft Defender for Identity (MDI) Advanced Hunting tables will follow in the near future. Supported Tables This release enables data lake tier ingestion for Advanced Hunting data from: Defender for Endpoint (MDE) – DeviceInfo, DeviceNetworkInfo, DeviceProcessEvents, DeviceNetworkEvents, DeviceFileEvents, DeviceRegistryEvents, DeviceLogonEvents, DeviceImageLoadEvents, DeviceEvents, DeviceFileCertificateInfo Defender for Office 365 (MDO) – EmailAttachmentInfo, EmailEvents, EmailPostDeliveryEvents, EmailUrlInfo, UrlClickEvents Defender for Cloud Apps (MDA) – CloudAppEvents Each source is ingested natively into Sentinel data lake, aligning with Microsoft’s broader lake‑centric security data strategy. As mentioned above, Microsoft Defender for Identity will be available in the near future. What’s New with data lake Tier Ingestion Until now, Advanced Hunting data was primarily optimized for near‑real‑time security operations and analytics. As users extend their detection strategies to include longer retention, retrospective analysis, AI‑driven investigations, and cross‑domain correlation, the need for a lake‑first architecture becomes critical. With data lake tier ingestion, Sentinel data lake becomes a must-have destination for XDR insights, enabling users to: Store high‑volume Defender Advanced Hunting data efficiently at scale while reducing operation overhead Extend security analytics and data beyond traditional analytics lifespans for investigation, compliance, and threat research with up to 12 years of retention Query data using KQL‑based experiences across unified datasets with the KQL explorer, KQL Jobs, and Notebook Jobs Integrate data with AI-driven tooling via MCP Server for quick and interactive insights into the environment Visualize threat landscapes and relational mappings while threat hunting with custom Sentinel graphs Decouple storage and retention decisions from real‑time SIEM operations while building a more flexible and futureproof Sentinel architecture Enabling Sentinel data lake Tier Ingestion for Advanced Hunting Tables The ingestion pipeline for sending Defender Advanced Hunting data to Sentinel data lake leverages existing infrastructure and UI experiences. To enable Advanced Hunting tables for Sentinel data lake ingestion: Within the Defender Portal, expand the Microsoft Sentinel section in the left navigation. Go to Configuration > Tables. Find any of the listed tables from above and select one. Within the side menu that opens, select Data Retention Settings. Once the options open, select the button next to ‘Data lake tier’ to set the table to ingest directly into Sentinel data lake. Set the desired total retention for the data. Click save. This configuration will allow Defender data to reside within each Advanced Hunting table for 30 days while remaining accessible via custom detections and queries, while a copy of the logs is sent to Sentinel data lake for usage with custom graphs, MCP server, and benefit from the option of retention up to 12 years. Why data lake Tier Ingestion Matters Built for Scale and Cost Efficiency Advanced Hunting data is rich—and voluminous. Sentinel data lake enables users to store this data using a lake‑optimized model, designed for high‑volume ingestion and long‑term analytical workloads while making it easy to manage table tiers and usage. A Foundation for Advanced Analytics With Defender data co‑located alongside other security and cloud signals, users can unlock: Cross‑domain investigations across endpoint, identity, cloud, and email Retrospective hunting without re‑ingestion AI‑assisted analytics and large‑scale pattern detection Flexible Architecture for Modern Security Teams Data lake tier ingestion supports a layered security architecture, where: Workspaces remain optimized for real‑time detection and SOC workflows The data lake serves as the cost-effective and durable system for security telemetry Users can choose the right level of ingestion depending on operational needs, without duplicating data paths or cost. Designed to Work with Existing Sentinel and XDR Experiences This GA release builds on Microsoft Sentinel’s ongoing investment in unified data configuration and management: Native integration with Microsoft Defender XDR Advanced Hunting schemas Alignment with existing Sentinel data lake query and exploration experiences Consistent management alongside other first‑party and third‑party data sources Consistent experiences within the Defender Portal No changes are required to existing Defender deployments to begin using data lake tier ingestion. Get started To learn more about Microsoft Sentinel Data Lake and managing Defender XDR data within Sentinel, visit the Microsoft Sentinel documentation and explore how lake‑based analytics can complement your existing security operations. We look forward to seeing how users use this capability to explore new detection strategies, perform deeper investigations, and build long‑term security habits.Case Management: Incidents, Cases, and When to Use Them
In March, Case Management went to GA status within the unified portal for customers. This introduced new functionality and experiences such as: A new case queue Custom statuses New Case task experience Linking incidents to cases This can be a little confusing for existing users who are familiar with incidents and the incident experience for either Microsoft Defender or Sentinel. Let’s break this down into more detail. What are Incidents? Incidents are artifacts that act as containers for alerts to signal that a noteworthy event took place that involves one or more malicious activities. These serve to be a single landing page for alerts, activities, entities, and more. When to use Incidents? Incidents are the default experience for analysts as they perform incident investigations and response. Incidents are where they will find any and all details available for alerts and entities while performing the basic tasks of a SOC analyst. Incidents should be used when investigating and responding to malicious activity within the environment. The current incident experience provides features such as: Alert timeline Entity mapping and tracking Entity investigation graph Copilot for Security Pre-performed investigations and responses What are Cases? Cases are artifacts that represent an actionable or trackable item, such as incident investigation, validating a threat hunting hypothesis, reviewing threat intelligence review, managing endpoint vulnerabilities, and more. They can exist without alerts or incidents. When to use Cases vs. Incidents? This section is not meant to put one over the other, but is meant to clear up some confusion. Cases serve as items that can be created to track important activities within the SOC, they don’t have to just be for incident response. A case can be created for any notable activity that the SOC performs, as mentioned above. Cases can be used as a collaboration tool within your SOC team. While cases may seem redundant to incident, that is not true one bit. Here are a few distinguishing points: As incidents are a container for alerts, cases can be a container for incidents, allowing multiple incidents to be worked on at once if they are related by threat actor, impacted entities, and more. Cases offer a native task experience, similar to the experience within Microsoft Sentinel in Azure. Cases offer attachment support, allowing analysts a more traditional case management experience that incidents do not have. Cases allow for more customization, such as custom statuses. Incidents do not offer custom statuses. Let’s look at two example scenarios: Cases with Incidents I am a SOC Analyst that is reviewing the incident queue. I find an incident that involves multiple threat types and scripts. I would like to work on this incident with my colleagues while tracking notable artifacts that we find in our investigation. For example: I visit the unified incident queue and see that I have a multi-stage incident, involving multiple alerts for multiple assets. I perform my initial triage and confirm that this is a true positive that should be addressed. I will then cut a case and attach this incident to it for collaboration. Within the case, I can add a code block to list any query that I have performed within Advanced Hunting, as well as paste results from my queries directly in the case for tracking. If using Copilot for Security, I can copy and paste the Copilot incident summary in the case so that my colleagues can get an incident summary without having to leave the case. Cases without Incidents I am a SOC Analyst that is responsible for remediating device vulnerabilities. I check our current CVE’s within Exposure Management and see that I have several devices that are currently vulnerable to CVE-2025-5419, a Microsoft Edge Chromium vulnerability. I save my list of devices to a CSV file so that I can attach it to my case. I also copy the description of the CVE to add the case notes to make it more convenient for my colleagues to join the case and not need to leave it. I then pivot to Advanced Hunting to review activities by any of these vulnerable devices. I have a match and would like to connect that result to my case, so I use Export > Copy to Clipboard so that I can paste it in the case. Back within the case, I begin uploading the CSV of exposed devices as evidence, I leave a message that is formatted to draw attention to the findings, and I paste my findings based on my query. Based on my findings, I begin generating new tasks for each device owner and pasting the instructions for remediation of the CVE. These are just some examples of the many uses for cases within the Defender Portal. Hopefully this highlights the versatility of case management today and how it can operate both with and without an incident involved. Keep an eye out for more improvements as Case Management matures. If looking to learn about case management, please check out the below resources: Public documentation: Manage security operations cases natively in the Microsoft Defender portal - Unified security operations | Microsoft Learn Video based learning: https://www.youtube.com/watch?v=G-vfMJSL11g Demo: Case Management in Microsoft DefenderTable Talk: Sentinel’s New ThreatIntel Tables Explained
Key updates On April 3, 2025, we publicly previewed two new tables to support STIX (Structured Threat Information eXpression) indicator and object schemas: ThreatIntelIndicators and ThreatIntelObjects. To summarize the important dates: 31 August 2025: We previously announced that data ingestion into the legacy ThreatIntelligenceIndicator table would cease on the 31 July 2025. This timeline has now been extended and the transition to the new ThreatIntelIndicators and ThreatIntelObjects tables will proceed gradually until the 31 st of August 2025. The legacy ThreatIntelligenceIndicator table (and its data) will remain accessible, but no new data will be ingested there. Therefore, any custom content, such as workbooks, queries, or analytic rules, must be updated to reference the new tables to remain effective. If you require additional time to complete the transition, you may opt into dual ingestion, available until the official retirement on the 21 st of May 2026, by submitting a service request. Update: The opt in to dual ingestion ended on the 31 st of August and is no longer available. 31 May 2026: ThreatIntelligenceIndicator table support will officially retire, along with ingestion for those who opt-in to dual ingestion beyond 31 st of August 2025. What’s changing: ThreatIntelligenceIndicator VS ThreatIntelIndicators and ThreatIntelObjects Let’s summarise some of the differences. ThreatIntelligenceIndicator ThreatIntelIndicators ThreatIntelObjects Status Extended data ingestion until the 31st of August 2025, opt-in for additional transition time available. Deprecating on the 31st of May 2026 — no new data will be ingested after this date. Active and recommended for use. Active and complementary to ThreatIntelIndicators. Purpose Originally used to store threat indicators like IPs, domains, file hashes, etc. Stores individual threat indicators (e.g. IPs, URLs, file hashes). Stores STIX objects that provide contextual information about indicators. Examples: threat actors, malware families, campaigns, attack patterns. Characteristics Limitations: o Less flexible schema. o Limited support for STIX (Structured Threat Information eXpression) objects. o Fewer contextual fields for advanced threat hunting. Enhancements: o Supports STIX indicator schema. o Includes a Data column with full STIX object data for advanced hunting. o More metadata fields (e.g. LastUpdateMethod, IsDeleted, ExpirationDateTime). o Optimized ingestion: excludes empty key-value pairs and truncates long fields over 1,000 characters. Enhancements: o Enables richer threat modelling and correlation. o Includes fields like StixType, Data.name, and Data.id. Use cases Legacy structure for storing threat indicators. Migration Note: All custom queries, workbooks, and analytics rules referencing this table must be updated to use the new tables . Ideal for identifying and correlating specific threat indicators. Threat Hunting: Enables hunting for specific Indicators of Compromise (IOCs) such as IP addresses, domains, URLs, and file hashes. Alerting and detection rules: Can be used in KQL queries to match against telemetry from other tables (e.g. Heartbeat, SecurityEvent, Syslog). Example query correlating threat indictors with threat actors: Identify threat actors associated with specific threat indicators Useful for understanding relationships between indicators and broader threat entities (e.g. linking an IP to a known threat actor). Threat Hunting: Adds context by linking indicators to threat actors, malware families, campaigns, and attack patterns. Alerting and Detection rules: Enrich alerts with context like threat actor names or malware types. Example query listing TI objects related to a threat actor, “Sangria Tempest.” : List threat intelligence data related to a specific threat actor Benefits of the new ThreatIntelIndicators and ThreatIntelObjects tables In addition to what’s mentioned in the table above. The main benefits of the new table include: Enhanced Threat Visibility More granular and complete representation of threat intelligence. Support for advanced hunting scenarios and complex queries. Enables attribution to threat actors and relationships. Improved Hunting Capabilities Generic parsing of STIX patterns. Support for all valid STIX IoCs, Threat Actors, Identity, and Relationships. Important considerations with the new TI tables Higher volume of data being ingested: o In the legacy ThreatIntelligenceIndicator table, only the IoCs with Domain, File, URL, Email, Network sources were ingested. o The new tables support a richer schema and more detailed data, which naturally increases ingestion volume. The Data column in both tables stores full STIX objects, which are often large and complex. o Additional metadata fields (e.g. LastUpdateMethod, StixType, ObservableKey, etc.) increase the size of each record. o Some fields like description and pattern are truncated if they exceed 1,000 characters, indicating the potential for large payloads. More Frequent Republishing: o Previously, threat intelligence data was republished over a 12-day cycle. Now, all data is republished every 7-10 days (depending on the volume), increasing the ingestion frequency and volume. o This change ensures fresher data but also leads to more frequent ingestion events. o Republishing is identifiable by LastUpdateMethod = "LogARepublisher" in the tables. Optimising data ingestion There are two mechanisms to optimise threat intelligence data ingestion and control costs. Ingestion Rules See ingestion rules in action: Introducing Threat Intelligence Ingestion Rules | Microsoft Community Hub Sentinel supports Ingestion Rules that allow organizations to curate data before it enters the system. In addition, it enables: Bulk tagging, expiration extensions, and confidence-based filtering, which may increase ingestion if more indicators are retained or extended. Custom workflows that may result in additional ingestion events (e.g. tagging or relationship creation). Reduce noise by filtering out irrelevant TI Objects such as low confidence indicators (e.g. drop IoCs with a confidence score of 0), suppressing known false positives from specific feeds. These rules act on TI objects before they are ingested into Sentinel, giving you control over what gets stored and analysed. Data Collection Rules/ Data transformation As mentioned above, the ThreatIntelIndicator and ThreatIntelObjects tables include a “Data” column which contains the full original STIX object and may or may not be relevant for your use cases. In this case, you can use a workspace transformation DCR to filter it out using a KQL query. An example of this KQL query is shown below, for more examples about using workspace transformations and data collection rules: Data collection rules in Azure Monitor - Azure Monitor | Microsoft Learn source | project-away Data A few things to note: o Your threat intelligence feeds will be sending the additional STIX objects data and IoCs, if you prefer not to receive these additional TI data, you can modify the filter out data according to your use cases as mentioned above. More examples are mentioned here: Work with STIX objects and indicators to enhance threat intelligence and threat hunting in Microsoft Sentinel (Preview) - Microsoft Sentinel | Microsoft Learn o If you are using a data collection rule to make schema changes such as dropping the fields, please make sure to modify the relevant Sentinel content (e.g. detection rules, Workbooks, hunting queries, etc.) that are using the tables. o There can be additional cost when using Azure Monitor data transformations (such as when adding extra columns or adding enrichments to incoming data), however, if Sentinel is enabled on the Log Analytics workspace, there is no filtering ingestion charge regardless of how much data the transformation filters. New Threat Intelligence solution pack available A new Threat Intelligence solution is now available in the Content Hub, providing out of the box content referencing the new TI tables, including 51 detection rules, 5 hunting queries, 1 Workbook, 5 data connectors and also includes 1 parser for the ThreatIntelIndicators. Please note, the previous Threat Intelligence solution pack will be deprecated and removed after the transition phase. We recommend downloading the new solution from the Content Hub as shown below: Conclusion The transition to the new ThreatIntelIndicators and ThreatIntelObjects tables provide enhanced support for STIX schemas, improved hunting and alerting features, and greater control over data ingestion allowing organizations to get deeper visibility and more effective threat detection. To ensure continuity and maximize value, it's essential to update existing content and adopt the new Threat Intelligence solution pack available in the Content Hub. Related content and references: Work with STIX objects and indicators to enhance threat intelligence and threat hunting in Microsoft Sentinel Curate Threat Intelligence using Ingestion Rules Announcing Public Preview: New STIX Objects in Microsoft SentinelLevel Up Your Security Skills with the New Microsoft Sentinel Ninja Training!
If you’ve explored our Microsoft Sentinel Ninja Training in the past, it’s time to revisit! Our training program has undergone some exciting changes to keep you ahead of the curve in the ever-evolving cybersecurity landscape. Microsoft Sentinel is a cutting-edge, cloud-native SIEM and SOAR solution designed to help security professionals protect their organizations from today’s complex threats. Our Ninja Training program is here to guide you through every aspect of this powerful tool. So, what’s new? In addition to the structured security roles format, the Ninja Training now offers a more interactive experience with updated modules, hands-on labs, and real-world scenarios. Whether you're focusing on threat detection, incident response, or automation, the training ensures you gain the practical skills needed to optimize your security operations. One of the biggest updates is the integration of Sentinel into the Defender XDR portal, creating a unified security platform. This merger simplifies workflows, speeds up incident response, and minimizes tool-switching, allowing for seamless operations. Other highlights include: Step-by-step guidance through the official Microsoft Sentinel documentation. Exclusive webinars and up-to-date blog posts from Microsoft experts. If you're ready to take your Sentinel skills to the next level or want to revisit the program’s new features, head over to the blog now and dive into the refreshed Microsoft Sentinel Ninja Training! Don’t miss out—your next cybersecurity breakthrough is just a click away!A Look at Different Options for Storing and Searching Sentinel Archived Logs
As an Azure Sentinel user, you know the importance of having a secure and accessible backup of your log data. In this blog, we'll show you the various options available for storing and searching Sentinel logs beyond the default 90-day retention period. Explore the features and benefits of each solution to find the best fit for your organization.
