Accelerate Agent Development: Hacks for Building with Microsoft Sentinel data lake
As a Senior Product Manager | Developer Architect on the App Assure team working to bring Microsoft Sentinel and Security Copilot solutions to market, I interact with many ISVs building agents on Microsoft Sentinel data lake for the first time. I’ve written this article to walk you through one possible approach for agent development – the process I use when building sample agents internally at Microsoft. If you have questions about this, or other methods for building your agent, App Assure offers guidance through our Sentinel Advisory Service. Throughout this post, I include screenshots and examples from Gigamon’s Security Posture Insight Agent. This article assumes you have: An existing SaaS or security product with accessible telemetry. A small ISV team (2–3 engineers + 1 PM). Focus on a single high value scenario for the first agent. The Composite Application Model (What You Are Building) When I begin designing an agent, I think end-to-end, from data ingestion requirements through agentic logic, following the Composite application model. The Composite Application Model consists of five layers: Data Sources – Your product’s raw security, audit, or operational data. Ingestion – Getting that data into Microsoft Sentinel. Sentinel data lake & Microsoft Graph – Normalization, storage, and correlation. Agent – Reasoning logic that queries data and produces outcomes. End User – Security Copilot or SaaS experiences that invoke the agent. This separation allows for evolving data ingestion and agent logic simultaneously. It also helps avoid downstream surprises that require going back and rearchitecting the entire solution. Optional Prerequisite You are enrolled in the ISV Success Program, so you can earn Azure Credits to provision Security Compute Units (SCUs) for Security Copilot Agents. Phase 1: Data Ingestion Design & Implementation Choose Your Ingestion Strategy The first choice I face when designing an agent is how the data is going to flow into my Sentinel workspace. Below I document two primary methods for ingestion. Option A: Codeless Connector Framework (CCF) This is the best option for ISVs with REST APIs. To build a CCF solution, reference our documentation for getting started. Option B: CCF Push (Public Preview) In this instance, an ISV pushes events directly to Sentinel via a CCF Push connector. Our MS Learn documentation is a great place to get started using this method. Additional Note: In the event you find that CCF does not support your needs, reach out to App Assure so we can capture your requirements for future consideration. Azure Functions remains an option if you’ve documented your CCF feature needs. Phase 2: Onboard to Microsoft Sentinel data lake Once my data is flowing into Sentinel, I onboard a single Sentinel workspace to data lake. This is a one-time action and cannot be repeated for additional workspaces. Onboarding Steps Go to the Defender portal. Follow the Sentinel Data lake onboarding instructions. Validate that tables are visible in the lake. See Running KQL Queries in data lake for additional information. Phase 3: Build and Test the Agent in Microsoft Foundry Once my data is successfully ingested into data lake, I begin the agent development process. There are multiple ways to build agents depending on your needs and tooling preferences. For this example, I chose Microsoft Foundry because it fit my needs for real-time logging, cost efficiency, and greater control. 1. Create a Microsoft Foundry Instance Foundry is used as a tool for your development environment. Reference our QuickStart guide for setting up your Foundry instance. Required Permissions: Security Reader (Entra or Subscription) Azure AI Developer at the resource group After setup, click Create Agent. 2. Design the Agent A strong first agent: Solves one narrow security problem. Has deterministic outputs. Uses explicit instructions, not vague prompts. Example agent responsibilities: To query Sentinel data lake (Sentinel data exploration tool). To summarize recent incidents. To correlate ISVs specific signals with Sentinel alerts and other ISV tables (Sentinel data exploration tool). 3. Implement Agent Instructions Well-designed agent instructions should include: Role definition ("You are a security investigation agent…"). Data sources it can access. Step by step reasoning rules. Output format expectations. Sample Instructions can be found here: Agent Instructions 4. Configure the Microsoft Model Context Protocol (MCP) tooling for your agent For your agent to query, summarize and correlate all the data your connector has sent to data lake, take the following steps: Select Tools, and under Catalog, type Sentinel, and then select Microsoft Sentinel Data Exploration. For more information about the data exploration tool collection in MCP server, see our documentation. I always test repeatedly with real data until outputs are consistent. For more information on testing and validating the agent, please reference our documentation. Phase 4: Migrate the Agent to Security Copilot Once the agent works in Foundry, I migrate it to Security Copilot. To do this: Copy the full instruction set from Foundry Provision a SCU for your Security Copilot workspace. For instructions, please reference this documentation. Make note of this process as you will be charged per hour per SCU Once you are done testing you will need to deprovision the capacity to prevent additional charges Open Security Copilot and use Create From Scratch Agent Builder as outlined here. Add Sentinel data exploration MCP tools (these are the same instructions from the Foundry agent in the previous step). For more information on linking the Sentinel MCP tools, please refer to this article. Paste and adapt instructions. At this stage, I always validate the following: Agent Permissions – I have confirmed the agent has the necessary permissions to interact with the MCP tool and read data from your data lake instance. Agent Performance – I have confirmed a successful interaction with measured latency and benchmark results. This step intentionally avoids reimplementation. I am reusing proven logic. Phase 5: Execute, Validate, and Publish After setting up my agent, I navigate to the Agents tab to manually trigger the agent. For more information on testing an agent you can refer to this article. Now that the agent has been executed successfully, I download the agent Manifest file from the environment so that it can be packaged. Click View code on the Agent under the Build tab as outlined in this documentation. Publishing to the Microsoft Security Store If I were publishing my agent to the Microsoft Security Store, these are the steps I would follow: Finalize ingestion reliability. Document required permissions. Define supported scenarios clearly. Package agent instructions and guidance (by following these instructions). Summary Based on my experience developing Security Copilot agents on Microsoft Sentinel data lake, this playbook provides a practical, repeatable framework for ISVs to accelerate their agent development and delivery while maintaining high standards of quality. This foundation enables rapid iteration—future agents can often be built in days, not weeks, by reusing the same ingestion and data lake setup. When starting on your own agent development journey, keep the following in mind: To limit initial scope. To reuse Microsoft managed infrastructure. To separate ingestion from intelligence. What Success Looks Like At the end of this development process, you will have the following: A Microsoft Sentinel data connector live in Content Hub (or in process) that provides a data ingestion path. Data visible in data lake. A tested agent running in Security Copilot. Clear documentation for customers. A key success factor I look for is clarity over completeness. A focused agent is far more likely to be adopted. Need help? If you have any issues as you work to develop your agent, please reach out to the App Assure team for support via our Sentinel Advisory Service . Or if you have any other tips, please comment below, I’d love to hear your feedback.
Do XDR Alerts cover the same alerts available in Alert Policies?
The alerts in question are the 'User requested to release a quarantined message', 'User clicked a malicious link', etc. About 8 of these we send to 'email address removed for privacy reasons'. That administrator account has an EOM license, so Outlook rules can be set. We set rules to forward those 8 alerts to our 'email address removed for privacy reasons' address. This is, very specifically, so the alert passes through the @tenant.com address, and our ticketing endpoint knows what tenant sent it. But this ISN'T ideal because it requires an EOP license (or similar - this actually hasn't been an issue until now just because of our customer environments). I've looked at the following alternatives: - Setting email address removed for privacy reasons as the recipient directly on the Alert Policies in question. This results in the mail going directly from microsoft to our Ticketing Portal - so it ends up sorted into Microsoft tickets. and the right team doesn't get it. SMTP Forwarding via either Exchange AC User controls or Mail Flow Rules. But these aren't traditional forwarding, and they have the same issue as above. Making administrator @tenant.com a SHARED mailbox that we can also login to (for administration purposes). But this doesn't allow you to set Outlook rules (or even login to Outlook). I've checked out the newer alerts under Defender's Settings panel - XDR alerts, I think they're called. Wondering if these can be leveraged at all for this? Essentially, trying to get these Alerts to come to our external ticketing address, from the tenants domain (instead of Microsoft). I could probably update Autotask's rules to check for a header, and set that header via Mail Flow rules, but.. just hoping I don't have to do that for everyone.Impersonation Protection: Users to Protect should also be Trusted Senders
Hey all, sort of a weird question here. Teaching my staff about Impersonation Protection, and it's kind of occurred to me that any external sender added to 'Senders to Protect' sort of implicitly should also be a 'Trusted Sender'. Example - we're an MSP, and we want our Help Desk (email address removed for privacy reasons) to be protected from impersonation. Specifically, we want to protect the 'Help Desk' name. So we add email address removed for privacy reasons to Senders to protect. However, we ALSO want to make sure our emails come thru. So we've ALSO had to add email address removed for privacy reasons to Trusted Senders on other tenants. Chats with Copilot have sort of given me an understanding that this is essentially a 'which is more usefuI' scenario. But CoPilot makes things up, and I want some human input. In theory, ANYONE we add to 'trusted senders' we ALSO want protected from Impersonation. Anyone we protect from Impersonation we ALSO want to trust. Copilot says you SHOULDN'T do both. Which is better / more practical?RSAC 2026: New Microsoft Sentinel Connectors Announcement
Microsoft Sentinel helps organizations detect, investigate, and respond to security threats across increasingly complex environments. With the rollout of the Microsoft Sentinel data lake in the fall, and the App Assure-backed Sentinel promise that supports it, customers now have access to long-term, cost-effective storage for security telemetry, creating a solid foundation for emerging Agentic AI experiences. Since our last announcement at Ignite 2025, the Microsoft Sentinel connector ecosystem has expanded rapidly, reflecting continued investment from software development partners building for our shared customers. These connectors bring diverse security signals together, enabling correlation at scale and delivering richer investigation context across the Sentinel platform. Below is a snapshot of Microsoft Sentinel connectors newly available or recently enhanced since our last announcement, highlighting the breadth of partner solutions contributing data into, and extending the value of, the Microsoft Sentinel ecosystem. New and notable integrations Acronis Cyber Protect Cloud Acronis Cyber Protect Cloud integrates with Microsoft Sentinel to bring data protection and security telemetry into a centralized SOC view. The connector streams alerts, events, and activity data - spanning backup, endpoint protection, and workload security - into Microsoft Sentinel for correlation with other signals. This integration helps security teams investigate ransomware and data-centric threats more effectively, leverage built-in hunting queries and detection rules, and improve visibility across managed environments without adding operational complexity. Anvilogic Anvilogic integrates with Microsoft Sentinel to help security teams operationalize detection engineering at scale. The connector streams Anvilogic alerts into Microsoft Sentinel, giving SOC analysts centralized visibility into high-fidelity detections and faster context for investigation and triage. By unifying detection workflows, reducing alert noise, and improving prioritization, this integration supports more efficient threat detection and response while helping teams extend coverage across evolving attack techniques. CyberArk Audit CyberArk Audit integrates with Microsoft Sentinel to centralize visibility into privileged identity and access activity. By streaming detailed audit logs - covering system events, user actions, and administrative activity - into Microsoft Sentinel, security teams can correlate identity-driven risks with broader security telemetry. This integration supports faster investigations, improved monitoring of privileged access, and more effective incident response through automated workflows and enriched context for SOC analysts. Cyera Cyera integrates with Microsoft Sentinel to extend AI-native data security posture management into security operations. The connector brings Cyera’s data context and actionable intelligence across multi-cloud, on-premises, and SaaS environments into Microsoft Sentinel, helping teams understand where sensitive data resides and how it is accessed, exposed, and used. Built on Sentinel’s modern framework, the integration feeds context-rich data risk signals into the Sentinel data lake, enabling more informed threat hunting, automation, and decision-making around data, user, and AI-related risk. TacitRed CrowdStrike IOC Automation Data443 TacitRed CS IOC Automation integrates with Microsoft Sentinel to streamline the operationalization of compromised credential intelligence. The solution uses Sentinel playbooks to automatically push TacitRed indicators of compromise into CrowdStrike via Sentinel playbooks, helping security teams turn identity-based threat intelligence into action. By automating IOC handling and reducing manual effort, this integration supports faster response to credential exposure and strengthens protection against account-driven attacks across the environment. TacitRed SentinelOne IOC Automation Data443 TacitRed SentinelOne IOC Automation integrates with Microsoft Sentinel to help operationalize identity-focused threat intelligence at the endpoint layer. The solution uses Sentinel playbooks to automatically consume TacitRed indicators and push curated indicators into SentinelOne via Sentinel playbooks and API-based enforcement, enabling faster enforcement of high-risk IOCs without manual handling. By automating the flow of compromised credential intelligence from Sentinel into EDR, this integration supports quicker response to identity-driven attacks and improves coordination between threat intelligence and endpoint protection workflows. TacitRed Threat Intelligence Data443 TacitRed Threat Intelligence integrates with Microsoft Sentinel to provide enhanced visibility into identity-based risks, including compromised credentials and high-risk user exposure. The solution ingests curated TacitRed intelligence directly into Sentinel, enriching incidents with context that helps SOC teams identify credential-driven threats earlier in the attack lifecycle. With built-in analytics, workbooks, and hunting queries, this integration supports proactive identity threat detection, faster triage, and more informed response across the SOC. Cyren Threat Intelligence Cyren Threat Intelligence integrates with Microsoft Sentinel to enhance detection of network-based threats using curated IP reputation and malware URL intelligence. The connector ingests Cyren threat feeds into Sentinel using the Codeless Connector Framework (CCF), transforming raw indicators into actionable insights, dashboards, and enriched investigations. By adding context to suspicious traffic and phishing infrastructure, this integration helps SOC teams improve alert accuracy, accelerate triage, and make more confident response decisions across their environments. TacitRed Defender Threat Intelligence Data443 TacitRed Defender Threat Intelligence integrates with Microsoft Sentinel to surface early indicators of credential exposure and identity-driven risk. The solution automatically ingests compromised credential intelligence from TacitRed into Sentinel and can support synchronization of validated indicators with Microsoft Defender Threat Intelligence through Sentinel workflows, helping SOC teams detect account compromise before abuse occurs. By enriching Sentinel incidents with actionable identity context, this integration supports faster triage, proactive remediation, and stronger protection against credential-based attacks. Datawiza Access Proxy (DAP) Datawiza Access Proxy integrates with Microsoft Sentinel to provide centralized visibility into application access and authentication activity. By streaming access and MFA logs from Datawiza into Sentinel, security teams can correlate identity and session-level events with broader security telemetry. This integration supports detection of anomalous access patterns, faster investigation through session traceability, and more effective response using Sentinel automation, helping organizations strengthen Zero Trust controls and meet auditing and compliance requirements. Endace Endace integrates with Microsoft Sentinel to provide deep network visibility by providing always-on, packet-level evidence. The connector enables one-click pivoting from Sentinel alerts directly to recorded packet data captured by EndaceProbes. This helps SOC and NetOps teams reconstruct events and validate threats with confidence. By combining Sentinel’s AI-driven analytics with Endace’s always-on, full-packet capture across on-premises, hybrid, and cloud environments, this integration supports faster investigations, improved forensic accuracy, and more decisive incident response. Feedly Feedly integrates with Microsoft Sentinel to ingest curated threat intelligence directly into security operations workflows. The connector automatically imports Indicators of Compromise (IoCs) from Feedly Team Boards and folders into Sentinel, enriching detections and investigations with context from the original intelligence articles. By bringing analyst‑curated threat intelligence into Sentinel in a structured, automated way, this integration helps security teams stay current on emerging threats and reduce the manual effort required to operationalize external intelligence. Gigamon Gigamon integrates with Microsoft Sentinel through a new connector that provides access to Gigamon Application Metadata Intelligence (AMI), delivering high-fidelity network-derived telemetry with rich application metadata from inspected traffic directly into Sentinel. This added context helps security teams detect suspicious activity, encrypted threats, and lateral movement faster and with greater precision. By enriching analytics without requiring full packet ingestion, organizations can reduce noise, manage SIEM costs, and extend visibility across hybrid cloud infrastructure. Halcyon Halcyon integrates with Microsoft Sentinel to provide purpose-built ransomware detection and automated containment across the Microsoft security ecosystem. The connector surfaces Halcyon ransomware alerts directly within Sentinel, enabling SOC teams to correlate ransomware behavior with Microsoft Defender and broader Microsoft telemetry. By supporting Sentinel analytics and automation workflows, this integration helps organizations detect ransomware earlier, investigate faster using native Sentinel tools, and isolate affected endpoints to prevent lateral spread and reinfection. Illumio The Illumio platform identifies and contains threats across hybrid multi-cloud environments. By integrating AI-driven insights with Microsoft Sentinel and Microsoft Graph, Illumio Insights enables SOC analysts to visualize attack paths, prioritize high-risk activity, and investigate threats with greater precision. Illumio Segmentation secures critical assets, workloads, and devices and then publishes segmentation policy back into Microsoft Sentinel to ensure compliance monitoring. Joe Sandbox Joe Sandbox integrates with Microsoft Sentinel to enrich incidents with dynamic malware and URL analysis. The connector ingests Joe Sandbox threat intelligence and automatically detonates suspicious files and URLs associated with Sentinel incidents, returning behavioral and contextual analysis results directly into investigation workflows. By adding sandbox-driven insights to indicators, alerts, and incident comments, this integration helps SOC teams validate threats faster, reduce false positives, and improve response decisions using deeper visibility into malicious behavior. Keeper Security The Keeper Security integration with Microsoft Sentinel brings advanced password and secrets management telemetry into your SIEM environment. By streaming audit logs and privileged access events from Keeper into Sentinel, security teams gain centralized visibility into credential usage and potential misuse. The connector supports custom queries and automated playbooks, helping organizations accelerate investigations, enforce Zero Trust principles, and strengthen identity security across hybrid environments. Lookout Mobile Threat Defense (MTD) Lookout Mobile Threat Defense integrates with Microsoft Sentinel to extend SOC visibility to mobile endpoints across Android, iOS, and Chrome OS. The connector streams device, threat, and audit telemetry from Lookout into Sentinel, enabling security teams to correlate mobile risk signals such as phishing, malicious apps, and device compromise, with broader enterprise security data. By incorporating mobile threat intelligence into Sentinel analytics, dashboards, and alerts, this integration helps organizations detect mobile driven attacks earlier and strengthen protection for an increasingly mobile workforce. Miro Miro integrates with Microsoft Sentinel to provide centralized visibility into collaboration activity across Miro workspaces. The connector ingests organization-wide audit logs and content activity logs into Sentinel, enabling security teams to monitor authentication events, administrative actions, and content changes alongside other enterprise signals. By bringing Miro collaboration telemetry into Sentinel analytics and dashboards, this integration helps organizations detect suspicious access patterns, support compliance and eDiscovery needs, and maintain stronger oversight of collaborative environments without disrupting productivity. Obsidian Activity Threat The Obsidian Threat and Activity Feed for Microsoft Sentinel delivers deep visibility into SaaS and AI applications, helping security teams detect account compromise and insider threats. By streaming user behavior and configuration data into Sentinel, organizations can correlate application risks with enterprise telemetry for faster investigations. Prebuilt analytics and dashboards enable proactive monitoring, while automated playbooks simplify response workflows, strengthening security posture across critical cloud apps. OneTrust for Purview DSPM OneTrust integrates with Microsoft Sentinel to bring privacy, compliance, and data governance signals into security operations workflows. The connector enriches Sentinel with privacy relevant events and risk indicators from OneTrust, helping organizations detect sensitive data exposure, oversharing, and compliance risks across cloud and non-Microsoft data sources. By unifying privacy intelligence with Sentinel analytics and automation, this integration enables security and privacy teams to respond more quickly to data risk events and support responsible data use and AI-ready governance. Pathlock Pathlock integrates with Microsoft Sentinel to bring SAP-specific threat detection and response signals into centralized security operations. The connector forwards security-relevant SAP events into Sentinel, enabling SOC teams to correlate SAP activity with broader enterprise telemetry and investigate threats using familiar SIEM workflows. By enriching Sentinel with SAP security context and focused detection logic, this integration helps organizations improve visibility into SAP landscapes, reduce noise, and accelerate detection and response for risks affecting critical business systems. Quokka Q-scout Quokka Q-scout integrates with Microsoft Sentinel to centralize mobile application risk intelligence across Microsoft Intune-managed devices. The connector automatically ingests app inventories from Intune, analyzes them using Quokka’s mobile app vetting engines, and streams security, privacy, and compliance risk findings into Sentinel. By surfacing app-level risks through Sentinel analytics and alerts, this integration helps security teams identify malicious or high-risk mobile apps, prioritize remediation, and strengthen mobile security posture without deploying agents or disrupting users. Synqly Synqly integrates with Microsoft Sentinel to simplify and scale security integrations through a unified API approach. The connector enables organizations and security vendors to establish a bi‑directional connection with Sentinel without relying on brittle, point‑to‑point integrations. By abstracting common integration challenges such as authentication handling, retries, and schema changes, Synqly helps teams orchestrate security data flows into and out of Sentinel more reliably, supporting faster onboarding of new data sources and more maintainable integrations at scale. Versasec vSEC:CMS Versasec vSEC:CMS integrates with Microsoft Sentinel to provide centralized visibility into credential lifecycle and system health events. The connector securely streams vSEC:CMS and vSEC:CLOUD alerts and status data into Sentinel using the Codeless Connector Framework (CCF), transforming credential management activity into correlation-ready security signals. By bringing smart card, token, and passkey management telemetry into Sentinel, this integration helps security teams monitor authentication infrastructure health, investigate credential-related incidents, and unify identity security operations within their SIEM workflows. VirtualMetric DataStream VirtualMetric DataStream integrates with Microsoft Sentinel to optimize how security telemetry is collected, normalized, and routed across the Microsoft security ecosystem. Acting as a high-performance telemetry pipeline, DataStream intelligently filters and enriches logs, sending high-value security data to Sentinel while routing less-critical data to Sentinel data lake or Azure Blob Storage for cost-effective retention. By reducing noise upstream and standardizing logs to Sentinel ready schemas, this integration helps organizations control ingestion costs, improve detection quality, and streamline threat hunting and compliance workflows. VMRay VMRay integrates with Microsoft Sentinel to enrich SIEM and SOAR workflows with automated sandbox analysis and high-fidelity, behavior-based threat intelligence. The connector enables suspicious files and phishing URLs to be submitted directly from Sentinel to VMRay for dynamic analysis, while validated, high-confidence indicators of compromise (IOCs) are streamed back into Sentinel’s Threat Intelligence repository for correlation and detection. By adding detailed attack-chain visibility and enriched incident context, this integration helps SOC teams reduce investigation time, improve detection accuracy, and strengthen automated response workflows across Sentinel environments. Zero Networks Segment Audit Zero Networks Segment integrates with Microsoft Sentinel to provide visibility into micro-segmentation and access-control activity across the network. The connector can collect audit logs or activities from Zero Networks Segment, enabling security teams to monitor policy changes, administrative actions, and access events related to MFA-based network segmentation. By bringing segmentation audit telemetry into Sentinel, this integration supports compliance monitoring, investigation of suspicious changes, and faster detection of attempts to bypass lateral-movement controls within enterprise environments. Zscaler Internet Access (ZIA) Zscaler Internet Access integrates with Microsoft Sentinel to centralize cloud security telemetry from web and firewall traffic. The connector enables ZIA logs to be ingested into Sentinel, allowing security teams to correlate Zscaler Internet Access signals with other enterprise data for improved threat detection, investigation, and response. By bringing ZIA web, firewall, and security events into Sentinel analytics and hunting workflows, this integration helps organizations gain broader visibility into internet-based threats and strengthen Zero Trust security operations. In addition to these solutions from our third-party partners, we are also excited to announce the following connector published by the Microsoft Sentinel team: GitHub Enterprise Audit Logs Microsoft’s Sentinel Promise For Customers Every connector in the Microsoft Sentinel ecosystem is built to work out of the box. In the unlikely event a customer encounters any issue with a connector, the App Assure team stands ready to assist. For Software Developers Software partners in need of assistance in creating or updating a Sentinel solution can also leverage Microsoft’s Sentinel Promise to support our shared customers. For developers seeking to build agentic experiences utilizing Sentinel data lake, we are excited to announce the launch of our Sentinel Advisory Service to guide developers across their Sentinel journey. Customers and developers alike can reach out to us via our intake form. Learn More Microsoft Sentinel data lake Microsoft Sentinel data lake: Unify signals, cut costs, and power agentic AI Introducing Microsoft Sentinel data lake What is Microsoft Sentinel data lake Unlocking Developer Innovation with Microsoft Sentinel data lake Microsoft Sentinel Codeless Connector Framework (CCF) Create a codeless connector for Microsoft Sentinel Public Preview Announcement: Microsoft Sentinel CCF Push What’s New in Microsoft Sentinel Monthly Blog Microsoft App Assure App Assure home page App Assure services App Assure blog App Assure Request Assistance Form App Assure Sentinel Advisory Services announcement App Assure’s promise: Migrate to Sentinel with confidence App Assure’s Sentinel promise now extends to Microsoft Sentinel data lake Ignite 2025 new Microsoft Sentinel connectors announcement Microsoft Security Microsoft’s Secure Future Initiative Microsoft Unified SecOps
Ingest Microsoft XDR Advanced Hunting Data into Microsoft Sentinel
I had difficulty finding a guide that can query Microsoft Defender vulnerability management Advanced Hunting tables in Microsoft Sentinel for alerting and automation. As a result, I put together this guide to demonstrate how to ingest Microsoft XDR Advanced Hunting query results into Microsoft Sentinel using Azure Logic Apps and System‑Assigned Managed Identity. The solution allows you to: Run Advanced Hunting queries on a schedule Collect high‑risk vulnerability data (or other hunting results) Send the results to a Sentinel workspace as custom logs Create alerts and automation rules based on this data This approach avoids credential storage and follows least privilege and managed identity best practices. Prerequisites Before you begin, ensure you have: Microsoft Defender XDR access Microsoft Sentinel deployed Azure Logic Apps permission Application Administrator or higher in Microsoft Entra ID PowerShell with Az modules installed Contributor access to the Sentinel workspace Architecture at a Glance Logic App (Managed Identity) ↓ Microsoft XDR Advanced Hunting API ↓ Logic App ↓ Log Analytics Data Collector API ↓ Microsoft Sentinel (Custom Log) Step 1: Create a Logic App In the Azure Portal, go to Logic Apps Create a new Consumption Logic App Choose the appropriate: Subscription Resource Group Region Step 2: Enable System‑Assigned Managed Identity Open the Logic App Navigate to Settings → Identity Enable System‑assigned managed identity Click Save Note the Object ID This identity will later be granted permission to run Advanced Hunting queries. Step 3: Locate the Logic App in Entra ID Go to Microsoft Entra ID → Enterprise Applications Change filter to All Applications Search for your Logic App name Select the app to confirm it exists Step 4: Grant Advanced Hunting Permissions (PowerShell) Advanced Hunting permissions cannot be assigned via the portal and must be done using PowerShell. Required Permission AdvancedQuery.Read.All PowerShell Script # Your tenant ID (in the Azure portal, under Azure Active Directory > Overview). $TenantID=”Your TenantID” Connect-AzAccount -TenantId $TenantID # Get the ID of the managed identity for the app. $spID = “Your Managed Identity” # Get the service principal for Microsoft Graph by providing the AppID of WindowsDefender ATP $GraphServicePrincipal = Get-AzADServicePrincipal -Filter "AppId eq 'fc780465-2017-40d4-a0c5-307022471b92'" | Select-Object Id # Extract the Advanced query ID. $AppRole = $GraphServicePrincipal.AppRole | ` Where-Object {$_.Value -contains "AdvancedQuery.Read.All"} # If AppRoleID comes up with blank value, it can be replaced with 93489bf5-0fbc-4f2d-b901-33f2fe08ff05 # Now add the permission to the app to read the advanced queries New-AzADServicePrincipalAppRoleAssignment -ServicePrincipalId $spID -ResourceId $GraphServicePrincipal.Id -AppRoleId $AppRole.Id # Or New-AzADServicePrincipalAppRoleAssignment -ServicePrincipalId $spID -ResourceId $GraphServicePrincipal.Id -AppRoleId 93489bf5-0fbc-4f2d-b901-33f2fe08ff05 After successful execution, verify the permission under Enterprise Applications → Permissions. Step 5: Build the Logic App Workflow Open Logic App Designer and create the following flow: Trigger Recurrence (e.g., every 24 hours Run Advanced Hunting Query Connector: Microsoft Defender ATP Authentication: System‑Assigned Managed Identity Action: Run Advanced Hunting Query Sample KQL Query (High‑Risk Vulnerabilities) Send Data to Log Analytics (Sentinel) On Send Data, create a new connection and provide the workspace information where the Sentinel log exists. Obtaining the Workspace Key is not straightforward, we need to retrieve using the PowerShell command. Get-AzOperationalInsightsWorkspaceSharedKey ` -ResourceGroupName "<ResourceGroupName>" ` -Name "<WorkspaceName>" Configuration Details Workspace ID Primary key Log Type (example): XDRVulnerability_CL Request body: Results array from Advanced Hunting Step 6: Run the Logic app to return results In the logic app designer select run, If the run is successful data will be sent to sentinel workspace. Step 7: Validate Data in Microsoft Sentinel In Sentinel, run the query: XDRVulnerability_CL | where TimeGenerated > ago(24h) If data appears, ingestion is successful. Step 8: Create Alerts & Automation Rules Use Sentinel to: Create analytics rules for: CVSS > 9 Exploit available New vulnerabilities in last 24 hours Trigger: Email notifications Incident creation SOAR playbooks Conclusion By combining Logic Apps, Managed Identities, Microsoft XDR, and Microsoft Sentinel, you can create a powerful, secure, and scalable pipeline for ingesting hunting intelligence and triggering proactive detections.RSAC 2026: What the Sentinel Playbook Generator actually means for SOC automation
RSAC 2026 brought a wave of Sentinel announcements, but the one I keep coming back to is the playbook generator. Not because it's the flashiest, but because it touches something that's been a real operational pain point for years: the gap between what SOC teams need to automate and what they can realistically build and maintain. I want to unpack what this actually changes from an operational perspective, because I think the implications go further than "you can now vibe-code a playbook." The problem it solves If you've built and maintained Logic Apps playbooks in Sentinel at any scale, you know the friction. You need a connector for every integration. If there isn't one, you're writing custom HTTP actions with authentication handling, pagination, error handling - all inside a visual designer that wasn't built for complex branching logic. Debugging is painful. Version control is an afterthought. And when something breaks at 2am, the person on call needs to understand both the Logic Apps runtime AND the security workflow to fix it. The result in most environments I've seen: teams build a handful of playbooks for the obvious use cases (isolate host, disable account, post to Teams) and then stop. The long tail of automation - the enrichment workflows, the cross-tool correlation, the conditional response chains - stays manual because building it is too expensive relative to the time saved. What's actually different now The playbook generator produces Python. Not Logic Apps JSON, not ARM templates - actual Python code with documentation and a visual flowchart. You describe the workflow in natural language, the system proposes a plan, asks clarifying questions, and then generates the code once you approve. The Integration Profile concept is where this gets interesting. Instead of relying on predefined connectors, you define a base URL, auth method, and credentials for any service - and the generator creates dynamic API calls against it. This means you can automate against ServiceNow, Jira, Slack, your internal CMDB, or any REST API without waiting for Microsoft or a partner to ship a connector. The embedded VS Code experience with plan mode and act mode is a deliberate design choice. Plan mode lets you iterate on the workflow before any code is generated. Act mode produces the implementation. You can then validate against real alerts and refine through conversation or direct code edits. This is a meaningful improvement over the "deploy and pray" cycle most of us have with Logic Apps. Where I see the real impact For environments running Sentinel at scale, the playbook generator could unlock the automation long tail I mentioned above. The workflows that were never worth the Logic Apps development effort might now be worth a 15-minute conversation with the generator. Think: enrichment chains that pull context from three different tools before deciding on a response path, or conditional escalation workflows that factor in asset criticality, time of day, and analyst availability. There's also an interesting angle for teams that operate across Microsoft and non-Microsoft tooling. If your SOC uses Sentinel for SIEM but has Palo Alto, CrowdStrike, or other vendors in the stack, the Integration Profile approach means you can build cross-vendor response playbooks without middleware. The questions I'd genuinely like to hear about A few things that aren't clear from the documentation and that I think matter for production use: Security Copilot dependency: The prerequisites require a Security Copilot workspace with EU or US capacity. Someone in the blog comments already flagged this as a potential blocker for organizations that have Sentinel but not Security Copilot. Is this a hard requirement going forward, or will there be a path for Sentinel-only customers? Code lifecycle management: The generated Python runs... where exactly? What's the execution runtime? How do you version control, test, and promote these playbooks across dev/staging/prod? Logic Apps had ARM templates and CI/CD patterns. What's the equivalent here? Integration Profile security: You're storing credentials for potentially every tool in your security stack inside these profiles. What's the credential storage model? Is this backed by Key Vault? How do you rotate credentials without breaking running playbooks? Debugging in production: When a generated playbook fails at 2am, what does the troubleshooting experience look like? Do you get structured logs, execution traces, retry telemetry? Or are you reading Python stack traces? Coexistence with Logic Apps: Most environments won't rip and replace overnight. What's the intended coexistence model between generated Python playbooks and existing Logic Apps automation rules? I'm genuinely optimistic about this direction. Moving from a low-code visual designer to an AI-assisted coding model with transparent, editable output feels like the right architectural bet for where SOC automation needs to go. But the operational details around lifecycle, security, and debugging will determine whether this becomes a production staple or stays a demo-only feature. Would be interested to hear from anyone who's been in the preview - what's the reality like compared to the pitch?Agentic Use Cases for Developers on the Microsoft Sentinel Platform
Interested in building an agent with Sentinel platform solutions but not sure where to start? This blog will help you understand some common use cases for agent development that we’ve seen across our partner ecosystem. SOC teams don’t need more alerts - they need fast, repeatable investigation and response workflows. Security Copilot agents can help orchestrate the steps analysts perform by correlating across the Sentinel data lake, executing targeted KQL queries, fetching related entities, enriching with context, and producing an evidence-backed decision without forcing analysts to switch tools. Microsoft Sentinel platform is a strong foundation for agentic experiences because it exposes a normalized security data layer, an investigation surface based on incidents and entities, and extensive automation capabilities. An agent can use these primitives to correlate identity, endpoint, cloud, and network telemetry; traverse entity relationships; and recommend remediation actions. In this blog, I will break down common agentic use cases that developers can implement on Sentinel platform, framed in buildable and repeatable patterns: Identify the investigation scenario Understand the required Sentinel data connectors and KQL queries Build enrichment and correlation logic Summarize findings with supporting evidence and recommended remediation steps Use Case 1: Identity & Access Intelligence Investigation Scenario: Is this risky sign-in part of an attack path? Signals Correlated: Identity access telemetry: Source user, IPs, target resources, MFA logs Authentication outcomes and diversity: Success vs. failure, Geographic spread Identity risk posture: User risk level/state Post-auth endpoint execution: Suspicious LOLBins Correlation Logic: An analyst receives a risky sign-in signal for a user and needs to determine whether the activity reflects expected behavior - such as travel, remote access, or MFA friction - or if it signals the early stage of an identity compromise that could escalate into privileged access and downstream workload impact. Practical Example: Silverfort Identity Threat Triage Agent, which is built on a similar framework, takes the user’s UPN as input and builds a bounded, last-24-hour investigation across authentication activity, MFA logs, user risk posture, and post-authentication endpoint behavior. Outcome: By correlating identity risk signals, MFA logs, sign-in success and failure patterns, and suspicious execution activity following authentication, the agent connects the initial risky sign-in to endpoint behavior, enabling the analyst to quickly assess compromise likelihood, identify escalation indicators, and determine appropriate remediation actions. “Our collaboration with Microsoft Sentinel and Security Copilot underscores the central role identity plays across every stage of attack path triage. By integrating Silverfort’s identity risk signals with Microsoft Entra ID and Defender for Endpoint, and sharing rich telemetry across platforms, we enable Security Copilot Agent to distinguish isolated anomalies from true identity-driven intrusions - while dramatically reducing the manual effort traditionally required for incident response and threat hunting. AI-driven agents accelerate analysis, enrich investigative context, reduce dwell time, and speed detection. Instead of relying on complex queries or deep familiarity with underlying data structures, security teams can now perform seamless, identity-centric reasoning within a single interaction.” - Frank Gasparovic, Director of Solution Architecture, Technology Alliances, Silverfort Use Case 2: Cyber Resilience, Backup & Recovery Investigation Scenario: Are the threats detected on a backup indicative of production impact and recovery risk? Signals Correlated: Backup threat telemetry: Backup threat scan alerts, risk analysis events, affected host/workload, detection timestamps Cross-vendor security alerts: Endpoint, network, and cloud security alerts for the same host/workload in the same time window Correlation Logic: The agent correlates threat signals originating from the backup environment with security telemetry associated with same host/workload to validate whether there is corroborating evidence in the production environment and whether activity aligns in time. Practical Example: Commvault Security Investigation Agent, which is built on a similar framework, takes a hostname as input and builds an investigation across Commvault Threat Scan / Risk Analysis events and third-party security telemetry. By correlating backup-originating detections with production security activity for the same host, the agent determines whether the backup threat signal aligns with observable production impact. Outcome: By correlating backup threat detections with endpoint, network, and cloud security telemetry while validating timing alignment, event spikes, and data coverage, the agent connects a backup originating threat signal to production evidence, enabling the analyst to quickly assess impact likelihood and determine appropriate actions such as containment or recovery-point validation. Use Case 3: Network, Exposure & Connectivity Investigation Scenario: Is this activity indicative of legitimate remote access, or does it demonstrate suspicious connectivity and access attempts that increase risk to private applications and internal resources. Signals Correlated: User access telemetry: Source user, source IPs/geo, device/context, destinations Auth and enforcement outcomes: Success vs. failure, MFA allow/block Behavior drift: new/rare IPs/locations, unusual destination/app diversity. Suspicious activity indicators: Risky URLs/categories, known-bad indicators, automated/bot-like patterns, repeated denied private app access attempts Correlation Logic: An analyst receives an alert for a specific user and needs to determine whether the activity reflects expected behavior such as travel, remote work, or VPN usage, or whether it signals the early stages of a compromise that could later extend into private application access. Practical Example: Zscaler ZIA ZPA Correlation Agent starts with a username and builds a bounded, last-24-hour investigation across Zscaler Internet Access and Zscaler Private Access activity. By correlating user internet behavior, access context, and private application interactions, the agent connects the initial Zscaler alert to any downstream access attempts or authentication anomalies, enabling the analyst to quickly assess risk, identify suspicious patterns, and determine whether Zscaler policy adjustments are required. Outcome: Provides a last‑24‑hour verdict on whether the activity reflects expected access patterns or escalation toward private application access, and recommends next actions—such as closing as benign drift, escalating for containment, or tuning access policy—based on correlated evidence. Use Case 4: Endpoint & Runtime Intelligence Investigation Scenario: Is this process malicious or a legitimate admin action? Signals Correlated: Execution context: Process chain, full command line, signer, unusual path Account & logon: Initiating user, logon type (RDP/service), recent risky sign-ins Tooling & TTPs: LOLBins, credential access hints, lateral movement tooling Network behavior: Suspicious connections, repeated callbacks/beaconing Correlation Logic: A PowerShell alert triggers on a production server. The agent ties the process to its parent (e.g., spawned by a web worker vs. an admin shell), validates the command-line indicators, correlates outbound connections from the same PID to a first-seen destination, and checks for immediate follow-on persistence and any adjacent runtime alerts in the same time window. Outcome: Classifies the activity as malicious vs. admin and produces an evidence pack (process tree, key command indicators, destinations, persistence/tamper artifacts) as well as the recommended containment step (isolate host and revoke/reset initiating credentials). Use Case 5: Exposure & Exploitability Investigation Scenario: What is the likelihood of exploitation and blast radius? Signals Correlated: Asset exposure: Internet-facing status, exposed services/ports, and identity or network paths required to reach the workload Exploit activity: Defender alerts on the resource, IDS/WAF hits, IOC matches, and first seen exploit or probing attempts Risk amplification signals: Internet communication, high privilege access paths, and indicators that the workload processes PII or sensitive data Blast radius: Downstream reachability to crown jewel systems (e.g., databases, key vaults) and trust relationships that could enable escalation Correlation Logic: An analyst receives a Medium/High Microsoft Defender for Cloud alert on a workload and needs to determine whether it’s a standalone detection or an exploitable exposure that can quickly progress into privilege abuse and data impact. The agent correlates exposure evidence signals such as internet reachability, high-privilege paths, and indicators that workload handles sensitive data by analyzing suspicious network connections in the same bounded time window. Outcome: Produces a resource-specific risk analysis that explains why the Defender for Cloud alert is likely to be exploited, based on asset attack surface and effective privileges, plus any supporting activity in the same 24-hour window. Use Case 6: Threat Intelligence & Adversary Context Investigation Scenario: Is this activity aligned with known attacker behavior? Signals Correlated: Behavior sequence: ordered events identity → execution → network. Technique mapping: MITRE ATT&CK technique IDs, typical progression, and required prerequisites. Threat intel match: campaign/adversary, TTPs, IOCs Correlation Logic: A chain of identity compromise, PowerShell obfuscation, and periodic outbound HTTPS is observed. The agent maps the sequence to ATT&CK techniques and correlates it with threat intel that matches a known adversary campaign. Outcome: Surfaces adversary-aligned behavioral insights and TTP context to help analysts assess intrusion likelihood and guide the next investigation steps. Summary This blog is intended to help developers better understand the key use cases for building agents with Microsoft Sentinel platform along with practical patterns to apply when designing and implementing agent scenarios. Need help? If you have any issues as you work to develop your agent, the App Assure team is available to assist via our Sentinel Advisory Service. Reach out via our intake form. Resources Learn more: For a practical overview of how ISVs can move from Sentinel data lake onboarding to building agents, see the Accelerate Agent Development blog - https://aka.ms/AppAssure_AccelerateAgentDev. Get hands-on: Explore the end-to-end journey from Sentinel data lake onboarding to a working Security Copilot agent through the accompanying lab modules available on GitHub Repo: https://github.com/suchandanreddy/Microsoft-Sentinel-Labs.Microsoft Sentinel data lake FAQ
Microsoft Sentinel data lake (generally available) is a purpose‑built, cloud‑native security data lake. It centralizes all security data in an open format, serving as the foundation for agentic defense, enhanced security insights, and graph-based enrichment. It offers cost‑effective ingestion, long‑term retention, and advanced analytics. In this blog we offer answers to many of the questions we’ve heard from our customers and partners. General questions What is the Microsoft Sentinel data lake? Microsoft has expanded its industry-leading SIEM solution, Microsoft Sentinel, to include a unified, security data lake, designed to help optimize costs, simplify data management, and accelerate the adoption of AI in security operations. This modern data lake serves as the foundation for the Microsoft Sentinel platform. It has a cloud-native architecture and is purpose-built for security—bringing together all security data for greater visibility, deeper security analysis, contextual awareness and agentic defense. It provides affordable, long-term retention, allowing organizations to maintain robust security while effectively managing budgetary requirements. What are the benefits of Sentinel data lake? Microsoft Sentinel data lake is purpose built for security offering flexible analytics, cost management, and deeper security insights. Sentinel data lake: Centralizes security data delta parquet and open format for easy access. This unified data foundation accelerates threat detection, investigation, and response across hybrid and multi-cloud environments. Enables data federation by allowing customers to access data in external sources like Microsoft Fabric, ADLS and Databricks from the data lake. Federated data appears alongside native Sentinel data, enabling correlated hunting, investigation, and custom graph analysis across a broader digital estate. Offers a disaggregated storage and compute pricing model, allowing customers to store massive volumes of security data at a fraction of the cost compared to traditional SIEM solutions. Allows multiple analytics engines like Kusto, Spark, and ML to run on a single data copy, simplifying management, reducing costs, and supporting deeper security analysis. Integrates with GitHub Copilot and VS Code empowering SOC teams to automate enrichment, anomaly detection, and forensic analysis. Supports AI agents via the MCP server, allowing tools like GitHub Copilot to query and automate security tasks. The MCP Server layer brings intelligence to the data, offering Semantic Search, Query Tools, and Custom Analysis capabilities that make it easier to extract insights and automate workflows. Provides streamlined onboarding, intuitive table management, and scalable multi-tenant support, making it ideal for MSSPs and large enterprises. The Sentinel data lake is designed for security workloads, ensuring that processes from ingestion to analytics meet evolving cybersecurity requirements. Is Microsoft Sentinel SIEM going away? No. Microsoft is expanding Sentinel into an AI powered end-to-end security platform that includes SIEM and new platform capabilities - Security data lake, graph-powered analytics and MCP Server. SIEM remains a core component and will be actively developed and supported. Getting started What are the prerequisites for Sentinel data lake? To get started: Connect your Sentinel workspace to Microsoft Defender prior to onboarding to Sentinel data lake. Once in the Defender experience see data lake onboarding documentation for next steps. Note: Sentinel is moving to the Microsoft Defender portal and the Sentinel Azure portal will be retired by March 31, 2027. I am a Sentinel-only customer, and not a Defender customer. Can I use the Sentinel data lake? Yes. You must connect Sentinel to the Defender experience before onboarding to the Sentinel data lake. Microsoft Sentinel is generally available in the Microsoft Defender portal, with or without Microsoft Defender XDR or an E5 license. If you have created a log analytics workspace, enabled it for Sentinel and have the right Microsoft Entra roles (e.g. Global Administrator + Subscription Owner, Security Administrator + Sentinel Contributor), you can enable Sentinel in the Defender portal. For more details on how to connect Sentinel to Defender review these sources: Microsoft Sentinel in the Microsoft Defender portal In what regions is Sentinel data lake available? For supported regions see: Geographical availability and data residency in Microsoft Sentinel | Azure Docs. Is there an expected release date for Microsoft Sentinel data lake in GCC, GCC-H, and DoD? While the exact date is not yet finalized, we plan to expand Sentinel data lake to the US Government environments. . How will URBAC and Entra RBAC work together to manage the data lake given there is no centralized model? Entra RBAC will provide broad access to the data lake (URBAC maps the right permissions to specific Entra role holders: GA/SA/SO/GR/SR). URBAC will become a centralized pane for configuring non-global delegated access to the data lake. For today, you will use this for the “default data lake” workspace. In the future, this will be enabled for non-default Sentinel workspaces as well – meaning all workspaces in the data lake can be managed here for data lake RBAC requirements. Azure RBAC on the Log Analytics (LA) workspace in the data lake is respected through URBAC as well today. If you already hold a built-in role like log analytics reader, you will be able to run interactive queries over the tables in that workspace. Or, if you hold log analytics contributor, you can read and manage table data. For more details see: Roles and permissions in the Microsoft Sentinel platform | Microsoft Learn Data ingestion and storage How do I ingest data into the Sentinel data lake? To ingest data into the Sentinel data lake, you can use existing Sentinel data connectors or custom connectors to bring data from Microsoft and third-party sources. Data can be ingested into the analytics tier or the data lake tier. Data ingested into the analytics tier is automatically mirrored to the lake (at no additional cost). Alternatively, data that is not needed in the analytics tier can be ingested directly into the data lake. Data retention is configured directly in table management, for both analytics retention and data lake storage. Note: Certain tables do not support data lake-only ingestion via either API or data connector UI. See here for more information: Custom log tables. What is Microsoft’s guidance on when to use analytics tier vs. the data lake tier? Sentinel data lake offers flexible, built-in data tiering (analytics and data lake tiers) to effectively meet diverse business use cases and achieve cost optimization goals. Analytics tier: Is ideal for high-performance, real-time, end-to-end detections, enrichments, investigation and interactive dashboards. Typically, high-fidelity data from EDRs, email gateways, identity, SaaS and cloud logs, threat intelligence (TI) should be ingested into the analytics tier. Data in the analytics tier is best monitored proactively with scheduled alerts and scheduled analytics to enable security detections Data in this tier is retained at no cost for up to 90 days by default, extendable to 2 years. A copy of the data in this tier is automatically available in the data lake tier at no extra cost, ensuring a unified copy of security data for both tiers. Data lake tier: Is designed for cost-effective, long-term storage. High-volume logs like NetFlow logs, TLS/SSL certificate logs, firewall logs and proxy logs are best suited for data lake tier. Customers can use these logs for historical analysis, compliance and auditing, incident response (IR), forensics over historical data, build tenant baselines, TI matching and then promote resulting insights into the analytics tier. Customers can run full Kusto queries, Spark Notebooks and scheduled jobs over a single copy of their data in the data lake. Customers can also search, enrich and promote data from the data lake tier to the analytics tier for full analytics. For more details see documentation. What does it mean that a copy of all new analytics tier data will be available in the data lake? When Sentinel data lake is enabled, a copy of all new data ingested into the analytics tier is automatically duplicated into the data lake tier. This means customers don’t need to manually configure or manage this process, every new log or telemetry added to the analytics tier becomes instantly available in the data lake. This allows security teams to run advanced analytics, historical investigations, and machine learning models on a single, unified copy of data in the lake, while still using the analytics tier for real-time SOC workflows. It’s a seamless way to support both operational and long-term use cases—without duplicating effort or cost. What is the guidance for customers using data federation capability in Sentinel data lake? Starting April 1, 2026, federate data from Microsoft Fabric, ADLS, and Azure Databricks into Sentinel data lake. Use data federation when data is exploratory, infrequently accessed, or must remain at source due to governance, compliance, sovereignty, or contractual requirements. Ingest data directly into Sentinel to unlock full SIEM capabilities, always-on detections, advanced automation, and AI‑driven defense at scale. This approach lets security teams start where their data already lives — preserving governance, then progressively ingest data into Sentinel for full security value. Is there any cost for retention in the analytics tier? Analytics ingestion includes 90 days of interactive retention, at no additional cost. Simply set analytics retention to 90 days or less. Analytics retention beyond 90 days will incur a retention cost. Data can be retained longer within the data lake by using the “total retention” setting. This allows you to extend retention within the data lake for up to 12 years. While data is retained within the analytics tier, there is no charge for the mirrored data within the lake. Retaining data in the lake beyond the analytics retention period incurs additional storage costs. See documentation for more details: Manage data tiers and retention in Microsoft Sentinel | Microsoft Learn What is the guidance for Microsoft Sentinel Basic and Auxiliary Logs customers? If you previously enabled Basic or Auxiliary Logs plan in Sentinel: You can view Basic Logs in the Defender portal but manage it from the Log Analytics workspace. To manage it in the Defender portal, you must change the plan from Basic to Analytics. Once the table is transitioned to the analytics tier, if desired, it can then be transitioned to the data lake. Existing Auxiliary Log tables will be available in the data lake tier for use once the Sentinel data lake is enabled. Billing for these tables will automatically switch to the Sentinel data lake meters. Microsoft Sentinel customers are recommended to start planning their data management strategy with the data lake. While Basic and Auxiliary Logs are still available, they are not being enhanced further. Sentinel data lake offers more capabilities at a lower price point. Please plan on onboarding your security data to the Sentinel data lake. Azure Monitor customers can continue to use Basic and Auxiliary Logs for observability scenarios. What happens to customers that already have Archive logs enabled? If a customer has already configured tables for Archive retention, existing retention settings will not change and will be automatically inherited by the Sentinel data lake. All data, including existing data in archive retention will be billed using the data lake storage meter, benefiting from 6x data compression. However, the data itself will not move. Existing data in archive will continue to be accessible through Sentinel search and restore experiences: o Data will not be backfilled into the data lake. o Data will be billed using the data lake storage meter. New data ingested after enabling the data lake: o Will be automatically mirrored to the data lake and accessible through data lake explorer. o Data will be billed using the data lake storage meter. Example: If a customer has 12 months of total retention enabled on a table, 2 months after enabling ingestion into the Sentinel data lake, the customer will still have access to 10 months of archived data (through Sentinel search and restore experiences), but access to only 2 months of data in the data lake (since the data lake was enabled). Key considerations for customers that currently have Archive logs enabled: The existing archive will remain, with new data ingested into the data lake going forward; previously stored archive data will not be backfilled into the lake. Archive logs will continue to be accessible via the Search and Restore tab under Sentinel. If analytics and data lake mode are enabled on table, which is the default setting for analytics tables when Sentinel data lake is enabled, all new data will be ingested into the Sentinel data lake. There will only be one storage meter (which is data lake storage) going forward. Archive will continue to be accessible via Search and Restore. If Sentinel data lake-only mode is enabled on table, new data will be ingested only into the data lake; any data that’s not already in the Sentinel data lake won’t be migrated/backfilled. Only data that was previously ingested under the archive plan will be accessible via Search and Restore. What is the guidance for customers using Azure Data Explorer (ADX) alongside Microsoft Sentinel? Some customers might have set up ADX cluster for their DIY lake setup. Customers can choose to continue using that setup and gradually migrate to Sentinel data lake for new data that they want to manage. The lake explorer will support federation with ADX to enable the customers to migrate gradually and simplify their deployment. What happens to the Defender XDR data after enabling Sentinel data lake? By default, Defender XDR tables are available for querying in advanced hunting, with 30 days of analytics tier retention included with the XDR license. To retain data beyond this period, an explicit change to the retention setting is required, either by extending the analytics tier retention or the total retention period. You can extend the retention period of supported Defender XDR tables beyond 30 days and ingest the data into the analytics tier. For more information see Manage XDR data in Microsoft Sentinel. You can also ingest XDR data directly into the data lake tier. See here for more information. A list of XDR advanced hunting tables supported by Sentinel are documented here: Connect Microsoft Defender XDR data to Microsoft Sentinel | Microsoft Learn. KQL queries and jobs Is KQL and Notebook supported over the Sentinel data lake? Yes, via the data lake KQL query experience along with a fully managed Notebook experience which enables spark-based big data analytics over a single copy of all your security data. Customers can run queries across any time range of data in their Sentinel data lake. In the future, this will be extended to enable SQL query over lake as well. Note: Triggering a KQL job directly via an API or Logic App is not yet supported but is on the roadmap. Why are there two different places to run KQL queries in Sentinel experience? Advanced hunting queries both XDR and analytics tables, with compute cost included. Data lake explorer only queries data in the lake and incurs a separate compute cost. Consolidating advanced hunting and KQL explorer user interfaces is on the roadmap. This will provide security analysts a unified query experience across both analytics and data lake tiers. Where is the output from KQL jobs stored? KQL jobs are written into existing or new custom tables in the analytics tier. Is it possible to run KQL queries on multiple data lake tables? Yes, you can run KQL interactive queries and jobs using operators like join or union. Can KQL queries (either interactive or via KQL jobs) join data across multiple workspaces? Security teams can run multi-workspace KQL queries for broader threat correlation Pricing and billing How does a customer pay for Sentinel data lake? Billing is automatically enabled at the time of onboarding based on Azure Subscription and Resource Group selections. Customers are then charged based on the volume of data ingested, retained, and analyzed (e.g. KQL Queries and Jobs). See Sentinel pricing page for more details. 2. What are the pricing components for Sentinel data lake? Sentinel data lake offers a flexible pricing model designed to optimize security coverage and costs. At a high level, pricing is based on the volume of data ingested/processed, the volume of data retained, and the volume of data processed. For specific meter definitions, see documentation. 3. How does the business model for Sentinel SIEM change with the introduction of the data lake? There is no change to existing Sentinel analytics tier ingestion business model. Sentinel data lake has separate meters for ingestion, storage and analytics. 4. What happens to the existing Sentinel SIEM and related Azure Monitor billing meters when a customer onboards to Sentinel data lake? When a customer onboards to the Sentinel data lake, nothing changes with analytic ingestion or retention. Customers using data archive and Auxiliary Logs will automatically transition to the new data lake meters. How does data lake storage affect cost efficiency for high volume data retention? Sentinel data lake offers cost-effective, long-term storage with uniform data compression of 6:1 across all data sources, applicable only to data lake storage. Example: For 600GB of data stored, you are only billed for 100GB compressed data. This approach allows organizations to retain greater volumes of security data over extended periods cost-effectively, thereby reducing security risks without compromising their overall security posture. here How “Data Processing” billed? To support the ingestion and standardization of diverse data sources, the Data Processing feature applies a $0.10 per GB (US East) charge for all data ingested into the data lake. This feature enables a broad array of transformations like redaction, splitting, filtering and normalization. The data processing charge is applied per GB of uncompressed data Note: For regional pricing, please refer to the “Data processing” meter within the Microsoft Sentinel Pricing official documentation. Does “Data processing” meter apply to analytics tier data mirrored in the data lake? No. Data processing charge will not be applied to mirrored data. Data mirrored from the analytic tier is not subject to either data ingestion or processing charges. How is retention billed for tables that use data lake-only ingestion & retention? Sentinel data lake decouples ingestion, storage, and analytics meters. Customers have the flexibility to pay based on how data is retained and used. For tables that use data lake‑only ingestion, there is no included free retention—unlike the analytics tier, which includes 90 days of analytics retention. Retention charges begin immediately once data is stored in the data lake. Data lake storage billing is based on compressed data size rather than raw ingested volume, which significantly reduces storage costs and delivers lower overall retention spend for customers. Does data federation incur charges? Data federation does not generate any ingestion or storage fees in Sentinel data lake. Customers are billed only when they run analytics or queries on federated data, with charges based on Sentinel data lake compute and analytics meters. This means customers pay solely for actual data usage, not mere connectivity. How do I understand Sentinel data lake costs? Sentinel data lake costs driven by three primary factors: how much data is ingested, how long that data is retained, and how the data is used. Customers can flexibly choose to ingest data into the analytics tier or data lake tier, and these architectural choices directly impact cost. For example, data can be ingested into the analytics tier—where commitment tiers help optimize costs for high data volumes—or ingested data directly into the Sentinel data lake for lower‑cost ingestion, storage, and on‑demand analysis. Customers are encouraged to work with their Microsoft account team to obtain an accurate cost estimate tailored to their environment. See Sentinel pricing page to understand Sentinel pricing. How do I manage Sentinel data lake costs? Built-in cost management experiences help customers with cost predictability, billing transparency, and operational efficiency. Reports provide customers with insights into usage trends over time, enabling them to identify cost drivers and optimize data retention and processing strategies. Set usage-based alerts on specific meters to monitor and control costs. For example, receive alerts when query or notebook usage passes set limits, helping avoid unexpected expenses and manage budgets. See our Sentinel cost management documentation to learn more. If I’m an Auxiliary Logs customer, how will onboarding to the Sentinel data lake affect my billing? Once a workspace is onboarded to Sentinel data lake, all Auxiliary Logs meters will be replaced by new data lake meters. Do we charge for data lake ingestion and storage for graph experiences? Microsoft Sentinel graph-based experiences are included as part of the existing Defender and Purview licenses. However, Sentinel graph requires Sentinel data lake and specific data sources to build the underlying graph. Enabling these data sources will incur ingestion and data lake storage costs. Note: For Sentinel SIEM customers, most required data sources are free for analytics ingestion. Non-entitled sources such as Microsoft Entra ID logs will incur ingestion and data lake storage costs. How is Entra asset data and ARG data billed? Data lake ingestion charges of $0.05 per GB (US EAST) will apply to Entra asset data and ARG data. Note: This was previously not billed during public preview and is billed since data lake GA. To learn more, see: https://learn.microsoft.com/azure/sentinel/datalake/enable-data-connectors When a customer activates Sentinel data lake, what happens to tables with archive logs enabled? To simplify billing, once the data lake is enabled, all archive data will be billed using the data lake storage meter. This provides consistent long-term retention billing and includes automatic 6x data compression. For most customers, this change results in lower long‑term retention costs. However, customers who previously had discounted archive retention pricing will not automatically receive the same discounts on the new data lake storage meters. In these cases, customers should engage their Microsoft account team to review pricing implications before enabling the Sentinel data lake. Thank you Thank you to our customers and partners for your continued trust and collaboration. Your feedback drives our innovation, and we’re excited to keep evolving Microsoft Sentinel to meet your security needs. If you have any questions, please don’t hesitate to reach out—we’re here to support you every step of the way. Learn more: Get started with Sentinel data lake today: https://aka.ms/Get_started/Sentinel_datalake Microsoft Sentinel AI-ready platform: https://aka.ms/Microsoft_Sentinel Sentinel data lake videos: https://aka.ms/Sentineldatalake_videos Latest innovations and updates on Sentinel: https://aka.ms/msftsentinelblog Sentinel pricing page: https://aka.ms/MicrosoftSentinel_Pricing
Part 3: Build custom email security reports with Power BI and workbooks in Microsoft Sentinel
TL;DR: We're releasing a brand-new Power BI template for email security reporting and a major update (v3) to the Microsoft Sentinel workbook. Both solutions share the same rich visuals and insights. Choose Power BI for quick deployment without Sentinel, or the Sentinel workbook for extended data retention and multi-tenant scenarios. Get started in minutes with either option. Introduction Security teams in both small and large organizations track key metrics to make critical security decisions and identify meaningful trends in their organizations. While Microsoft Defender for Office 365 provides rich, built-in reporting capabilities, many security teams need custom reporting solutions to create dedicated views, combine multiple data sources, and derive deeper insights tailored to their unique requirements. Earlier last year (Part 1 and Part 2) we shared examples of how you can use workbooks in Microsoft Sentinel to build a custom email security insights dashboard for Microsoft Defender for Office 365. Today, we are excited to announce the release of a new Power BI template file for Microsoft Defender for Office 365 customers, along with an updated version of the Microsoft Defender for Office 365 Detections and Insights workbook in Microsoft Sentinel. Both solutions share the same visual design and structure, giving you a consistent experience regardless of which platform you choose. Power BI template file - Microsoft Defender for Office 365 Detections and Insights: Microsoft Sentinel workbook - Microsoft Defender for Office 365 Detections and Insights: NEW: Power BI template file for Microsoft Defender for Office 365 Detections and Insights This custom reporting template file utilizes Power BI and Microsoft Defender XDR Advanced Hunting through the Microsoft Graph security API. It is designed for Microsoft Defender for Office 365 customers who have access to Advanced Hunting but are not using Microsoft Sentinel. Advanced Hunting data in Microsoft Defender for Office 365 tables is available for up to 30 days. The reporting template uses these same data tables to visualize insights into an organization's email security, including protection, detection, and response metrics provided by Microsoft Defender for Office 365. Note: If data retention beyond 30 days is required, customers can use the Microsoft Defender for Office 365 Detections and Insights workbook in Microsoft Sentinel. You can find the new .pbit template file and detailed instructions on how to set up and use it in the unified Microsoft Sentinel and Microsoft 365 Defender GitHub repository. This new Power BI template uses the same visuals and structure as the Microsoft Defender for Office 365 Detections and Insights workbook in Microsoft Sentinel, providing an easy way to gain deep email security insights across a wide range of use cases. UPDATED: Microsoft Defender for Office 365 Detections and Insights workbook in Microsoft Sentinel We are excited to announce the release of a new version (3.0.0) of the Microsoft Defender for Office 365 Detections and Insights workbook in Microsoft Sentinel. The workbook is part of the Microsoft Defender XDR solution in Microsoft Sentinel and can be installed and started to use with a few simple clicks. In this new release we incorporated feedback we have received from many customers in the past few months to add new visuals, updated existing visuals and add insights focusing on security operations. What’s New Here are some notable changes and new capabilities available in the updated workbook template. Improved structure: Headings and grouped insights have been added to tabs for easier navigation and understanding of metrics. Contextual explanations: Each tab, section, and visual now includes descriptions to help users interpret insights effectively. Drill-down capability: A single “Open query link” action allows users to view the underlying KQL query for each visual, enabling quick investigation and hunting by modifying conditions or removing summaries to access raw data. Detection Dashboard tab enhancements: Added an example Effectiveness metric, updated visuals to focus on overall Microsoft Defender for Office 365 protection values, and introduced new sections for Emerging Threats and Microsoft 365 Secure Email Gateway Performance. New Security Operations Center (SOC) Insights tab: Provides operational metrics such as Security Incident Response, Investigation, and Response Actions for SOC teams. Advanced threat insights: Includes new LLM-based content analysis detections and threat classification insights on the Emails – Phish Detections tab. External forwarding insights: Added deep visibility into Inbox rules and SMTP forwarding in Outlook, including destination details to assess potential data leakage risks. Geo-location improvements: Sender IPv4 insights now include top countries for better geographic context for each Threat types (Malware, Spam, Phish). Enhanced top attacked users and top senders: Added TotalEmailCount and Bad_Traffic_Percentage for richer context in top attacked users and senders charts. Expanded URL click insights: URL click-based threat detection visuals now include Microsoft 365 Copilot as a workload. How to use the workbook across multiple tenants If you manage multiple environments with Microsoft Sentinel — or you are an MSSP (Managed Security Service Provider) working across multiple customer tenants — you can also use the workbook in multi‑tenant scenarios. Once the required configuration is in place, you can change the Subscription and Workspace parameters in the workbook to be multi select and load data from one or multiple tenants. This enables to see deep email security insights in multi‑tenant environments, including: Aggregated multi‑tenant view: You can view aggregated insights across tenants in a single workbook view. By multi‑selecting tenants in the Subscription and Workspace parameters, the workbook automatically loads and combines data from all selected environments for all visuals on all tabs. Side‑by-side‑ comparison: For example, you can compare phishing detection trends or top attacked users across two or more tenants simply by opening the workbook in two browser windows placed side by side. Note: For the multiselect option‑ to work in the current workbook version, you need to manually adjust the Subscription and Workspace parameters. This configuration is planned to become the default in the next release of the workbook. Until then, you can simply apply this change using the workbook’s Edit mode. How to get the updated workbook version The latest version of the Microsoft Defender for Office 365 Detections and Insights workbook is available as part of the Microsoft Defender XDR solution in the Microsoft Sentinel - Content hub. Version 3.0.13 of the solution has the updated workbook template. If you already have the Microsoft Defender XDR solution deployed, version 3.0.13 is available now as an update. After you install the update, you will have the new workbook template available to use. Note: If you had the workbook saved from a previous template version, make sure you delete the old workbook and use the save button on the new template to recreate a new local version with the latest updates. If you install the Microsoft Defender XDR solution for the first time, you are deploying the latest version and will have the updated template ready to use. How to edit and share the workbook with others You can customize each visual easily. Simply edit the workbook after saving, then adjust the underlying KQL query, change the type of the visual, or create new insights. More information: Visualize your data using workbooks in Microsoft Sentinel | Microsoft Learn Granting other users access to the workbook also possible, see the Manage Access to Microsoft Sentinel Workbooks with Lower Scoped RBAC on the Microsoft Sentinel Blog. Do you have feedback related to reporting in Microsoft Defender for Office 365? You can provide direct feedback via filling the form: aka.ms/mdoreportingfeedback Do you have questions or feedback about Microsoft Defender for Office 365? Engage with the community and Microsoft experts in the Defender for Office 365 forum. More information Integrate Microsoft Defender XDR with Microsoft Sentinel Learn more about Microsoft Sentinel workbooks Learn more about Microsoft Defender XDR
