Step-by-Step Guide: Integrating Microsoft Purview with Azure Databricks and Microsoft Fabric
Co-Authored By: aryananmol, laurenkirkwood and mmanley This article provides practical guidance on setup, cost considerations, and integration steps for Azure Databricks and Microsoft Fabric to help organizations plan for building a strong data governance framework. It outlines how Microsoft Purview can unify governance efforts across cloud platforms, enabling consistent policy enforcement, metadata management, and lineage tracking. The content is tailored for architects and data leaders seeking to execute governance in scalable, hybrid environments. Note: this article focuses mainly on Data Governance features for Microsoft Purview. Why Microsoft Purview Microsoft Purview enables organizations to discover, catalog, and manage data across environments with clarity and control. Automated scanning and classification build a unified view of your data estate enriched with metadata, lineage, and sensitivity labels, and the Unified Catalog gives business-friendly search and governance constructs like domains, data products, glossary terms, and data quality. Note: Microsoft Purview Unified Catalog is being rolled out globally, with availability across multiple Microsoft Entra tenant regions; this page lists supported regions, availability dates, and deployment plans for the Unified Catalog service: Unified Catalog Supported Regions. Understanding Data Governance Features Cost in Purview Under the classic model: Data Map (Classic), users pay for an “always-on” Data Map capacity and scanning compute. In the new model, those infrastructure costs are subsumed into the consumption meters – meaning there are no direct charges for metadata storage or scanning jobs when using the Unified Catalog (Enterprise tier). Essentially, Microsoft stopped billing separately for the underlying data map and scan vCore-hours once you opt into the new model or start fresh with it. You only incur charges when you govern assets or run data processing tasks. This makes costs more predictable and tied to governance value: you can scan as much as needed to populate the catalog without worrying about scan fees and then pay only for the assets you actively manage (“govern”) and any data quality processes you execute. In summary, Purview Enterprise’s pricing is usage-based and divided into two primary areas: (1) Governed Assets and (2) Data Processing (DGPUs). Plan for Governance Microsoft Purview’s data governance framework is built on two core components: Data Map and Unified Catalog. The Data Map acts as the technical foundation, storing metadata about assets discovered through scans across your data estate. It inventories sources and organizes them into collections and domains for technical administration. The Unified Catalog sits on top as the business-facing layer, leveraging the Data Map’s metadata to create a curated marketplace of data products, glossary terms, and governance domains for data consumers and stewards. Before onboarding sources, align Unified Catalog (business-facing) and Data Map (technical inventory) and define roles, domains, and collections so ownership and access boundaries are clear. Here is a documentation that covers roles and permissions in Purview: Permissions in the Microsoft Purview portal | Microsoft Learn. The imageabove helps understand therelationship between the primary data governance solutions, Unified Catalog and Data Map, and the permissions granted by the roles for each solution. Considerations and Steps for Setting up Purview Steps for Setting up Purview: Step 1: Create a Purview Account. In the Azure Portal, use the search bar at the top to navigate to Microsoft Purview Accounts. Once there, click “Create”. This will take you to the following screen: Step 2: Click Next: Configuration and follow the Wizard, completing the necessary fields, including information on Networking, Configurations, and Tags. Then click Review + Create to create your Purview account. Consideration: Private networking: Use Private Endpoints to secure Unified Catalog/Data Map access and scan traffic; follow the new platform private endpoints guidance in the Microsoft Purview portal or migrate classic endpoints. Once your Purview Account is created, you’ll want to set up and manage your organization’s governance strategy to ensure that your data is classified and managed according to the specific lifecycle guidelines you set. Note: Follow the steps in this guide to set up Microsoft Purview Data Lifecycle Management: Data retention policy, labeling, and records management. Data Map Best Practices Design your collections hierarchy to align with organizational strategy—such as by geography, business function, or data domain. Register each data source only once per Purview account to avoid conflicting access controls. If multiple teams consume the same source, register it at a parent collection and create scans under subcollections for visibility. The imageaboveillustrates a recommended approach for structuring your Purview DataMap. Why Collection Structure Matters A well-structured Data Map strategy, including a clearly defined hierarchy of collections and domains, is critical because the Data Map serves as the metadata backbone for Microsoft Purview. It underpins the Unified Catalog, enabling consistent governance, role-based access control, and discoverability across the enterprise. Designing this hierarchy thoughtfully ensures scalability, simplifies permissions management, and provides a solid foundation for implementing enterprise-wide data governance. Purview Integration with Azure Databricks Databricks Workspace Structure In Azure Databricks, each region supports a single Unity Catalog metastore, which is shared across all workspaces within that region. This centralized architecture enables consistent data governance, simplifies access control, and facilitates seamless data sharing across teams. As an administrator, you can scan one workspace in the region using Microsoft Purview to discover and classify data managed by Unity Catalog, since the metastore governs all associated workspaces in a region. If your organization operates across multiple regions and utilizes cross-region data sharing, please review the consideration and workaround outlined below to ensure proper configuration and governance. Follow pre-requisite requirements here, before you register your workspace: Prerequisites to Connect and manage Azure Databricks Unity Catalog in Microsoft Purview. Steps to Register Databricks Workspace Step 1: In the Microsoft Purview portal, navigate to the Data Map section from the left-hand menu. Select Data Sources. Click on Register to begin the process of adding your Databricks workspace. Step 2: Note: There are two Databricks data sources, please review documentation here to review differences in capability: Connect to and manage Azure Databricks Unity Catalog in Microsoft Purview | Microsoft Learn. You can choose either source based on your organization’s needs. Recommended is “Azure Databricks Unity Catalog”: Step 3: Register your workspace. Here are the steps to register your data source: Steps to Register an Azure Databricks workspace in Microsoft Purview. Step 4: Initiate scan for your workspace, follow steps here: Steps to scan Azure Databricks to automatically identify assets. Once you have entered the required information test your connection and click continue to set up scheduled scan trigger. Step 5: For Scan trigger, choose whether to set up a schedule or run the scan once according to your business needs. Step 6: From the left pane, select Data Map and select your data source for your workspace. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab. Review further options here: Manage and Review your Scans. You can review your scanned data sources, history and details here: Navigate to scan run history for a given scan. Limitation: The “Azure Databricks Unity Catalog” data source in Microsoft Purview does not currently support connection via Managed Vnet. As a workaround, the product team recommends using the “Azure Databricks Unity Catalog” source in combination with a Self-hosted Integration Runtime (SHIR) to enable scanning and metadata ingestion. You can find setup guidance here: Create and manage SHIR in Microsoft Purview Choose the right integration runtime configuration Scoped scan support for Unity Catalog is expected to enter private preview soon. You can sign up here: https://aka.ms/dbxpreview. Considerations: If you have delta-shared Databricks-to-Databricks workspaces, you may have duplication in your data assets if you are scanning both Workspaces. The workaround for this scenario is as you add tables/data assets to a Data Product for Governance in Microsoft Purview, you can identify the duplicated tables/data assets using their Fully Qualified Name (FQN). To make identification easier: Look for the keyword “sharing” in the FQN, which indicates a Delta-Shared table. You can also apply tags to these tables for quicker filtering and selection. The screenshot highlights how the FQN appears in the interface, helping you confidently identify and manage your data assets. Purview Integration with Microsoft Fabric Understanding Fabric Integration: Connect Cross-Tenant: This refers to integrating Microsoft Fabric resources across different Microsoft Entra tenants. It enables organizations to share data, reports, and workloads securely between separate tenants, often used in multi-organization collaborations or partner ecosystems. Key considerations include authentication, data governance, and compliance with cross-tenant policies. Connect In-Same-Tenant: This involves connecting Fabric resources within the same Microsoft Entra tenant. It simplifies integration by leveraging shared identity and governance models, allowing seamless access to data, reports, and pipelines across different workspaces or departments under the same organizational umbrella. Requirements: An Azure account with an active subscription. Create an account for free. An active Microsoft Purview account. Authentication is supported via: Managed Identity. Delegated Authentication and Service Principal. Steps to Register Fabric Tenant Step 1: In the Microsoft Purview portal, navigate to the Data Map section from the left-hand menu. Select Data Sources. Click on Register to begin the process of adding your Fabric Tenant (which also includes PowerBI). Step 2: Add in Data Source Name, keep Tenant ID as default (auto-populated). Microsoft Fabric and Microsoft Purview should be in the same tenant. Step 3: Enter in Scan name, enable/disable scanning for personal workspaces. You will notice under Credentials automatically created identity for authenticating Purview account. Note: If your Purview is behind Private Network, follow the guidelines here: Connect to your Microsoft Fabric tenant in same tenant as Microsoft Purview. Step 4: From your Microsoft Fabric, open Settings, Click on Tenant Settings and enable “Service Principals can access read-only admin APIs”, “Enhanced admin API responses within detailed metadata” and “Enhance Admin API responses with DAX and Mashup Expressions” within Admin API Settings section. Step 5: You will need to create a group, add the Purviews' managed identity to the group and add the group under “Service Principals can access read-only admin APIs” section of your tenant settings inside Microsoft Fabric Step 6: Test your connection and setup scope for your scan. Select the required workspaces, click continue and automate a scan trigger. Step 7: From the left pane, select Data Map and select your data source for your workspace. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab. Review further options here: Manage and Review your Scans. You can review your scanned data sources, history and details here: Navigate to scan run history for a given scan. Why Customers Love Purview Kern County unified its approach to securing and governing data with Microsoft Purview, ensuring consistent compliance and streamlined data management across departments. EY accelerated secure AI development by leveraging the Microsoft Purview SDK, enabling robust data governance and privacy controls for advanced analytics and AI initiatives. Prince William County Public Schools created a more cyber-safe classroom environment with Microsoft Purview, protecting sensitive student information while supporting digital learning. FSA (Food Standards Agency) helps keep the UK food supply safe using Microsoft Purview Records Management, ensuring regulatory compliance and safeguarding critical data assets. Conclusion Purview’s Unified Catalog centralizes governance across Discovery, Catalog Management, and Health Management. The Governance features in Purview allow organizations to confidently answer critical questions: What data do we have? Where did it come from? Who is responsible for it? Is it secure and compliant? Can we trust its quality? Microsoft Purview, when integrated with Azure Databricks and Microsoft Fabric, provides a unified approach to cataloging, classifying, and governing data across diverse environments. By leveraging Purview’s Unified Catalog, Data Map, and advanced governance features, organizations can achieve end-to-end visibility, enforce consistent policies, and improve data quality. You might ask, why does data quality matter? Well, in today’s world, data is the new gold. References Microsoft Purview | Microsoft Learn Pricing - Microsoft Purview | Microsoft Azure Use Microsoft Purview to Govern Microsoft Fabric Connect to and manage Azure Databricks Unity Catalog in Microsoft PurviewIntroducing Microsoft Sentinel graph (Public Preview)
Security is being reengineered for the AI era—moving beyond static, rulebound controls and after-the-fact response toward platform-led, machine-speed defense. The challenge is clear: fragmented tools, sprawling signals, and legacy architectures that can’t match the velocity and scale of modern attacks. What’s needed is an AI-ready, data-first foundation—one that turns telemetry into a security graph, standardizes access for agents, and coordinates autonomous actions while keeping humans in command of strategy and high-impact investigations. Security teams already center operations on their SIEM for end-to-end visibility, and we’re advancing that foundation by evolving Microsoft Sentinel into both the SIEM and the platform for agentic defense—connecting analytics and context across ecosystems. And today, we announced the general availability of Sentinel data lake and introduced new preview platform capabilities that are built on Sentinel data lake (Figure 1), so protection accelerates to machine speed while analysts do their best work. We are excited to announce the public preview of Microsoft Sentinel graph, a deeply connected map of your digital estate across endpoints, cloud, email, identity, SaaS apps, and enriched with our threat intelligence. Sentinel graph, a core capability of the Sentinel platform, enables Defenders and Agentic AI to connect the dots and bring deep context quickly, enabling modern defense across pre-breach and post-breach. Starting today, we are delivering new graph-based analytics and interactive visualization capabilities across Microsoft Defender and Microsoft Purview. Attackers think in graphs. For a long time, defenders have been limited to querying and analyzing data in lists forcing them to think in silos. With Sentinel graph, Defenders and AI can quickly reveal relationships, traversable digital paths to understand blast radius, privilege escalation, and anomalies across large, cloud-scale data sets, deriving deep contextual insight across their digital estate, SOC teams and their AI Agents can stay proactive and resilient. With Sentinel graph-powered experiences in Defender and Purview, defenders can now reason over assets, identities, activities, and threat intelligence to accelerate detection, hunting, investigation, and response. Incident graph in Defender. The incident graph in the Microsoft Defender portal is now enriched with ability to analyze blast radius of the active attack. During an incident investigation, the blast radius analysis quickly evaluates and visualizes the vulnerable paths an attacker could take from a compromise entity to a critical asset. This allows SOC teams to effectively prioritize and focus their attack mitigation and response saving critical time and limiting impact. Hunting graph in Defender. Threat hunting often requires connecting disparate pieces of data to uncover hidden paths that attackers exploit to reach your crown jewels. With the new hunting graph, analysts can visually traverse the complex web of relationships between users, devices, and other entities to reveal privileged access paths to critical assets. This graph-powered exploration transforms threat hunting into a proactive mission, enabling SOC teams to surface vulnerabilities and intercept attacks before they gain momentum. This approach shifts security operations from reactive alert handling to proactive threat hunting, enabling teams to identify vulnerabilities and stop attacks before they escalate. Data risk graph in Purview Insider Risk Management (IRM). Investigating data leaks and insider risks is challenging when information is scattered across multiple sources. The data risk graph in IRM offers a unified view across SharePoint and OneDrive, connecting users, assets, and activities. Investigators can see not just what data was leaked, but also the full blast radius of risky user activity. This context helps data security teams triage alerts, understand the impact of incidents, and take targeted actions to prevent future leaks. Data risk graph in Purview Data Security Investigation (DSI). To truly understand a data breach, you need to follow the trail—tracking files and their activities across every tool and source. The data risk graph does this by automatically combining unified audit logs, Entra audit logs, and threat intelligence, providing an invaluable insight. With the power of the data risk graph, data security teams can pinpoint sensitive data access and movement, map potential exfiltration paths, and visualize the users and activities linked to risky files, all in one view. Getting started Microsoft Defender If you already have the Sentinel data lake, the required graph will be auto provisioned when you login into the Defender portal; hunting graph and incident graph experience will appear in the Defender portal. New to data lake? Use the Sentinel data lake onboarding flow to provision the data lake and graph. Microsoft Purview Follow the Sentinel data lake onboarding flow to provision the data lake and graph. In Purview Insider Risk Management (IRM), follow the instructions here. In Purview Data Security Investigation (DSI), follow the instructions here. Reference links Watch Microsoft Secure Microsoft Secure news blog Data lake blog MCP server blog ISV blog Security Store blog Copilot blog Microsoft Sentinel—AI-Powered Cloud SIEM | Microsoft SecurityTrusted Signing Public Preview Update
Nearly a year ago we announced the Public Preview of Trusted Signing with availability for organizations with 3 years or more of verifiable history to onboard to the service to get a fully managed code signing experience to simplify the efforts for Windows app developers. Over the past year, we’ve announced new features including the Preview support for Individual Developers, and we highlighted how the service contributes to the Windows Security story at Microsoft BUILD 2024 in the Unleash Windows App Security & Reputation with Trusted Signing session. During the Public Preview, we have obtained valuable insights on the service features from our customers, and insights into the developer experience as well as experience for Windows users. As we incorporate this feedback and learning into our General Availability (GA) release, we are limiting new customer subscriptions as part of the public preview. This approach will allow us to focus on refining the service based on the feedback and data collected during the preview phase. The limit in new customer subscriptions for Trusted Signing will take effect Wednesday, April 2, 2025, and make the service only available to US and Canada-based organizations with 3 years or more of verifiable history. Onboarding for individual developers and all other organizations will not be directly available for the remainder of the preview, and we look forward to expanding the service availability as we approach GA. Note that this announcement does not impact any existing subscribers of Trusted Signing, and the service will continue to be available for these subscribers as it has been throughout the Public Preview. For additional information about Trusted Signing please refer to Trusted Signing documentation | Microsoft Learn and Trusted Signing FAQ | Microsoft Learn.Phishing Triage Agent in Defender XDR: Say Goodbye to False Positives and Analyst Fatigue
Phishing remains one of the most common and dangerous attack vectors in cybersecurity. With the rise of user-reported suspicious emails, Security Operations Center (SOC) teams are overwhelmed by the volume and complexity of triage. Enter the Phishing Triage Agent, a new capability within Microsoft Defender XDR and Security Copilot that uses AI to automate phishing classification, reduce false positives, and accelerate incident response. Image from Microsoft Learn - Microsoft Security Copilot Agents What’s the Issue? SOC analysts regularly handle a high volume of suspicious email reports, dedicating substantial time to reviewing each submission, though many prove to be non-threatening. More than 90% of cyberattacks originate from phishing, making it a primary method used to breach organizational defenses. This results in numerous alerts and potential incidents that must be triaged, prioritized, and investigated. Traditional rule-based systems, which were once effective for detecting known threats, now face challenges as attackers adapt their tactics and techniques. The continually changing threat landscape requires defenders to address not only advanced phishing attempts but also alert fatigue and the possibility of missing significant incidents. In this context, scalable and efficient solutions are important for enabling defenders to focus on investigating and mitigating real threats rather than addressing false positives. Image from Microsoft Learn - Type view for the Mailflow status report Why It’s Urgent Phishing is a very popular entry point for attackers, with such attacks growing more frequent and advanced, leaving SOC teams struggling with incident management. The Phishing Triage Agent uses LLMs and state of the art Threat Intelligence to quickly analyze and categorize reported emails, helping analysts focus on real threats. Integrating easily with current workflows, it offers adaptive, AI-driven insights for rapid threat detection and improved situational awareness. Through ongoing learning, it stays aligned with evolving attacker tactics and helps strengthen email security. Image from Microsoft Learn - Defender for Office 365 Phishing block Use Cases Automated Triage: Classify phishing emails without manual rules. False Positive Filtering: Reduce noise and analyst fatigue. Explainable AI: Provide clear reasoning behind verdicts. Threat Prioritization: Focus on high-risk incidents with enriched context. Compliance Auditing: Maintain logs and transparency for regulatory needs. Image from Microsoft Learn – Incident Queue with Phishing Triage Agent How It Works The agent activates when a user reports a suspicious email and does the following: Analyzes the message using LLMs. Classifies it as normal email or phishing. Enriches the incident with threat intelligence. Provides a verdict with natural-language explanation. Escalates or resolves based on severity and confidence. Image was created with AI It integrates with Security Copilot, enabling AI-assisted investigations and automation across Microsoft Defender XDR. Image from Microsoft Learn - Transparency and explainability in phishing triage Pros and Cons This section outlines the main advantages, limitations, and licensing requirements of the Phishing Triage Agent solution. Pros Cons License Needed Scales phishing triage across the enterprise Requires SCU provisioning and Defender licensing Microsoft Defender for Office 365 Plan 2 Reduces false positives and analyst fatigue Currently in preview; may evolve Security Copilot subscription Provides explainable decisions Requires integration with Defender XDR SCUs and plugin configuration The Phishing Triage Agent is a game-changer for SOC teams. By combining AI-powered analysis with human oversight, it accelerates detection, sharpens response, and strengthens organizational security posture. As phishing tactics evolve, this agent ensures your defenses stay ahead. Getting Started with Phishing Triage Agent The Phishing Triage Agent in Microsoft Defender XDR and Security Copilot helps SOC teams automate and accelerate phishing email analysis. Here’s how to get started: Check Prerequisites Ensure your organization has the necessary licenses: Microsoft Defender for Office 365 Plan 2 Security Copilot subscription Security Compute Units (SCUs) provisioned Defender XDR integration enabled Microsoft Defender for Office 365 service description License options for Microsoft 365 Copilot Enable Phishing Triage Agent Go to the Microsoft Defender portal: Settings > Email & Collaboration > Policies & Rules Enable the Phishing Triage Agent under Automated Investigation & Response (AIR). Automated investigation and response examples - Microsoft Defender for Office 365 Integrate with Security Copilot In the Security Copilot interface: Add the Phishing Triage Agent as a plugin Configure it to trigger when users report suspicious emails via Outlook or Defender for Office 365 Use plugins in Microsoft Security Copilot Test the Workflow Simulate a phishing report by submitting a suspicious email. The agent will: Use LLMs to analyze the message Classify it as phishing or safe Enriching the incident with threat intelligence Provide a natural-language explanation Escalate or resolve based on severity Security Copilot Phishing Triage Agent in Microsoft Defender Review and Tune Use the Mailflow status report and Incident Queue to monitor: Classification accuracy False positives Analyst workload reduction Mail flow insights in the new EAC in Exchange Online Prioritize incidents in the Microsoft Defender portal Train Your SOC Team Share explainable AI outputs with analysts to build trust Use the agent’s verdicts to guide manual investigations and reinforce learning Security Copilot Phishing Triage Agent in Microsoft Defender (Preview) Iterate and Improve Review phishing trends Update triage policies Leverage Security Copilot’s adaptive learning to stay ahead of evolving threats What is Microsoft Security Copilot? About the Author: Greetings! Jacques “Jack” here. I am excited to share this remarkable technology with our Defender community, as it has the potential to greatly enhance organizational protection. My role as a Microsoft Technical Trainer has shown me how valuable solutions like Security Copilot and Security AI Agents can be in strengthening defenses and accelerating response to threats. By sharing these advancements, I hope to empower you with the tools needed to safeguard your environment in an ever-evolving security landscape. #MicrosoftLearn #SkilledByMTTNo More Guesswork—Copilot Makes Azure Security Crystal Clear
Elevating Azure Security and Compliance In today’s rapidly evolving digital landscape, security and compliance are more critical than ever. As organizations migrate workloads to Azure, the need for robust security frameworks and proactive compliance strategies grows. Security Copilot, integrated with Azure, is transforming how technical teams approach these challenges, empowering users to build secure, compliant environments with greater efficiency and confidence. As a security expert, I’d like to provide clear guidance on how to effectively utilize Security Copilot in the ever-evolving landscape of security and compliance. Security Copilot is a premium offering; it includes advanced capabilities that go beyond standard Azure security tools. These features may require specific licensing or subscription tiers. It provides deeper insights, enhanced automation, and tailored guidance for complex security scenarios. Below, I’ll highlight a range of security topics with sample Copilot prompts that you can use to help create a more secure and compliant environment. Getting Started with Microsoft Security Copilot Before leveraging the advanced capabilities of Security Copilot, it's important to understand the foundational requirements and setup steps: Azure Subscription Requirement Security Copilot is not automatically available in all Azure subscriptions. To use it, your organization must have an active Azure subscription. This is necessary to provision Security Compute Units (SCUs), which are the core resources that power Copilot workloads. Provisioning Security Compute Units (SCUs) SCUs are billed hourly and can be scaled based on workload needs. At least one SCU must be provisioned to activate Security Copilot. You can manage SCUs via the Azure portal or the Security Copilot portal, adjusting capacity as needed for performance and cost optimization. Role-Based Access Control To set up and manage Security Copilot: You need to be an Azure Owner or Contributor to provision SCUs. Users must be assigned appropriate Microsoft Entra roles (e.g., Security Administrator) to access and interact with Copilot features. Embedded Experience Security Copilot can be used as a standalone tool or embedded within other Microsoft services like Defender for Endpoint, Intune, and Purview, offering unified security management experience. Data Privacy and Security: Foundational Best Practices Why settle for generic security advice when Security Copilot delivers prioritized, actionable guidance backed by Microsoft’s best practices? Copilot doesn’t just recommend security measures, it actively helps you implement them, leveraging advanced features like encryption and granular access controls to safeguard every layer of your Azure environment. While Security Copilot doesn’t directly block threats like a firewall or Web Application Firewall (WAF), it enhances data integrity and confidentiality by analyzing security signals across Azure, identifying vulnerabilities, and guiding teams with prioritized, actionable recommendations. It helps implement encryption, access controls, and compliance-aligned configurations, while integrating with existing security tools to interpret logs and suggest containment strategies. By automating investigations and supporting secure-by-design practices, Copilot empowers organizations to proactively reduce breach risks and maintain a strong security posture. Secure Coding and Developer Productivity While Security Copilot supports secure coding by identifying vulnerabilities like SQL injection, Cross-Site Scripting (XSS), and buffer overflows, it is not a direct replacement for traditional code scanning tools, instead, it complements these tools by leveraging telemetry from integrated Microsoft services and applying AI-driven insights to prioritize risks and guide remediation. Copilot enhances developer productivity by interpreting signals, offering tailored recommendations, and embedding security practices throughout the software lifecycle. Understanding Security Protocols and Mechanisms Azure’s security stands on robust protocols and mechanisms but understanding them shouldn’t require a cryptography degree. Security Copilot demystifies encryption, authentication, and secure communications—making complex concepts accessible and actionable. With Security Copilot as your guide, teams can confidently configure Azure resources and respond to threats with informed, best-practice decisions. Compliance and Regulatory Alignment Regulatory requirements such as GDPR, HIPAA, and PCI-DSS don’t have to slow you down. Security Copilot streamlines Azure compliance with ready-to-use templates, clear guidelines, and robust documentation support. From maintaining audit logs to generating compliance reports, Security Copilot keeps every action tracked and organized—reducing non-compliance risk and making audits a breeze. Incident Response Planning No security strategy is complete without a solid incident response plan. Security Copilot equips Azure teams with detailed protocols for identifying, containing, and mitigating threats. It enhances Security Information and Event Management (SIEM) and Security Orchestration, Automation, and Response (SOAR) solutions through ready-made playbooks tailored to diverse scenarios. With built-in incident simulations, Copilot enables teams to rehearse and refine their responses—minimizing breach impact and accelerating recovery. Security Best Practices for Azure Staying ahead of threats means never standing still. Security Copilot builds on Azure’s proven security features—like multi-factor authentication, regular updates, and least privilege access—by automating their implementation, monitoring usage patterns, and surfacing actionable insights. It connects with tools like Microsoft Defender and Entra ID to interpret signals, recommend improvements, and guide teams in real time. With Copilot, your defenses don’t just follow best practices, they evolve dynamically to meet emerging threats, keeping your team sharp and your environment secure. Integrating Copilot into Your Azure Security Strategy Security Copilot isn’t just a technical tool—it’s your strategic partner for Azure security. By weaving Copilot into your workflows, you unlock advanced security enhancements, optimized code, and robust privacy protection. Its holistic approach ensures security and compliance are seamlessly integrated into every corner of your Azure environment. Conclusion Security Copilot is changing the game for Azure security and compliance. By blending secure coding, advanced security expertise, regulatory support, incident response playbooks, and best practices, Copilot empowers technical teams to build resilient, compliant cloud environments. As threats evolve, Copilot keeps your data protected and your organization ahead of the curve. Ready to take your Azure security and compliance to the next level? Start leveraging Security Copilot today to empower your team, streamline operations, and stay ahead of evolving threats. Dive deeper into best practices, hands-on tutorials, and expert guidance to maximize your security posture and unlock the full potential of Copilot in your organization. Explore, learn, and secure your cloud—your journey starts now! Further Reading & Resources Microsoft Security Copilot documentation Get started with Microsoft Security Copilot Microsoft Copilot in Azure Overview Security best practices and patterns - Microsoft Azure Azure compliance documentation Copilot Learning Hub Microsoft Security Copilot Blog Author: Microsoft Principal Technical Trainer, https://www.linkedin.com/in/eliasestevao/ #MicrosoftLearn #SkilledByMTTPlanning your move to Microsoft Defender portal for all Microsoft Sentinel customers
In November 2023, Microsoft announced our strategy to unify security operations by bringing the best of XDR and SIEM together. Our first step was bringing Microsoft Sentinel into the Microsoft Defender portal, giving teams a single, comprehensive view of incidents, reducing queue management, enriching threat intel, streamlining response and enabling SOC teams to take advantage of Gen AI in their day-to-day workflow. Since then, considerable progress has been made with thousands of customers using this new unified experience; to enhance the value customers gain when using Sentinel in the Defender portal, multi-tenancy and multi-workspace support was added to help customers with more sophisticated deployments. Our mission is to unify security operations by bringing all your data, workflows, and people together to unlock new capabilities and drive better security outcomes. As a strong example of this, last year we added extended posture management, delivering powerful posture insights to the SOC team. This integration helps build a closed-loop feedback system between your pre- and post-breach efforts. Exposure Management is just one example. By bringing everything together, we can take full advantage of AI and automation to shift from a reactive to predictive SOC that anticipates threats and proactively takes action to defend against them. Beyond Exposure Management, Microsoft has been constantly innovating in the Defender experience, adding not just SIEM but also Security Copilot. The Sentinel experience within the Defender portal is the focus of our innovation energy and where we will continue to add advanced Sentinel capabilities going forward. Onboarding to the new unified experience is easy and doesn’t require a typical migration. Just a few clicks and permissions. Customers can continue to use Sentinel in the Azure portal while it is available even after choosing to transition. Today, we’re announcing that we are moving to the next phase of the transition with a target to retire the Azure portal for Microsoft Sentinel by July 1, 2026. Customers not yet using the Defender portal should plan their transition accordingly. “Really amazing to see that coming, because cross querying with tables in one UI is really cool! Amazing, big step forward to the unified [Defender] portal.” Glueckkanja AG “The biggest benefit of a unified security operations solution (Microsoft Sentinel + Microsoft Defender XDR) has been the ability to combine data in Defender XDR with logs from third party security tools. Another advantage developed has been to eliminate the need to switch between Defender XDR and Microsoft Sentinel portals, now having a single pane of glass, which the team has been wanting for some years.” Robel Kidane, Group Information Security Manager, Renishaw PLC Delivering the SOC of the future Unifying threat protection, exposure management and security analytics capabilities in one pane of glass not only streamlines the user experience, but also enables Sentinel customers to realize security outcomes more efficiently: Analyst efficiency: A single portal reduces context switching, simplifies workflows, reduces training overhead, and improves team agility. Integrated insights: SOC-focused case management, threat intelligence, incident correlation, advanced hunting, exposure management, and a prioritized incident queue enriched with business and sensitivity context—enabling faster, more informed detection and response across all products. SOC optimization: Security controls that can be adjusted as threats and business priorities change to control costs and provide better coverage and utilization of data, thus maximizing ROI from the SIEM. Accelerated response: AI-driven detection and response which reduces mean time to respond (MTTR) by 30%, increases security response efficiency by 60%, and enables embedded Gen AI and agentic workflows. What’s next: Preparing for the retirement of the Sentinel Experience in the Azure Portal Microsoft is committed to supporting every single customer in making that transition over the next 12 months. Beginning July 1, 2026, Sentinel users will be automatically redirected to the Defender portal. After helping thousands of customers smoothly make the transition, we recommend that security teams begin planning their migration and change management now to ensure continuity and avoid disruption. While the technical process is very straightforward, we have found that early preparation allows time for workflow validation, training, and process alignment to take full advantage of the new capabilities and experience. Tips for a Successful Migration to Microsoft Defender 1. Leverage Microsoft’s help: Leverage Microsoft documentation, instructional videos, guidance, and in-product support to help you be successful. A good starting point is the documentation on Microsoft Learn. 2. Plan early: Engage stakeholders early including SOC and IT Security leads, MSSPs, and compliance teams to align on timing, training and organizational needs. Make sure you have an actionable timeline and agreement in the organization around when you can prioritize this transition to ensure access to the full potential of the new experience. 3. Prepare your environment: Plan and design your environment thoroughly. This includes understanding the prerequisites for onboarding Microsoft Sentinel workspaces, reviewing and deciding on access controls, and planning the architecture of your tenant and workspace. Proper planning will ensure a smooth transition and help avoid any disruptions to your security operations. 4. Leverage Advanced Threat Detection: The Defender portal offers enhanced threat detection capabilities with advanced AI and machine learning for Microsoft Sentinel. Make sure to leverage these features for faster and more accurate threat detection and response. This will help you identify and address critical threats promptly, improving your overall security posture. 5. Utilize Unified Hunting and Incident Management: Take advantage of the enhanced hunting, incident, and investigation capabilities in Microsoft Defender. This provides a comprehensive view for more efficient threat detection and response. By consolidating all security incidents, alerts, and investigations into a single unified interface, you can streamline your operations and improve efficiency. 6. Optimize Cost and Data Management The Defender portal offers cost and data optimization features, such as SOC Optimization and Summary Rules. Make sure to utilize these features to optimize your data management, reduce costs, and increase coverage and SIEM ROI. This will help you manage your security operations more effectively and efficiently. Unleash the full potential of your Security team The unified SecOps experience available in the Defender portal is designed to support the evolving needs of modern SOCs. The Defender portal is not just a new home for Microsoft Sentinel - it’s a foundation for integrated, AI-driven security operations. We’re committed to helping you make this transition smoothly and confidently. If you haven’t already joined the thousands of security organizations that have done so, now is the time to begin. Resources AI-Powered Security Operations Platform | Microsoft Security Microsoft Sentinel in the Microsoft Defender portal | Microsoft Learn Shifting your Microsoft Sentinel Environment to the Defender Portal | Microsoft Learn Microsoft Sentinel is now in Defender | YouTubeMicrosoft Sentinel’s New Data Lake: Cut Costs & Boost Threat Detection
Microsoft Sentinel is leveling up! Already a trusted cloud-native Security Information and Event Management (SIEM) and Security Orchestration, Automation and Response (SOAR) solution, it empowers security teams to detect, investigate, and respond to threats with speed and precision. Now, with the introduction of its new Data Lake architecture, Sentinel is transforming how security data is stored, accessed, and analyzed, bringing unmatched flexibility and scale to threat investigation. Unlike Microsoft Fabric OneLake, which supports analytics across the organization, Sentinel’s Data Lake is purpose-built for security. It centralizes raw structured, semi-structured, and unstructured data in its original format, enabling advanced analytics without rigid schemas. This article is written by someone who’s spent years helping security teams navigate Microsoft’s evolving ecosystem, translating complex capabilities into practical strategies. What follows is a hands-on look at the key features, benefits, and challenges of Sentinel’s Data Lake, designed to help you make the most of this powerful new architecture. Current Sentinel Features To tackle the challenges security teams, face today—like explosive data growth, integration of varied sources, and tight compliance requirements—organizations need scalable, efficient architectures. Legacy SIEMs often become costly and slow when analyzing multi-year data or correlating diverse events. Security data lakes address these issues by enabling seamless ingestion of logs from any source, schema-on-read flexibility, and parallelized queries over massive datasets. This schema-on-read allows SOC analysts to define how data is interpreted at the time of analysis, rather than when it is stored. This means analysts can flexibly adapt queries and threat detection logic to evolving threats, without reformatting historical data making investigations more agile and responsive to change. This empowers security operations to conduct deep historical analysis, automate enrichment, and apply advanced analytics, such as machine learning, while retaining strict control over data access and residency. Ultimately, decoupling storage and compute allows teams to boost detection and response speed, maintain compliance, and adapt their Security Operation Center (SOC) to future security demands. As organizations manage increasing data and limited budgets, many are moving from legacy SIEMs to advanced cloud-native options. Microsoft Sentinel’s Data Lake separates storage from computing, offering scalable and cost-effective analytics and compliance. For instance, storing 500 TB of logs in Sentinel Data Lake can cut costs by 60–80% compared to Log Analytics, due to lower storage costs and flexible retention. Integration with modern tools and open formats enables efficient threat response and regulatory compliance. Microsoft Sentinel data lake pricing (preview) Sentinel Data Lake Use Cases Log Retention: Long-term retention of security logs for compliance and forensic investigations Hunting: Advanced threat hunting using historical data Interoperability: Integration with Microsoft Fabric and other analytics platforms Cost: Efficient storage prices for high-volume data sources How Microsoft Sentinel Data Lake Helps Microsoft Sentinel’s Data Lake introduces a powerful paradigm shift for security operations by architecting the separation of storage and compute, enabling organizations to achieve petabyte-scale data retention without the traditional overhead and cost penalties of legacy SIEM solutions. Built atop highly scalable, cloud-native infrastructure, Sentinel Data Lake empowers SOCs to ingest telemetry from virtually unlimited sources ranging from on-premises firewalls, proxies, and endpoint logs to SaaS, IaaS, and PaaS environments—while leveraging schema-on-read, a method that allows analysts to define how data is interpreted at query time rather than when it is stored, offering greater flexibility in analytics. For example, a security analyst can adapt to the way historical data is examined as new threats emerge, without needing to reformat or restructure the data stored in the Data Lake. From Microsoft Learn – Retention and data tiering Storing raw security logs in open formats like Parquet (this is a columnar storage file format optimized for efficient data compression and retrieval, commonly used in big data processing frameworks like Apache Spark and Hadoop) enables easy integration with analytics tools and Microsoft Fabric, letting analysts efficiently query historical data using KQL, SQL, or Spark. This approach eliminates the need for complex ETL and archived data rehydration, making incident response faster; for instance, a SOC analyst can quickly search for years of firewall logs for threat detection. From Microsoft Learn – Flexible querying with Kusto Query Language Granular data governance and access controls allow organizations to manage sensitive information and meet legal requirements. Storing raw security logs in open formats enables fast investigations of long-term data incidents, while automated lifecycle management reduces costs and ensures compliance. Data Lakes integrate with Microsoft platforms and other tools for unified analytics and security. Machine learning helps detect unusual login activity across years, overcoming previous storage issues. From Microsoft Learn – Powerful analytics using Jupyter notebooks Pros and Cons The following table highlights the advantages and potential opportunities that Microsoft Sentinel Data Lake offers. This follows the same Pay-As-You-Go pricing model as currently available with Sentinel. Pros Cons License Needed Scalable, cost-effective long-term retention of security data Requires adaptation to new architecture Pay-As-You-Go model Seamless integration with Microsoft Fabric and open data formats Initial setup and integration may involve a learning curve Pay-As-You-Go model Efficient processing of petabyte-scale datasets Transitioning existing workflows may require planning Pay-As-You-Go model Advanced analytics, threat hunting, and AI/ML across historical data Some features may depend on integration with other services Pay-As-You-Go model Supports compliance use cases with robust data governance and audit trails Complexity in new data governance features Pay-As-You-Go model Microsoft Sentinel Data Lake solution advances cloud-native security by overcoming traditional SIEM limitations, allowing organizations to better retain, analyze, and respond to security data. As cyber threats grow, Sentinel Data Lake offers flexible, cost-efficient storage for long-term retention, supporting detection, compliance, and audits without significant expense or complexity. Quick Guide: Deploy Microsoft Sentinel Data Lake Assess Needs: Identify your security data volume, retention, and compliance requirements - Sentinel Data Lake Overview. Prepare Environment: Ensure Azure permissions and workspace readiness - Onboarding Guide. Enable Data Lake: Use Azure CLI or Defender portal to activate - Setup Instructions. Ingest & Import Data: Connect sources and migrate historical logs - Microsoft Sentinel Data Connectors. Integrate Analytics: Use KQL, notebooks, and Microsoft Fabric for scalable analysis - Fabric Overview Train & Optimize: Educate your team and monitor performance - Best Practices. About the Author: Hi! Jacques “Jack” here, I’m a Microsoft Technical Trainer at Microsoft. I wanted to share this as it’s something I often asked during my Security Trainings. This improves the already impressive Microsoft Sentinel feature stack helping the Defender Community to secure their environment in this ever-growing hacked world. I’ve been working with Microsoft Sentinel since September 2019, and I have been teaching learners about this SIEM since March 2020. I have experience using Security Copilot and Security AI Agents, which have been effective in improving my incident response and compromise recovery times.Microsoft Purview: The Ultimate AI Data Security Solution
Introduction AI is transforming the way enterprises operate, however with great innovation comes great responsibility. I’ve spent the last few years helping organizations secure their data with tools like Azure Information Protection, Data Loss Prevention, and now Microsoft Purview. As generative AI tools like Microsoft Copilot become embedded in everyday workflows, the need for clear governance and robust data protection is more urgent than ever. Through this blog post, let's explore how Microsoft Purview can help organizations stay ahead of securing AI interactions without slowing down innovation. What’s the Issue? AI agents are increasingly used to process sensitive data, often through natural language prompts. Without proper oversight, this can lead to data oversharing, compliance violations, and security risks. Why It’s Urgent? According to the recent trends of 2025, over half of corporate users bring their own AI tools to work, often consumer-grade apps like ChatGPT or DeepSeek. These tools bypass enterprise protections, making it difficult to monitor and control data exposure. Use Cases Enterprise AI Governance: Apply consistent policies across Microsoft and third-party AI tools. Compliance Auditing: Generate audit logs for AI interactions to meet regulatory requirements. Risk Mitigation: Block risky uploads and enforce adaptive protection based on user behavior. How Microsoft Purview Solves It Data Security Posture Management (DSPM) for AI Purview’s DSPM for AI provides a centralized dashboard to monitor AI activity, assess data risks, and enforce compliance policies across Copilots, agents, and third-party AI apps. It correlates data classification, user behavior, and policy coverage to surface real-time risks, such as oversharing via AI agents, and generates actionable recommendations to remediate gaps. DSPM integrates with tools like Microsoft Security Copilot for AI-assisted investigations and supports automated scanning, trend analytics, and posture reporting. It also extends protection to third-party AI tools like ChatGPT through endpoint DLP and browser extensions, ensuring consistent governance across both managed and unmanaged environments 2. Unified Protection Across AI Agents Whether you're using Microsoft 365 Copilot, Security Copilot, or Azure AI services, Purview applies consistent security and compliance controls. Agents inherit protection from their parent apps, including sensitivity labels, data loss prevention (DLP), and Insider Risk Management. Real-Time Risk Detection Purview enables real-time monitoring of prompts and responses, helping security teams detect oversharing and policy violations instantly. From Microsoft Learn – Insider Risk 4. One-Click Policy Activation Administrators can leverage Microsoft Purview’s Data Security Posture Management (DSPM) for AI to rapidly deploy comprehensive security and compliance controls via one-click policy activation. This streamlined mechanism enables organizations to enforce prebuilt policy templates across AI ecosystems, ensuring prompt implementation of data loss prevention (DLP), sensitivity labeling, and Insider Risk Management on both Microsoft and third-party AI services. Through DSPM’s unified policy orchestration layer, security teams gain granular telemetry into prompt and response flows, real-time policy enforcement, and detailed incident reporting. Automated analytics continuously assess risk posture, enabling adaptive policy adjustments and scalable governance as new AI tools and user workflows are introduced into the enterprise environment. Please note: After implementing policy changes, it can take up to 24 hours for changes to become visible and take full effect across your environment. From Microsoft Learn – Purview Data Security Posture Management (DSPM) portal 5. Support for Third-Party AI Apps Purview extends robust data security and compliance to browser-based AI tools such as ChatGPT and Google Gemini by employing endpoint Data Loss Prevention (DLP) and browser extensions that monitor and control data flows in real time. Through Microsoft Purview’s Data Security Posture Management (DSPM) for AI, organizations can implement granular controls for sensitive data accessed during both Microsoft-native and third-party AI interactions. DSPM offers continuous discovery and classification of data assets, linking AI prompts and responses to their original data sources to automatically enforce data protection policies, including sensitivity labeling, adaptive access controls, and comprehensive content inspection, contextually for each AI transaction. For unsanctioned AI services reached via browsers, the Purview browser extension inspects both input and output, enabling endpoint DLP to block, alert, or redact sensitive material instantly, thus preventing unauthorized uploads, downloads, or copy/paste activities. Security teams benefit from rich telemetry on AI usage patterns, which integrate with user risk profiles and anomaly detection to identify and flag suspicious attempts to extract confidential information. Close integration with Microsoft Security Copilot and automated analytics further enhances visibility across all AI data flows, supporting incident response, audit, and compliance reporting needs. Purview’s adaptive policy orchestration ensures that evolving AI services and workflows are continuously assessed for risk, and that controls are dynamically aligned with business, regulatory, and security requirements, enabling scalable, policy-driven governance for the expanding enterprise AI ecosystem. Pros and Cons The following table outlines the key advantages and potential limitations of implementing AI and agent data security controls within Microsoft Purview. Pros Cons License Needed Centralized AI governance Requires proper licensing and setup Microsoft 365 E5 or equivalent Purview add-on license Real-time risk detection May need browser extensions for full coverage Microsoft 365 E5 or Purview add-on Supports both Microsoft and third-party AI apps Some features limited to enterprise versions Microsoft 365 E5, E5 Compliance, or equivalent Purview add-on Conclusion Microsoft Purview offers a comprehensive solution for securing AI agents and their data interactions. By leveraging DSPM for AI, organizations can confidently adopt AI technologies while maintaining control over sensitive information. Explore Microsoft Purview’s DSPM for AI here. Start by assessing your current AI usage and activate one-click policies to secure your environment today! FAQ 1. What is the purpose of Microsoft Purview’s AI and agent data security controls? The purpose is to ensure that sensitive data accessed or processed by AI systems and agents is governed, protected, and monitored using Microsoft Purview’s compliance and security capabilities. Microsoft Purview data security and compliance protection 2. How does Microsoft Purview help secure AI-generated content? Microsoft Purview applies data loss prevention (DLP), sensitivity labels, and information protection policies to AI-generated content, ensuring it adheres to organizational compliance standards. Microsoft Purview Information Protection 3. Can Microsoft Purview track and audit AI interactions with sensitive data? Yes. Microsoft Purview provides audit logs and activity explorer capabilities that allow organizations to monitor how AI systems and agents interact with sensitive data. Search the audit log 4. What role do sensitivity labels play in AI data governance? Sensitivity labels classify and protect data based on its sensitivity level. When applied, they enforce encryption, access restrictions, and usage rights, even when data is processed by AI. Learn about sensitivity labels 5. How does Microsoft Purview integrate with Copilot and other AI tools? Microsoft Purview extends its data protection and compliance capabilities to Microsoft 365 Copilot and other AI tools by ensuring that data accessed by these tools is governed under existing policies. Microsoft 365 admin center Microsoft 365 Copilot usage 6. Are there specific controls for third-party AI agents? Yes. Microsoft Purview supports conditional access, DLP, and access reviews to manage and monitor third-party AI agents that interact with organizational data. What is Conditional Access in Microsoft Entra ID? 7. How can organizations ensure AI usage complies with regulatory requirements? By using Microsoft Purview’s compliance manager, organizations can assess and manage regulatory compliance risks associated with AI usage. Microsoft Purview Compliance Manager About the Author: Hi! Jacques “Jack” here, I’m a Microsoft Technical Trainer at Microsoft. I wanted to share a topic that is often top of mind, AI governance. I’ve been working with Microsoft Purview since its launch in 2022, building on prior experience with Azure Information Protection and Data Loss Prevention. I also have great expertise with Generative AI technologies since their public release in November 2022, including Microsoft Copilot and other enterprise-grade AI solutions.
Introducing Microsoft Sentinel data lake
Today, we announced a significant expansion of Microsoft Sentinel’s capabilities through the introduction of Sentinel data lake, now rolling out in public preview. Security teams cannot defend what they cannot see and analyze. With exploding volumes of security data, organizations are struggling to manage costs while maintaining effective threat coverage. Do-it-yourself security data architectures have perpetuated data silos, which in turn have reduced the effectiveness of AI solutions in security operations. With Sentinel data lake, we are taking a major step to address these challenges. Microsoft Sentinel data lake enables a fully managed, cloud-native, data lake that is purposefully designed for security, right inside Sentinel. Built on a modern lake architecture and powered by Azure, Sentinel data lake simplifies security data management, eliminates security data silos, and enables cost-effective long-term security data retention with the ability to run multiple forms of analytics on a single copy of that data. Security teams can now store and manage all security data. This takes the market-leading capabilities of Sentinel SIEM and supercharges it even further. Customers can leverage the data lake for retroactive TI matching and hunting over a longer time horizon, track low and slow attacks, conduct forensics analysis, build anomaly insights, and meet reporting & compliance needs. By unifying security data, Sentinel data lake provides the AI ready data foundation for AI solutions. Let’s look at some of Sentinel data lake’s core features. Simplified onboarding and enablement inside Defender Portal: Customers can easily discover and enable the new data lake from within the Defender portal, either from the banner on the home page or from settings. Setting up a modern data lake now is just a click away, empowering security teams to get started quickly without a complex setup. Simplified security data management: Sentinel data lake works seamlessly with existing Sentinel connectors. It brings together security logs from Microsoft services across M365, Defender, Azure, Entra, Purview, Intune plus third-party sources like AWS, GCP, network and firewall data from 350+ connectors and solutions. The data lake supports Sentinel’s existing table schemas while customers can also create custom connectors to bring raw data into the data lake or transform it during ingestion. In the future, we will enable additional industry-standard schemas. The data lake expands beyond just activity logs by including a native asset store. Critical asset information is added to the data lake using new Sentinel data connectors for Microsoft 365, Entra, and Azure, enabling a single place to analyze activity and asset data enriched with Threat intelligence. A new table management experience makes it easy for customers to choose where to send and store data, as well as set related retention policies to optimize their security data estate. Customers can easily send critical, high-fidelity security data to the analytics tier or choose to send high-volume, low fidelity logs to the new data lake tier. Any data brought into the analytics tier is automatically mirrored into the data lake at no additional charge, making data lake the central location for all security data. Advanced data analysis capabilities over data in the data lake: Sentinel data lake stores all security data in an open format to enable analysts to do multi-modal security analytics on a single copy of data. Through the new data lake exploration experience in the Defender portal, customers can leverage Kusto query language to analyze historical data using the full power of Kusto. Since the data lake supports the Sentinel table schema, advanced hunting queries can be run directly on the data lake. Customers can also schedule long-running jobs, either once or on a schedule, that perform complex analysis on historical data for in-depth security insights. These insights generated from the data lake can be easily elevated to analytics tier and leveraged in Sentinel for threat investigation and response. Additionally, as part of the public preview, we are also releasing a new Sentinel Visual Studio Code extension that enables security teams to easily connect to the same data lake data and use Python notebooks, as well as spark and ML libraries to deeply analyze lake data for anomalies. Since the environment is fully managed, there is no compute infrastructure to set up. Customers can just install the Visual Studio Code extension and use AI coding agents like GitHub Copilot to build a notebook and execute it in the managed environment. These notebooks can also be scheduled as jobs and the resulting insights can be elevated to analytics tier and leveraged in Sentinel for threat investigation and response. Flexible business model: Sentinel data lake enables customers to separate their data ingestion and retention needs from their security analytics needs, allowing them to ingest and store data cost effectively and then pay separately when analyzing data for their specific needs. Let’s put this all together and show an example of how a customer can operationalize and derive value from the data lake for retrospective threat intelligence matching in Microsoft Sentinel. Network logs are typically high-volume logs but can often contain key insights for detecting initial entry point of an attack, command and control connection, lateral movement or an exfiltration attempt. Customers can now send these high-volume logs to the data lake tier. Next, they can create a python notebook that can join latest threat intelligence from Microsoft Defender Threat Intelligence to scan network logs for any connections to/from a suspicious IP or domain. They can schedule this notebook to run as a scheduled job, and any insights can then be promoted to analytics tiers and leveraged to enrich ongoing investigation, hunts, response or forensics analysis. All this is possible cost-effectively without having to set up any complex infrastructure, enabling security teams to achieve deeper insights. This preview is now rolling out for customers in Defender portal in our supported regions. To learn more, check out our Mechanics video and our documentation or talk to your account teams. Get started today Join us as we redefine what’s possible in security operations: Onboard Sentinel data lake: https://aka.ms/sentineldatalakedocs Explore our pricing: https://aka.ms/sentinel/pricingblog For the supported regions, please refer to https://aka.ms/sentinel/datalake/geos Learn more about our MDTI news: http://aka.ms/mdti-convergence General Availability of Auxiliary Logs and Reduced Pricing
