data retention
21 TopicsWhat’s New in Microsoft Sentinel: November 2025
Welcome to our new Microsoft Sentinel blog series! We’re excited to launch a new blog series focused on Microsoft Sentinel. From the latest product innovations and feature updates to industry recognition, success stories, and major events, you’ll find it all here. This first post kicks off the series by celebrating Microsoft’s recognition as a Leader in the 2025 Gartner Magic Quadrant for SIEM 1 . It also introduces the latest innovations designed to deliver measurable impact and empower defenders with adaptable, collaborative tools in an evolving threat landscape. Microsoft is recognized as a Leader in 2025 Gartner Magic Quadrant for Security Information and Event Management (SIEM) Microsoft Sentinel continues to drive security innovation—and the industry is taking notice. Microsoft was named a leader in the 2025 Gartner Magic Quadrant for Security Information and Event Management (SIEM) 1 , published on October 8, 2025. We believe this acknowledgment reinforces our commitment to helping organizations stay secure in a rapidly changing threat landscape. Read blog for more information. Take advantage of M365 E5 benefit and Microsoft Sentinel promotional pricing Microsoft 365 E5 benefit Customers with Microsoft 365 E5, A5, F5, or G5 licenses automatically receive up to 5 MB of free data ingestion per user per day, covering key security data sources like Azure AD sign-in logs and Microsoft Cloud App Security discovery logs—no enrollment required. Read more about M365 benefits for Microsoft Sentinel. New 50GB promotional pricing To make Microsoft Sentinel more accessible to small and mid-sized organizations, we introduced a new 50 GB commitment tier in public preview, with promotional pricing starting October 1, 2025, through March 31, 2026. Customers who choose the 50 GB commitment tier during this period will maintain their promotional rate until March 31, 2027. Available globally with regional variations in regional pricing it is accessible through EA, CSP, and Direct channels. For more information see Microsoft Sentinel pricing page. Partner Integrations: Strengthening TI collaboration and workflow automation Microsoft Sentinel continues to expand its ecosystem with powerful partner integrations that enhance security operations. With Cyware, customers can now share threat intelligence bi-directionally across trusted destinations, ISACs, and multi-tenant environments—enabling real-time intelligence exchange that strengthens defenses and accelerates coordinated response. Learn more about the Cyware integration. Learn more about the Cyware integration here. Meanwhile, BlinkOps integration combined with Sentinel’s SOAR capabilities empowers SOC teams to automate repetitive tasks, orchestrate complex playbooks, and streamline workflows end-to-end. This automation reduces operational overhead, cuts Mean Time to Respond (MTTR) and frees analysts for strategic threat hunting. Learn more about the BlinkOps integration. Learn more about the BlinkOps integration. Harnessing Microsoft Sentinel Innovations Security is being reengineered for the AI era, moving beyond static, rule-based controls and reactive post-breach response toward platform-led, machine-speed defense. To overcome fragmented tools, sprawling signals, and legacy architectures that cannot keep pace with modern attacks, Microsoft Sentinel has evolved into both a SIEM and a unified security platform for agentic defense. These updates introduce architectural enhancements and advanced capabilities that enable AI-driven security operations at scale, helping organizations detect, investigate, and respond with unprecedented speed and precision. Microsoft Sentinel graph – Public Preview Unified graph analytics for deeper context and threat reasoning. Microsoft Sentinel graph delivers an interactive, visual map of entity relationships, helping analysts uncover hidden attack paths, lateral movement, and root causes for pre- and post-breach investigations. Read tech community blog for more details. Microsoft Sentinel Model Context Protocol (MCP) server – Public Preview Context is key to effective security automation. Microsoft Sentinel MCP server introduces a standardized protocol for building context-aware solutions, enabling developers to create smarter integrations and workflows within Sentinel. This opens the door to richer automation scenarios and more adaptive security operations. Read tech community blog for more details. Enhanced UEBA with New Data Sources – Public Preview We are excited to announce support for six new sources in our user entity and behavior analytics algorithm, including AWS, GCP, Okta, and Azure. Now, customers can gain deeper, cross-platform visibility into anomalous behavior for earlier and more confident detection. Read our blog and check out our Ninja Training to learn more. Developer Solutions for Microsoft Sentinel platform – Public Preview Expanded APIs, solution templates, and integration capabilities empower developers to build and distribute custom workflows and apps via Microsoft Security Store. This unlocks faster innovation, streamlined operations, and new revenue opportunities, extending Sentinel beyond out-of-the-box functionality for greater agility and resilience. Read tech community blog for more details. Growing ecosystem of Microsoft Sentinel data connectors We are excited to announce the general availability of four new data connectors: AWS Server Access Logs, Google Kubernetes Engine, Palo Alto CSPM, and Palo Alto Cortex Xpanse. Visit find your Microsoft Sentinel data connector page for the list of data connectors currently supported. We are also inviting Private Previews for four additional connectors: AWS EKS, Qualys VM KB, Alibaba Cloud Network, and Holm Security towards our commitment to expand the breadth and depth to support new data sources. Our customer support team can help you sign up for previews. New agentless data connector for Microsoft Sentinel Solution for SAP applications We’re excited to announce the general availability of a new agentless connector for Microsoft Sentinel solution for SAP applications, designed to simplify integration and enhance security visibility. This connector enables seamless ingestion of SAP logs and telemetry directly into Microsoft Sentinel, helping SOC teams monitor critical business processes, detect anomalies, and respond to threats faster—all while reducing operational overhead. Events, Webinars and Training Stay connected with the latest security innovation and best practices. From global conferences to expert-led sessions, these events offer opportunities to learn, network, and explore how Microsoft is shaping AI-driven, end-to-end security for the modern enterprise. Microsoft Ignite 2025 Security takes center stage at Microsoft Ignite, with dedicated sessions and hands-on experiences for security professionals and leaders. Join us in San Francisco, November 17–21, 2025, or online, to explore our AI-first, end-to-end security platform designed to protect identities, devices, data, applications, clouds, infrastructure—and critically—AI systems and agents. Register today! Microsoft Security Webinars Stay ahead of emerging threats and best practices with expert-led webinars from the Microsoft Security Community. Discover upcoming sessions on Microsoft Sentinel SIEM & platform, Defender, Intune, and more. Sign up today and be part of the conversation that shapes security for everyone. Learn more about upcoming webinars. Onboard Microsoft Sentinel in Defender – Video Series Microsoft leads the industry in both SIEM and XDR, delivering a unified experience that brings these capabilities together seamlessly in the Microsoft Defender portal. This integration empowers security teams to correlate insights, streamline workflows, and strengthen defenses across the entire threat landscape. Ready to get started? Explore our video series to learn how to onboard your Microsoft Sentinel experience and unlock the full potential of integrated security. Watch Microsoft Sentinel is now in Defender video series. MDTI Convergence into Microsoft Sentinel & Defender XDR overview Discover how Microsoft Defender Threat Intelligence Premium is transforming cybersecurity by integrating into Defender XDR, Sentinel, and the Defender portal. Watch this session to learn about new features, expanded access to threat intelligence, and how these updates strengthen your security posture. Partner Sentinel Bootcamp Transform your security team from Sentinel beginners to advanced practitioners. This comprehensive 2-day bootcamp helps participants master architecture design, data ingestion strategies, multi-tenant management, and advanced analytics while learning to leverage Microsoft's AI-first security platform for real-world threat detection and response. Register here for the bootcamp. Looking to dive deeper into Microsoft Sentinel development? Check out the official https://aka.ms/AppAssure_SentinelDeveloper. It’s the central reference for developers and security teams who want to build custom integrations, automate workflows, and extend Sentinel’s capabilities. Bookmark this link as your starting point for hands-on guidance and tools. Stay Connected Check back each month for the latest innovations, updates, and events to ensure you’re getting the most out of Microsoft Sentinel. 1 Gartner® Magic Quadrant™ for Security Information and Event Management, Andrew Davies, Eric Ahlm, Angel Berrios, Darren Livingstone, 8 October 20252.2KViews2likes3CommentsRun agentless SAP connector cost-efficiently
The SAP agentless connector uses SAP Integration Suite (Cloud Integration/CPI) to fetch SAP audit log data and forward it to Microsoft Sentinel. Because SAP CPI billing typically reflects message counts and data volume, you can tune the connector to control costs—while preserving reliability and timeliness. Cost reductions primarily come from sending fewer CPI messages by increasing the polling interval. The max-rows parameter is a stability safeguard that caps events per run to protect CPI resources; it is not a direct cost-optimization lever. After any change, monitor CPI execution time and resource usage. ☝️Note: It may not be feasible to increase the polling interval on busy systems processing large data volumes. Larger intervals can lengthen CPI execution time and cause truncation when event spikes exceed max-rows. Cost optimization via longer intervals generally works best on lower-utilization environments (for example, dev and test) where event volume is modest and predictable. Tunable parameters Setting Default Purpose Cost impact Risk / trade-off Polling interval 1 minute How often the connector queries SAP and triggers a CPI message. Lower message count at longer intervals → potential cost reduction. Larger batches per run can extend CPI execution time; spikes may approach max-rows after which message processing for that interval is truncated. max-rows 150,000 Upper bound on events packaged per run to protect CPI stability. None (safeguard)—does not reduce message count on its own. If too low, frequent truncation; if too high, runs may near CPI resource limits. Adjust cautiously and observe. ☝️Note: When event volume within one interval exceeds max-rows, the batch is truncated by design. Remaining events are collected on subsequent runs. Recommended approach Start with defaults. Use a 1-minute polling interval and max-rows = 150,000. Measure your baseline. Understand average and peak ingestion per minute (see KQL below). Optimize the polling interval first to reduce message count when costs are a concern. Treat max-rows as a guardrail. Change only if you consistently hit the cap; increase in small steps. Monitor after each change. Track CPI run duration, CPU/memory, retries/timeouts, and connector health in both SAP CPI and Sentinel. 💡Aim for the lowest interval that keeps CPI runs comfortably within execution-time and resource limits. Change one variable at a time and observe for at least a full business cycle. 🧐Consider the Azure Monitor Log Ingestion API limits to close the loop on your considerations. Analyze ingestion profile (KQL) ABAPAuditLog | where TimeGenerated >= ago(90d) | summarize IngestedEvents = count() by bin(UpdatedOn, 1m) | summarize MaxEvents = max(IngestedEvents), AverageEvents = toint(avg(IngestedEvents)), P95_EventsPerMin = percentile(IngestedEvents, 95) How to use these metrics? AverageEvents → indicates typical per-minute volume. P95_EventsPerMin → size for spikes: choose a polling interval such that P95 × interval (minutes) remains comfortably below max-rows. If MaxEvents × interval approaches max-rows, expect truncation and catch-up behavior—either shorten the interval or, if safe, modestly raise max-rows. Operational guidance ⚠️❗Large jumps (for example, moving from a 1-minute interval to 5 minutes and raising max-rows simultaneously) can cause CPI runs to exceed memory/time limits. Adjust gradually and validate under peak workloads (e.g., period close, audit windows). Document changes (interval, max-rows, timestamp, rationale). Alert on CPI anomalies (timeouts, retries, memory warnings). Re-evaluate regularly in higher-risk periods when SAP event volume increases. Balancing Audit Log Tuning and Compliance in SAP NetWeaver: Risks of Excluding Users and Message Classes When tuning SAP NetWeaver audit logging via transaction SM19 (older releases) or RSAU_CONFIG (newer releases), administrators can filter by user or message class to reduce log volume - such as excluding high-volume batch job users or specific event types - but these exclusions carry compliance risks: omitting audit for certain users or classes may undermine traceability, violate regulatory requirements, or mask unauthorized activities, especially if privileged or technical users are involved. Furthermore, threat hunting in Sentinel for SAP gets "crippled" due to missing insights. Best practice is to start with comprehensive logging, only apply exclusions after a documented risk assessment, and regularly review settings to ensure that all critical actions remain auditable and compliant with internal and external requirements. Cost-Efficient Long-Term Storage for Compliance Microsoft Sentinel Data Lake enables organizations to retain security logs - including SAP audit data - for up to 12 years at a fraction of traditional SIEM storage costs, supporting compliance with regulations such as NIS2, DORA and more. By decoupling storage from compute, Sentinel Data Lake allows massive volumes of security data to be stored cost-effectively in a unified, cloud-native platform, while maintaining full query and analytics capabilities for forensic investigations and regulatory reporting. This approach ensures that organizations can meet strict data retention and auditability requirements without compromising on cost or operational efficiency. Summary Use the polling interval to reduce message count (primary cost lever). Keep max-rows as a safety cap to protect CPI stability. Measure → adjust → monitor to achieve a stable, lower-cost configuration tailored to your SAP workload. Use built-in mirroring to the Sentinel Data Lake to store the SAP audit logs cost-efficient for years Next Steps Explore agentless SAP deployment config guide on Microsoft Learn Expand deployment to SAP Business Technology Platform Use the insightful blog series by AryaG for details on how to move to production with the built-in SAP content of agentless216Views0likes0CommentsComprehensive coverage and cost-savings with Microsoft Sentinel’s new data tier
Microsoft is excited to announce the public preview of a new data tier Auxiliary Logs and Summary Rules in Microsoft Sentinel to further increase security coverage for high-volume data at an affordable price.16KViews3likes2CommentsAdaptive Scopes
I'm setting up adaptive scopes in MS Purview for data retention testing, focusing on Entra groups. However, when I create a test adaptive scope using the 365 groups scope and add a query with the group's display name, it doesn't populate. Some scopes are over 7 days old, despite MS stating it can take up to 3 days for queries to sync. Does anyone have a better method for creating adaptive scopes for Entra groups?196Views0likes1CommentFetching alerts from Sentinel using logic apps
Hello everyone, I have a requirement to archive alerts from sentinel. To do that I need to do the following: Retrieve the alerts from Sentinel Send the data to an external file share As a solution, I decided to proceed with using logic apps where I will be running a script to automate this process. My questions are the following: -> Which API endpoints in sentinel are relevant to retrieve alerts or to run kql queries to get the needed data. -> I know that I will need some sort of permissions to interact with the API endpoint. What type of service account inside azure should I create and what permissions should I provision to it ? -> Is there any existing examples of logic apps interacting with ms sentinel ? That would be helpful for me as I am new to Azure. Any help is much appreciated !431Views1like4CommentsData archiving of delta table in Azure Databricks
Hi all, Currently I am researching on data archiving for delta table data on Azure platform as there is data retention policy within the company. I have studied the documentation from Databricks official (https://docs.databricks.com/en/optimizations/archive-delta.html) which is about archival support in Databricks. It said "If you enable this setting without having lifecycle policies set for your cloud object storage, Databricks still ignores files based on this specified threshold, but no data is archived." Therefore, I am thinking how to configure the lifecycle policy in azure storage account. I have read the documentation on Microsoft official (https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview) Let say the delta table data are stored in "test-container/sales" and there are lots of "part-xxxx.snappy.parquet" data file stored in that folder. Should I simply specify "tierToArchive", "daysAfterCreationGreaterThan: 1825", "prefixMatch: ["test-container/sales"]? However, I am worried that will this archive mechanism impact on normal delta table operation? Besides, I am worried that what if the parquet data file moved to archive tier contains both data created before 5 years and after 5 years, it is possible? Will it by chance move data earlier to archive tier before 5 years? Highly appreciate if someone could help me out with the questions above. Thanks in advance.335Views0likes1CommentFinOps In Microsoft Sentinel
Microsoft Sentinel's security analytics and operations data is stored in an Azure Monitor Log Analytics workspace. Billing is based on the volume of data analyzed in Microsoft Sentinel and stored in the Log Analytics workspace. The cost of both is combined in a simplified pricing tier. Microsoft 365 data sources are always free to ingest for all Microsoft Sentinel users: Billable data sources: Although alerts are free, the raw logs for Microsoft Endpoint Defender, Defender for Cloud Apps, Microsoft Entra ID sign in and audit logs, and Azure Information Protection (AIP) data types are paid: Microsoft Sentinel data retention is free for the first 90 days. Enable Microsoft Sentinel on an Azure Monitor Log Analytics workspace and the first 10 GB/day is free for 31 days. The cost for both Log Analytics data ingestion and Microsoft Sentinel analysis charges up to the 10 GB/day limit are waived during the 31-day trial period. This free trial is subject to a 20 workspace limit per Azure tenant • By default, all tables in your workspace inherit the workspace's interactive retention setting and have no archive. • You can modify the retention and archive settings of individual tables Azure Monitor Logs retains data in two states: - Interactive retention: Lets you retain Analytics logs for interactive queries of up to 2 years. - Archive: Lets you keep older, less used data in your workspace at a reduced cost. • You can access data in the archived state by using search jobs, restore and keep data in archived state for up to 12 years • Its very important for cost management in MS Sentinel when you define short data retention period, but firstly go in Log Analytics WS | Workbooks | Workspace Usage in order to see tables size Use this workbook to analyze the the sizes of the different tables in your workspace: Where can save your money? Ingestion • Carefully plan what data is sent into your Microsoft sentinel workspace • Utilize filtering mechanisms to reduce ingestions to what the SOC needs • Set daily cap (good for PoC scenarios but not recommend for production) Retention • Send data to other storage platforms that have cheaper storage costs (Azure blob storage, Azure data explorer) Compute • Shutdown Azure machine learning compute during off hours, consider using reserved instances pricing • Set quotas on your subscription and workspaces • Use low-priority virtual machine (VM) Bandwidth • Sending data across Azure regions might incur into additional costs Ingestion planning • Analyze your data sources and decides what data is needed by your SOC for detection, investigations, hunting and enrichment. Take use-driven approach • Plan your workspace design • Existing workspaces might be ingesting data not needed by the SOC • Consider using separate workspace for Microsoft Sentinel • When possible enable Defender for Servers on the same workspace where you enable Microsoft Sentinel, you get 500 MB of free data ingestion per day • If you configure your Log Analytics agent to send data to two or more different Log Analytics workspaces (multi-homing), you'll get 500-MB free data ingestion for each workspace. Retention • Microsoft Sentinel retention is charged ($0.1/GB/month) and can become a big portion of the Microsoft Sentinel cost • 1.2 TB/day ingestion with 1-year retention (East US list prices) Ingestion: ~ $89К/month Retention: ~ $33К/month • If you require more than 90 days retention, determine if you need it for the whole workspace or just some tables • Consider using other storage platform for long storage retention (Azure blob storage, Azure data explorer) Long term retention options: • Azure blob storage • Cheaper than Microsoft sentinel retention • Difficult for query • Ideal for audit/compliance purposes Azure Data explorer Stores security logs in Azure Data Explorer on a long-term basis. Minimizes costs and provides easy access when you need to query the data and stores most of the data in the cold cache, minimizing the computing cost. Log Analytics doesn't currently support exporting custom log tables. In this scenario, you can use Azure Logic Apps to export data from Log Analytics workspaces. Because Azure Data Explorer provides long-term storage, you can reduce your Sentinel retention costs with this approach and ideal for forensic investigation and hunting on older data Can achieve up to 75% saving on retention costs Instead of using Azure Data Explorer for long-term storage of security logs, you can use Storage. This approach simplifies the architecture and can help control the cost. A disadvantage is the need to rehydrate the logs for security audits and interactive investigative queries. With Azure Data Explorer, you can move data from the cold partition to the hot partition by changing a policy. This functionality speeds up data exploration. Bandwidth Sending telemetry from one Azure region to another can incur in bandwidth costs this only affect Azure VMs that send telemetry across Azure regions data sources based on diagnostics settings are not affected not a big cost component compared to ingestion or retention Example: 1000 VMs, where each generates 1GB/day, sending data from US to EU: 1000 VMs * 1GB/day *30 days/month*$0.05/GB =$1.500/month Ingestion Cost Alert Playbook Managing cost for cloud services is an essential part of ensuring that you get maximum value for your investment in solutions running on this computing platform. Azure Sentinel is no different. To help you exercise greater control over your budget for Azure Sentinel this playbook will send you an alert should you exceed a budget that you define for your Azure Sentinel Workspace within a given time-frame With the ingestion cost alert playbook, you can set up an alert based on the budget defined in your Microsoft Sentinel workspace within a given timeframe. Ingestion Anomaly Alert Playbook This playbook sends you an alert should there be an ingestion spike into your workspace. The playbook uses the series_decompose_anomalies KQL function to determine anomalous ingestion The Workspace Usage Report workbook The Workspace Usage Report workbook provides your workspace's data consumption, cost, and usage statistics. The workbook gives the workspace's data ingestion status and amount of free and billable data. You can use the workbook logic to monitor data ingestion and costs, and to build custom views and rule-based alerts. This workbook also provides granular ingestion details. The workbook breaks down the data in your workspace by data table, and provides volumes per table and entry to help you better understand your ingestion patterns. Azure pricing model – based on volume of data ingested User Entity Behavior Analytics Approximately 10% of the cost of logs selected for UEBA Reduce To change your pricing tier commitment, select one of the other tiers on the pricing page, and then select Apply. You must have Contributor or Owner role in Microsoft Sentinel to change the pricing tier costs for Microsoft Sentinel Useful links: Tools that are related to FinOps on Azure Sentinel (Azure Pricing Calculator, Azure Cost Management, Azure Advisor, TCO Calculator, Azure Hybrid Benefit Savings Calculator) https://techcommunity.microsoft.com/t5/fasttrack-for-azure/the-azure-finops-guide/ba-p/3704132 Manage and monitoring Costs for Microsoft Sentinel https://learn.microsoft.com/en-us/azure/sentinel/billing-monitor-costs Reduce costs for Microsoft Sentinel https://learn.microsoft.com/en-us/azure/sentinel/billing-reduce-costs Ingestion Cost Spike Detection Playbook https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/ingestion-cost-spike-detection-playbook/ba-p/2591301 Ingestion Cost Alert Playbook https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/ingestion-cost-alert-playbook/ba-p/2006003 Introducing Microsoft Sentinel Optimization Workbook https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/introducing-microsoft-sentinel-optimization-workbook/ba-p/39014892.2KViews1like0CommentsEdit a single watchlist item
Hi, I have a watchlist, let's say it has 3000 watchlist items present. I want to modify the value of an entry - an IP address that is no longer required. Can I edit this programmatically? At present I manually have to scroll through each page of results with a CTRL+F to find on the page which is really lame. Surely there must be a better way?715Views0likes3Comments