Understand New Sentinel Pricing Model with Sentinel Data Lake Tier
Introduction on Sentinel and its New Pricing Model Microsoft Sentinel is a cloud-native Security Information and Event Management (SIEM) and Security Orchestration, Automation, and Response (SOAR) platform that collects, analyzes, and correlates security data from across your environment to detect threats and automate response. Traditionally, Sentinel stored all ingested data in the Analytics tier (Log Analytics workspace), which is powerful but expensive for high-volume logs. To reduce cost and enable customers to retain all security data without compromise, Microsoft introduced a new dual-tier pricing model consisting of the Analytics tier and the Data Lake tier. The Analytics tier continues to support fast, real-time querying and analytics for core security scenarios, while the new Data Lake tier provides very low-cost storage for long-term retention and high-volume datasets. Customers can now choose where each data type lands—analytics for high-value detections and investigations, and data lake for large or archival types—allowing organizations to significantly lower cost while still retaining all their security data for analytics, compliance, and hunting. Please flow diagram depicts new sentinel pricing model: Now let's understand this new pricing model with below scenarios: Scenario 1A (PAY GO) Scenario 1B (Usage Commitment) Scenario 2 (Data Lake Tier Only) Scenario 1A (PAY GO) Requirement Suppose you need to ingest 10 GB of data per day, and you must retain that data for 2 years. However, you will only frequently use, query, and analyze the data for the first 6 months. Solution To optimize cost, you can ingest the data into the Analytics tier and retain it there for the first 6 months, where active querying and investigation happen. After that period, the remaining 18 months of retention can be shifted to the Data Lake tier, which provides low-cost storage for compliance and auditing needs. But you will be charged separately for data lake tier querying and analytics which depicted as Compute (D) in pricing flow diagram. Pricing Flow / Notes The first 10 GB/day ingested into the Analytics tier is free for 31 days under the Analytics logs plan. All data ingested into the Analytics tier is automatically mirrored to the Data Lake tier at no additional ingestion or retention cost. For the first 6 months, you pay only for Analytics tier ingestion and retention, excluding any free capacity. For the next 18 months, you pay only for Data Lake tier retention, which is significantly cheaper. Azure Pricing Calculator Equivalent Assuming no data is queried or analyzed during the 18-month Data Lake tier retention period: Although the Analytics tier retention is set to 6 months, the first 3 months of retention fall under the free retention limit, so retention charges apply only for the remaining 3 months of the analytics retention window. Azure pricing calculator will adjust accordingly. Scenario 1B (Usage Commitment) Now, suppose you are ingesting 100 GB per day. If you follow the same pay-as-you-go pricing model described above, your estimated cost would be approximately $15,204 per month. However, you can reduce this cost by choosing a Commitment Tier, where Analytics tier ingestion is billed at a discounted rate. Note that the discount applies only to Analytics tier ingestion—it does not apply to Analytics tier retention costs or to any Data Lake tier–related charges. Please refer to the pricing flow and the equivalent pricing calculator results shown below. Monthly cost savings: $15,204 – $11,184 = $4,020 per month Now the question is: What happens if your usage reaches 150 GB per day? Will the additional 50 GB be billed at the Pay-As-You-Go rate? No. The entire 150 GB/day will still be billed at the discounted rate associated with the 100 GB/day commitment tier bucket. Azure Pricing Calculator Equivalent (100 GB/ Day) Azure Pricing Calculator Equivalent (150 GB/ Day) Scenario 2 (Data Lake Tier Only) Requirement Suppose you need to store certain audit or compliance logs amounting to 10 GB per day. These logs are not used for querying, analytics, or investigations on a regular basis, but must be retained for 2 years as per your organization’s compliance or forensic policies. Solution Since these logs are not actively analyzed, you should avoid ingesting them into the Analytics tier, which is more expensive and optimized for active querying. Instead, send them directly to the Data Lake tier, where they can be retained cost-effectively for future audit, compliance, or forensic needs. Pricing Flow Because the data is ingested directly into the Data Lake tier, you pay both ingestion and retention costs there for the entire 2-year period. If, at any point in the future, you need to perform advanced analytics, querying, or search, you will incur additional compute charges, based on actual usage. Even with occasional compute charges, the cost remains significantly lower than storing the same data in the Analytics tier. Realized Savings Scenario Cost per Month Scenario 1: 10 GB/day in Analytics tier $1,520.40 Scenario 2: 10 GB/day directly into Data Lake tier $202.20 (without compute) $257.20 (with sample compute price) Savings with no compute activity: $1,520.40 – $202.20 = $1,318.20 per month Savings with some compute activity (sample value): $1,520.40 – $257.20 = $1,263.20 per month Azure calculator equivalent without compute Azure calculator equivalent with Sample Compute Conclusion The combination of the Analytics tier and the Data Lake tier in Microsoft Sentinel enables organizations to optimize cost based on how their security data is used. High-value logs that require frequent querying, real-time analytics, and investigation can be stored in the Analytics tier, which provides powerful search performance and built-in detection capabilities. At the same time, large-volume or infrequently accessed logs—such as audit, compliance, or long-term retention data—can be directed to the Data Lake tier, which offers dramatically lower storage and ingestion costs. Because all Analytics tier data is automatically mirrored to the Data Lake tier at no extra cost, customers can use the Analytics tier only for the period they actively query data, and rely on the Data Lake tier for the remaining retention. This tiered model allows different scenarios—active investigation, archival storage, compliance retention, or large-scale telemetry ingestion—to be handled at the most cost-effective layer, ultimately delivering substantial savings without sacrificing visibility, retention, or future analytical capabilities.
need to create monitoring queries to track the health status of data connectors
I'm working with Microsoft Sentinel and need to create monitoring queries to track the health status of data connectors. Specifically, I want to: Identify unhealthy or disconnected data connectors, Determine when a data connector last lost connection Get historical connection status information What I'm looking for: A KQL query that can be run in the Sentinel workspace to check connector status OR a PowerShell script/command that can retrieve this information Ideally, something that can be automated for regular monitoring Looking at the SentinelHealth table, but unsure about the exact schema,connector, etc Checking if there are specific tables that track connector status changes Using Azure Resource Graph or management APIs Ive Tried multiple approaches (KQL, PowerShell, Resource Graph) however I somehow cannot get the information I'm looking to obtain. Please assist with this, for example i see this microsoft docs page, https://learn.microsoft.com/en-us/azure/sentinel/monitor-data-connector-health#supported-data-connectors however I would like my query to state data such as - Last ingestion of tables? How much data has been ingested by specific tables and connectors? What connectors are currently connected? The health of my connectors? Please helpSentinel Log Volume vs Defender Log Volume
Dear community, we're currently building up our first SOC service and wondering about the costs (not realy, we know that SIEM is expencive, but we don't understand the log volumes). We started with sentinel and enabled the XDR connector with all log sources (Device information, Defender for Identity etc). If we take a look into the LAW and log volumes for those tables, we've an ingest as shown in the following exibit: BUT, in comparison, we did a cost analysis with the real defender information (from Defender portal). We come to an estimated log volume from approx. 45GB (and usually billable around 25GB, because the customer has E5 licenses, which should include the AADNonInteractiveUserSignInLogs table (5MB/user/day for free) We're asking ourself why we got this differences in comparison to the last few days, shown in the exibit (attention: we started with a daily ingest cap, so the daily ingest would be around 90 - 100GB/day). Where is our mistake/lag of knowledge? Here are the KQL, sent wihtin the Defender portal to get the volumes from there AlertInfo | union AlertEvidence | summarize RecordCount = count(), AlertTotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2) EmailEvents | union EmailUrlInfo | union EmailAttachmentInfo | union EmailPostDeliveryEvents | union UrlClickEvents | summarize RecordCount = count(), MDOTotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2) IdentityLogonEvents | union IdentityQueryEvents | union IdentityDirectoryEvents | summarize RecordCount = count(), IDTotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2) DeviceInfo | union DeviceNetworkInfo | union DeviceProcessEvents | union DeviceNetworkEvents | union DeviceFileEvents | union DeviceRegistryEvents | union DeviceLogonEvents | union DeviceImageLoadEvents | union DeviceEvents | union DeviceFileCertificateInfo | summarize RecordCount = count(), MDETotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2) CloudAppEvents | summarize RecordCount = count(), CAppsTotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2) Thanks a lot! Best & have nice a nice easter :-)Cribl o Logstash vs AMA CEF: What’s the Best Choice for Ingesting Firewall Logs?
Hi everyone, what are the advantages of using Cribl or Logstash over a CEF log collector via AMA for ingesting firewall logs such as Palo Alto for example into Microsoft Sentinel? In a typical scenario, how would you configure the ingestion to optimize performance, scalability, and cost? What do you think? Let’s discuss and share experiences!
Feed data location to run against Sentinel's KQL function
Hi, We have a feed consisting of around 250,000-300,000 entries and will be imported daily. We do not intend to store this data in Sentinel as a table and would like to store it somewhere else (Cosmos, storage, etc.) from where we can grab this data and run it against one of our Sentinel's KQL functions to generate Alerts. Planning to use Logic Apps/Functions to do the above actions. But would like to know what would be the right solution here so that comparing the feed data against KQL function results would be fast and not of high cost Thank you !!
Defender advanced hunting, data-grant from Defender for Servers licensing.
Hi, when configuring Defender for Servers P2 in Defender for Cloud it states that you would be granted a 500 MB per day free ingestion to a log analytics workspace, such as in Sentinel. However, when looking into the supported data sources I do not find the advanced hunting data that would be my first go-to data source when setting up Sentinel, how come? Here is a screenshot of how data-ingestion changed once i turned on the XDR connector, am I to understand that the 500MB ingestion per device we're paying for will do nothing to cover this cost? The E5 grant of 5MB/user/day is nowhere near this amount of data. Is there a way to utilize the 500MB ingestion per device grant for the advanced hunting data?
Linux AMA log ingestion filtering specific logs
I had previously applied ingestion time data transformation for few incoming logs in syslog table when I was using MMA agent for linux. Now I am moving to AMA for Linux servers. How do I apply specific log filtering on AMA for linux logsources? such as if ip is 1.1.1.1 and it contains err logs, drop them. I know it is possible in windows DCR but how can I built same DCR for linux in AMA to filter out them.AMA agent DCR log filtering
Hi, I have previously created KQL queries for ingestion time transformation and was filtering out certain event ids and few other logs (e.g. | where not(EventID == 4799 and CallerProcessName contains "C:\\Program Files\\Qualys\\QualysAgent\\QualysAgent.exe") ) . Now I have almost 80+ filtering KQL queries which I have applied on securityEvent table to filter out specific logs. I have shifted my servers from MMA agent to AMA agent and AMA agent has its down DCR and my existing ingestion time transformation won't work now. I need to create xpath queries in new DCR. Is there anyway I can convert all of the existing ingestion time transformation applied KQLs (example already mentioned above)? OR Do I need to create separate DCRs for AMA to filterout specific events which are 80+?
