Forum Discussion
Sentinel Log Volume vs Defender Log Volume
Dear community,
we're currently building up our first SOC service and wondering about the costs (not realy, we know that SIEM is expencive, but we don't understand the log volumes).
We started with sentinel and enabled the XDR connector with all log sources (Device information, Defender for Identity etc). If we take a look into the LAW and log volumes for those tables, we've an ingest as shown in the following exibit:
BUT, in comparison, we did a cost analysis with the real defender information (from Defender portal).
We come to an estimated log volume from approx. 45GB (and usually billable around 25GB, because the customer has E5 licenses, which should include the AADNonInteractiveUserSignInLogs table (5MB/user/day for free)
We're asking ourself why we got this differences in comparison to the last few days, shown in the exibit (attention: we started with a daily ingest cap, so the daily ingest would be around 90 - 100GB/day).
Where is our mistake/lag of knowledge?
Here are the KQL, sent wihtin the Defender portal to get the volumes from there
AlertInfo
| union AlertEvidence
| summarize RecordCount = count(), AlertTotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2)EmailEvents
| union EmailUrlInfo
| union EmailAttachmentInfo
| union EmailPostDeliveryEvents
| union UrlClickEvents
| summarize RecordCount = count(), MDOTotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2)IdentityLogonEvents
| union IdentityQueryEvents
| union IdentityDirectoryEvents
| summarize RecordCount = count(), IDTotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2)DeviceInfo
| union DeviceNetworkInfo
| union DeviceProcessEvents
| union DeviceNetworkEvents
| union DeviceFileEvents
| union DeviceRegistryEvents
| union DeviceLogonEvents
| union DeviceImageLoadEvents
| union DeviceEvents
| union DeviceFileCertificateInfo
| summarize RecordCount = count(), MDETotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2)CloudAppEvents
| summarize RecordCount = count(), CAppsTotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2)
Thanks a lot!
Best & have nice a nice easter :-)
The volume reported in Microsoft Defender is only an estimate of raw telemetry size, not the actual amount that gets ingested and billed in Sentinel.
Sentinel adds overhead, normalization, indexing, and data duplication (from multiple connectors), leading to much higher volume than raw Defender estimates.Data Normalization:
Defender’s estimate data size may exclude bits like metadata and unused columns. Sentinel ingestion includes all fields.
2 Replies
btw, you could have saved some typing and made sure all Tables were included, you can group with the union for example all tables that start with DEVICE or even multiple groups, as per this example
union Device* , Email*
| summarize RecordCount = count(), MDETotalSizeMB = round(sum(estimate_data_size(*))/pow(1024,2),2)
You also have to assess if you need all the Tables duplicated in Sentinel, you normally do it for one of two reasons:
1. You need to retain the data for a longer period for ad-hoc Hunting/reporting or to meet a compliance obligation2. You have Analytics that need the data in Sentinel - so make sure you are using the data you are syncing
The volume reported in Microsoft Defender is only an estimate of raw telemetry size, not the actual amount that gets ingested and billed in Sentinel.
Sentinel adds overhead, normalization, indexing, and data duplication (from multiple connectors), leading to much higher volume than raw Defender estimates.Data Normalization:
Defender’s estimate data size may exclude bits like metadata and unused columns. Sentinel ingestion includes all fields.
