Forum Discussion
Increasing scheduled analytic frequency
We've seen a number of template analytics with search frequency set to 1d. We have some use cases where we'd like to be notified much sooner than the incident + ~1d. What we're struggling with is understanding the ingestion latency for certain log sources. Particularly OfficeActivity where we'd like to know about malicious behaviour within 1-2hrs, but we're seeing ingest latency fluctuations up to 90 minutes based on the query we're looking at which deviates from the https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-ingestion-time#azure-activity-logs-resource-logs-and-metrics:
// Calculate the latency of each log type
union withsource = tt *
| project TimeGenerated, Type
| extend latency = ingestion_time() - TimeGenerated
| where latency > 0s
| summarize
max = max(latency)/1m,
min = min(latency)/1m,
avg = avg(latency)/1m,
p50 = totimespan(percentiles(latency, 50)[0])/1m,
p95 = totimespan(percentiles(latency, 95)[0])/1m by Type
| order by Type asc
Are there any absolute time guarantees for Azure cloud service logs?
The link you provided discussed the push from O365 to Sentinel:
"Office 365 solution polls activity logs using the Office 365 Management Activity API, which currently does not provide any near-real time latency guarantees.".
I'm sure you see from your own query that many tables have acceptable latency. Its an area always under review and being optimised.
2 Replies
- CliveWatsonFormer Employee
The link you provided discussed the push from O365 to Sentinel:
"Office 365 solution polls activity logs using the Office 365 Management Activity API, which currently does not provide any near-real time latency guarantees.".
I'm sure you see from your own query that many tables have acceptable latency. Its an area always under review and being optimised.
- pemonttoBrass Contributor
CliveWatson thanks! And thanks for the https://github.com/CliveW-MSFT/KQLpublic/blob/master/KQL/Workbooks/Workspace%20Usage%20report.workbook that wraps everything up nicely, including latency!