SOLVED

Increasing scheduled analytic frequency

%3CLINGO-SUB%20id%3D%22lingo-sub-1219944%22%20slang%3D%22en-US%22%3EIncreasing%20scheduled%20analytic%20frequency%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1219944%22%20slang%3D%22en-US%22%3E%3CP%20data-unlink%3D%22true%22%3EWe've%20seen%20a%20number%20of%20template%20analytics%20with%20search%20frequency%20set%20to%201d.%26nbsp%3BWe%20have%20some%20use%20cases%20where%20we'd%20like%20to%20be%20notified%20much%20sooner%20than%20the%20incident%20%2B%20~1d.%20What%20we're%20struggling%20with%20is%20understanding%20the%20ingestion%20latency%20for%20certain%20log%20sources.%20Particularly%20OfficeActivity%20where%20we'd%20like%20to%20know%20about%20malicious%20behaviour%20within%201-2hrs%2C%20but%20we're%20seeing%20ingest%20latency%20fluctuations%20up%20to%2090%20minutes%20based%20on%20the%20query%20we're%20looking%20at%20which%20deviates%20from%20the%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Fplatform%2Fdata-ingestion-time%23azure-activity-logs-resource-logs-and-metrics%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3Edata%20ingestion%20time%20documentation%3C%2FA%3E%3A%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3E%2F%2F%20Calculate%20the%20latency%20of%20each%20log%20type%0Aunion%20withsource%20%3D%20tt%20*%0A%7C%20project%20TimeGenerated%2C%20Type%0A%7C%20extend%20latency%20%3D%20ingestion_time()%20-%20TimeGenerated%0A%7C%20where%20latency%20%26gt%3B%200s%0A%7C%20summarize%0Amax%20%3D%20max(latency)%2F1m%2C%0Amin%20%3D%20min(latency)%2F1m%2C%0Aavg%20%3D%20avg(latency)%2F1m%2C%0Ap50%20%3D%20totimespan(percentiles(latency%2C%2050)%5B0%5D)%2F1m%2C%0Ap95%20%3D%20totimespan(percentiles(latency%2C%2095)%5B0%5D)%2F1m%20by%20Type%0A%7C%20order%20by%20Type%20asc%3C%2FCODE%3E%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%20data-unlink%3D%22true%22%3EAre%20there%20any%20absolute%20time%20guarantees%20for%20Azure%20cloud%20service%20logs%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1220073%22%20slang%3D%22en-US%22%3ERe%3A%20Increasing%20scheduled%20analytic%20frequency%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1220073%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F456796%22%20target%3D%22_blank%22%3E%40pemontto%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20link%20you%20provided%20discussed%20the%20%3CSTRONG%3Epush%3C%2FSTRONG%3E%20from%20O365%20to%20Sentinel%3A%3C%2FP%3E%0A%3CP%3E%3CEM%3E%22Office%20365%20solution%20polls%20activity%20logs%20using%20the%20Office%20365%20Management%20Activity%20API%2C%20%3CFONT%20color%3D%22%23FF0000%22%3Ewhich%20currently%20does%20not%20provide%20any%20near-real%20time%20latency%20guarantees.%3C%2FFONT%3E%22.%26nbsp%3B%3C%2FEM%3E%3C%2FP%3E%0A%3CP%3EI'm%20sure%20you%20see%20from%20your%20own%20query%20that%20many%20tables%20have%20acceptable%20latency.%26nbsp%3B%20Its%20an%20area%20always%20under%20review%20and%20being%20optimised.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1275701%22%20slang%3D%22en-US%22%3ERe%3A%20Increasing%20scheduled%20analytic%20frequency%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1275701%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F239477%22%20target%3D%22_blank%22%3E%40Clive%20Watson%3C%2FA%3E%26nbsp%3Bthanks!%20And%20thanks%20for%20the%20%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FCliveW-MSFT%2FKQLpublic%2Fblob%2Fmaster%2FKQL%2FWorkbooks%2FWorkspace%2520Usage%2520report.workbook%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3Eworkbook%3C%2FA%3E%20that%20wraps%20everything%20up%20nicely%2C%20including%20latency!%3C%2FP%3E%3C%2FLINGO-BODY%3E
Contributor

We've seen a number of template analytics with search frequency set to 1d. We have some use cases where we'd like to be notified much sooner than the incident + ~1d. What we're struggling with is understanding the ingestion latency for certain log sources. Particularly OfficeActivity where we'd like to know about malicious behaviour within 1-2hrs, but we're seeing ingest latency fluctuations up to 90 minutes based on the query we're looking at which deviates from the data ingestion time documentation:

// Calculate the latency of each log type
union withsource = tt *
| project TimeGenerated, Type
| extend latency = ingestion_time() - TimeGenerated
| where latency > 0s
| summarize
max = max(latency)/1m,
min = min(latency)/1m,
avg = avg(latency)/1m,
p50 = totimespan(percentiles(latency, 50)[0])/1m,
p95 = totimespan(percentiles(latency, 95)[0])/1m by Type
| order by Type asc

 

Are there any absolute time guarantees for Azure cloud service logs?

2 Replies
best response confirmed by rodtrent (Microsoft)
Solution

@pemontto 

 

The link you provided discussed the push from O365 to Sentinel:

"Office 365 solution polls activity logs using the Office 365 Management Activity API, which currently does not provide any near-real time latency guarantees.". 

I'm sure you see from your own query that many tables have acceptable latency.  Its an area always under review and being optimised.

 

 

 

 

@CliveWatson thanks! And thanks for the workbook that wraps everything up nicely, including latency!