Azure Sentinel - Logs delay?

%3CLINGO-SUB%20id%3D%22lingo-sub-1206134%22%20slang%3D%22en-US%22%3EAzure%20Sentinel%20-%20Logs%20delay%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1206134%22%20slang%3D%22en-US%22%3E%3CP%3EHello%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI've%20connected%20my%20windows%20server%20with%20Azure%20Sentinel%20via%20Security%20events%20data%20connector.%3C%2FP%3E%3CP%3EA%20few%20days%20ago%2C%20the%20delay%20of%20getting%20logs%20from%20windows%20event%20manager%20to%20Azure%20sentinel%20logs%20was%2050%20secs%2C%20now%20it%20is%2010%20minutes.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAny%20idea%20on%20how%20I%20can%20see%20why%20it%20takes%20so%20long%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1206289%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Sentinel%20-%20Logs%20delay%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1206289%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F567396%22%20target%3D%22_blank%22%3E%40FeintBE%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ETo%20the%20Events%20table%3F%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIs%20this%20all%20Computers%2C%20or%20just%20some%3F%26nbsp%3B%20Are%20they%20in%20the%20same%20Azure%20Region%20as%20Log%20Analytics%3F%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFactors%20that%20affect%20latency%20are%20discussed%20here%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Fplatform%2Fdata-ingestion-time%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Fplatform%2Fdata-ingestion-time%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20query%20may%20help%20to%20see%20if%20its%20a%20particular%20Computer%20-%20assuming%20its%20the%20Events%20table%20(if%20not%20you%20will%20need%20to%20edit)%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EEvent%0A%7C%20where%20TimeGenerated%20%26gt%3B%20startofday(ago(7day))%0A%7C%20extend%20E2EIngestionLatency%20%3D%20ingestion_time()%20-%20TimeGenerated%20%0A%2F%2F%7C%20extend%20AgentLatency%20%3D%20_TimeReceived%20-%20TimeGenerated%20%0A%7C%20summarize%20%0A%20%20%20%20%20%20%5B'average%20E2E%20IngestionLatency'%5D%20%3D%20round(avg(todouble(datetime_diff(%22Second%22%2Cingestion_time()%2CTimeGenerated))%2F60%20)%2C2)%0A%20%20%20%20%2C%20%5B'minimun%20E2E%20IngestionLatency'%5D%20%3D%20round(min(todouble(datetime_diff(%22Second%22%2Cingestion_time()%2CTimeGenerated))%2F60%20)%2C2)%20%0A%20%20%20%20%2C%20%5B'maximum%20E2E%20IngestionLatency'%5D%20%3D%20round(max(todouble(datetime_diff(%22Second%22%2Cingestion_time()%2CTimeGenerated))%2F60%20)%2C2)%0A%20%20by%20Computer%20%2C%20bin(TimeGenerated%2C%201d)%0A%7C%20order%20by%20%5B'average%20E2E%20IngestionLatency'%5D%20desc%0A%2F%2F%7C%20render%20timechart%20%20%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1206336%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Sentinel%20-%20Logs%20delay%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1206336%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F239477%22%20target%3D%22_blank%22%3E%40Clive%20Watson%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThese%20are%20the%20results%20of%20the%20query%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22res.PNG%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F174684i620A8EDA63648E47%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22res.PNG%22%20alt%3D%22res.PNG%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1206374%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Sentinel%20-%20Logs%20delay%3F%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1206374%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F567396%22%20target%3D%22_blank%22%3E%40FeintBE%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESo%20your%20average%20and%20minimum%20look%20good%2C%20but%20the%20maximum%20latency%20was%20high%20yesterday%20(over%2010mins).%26nbsp%3B%20I%20assume%20that%20is%20the%20same%20computer%20(behind%20the%20red%20squiggle)%3F%26nbsp%3B%20%3CBR%20%2F%3E%3CBR%20%2F%3EWas%20the%20computer%20online%20yesterday%2C%20you%20should%20get%20~60%20heartbeats%20per%20full%20minute%3F%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3EHeartbeat%20%0A%7C%20where%20Computer%20startswith%20%22%26lt%3B%20insert%20computer%20name%26gt%3B%22%20%0A%7C%20where%20TimeGenerated%20%26gt%3B%20ago(1d)%0A%7C%20summarize%20count()%20by%20bin(TimeGenerated%2C1h)%0A%7C%20render%20columnchart%20%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWere%20other%20computers%20affected%20-%20that%20would%20help%20to%20rule%20out%20Network%20issues%20between%20your%20machine%20and%20Log%20Analytics%3F%3CBR%20%2F%3E%3CBR%20%2F%3EWas%20it%20the%20Agent%20compared%20to%20the%20E2E%3F%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3ESecurityEvent%0A%7C%20where%20TimeGenerated%20%26gt%3B%20startofday(ago(7day))%0A%2F%2F%7C%20extend%20E2EIngestionLatency%20%3D%20ingestion_time()%20-%20TimeGenerated%20%0A%2F%2F%7C%20extend%20AgentLatency%20%3D%20_TimeReceived%20-%20TimeGenerated%20%0A%7C%20summarize%20%0A%20%20%20%20%20%20%5B'average%20E2E%20IngestionLatency'%5D%20%3D%20round(avg(todouble(datetime_diff(%22Second%22%2Cingestion_time()%2CTimeGenerated))%2F60%20)%2C2)%0A%20%20%20%20%2C%20%5B'average%20Agent%20Latency'%5D%20%20%20%20%20%20%20%20%3D%20round(avg(todouble(datetime_diff(%22Second%22%2C_TimeReceived%20%20%20%2CTimeGenerated))%2F60%20)%2C2)%20%0A%20%20by%20Computer%20%2C%20bin(TimeGenerated%2C%201d)%0A%7C%20order%20by%20%5B'average%20Agent%20Latency'%5D%20%20desc%3C%2FCODE%3E%3C%2FPRE%3E%3C%2FLINGO-BODY%3E
Occasional Contributor

Hello,

 

I've connected my windows server with Azure Sentinel via Security events data connector.

A few days ago, the delay of getting logs from windows event manager to Azure sentinel logs was 50 secs, now it is 10 minutes.

 

Any idea on how I can see why it takes so long?

 

Thanks!

3 Replies

@FeintBE 

 

To the Events table?

 

Is this all Computers, or just some?  Are they in the same Azure Region as Log Analytics?

 

Factors that affect latency are discussed here: https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-ingestion-time

 

This query may help to see if its a particular Computer - assuming its the Events table (if not you will need to edit)

 

Event
| where TimeGenerated > startofday(ago(7day))
| extend E2EIngestionLatency = ingestion_time() - TimeGenerated 
//| extend AgentLatency = _TimeReceived - TimeGenerated 
| summarize 
      ['average E2E IngestionLatency'] = round(avg(todouble(datetime_diff("Second",ingestion_time(),TimeGenerated))/60 ),2)
    , ['minimun E2E IngestionLatency'] = round(min(todouble(datetime_diff("Second",ingestion_time(),TimeGenerated))/60 ),2) 
    , ['maximum E2E IngestionLatency'] = round(max(todouble(datetime_diff("Second",ingestion_time(),TimeGenerated))/60 ),2)
  by Computer , bin(TimeGenerated, 1d)
| order by ['average E2E IngestionLatency'] desc
//| render timechart  

   

@CliveWatson 

 

These are the results of the query

res.PNG

@FeintBE 

 

So your average and minimum look good, but the maximum latency was high yesterday (over 10mins).  I assume that is the same computer (behind the red squiggle)? 

Was the computer online yesterday, you should get ~60 heartbeats per full minute?

Heartbeat 
| where Computer startswith "< insert computer name>" 
| where TimeGenerated > ago(1d)
| summarize count() by bin(TimeGenerated,1h)
| render columnchart 

 

Were other computers affected - that would help to rule out Network issues between your machine and Log Analytics?

Was it the Agent compared to the E2E? 

SecurityEvent
| where TimeGenerated > startofday(ago(7day))
//| extend E2EIngestionLatency = ingestion_time() - TimeGenerated 
//| extend AgentLatency = _TimeReceived - TimeGenerated 
| summarize 
      ['average E2E IngestionLatency'] = round(avg(todouble(datetime_diff("Second",ingestion_time(),TimeGenerated))/60 ),2)
    , ['average Agent Latency']        = round(avg(todouble(datetime_diff("Second",_TimeReceived   ,TimeGenerated))/60 ),2) 
  by Computer , bin(TimeGenerated, 1d)
| order by ['average Agent Latency']  desc