Handling ingestion delay in Azure Sentinel scheduled alert rules

Published Jan 12 2021 06:13 AM 6,095 Views
Microsoft

 

At Azure Sentinel we take pride in the ability to ingest data from a variety of sources(learn more).

However, data ingestion time may vary for different data sources under different circumstances.

 

In this blog post we will address the delay challenge: understanding the impact of the ingestion delay and how to fix it.

 

Why is the delay significant?

When ingesting data into Sentinel we might encounter delays for a variety of reasons.

These delays might affect our scheduled queries in the following way:

 

When writing a custom detection rule, we set the “Run query every” and the “Lookup data from the last” parameters.

For our , let's assume we run our query every five minutes and lookup data in a five-minute "look-back" window.

 

 

romarsia_0-1609858039791.png

 

Ideally (when there is no delay), this detection will not miss any events.

 

So, how would it look?

romarsia_1-1609858039796.png

 

The event arrives as it is generated, and the window contains the event as we wanted.

Now, assume there is some delay for our data source.

For this example, let’s say the event was ingested 2 minutes after it was generated (delay of 2 minutes).

Example #1:

 

romarsia_2-1609858039801.png

 

 

As we see, the event is generated within the first "look-back" window, but on the first run the event is not ingested into the workspace.

No big deal – after all, we’ll catch it on the second run, right?

Wrong! The next time the query is scheduled to run, the event has been ingested, however it will be filtered out by the time-generated filter (since it happened more than five minutes ago), and the rule won’t fire an alert.

So, what can I do?

 

How to handle the delay:

To solve the issue, we’ll need to be able to know what is the delay for our data type.

In our example it was easy since we knew it was 2 minutes.

But we can figure it out by using the Kusto function “ingestion_time()” and calculating the difference between “TimeGenerated” and the ingestion time - we will talk more about it later.

So, after finalizing the number we can try to address our problem.

 

The basic intuition is probably telling you - "We need to increase the window size. It will help us".

You are correct: this is indeed a part of the solution.

Since our "look-back" window is 5 minutes, and our delay is 2 minutes we will set the "look-back" window to 7 minutes.

romarsia_3-1609858039802.png

We can see now that our missed event is contained in the "look-back" window (7 minutes), problems solved.

romarsia_4-1609858039804.png

 

As you can probably realize by now, we can create duplication since the look back windows overlap.

For a different event, the following diagram applies.

Example #2:

romarsia_5-1609858039810.png

Since the event “TimeGenerated” is in both windows it will fire 2 alerts meaning we need to find a way to solve the duplication.
Going back to our original problem (example #1), we missed events since our data wasn't ingested when our scheduled query ran.
Then we extended the "look-back" to include the event but caused duplication.
So, we would like to associate the event to the window we extended to contain it.
Meaning: ingestion_time() > ago(5m) (original rule "look-back" = 5min).
This will associate our event from the previous example to the first window.

 
 
 
 
 
 
 
 
 
 

Snip20210105_10.png

As we can see the ingestion time restriction trims the additional 2 minutes added to the look back.

And for our previous example we can see that the event is captured by the second run look-back window.

Snip20210105_12.png

 

This query summarizes it up:

 

 

 

 

 

let ingestion_delay= 2min;
let rule_look_back = 5min;
CommonSecurityLog
| where TimeGenerated >= ago(ingestion_delay + rule_look_back)
| where ingestion_time() > (rule_look_back)

 

 

 

 

 

Now when we know how to tackle the ingestion delay that was given to us let’s see how we can determine the ingestion delay by ourselves.

Note: when joining multiple tables each has its own delay, we need to take it into consideration.

 

Calculating ingestion delay:

Ingestion delay may be caused from a verity of reasons and may change according to the source of the data.

Azure Sentinel collects logs from multiple sources which means that we may see different delays.

This blog post provides a dashboard holding latency and delays for the different data types flowing into the workspace.

In the following example we calculated the 99 and 95 percentiles of delay by the Device Vendor and Device product (for the common security logs table).

 

Note: when joining multiple data types, we need to understand what kind of changes to apply to the "look-back".

Snip20210106_14.png

 

The query:

 

 

 

 

CommonSecurityLog
| extend delay = ingestion_time() - TimeGenerated
| summarize percentiles(delay,95,99) by DeviceVendor, DeviceProduct

 

 

 

 

Now that we have this data, we can know what kind of changes to apply to the "look-back" based on the delay that we have.

The fact that we distinguish between different sources emphasize the different delay from different data types.

 

Important to know: Azure Sentinel scheduled alert rules are delayed by 5min.

This allows data types with a smaller delay to be ingested on time for the scheduled run.

For more information, please visit our documentation.

 

 

 

 

5 Comments
Occasional Contributor

@romarsia This is a very interesting article, thank you for the effort.

 

 Few doubts :

 

1. 

Azure Sentinel scheduled alert rules are delayed by 5min.

This allows data types with a smaller delay to be ingested on time for the scheduled run.

For more information, please visit our documentation.

 

--> Does this 5 min delay mean that if I set the alert to run every 5 mins and lookup data from past 10 mins, it actually looks up data from the last 15 mins?

If so does this not bring about duplication as discussed above.

Could you please share the link to the documentation that is mentioned above that discusses this in detail?

 

 

2. 

 

 

let ingestion_delay= 2min;
let rule_look_back = 5min;
CommonSecurityLog
| where TimeGenerated >= ago(ingestion_delay + rule_look_back)
| where ingestion_time() > (rule_look_back)

 

 

 

In this query,

ingestion_time() returns a date time object (like 1/14/2021, 2:26:07.477 PM) and rule_look_back is in minutes,

 

Ex:

where ingestion_time() > 5min
 
When I tested this this comparison operation shows an output set of rows but does not filter correctly probably due to mismatch in data type. 
 
3.
So do we need to design all our analytic queries with the above query conditions in mind ?
Also when setting the rule logic, does the rule_look_back set in the query take precedence over the lookup data from field set in UI ?
 
4. 
Instead of applying such modifications per rule, would it not solve these issues if the Azure Sentinel platform considers Ingestion time for alert generation and not TimeGenerated. Would it not remove the need to keep track of the delay and per table type query modification.
 
Occasional Contributor


Very good content, @romarsia .

Thanks.

I have a doubt.

 

The timegenereted() is the time that the event was generated at the source and is informed in the source payload, right?

 

If for some reason my source does not send the time the log was created, what will be the timegenereted time?

 

If I'm not mistaken, for the MMA collector, if I don't have the timegenereted specified in the log, I'll specify _TimeReceived, which is the time that the log arrived at the installed agent. That's it?

 

What would it be like for firewall logs, for example?

Microsoft

@luizao_lf  you are correct about the MMA agent.

The Timegenerated should represent the time the event was created.

How the time generated is set depends on the source and the pipe used for ingestion 

Occasional Contributor

If the event is from windows, for example, if you do not have timegenereted () at the source, the value of timegenereted() will be the moment the log arrived at the MMA agent.

 

But what about a firewall that does not have an MMA agent? If the team does not contain the source, what value will be sent in the timegenereted() field? Will it be the value when it arrives at the CEF concentrator? or would it be otherwise?

Frequent Visitor

@romarsia 

Thanks for the in-detail information.

 

I was just wondering, is Line 1 below actually required, when the "real" filtering is all done with Line 2?

| where TimeGenerated >= ago(ingestion_delay + rule_look_back)
| where ingestion_time() > (rule_look_back) 

 

In regards to monitoring the latest logs arriving in the last x minutes/hours, there does not seem to be any use for Line 1,

as long as the "lookup data from the last" setting in the rule accounts for any possible ingestion delay.

 

Also, I think line 2 is missing an "ago".

| where ingestion_time() > ago(rule_look_back)

 

%3CLINGO-SUB%20id%3D%22lingo-sub-2052851%22%20slang%3D%22en-US%22%3EHandling%20ingestion%20delay%20in%20Azure%20Sentinel%20scheduled%20alert%20rules%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2052851%22%20slang%3D%22en-US%22%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAt%20Azure%20Sentinel%20we%20take%20pride%20in%20the%20ability%20to%20ingest%20data%20from%20a%20variety%20of%20sources(%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fsentinel%2Fconnect-data-sources%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Elearn%20more%3C%2FA%3E).%3C%2FP%3E%0A%3CP%3EHowever%2C%20data%20ingestion%20time%20may%20vary%20for%20different%20data%20sources%20under%20different%20circumstances.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20this%20blog%20post%20we%20will%20address%20the%20delay%20challenge%3A%20understanding%20the%20impact%20of%20the%20ingestion%20delay%20and%20how%20to%20fix%20it.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EWhy%20is%20the%20delay%20significant%3F%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EWhen%20ingesting%20data%20into%20Sentinel%20we%20might%20encounter%20delays%20for%20a%20variety%20of%20reasons.%3C%2FP%3E%0A%3CP%3EThese%20delays%20might%20affect%20our%20scheduled%20queries%20in%20the%20following%20way%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhen%20writing%20a%20custom%20detection%20rule%2C%20we%20set%20the%20%E2%80%9CRun%20query%20every%E2%80%9D%20and%20the%20%E2%80%9CLookup%20data%20from%20the%20last%E2%80%9D%20parameters.%3C%2FP%3E%0A%3CP%3EFor%20our%20%2C%20let's%20assume%20we%20run%20our%20query%20every%20five%20minutes%20and%20lookup%20data%20in%20a%20five-minute%20%22%3CEM%3Elook-back%3C%2FEM%3E%22%20window.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22romarsia_0-1609858039791.png%22%20style%3D%22width%3A%20787px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F244299i1AA2CC863E886D65%2Fimage-dimensions%2F787x537%3Fv%3D1.0%22%20width%3D%22787%22%20height%3D%22537%22%20role%3D%22button%22%20title%3D%22romarsia_0-1609858039791.png%22%20alt%3D%22romarsia_0-1609858039791.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIdeally%20(when%20there%20is%20no%20delay)%2C%20this%20detection%20will%20not%20miss%20any%20events.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESo%2C%20how%20would%20it%20look%3F%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22romarsia_1-1609858039796.png%22%20style%3D%22width%3A%20799px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F244300i7BF526D1B2E4918B%2Fimage-dimensions%2F799x403%3Fv%3D1.0%22%20width%3D%22799%22%20height%3D%22403%22%20role%3D%22button%22%20title%3D%22romarsia_1-1609858039796.png%22%20alt%3D%22romarsia_1-1609858039796.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20event%20arrives%20as%20it%20is%20generated%2C%20and%20the%20window%20contains%20the%20event%20as%20we%20wanted.%3C%2FP%3E%0A%3CP%3ENow%2C%20assume%20there%20is%20some%20delay%20for%20our%20data%20source.%3C%2FP%3E%0A%3CP%3EFor%20this%20example%2C%20let%E2%80%99s%20say%20the%20event%20was%20ingested%202%20minutes%20after%20it%20was%20generated%20(delay%20of%202%20minutes).%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EExample%20%231%3A%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22romarsia_2-1609858039801.png%22%20style%3D%22width%3A%20925px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F244298i9514D03EAEA00DC5%2Fimage-dimensions%2F925x306%3Fv%3D1.0%22%20width%3D%22925%22%20height%3D%22306%22%20role%3D%22button%22%20title%3D%22romarsia_2-1609858039801.png%22%20alt%3D%22romarsia_2-1609858039801.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAs%20we%20see%2C%20the%20event%20is%20generated%20within%20the%20first%20%22%3CEM%3Elook-back%3C%2FEM%3E%22%20window%2C%20but%20on%20the%20first%20run%20the%20event%20is%20not%20ingested%20into%20the%20workspace.%3C%2FP%3E%0A%3CP%3ENo%20big%20deal%20%E2%80%93%20after%20all%2C%20we%E2%80%99ll%20catch%20it%20on%20the%20second%20run%2C%20right%3F%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EWrong!%3C%2FSTRONG%3E%20The%20next%20time%20the%20query%20is%20scheduled%20to%20run%2C%20the%20event%20has%20been%20ingested%2C%20however%20it%20will%20be%20filtered%20out%20by%20the%20time-generated%20filter%20(since%20it%20happened%20more%20than%20five%20minutes%20ago)%2C%20and%20the%20rule%20won%E2%80%99t%20fire%20an%20alert.%3C%2FP%3E%0A%3CP%3ESo%2C%20what%20can%20I%20do%3F%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EHow%20to%20handle%20the%20delay%3A%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3ETo%20solve%20the%20issue%2C%20we%E2%80%99ll%20need%20to%20be%20able%20to%20know%20what%20is%20the%20delay%20for%20our%20data%20type.%3C%2FP%3E%0A%3CP%3EIn%20our%20example%20it%20was%20easy%20since%20we%20knew%20it%20was%202%20minutes.%3C%2FP%3E%0A%3CP%3EBut%20we%20can%20figure%20it%20out%20by%20using%20the%20Kusto%20function%20%E2%80%9Cingestion_time()%E2%80%9D%20and%20calculating%20the%20difference%20between%20%E2%80%9CTimeGenerated%E2%80%9D%20and%20the%20ingestion%20time%20-%20we%20will%20talk%20more%20about%20it%20later.%3C%2FP%3E%0A%3CP%3ESo%2C%20after%20finalizing%20the%20number%20we%20can%20try%20to%20address%20our%20problem.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20basic%20intuition%20is%20probably%20telling%20you%20-%20%22We%20need%20to%20increase%20the%20window%20size.%20It%20will%20help%20us%22.%3C%2FP%3E%0A%3CP%3EYou%20are%20correct%3A%20this%20is%20indeed%20a%20part%20of%20the%20solution.%3C%2FP%3E%0A%3CP%3ESince%20our%20%22%3CEM%3Elook-back%3C%2FEM%3E%22%20window%20is%205%20minutes%2C%20and%20our%20delay%20is%202%20minutes%20we%20will%20set%20the%20%22%3CEM%3Elook-back%3C%2FEM%3E%22%20window%20to%207%20minutes.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22romarsia_3-1609858039802.png%22%20style%3D%22width%3A%20844px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F244301iCD9E6E7C08DD4876%2Fimage-dimensions%2F844x152%3Fv%3D1.0%22%20width%3D%22844%22%20height%3D%22152%22%20role%3D%22button%22%20title%3D%22romarsia_3-1609858039802.png%22%20alt%3D%22romarsia_3-1609858039802.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EWe%20can%20see%20now%20that%20our%20missed%20event%20is%20contained%20in%20the%20%22%3CEM%3Elook-back%3C%2FEM%3E%22%20window%20(7%20minutes)%2C%20problems%20solved.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22romarsia_4-1609858039804.png%22%20style%3D%22width%3A%20774px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F244302i330E7A6A0A1DB622%2Fimage-dimensions%2F774x300%3Fv%3D1.0%22%20width%3D%22774%22%20height%3D%22300%22%20role%3D%22button%22%20title%3D%22romarsia_4-1609858039804.png%22%20alt%3D%22romarsia_4-1609858039804.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAs%20you%20can%20probably%20realize%20by%20now%2C%20we%20can%20create%20%3CU%3Eduplication%3C%2FU%3E%20since%20the%20look%20back%20windows%20overlap.%3C%2FP%3E%0A%3CP%3EFor%20a%20different%20event%2C%20the%20following%20diagram%20applies.%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EExample%20%232%3A%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22romarsia_5-1609858039810.png%22%20style%3D%22width%3A%20716px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F244303i7EABE90CA3A382C4%2Fimage-dimensions%2F716x299%3Fv%3D1.0%22%20width%3D%22716%22%20height%3D%22299%22%20role%3D%22button%22%20title%3D%22romarsia_5-1609858039810.png%22%20alt%3D%22romarsia_5-1609858039810.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3ESince%20the%20event%20%E2%80%9CTimeGenerated%E2%80%9D%20is%20in%20both%20windows%20it%20will%20fire%202%20alerts%20meaning%20we%20need%20to%20find%20a%20way%20to%20solve%20the%20duplication.%3CBR%20%2F%3EGoing%20back%20to%20our%20original%20problem%20(example%20%231)%2C%20we%20missed%20events%20since%20our%20data%20wasn't%20ingested%20when%20our%20scheduled%20query%20ran.%3CBR%20%2F%3EThen%20we%20extended%20the%20%22%3CEM%3Elook-back%3C%2FEM%3E%22%20to%20include%20the%20event%20but%20caused%20duplication.%3CBR%20%2F%3ESo%2C%20we%20would%20like%20to%20associate%20the%20event%20to%20the%20window%20we%20extended%20to%20contain%20it.%3CBR%20%2F%3EMeaning%3A%20%3CEM%3Eingestion_time()%20%26gt%3B%20ago(5m)%3C%2FEM%3E%20(original%20rule%20%22%3CEM%3Elook-back%3C%2FEM%3E%22%20%3D%205min).%3CBR%20%2F%3EThis%20will%20associate%20our%20event%20from%20the%20previous%20example%20to%20the%20first%20window.%3C%2FP%3E%0A%3CDIV%20id%3D%22tinyMceEditorromarsia_10%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorromarsia_11%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorromarsia_12%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorromarsia_16%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorromarsia_17%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorromarsia_18%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorromarsia_19%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorromarsia_20%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorromarsia_21%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CDIV%20id%3D%22tinyMceEditorromarsia_23%22%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Snip20210105_10.png%22%20style%3D%22width%3A%20745px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F244307i135C35F78E1554CB%2Fimage-dimensions%2F745x328%3Fv%3D1.0%22%20width%3D%22745%22%20height%3D%22328%22%20role%3D%22button%22%20title%3D%22Snip20210105_10.png%22%20alt%3D%22Snip20210105_10.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EAs%20we%20can%20see%20the%20ingestion%20time%20restriction%20trims%20the%20additional%202%20minutes%20added%20to%20the%20look%20back.%3C%2FP%3E%0A%3CP%3EAnd%20for%20our%20previous%20example%20we%20can%20see%20that%20the%20event%20is%20captured%20by%20the%20second%20run%20look-back%20window.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Snip20210105_12.png%22%20style%3D%22width%3A%20735px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F244310i625E160D58214851%2Fimage-dimensions%2F735x316%3Fv%3D1.0%22%20width%3D%22735%22%20height%3D%22316%22%20role%3D%22button%22%20title%3D%22Snip20210105_12.png%22%20alt%3D%22Snip20210105_12.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EThis%20query%20summarizes%20it%20up%3A%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-applescript%22%3E%3CCODE%3Elet%20ingestion_delay%3D%202min%3B%0Alet%20rule_look_back%20%3D%205min%3B%0ACommonSecurityLog%0A%7C%20where%20TimeGenerated%20%26gt%3B%3D%20ago(ingestion_delay%20%2B%20rule_look_back)%0A%7C%20where%20ingestion_time()%20%26gt%3B%20(rule_look_back)%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ENow%20when%20we%20know%20how%20to%20tackle%20the%20ingestion%20delay%20that%20was%20given%20to%20us%20let%E2%80%99s%20see%20how%20we%20can%20determine%20the%20ingestion%20delay%20by%20ourselves.%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3ENote%3A%20%3C%2FSTRONG%3Ewhen%20joining%20multiple%20tables%20each%20has%20its%20own%20delay%2C%20we%20need%20to%20take%20it%20into%20consideration.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CSPAN%3ECalculating%20ingestion%20delay%3A%3C%2FSPAN%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3EIngestion%20delay%20may%20be%20caused%20from%20a%20%3C%2FSPAN%3E%3CSPAN%3Everity%3C%2FSPAN%3E%3CSPAN%3E%20of%20reasons%20and%20may%20change%20according%20to%20the%20%3C%2FSPAN%3E%3CSPAN%3Esource%20of%20the%20data.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3EAzure%20Sentinel%20collects%20logs%20from%20multiple%20sources%20which%20means%20that%20we%20may%20see%20different%20d%3C%2FSPAN%3E%3CSPAN%3Eelays.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fazure-sentinel%2Fusage-reporting-for-azure-sentinel%2Fba-p%2F1267383%22%20target%3D%22_blank%22%3EThis%3C%2FA%3E%20blog%20post%20%3C%2FSPAN%3E%3CSPAN%3Eprovides%20a%20dashboard%20holding%20latency%20and%20delays%20for%20the%20different%20data%20types%20flowing%20into%20the%20workspace.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3EIn%20the%20following%20example%20we%20calculated%20the%2099%20and%2095%20percentiles%20%3C%2FSPAN%3E%3CSPAN%3Eof%20delay%20by%20the%20Device%20Vendor%20and%20Device%20product%20(for%20the%20common%20security%20logs%20table)%3C%2FSPAN%3E%3CSPAN%3E.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CSPAN%3ENote%3A%3C%2FSPAN%3E%3C%2FSTRONG%3E%3CSPAN%3E%20when%20joining%20multiple%20data%20types%2C%20we%20need%20to%20understand%20what%20kind%20of%20changes%20to%20apply%20to%20the%20%22%3CEM%3Elook-back%3C%2FEM%3E%22.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Snip20210106_14.png%22%20style%3D%22width%3A%20563px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F244490i50DC42EDE11CDBB6%2Fimage-dimensions%2F563x283%3Fv%3D1.0%22%20width%3D%22563%22%20height%3D%22283%22%20role%3D%22button%22%20title%3D%22Snip20210106_14.png%22%20alt%3D%22Snip20210106_14.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3EThe%20query%3A%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CPRE%20class%3D%22lia-code-sample%20language-applescript%22%3E%3CCODE%3ECommonSecurityLog%0A%7C%20extend%20delay%20%3D%20ingestion_time()%20-%20TimeGenerated%0A%7C%20summarize%20percentiles(delay%2C95%2C99)%20by%20DeviceVendor%2C%20DeviceProduct%3C%2FCODE%3E%3C%2FPRE%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%3ENow%20that%20we%20have%20this%20data%2C%20we%20can%20know%20what%20kind%20of%20changes%20to%20apply%20to%20the%20%22%3CEM%3Elook-back%3C%2FEM%3E%22%26nbsp%3Bbased%20on%20the%20delay%20that%20we%20have.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3EThe%20fact%20that%20we%20distinguish%20between%20different%20sources%20emphasize%20the%20different%20delay%20from%20different%20data%20types.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20style%3D%22font-weight%3A%20normal%20!msorm%3B%22%3E%3CSTRONG%3E%3CSPAN%3EImportant%20to%20know%3A%3C%2FSPAN%3E%3C%2FSTRONG%3E%3C%2FSPAN%3E%20%3CSPAN%3EAzure%20Sentinel%20scheduled%20alert%20rules%20are%20delayed%20by%205min.%20%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3EThis%20allows%20data%20types%20with%20a%20smaller%20delay%20to%20be%20ingested%20on%20time%20for%20the%20scheduled%20run.%20%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3EFor%20more%20information%2C%20please%20visit%20our%20documentation.%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%E2%80%83%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2052851%22%20slang%3D%22en-US%22%3E%3CP%3EAt%20Azure%20Sentinel%20we%20take%20pride%20in%20the%20ability%20to%20ingest%20data%20from%20a%20variety%20of%20sources.%3C%2FP%3E%0A%3CP%3EHowever%2C%20data%20ingestion%20time%20may%20vary%20for%20different%20data%20sources%20under%20different%20circumstances.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20this%20blog%20post%20we%20will%20address%20the%20%3CSTRONG%3Edelay%20challenge%3A%20understanding%20the%20impact%20of%20the%20ingestion%20delay%20and%20how%20to%20fix%20it.%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22romarsia_0-1610459712152.png%22%20style%3D%22width%3A%20700px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F245652iB47412F36A8BA0E0%2Fimage-dimensions%2F700x294%3Fv%3D1.0%22%20width%3D%22700%22%20height%3D%22294%22%20role%3D%22button%22%20title%3D%22romarsia_0-1610459712152.png%22%20alt%3D%22romarsia_0-1610459712152.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2052851%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAnalytics%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2059934%22%20slang%3D%22en-US%22%3ERe%3A%20Handling%20ingestion%20delay%20in%20Azure%20Sentinel%20scheduled%20alert%20rules%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2059934%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F918082%22%20target%3D%22_blank%22%3E%40romarsia%3C%2FA%3E%26nbsp%3BThis%20is%20a%20very%20interesting%20article%2C%20thank%20you%20for%20the%20effort.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3BFew%20doubts%20%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E1.%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3EAzure%20Sentinel%20scheduled%20alert%20rules%20are%20delayed%20by%205min.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EThis%20allows%20data%20types%20with%20a%20smaller%20delay%20to%20be%20ingested%20on%20time%20for%20the%20scheduled%20run.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EFor%20more%20information%2C%20please%20visit%20our%20documentation.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3E--%26gt%3B%20Does%20this%205%20min%20delay%20mean%20that%20if%20I%20set%20the%20alert%20to%20run%20every%205%20mins%20and%20lookup%20data%20from%20past%2010%20mins%2C%20it%20actually%20looks%20up%20data%20from%20the%20last%2015%20mins%3F%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EIf%20so%20does%20this%20not%20bring%20about%20duplication%20as%20discussed%20above.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3ECould%20you%20please%20share%20the%20link%20to%20the%20documentation%20that%20is%20mentioned%20above%20that%20discusses%20this%20in%20detail%3F%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3E2.%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-markup%22%3E%3CCODE%3Elet%20ingestion_delay%3D%202min%3B%0Alet%20rule_look_back%20%3D%205min%3B%0ACommonSecurityLog%0A%7C%20where%20TimeGenerated%20%26gt%3B%3D%20ago(ingestion_delay%20%2B%20rule_look_back)%0A%7C%20where%20ingestion_time()%20%26gt%3B%20(rule_look_back)%3C%2FCODE%3E%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIn%20this%20query%2C%3C%2FP%3E%3CP%3Eingestion_time()%20returns%20a%20date%20time%20object%20(like%26nbsp%3B%3CSPAN%3E1%2F14%2F2021%2C%202%3A26%3A07.477%20PM%3C%2FSPAN%3E)%20and%20rule_look_back%20is%20in%20minutes%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EEx%3A%3C%2FP%3E%3CDIV%3E%3CDIV%3E%3CSPAN%3E%7C%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3Ewhere%3C%2FSPAN%3E%3CSPAN%3E%26nbsp%3Bingestion_time()%26nbsp%3B%26gt%3B%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3E5%3C%2FSPAN%3E%3CSPAN%3Emin%3C%2FSPAN%3E%3C%2FDIV%3E%3CDIV%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%3EWhen%20I%20tested%20this%20this%20comparison%20operation%20shows%20an%20output%20set%20of%20rows%20but%20does%20not%20filter%20correctly%20probably%20due%20to%20mismatch%20in%20data%20type.%26nbsp%3B%3C%2FDIV%3E%3CDIV%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%3E3.%3C%2FDIV%3E%3CDIV%3ESo%20do%20we%20need%20to%20design%20all%20our%20analytic%20queries%20with%20the%20above%20query%20conditions%20in%20mind%20%3F%3C%2FDIV%3E%3CDIV%3EAlso%20when%20setting%20the%20rule%20logic%2C%20does%20the%26nbsp%3Brule_look_back%20set%20in%20the%20query%20take%20precedence%20over%20the%20lookup%20data%20from%20field%20set%20in%20UI%20%3F%3C%2FDIV%3E%3CDIV%3E%26nbsp%3B%3C%2FDIV%3E%3CDIV%3E4.%26nbsp%3B%3C%2FDIV%3E%3CDIV%3EInstead%20of%20applying%20such%20modifications%20per%20rule%2C%20would%20it%20not%20solve%20these%20issues%20if%20the%20Azure%20Sentinel%20platform%20considers%20Ingestion%20time%20for%20alert%20generation%20and%20not%20TimeGenerated.%20Would%20it%20not%20remove%20the%20need%20to%20keep%20track%20of%20the%20delay%20and%20per%20table%20type%20query%20modification.%3C%2FDIV%3E%3CDIV%3E%26nbsp%3B%3C%2FDIV%3E%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2253087%22%20slang%3D%22en-US%22%3ERe%3A%20Handling%20ingestion%20delay%20in%20Azure%20Sentinel%20scheduled%20alert%20rules%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2253087%22%20slang%3D%22en-US%22%3E%3CP%3E%3CBR%20%2F%3EVery%20good%20content%2C%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F918082%22%20target%3D%22_blank%22%3E%40romarsia%3C%2FA%3E%26nbsp%3B.%3C%2FP%3E%3CP%3EThanks.%3C%2FP%3E%3CP%3EI%20have%20a%26nbsp%3Bdoubt.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20%3CU%3Etimegenereted()%3C%2FU%3E%20is%20the%20time%20that%20the%20event%20was%20generated%20at%20the%20source%20and%20is%20informed%20in%20the%20source%20payload%2C%20right%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIf%20for%20some%20reason%20my%20source%20does%20not%20send%20the%20time%20the%20log%20was%20created%2C%20what%20will%20be%20the%20timegenereted%20time%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIf%20I'm%20not%20mistaken%2C%20for%20the%20MMA%20collector%2C%20if%20I%20don't%20have%20the%20timegenereted%20specified%20in%20the%20log%2C%20I'll%20specify%20_TimeReceived%2C%20which%20is%20the%20time%20that%20the%20log%20arrived%20at%20the%20installed%20agent.%20That's%20it%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWhat%20would%20it%20be%20like%20for%20firewall%20logs%2C%20for%20example%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2253680%22%20slang%3D%22en-US%22%3ERe%3A%20Handling%20ingestion%20delay%20in%20Azure%20Sentinel%20scheduled%20alert%20rules%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2253680%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F733311%22%20target%3D%22_blank%22%3E%40luizao_lf%3C%2FA%3E%26nbsp%3B%20you%20are%20correct%20about%20the%20MMA%20agent.%3C%2FP%3E%0A%3CP%3EThe%20Timegenerated%20should%20represent%20the%20time%20the%20event%20was%20created.%3C%2FP%3E%0A%3CP%3EHow%20the%20time%20generated%20is%20set%20depends%20on%20the%20source%20and%20the%20pipe%20used%20for%20ingestion%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2257926%22%20slang%3D%22en-US%22%3ERe%3A%20Handling%20ingestion%20delay%20in%20Azure%20Sentinel%20scheduled%20alert%20rules%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2257926%22%20slang%3D%22en-US%22%3E%3CP%3EIf%20the%20event%20is%20from%20windows%2C%20for%20example%2C%20if%20you%20do%20not%20have%20timegenereted%20()%20at%20the%20source%2C%20the%20value%20of%20timegenereted()%20will%20be%20the%20moment%20the%20log%20arrived%20at%20the%20MMA%20agent.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EBut%20what%20about%20a%20firewall%20that%20does%20not%20have%20an%20MMA%20agent%3F%20If%20the%20team%20does%20not%20contain%20the%20source%2C%20what%20value%20will%20be%20sent%20in%20the%20timegenereted()%20field%3F%20Will%20it%20be%20the%20value%20when%20it%20arrives%20at%20the%20CEF%20concentrator%3F%20or%20would%20it%20be%20otherwise%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Jan 12 2021 06:43 AM
Updated by: