Azure Monitor - LogAnalytics - Delay in sending alerts

%3CLINGO-SUB%20id%3D%22lingo-sub-1498916%22%20slang%3D%22en-US%22%3EAzure%20Monitor%20-%20LogAnalytics%20-%20Delay%20in%20sending%20alerts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1498916%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20all...%26nbsp%3B%3C%2FP%3E%3CP%3EI'm%20currently%20using%20log%20analytics%20and%20alerts%20for%20our%20company%20and%20implementing%20monitoring%20only%20through%20Azure%20Monitor.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%C2%B4m%20experiencing%20a%20lot%20of%20delay%20in%20receiving%20the%20alerts%20from%20the%20monitoring%20platform%20over%20the%20last%20days.%26nbsp%3B%3C%2FP%3E%3CP%3EThus%20this%20resulting%20in%20massive%20spam%20Messages%20to%20the%20TeamsChannel%2Femail%2FSMS%20Contacts%2C%20for%20that%20purpose.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHere's%20a%20preview%20of%20what%20I'm%20talking%20about%3A%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22loadedlouie27_0-1593530772591.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F202195i39967C581D38528C%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20title%3D%22loadedlouie27_0-1593530772591.png%22%20alt%3D%22loadedlouie27_0-1593530772591.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3EThe%20same%20alert%20was%20triggered%20at%20the%20same%20minute%20a%20couple%20of%20times%2C%20I'm%20guessing%20by%20delayed%20ingestion%20on%20the%20component%20that%20actually%20triggers%20the%20alerts%2C%20and%20not%20the%20log%20analytics%20itself%20since%20I%20can%20see%20a%20process%20running%20on%20a%20machine%20within%20a%20few%20minutes.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3EFirst%20question%3A%26nbsp%3B%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3EIs%20this%20supposed%20to%20be%20a%20viable%20product%20at%20this%20point%3F%3C%2FP%3E%3CP%3EIs%20this%20something%20companies%20can%20really%20rely%20on%20or%20am%20I%20pushing%20too%20much%20and%20expecting%20much%20more%20than%20this%20type%20of%20behavior%20from%20this%20solution%3F%3C%2FP%3E%3CP%3E%3CBR%20%2F%3E%3CSTRONG%3ESecond%20Question%3A%26nbsp%3B%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3EIs%20there%20a%20way%20to%20see%20whats%20the%20actual%20delay%20on%20the%20alert%20side%3F%26nbsp%3B%3C%2FP%3E%3CP%3EI%20know%20I%20can%20see%20the%20ingest%20time%20of%20the%20log%20analytics%2C%20however%2C%20I%20can%20tell%20nothing%20is%20wrong%20on%20the%20time%20its%20ingested%2C%20instead%2C%20we%20see%20a%20massive%20delay%20on%20sending%20the%20alerts%2C%20and%20showing%20them%20on%20the%20alert%20console.%20resulting%20in%20a%20non-reliable%20product...%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3EThird%20Question%3A%26nbsp%3B%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3EIs%20there%20a%20place%20we%20find%20a%20%22bible%22%20on%20monitoring%20with%20azure%2C%20I%20find%20the%20documentation%2C%20sometimes%20too%20much%20vague.%20And%20there's%20not%20that%20much%20info%20about%20monitoring%20with%20azure%20monitor%2C%20sentinel%2C%20log%20analytics%2C%20create%20metrics%20from%20log%20analytics%2C%20etc...%3CBR%20%2F%3EBut%20it%20can%20be%20me...%20I'm%20new%20in%20the%20cloud...%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3EFourth%20Question%3A%26nbsp%3B%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3EWhy%20do%20alerts%2C%20that%20are%20fired%2C%20using%20a%20custom%20search%2C%20don't%20change%20the%20monitor%20condition%20to%20resolved%3F%3CBR%20%2F%3Eeven%20with%20metric%20type%2C%20using%20aggregated%20and%20time%20generated%26nbsp%3B%3C%2FP%3E%3CP%3EAnything%20I%20need%20to%20do%20in%20my%20query%3F%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20in%20advance%20for%20your%20time%20and%20help.%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1505731%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Monitor%20-%20LogAnalytics%20-%20Delay%20in%20sending%20alerts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1505731%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F714167%22%20target%3D%22_blank%22%3E%40loadedlouie27%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAdding%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F65839%22%20target%3D%22_blank%22%3E%40Robi%20Czitron%3C%2FA%3E%26nbsp%3Band%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F98398%22%20target%3D%22_blank%22%3E%40Oleg%20Ananiev%3C%2FA%3E%26nbsp%3B%20from%20Alerts%20and%20Log%20Analytics%20team%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1512621%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Monitor%20-%20LogAnalytics%20-%20Delay%20in%20sending%20alerts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1512621%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F54749%22%20target%3D%22_blank%22%3E%40Ketan%20Ghelani%3C%2FA%3E%26nbsp%3BTks%20for%20the%20help%2C%20but%20i%20dont%20think%20anybody%20can%20respond%20to%20this.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3A)%3C%2Fimg%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1516819%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Monitor%20-%20LogAnalytics%20-%20Delay%20in%20sending%20alerts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1516819%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F714167%22%20target%3D%22_blank%22%3E%40loadedlouie27%3C%2FA%3E%26nbsp%3BThere%20is%20a%20lot%20of%20questions%2C%20but%20I'll%20answer%20generally.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ELog%20alerts%20is%20fully%20GA%20and%20we%20can%20assist%20you%20in%20these%20cases%20via%20the%20official%20support%20channels.%20Our%20documentation%20is%20available%20for%20assisting%20getting%20you%20started%20with%20the%20different%20monitoring%20options.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ELog%20alerts%20works%20best%20when%20looking%20for%20data%20in%20the%20log%20and%20less%20well%20when%20looking%20for%20lack%20of%20data%20(such%20as%20heartbeat).%20Ingestion%20delay%20can%20impact%20these%20alerts%3A%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Fplatform%2Fdata-ingestion-time%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ehttps%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-monitor%2Fplatform%2Fdata-ingestion-time%3C%2FA%3E%3C%2FP%3E%0A%3CP%3EThis%20means%20when%20this%20happens%2C%20you%20could%20experience%20false%20alerts%20or%20late%20alerts.%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20would%20recommend%20you%20use%20metric%20alerts%20for%20those%20use%20cases%20unless%20you%20need%20the%20power%20of%20a%20log%20alert%20custom%20query.%3C%2FP%3E%0A%3CP%3ESaying%20that%20we%20are%20introducing%20a%20new%20flow%20this%20month%20that%20should%20improve%20accuracy%20of%20the%20alerts%20and%20lower%20the%20chances%20of%20you%20hitting%20issues.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ELog%20search%20alerts%20are%20stateless%20by%20design.%20We%20are%20working%20on%20adding%20stateful%20log%20alerts%20that%20also%20resolve.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1533160%22%20slang%3D%22en-US%22%3ERe%3A%20Azure%20Monitor%20-%20LogAnalytics%20-%20Delay%20in%20sending%20alerts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1533160%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F390831%22%20target%3D%22_blank%22%3E%40yalavi%3C%2FA%3Ehi%20thank%20you%20for%20your%20time%2C%26nbsp%3B%3C%2FP%3E%3CP%3EIf%20you%20don't%20mind%20I%20have%20a%20few%20questions%2C%20I%20encounter%20a%20major%20issue%2C%20in%20my%20opinion%2C%3C%2FP%3E%3CP%3Eusing%20the%20current%20solution%2C%20and%20I%20would%20like%20to%20know%20if%20they%20are%20gonna%20be%20addressed%2C%3CBR%20%2F%3Eor%20if%20they%20are%20out%20of%20the%20scope%20for%20the%20current%20road%20map.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3EAre%20you%20guys%20thinking%20about%20making%20the%20alerts%20fired%20being%20grouped%3F%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3EThis%20is%20one%20of%20the%20major%20issues%20I%20currently%20see%20in%20using%20the%20Solution.%26nbsp%3B%3C%2FP%3E%3CP%3EWhat%20I%20mean%20is%3A%20I%20have%20the%20same%20alert%20been%20checked%20every%205%20minutes%2C%20and%20if%20it%20triggered%2C%20the%20alert%20just%20keeps%20on%20repeating%20itself%20and%20having%20like%202000%20alerts%20for%20the%20same%20threshold%2Frule%2C%20its%20kind%20of%20a%20killer%2C%20for%20using%20the%20tool%20correctly%2C%20in%20my%20opinion.%26nbsp%3B%3C%2FP%3E%3CP%3EIf%20you%20go%20asleep%20at%20night%2C%20you%20might%20wake%20up%20the%20next%20morning%2C%20for%20a%20rule%20that%20has%20created%202000%20alerts%20in%208%20hours%2C%20and%20have%20to%20close%20the%20alerts%20%22by%20hand%22.%3C%2FP%3E%3CP%3EWhat%20I'm%20suggesting%20its%20something%20a%20bit%20kind%20of%20Azure%20Sentinel%20grouping.%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3EIs%20the%20%22Alert%20Console%22%20going%20to%20be%20reworked%2C%20or%20allow%20further%20customization%3F%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EOn%20the%20Monitoring%20Side%20is%20there%20any%20place%20i%20can%20find%20a%20direct%20match%20from%20the%20tables%20been%20monitored%3F%26nbsp%3B%3C%2FP%3E%3CP%3EWhat%20I%20mean%20is%3A%26nbsp%3B%20%3CSTRONG%3Eis%20there%20a%20way%20I%20can%20see%20where%20to%20activate%2C%20and%20what%2C%20%3C%2FSTRONG%3E%3CBR%20%2F%3E%3CSTRONG%3Ein%20order%20to%20get%20data%20into%20a%20given%20table%20in%20log%20analytics%3F%3C%2FSTRONG%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20in%20advance%2C%20and%20I'm%20sorry%20for%20my%20questions%2C%20they%20may%20be%20seen%20a%20bit%20noobish%2C%3CBR%20%2F%3Ebut%20I%20think%20some%20of%20the%20topics%20are%20like%20elephants%20in%20the%20room%2C%20at%20least%20in%20some%20documentations%20in%20Microsoft.%26nbsp%3B%3C%2FP%3E%3CP%3EOr%20the%20information%20is%20so%20dispersed%2C%20that%20I%20have%20trouble%20getting%20it.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Occasional Contributor

Hi all... 

I'm currently using log analytics and alerts for our company and implementing monitoring only through Azure Monitor. 

 

I´m experiencing a lot of delay in receiving the alerts from the monitoring platform over the last days. 

Thus this resulting in massive spam Messages to the TeamsChannel/email/SMS Contacts, for that purpose.  

 

Here's a preview of what I'm talking about: 

loadedlouie27_0-1593530772591.png

The same alert was triggered at the same minute a couple of times, I'm guessing by delayed ingestion on the component that actually triggers the alerts, and not the log analytics itself since I can see a process running on a machine within a few minutes. 

 

First question: 

Is this supposed to be a viable product at this point?

Is this something companies can really rely on or am I pushing too much and expecting much more than this type of behavior from this solution?


Second Question: 

Is there a way to see whats the actual delay on the alert side? 

I know I can see the ingest time of the log analytics, however, I can tell nothing is wrong on the time its ingested, instead, we see a massive delay on sending the alerts, and showing them on the alert console. resulting in a non-reliable product... 

 

Third Question: 

Is there a place we find a "bible" on monitoring with azure, I find the documentation, sometimes too much vague. And there's not that much info about monitoring with azure monitor, sentinel, log analytics, create metrics from log analytics, etc...
But it can be me... I'm new in the cloud... 

 

Fourth Question: 

Why do alerts, that are fired, using a custom search, don't change the monitor condition to resolved?
even with metric type, using aggregated and time generated 

Anything I need to do in my query? 

 

 

Thanks in advance for your time and help. 

4 Replies

@loadedlouie27 

Adding @Robi Czitron and @OlegAnaniev  from Alerts and Log Analytics team

@Ketan Ghelani Tks for the help, but i dont think anybody can respond to this. 

 

:) 

 

@loadedlouie27 There is a lot of questions, but I'll answer generally.

 

Log alerts is fully GA and we can assist you in these cases via the official support channels. Our documentation is available for assisting getting you started with the different monitoring options.

 

Log alerts works best when looking for data in the log and less well when looking for lack of data (such as heartbeat). Ingestion delay can impact these alerts:

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-ingestion-time

This means when this happens, you could experience false alerts or late alerts. 

I would recommend you use metric alerts for those use cases unless you need the power of a log alert custom query.

Saying that we are introducing a new flow this month that should improve accuracy of the alerts and lower the chances of you hitting issues.

 

Log search alerts are stateless by design. We are working on adding stateful log alerts that also resolve.

 

@yalavihi thank you for your time, 

If you don't mind I have a few questions, I encounter a major issue, in my opinion,

using the current solution, and I would like to know if they are gonna be addressed,
or if they are out of the scope for the current road map. 

 

Are you guys thinking about making the alerts fired being grouped?

This is one of the major issues I currently see in using the Solution. 

What I mean is: I have the same alert been checked every 5 minutes, and if it triggered, the alert just keeps on repeating itself and having like 2000 alerts for the same threshold/rule, its kind of a killer, for using the tool correctly, in my opinion. 

If you go asleep at night, you might wake up the next morning, for a rule that has created 2000 alerts in 8 hours, and have to close the alerts "by hand".

What I'm suggesting its something a bit kind of Azure Sentinel grouping. 

Is the "Alert Console" going to be reworked, or allow further customization?

 

On the Monitoring Side is there any place i can find a direct match from the tables been monitored? 

What I mean is:  is there a way I can see where to activate, and what,
in order to get data into a given table in log analytics? 

 

Thanks in advance, and I'm sorry for my questions, they may be seen a bit noobish,
but I think some of the topics are like elephants in the room, at least in some documentations in Microsoft. 

Or the information is so dispersed, that I have trouble getting it.