SOLVED

Alerting on Heartbeat issue

%3CLINGO-SUB%20id%3D%22lingo-sub-455450%22%20slang%3D%22en-US%22%3EAlerting%20on%20Heartbeat%20issue%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-455450%22%20slang%3D%22en-US%22%3E%3CP%3ESurely%20there%20is%20a%20better%20solution%20for%20this%3F%20My%20use%20case%20doesn't%20work%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E1.%20Create%20a%20computer%20group%3C%2FP%3E%3CP%3E2.%20Alert%20when%20an%20agent%20in%20computer%20group%20has%20not%20%22heartbeated%22%20for%20%3CSTRONG%3Eover%2024%20hours%3C%2FSTRONG%3E.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EBy%20the%20logic%20in%20Alerts%2C%20even%20if%20I%20set%20the%20query%20as%20I%20do%20below%2C%20the%20time%20span%20that%26nbsp%3B%3CSTRONG%3EI%20define%3C%2FSTRONG%3Eis%26nbsp%3Bignored%20because%20of%20the%20%22Period%22%20in%20Alerts%3A%3C%2FP%3E%3CP%3E%3CBR%20%2F%3EHeartbeat%3CBR%20%2F%3E%7C%20project%20TimeGenerated%2C%20Computer%3CBR%20%2F%3E%7C%20where%20TimeGenerated%20%26lt%3B%20now()%3CBR%20%2F%3E%7C%20where%20Computer%20in%20(COMPUTERGROUP)%3CBR%20%2F%3E%7C%20summarize%20%5B%22Last%20Heartbeat%22%5D%3Dmax(TimeGenerated)%20by%20Computer%3CBR%20%2F%3E%7C%20where%20%5B%22Last%20Heartbeat%22%5D%20%26lt%3B%20ago(24h)%3CBR%20%2F%3E%3CBR%20%2F%3EThis%20query--when%20run%20outside%20of%20Alerts--returns%20several%20machines%20that%20have%20no%20heartbeat%20in%20the%20last%2024%20hours%2C%20going%20back%20to%20as%20long%20as%20we've%20been%20collecting%20the%20data.%20But%20because%20Alerts%20confines%20me%20to%20a%20max%2024%20hour%20period%20to%20check%20against%2C%20I%20get%200%20results.%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3EI%20essentially%20want%20an%20alert%20generated%20every%2024%20hours%20as%20a%20%22nag%20alert%22%20with%20a%20list%20of%20the%20machines%20that%20have%20not%20sent%20heartbeat%20data%20in%20over%2024%20hours.%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3EIs%20there%20any%20way%20to%20get%20around%20this%20extremely%20limiting%20design%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-455450%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAlerts%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Log%20Analytics%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Monitor%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-455599%22%20slang%3D%22en-US%22%3ERe%3A%20Alerting%20on%20Heartbeat%20issue%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-455599%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F151992%22%20target%3D%22_blank%22%3E%40Scott%20Allison%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAlerts%20are%20designed%20to%20look%20back%20on%2024hrs%20as%20you%20state%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EFor%20a%20report%20of%20this%20nature%2C%20I'd%20suggest%20a%20%3CSTRONG%3ELogic%20App%3C%2FSTRONG%3E%2C%20something%20like%20this%20mock%20up%3F%26nbsp%3B%20This%20fires%20at%20a%20pre-set%20time%20(recurrence)%2C%20then%20runs%20your%20query%2C%20then%20sends%20an%20email%20(you%20could%20send%20to%20teams%2FSlack%2FServiceNow%20etc..%20instead%20or%20in%20parallel)%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F108909iC1B907F38653C84F%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22Annotation%202019-04-15%20171941.jpg%22%20title%3D%22Annotation%202019-04-15%20171941.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20also%20like%20the%20example%20availability%20rate%20query%3A%3C%2FP%3E%0A%3CPRE%3E%2F%2F%20Availability%20rate%0A%2F%2F%20Calculate%20the%20availability%20rate%20of%20each%20connected%20computer%0AHeartbeat%0A%2F%2F%20bin_at%20is%20used%20to%20set%20the%20time%20grain%20to%201%20hour%2C%20starting%20exactly%2024%20hours%20ago%0A%7C%20summarize%20heartbeatPerHour%20%3D%20count()%20by%20bin_at(TimeGenerated%2C%201h%2C%20ago(24h))%2C%20Computer%0A%7C%20extend%20availablePerHour%20%3D%20iff(heartbeatPerHour%20%26gt%3B%200%2C%20true%2C%20false)%0A%7C%20summarize%20totalAvailableHours%20%3D%20countif(availablePerHour%20%3D%3D%20true)%20by%20Computer%20%0A%7C%20extend%20availabilityRate%20%3D%20totalAvailableHours*100.0%2F24%0A%7C%20project-away%20totalAvailableHours%20%0A%7C%20render%20barchart%20%3C%2FPRE%3E%0A%3CP%3ENote%3A%20I%20added%20the%20last%20two%20lines%2C%20as%20I%20prefer%20how%20it%20looks%20as%20a%20chart%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-460129%22%20slang%3D%22en-US%22%3ERe%3A%20Alerting%20on%20Heartbeat%20issue%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-460129%22%20slang%3D%22en-US%22%3E%3CP%3EThanks%20Clive.%20T%3CBR%20%2F%3E%3CBR%20%2F%3Ehis%20is%20a%20pretty%20straightforward%20recommendation.%20It%20does%2C%20however%2C%20stray%20from%20our%20deliberate%20move%20to%20Azure%20Monitor%20for%20all%20alerting%20(taking%20advantage%20of%20Action%20Groups%20and%20automation).%20It%20would%20be%20nice%20to%20have%20the%20option%20to%20remove%20some%20of%20these%20%22guardrails%22%20for%20Alerts...%20or%20at%20the%20very%20least%2C%20have%20a%20viable%20explanation%20as%20to%20why%20the%20guardrail%20is%20necessary.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Ecc%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F152080%22%20target%3D%22_blank%22%3E%40Daniel%20Thilagan%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-460854%22%20slang%3D%22en-US%22%3ERe%3A%20Alerting%20on%20Heartbeat%20issue%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-460854%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F151992%22%20target%3D%22_blank%22%3E%40Scott%20Allison%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20best%20public%20explanation%20I've%20seen%20is%3A%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ffeedback.azure.com%2Fforums%2F267889-log-analytics%2Fsuggestions%2F32043751-alerting-timewindow-limitation-of-24-hours-makes-a%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Ffeedback.azure.com%2Fforums%2F267889-log-analytics%2Fsuggestions%2F32043751-alerting-timewindow-limitation-of-24-hours-makes-a%3C%2FA%3E%26nbsp%3Bwhich%20gives%20an%20explanation%20-%20you%20could%20add%20your%20'vote'%20to%20this%3F%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThanks%20Clive%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-466141%22%20slang%3D%22en-US%22%3ERe%3A%20Alerting%20on%20Heartbeat%20issue%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-466141%22%20slang%3D%22en-US%22%3EVoted%20and%20commented.%3C%2FLINGO-BODY%3E
Highlighted
Frequent Contributor

Surely there is a better solution for this? My use case doesn't work:

 

1. Create a computer group

2. Alert when an agent in computer group has not "heartbeated" for over 24 hours

 

By the logic in Alerts, even if I set the query as I do below, the time span that I defineis ignored because of the "Period" in Alerts:


Heartbeat
| project TimeGenerated, Computer
| where TimeGenerated < now()
| where Computer in (COMPUTERGROUP)
| summarize ["Last Heartbeat"]=max(TimeGenerated) by Computer
| where ["Last Heartbeat"] < ago(24h)

This query--when run outside of Alerts--returns several machines that have no heartbeat in the last 24 hours, going back to as long as we've been collecting the data. But because Alerts confines me to a max 24 hour period to check against, I get 0 results. 

I essentially want an alert generated every 24 hours as a "nag alert" with a list of the machines that have not sent heartbeat data in over 24 hours. 

Is there any way to get around this extremely limiting design?

4 Replies
Highlighted
Solution

@Scott Allison 

 

Alerts are designed to look back on 24hrs as you state 

 

For a report of this nature, I'd suggest a Logic App, something like this mock up?  This fires at a pre-set time (recurrence), then runs your query, then sends an email (you could send to teams/Slack/ServiceNow etc.. instead or in parallel)

 

Annotation 2019-04-15 171941.jpg

 

I also like the example availability rate query:

// Availability rate
// Calculate the availability rate of each connected computer
Heartbeat
// bin_at is used to set the time grain to 1 hour, starting exactly 24 hours ago
| summarize heartbeatPerHour = count() by bin_at(TimeGenerated, 1h, ago(24h)), Computer
| extend availablePerHour = iff(heartbeatPerHour > 0, true, false)
| summarize totalAvailableHours = countif(availablePerHour == true) by Computer 
| extend availabilityRate = totalAvailableHours*100.0/24
| project-away totalAvailableHours 
| render barchart 

Note: I added the last two lines, as I prefer how it looks as a chart  

Highlighted

Thanks Clive. T

his is a pretty straightforward recommendation. It does, however, stray from our deliberate move to Azure Monitor for all alerting (taking advantage of Action Groups and automation). It would be nice to have the option to remove some of these "guardrails" for Alerts... or at the very least, have a viable explanation as to why the guardrail is necessary.

 

cc: @Daniel Thilagan 

Highlighted

Hi @Scott Allison 

 

The best public explanation I've seen is:

https://feedback.azure.com/forums/267889-log-analytics/suggestions/32043751-alerting-timewindow-limi... which gives an explanation - you could add your 'vote' to this?

 

Thanks Clive 

Highlighted
Voted and commented.