Jan 11 2019
03:05 PM
- last edited on
Apr 07 2022
05:35 PM
by
TechCommunityAP
Jan 11 2019
03:05 PM
- last edited on
Apr 07 2022
05:35 PM
by
TechCommunityAP
I've got an Application Insight set up for one of our App Services, and what I'm trying to do is get alerted when more than a certain percentage of requests fail. We're not looking to hardcode values due to the fact that our traffic levels vary wildly throughout the year and having to manually enter them over and over seems unnecessary.
Let's pretend I want to be alerted when more than 1% of HTTP requests fail over a rolling period of 5 minutes. At a high level, I understand I need to create a Custom Log Alert that does something to the effect of:
- Count the total number of HTTP requests in the previous 5 minutes (totalRequests)
- Count the number of failed HTTP requests in the previous 5 minutes (failedRequests)
- Math: failedRequests / totalRequests = failedPercentage
- If: failedPercentage > 0.01, trigger alert
But what I don't understand is how I'd implement that logic in the custom log alerts. I've got as far as creating this search query:
requests
| summarize failedRequests = countif(resultCode == "500"), totalRequests = count()
| project failedPercentage = failedRequests / totalRequests
| extend should_alert = failedPercentage > 0.01
which is just based on some example code sent to me by an MS Support Engineer (so I apologize if I've made some assumptions with the summarize/project/extend functions), but rather than doing what I want, it updates the chart on the top part of the blade to be a perfect 1.0 which is definitely incorrect, since while we have had some errors in the previous 5 minute interval, the site is most certainly up and generally responding. I've also changed the period and the frequency, yet no matter what options I try, the line graph stays at 1.0.
Can someone tell me what I'm missing here, and then also, what I should do with "should_alert" to get it to trigger, or if that's a reserved keyword for this service that takes the place of the fields at the bottom of the window?
Jan 12 2019 08:04 AM
I did a post a while ago that may help. Different scenario, but similar idea. I did a lookback average from 30 minutes ago to 5 minutes ago, and compared that to the last 5 minute average. If the 5 minute average was above a threshold, create an alert.
https://www.ciraltos.com/alerts-based-on-rolling-averages-in-log-analytics/
Hope this helps.