Forum Discussion
Reliably trigger alerts for Log Analytics log entries
Hi,
The new API is discussed here:
I haven't published examples on my blog as I try to avoid publishing things before they are are announced officially but I have been using the API for several weeks now. It had some bugs that I hope are fixed/or will be fixed before official release.
Hi,
The first thing you should do is to never use search operator in alerts or any kind of saved queries. Search operator is usually used on discovering data initially but once you know where your data is you should stop using it. This is also described here:
https://azure.microsoft.com/en-us/blog/best-practices-for-queries-used-in-log-alerts-rules/
I am assuming that for Data factory you use diagnostic logs which are send to Log Analytics. In that case your data is in AzureDiagnostics table so your query should look like:
AzureDiagnostics | where ResourceProvider == "MICROSOFT.DATAFACTORY" and (Level == "Error" or status_s == "Failed") | order by TimeGenerate
You can also skip | order by TimeGenerated when you use it in alert as it does not have any affect there.
Here is some information how ingestion time can be checked although I do not think that is the problem for you:
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-ingestion-time
Keep in mind that in e-mail only 10 results will be added to the e-mail but if you go to the link of the alert query results you will see all the results.
The only reason why you are not seeing the alerts in Azure Monitor if you haven't selected the subscription of the where the Log Analytics wokrspace is located. Azure Monitor can display alerts from 5 subscription maximum.
I usually use metric measurement based alerts rather number of results type as that way I can get alert per instance. I have a small blog describing that scenario here:
Thanks for reply. I have one more question regarding "The only reason why you are not seeing the alerts in Azure Monitor if you haven't selected the subscription of the where the Log Analytics wokrspace is located". I have only one subscription to choose from, so there is no chance I can select the wrong one. Additional investigation revealed that my colleague can see alerts, but he is an Owner of the Subscription, while I am a Contributor of Resource Group where alerts are created. I can create alerts, change them, but cannot see which alerts were triggered. Unfortunately I cannot be granted any rights on global Subscription level, are there any way to configure a per Resource Group access so I'll be able to see alerts?
I think the alert instances itself are subscription level resource so you will need some access at subscription level not resource group. You do not need necessary to be Contributor at subscription level, for example you can be Monitoring contributor. You can also opt-in at creating your own custom role and have access to resources like Microsoft.AlertsManagement/alerts/ and Microsoft.AlertsManagement/alertsSummary/ at subscription level.
Stanislav, thank you a lot for your replies. Just for completeness I'd like to provide an update on my issue. After getting access to Alerts on a Subscription level we've realized that all alerts are actually triggered, so there are no bugs in the documentation, everything works fine even with 5 minutes Interval.
The actual but happened to be that triggered alerts have theit Action Group property empty, thats why emails are not sent. Action Group is not empty when viewing alert rules from Monitor -> Alerts -> "Manage alert rules" page, but it is empty when navigating to alert rule via link inside triggered alert instance. This definitely looks strange, we're already working on this issue with MS support. The initial investigation showed that problem might be with deploying Log Analytics alerts using ARM template. Whn we manuall create alert rule from the Portal all is fine, but when they are created with the help of ARM - Action Goup on triggered alerts is empty for some reason. The currently found workarounds:
1. After ARM deployment go to the Portal and manually re-save alert rules.
2. After ARM deployment use REST API to get and set alert actions (this is also just "re-save" with no modification).
The ARM is based on examples from page https://docs.microsoft.com/en-us/azure/azure-monitor/insights/solutions-resources-searches-alerts.
If the action group is not attached to the alert via ARM that means the action group was not referenced correctly. You have to make sure that the resource id of the action group is correct. I think there was also some bug that I've reported some time ago on the old Log Analytics alerts API where if the action group name contains white spaces the API cannot find the Action Group resource. The API also does not verifies if the action group exists so if it does not exist it will create the alert anyway. Workaround for that bug was to use name without white spaces so the resource ID can be correct or or to encode the name of the action group resource when you construct the resource id.
I see that you provide link to the old API so probably that bug still exists.