User Profile
Roman_Turovskyy
Copper Contributor
Joined Dec 26, 2018
User Widgets
Recent Discussions
Reliably trigger alerts for Log Analytics log entries
MSDN documentation at https://docs.microsoft.com/en-us/azure/azure-monitor/platform/alert-log-troubleshoot states: "To mitigate data ingestion delay, the system waits and retries the alert query multiple times if it finds the needed data is not yet ingested". We have an issue with triggering alerts and the issue suggests that described behavior is not very reliable as a lot of our alerts aren't fired. To be more precise - we ingress logs from Data Factory V2 into Log Analytics and watch for log entries with Level == "Error", based on number of results greater that 0 (Period = Frequency = 30 minutes). We expect that in case a log entry with Level == "Error" is generated by Data Factory and ingested into Log Analytics we shall receive an alert, but very often we don't. We tried to change Period to larger values (30 minutes) leaving Frequency at 15 but in this case there is a big chance to receive duplicated alerts which also is not good. Are there any recommended and reliable Period/Frequency/Query configuration strategy that guarantees no alerts are missing and also does not produce duplicated alerts?Solved12KViews0likes10CommentsHow to preserve existing tags when re-deploying resource via ARM template
I've noticed that each time we deploy our resource via ARM template this results in all tags that was set manually on these resource to be removed (more precicely - replaced to ones specified in ARM template). In this an expected behavior? Are there any easy way to preserve custom tags and not erase them? (non-easy way example is: read tags on resource before running ARM template and pass all these tags back to ARM). ARM template mode is Incremental.1.4KViews0likes1CommentRe: Reliably trigger alerts for Log Analytics log entries
The weird thing is that action group seems to be attached when I create alert via ARM. At least I can see it on Monitor -> Alerts -> "Manage alert rules" page (image 1 in attached screenshot). Action group is only missing when looking at alert rule via link from triggered alert instance ("Alert rule" in the "Essentials" section, image 2 in attached screenshot). A tried to get alert action JSON using REST API - reference to action group it is there. After re-saving an alert from the Portal or by get/put REST API calls nothing changes in action JSON (except etag), but somehow such re-save fixes the issue, so something internal is definitely changed. Here goes sample request I used to get alert action: $actionUrl = "/subscriptions/{subscription id}/resourceGroups/{res group name}/providers/Microsoft.OperationalInsights/workspaces/cdm1drepomsf01/savedSearches/saved_search60eee2d2dc0b42dd87cd0a06b1c3f335/schedules/schedule_60eee2d2dc0b42dd87cd0a06b1c3f335/actions/action_60eee2d2dc0b42dd87cd0a06b1c3f335?api-version=2015-03-20" $jsonStr = armclient get $actionUrl And here is what I used to re-save alert action via REST API: $json = $jsonStr | ConvertFrom-Json $json2 = @{ etag=$json.etag properties=$json.properties } $json2 = $json2 | ConvertTo-Json -Depth 3 $json2 | armclient put $actionUrl API samples may be found here: https://docs.microsoft.com/en-us/azure/azure-monitor/platform/api-alerts. You've mentioned that link points to an old API (it has "(Preview)" in title). Do you have a link to a new ARM API for Log Analytics alerts creation?12KViews0likes2CommentsRe: Reliably trigger alerts for Log Analytics log entries
Stanislav, thank you a lot for your replies. Just for completeness I'd like to provide an update on my issue. After getting access to Alerts on a Subscription level we've realized that all alerts are actually triggered, so there are no bugs in the documentation, everything works fine even with 5 minutes Interval. The actual but happened to be that triggered alerts have theit Action Group property empty, thats why emails are not sent. Action Group is not empty when viewing alert rules from Monitor -> Alerts -> "Manage alert rules" page, but it is empty when navigating to alert rule via link inside triggered alert instance. This definitely looks strange, we're already working on this issue with MS support. The initial investigation showed that problem might be with deploying Log Analytics alerts using ARM template. Whn we manuall create alert rule from the Portal all is fine, but when they are created with the help of ARM - Action Goup on triggered alerts is empty for some reason. The currently found workarounds: 1. After ARM deployment go to the Portal and manually re-save alert rules. 2. After ARM deployment use REST API to get and set alert actions (this is also just "re-save" with no modification). The ARM is based on examples from page https://docs.microsoft.com/en-us/azure/azure-monitor/insights/solutions-resources-searches-alerts.12KViews0likes4CommentsRe: Reliably trigger alerts for Log Analytics log entries
Thanks for reply. I have one more question regarding "The only reason why you are not seeing the alerts in Azure Monitor if you haven't selected the subscription of the where the Log Analytics wokrspace is located". I have only one subscription to choose from, so there is no chance I can select the wrong one. Additional investigation revealed that my colleague can see alerts, but he is an Owner of the Subscription, while I am a Contributor of Resource Group where alerts are created. I can create alerts, change them, but cannot see which alerts were triggered. Unfortunately I cannot be granted any rights on global Subscription level, are there any way to configure a per Resource Group access so I'll be able to see alerts?12KViews0likes6CommentsRe: Reliably trigger alerts for Log Analytics log entries
The exact query is: search * | where ResourceProvider == "MICROSOFT.DATAFACTORY" and (Level == "Error" or status_s == "Failed") | order by TimeGenerated Query is running over Log Analytics to which Data Factory V2 writes them (with several minutes delay, but it is hard to tell the exact numbers). When I set Period = Frequency = 5 minutes then more than 50% of alert emails are missing, for Period = Frequency = 15 almost all logs relult in alert email, but not 100% all. Except described issue there is a more severe issue, which may be related to the described one. When I navigate to Monitor -> Alerts I always see "All is good! You have no alerts." message which is really strange. I expect to see statistics about triggered alerts. Because of this "You have no alerts." message it is hard to be sure that the issue is with alerts but not with emails (configured via Action Group). Our assumption was "there might be an issue with emails delivering, e.g. because of spam filters" but this assumption was dismissed after we configured Azure Function action type - azure functions are not invoked when emails are missing and are invoked when emails are delivered, so at least there is consistency with emails and Azure Function action types. What may be the reason of "All is good! You have no alerts." message is always present?12KViews0likes8Comments
Recent Blog Articles
No content to show