Forum Discussion

Roman_Turovskyy's avatar
Roman_Turovskyy
Copper Contributor
Jan 07, 2019
Solved

Reliably trigger alerts for Log Analytics log entries

MSDN documentation at https://docs.microsoft.com/en-us/azure/azure-monitor/platform/alert-log-troubleshoot states: "To mitigate data ingestion delay, the system waits and retries the alert query mult...
Azure Log Analytics
azure monitor
Roman_Turovskyy's avatar
Roman_Turovskyy
Copper Contributor
Jan 16, 2019

Stanislav, thank you a lot for your replies. Just for completeness I'd like to provide an update on my issue. After getting access to Alerts on a Subscription level we've realized that all alerts are actually triggered, so there are no bugs in the documentation, everything works fine even with 5 minutes Interval.

The actual but happened to be that triggered alerts have theit Action Group property empty, thats why emails are not sent. Action Group is not empty when viewing alert rules from Monitor -> Alerts -> "Manage alert rules" page, but it is empty when navigating to alert rule via link inside triggered alert instance. This definitely looks strange, we're already working on this issue with MS support. The initial investigation showed that problem might be with deploying Log Analytics alerts using ARM template. Whn we manuall create alert rule from the Portal all is fine, but when they are created with the help of ARM - Action Goup on triggered alerts is empty for some reason. The currently found workarounds:

1. After ARM deployment go to the Portal and manually re-save alert rules.

2. After ARM deployment use REST API to get and set alert actions (this is also just "re-save" with no modification).

The ARM is based on examples from page https://docs.microsoft.com/en-us/azure/azure-monitor/insights/solutions-resources-searches-alerts.

Stanislav_Zhelyazkov's avatar
Stanislav_Zhelyazkov
MVP
Jan 16, 2019

If the action group is not attached to the alert via ARM that means the action group was not referenced correctly. You have to make sure that the resource id of the action group is correct. I think there was also some bug that I've reported some time ago on the old Log Analytics alerts API where if the action group name contains white spaces the API cannot find the Action Group resource. The API also does not verifies if the action group exists so if it does not exist it will create the alert anyway. Workaround for that bug was to use name without white spaces so the resource ID can be correct or or to encode the name of the action group resource when you construct the resource id.

 

I see that you provide link to the old API so probably that bug still exists.

  • Roman_Turovskyy's avatar
    Roman_Turovskyy
    Copper Contributor
    Jan 17, 2019

    Stanislav, thank you very much! I just tried that new API and it works - emails are properly sent, action group does not disapper. Finally!

  • Roman_Turovskyy's avatar
    Roman_Turovskyy
    Copper Contributor
    Jan 16, 2019

    The weird thing is that action group seems to be attached when I create alert via ARM. At least I can see it on Monitor -> Alerts -> "Manage alert rules" page (image 1 in attached screenshot). Action group is only missing when looking at alert rule via link from triggered alert instance ("Alert rule" in the "Essentials" section, image 2 in attached screenshot).

    A tried to get alert action JSON using REST API - reference to action group it is there. After re-saving an alert from the Portal or by get/put REST API calls nothing changes in action JSON (except etag), but somehow such re-save fixes the issue, so something internal is definitely changed.

     

    Here goes sample request I used to get alert action:

    $actionUrl = "/subscriptions/{subscription id}/resourceGroups/{res group name}/providers/Microsoft.OperationalInsights/workspaces/cdm1drepomsf01/savedSearches/saved_search60eee2d2dc0b42dd87cd0a06b1c3f335/schedules/schedule_60eee2d2dc0b42dd87cd0a06b1c3f335/actions/action_60eee2d2dc0b42dd87cd0a06b1c3f335?api-version=2015-03-20"
    $jsonStr = armclient get $actionUrl

     

    And here is what I used to re-save alert action via REST API:

    $json = $jsonStr | ConvertFrom-Json
    $json2 = @{
      etag=$json.etag
      properties=$json.properties
    }
    $json2 = $json2 | ConvertTo-Json -Depth 3
    $json2 | armclient put $actionUrl

     

    API samples may be found here: https://docs.microsoft.com/en-us/azure/azure-monitor/platform/api-alerts.

     

    You've mentioned that link points to an old API (it has "(Preview)" in title). Do you have a link to a new ARM API for Log Analytics alerts creation?

Resources