I've been in discussion with Azure Monitor Support (2205240050001397) regarding alerts for resources created through ARM deployments, and it was suggested I also raise this here.
To sum up; we configured alerts in Azure Monitor when custom roles are created or updated, or users are assigned to roles. We deploy applications from ARM templates which include required role assignments for their resource groups, for example, Key Vault Azure RBAC.
We deploy via a CI/CD pipeline, which could be 10-12 times a day. The deployment triggers 12 notifications.
We have two key problems;
1. The notification content is not great, its not easy to see that "User first.last was added to the Owner role in the ResourceGroup resource group".
2. Because the operations against ARM are PUT operations, an activity log entry is placed to say the entry is created for every deployment, even if the role or assignment has not actually changed. Given we're operating CI/CD, this can be well over 100 notifications a day for a resource assignment that has not changed - this renders the alerts unusable.
The support agent has been very helpful and is raising with the product teams to see if this was a design consideration or an oversight .... Are we missing something that would make this possible?
A workaround has been suggested which involves changing the template to set a new name each time so the role assignment would be seen as a duplicate, and will fail deployment - while this is the case, it means all our deployments will fail, and will report back to the CI/CD pipeline as a failure.
Role assignment in activity log:
Attempt to re-deploy the same assignment with new name: