How to Create Azure Monitor Alerts for Non-Compliant Azure Policies

[Tao Yang]

Azure Policy allows organization to define and enforce how Azure resources should or should not be used. It allows for the ability to block resources with certain configurations from being created, or generate audit events when a particular configuration is used. It can also modify resources at creation stage to ensure it meets standards set by your organization.

Azure Policy can be used to:

• Restrict users only use a list of Azure regions approved by your organization
• Audit if a Public IP is being used
• Enforce Hybrid Use Benefit is enabled on all Windows Server VMs
• Automatically deploy the Log Analytics agent on any newly created VMs

The list can go on and on. Currently Azure provides over one hundred built-in policy definitions that can be utilized to consume free of charge, and also provides the ability to develop your own policy definitions if none of the built-in policy meets your requirements.

Azure policy engine evaluates policy assignments when ARM resources are created or updated within the assignment scope. It also runs delta and full scans on a schedule – to detect non-compliant resources.

A resource can be marked as non-compliant by audit policies, or detected by the periodic scans (for example, resources that were created before the policies were assigned).

In the Azure portal, you can view the list of non-compliant policies (as shown below):

Azure Monitor can generate alerts and trigger pre-defined actions when non-complaint resources are detected.

This blog post detail how to use Kusto query language to detect non-compliant resources by querying your Log Analytics workspace, and then how to create Azure monitor alert rules to take actions when non-compliant resources are detected by Azure Policy.

**** UPDATE Dec 2021: The following KQL scenario no longer functions, because Azure Policy no longer publishes resource compliance states to the Azure activity log. Author Tao Yang has researched an alternative solution using Azure Event Grid - find the details here: https://blog.tyang.org/2021/12/06/monitoring-azure-policy-compliance-states-2021-edition/ **** 

The Azure policy compliance status is logged in the Azure subscription’s Activity logs. The Azure Log Analytics workspace can be configured to collect Azure Activity logs from any subscriptions in the same tenant. Azure Monitor alert rules can then be created to execute queries in the Log Analytics workspace on a schedule and generate alerts when non-compliant resources are detected by the query.

Before we start creating alert rules, we need to make sure Azure Activity logs are being collected by a Log Analytics workspace. To configure a Log Analytics workspace to collect Azure Activity logs, you need to navigate to the Azure Activity log blade of the workspace in the portal, and connect the subscriptions (as shown below):

Once the Log Analytics workspace started collecting logs, the policy compliance status will be collected to your workspace.

The following Kusto queries can be used to check these logs in your Log Analytics workspace:

1. Get a list of non-compliant policy with resource count

AzureActivity | where Category == 'Policy' and Level != 'Informational' | extend p=todynamic(Properties) | extend policies=todynamic(tostring(p.policies)) | mvexpand policy = policies | summarize resource_count=count() by tostring(policy.policyDefinitionName)

2. Get a list of non-compliant resources from a single policy (using “audit-resources-without-tags-policyDef” definition as an example)

let policyDefId = 'audit-resources-without-tags-policyDef'; AzureActivity | where Category == 'Policy' and Level != 'Informational' | extend p=todynamic(Properties) | extend policies=todynamic(tostring(p.policies)) | mvexpand policy = policies | where policy.policyDefinitionName in (policyDefId) | distinct ResourceId

3. Get newly detected non-compliance resources from a single policy (using using “audit-resources-without-tags-policyDef” definition as an example):

let policyDefId = 'audit-resources-without-tags-policyDef'; AzureActivity | where Category == 'Policy' and Level != 'Informational' | extend p=todynamic(Properties) | extend policies=todynamic(tostring(p.policies)) | mvexpand policy = policies | where policy.policyDefinitionName in (policyDefId) and p.isComplianceCheck == 'False'

NOTE: As previously mentioned, the Azure Policy engine runs periodic scans across all resources (compliance check). The full scan runs every 24 hours. In this example, we have added a condition in the end to only retrieve non-compliant resources that are not detected by the compliance check (p.isComplianceCheck == ‘False”). Without this condition, you will get spammed with any new and existing non-compliant resources after every compliance check scan. This will most likely cause duplicate alerts being generated too.

Based on these sample queries, you can use the following queries to build your alert rules based on your scenarios:

1. To alert on any new non-compliant resources:

AzureActivity | where Category == 'Policy' and Level != 'Informational' | extend p=todynamic(Properties) | extend policies=todynamic(tostring(p.policies)) | mvexpand policy = policies | where p.isComplianceCheck == 'False'

2. To alert on new non-compliant resources detected by a particular policy:

let policyDefId = '<enter-policy-definition-id-here>'; AzureActivity | where Category == 'Policy' and Level != 'Informational' | extend p=todynamic(Properties) | extend policies=todynamic(tostring(p.policies)) | mvexpand policy = policies | where policy.policyDefinitionName in (policyDefId) and p.isComplianceCheck == 'False'

The alert rule can be created from the Azure portal or by using ARM templates.

Follow the following steps to create the alert rule via the portal:

1. Browse to Azure Monitor and click on Alerts blade

2. Click on New alert rules

3. On the Create rule page, select the appropriate subscription and the Log Analytics workspace

4. Add a condition, on the Configure signal logic page, select Custom log search

5. Enter the search query, and configure the alert logic as Number of results greater than 0, specify the period and frequency (or leave it as default)6. Select an existing action group, or create a new one

7. Provide alert name, severity and other optional information such as email subject, and finish the wizard

Since I have configured the action group to send me an email, when next time a non-compliant resource is created, I will get an email within few minutes (as shown below):

NOTE: Action groups in Azure Monitor supports other action types such as webhook, mobile push notifications, SMS, LogicApp, Azure Function, ITSM, etc. Although I have only used email in this example, you can customise the action group to suit your requirements (i.e. creating a ticket in ServiceNow via the ITSM connector.

A sample template is available in my GitHub repo if there is a need to create the alert rule using ARM template: https://github.com/tyconsulting/azurepolicy/tree/master/arm-templates/monitor-alert-rule-for-non-com...

This sample template is a subscription level ARM template, it creates a resource group, Azure monitor action group and alert rule within the resource group. Since it is a subscription level template, you will need to use new-azdeployment cmdlet to deploy it. For example:

new-azdeployment -Name sampleAlertRule -Location australiasoutheast -TemplateFile .\azuredeploy.json -logAnalyticsWorkspaceResourceId /subscriptions/<sub-id>/resourcegroups/<log-analytics-rg>/providers/microsoft.operationalinsights/workspaces/workspace-name

NOTE:

The “new-azdeployment” cmdlet is part of the Azure PowerShell module. To execute this command, you must firstly install the “Az” PowerShell module and login to your Azure subscription using the following commands:

#Install Az module from the PowerShell gallery. Run it as administrator
Install-Module Az -Repository PSGallery -Force -AllowClobber

#Login to Azure
Connect-AzAccount

To make use of the sample ARM template, you also need to update the PolicyDefId from line 22 of the template JSON file. To retrieve the full list of available Azure policy definitions in your environment, execute the command below once you have signed in to Azure using Connect-AzAccount cmdlet:

get-azpolicydefinition |format-table ResourceName, @{Name='DisplayName'; Expression={$_.Properties.DisplayName}}, ResourceId

The policy definition Id is listed under the ResourceName column (as shown below):

In this post, I have used a custom policy definition so the policy Id is more readable (the names of built-in policies are all GUIDs). The custom policy can be found at my GitHub repo: https://github.com/tyconsulting/azurepolicy/tree/master/policy-definitions/audit-resources-without-t...

In this post, I have demonstrated how to use Kusto search queries in Azure Log Analytics to detect non-compliant resources detected by Azure Policy, and how to create alert rules in Azure Monitor when a new non-compliant resource is detected. If you’d like to learn more about Azure Policy and Azure Monitor, make sure you check out our free eBook Inside Azure Management (https://bit.ly/InsideAzureMgmt)

Lastly, I’d like to thank my friend and fellow MVP Stanislav Zhelyazkov (@StanZhelyazkov) for helping me out with fine-tuning the Kusto queries used in the alert rule.

Tao

[Maxim_M]

@Tao Yang  , How did you get policy_policyDefinitionName field to show the actual name of the policy instead of just the ID ? I get the ID in that field regardless of whether it is a custom policy or a built-in one.

EDIT: answered my own question - creating custom policies through the portal results in the 'name' attribute being assigned the definition ID. Same thing happens when duplicating definitions and customizing them later on. 

Creating the custom policy with PowerShell sets the name correctly. I used the following: New-AzPolicyDefinition -Name "<PolicyName>" -DisplayName "<PolicyDisplayName>" -Policy <Policy JSON file>.

I've yet to test that with the ARMClient, but suspect the result will be the same using the below:

# For defining a policy in a subscription armclient

PUT "/subscriptions/{subscriptionId}/providers/Microsoft.Authorization/policyDefinitions/AuditStorageAccounts?api-version=2019-09-01" @<path to policy definition JSON file>

# For defining a policy in a management group armclient

PUT /providers/Microsoft.Management/managementgroups/{managementGroupId}/providers/Microsoft.Authorization/policyDefinitions/AuditStorageAccounts?api-version=2019-09-01" @<path to policy definition JSON file>

[Jeff Walzer]

@Maxim_M - when you say, "Creating the custom policy with PowerShell sets the name correctly." does that mean I can simply overwrite an existing custom policy using the PS command?

I have an existing custom policy that has the name set as the definition ID and I'd like to assign a proper name, but I'm unsure if using the PS command will overwrite the policy altogether, or simply change the name and keep all existing policy parameters,

Thx

[Maxim_M]

@Jeff Walzer In my post i meant that if I create a brand new policy using the powershell cmdlet then it would create it with the proper name. If you already have an existing policy that has the Name attribute set as the Definition Id then it means it was either created thru the portal (how i did got confused when I was first testing), or via powershell omitting the attribute.

In your case I'd explore creating a new identical policy with the correct descriptive name and assign it to the same scopes as the one with the ID as the descriptive name. Then simply deprovision the old one. This is made a bit easier if you have initiatives as you can simply swap them out. Do keep in mind that if you have any reports/historical data you need to reference as part of reports/compliance audits then you'd need to document this change as it might be confusing someone who might be looking at let's say 2 yrs worth of compliance data to see the policy change all of the sudden.

If you are simply testing like i was then it's as easy as blowing away the old policy! :)

[rbrewer938]

What is the indicator/parameter in the Activity log that determines a resource "non-compliance" or "compliance" result and how do you know?

The key filters I saw in your code were:

where Category == 'Policy' and Level != 'Informational'

and

where policy.policyDefinitionName in (policyDefId) and p.isComplianceCheck == 'False'

From what I understand "isComplianceCheck": "True" == 'False' is related to the once every 24hr check that Azure does and Level != "Informational" is related to the event category. 

[dsv591]

@Tao Yang Azure Policy does no longer write to Activity Log on daily scans/manually triggered scans (aka brownfield).

Can you please call that out in the article? This is causing some confusion to customers.

The approach is still valid but only for entries in Activity Log generated by PUT/PATCH calls where the policy is evaluated in enforcement mode.

[Tao Yang]

December 2021 Update: I have created a brand new monitoring solution for Azure Policy compliance status since the solution introduced on this article no longer works. the new solution can be found on my blog post Monitoring Azure Policy Compliance States - 2021 Edition

[spacex9]

Hello Team, Can you please help me to create a new Azure monitor alert when there is a Non-compliant in azure policy?