Optimize Azure Log Costs: Split Tables and Use the Auxiliary Tier with DCR
This blog is continuation of my previous blog where I discussed about saving ingestion costs by splitting logs into multiple tables and opting for the basic tier! Now that the transformation feature for Auxiliary logs has entered Public Preview stage, I’ll take a deeper dive, showing how to implement transformations to split logs across tables and route some of them to the Auxiliary tier. A quick refresher: Azure Monitor offers several log plans which our customers can opt for depending on their use cases. These log plans include: Analytics Logs – This plan is designed for frequent, concurrent access and supports interactive usage by multiple users. This plan drives the features in Azure Monitor Insights and powers Microsoft Sentinel. It is designed to manage critical and frequently accessed logs optimized for dashboards, alerts, and business advanced queries. Basic Logs – Improved to support even richer troubleshooting and incident response with fast queries while saving costs. Now available with a longer retention period and the addition of KQL operators to aggregate and lookup. Auxiliary Logs – Our new, inexpensive log plan that enables ingestion and management of verbose logs needed for auditing and compliance scenarios. These may be queried with KQL on an infrequent basis and used to generate summaries. Following diagram provides detailed information about the log plans and their use cases: More details about Azure Monitor Logs can be found here: Azure Monitor Logs - Azure Monitor | Microsoft Learn **Note** This blog will be focussed on switching to Auxiliary logs only. I would recommend going through our public documentation for detailed insights about feature-wise comparison for the log plans which should help you in taking right decisions for choosing the correct log plans. At this stage, I assume you’re aware about different log tiers that Azure Monitor offers and you’ve decided to switch to Auxiliary logs for high volume, low-fidelity logs. Let’s look at the high-level approach we’re going to follow to achieve this: Review the relevant tables and figure out which portion of the log can be moved to Auxiliary tier Create a DCR-based custom table which same schema as of the original table. For Ex. If you wish to split Syslog table and ingest a portion of the table into Auxiliary tier, then create a DCR-based custom table with same schema as of the Syslog table. At this point, switching table plan via UI is not possible, so I’d recommend using PowerShell script to create the DCR-based custom table. Once DCR-based custom table is created, implement DCR transformation to split the table. Configure total retention period of the Auxiliary table (this configuration will be done while creating the table) Let’s get started Use Case: In this demo, I’ll split Syslog table and route “Informational” logs to the Auxiliary table. Creating a DCR-based custom table: Previously a complex task, creating custom tables is now easy, thanks to a PowerShell script by MarkoLauren. Simply input the name of an existing table, and the script creates a DCR-based custom table with the same schema. Let’s see it in action now: Download the script locally. Update the resourceID details in this script and save it. Upload the updated script in Azure Shell. Load the file and enter the table name from which you wish to copy the schema. In my case, it's going to be "Syslog" table. Enter new table name, table type and total retention period, shown below: **Note** We highly recommend you review the PowerShell script thoroughly and do proper testing before executing it in production. We don't take any responsibility for the script. As you can see, Aux_Syslog_CL table has been created. Let’s validate in log analytics workspace > table section. Since the Auxiliary table has been created now, next step is to implement transformation logic at data collection rule level. The next step is to update the Data Collection Rule template to split the logs Since we already created custom table, we should create a transformation logic to split the Syslog table and route the logs with SeverityLevel “info” to the Auxiliary table. Let’s see how it works: Browse to Data Collection Rule blade. Open the DCR for Syslog table, click on Export template > Deploy > Edit Template as shown below: In the dataFlows section, I’ve created 2 streams for splitting the logs. Details about the streams as follows: 1 st Stream: It’s going to drop the Syslog messages where SeverityLevel is “info” and send rest of the logs to Syslog table. 2 nd Stream: It’s going to capture all Syslog messages where SeverityLevel is “info” and send the logs to Aux_Syslog_CL table. Save and deploy the updated template. Let’s see if it works as expected Browse to Azure > Microsoft Sentinel > Logs; and query the Auxiliary table to confirm if data is being ingested into this table. As we can see, the logs where SeverityLevel is “info” is being ingested in the Aux_Syslog_CL table and rest of the logs are flowing into Syslog table. Some nice cost savings are coming your way, hope this helps!From Healthy to Unhealthy: Alerting on Defender for Cloud Recommendations with Logic Apps
In today's cloud-first environments, maintaining strong security posture requires not just visibility but real-time awareness of changes. This blog walks you through a practical solution to monitor and alert on Microsoft Defender for Cloud recommendations that transition from Healthy to Unhealthy status. By combining the power of Kusto Query Language (KQL) with the automation capabilities of Azure Logic Apps, you’ll learn how to: Query historical and current security recommendation states using KQL Detect resources that have degraded in compliance over the past 14 days Send automatic email alerts when issues are detected Customize the email content with HTML tables for easy readability Handle edge cases, like sending a “no issues found” email when nothing changes Whether you're a security engineer, cloud architect, or DevOps practitioner, this solution helps you close the gap between detection and response and ensure that no security regressions go unnoticed. Prerequisites Before implementing the monitoring and alerting solution described in this blog, ensure the following prerequisites are met: Microsoft Defender for Cloud is Enabled Defender for Cloud must be enabled on the target Azure subscriptions/management group. It should be actively monitoring your resources (VMs, SQL, App Services, etc.). Make sure the recommendations are getting generated. Continuous Export is Enabled for Security Recommendations Continuous export should be configured to send security recommendations to a Log Analytics workspace. This enables you to query historical recommendation state using KQL. You can configure continuous export by going to: Defender for Cloud → Environment settings → Select Subscription → Continuous Export Then enable export for Security Recommendations to your chosen Log Analytics workspace. Detailed guidance on setting up continuous export can be found here: Set up continuous export in the Azure portal - Microsoft Defender for Cloud | Microsoft Learn High-Level Summary of the Automation Flow This solution provides a fully automated way to track and alert on security posture regressions in Microsoft Defender for Cloud. By integrating KQL queries with Azure Logic Apps, you can stay informed whenever a resource's security recommendation changes from Healthy to Unhealthy. Here's how the flow works: Microsoft Defender for Cloud evaluates Azure resources and generates security recommendations based on best practices and potential vulnerabilities. These recommendations are continuously exported to a Log Analytics workspace, enabling historical analysis over time. A scheduled Logic App runs a KQL query that compares: Recommendations from ~14 days ago (baseline), With those from the last 7 days (current state). If any resources are found to have shifted from Healthy to Unhealthy, the Logic App: Formats the data into an HTML table, and Sends an email alert with the affected resource details and recommendation metadata. If no such changes are found, an optional email can be sent stating that all monitored resources remain compliant — providing peace of mind and audit trail coverage. This approach enables teams to proactively monitor security drift, reduce manual oversight, and ensure timely remediation of emerging security issues. Logic Apps Flow This Logic App is scheduled to trigger daily. It runs a KQL query against a Log Analytics workspace to identify resources that have changed from Healthy to Unhealthy status over the past two weeks. If such changes are detected, the results are formatted into an HTML table and emailed to the security team for review and action. KQL Query used here: // Get resources that are currently unhealthy within the last 7 days let now_unhealthy = SecurityRecommendation | where TimeGenerated > ago(7d) | where RecommendationState == "Unhealthy" // For each resource and recommendation, get the latest record | summarize arg_max(TimeGenerated, *) by AssessedResourceId, RecommendationDisplayName; // Get resources that were healthy approximately 14 days ago (between 12 and 14 days ago) let past_healthy = SecurityRecommendation | where TimeGenerated between (ago(14d) .. ago(12d)) | where RecommendationState == "Healthy" // For each resource and recommendation, get the latest record in that time window | summarize arg_max(TimeGenerated, *) by AssessedResourceId, RecommendationDisplayName; // Join current unhealthy resources with their healthy state 14 days ago now_unhealthy | join kind=inner past_healthy on AssessedResourceId, RecommendationDisplayName | project AssessedResourceId, // Unique ID of the assessed resource RecommendationDisplayName, // Name of the security recommendation RecommendationSeverity, // Severity level of the recommendation Description, // Description explaining the recommendation State_14DaysAgo = RecommendationState1,// Resource state about 14 days ago (should be "Healthy") State_Recent = RecommendationState, // Current resource state (should be "Unhealthy") Timestamp_14DaysAgo = TimeGenerated1, // Timestamp from ~14 days ago Timestamp_Recent = TimeGenerated // Most recent timestamp Once this logic app executes successfully, you’ll get an email as per your configuration. This email includes: A brief introduction explaining the situation. The number of affected recommendations. A formatted HTML table with detailed information: AssessedResourceId: The full Azure resource ID. RecommendationDisplayName: What Defender recommends (e.g., “Enable MFA”). Severity: Low, Medium, High. Description: What the recommendation means and why it matters. State_14DaysAgo: The previous (Healthy) state. State_Recent: The current (Unhealthy) state. Timestamps: When the states were recorded. Sample Email for reference: What the Security Team Can Do with It? Review the Impact Quickly identify which resources have degraded in security posture. Assess if the changes are critical (e.g., exposed VMs, missing patching). Prioritize Remediation Use the severity level to triage what needs immediate attention. Assign tasks to the right teams — infrastructure, app owners, etc. Correlate with Other Alerts Cross-check with Microsoft Sentinel, vulnerability scanners, or SIEM rules. Investigate whether these changes are expected, neglected, or malicious. Track and Document Use the email as a record of change in security posture. Log it in ticketing systems (like Jira or ServiceNow) manually or via integration. Optional Step: Initiate Remediation Playbooks Based on the resource type and issue, teams may: Enable security agents, Update configurations, Apply missing patches, Isolate the resource (if necessary). Automating alerts for resources that go from Healthy to Unhealthy in Defender for Cloud makes life a lot easier for security teams. It helps you catch issues early, act faster, and keep your cloud environment safe without constantly watching dashboards. Give this Logic App a try and see how much smoother your security monitoring and response can be! Access the JSON deployment file for this Logic App here: Microsoft-Unified-Security-Operations-Platform/Microsoft Defender for Cloud/ResourcesMovingFromHealthytoUnhealthyState/ARMTemplate-HealthytoUnhealthyResources(MDC).json at main · Abhishek-Sharan/Microsoft-Unified-Security-Operations-Platform
