Introducing the enhanced query diagnostics in Azure Log Analytics
We’re excited to introduce a set of new capabilities that simplify query diagnostics and troubleshooting in Azure Log Analytics. With this new experience, users can now easily resolve query issues that may surface using the suggested recommendations, have a deeper visibility into query execution statistics and utilize those to optimize query performances. What’s new? Improved query notification experience Concise details providing a clear understanding of query status, errors, and root cause Actionable recommendations and suggestions to quickly resolve query issues Contextual linkage to the Query Details pane for advanced diagnostics The revamped 'Query Details' pane After running a query, you can navigate to the revamp Query details to explore multiple query aspects over the new tabs: Overview – A curated set of key performance indicators, including: Execution time breakdown: Engine, Log Analytics Service, and Queue times Metrics such as Total CPU, Memory peak, Response size and more Raw statistics – Complete execution details, referenced workspaces, and technical metadata Errors – Raw error payloads for precise and advanced debugging With these enhancements, query troubleshooting is made easier - All within the familiar Azure Log Analytics experience over the Azure Portal. Learn more: Optimize log queries in Azure Monitor - Azure Monitor | Microsoft LearnGeneral Availability of Auxiliary Logs and Reduced Pricing
Azure Monitor logs are trusted by hundreds of thousands of organizations to monitor mission-critical workloads. But with such a diverse customer base, there’s no one-size-fits-all solution. That’s why we’re excited to announce a series of major advancements to Auxiliary Logs, the Azure Monitor plan that is designed for high volume logs. Auxiliary Logs works in tandem with all other Azure Monitor tools including the more powerful Basic and Analytics Logs plans. Together, they are the one-stop-shop for all the logging needs of an organization. Auxiliary Logs were introduced last year and have gained a lot of traction since. There are many customers that ingest data into Auxiliary Logs, with several teams ingesting more than a petabyte of logs per day. Over the last few months, we have moved Auxiliary Logs to General Availability status, made them available in all regions, and made numerous enhancements to the service. Auxiliary Logs were first introduced with support for Custom Logs only; security data was added shortly afterwards and is now also supported. Additional table support will be available soon. Learn more about table plans here. We’re also announcing a significant price reduction for Auxiliary Logs, making them even more cost-effective and accessible for high-volume scenarios. For detailed pricing information and charges, visit the Azure Monitor pricing page. This is part of a broader strategy as we align and evolve our data lake assets and pricing models, with the goal of enabling customers to benefit from modern data lake technology including batch computing and federated access without duplication for multiple use-cases spanning security and observability on a common technology stack. The Sentinel data lake announced recently is a key part of this evolution. Data ingested into the Sentinel data lake can be accessed in Auxiliary logs without copy and vice-versa. Stay tuned for more information later this year on how we’re evolving our data strategy for operational scenarios. Enhanced Query Capabilities We have worked to make queries on Auxiliary Logs faster and more powerful. That includes: Expanded KQL Support: All KQL operators on a single table are now supported, including the lookup operator to Analytics tables. Performance Boosts: Built on Delta Parquet, Auxiliary Logs now benefit from improved encoding and partitioning to make queries much more efficient, though indexed technologies like Basic Logs and Analytics Logs will perform better. Extended Time Range: Queries are no longer limited to the last 30 days - you can now query across any time period. Cost Estimation Preview: Get a cost estimate before running your query. General Availability of Summary Rules We are also announcing the General Availability of summary rules. Summary rules have quickly become a key resource for optimizing data ingestion and analysis, having been adopted by a significant number of customers during the preview period. Summary rules enable users to efficiently summarize high-ingestion-rate streams across Analytics, Basic, or Auxiliary plans, supporting robust analysis, dashboarding, and long-term reporting via summarized Analytics tables. Unlike conventional ETL processes, raw data remains in its original tables, allowing for detailed investigations as needed. Key enhancements include: Increased rule limits per workspace Enabling users to retry bins affected by incidents Expanded regional availability Customers can now utilize summary rules on a greater scale with increased confidence. Learn more about summary rules here. Search Jobs: More Power, More Flexibility Search jobs allow users to scan vast amounts of data asynchronously and ingest the results into Analytics table for further investigation Based on customer feedback, we’ve made the following improvements: Enabling more results to be loaded, up to 100 million records (coming soon) Improved user interface that streamlines the search job execution Providing a cost prediction before running a search job Increasing concurrently and removing additional limits. Added support for all KQL operators on a single table with the lookup operator to Analytics tables (coming soon). Learn more about search jobs here. Public Preview of KQL Transformations for Auxiliary Logs Last, but not least, we’re excited to announce the public preview of KQL-based transformations for Auxiliary Logs. This milestone brings Auxiliary Logs to feature parity in terms of ingestion-time transformations with other Azure Monitor log tiers, eliminating the previous limitation of ingesting only raw custom logs into Auxiliary storage. With this new capability, you can now apply filtering and transformation logic at ingestion time, enabling a more strategic and cost-effective approach to managing high-volume, low-fidelity logs. By using Data Collection Rules (DCRs) with Kusto Query Language (KQL) expressions, you can: Filter out noise to reduce data volume. Parse and shape fields to prepare logs for efficient downstream consumption. Split data across multiple tables or tiers, for cost-performance optimization. What makes this especially powerful is that transformations apply to both custom and standard log streams directed to custom tables in the Auxiliary tier. For example, you can now route a portion, or the entirety, of specific platform logs to a custom table in Auxiliary storage, applying transformations as needed. Applying custom transformations and filtering data ingested into the Auxiliary tier will incur a log processing charge. For detailed pricing information and charges, please refer to the Azure Monitor pricing page. Learn more about ingestion-time transformations here.
Announcing resource-scope query for Azure Monitor Workspaces
We’re excited to announce the public preview of resource-scope query for Azure Monitor Workspaces (AMWs)—a major step forward in simplifying observability, improving access control, and aligning with Azure-native experiences. This new capability builds on the successful implementation of resource-scope query in Log Analytics Workspaces (LAWs), which transformed how users access logs by aligning them with Azure resource scopes. We’re now bringing the same power and flexibility to metrics in AMWs. What is resource-scope query? Resource-scope query has been a frequently requested capability that allows users to query metrics scoped to a specific resource, resource group, or subscription—rather than needing to know which AMW the metrics are stored in. This means: Simpler querying: users can scope to the context of one or more resources directly, without knowledge of where metrics are stored. Granular Azure RBAC control: if the AMW is configured in resource-centric access mode, user permissions are checked against the resources they are querying for, rather than access to the workspace itself - just like how LAW works today. This supports security best practices for least privileged access requirements. Why use resource-centric query? Traditional AMW querying required users to: Know the exact AMW storing their metrics. Have access to the AMW. Navigate away from the resource context to query metrics. This created friction for DevOps teams and on-call engineers who do not necessarily know which AMW to query when responding to an alert. With resource-centric querying: Users can query metrics directly from the resource’s Metrics blade. Least privilege access is respected—users only need access to the resource(s) they are querying about. Central teams can maintain control of AMWs while empowering app teams to self-monitor. How does it work? All metrics ingested via Azure Monitor Agent are automatically stamped with dimensions like Microsoft.resourceid, Microsoft.subscriptionid, and Microsoft.resourcegroupname to enable this experience. The addition of these dimensions does not have any cost implications to end users. Resource-centric queries use a new endpoint: https://query.<region>.prometheus.monitor.azure.com We will re-route queries as needed from any region, but we recommend choosing the one nearest to your AMWs for the best performance. Users can query via: Azure Portal PromQL Editor Grafana dashboards (with data source configuration) Query-based metric alerts Azure Monitor solutions like Container Insights and App Insights (when using OTel metrics with AMW as data source) Prometheus HTTP APIs When querying programmatically, users pass an HTTP header: x-ms-azure-scoping: <ARM Resource ID> Scoping supports a single: Individual resource Resource group Subscription At this time, scoping is only support at a single-resource level, but comma-separated multi-resource scoping will be added by the end of 2025. Who Can Benefit? Application Teams: Query metrics for their own resources without needing AMW access. Central Monitoring Teams: Maintain control of AMWs while enabling secure, scoped access for app teams. DevOps Engineers: Respond to alerts and troubleshoot specific resources without needing to locate the AMW(s) storing the metrics they need. Grafana Users: Configure dashboards scoped to subscriptions or resource groups with dynamic variables without needing to identify the AMW(s) storing their metrics. When Is This Available? Microsoft. dimension stamping* is already complete and ongoing for all AMWs. Public Preview of the resource-centric query endpoint begins October 10th, 2025. Starting on that date, all newly created AMWs will default to resource-context access mode. What is the AMW “access control mode”? The access control mode is a setting on each workspace that defines how permissions are determined for the workspace. Require workspace permissions. This control mode does NOT allow granular resource-level Azure RBAC. To access the workspace, the user must be granted permissions to the workspace. When a user scopes their query to a workspace, workspace permissions apply. When a user scopes their query to a resource, both workspace permissions AND resource permissions are verified. This setting is the default for all workspaces created before October 2025. Use resource or workspace permissions. This control mode allows granular Azure RBAC. Users can be granted access to only data associated with resources they can view by assigning Azure read permission. When a user scopes their query to a workspace, workspace permissions apply. When a user scopes their query to a resource, only resource permissions are verified, and workspace permissions are ignored. This setting is the default for all workspaces created after October 2025. Read about how to change the control mode for your workspaces here. Final Thoughts Resource-centric query brings AMWs in line with Azure-native experiences, enabling secure, scalable, and intuitive observability. Whether you’re managing thousands of VMs, deploying AKS clusters, or building custom apps with OpenTelemetry, this feature empowers you to monitor in the context of your workloads or resources rather than needing to first query the AMW(s) and then filter down on what you’re looking for. To get started, simply navigate to your resource’s Metrics blade after October 10 th , 2025 or configure your Grafana data source to use the new query endpoint.
Optimizing Microsoft Sentinel: Resolving AMA-Induced Syslog & CEF Duplicates
2) Recommended Solutions When collecting both Syslog and CEF logs from the same Linux collector using the Azure Monitor Agent (AMA) in Microsoft Sentinel, duplicate log entries can occur. These duplicates arise because the same event may be ingested through both the Syslog and CEF pipelines, leading to redundancy in the Log Analytics Workspace (LAW). The following solutions aim to eliminate or reduce duplicate log ingestion, ensuring that: CEF events are parsed correctly and only once. Syslog data remains clean and non-redundant. Storage and analytics efficiency is improved. Alerting and incident investigation are not skewed by duplicate entries. Each option provides a different strategy based on your environment’s flexibility and configuration capabilities—from facility-level separation, to ingestion-time filtering, to daemon-side log routing. Option 1: Facility Separation (Preferred) Configure devices to emit CEF logs on a dedicated facility (for example, 'local4'), and adjust the Data Collection Rules (DCRs) so that the CEF stream includes only that facility, while the Syslog stream excludes it. This ensures CEF events are parsed once into 'CommonSecurityLog' and never land in 'Syslog'. CEF via AMA DCR (include only CEF facility): { "properties": { "dataSources": { "syslog": [ { "streams": ["Microsoft-CommonSecurityLog"], "facilityNames": ["local4"], "logLevels": ["*"], "name": "cefDataSource" } ] }, "dataFlows": [ { "streams": ["Microsoft-CommonSecurityLog"], "destinations": ["laDest"] } ] } } Syslog via AMA DCR (exclude CEF facility): { "properties": { "dataSources": { "syslog": [ { "streams": ["Microsoft-Syslog"], "facilityNames": [ "auth","authpriv","cron","daemon","kern","mail", "syslog","user","local0","local1","local2","local3", "local5","local6","local7" ], "logLevels": ["*"], "name": "syslogDataSource" } ] }, "dataFlows": [ { "streams": ["Microsoft-Syslog"], "destinations": ["laDest"] } ] } } Option 2: Ingest-time Transform (Drop CEF from Syslog) If facility separation is not feasible, apply a transformation to the Syslog stream in the DCR so that any CEF-formatted messages are dropped during ingestion. Syslog stream transformKql: { "properties": { "dataFlows": [ { "streams": ["Microsoft-Syslog"], "transformKql": "source | where not(SyslogMessage startswith 'CEF:')", "destinations": ["laDest"] } ] } } Option 3: Daemon-side Filtering/Rewriting (rsyslog/syslog-ng) Filter or rewrite CEF messages before AMA sees them. For example, route CEF messages to a dedicated facility using syslog-ng and stop further processing: # Match CEF filter f_cef { message("^CEF:"); }; # Send CEF to local5 and stop further processing log { source(s_src); filter(f_cef); rewrite { set_facility(local5); }; destination(d_azure_mdsd); flags(final); } 3) Verification Steps with KQL Queries Detect CEF messages that leaked into Syslog: Syslog | where TimeGenerated > ago(1d) | where SyslogMessage startswith "CEF:" | summarize count() by Computer | order by count_ desc Estimate duplicate count across Syslog and CommonSecurityLog: let sys = Syslog | where TimeGenerated > ago(1d) | where SyslogMessage startswith "CEF:" | extend key = hash_sha256(SyslogMessage); let cef = CommonSecurityLog | where TimeGenerated > ago(1d) | extend key = hash_sha256(RawEvent); cef | join kind=innerunique (sys) on key | summarize duplicates = count() Note : You should identify the RawEvent that might be causing the duplicates. 3.1) Duplicate Detection Query Explained This query helps quantify duplicate ingestion when both Syslog and CEF connectors ingest the same events. It works as follows: Build the Syslog set (sys): Filter the 'Syslog' table for the last day and keep only messages that start with 'CEF:'. Compute a SHA-256 hash of the entire message as a stable join key ("key"). Build the CEF set (cef): Filter the 'CommonSecurityLog' table for the last day and compute a SHA-256 hash of the 'RawEvent' field as the same-style join key. Join on the key: Use 'join kind=innerunique' to find messages that exist in both sets (i.e., duplicates). Summarize: Count the number of matching rows to get a duplicate total. 4) Common Pitfalls - Overlapping DCRs applied to the same collector VM causing overlapping facilities/severities. - CEF and Syslog using the same facility on sources, leading to ingestion on both streams. - rsyslog/syslog-ng filters placed after AMA’s own configuration include (ensure your custom rules run before '10-azuremonitoragent.conf'). 5) References - Microsoft Learn: Ingest syslog and CEF messages to Microsoft Sentinel with AMA (https://learn.microsoft.com/en-us/azure/sentinel/connect-cef-syslog-ama)
Azure Monitor managed service for Prometheus now includes native Grafana dashboards
We are excited to announce that Azure Monitor managed service for Prometheus now includes native Grafana dashboards within the Azure portal at no additional cost. This integration marks a major milestone in our mission to simplify observability reducing the administrative overhead and complexity compared to deploying and maintaining your own Grafana instances. The use of open-source observability tools continues to grow for cloud-native scenarios such as application and infrastructure monitoring using Prometheus metrics and OpenTelemetry logs and traces. For these scenarios, DevOps and SRE teams need streamlined and cost-effective access to industry-standard tooling like Prometheus metrics and Grafana dashboards within their cloud-hosted environments. For many teams, this usually means deploying and managing separate monitoring stacks with some versions self-hosted or partner-managed Prometheus and Grafana. However, Azure Monitor's latest integrations with Grafana provides this capability out-of-the-box by enabling you to view Prometheus metrics and Azure other observability data in Grafana dashboards fully integrated into the Azure portal. Azure Monitor dashboards with Grafana delivers powerful visualization and data transformation capabilities on Prometheus metrics, Azure resource metrics, logs, and traces stored in Azure Monitor. Pre-built dashboards are included for several key scenarios like Azure Kubernetes Service, Azure Container Apps, Container Insights, and Application Insights. Why Grafana in Azure portal? Grafana dashboards are widely adopted visualization tool used with Prometheus metrics and cloud-native observability tools. Embedding it natively in Azure Portal offers: Unified Azure experience: No additional RBAC or network configuration required, users Azure login credentials and Azure RBAC are used to access dashboards and data. View Grafana dashboards alongside all your other Azure resources and Azure Monitor views in the same portal. No management overhead or compute costs: Dashboards with Grafana use a fully SaaS model built into Azure Monitor, where you do not have to administer the Grafana server or the compute on which it runs. Access to community dashboards: Open-source and Grafana community dashboards using Prometheus or Azure Monitor data sources can be imported with no modifications. These capabilities mean faster troubleshooting, deeper insights, and a more consistent observability platform for Azure-centric workloads. Figure 1: Dashboards with Grafana landing page in the context of Azure Monitor Workspace in the Azure portal Getting Started To get started, enable Managed Prometheus for your AKS cluster and then navigate to the Azure Monitor workspace or AKS cluster in the Azure portal and select Monitoring > Dashboards with Grafana (preview). From this page you can view, edit, create and import Grafana dashboards. Simply click on one of the pre-built dashboards to get started. You may use these dashboards as they have been provided or edit and add panels, update visualizations and create variables to create your own custom dashboards. With this approach, no Grafana servers or additional Azure resources need to be provisioned or maintained. Teams can quickly leverage and customize Grafana dashboards within the Azure portal, reducing their deployment and management time while still gaining the benefits of dashboards and visualizations to improve monitoring and troubleshooting times. Figure 2: Kubernetes Compute Resources dashboard being viewed in the context of Azure Monitor Workspace in the Azure portal When to upgrade to Azure Managed Grafana? Dashboards with Grafana in the Azure portal cover most common Prometheus scenarios but, Azure Managed Grafana remains the right choice for several advanced use cases, including: Extended data source support for non-Azure data sources e.g. open-source and third-party data stores Private networking and advanced authentication options Multi-cloud, hybrid and on-premises data source connectivity. See When to use Azure Managed Grafana for more details. Get started with Azure Monitor dashboards with Grafana today.Generally Available - High scale mode in Azure Monitor - Container Insights
Container Insights is Azure Monitor’s solution for collecting logs from your Azure Kubernetes Service (AKS) clusters. As the adoption of AKS continues to grow, we are seeing an increasing number of customers with log scaling needs that hit the limits of log collection in Container Insights. Last August, we announced the public preview of High Scale mode in Container Insights to help customers achieve a higher log collection throughput from their AKS clusters. Today, we are happy to announce the General Availability of High Scale mode. High scale mode is ideal for customers approaching or above 10,000 logs/sec from a single node. When High Scale mode is enabled, Container Insights does multiple configuration changes leading to a higher overall throughput. These include using a more powerful agent setup, using a different data pipeline, allocating more memory for the agent, and more. All these changes are made in the background by the service and do not require input or configuration from customers. High Scale mode impacts only the data collection layer (with a new DCR) – the rest of the experience remains the same. Data flows to our existing tables, your queries and alerts work as before too. High Scale mode is available to all customers. Today, High scale is turned off by default. In the future, we plan to enable High Scale mode by default for all customers to reduce the chances of log loss when workloads scale. To get started with High Scale mode, please see our documentation at https://aka.ms/cihsmodeAutomate Your Log Analytics Workflows with AI and Logic Apps
In this post, we’ll demonstrate how to build a simple yet powerful workflow using Azure Logic Apps, Log Analytics queries, and LLMs to automate log analysis, save time, and spot issues faster. While we focus here on an example using Application Insights data with Azure OpenAI, the same approach can be applied to any Log Analytics source - whether raw logs, security events, or custom logs. By customizing your queries and AI prompts to match your data and the model’s capabilities, you can easily adapt this workflow to meet your specific needs. Note: This blog post offers guidance for automating workflows with Log Analytics data and LLMs using existing Azure Monitor products. It’s intended as a flexible approach based on user needs and preferences, providing an additional option alongside other Microsoft experiences, such as Azure Monitor issues and investigations (preview). Application Insights as a Use Case Imagine you’re an Application Insights user relying on the AppTraces table - detailed logs of events, errors, and critical traces. You need to spot hour-over-hour spikes or drops, identify operations causing the most issues, and detect recurring patterns or keywords that reveal deeper problems. These insights help turn raw data into actionable information. Running queries and analyzing logs regularly is essential, and automation offers a way to make this process more efficient. This saves time and helps you focus on the most impactful insights - so you can quickly move on to what matters next. With Azure Logic Apps, you can create a recurring workflow that automatically runs your Log Analytics queries, sends the summarized results to Azure OpenAI for analysis, and delivers a clear, actionable report straight to your inbox on your preferred schedule. From Logs to Insights: Step-by-Step AI Workflow 1. Create a Logic App Go to the Azure Portal and create a new Logic App. Open the Logic App Designer to start building your workflow. Helpful resource: Overview - Azure Logic Apps | Microsoft Learn 2. Set a Trigger Add a trigger to start your flow - for this scenario, we recommend using the Recurrence trigger to schedule it on a weekly basis (or any frequency you prefer). Of course, you can choose other triggers depending on your specific needs. 3. Query Your Log Analytics Data Add the Azure Monitor Logs - “Run query and list results” connector to your Logic App. Connect it to your Log Analytics workspace (or another relevant resource). Write a Kusto Query Language (KQL) query to pull data from Log Analytics Tables. In our example, the query retrieves aggregated error-level (SeverityLevel = 3) and critical-level (SeverityLevel = 4) traces from the last week, grouped by hour and operation name, with three sample messages for context. This not only shows the number of errors, when they occurred, and which operations were most affected, but also gives the LLM in the next step a solid foundation for uncovering deeper insights and trends. The query: AppTraces | where TimeGenerated > startofday(ago(7d)) | where SeverityLevel in (3, 4) // Error = 3, Critical = 4 | summarize TracesCount = count(), SampleMessages = make_list(Message, 3) by bin(TimeGenerated, 1h), SeverityLevel, OperationName | order by TimeGenerated asc Tip: Log datasets can be huge - use the summarize operator to aggregate results and reduce the volume for the AI model. Helpful resource: Connect to Log Analytics or Application Insights - Azure Logic Apps | Microsoft Learn 4. Prerequisite - Azure OpenAI Resource Configuration Make sure you have an Azure OpenAI resource set up and an AI model (e.g., GPT-4) deployed before continuing with your workflow. Helpful resource: What is Azure OpenAI in Azure AI Foundry Models? | Microsoft Learn 5. Analyze and Summarize with Azure OpenAI In Logic Apps, add an HTTP action and set all the parameters to call the OpenAI API endpoint. Pass the query results from the previous step (step 3) as input and instruct the OpenAI model to: Summarize key findings - for example, the total number of errors and critical events, and identify the top operations generating the most issues. Highlight anomalies or trends - such as trends and spikes in errors over time (hour-by-hour), and detection of recurring error patterns or keywords. Provide recommendations prioritized by urgency to guide the next steps. Format the output in HTML for easy email rendering. Tip: The body structure sent to the AI includes both System and User rules, formatted together as one string (see below). Helpful resource: How to use Assistants with Logic apps - Azure OpenAI | Microsoft Learn Here’s the prompt example: "messages": [ { "role": "system", "content": "You are an Azure AI tool that creates a weekly report based solely on this prompt and input JSON data from Log Analytics. The input is a list of records, each with these fields: TimeGenerated (ISO 8601 timestamp string), SeverityLevel (integer, where 3=Error, 4=Critical), OperationName (string), TracesCount (integer), SampleMessages (JSON string representing a list of up to 3 messages). Your tasks: 1) Sum the TracesCount values accurately to provide total counts for the entire week and broken down by day and SeverityLevel. 2) Present TracesCount counts per OperationName, grouped by hour and day with severity-level breakdowns. 3) Identify and list the top 10 OperationNames by combined Error and Critical TracesCount for the week, including up to 3 unique sample messages per OperationName, removing duplicates. 4) Compare TracesCount hour-by-hour and day-by-day, calculating percentage changes and highlighting spikes (>100% increase) and significant drops. 5) Detect any new OperationNames appearing during the week that did not appear before. 6) Highlight recurring Errors and Critical issues based on keywords: timeout, exception, outofmemory, connection refused. 7) Assign urgency levels based on frequency, impact, and trends. 8) Provide clear, prioritized recommendations for resolving the main issues. Format your output as valid inline-styled HTML using only these tags: <h2>, <h3>, <p>, <ul>, <li>, and <hr>. Include these report sections in this order: Executive Summary, Weekly Totals and Daily Breakdown, Hourly and Daily Trend Comparison, New & Emerging OperationNames, Detailed Operation Errors, Data Quality & Confidence, Recommendations. Include an opening title with the report’s time period." }, { "role": "user", "content": "string(outputs('Run_query_and_list_results'))" } ] } 6. Send the Report via Email Use the Send an email (V2) connector, or another endpoint connector, such as Teams. Send the AI-generated report to your team, stakeholders, or yourself. Customize the email subject, body, and importance level as needed. Section of the final email report: Important reminder: Once your flow is ready, enable it in Logic Apps to ensure it starts running according to the schedule. Key Takeaways By combining Azure Logic Apps, Log Analytics, and Azure OpenAI, you can turn raw, complex logs into clear, actionable insights - automatically. This workflow helps reduce manual analysis time and enables faster responses to critical issues. Ready to try? Build your own automated log insights workflow today and empower your team with AI-driven clarity.Optimize Azure Log Costs: Split Tables and Use the Auxiliary Tier with DCR
This blog is continuation of my previous blog where I discussed about saving ingestion costs by splitting logs into multiple tables and opting for the basic tier! Now that the transformation feature for Auxiliary logs has entered Public Preview stage, I’ll take a deeper dive, showing how to implement transformations to split logs across tables and route some of them to the Auxiliary tier. A quick refresher: Azure Monitor offers several log plans which our customers can opt for depending on their use cases. These log plans include: Analytics Logs – This plan is designed for frequent, concurrent access and supports interactive usage by multiple users. This plan drives the features in Azure Monitor Insights and powers Microsoft Sentinel. It is designed to manage critical and frequently accessed logs optimized for dashboards, alerts, and business advanced queries. Basic Logs – Improved to support even richer troubleshooting and incident response with fast queries while saving costs. Now available with a longer retention period and the addition of KQL operators to aggregate and lookup. Auxiliary Logs – Our new, inexpensive log plan that enables ingestion and management of verbose logs needed for auditing and compliance scenarios. These may be queried with KQL on an infrequent basis and used to generate summaries. Following diagram provides detailed information about the log plans and their use cases: More details about Azure Monitor Logs can be found here: Azure Monitor Logs - Azure Monitor | Microsoft Learn **Note** This blog will be focussed on switching to Auxiliary logs only. I would recommend going through our public documentation for detailed insights about feature-wise comparison for the log plans which should help you in taking right decisions for choosing the correct log plans. At this stage, I assume you’re aware about different log tiers that Azure Monitor offers and you’ve decided to switch to Auxiliary logs for high volume, low-fidelity logs. Let’s look at the high-level approach we’re going to follow to achieve this: Review the relevant tables and figure out which portion of the log can be moved to Auxiliary tier Create a DCR-based custom table which same schema as of the original table. For Ex. If you wish to split Syslog table and ingest a portion of the table into Auxiliary tier, then create a DCR-based custom table with same schema as of the Syslog table. At this point, switching table plan via UI is not possible, so I’d recommend using PowerShell script to create the DCR-based custom table. Once DCR-based custom table is created, implement DCR transformation to split the table. Configure total retention period of the Auxiliary table (this configuration will be done while creating the table) Let’s get started Use Case: In this demo, I’ll split Syslog table and route “Informational” logs to the Auxiliary table. Creating a DCR-based custom table: Previously a complex task, creating custom tables is now easy, thanks to a PowerShell script by MarkoLauren. Simply input the name of an existing table, and the script creates a DCR-based custom table with the same schema. Let’s see it in action now: Download the script locally. Update the resourceID details in this script and save it. Upload the updated script in Azure Shell. Load the file and enter the table name from which you wish to copy the schema. In my case, it's going to be "Syslog" table. Enter new table name, table type and total retention period, shown below: **Note** We highly recommend you review the PowerShell script thoroughly and do proper testing before executing it in production. We don't take any responsibility for the script. As you can see, Aux_Syslog_CL table has been created. Let’s validate in log analytics workspace > table section. Since the Auxiliary table has been created now, next step is to implement transformation logic at data collection rule level. The next step is to update the Data Collection Rule template to split the logs Since we already created custom table, we should create a transformation logic to split the Syslog table and route the logs with SeverityLevel “info” to the Auxiliary table. Let’s see how it works: Browse to Data Collection Rule blade. Open the DCR for Syslog table, click on Export template > Deploy > Edit Template as shown below: In the dataFlows section, I’ve created 2 streams for splitting the logs. Details about the streams as follows: 1 st Stream: It’s going to drop the Syslog messages where SeverityLevel is “info” and send rest of the logs to Syslog table. 2 nd Stream: It’s going to capture all Syslog messages where SeverityLevel is “info” and send the logs to Aux_Syslog_CL table. Save and deploy the updated template. Let’s see if it works as expected Browse to Azure > Microsoft Sentinel > Logs; and query the Auxiliary table to confirm if data is being ingested into this table. As we can see, the logs where SeverityLevel is “info” is being ingested in the Aux_Syslog_CL table and rest of the logs are flowing into Syslog table. Some nice cost savings are coming your way, hope this helps!General Availability of Azure Monitor Network Security Perimeter Features
We’re excited to announce that Azure Monitor Network Security Perimeter features are now generally available! This update is an important step forward for Azure Monitor’s security, providing comprehensive network isolation for your monitoring data. In this post, we’ll explain what Network Security Perimeter is, why it matters, and how it benefits Azure Monitor users. Network Security Perimeter is purpose-built to strengthen network security and monitoring, enabling customers to establish a more secure and isolated environment. As enterprise interest grows, it’s clear that this feature will play a key role in elevating the protection of Azure PaaS resources against evolving security threats. What is Network Security Perimeter and Why Does It Matter? Network Security Perimeter is a network isolation feature for Azure PaaS services that creates a trusted boundary around your resources. Azure Monitor’s key components (like Log Analytics workspaces and Application Insights) run outside of customer virtual networks; Network security perimeter allows these services to communicate only within an explicit perimeter and blocks any unauthorized public access. In essence, the security perimeter acts as a virtual firewall at the Azure service level – by default it restricts public network access to resources inside the perimeter, and only permits traffic that meets your defined rules. This prevents unwanted network connections and helps prevent data exfiltration (sensitive monitoring data stays within your control). For Azure Monitor customers, Network Security Perimeter is a game-changer. It addresses a common ask from enterprises for “zero trust” network security on Azure’s monitoring platform. Previously, while you could use Private Link to secure traffic from your VNets to Azure Monitor, Azure Monitor’s own service endpoints were still accessible over the public internet. The security perimeter closes that gap by enforcing network controls on Azure’s side. This means you can lock down your Log Analytics workspace or Application Insights to only accept data from specific sources (e.g. certain IP ranges, or other resources in your perimeter) and only send data out to authorized destinations. If anything or anyone outside those rules attempts to access your monitoring resources, Network Security Perimeter will deny it and log the attempt for auditing. In short, Network Security Perimeter brings a new level of security to Azure Monitor: it allows organizations to create a logical network boundary around their monitoring resources, much like a private enclave. This is crucial for customers in regulated industries (finance, government, healthcare) who need to ensure their cloud services adhere to strict network isolation policies. By using the security perimeter, Azure Monitor can be safely deployed in environments that demand no public exposure and thorough auditing of network access. It’s an important step in strengthening Azure Monitor’s security posture and aligning with enterprise zero-trust networking principles. Key Benefits of Network Security Perimeter in Azure Monitor With Network Security Perimeter now generally available, Azure Monitor users gain several powerful capabilities: 🔒 Enhanced Security & Data Protection: Azure PaaS resources in a perimeter can communicate freely with each other, but external access is blocked by default. You define explicit inbound/outbound rules for any allowed public traffic, ensuring no unauthorized network access to your Log Analytics workspaces, Application Insights components, or other perimeter resources. This greatly reduces the risk of data exfiltration and unauthorized access to monitoring data. ⚖️ Granular Access Control: Network Security Perimeter supports fine-grained rules to tailor access. You can allow inbound access by specific IP address ranges or Azure subscription IDs, and allow outbound calls to specific Fully Qualified Domain Names (FQDNs). For example, you might permit only your corporate IP range to send telemetry to a workspace, or allow a workspace to send data out only to contoso-api.azurewebsites.net. This level of control ensures that only trusted sources and destinations are used. 📜 Comprehensive Logging & Auditing: Every allowed or denied connection governed by Network Security Perimeter can be logged. Azure Monitor’s Network Security Perimeter integration provides unified access logs for all resources in the perimeter. These logs give you visibility into exactly what connections were attempted, from where, and whether they were permitted or blocked. This is invaluable for auditing and compliance – for instance, proving that no external IPs accessed your workspace, or detecting unexpected outbound calls. The logs can be sent to a Log Analytics workspace or storage for retention and analysis. 🔧 Seamless Integration with Azure Monitor Services: Network Security Perimeter is natively integrated across Azure Monitor’s services and workflows. Log Analytics workspaces and Application Insights components support Network Security Perimeter out-of-the-box, meaning ingestion, queries, and alerts all enforce perimeter rules behind the scenes. Azure Monitor Alerts (scheduled query rules) and Action Groups also work with Network Security Perimeter , so that alert notifications or automation actions respect the perimeter (for example, an alert sending to an Event Hub will check Network Security Perimeter rules). This end-to-end integration ensures that securing your monitoring environment with Network Security Perimeter doesn’t break any functionality – everything continues to work, but within your defined security boundary. 🤝 Consistent, Centralized Management: Network Security Perimeter introduces a uniform way to manage network access for multiple resources. You can group resources from different services (and even different subscriptions) into one perimeter and manage network rules in one place. This “single pane of glass” approach simplifies operations: network admins can define a perimeter once and apply it to all relevant Azure Monitor components (and other supported services). It’s a more scalable and consistent method than maintaining disparate firewall settings on each service. Network Security Perimeter uses Azure’s standard API and portal experience, so setting up a perimeter and rules is straightforward. 🌐 No-Compromise Isolation (with Private Link): Network Security Perimeter complements existing network security options. If you’re already using Azure Private Link to keep traffic off the internet, Network Security Perimeter adds another layer of protection. Private Link secures traffic between your VNet and Azure Monitor; Network Security Perimeter secures Azure Monitor’s service endpoints themselves. Used together, you achieve defense-in-depth: e.g., a workspace can be accessible only via private endpoint and only accept data from certain sources due to Network Security Perimeter . This layered approach helps meet even the most stringent security requirements. In conclusion, Network Security Perimeter for Azure Monitor provides strong network isolation, flexible control, and visibility – all integrated into the Azure platform. It helps organizations confidently use Azure Monitor in scenarios where they need to lock down network access and simplify compliance. For detailed information on configuring Azure Monitor with a Network Security Perimeter, please refer to the following link: Configure Azure Monitor with Network Security Perimeter.Introducing the Improved Search Job Experience in Azure Monitor Log Analytics
A search job is an asynchronous query that runs on any data in your Log Analytics workspace, including data from the long-term retention, making the results available for further queries in a new Analytics table within your workspace. To efficiently search massive datasets, Search Job divides queries into smaller time-based segments, processes them in parallel, and returns the results. This approach optimizes scalability and enables reliable analysis, even over petabytes of data. We’re excited to announce significant enhancements to Search Jobs, designed to make large-scale data exploration faster, easier, and more efficient. What’s New in Search Job Our latest update includes several powerful improvements: Intuitive and streamlined UI experience for faster and simpler setup. Cost estimation preview before running a Search Job. Previously, we had system limitations in place to ensure stability. Now, as more customers use Search Job, we’re removing most of these limits to enhance your experience: Result limits are being increased, with support for up to 100 million records coming soon. Enhanced concurrency, allowing more jobs to run in parallel. Removed the search date-range limit, now supporting any date range over the table’s retention. These updates make it easier to explore massive datasets while giving you greater control over costs and performance. Explore the New UI Experience Let’s walk through a familiar scenario to showcase the new UI. Imagine you want to check if a specific client IP address has repeatedly accessed your system over the past year, as part of investigating suspicious activity. With the new Search Job experience, scanning through massive volumes of logs is now fast, simple, and intuitive. Step-by-Step: Start by typing your query or selecting the relevant table - here, we’re querying the SecurityEvent table for a suspicious IP address. Open the ellipsis menu (…) on the right and choose "Search Job". Use the time picker to set your date range. For example, select ‘Last year’ to view a full year of activity, or choose a longer period if needed. Name your new results table, such as SecurityEventJuly25. Before running the job, you’ll see an approximate cost estimation, helping you decide if you want to proceed with the query. Click Run to launch the Search Job. A new table is created in your workspace, allowing you to analyze results efficiently without impacting performance. This new UI flow makes it seamless to handle even large-scale investigations like this, with fewer clicks and better visibility along the way. What’s Next? We’re continuing to enhance Search Job with broader KQL operator support and additional features. Stay tuned for more updates! For a deeper dive into all these improvements, check out the full documentation https://aka.ms/LogAnalyticsSearchJobs. For questions or feedback, feel free to leave a comment on the blog or use the “Give feedback” form directly in the Logs UI.
