Making Azure the Best Place to Observe Your Apps with OpenTelemetry
Our goal is to make Azure the most observable cloud. To that end, we are refactoring Azure’s native observability platform to be based on OpenTelemetry, an industry standard for instrumenting applications and transmitting telemetry.What’s new in Observability at Build 2025
At Build 2025, we are excited to announce new features in Azure Monitor designed to enhance observability for developers and SREs, making it easier for you to streamline troubleshooting, improve monitoring efficiency, and gain deeper insights into application performance. With our new AI-powered tools, customizable alerts, and advanced visualization capabilities, we’re empowering developers to deliver high-quality, resilient applications with greater operational efficiency. AI-Powered Troubleshooting Capabilities We are excited to disclose two new AI-powered features, as well as share an update to a GA feature, which enhance troubleshooting and monitoring: AI-powered investigations (Public Preview): Identifies possible explanations for service degradations via automated analyses, consolidating all observability-related data for faster problem mitigation. Attend our live demo at Build and learn more here. Health models (Public Preview – coming in June 2025): Significantly improves the efficiency of detecting business-impacting issues in workloads, empowering organizations to deliver applications with operational efficiency and resilience through a full-stack view of workload health. Attend our live demo at Build to get a preview of the experience and learn more here. AI-powered Application Insights Code Optimizations (GA): Provides code-level suggestions for running .NET apps on Azure. Now, it’s easier to get code-level suggestions with GitHub Copilot coding agent (preview) and GitHub Copilot for Azure in VS Code. Learn more here. Enhanced AI and agent observability Azure Monitor and Azure AI Foundry now jointly offer real-time monitoring and continuous evaluation of AI apps and agentic systems in production. These capabilities are deeply integrated with the Foundry Observability experience and allow you to track key metrics such as performance, quality, safety, and resource usage. Features include: Unified observability dashboard for generative AI apps and agents (Public Preview): Provides full-stack visibility of AI apps and infrastructure with AI app metrics surfaced in both Azure Monitor and Foundry Observability. Alerts: Data is published to Azure Monitor Application Insights, allowing users to set alerts and analyze them for troubleshooting. Debug with tracing capabilities: Enables detailed root-cause analysis of issues like groundedness regressions. Learn more in our breakout session at Build! Improved Visualization We have expanded our visualization capabilities, particularly for Kubernetes services: Azure Monitor dashboards with Grafana (Public Preview): Create and edit Grafana dashboards directly in the Azure Portal with no additional cost. This includes dashboards for Azure Kubernetes Services (AKS) and other Azure resources. Learn more. Managed Prometheus Visualizations: Supports managed Prometheus visualizations for both AKS clusters (GA) and Arc-enabled Kubernetes clusters (Public Preview), offering a more cost-efficient and performant solution. Learn more. Customized and Simplified Monitoring Through enhancements to alert customization, we’re making it easier for you to get started with monitoring: Prometheus community recommended alerts: Offers one-click enablement of Prometheus recommended alerts for AKS clusters (GA) and Arc-enabled Kubernetes clusters (Public Preview), providing comprehensive alerting coverage across cluster, node, and pod levels. Simple log alerts (Public Preview): Designed to provide a simplified and more intuitive experience for monitoring and alerting, Simple log alerts evaluate each row individually, providing faster alerting compared to traditional log alerts. Simple log alerts support multiple log tiers, including Analytics and Basic Logs, which previously did not have any alerting solution. Learn more. Customizable email subjects for log search alerts (Public Preview): Allows customers to personalize the subject lines of alert emails including dynamic values, making it easier to quickly identify and respond to alerts. Send a custom event from the Azure Monitor OpenTelemetry Distro (GA): Offers developers a way to track user or system actions that matter the most to their business objectives, now available in the Azure Monitor OpenTelemetry Distro. Learn more. Application Insights auto-instrumentation for Java and Node Microservices on AKS (Public Preview): Easily monitor your Java and Node deployments without changing your code by leveraging auto-instrumentation that is integrated into the AKS cluster. These capabilities will help you easily assess the performance of your application and identify the cause of incidents efficiently. Learn more. Enhancements for Large Enterprises and Government Entities Azure Monitor Logs is introducing several new features aimed at supporting highly sensitive and high-volume logs, empowering large enterprises and government entities. With better data control and access, developers at these organizations can work better with IT Professionals to improve the reliability of their applications. Workspace replication (GA): Enhances resilience to regional incidents by enabling cross-regional workspace replication. Logs are ingested in both regions, ensuring continued observability through dashboards, alerts, and advanced solutions like Microsoft Sentinel. Granular RBAC (Public Preview): Supports granular role-based access control (RBAC) using Azure Attribute-Based Access Control (ABAC). This allows organizations to have row-level control on which data is visible to specific users. Data deletion capability (GA): Allows customers to quickly mark unwanted log entries, such as sensitive or corrupt data, as deleted without physically removing them from storage. It’s useful for unplanned deletions using filters to target specific records, ensuring data integrity for analysis. Process more log records in the Azure Portal (GA): Supports up to 100,000 records per query in the Azure Portal, enabling deeper investigations and broader data analysis directly within the portal without need for additional tools. We’re proud to further Azure Monitor's commitment to providing comprehensive and efficient observability solutions for developers, SREs, and IT Professionals alike. For more information, chat with Observability experts through the following sessions at Build 2025: BRK168: AI and Agent Observability with Azure AI Foundry and Azure Monitor BRK188: Power your AI Apps Across Cloud and Edge with Azure Arc DEM547: Enable application monitoring and troubleshooting faster with Azure Monitor DEM537: Mastering Azure Monitor: Essential Tips in 15 Minutes Expo Hall (Meet the Experts): Azure Arc and Azure Monitor boothAzure Monitor Application Insights Auto-Instrumentation for Java and Node Microservices on AKS
Key Takeaways (TLDR) Monitor Java and Node applications with zero code changes Fast onboarding: just 2 steps Supports distributed tracing, logs, and metrics Correlates application-level telemetry in Application Insights with infrastructure-level telemetry in Container Insights Available today in public preview Introduction Monitoring your applications is now easier than ever with the public preview release of Auto-Instrumentation for Azure Kubernetes Service (AKS). You can now easily monitor your Java and Node deployments without changing your code by leveraging auto-instrumentation that is integrated into the AKS cluster. This feature is ideal for developers or operators who are... Looking to add monitoring in the easiest way possible, without modifying code and avoiding ongoing SDK update maintenance. Starting out on their monitoring journey and looking to benefit from carefully chosen default configurations with the ability to tweak them over time. Working with someone else’s code and looking to instrument at scale. Or considering monitoring for the first time at the time of deployment. Before the introduction of this feature, users needed to manually instrument code, install language-specific SDKs, and manage updates on their own—a process that involved significant effort and numerous opportunities for errors. Now, all you need to do is follow a simple two-step process to instrument your applications and automatically send correlated OpenTelemetry-based application-level logs, metrics, and distributed tracing to your Application Insights resource. With AKS Auto-Instrumentation, you will be able to assess the performance of your application and identify the cause of any incidents more efficiently using the robust application performance monitoring capabilities of Azure Monitor Application Insights. This streamlined approach not only saves time but also ensures that your monitoring setup is both reliable and scalable. Feature Enablement and Onboarding To onboard to this feature, you will need to follow a two-step process: Prepare your cluster by installing the application monitoring webhook. Choose between namespace-wide onboarding or per-deployment onboarding by creating K8’s custom resources. Namespace-wide onboarding is the easiest method. It allows you to instrument all Java or Node deployments in your namespace and direct telemetry to a single Application Insights resource. Per-deployment onboarding allows more control by targeting specific deployments and directing telemetry to different Application Insights resources. Once the custom resource is created, you will need to deploy or redeploy your application, and telemetry will start flowing to Application Insights. For step-by-step instructions and to learn more about onboarding visit our official documentation on MS Learn. The Application Insights experience Once telemetry begins flowing, you can take advantage of Application Insights features such as Application Map, Failures/Performance Views, Availability, and more to help you efficiently diagnose and troubleshoot application issues. Let’s look at an example: I have an auto-instrumented distributed application running in the demoapp namespace of my AKS cluster. It consists of: One Java microservice Two Node.js microservices MongoDB and Redis as its data layer Scenario: End users have been complaining about some latency in the application. As the DRI, I can start my troubleshooting journey by going to the Application Map to get a topological view of my distributed application. I open Application Map and notice MicroserviceA has a red border - 50% of calls are erroring. The Container Insights card shows healthy pods - no failed pods or high CPU/memory usage. I can eliminate infrastructure issues as the cause of the slowness. In the Performance card, I spot that the rescuepet operation has an average duration of 10 seconds. That's pretty long. I drill in to get a distributed trace of the operation and find the root cause: an OutOfMemoryError. In this scenario, the issue has been identified as an out-of-memory error at the application layer. However, when the root cause is not in the code but in the infrastructure I get a full set of resource properties with every distributed trace so I can easily identify the infra resources running each span of my trace. I can click the investigate pods button to transition to Azure Monitor Container Insights and investigate my pods further. This correlation between application-level and infrastructure-level telemetry makes it much easier to determine whether the issue is caused by the application or the infrastructure. Pricing There is no additional cost to use AKS auto-instrumentation to send data to Azure Monitor. You will be only charged as per the current pricing. What’s Next Language Support This integration supports Java and Node workloads by leveraging the Azure Monitor OpenTelemetry distro. We have distros for .NET and Python as well and we are working to integrate these distros into this solution. At that point, this integration will support .NET, Python, Java and Node.js. For customers that want to instrument workloads in other languages such as Go, Ruby, PHP, etc. we plan to leverage open-source instrumentations available in the Open Telemetry community. In this scenario, customers will instrument their code using open source OpenTelemetry instrumentations, and we will provide mechanisms that will make it easy to channel the telemetry to Application Insights. Application Insights will expose an endpoint that accepts OpenTelemetry Language Protocol (OTLP) signals and configure the instrumented workload to channel the telemetry to this endpoint. Operating Systems and K8’s Controllers Right now, you can only instrument kubernetes deployments running on Linux node pools, but we plan to expand support to introduce support for Linux ARM64 node pools as well as support for StatefulSet, Job, Cronjob, and Replicaset controller types. Portal Experiences We are also working on Azure portal experiences to make onboarding easier. When our portal experiences for onboarding are released, users will be able to install the Application Insights extension for AKS using the portal and use a portal user interface to instrument their workloads instead of having to create custom resources. Beyond onboarding, we are working to build Application Insights consumption experiences within the AKS namespace and workloads blade. You will be able to see application-level telemetry right there in the AKS portal without having to navigate away from your cluster to Application Insights. FAQs: What are the advantages of AKS Auto-Instrumentation? No code changes required No access to source code required No configuration changes required Eliminates instrumentation maintenance What languages are supported by AKS Auto-Instrumentation? Currently, AKS Auto-Instrumentation supports Java and Node.js applications. Python and .NET support is coming soon. Moreover, we will be adding support for all OTel supported languages like Go soon via native OTLP ingestion. Does AKS Auto-Instrumentation support custom metrics? For Node.js applications, custom metrics require manual instrumentation with the Azure Monitor OpenTelemetry Distro. Java applications allow custom metrics with auto-instrumentation. Click here for more FAQs. This article was co-authored by Rishab Jolly and Abinet AbateAzure Monitor Private Link Scope (AMPLS) Scale Limits Increased by 10x!
What is Azure Monitor Private Link Scope (AMPLS)? Azure Monitor Private Link Scope (AMPLS) is a feature that allows you to securely connect Azure Monitor resources to your virtual network using private endpoints. This ensures that your monitoring data is accessed only through authorized private networks, preventing data exfiltration and keeping all traffic inside the Azure backbone network. AMPLS – Scale Limits Increased by 10x in Public Cloud - Public Preview In a groundbreaking development, we are excited to share that the scale limits for Azure Monitor Private Link Scope (AMPLS) have been significantly increased by tenfold (10x) in Public Cloud regions as part of the Public Preview! This substantial enhancement empowers our customers to manage their resources more efficiently and securely with private links using AMPLS, ensuring that workload logs are routed via the Microsoft backbone network. Addressing Customer Challenges Top Azure Strategic 500 customers, including leading Telecom service providers, Banking & Financial services customers, have reported that the previous limits of AMPLS were insufficient to meet their growing demands. The need for private links has surged 3-5 times beyond capacity, impacting network isolation and integration of critical workloads. Real-World Impact Our solution now enables customers to scale their Azure Monitor resources significantly, ensuring seamless network configurations and enhanced performance. Scenario 1: A Leading Telecom Service Provider known for its micro-segmentation architecture, have faced challenges with large-scale monitoring and reporting due to limitations on AMPLS. With the new solution, the customer can now scale up to 3,000 Log Analytics and 10,000 Application Insights workspaces with a single AMPLS resource, allowing them to configure over 13,000 Azure Monitor resources effortlessly. Scenario 2: A Leading Banking & Financial Services Customer have faced the scale challenges in delivering personalized insights due to complex workflows. By utilizing Azure Monitor with network isolation configurations, the customer can now scale their Azure Monitor resources to ensure secure telemetry flow and compliance. They have enabled thousands of Azure Monitor resources configured with AMPLS. Key Benefits to the Customer We believe that the solution our team has developed will significantly improve our customers' experience, allowing them to manage their resources more efficiently and effectively with private links using AMPLS. An AMPLS object can now connect up to 3,000 Log Analytics workspaces and 10,000 Application Insights components. (10x Increase) The Log Analytics workspace limit has been increased from 300 to 3,000 (10x increase). The Application Insights limit has increased from 1,000 to 10,000 (10x increase). An Azure Monitor resources can now connect up to 100 AMPLSs (20x increase). Data Collection Endpoint (DCE) Log Analytics Workspace (LA WS) Application Insights components (AI) An AMPLS object can connect to 10 private endpoints at most. Redesign of AMPLS – User experience to load 13K+ resources with Pagination Call to Action Explore the new capabilities of Azure Monitor Private Link Scope (AMPLS) and see how it can transform your network isolation and resource management. Visit our Azure Monitor Private Link Scope (AMPLS) documentation page for more details and start leveraging these enhancements today! For detailed information on configuring Azure Monitor private link scope and azure monitor resources, please refer to the following link: Configure Azure Monitor Private Link Scope (AMPLS) Configure Private Link for Azure MonitorAccelerate your observability journey with Azure Monitor pipeline (preview)
In the ever-evolving landscape of digital infrastructure, transparency in resource and application performance is imperative. Success hinges on visibility, and that’s true whether you’re operating on Azure, on-premise, or at the edge. As organizations scale their infrastructures and applications, the volume of observability data naturally increases. This surge can complicate the management of networking, data storage and ingestion, often forcing a trade-off between cost management and observability. The complexity doesn’t end there. The very tools designed to ingest, process, and route this data can be both costly and complex, adding layers of operational challenges. Moreover, edge infrastructure is deployed near IoT devices for optimal data processing, high availability, and reduced latency. This adds its own set of challenges when it comes to collecting telemetry from such constrained environments. Recognizing these challenges, our team has been focused on providing a robust, highly scalable, and secure data ingestion solution through Azure Monitor. We are thrilled to announce the preview of the Azure Monitor pipeline at edge. What is Azure Monitor pipeline? Azure Monitor pipeline, similar to ETL (Extract, Transform, Load) process, enhances traditional data collection methods. It streamlines data collection from various sources through a unified ingestion pipeline and utilizes a standardized configuration approach that is more efficient and scalable. This is particularly beneficial for cloud-based monitoring in Azure. We are now extending our Azure Monitor pipeline capabilities from the cloud to the edge, enabling high-scale data ingestion with centralized configuration management. What is Azure Monitor pipeline at edge? Azure Monitor pipeline at edge is a powerful solution designed to facilitate high-scale data ingestion and routing from edge environments to Azure Monitor for observability. It leverages the robust capabilities of the vendor-agnostic tool - OpenTelemetry Collector, which is used by enterprises worldwide to manage high volumes of telemetry each month. With the Azure Monitor pipeline at edge, organizations can tap into the same highly scalable platform with a standardized configuration and reliability. Whether dealing with petabytes of data or seeking consistent observability experience across Azure, edge, and multi-cloud, this solution empowers organizations to reliably collect telemetry and drive operational excellence. The Azure Monitor pipeline at edge is equipped with out-of-the-box capabilities to receive telemetry from a diverse range of resources and route it to Azure Monitor. Here are some key features: High scale data ingestion: Customers have various devices and resources at edge, emitting high volume of data. With Azure Monitor pipeline at edge, you can seamlessly scale to support ingestion of high volume of data in the cloud. Azure Monitor pipeline can be deployed on your on-premises Kubernetes cluster as an Arc Kubernetes cluster extension. This allows it to adapt to your data scaling needs by running multiple replica sets and provides you with full control to define workflows and route high-volume data to Azure Monitor. Observing resources in isolated environments: In the manufacturing sector, resources are often located in isolated network zones without direct cloud connectivity, posing challenges for telemetry collection. With the Azure Monitor pipeline at edge, combined with Azure IoT Layered Network Management, you can facilitate a connection between Azure and Kubernetes clusters in isolated networks, deploy the Azure Monitor pipeline at edge, collect data from resources in segmented networks, and route it to Azure Monitor for comprehensive observability. Reliable data ingestion and prevent data loss: Edge environments frequently encounter intermittent connectivity, leading to potential data loss and disrupting data continuity. The Azure Monitor pipeline at edge allows you to cache logs during periods of intermittent connectivity. When connectivity is re-established, your data is synchronized with Azure Monitor, preventing data loss. Getting started It’s super easy to get started! You need to deploy the Azure Monitor pipeline on a single Arc-enabled Kubernetes cluster in your environment. Once that is done, you can configure your resources to emit the telemetry to Azure Monitor pipeline at edge and ingest into Azure Monitor for observability. Once you Arc-enable your on-prem Kubernetes cluster and the prerequisites are met, go the Extension section, select Azure Monitor pipeline extension (preview) and create the instance. Alternatively, from the search bar in the Azure portal, select Azure Monitor pipeline and then click Create. Enter the information related to the pipeline instance. The Dataflow tab allows you to create and edit dataflows for the pipeline instance. Configure your resources to emit the telemetry to the Azure Monitor pipeline. Learn more in our documentation. Pricing There is no additional cost to use Azure Monitor pipeline to send data to Azure Monitor. You will be only charged for data ingestion as per the current pricing. FAQ What telemetry can be collected using Azure Monitor pipeline? Currently, in public preview, you can collect syslogs and OTLP logs using Azure Monitor pipeline at edge. We will keep expanding the data collection capabilities based on your feedback and requirements. How can I perform transformations on the telemetry that is collected? You can certainly transform your telemetry! Since this is an extension of Azure Monitor pipeline, you can perform the data collection transformations in the Azure Monitor pipeline at cloud. Is this another agent for data collection? Azure Monitor pipeline at edge is engineered to function in environments where installing agents on resources is not feasible, whether due to technical limitations or warranty concerns. It enables you to get the telemetry from these resources and acts as a central forwarding component to ingest high volume data. I have 100 Linux servers in my on-prem environment. Do I need to deploy Azure Monitor pipeline at edge on all of them? You need to deploy the Azure Monitor pipeline at edge on a single Arc-enabled Kubernetes cluster and configure it to ingest data into Azure Monitor. Once that is completed, you can configure your Linux servers to emit telemetry to the Azure Monitor pipeline at edge instance.Optimizing Cost using the Azure Monitor OpenTelemetry Distro
Immediately after Willow deployed the Azure Monitor OpenTelemetry Distro, their Azure Costs spiked, and they approached Azure Monitor Team for strategies to optimize cost. By following the strategies in this blog, they were able to reduce costs by over 90%!
