Making Azure the Best Place to Observe Your Apps with OpenTelemetry
Our goal is to make Azure the most observable cloud. To that end, we are refactoring Azure’s native observability platform to be based on OpenTelemetry, an industry standard for instrumenting applications and transmitting telemetry.
Query Azure Resource Graph from Azure Monitor
Azure Monitor now lets you query data in Azure Resource Graph from your Log Analytics workspace. You can utilize this feature to make your Azure Log Analytics queries aware of your resource’s properties such as tags, resource attributes and much more. You can run cross-service queries by using any client tools that support Kusto Query Language (KQL) queries, including the Log Analytics web UI, workbooks, PowerShell, and the REST API. Read more here
Log Analytics results upgrade
Log Analytics team has shipped one of it's most significant updates to date - a new and upgraded results set grid. The new results set boasts a brand new experience and designed aimed at empowering Log Analytics users to do more with Logs. The new result set introduces numerous new features and improvements and is designed to be more accessible and easy to use. Users can now use on grid filters, drag and drop actions and even Pivot mode to easily analyze their query results with no KQL knowledge needed.Accelerate your observability journey with Azure Monitor pipeline (preview)
In the ever-evolving landscape of digital infrastructure, transparency in resource and application performance is imperative. Success hinges on visibility, and that’s true whether you’re operating on Azure, on-premise, or at the edge. As organizations scale their infrastructures and applications, the volume of observability data naturally increases. This surge can complicate the management of networking, data storage and ingestion, often forcing a trade-off between cost management and observability. The complexity doesn’t end there. The very tools designed to ingest, process, and route this data can be both costly and complex, adding layers of operational challenges. Moreover, edge infrastructure is deployed near IoT devices for optimal data processing, high availability, and reduced latency. This adds its own set of challenges when it comes to collecting telemetry from such constrained environments. Recognizing these challenges, our team has been focused on providing a robust, highly scalable, and secure data ingestion solution through Azure Monitor. We are thrilled to announce the preview of the Azure Monitor pipeline at edge. What is Azure Monitor pipeline? Azure Monitor pipeline, similar to ETL (Extract, Transform, Load) process, enhances traditional data collection methods. It streamlines data collection from various sources through a unified ingestion pipeline and utilizes a standardized configuration approach that is more efficient and scalable. This is particularly beneficial for cloud-based monitoring in Azure. We are now extending our Azure Monitor pipeline capabilities from the cloud to the edge, enabling high-scale data ingestion with centralized configuration management. What is Azure Monitor pipeline at edge? Azure Monitor pipeline at edge is a powerful solution designed to facilitate high-scale data ingestion and routing from edge environments to Azure Monitor for observability. It leverages the robust capabilities of the vendor-agnostic tool - OpenTelemetry Collector, which is used by enterprises worldwide to manage high volumes of telemetry each month. With the Azure Monitor pipeline at edge, organizations can tap into the same highly scalable platform with a standardized configuration and reliability. Whether dealing with petabytes of data or seeking consistent observability experience across Azure, edge, and multi-cloud, this solution empowers organizations to reliably collect telemetry and drive operational excellence. The Azure Monitor pipeline at edge is equipped with out-of-the-box capabilities to receive telemetry from a diverse range of resources and route it to Azure Monitor. Here are some key features: High scale data ingestion: Customers have various devices and resources at edge, emitting high volume of data. With Azure Monitor pipeline at edge, you can seamlessly scale to support ingestion of high volume of data in the cloud. Azure Monitor pipeline can be deployed on your on-premises Kubernetes cluster as an Arc Kubernetes cluster extension. This allows it to adapt to your data scaling needs by running multiple replica sets and provides you with full control to define workflows and route high-volume data to Azure Monitor. Observing resources in isolated environments: In the manufacturing sector, resources are often located in isolated network zones without direct cloud connectivity, posing challenges for telemetry collection. With the Azure Monitor pipeline at edge, combined with Azure IoT Layered Network Management, you can facilitate a connection between Azure and Kubernetes clusters in isolated networks, deploy the Azure Monitor pipeline at edge, collect data from resources in segmented networks, and route it to Azure Monitor for comprehensive observability. Reliable data ingestion and prevent data loss: Edge environments frequently encounter intermittent connectivity, leading to potential data loss and disrupting data continuity. The Azure Monitor pipeline at edge allows you to cache logs during periods of intermittent connectivity. When connectivity is re-established, your data is synchronized with Azure Monitor, preventing data loss. Getting started It’s super easy to get started! You need to deploy the Azure Monitor pipeline on a single Arc-enabled Kubernetes cluster in your environment. Once that is done, you can configure your resources to emit the telemetry to Azure Monitor pipeline at edge and ingest into Azure Monitor for observability. Once you Arc-enable your on-prem Kubernetes cluster and the prerequisites are met, go the Extension section, select Azure Monitor pipeline extension (preview) and create the instance. Alternatively, from the search bar in the Azure portal, select Azure Monitor pipeline and then click Create. Enter the information related to the pipeline instance. The Dataflow tab allows you to create and edit dataflows for the pipeline instance. Configure your resources to emit the telemetry to the Azure Monitor pipeline. Learn more in our documentation. Pricing There is no additional cost to use Azure Monitor pipeline to send data to Azure Monitor. You will be only charged for data ingestion as per the current pricing. FAQ What telemetry can be collected using Azure Monitor pipeline? Currently, in public preview, you can collect syslogs and OTLP logs using Azure Monitor pipeline at edge. We will keep expanding the data collection capabilities based on your feedback and requirements. How can I perform transformations on the telemetry that is collected? You can certainly transform your telemetry! Since this is an extension of Azure Monitor pipeline, you can perform the data collection transformations in the Azure Monitor pipeline at cloud. Is this another agent for data collection? Azure Monitor pipeline at edge is engineered to function in environments where installing agents on resources is not feasible, whether due to technical limitations or warranty concerns. It enables you to get the telemetry from these resources and acts as a central forwarding component to ingest high volume data. I have 100 Linux servers in my on-prem environment. Do I need to deploy Azure Monitor pipeline at edge on all of them? You need to deploy the Azure Monitor pipeline at edge on a single Arc-enabled Kubernetes cluster and configure it to ingest data into Azure Monitor. Once that is completed, you can configure your Linux servers to emit telemetry to the Azure Monitor pipeline at edge instance.Join Azure Monitor 'Monthly Connections' on June 30 for live Q&A
Join the Azure Monitor engineering team for our first 'Monthly Connections' event on June 30, 2020. This will be an "Ask-Me-Anything" event where you can have all of your monitoring questions answered live by our panel of experts. We will also highlight all the new scenarios & capabilities launched in the last month, that you can start using today!Making it easier to manage your workspace with a clearer Log Analytics Workspace menu
Customers have been using Azure Monitor Log Analytics workspaces to store and analyze logs for almost a decade. Over these years, the service has evolved, with new options being added and old technologies phased out. We’ve tried to keep the workspace administration UI stable without making too many changes, but it’s time for a refresh. Today we start rolling out a revamped workspace menu that will help you get to the things you need quicker. Key items are now right at the top, where you expect them to be: The new menu is gradually being exposed, so you might not see it immediately. As part of this change, we're also clearly indicating menu options that are in the process of replacement or deprecation. Here are the relevant options and their replacement: Menu option Replacement technology The MMA and OMS agents are being replaced by the AMA agent. For more details and assistance in the process, see https://aka.ms/migrateToAMA. All table management operations are now done on a single table management UI, which covers all aspects of workspace tables, including custom logs. For details, see Manage tables in a Log Analytics workspace - Azure Monitor | Microsoft Learn. You no longer need connectors to export Azure Activity Logs to Log Analytics workspaces. For instructions, see Azure activity log - Azure Monitor | Microsoft Learn. Some agents in specific scenarios ingest logs to storage accounts. This option exists to collect logs from such accounts. See more details how to collect logs directly to Azure Monitor in https://aka.ms/migrateToAMA. Workspace summary is the old OMS portal for solutions built using View Designer. This technology was replaced with Workbooks and Insights several years ago. For more details, see View Designer in Azure Monitor is retiring on 31 August 2023 | Azure updates | Microsoft Azure. Azure Monitor VM Insights has evolved to use the new AMA agent, which also includes the Service Map functionality. For more details, see Migrate to Azure Monitor VM insights by 30 September 2025 when Service Map will be retired | Azure updates | Microsoft Azure. Azure Monitor had several technologies to help deploy and manage the old agent's data collection and logic. These technologies became obsolete with the move to the new AMA agent, which offers rich management and policy mechanisms. For more details, see https://aka.ms/migrateToAMA.Azure Monitor- Announcing Public Preview of Azure Metrics Data plane API
The Azure Monitor REST API , hosted behind Azure Resource Manager, is a widely used API service in Azure that allows customers to gain insights into Azure resources. However, customers who leverage this API to query metrics in bulk at high frequency and push metrics to other datastores for the purpose of deep analytics may experience throttling issues with the current API. To address this, and better empower these use cases, Microsoft Azure has introduced the Azure Metrics Data plane API, a new batch API that enables efficient and large-scale queries. We are excited today to announce the Public Preview of the Azure Metrics Data plane API. This powerful addition allows customers to query metrics for their Azure resources at a large scale. So, what is Azure Monitor Metrics Data plane API? Azure Metrics Data plane API is a new paradigm in Azure Monitoring which makes resource insight gathering more efficient with a higher capacity querying experience. It provides the capability to retrieve the metric data for up to 50 resource Ids in the same subscription and region in a single batch API call thus improving query throughput and reducing the risk of throttling. Here are some of the key features and benefits of Azure Metrics Data plane API - Higher Querying Limits: Azure Metrics Data plane API is designed for querying metric data for your resources with higher query limits than existing Azure Resource Manager APIs. This is particularly beneficial for customers who have large subscriptions with multiple resources, as they can query data without worrying about being throttled. Compared to the REST API, which only allows 12,000 API calls per hour, this new API enables you to perform up to 360,000 API calls per hour. This improvement in query throughput provides a more efficient and streamlined experience for customers. Efficiency: Collecting metrics for multiple resources in a single API call is more efficient than making multiple API calls for each resource. The Azure Metrics Data plane API offers a single batch API call which can accommodate up to 50 resource IDs, resulting in higher throughput and more efficient querying. Improved Performance: The Azure Metrics Data plane API can enhance the performance of client-side services by reducing the number of calls needed to extract the same amount of metric data for resources, resulting in improved performance. Overall, this API provides a powerful and efficient mechanism for organizations looking to monitor and manage their resources more effectively at a scale. By using this API, organizations can streamline their monitoring processes and improve their infrastructure's overall performance. We look forward to seeing the innovative solutions and exciting use cases that developers and organizations will create using the Azure Metrics Data plane API. Our API is being used by several customers in Private Preview and here is what some of them have to say about their experience with the Azure Metrics Data plane API: Customer Quote “Datadog uses Azure Monitor APIs to ingest metrics on behalf of thousands of organizations, including some of the largest Azure customers in the world. Through our partnership with the Azure Monitor team, we’ve been able to leverage the Data plane API to ingest data with minimal latency, enabling faster alerting and near real-team evaluation of health and performance even for enterprise-scale Azure environments,” says Jimmy Caputo, Director of Product Management at Datadog.” The new metrics Data plane API will be charged at a rate of $0.01/1000 API calls (Refer to pricing details here- Pricing - Azure Monitor | Microsoft Azure). The service is currently free to use, with billing set to begin on 10/1/2023. Please note current users of the Metrics REST API hosted behind the ARM APIs will see no change in pricing. Learn more about Azure Metrics Data plane API - Metrics in Azure Monitor - Azure Monitor | Microsoft Learn.Join Azure Monitor Monthly Connections on July 30 for live Q&A
Join the Azure Monitor engineering team for our next 'Monthly Connections' event on July 30, 2020. This will be an "Ask-Me-Anything (AMA)" event where you can have all of your monitoring questions answered live by our panel of experts. We will also highlight all the new scenarios & capabilities launched in the last month, that you can start using today!
