Seamless Metric Export: Simplifying AKS Platform Metrics Routing to Any Destination

Platform metrics in Azure

Platform or resource metrics are telemetry data automatically collected by Azure Monitor for resources running within the Azure environment. These metrics include performance indicators such as CPU usage, memory consumption, network traffic, and disk I/O, which are critical for resource monitoring and performance tuning.

For Azure Kubernetes Service (AKS), platform metrics provide insights into both the Kubernetes cluster and its underlying infrastructure. Examples include:

• Node metrics: CPU utilization, memory usage
• Pod metrics: Pod status
• Control plane metrics: API server inflight requests

These metrics enable administrators and developers to monitor, troubleshoot, and optimize their applications effectively. The list of platform metrics for AKS is available here. This blog explores how to export and utilize platform metrics from AKS to other destinations like Log Analytics, Event Hub and Storage Accounts, with a step-by-step example.

Exporting AKS Platform metrics

This feature is in Preview as of September 2024. Please make sure to read the official preview policy before using this in production

Azure Monitor Metrics Export is configurable through Data Collection Rules (DCR), which provides the capability to route Azure resource metrics data to Azure Storage Accounts, Azure Event Hubs and Azure Log Analytics Workspace for 18 resource types and 10 Azure public regions, which includes AKS. Metrics Export feature provides a more scalable, flexible and reliable way to export platform metrics, in comparison to Azure Monitor Diagnostic Settings.

Exporting platform metrics enables users to co-locate their metrics in a single store so that they can use a wide variety of monitoring and dashboarding tools. Additionally, since platform metrics are retained in Azure Monitor only for 93 days, exporting these metrics are crucial while making long term business critical decisions. Metrics Export also enables users to export these metrics in near-real time, with full fidelity and at scale.

Using Metrics Export, platform metrics can be sent to the following destinations:

1. Log Analytics Workspaces
• Metrics are stored in the AzureMetricsV2 table.
• The workspace and DCR must reside in the same region, but the monitored resources can be in any region.
2. Azure Event Hubs
• Enables integration with external systems for real-time analytics.
• The Event Hub, DCR, and monitored resources must be in the same region.
3. Azure Storage Accounts
• Suitable for long-term storage.
• Similar regional constraints apply.

Example: Exporting AKS platform metrics (CLI and Portal)

Step 1: Create an AKS Cluster

First, create a new AKS cluster using the Azure CLI / UX. You can skip this step if you have an existing AKS cluster

CLI

az aks create --resource-group myResourceGroup --name myAKSCluster  --node-count 2 

Portal 

1.  Search for AKS on the Azure Portal Marketplace

2.  Start creating AKS Cluster using the creation wizard.  You can choose the defaults 

Save the resource ID of the AKS cluster for further steps. You can find the resource ID in the Properties tab under Settings in Portal or use the following command 

az aks show --resource-group $myResourceGroup --name $aksClusterName --query id --output tsv

Step 2: Configure Data Collection Rules (DCR) in Azure Monitor

Create a DCR to specify the metrics to collect and the destination. We will look at examples of sending metrics to both Log Analytics and Event Hubs as destination.

CLI - Log Analytics as destination

1. Creating a DCR with Log Analytics as a destination. In this example, we are exporting two metrics all metrics. If you are interested in specific metrics, you can specify them in the streams field by following the documentation here


First, we need to create a rule file  (named rule-file.json) with the details for the destination and the source.  Make sure you to create a Log analytics workspace and have the workspace ID handy for this step. The log Analytics workspace can be located in a different region from the cluster or the DCR

{
    "identity": {
        "type": "systemAssigned"
    },
    "kind": "PlatformTelemetry",
    "location": "westus2",
    "properties": {
        "dataSources": {
            "platformTelemetry": [
                {
                    "streams": [
                         "Microsoft.ContainerService/managedClusters:Metrics-Group-All"
                    ],
                    "name": "myPlatformTelemetryDatasource"
                }
            ]
        },
        "destinations": {
            "logAnalytics": [ 
                 { 
                    "workspaceResourceId": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourcegroups/rg-001/providers/microsoft.operationalinsights/workspaces/laworkspace001", 
                    "name": "ladestination" 
                 } 
             ]     },
        "dataFlows": [
            {
                "streams": [
                            "Microsoft.ContainerService/managedClusters:Metrics-Group-All"            
                         ],
                "destinations": [
                    "ladestination"               
                ]
            }
        ]
    }
    
}

az monitor data-collection rule create  --name AKSMetricsToLogAnalytics  --location myRegion -g myResourceGroup --rule-file rule.json

Save the resource ID of the DCR for the next step 

az monitor data-collection rule show --name aksMetricstoLogAnalytics -g test-rg --query id --output tsv

2. Linking DCR to the AKS resource using DCRA (Data Collection Rules Association)

az monitor data-collection rule association create --resource-group myResourceGroup -n logAnalyticsDCRAssociation  --rule-id "<DCR resourceid>" --resource "<AKS cluster ID>"

Replace the rule ID with the resource ID of the DCR and AKS cluster ID with the resource ID of the AKS cluster 

CLI - Event Hubs as destination

1. Create an Event Hub in your desired namespace:

az eventhubs namespace create \ 

    --name myEventHubNamespace \ 

    --resource-group myResourceGroup

az eventhubs eventhub create --name $eventhubName --resource-group $rgName --namespace-name $namespaceName

Save the resource ID of the event hub created for the next step. You can find it through CLI using the following command 

az eventhubs eventhub show --name $eventHubName --namespace-name $namespaceName --resource-group $resourceGroup --query id --output tsv

2. Creating a DCR with Event Hub as the destination:

We first create the rule file with the details of the destination. Replace the eventHubResourceID with the ID of the Event hub created in Step 1. 

{
    "identity": {
        "type": "systemAssigned"
    },
    "kind": "PlatformTelemetry",
    "location": "westus2",
    "properties": {
        "dataSources": {
            "platformTelemetry": [
                {
                    "streams": [
                         "Microsoft.ContainerService/managedClusters:Metrics-Group-All"
                    ],
                    "name": "myPlatformTelemetryDatasource"
                }
            ]
        },
        "destinations": {
            "eventHubs": [ 
                 { 
                    "eventHubResourceId": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/rg-001/providers/Microsoft.EventHub/namespaces/event-hub-001/eventhubs/hub-001", 
                    "name": "myHub" 
                 } 
             ]     },
        "dataFlows": [
            {
                "streams": [
                            "Microsoft.ContainerService/managedClusters:Metrics-Group-All"            
                         ],
                "destinations": [
                    "myHub"               
                ]
            }
        ]
    }
    
}

We create the DCR based on the rule file above

az monitor data-collection rule create  --name AKSMetricsToEventHub  --location myRegion -g myResourceGroup --rule-file rule.json

Save the resource ID of the event hub created for the next step. You can find it through CLI using the following command 

az eventhubs eventhub show --name $eventHubName --namespace-name $namespaceName --resource-group $resourceGroup --query id --output tsv

3. Linking DCR to the AKS resource using DCRA

az monitor data-collection rule association create --resource-group myResourceGroup -n eventHubDCRAssociation  --rule-id "<DCR resourceid>" --resource "<AKS cluster ID>"

Once the data is routed to Event Hub, you can then integrate to external tools such as Splunk 

Portal- Log Analytics as destination

1. In the DCR Creation Wizard, click the information wizard that specifies creating DCRs for Platform metrics

    

2. Specify the properties for DCR as shown below (you can ignore the managed identity configuration option as it is not required for Log Analytics as a destination)
3. Select the AKS resource(s) to export platform metrics. Note that you can select multiple resources across subscriptions here (without any region restrictions for Log Analytics as a destination)

    

4. In the next "Collect and Deliver" step, click on "Add new dataflow" button. In the side panel, you will see that the "Data source type" and "Resource types" are already populated to Platform metrics and Kubernetes service. If you wish to add more resource types in the same DCR, make sure to either add those resources in the step 3 above - or you can opt not to include any resources in the step 3 above and include resources after DCR creation (which will be described in the steps later). 

   Click "Next: Destinations" in the side panel to add a destination Log Analytics workspace.

   

    

5. Select the "Azure Monitor Logs" destination type, and it can be in any accessible subscription, as long as the Log Analytics region is same as the DCR region. Click on "Save" in the side panel to add this workflow.

    

6. You can optionally add tags, and then click on "Review+Create" to create you DCR and start platform metrics export.
7. In the DCR, you can always associate more resource types and resources to a single DCR. Please note that there is only one destination allowed per DCR for platform metrics export

    

Portal- Event Hubs as destination

1. In the DCR creation wizard, make sure to select the "Enabled Managed Identity" checkbox. You can choose either System Assigned or User Assigned to enable export to Event Hub.

    

2. Add the resources as described in the Log Analytics destination export section above. Please note that the resource(s) must be in the same region as the DCR for Event Hubs export.
3. In the "Collect and Deliver" tab, in the Destinations tab of "Add new dataflow", make sure to select the appropriate Event Hub 

    

4. You can optionally add tags, and then click on "Review+Create" to create you DCR and start platform metrics export.
5. In the DCR, you can always associate more resource types and resources to a single DCR. Please note that there is only one destination allowed per DCR for platform metrics export

Step 3: Verify the Export

• For Log Analytics, navigate to the AzureMetricsV2 table in your workspace to view the exported metrics.

• For Event Hub, set up a consumer application or use Azure Stream Analytics to verify incoming metrics.

Summary 

Platform metrics are a powerful feature for monitoring AKS clusters and their workloads. By leveraging Azure Monitor’s Data Collection Rules, you can seamlessly export metrics to destinations like Log Analytics, Event Hub, and Storage Accounts enabling advanced analysis and integration. Start using these tools today to gain deeper insights into your AKS clusters!