Use Azure tags in Azure Kubernetes Service (AKS)

paolosalvatori · ‎Aug 29 2022

As documented in Use Azure tags in Azure Kubernetes Service (AKS), you can use Azure tags on an AKS cluster to associate its related resources to a given workload or tenant. For some resources, such as a managed data disk created via a persistent volume claim or an Azure Public IP created by a public Kubernetes service, you can also use Kubernetes manifests to set Azure tags. Azure tags are a helpful mechanism to track resource usage and charge back their costs to separate tenants or business units within an organization. This article explains how to set Azure tags for AKS clusters and related resources.

Azure Tags

Tags are metadata elements that you apply to your Azure resources. They're key-value pairs that help you identify resources based on settings relevant to your organization. If you want to track the deployment environment for your resources, add a key named Environment. To identify the resources deployed to production, give them a value of Production. Fully formed, the key-value pair becomes Environment = Production. You can apply tags to your Azure resources, resource groups, and subscriptions. For more information on Azure Tags, see Use tags to organize your Azure resources and management hierarchy.

You can use Azure Policy to create tagging rules and conventions for Azure resources, including AKS clusters or its resources. By creating a policy, you avoid the scenario of resources being deployed to your subscription that don't have the expected tags for your organization. Instead of manually applying tags or searching for resources that aren't compliant, you create a policy that automatically applies the needed tags during deployment. Tags can also now be applied to existing resources with the new Modify effect and a remediation task. For more information, see Assign policy definitions for tag compliance.

How tags work in AKS

Before diving into how using Azure tags in AKS, let's see what happens when you set and update Azure tags with AKS clusters and their related resources:

Tags set on an AKS cluster apply to all resources related to the cluster, but not the node pools and related virtual machine scale sets (VMSS) in the node resource group. This operation overwrites the values of existing keys.
Tags set on a node pool apply only to resources related to that node pool, mainly the virtual machine scale set (VMSS) used by the node pool. This operation overwrites the values of existing keys. Resources outside that node pool, including resources for the rest of the cluster and other node pools, are unaffected.
Azure public IPs, Azure files, and managed disks can have tags set by Kubernetes through a YAML Kubernetes manifest. Azure tags set in this way will maintain the Kubernetes values, even if you update them later using another method. When Azure public IPs, Azure files, or managed disks are removed through Kubernetes, any tags set by Kubernetes are removed. Tags on those resources that aren't tracked by Kubernetes remain unaffected.

Prerequisites

The Azure CLI version 2.0.59 or later, installed and configured. To find your version, run az --version. If you need to install it or update your version, see Install Azure CLI.
Kubernetes version 1.20 or later, installed.

Limitations

Azure tags have keys that are case-insensitive for operations, such as when you're retrieving a tag by searching the key. In this case, a tag with the specified key will be updated or retrieved regardless of the casing. Tag values are case-sensitive.
In AKS, if multiple tags are set with identical keys but different casings, the tags are used in alphabetical order. For example, {"Key1": "val1", "kEy1": "val2", "key1": "val3"} results in Key1 and val1 being set.
For shared resources, tags aren't able to determine the split in resource usage on their own.

Add tags to the cluster

When you create or update an AKS cluster with the --tags parameter, the following are assigned the Azure tags that you've specified:

The AKS cluster
The route table that's associated with the cluster
The public IP that's associated with the cluster
The load balancer that's associated with the cluster
The network security group that's associated with the cluster
The virtual network that's associated with the cluster
The AKS managed kubelet managed identity associated with the cluster
The AKS managed addon managed identity associated with the cluster
The private DNS zone associated with the private cluster
The private endpoint associated with the private cluster

NOTE: Azure Private DNS only supports 15 tags. tag resources.

To create a cluster and assign Azure tags, you can run az aks create with the --tags parameter, as shown in the following command. Running the command creates a myAKSCluster in the myResourceGroup with the tags dept=IT and costcenter=9999.

az aks create \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --tags dept=IT costcenter=9999 \
    --generate-ssh-keys

NOTE: To set tags on the initial node pool, the node resource group, the virtual machine scale set, and each virtual machine scale set instance that's associated with the initial node pool, also set the --nodepool-tags parameter.

az aks create \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --tags dept=IT costcenter=9999 \
    --nodepool-tags dept=IT costcenter=9999 \
    --generate-ssh-keys

IMPORTANT: If you're using existing resources when you're creating a new cluster, such as an IP address or route table, az aks create overwrites the set of tags. If you delete that cluster later, any tags set by the cluster will be removed.

You can use the az aks show command to verify that the tags have been properly applied to the cluster and related resources. The cluster tags for myAKSCluster are shown in the following example:

$ az aks show -g myResourceGroup -n myAKSCluster --query '[tags]'
{
  "clusterTags": {
    "costcenter": "9999",
    "dept": "IT"
  }
}

To update the tags on an existing cluster, you can run the az aks update command with the --tags parameter. Running the command updates the myAKSCluster with the tags team=alpha and costcenter=1234.

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --tags team=alpha costcenter=1234

You can use the az aks show command to verify that the tags have been applied to the cluster. For example:

$ az aks show -g myResourceGroup -n myAKSCluster --query '[tags]'
{
  "clusterTags": {
    "costcenter": "1234",
    "team": "alpha"
  }
}

IMPORTANT: Setting tags on a cluster by using az aks update overwrites the set of tags. For example, if your cluster has the tags dept=IT and costcenter=9999 and you use az aks update with the tags team=alpha and costcenter=1234, the new list of tags would be team=alpha and costcenter=1234.

In the following example, the businessUnit=FastTrack for Azure and supportedBy=Paolo Salvatori tags of the AKS cluster were created via Azure Policy, while the IaC=Bicep and workload=tags tags were created at creation time via Bicep.

Adding tags to node pools

You can apply one or more Azure tags to a new or existing node pool in your AKS cluster. Tags applied to a node pool are applied to each node within the node pool and are persisted through upgrades. Tags are also applied to new nodes that are added to a node pool during scale-out operations. Adding a tag can help with policy tracking or cost estimation tasks.

When you create or update a node pool with the --tags parameter, the tags that you specify are assigned to the following resources:

The node pool
The node resource group
The virtual machine scale set and each virtual machine scale set instance that's associated with the node pool

To create a node pool with an Azure tag, run az aks nodepool add with the --tags parameter. Running the following command creates a contosoNodePool node pool with the tags tenant=contoso and costcenter=5555 in the myAKSCluster.

az aks nodepool add \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name contosoNodePool \
    --node-count 1 \
    --tags tenant=contoso costcenter=5555 \
    --no-wait

You can use the az aks show command to verify that the tags have been applied to the contosoNodePool node pool.

$ az aks show -g myResourceGroup -n myAKSCluster --query 'agentPoolProfiles[].{nodepoolName:name,tags:tags}'
[
  {
    "nodepoolName": "nodepool1",
    "tags": null
  },
  {
    "nodepoolName": "contosoNodePool",
    "tags": {
      "tenant": "contoso",
      "costcenter": "5555"
    }
  }
]

To update a node pool with an Azure tag, you can run the az aks nodepool update command with the --tags parameter. Running the following command updates the contosoNodePool node pool with the tags appversion=1.0.0 and costcenter=4444 in the myAKSCluster, which already has the tags tenant=contoso and costcenter=5555.

az aks nodepool update \
    --resource-group myResourceGroup \
    --cluster-name myAKSCluster \
    --name contosoNodePool \
    --tags appversion=1.0.0 costcenter=4444 \
    --no-wait

IMPORTANT: Setting tags on a node pool by using the az aks nodepool update command overwrites the set of tags. For example, if your node pool has the tags tenant=contoso and costcenter=5555, and you use the az aks nodepool update command with the tags appversion=1.0.0 and costcenter=4444, the new list of tags would be appversion=1.0.0 and costcenter=4444.

You can use the az aks show command to verify that the tags have been appropriately updated on the node pool.

$ az aks show -g myResourceGroup -n myAKSCluster --query 'agentPoolProfiles[].{nodepoolName:name,tags:tags}'
[
  {
    "nodepoolName": "nodepool1",
    "tags": null
  },
  {
    "nodepoolName": "contosoNodePool",
    "tags": {
      "appversion": "1.0.0",
      "costcenter": "4444"
    }
  }
]

In the following example, the osDiskType=ephemeral and osType=Windows2022 tags of the node pool VMSS were created using the az aks nodepool create command with the --tags parameter at provisioning time.

Add tags by using Kubernetes manifests

You can apply Azure tags to Azure public IPs, managed disks, and Azure files by using a Kubernetes YAML manifest.

Azure Public IPs

The service.beta.kubernetes.io/azure-pip-tags annotation can be to determines what Azure tags should be applied to the Azure public IP used by a Kubernetes service. The supported format is a=b,c=d,.... After updated, the old user-assigned tags would not be replaced by the new ones.

For example, the following YAML file creates a Kubernetes deployment that uses NGINX container image, and exposes it via a public LoadBalancer Kubernetes service. The service definition uses the costcenter=1234,tenant=Contoso annotation to set Azure tags on the Azure public IP resource created by the cloud controller manager.

---
apiVersion: apps/v1 
kind: Deployment
metadata:
  name: nginx
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nginx
  replicas: 3 
  template: 
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "128Mi"
            cpu: "250m"
          limits:
            memory: "256Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /
            port: 80
          failureThreshold: 1
          initialDelaySeconds: 1
          periodSeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /
            port: 80
          failureThreshold: 1
          initialDelaySeconds: 1
          periodSeconds: 30
          timeoutSeconds: 5
        startupProbe:
          httpGet:
            path: /
            port: 80
          failureThreshold: 1
          initialDelaySeconds: 1
          periodSeconds: 30
          timeoutSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    service.beta.kubernetes.io/azure-pip-tags: costcenter=1234
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

In the following example, the costcenter=1234 tag of the Azure Public IP in the node resource group of the AKS cluster was created by the service.beta.kubernetes.io/azure-pip-tag annotation in the YAML manifest used to deploy the Kubernetes service.

IMPORTANT: Setting tags on files, disks, and public IPs by using Kubernetes updates the set of tags. For example, if your disk has the tags dept=IT and costcenter=5555, and you use Kubernetes to set the tags team=beta and costcenter=3333, the new list of tags would be dept=IT, team=beta, and costcenter=3333. Any updates that you make to tags through Kubernetes will retain the value that's set through Kubernetes. For example, if your disk has tags dept=IT and costcenter=5555 set by Kubernetes, and you use the portal to set the tags team=beta and costcenter=3333, the new list of tags would be dept=IT, team=beta, and costcenter=5555. If you then remove the disk through Kubernetes, the disk would have the tag team=beta.

Azure Disks and Files

You can apply Azure tags to Azure managed disks and Azure files by using a Kubernetes YAML manifest created via a persistent volume claim PVC. Today you cannot specify metadata and, in particular, tags in a PVC that are passed down to a CSI driver. Today, the only way to tag managed resources dynamically created by a PVC on a cloud platform is using the tags parameter of the Azure Disks CSI driver:

Create a custom storage class and specify tags in the parameters:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: contoso-managed-csi-premium
provisioner: disk.csi.azure.com
parameters:
  skuname: Premium_LRS
  tags: costcenter=1234,tenant=Contoso
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Create a PVC based on the custom storage class:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: contoso-managed-csi-premium
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: contoso-managed-csi-premium
  resources:
    requests:
      storage: 50Gi

Use the PVC in your deployment definition

In a multi-tenant environment, this approach can lead to two kinds of problems:

The proliferation of custom storage classes, one for each tenant. If you share an AKS cluster across hundreds of tenants each using PVs, and you want to use this technique to tag managed disks, you can quickly end up with hundreds of custom storage classes
Storage classes are not namespaced resources, so this can lead to issues when trying to segregate or restrict security using Kubernetes RBAC or Azure AD integrated RBAC

You can use the following workaround to avoid the proliferation of storage classes:

You can create a custom storage class with the proper tags for a tenant
Based on the custom storage class, create one or more PVCs, which dynamically create related managed resources on the current cloud platform.
Delete the custom storage class.

In the following example, the costcenter=1234 and tenant=Contoso tags of the Azure managed disk in the node resource group of the AKS cluster were created by the tags parameter in the YAML manifest used to create the contoso-managed-csi-premium storage class used by the persistent volume claim (PVC).

Products (65)

Special Topics (44)

Video Hub (978)

Most Active Hubs