Beyond iptables: Scaling AKS Networking with nftables and Project Calico
Author : Reza Ramezanpour Senior Developer Advocate @ Tigera For most of Kubernetes’ life, service networking has relied on iptables. It was practical, widely available on all Linux distributions, and good enough for clusters with modest scale with predictable workloads. However, as cloud providers are increasing the pod limits for clusters and deployments are taking advantage of their multi-regional, highly available infrastructure the need for running more workloads shines a light on an old design problem. Today’s production clusters by taking advantage of cloud provider infrastructure run thousands of services that may experience constant endpoint churn, and must satisfy strict requirements around performance, security, and compliance. In this case the old iptables model which we will discuss here is inefficient and it was never designed with these environments in mind. This is why the Kubernetes community started to move away from iptables toward nftables, and the upstream Kubernetes graduated the kube-proxy support for nftables mode to stable in the v1.33 release which was followed by Tigera’s free and open source Project Calico v3.29. Last year, Microsoft’s decision to support kube-proxy in nftables mode reflects a broader reality, the traditional iptables model is becoming a structural bottleneck for modern Kubernetes platforms, including managed environments such as AKS. In this blog we are going to use Microsoft’s latest kube-proxy preview features to create a Bring Your Own CNI cluster configured for nftables, and use Project Calico to establish networking and security on it. We will also look at why a shift from iptabels to nftables is recommended and should happen sooner than later. The Hidden Tax of iptables You might be using iptables in a large cluster today and thinking, everything works, why change it? But that’s how the problem usually starts. The iptables limitations don’t show up as a failure. They show up gradually: higher CPU usage, slower updates, harder debugging. Then one day, they’re unavoidable. Think of it like this, You have two security guards checking people entering a stadium. Both have the same guest list. The first guard (iptables) holds the list on paper. For every person, he starts at the top of the list, scans name by name, finds a match, then goes back to the top for the next person. Each time a new person joins the line, he needs to re-write the whole/or part of the list again and start from scratch. The second guard (nftables) has the list in a searchable database. He types the name, gets an instant answer, and moves on. Both let the right people in. However, one of these security guards slows down as the crowd grows, and at some point it will just give up scanning for new people. That’s the hidden cost of sticking to iptables, lookup time grows with the number of services that you have in your cluster, and updates get more expensive as the system scales. Creating Your First AKS Managed Nftables Ready Cluster Microsoft Azure Kubernetes Service (AKS) now includes preview support for running kube-proxy in nftables mode. This preview is opt-in and not enabled by default, because it’s still under development. To use it in AKS you must explicitly opt into the preview : az extension add --name aks-preview az extension update --name aks-preview Next, you have to register the kubeproxy custom configuration preview feature. az feature register --namespace "Microsoft.ContainerService" --name "KubeProxyConfigurationPreview" Once the feature is registered (this may take a few minutes to complete), you can customize the kube-proxy deployment. To enable nftables mode, define a minimal kube-proxy configuration like the following and save it as kube-proxy.json, which will be referenced when creating the AKS cluster in the next step: { "enabled": true, "mode": "NFTABLES" } With that config saved in a file (e.g., kube-proxy.json), issue the following command and create your cluster: az group create --name nftables-demo --location canadacentral az aks create \ --resource-group nftables-demo \ --name calico-nftables \ --kube-proxy-config kube-proxy.json \ --network-plugin none \ --pod-cidr "10.10.0.0/16" \ --generate-ssh-keys \ --location canadacentral \ --node-count 2 \ --vm-size Standard_A8m_v2 After the cluster is created, verify that kube-proxy is running in nftables mode by issuing the following command: kubectl logs -n kube-system ds/kube-proxy | egrep "Proxier" Output: I0129 21:09:06.613047 1 server_linux.go:259] "Using nftables Proxier" After you’ve confirmed nftables is active, install Calico. kubectl create -f https://docs.tigera.io/calico/latest/manifests/tigera-operator.yaml Next, install the cluster installation resource which instructs Tigera operator about the Calico features that should be enabled in your environment. kubectl create -f https://gist.githubusercontent.com/frozenprocess/6932bae9e33468b53696f1a901f2aa76/raw/d74ba8ef30a5657e7474a1fe22a650f87c08ecaf/installation.yaml If you already installed Calico and are using any of its dataplanes such as iptables, IPVS or Calico eBPF mode simply use the following command to only change the dataplane to nftables. kubectl patch installation default --type=merge -p='{"spec":{"calicoNetwork":{"linuxDataplane":"Nftables"}}}' A Tale of Two Backends - How Services Works In Nftables Now that we have the analogy and an environment, let's look at the actual differences between these two backends. For that, first deploy an application on your cluster. This demo app creates a deployment with 1 replica and a service with the type of LoadBalancer. NAME READY STATUS RESTARTS AGE pod/anp-demo-app-5669879946-rb28v 1/1 Running 0 91s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/container-service LoadBalancer 10.0.92.204 40.82.188.95 80:31907/TCP 90s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/anp-demo-app 1/1 1 1 91s NAME DESIRED CURRENT READY AGE replicaset.apps/anp-demo-app-5669879946 1 1 1 92s 1. Service Maps (The list in our analogy) In nftables mode, kube-proxy doesn't just create a long list of linear rules. Instead, it leverages maps. Think of a map as a high-speed lookup table. Our first step is to see how Kubernetes stores the mapping between a Service's Public/Cluster IP and its internal handling logic. kubectl exec -itn calico-system ds/calico-node -c calico-node -- nft list map ip kube-proxy service-ips Output: type ipv4_addr . inet_proto . inet_service : verdict elements = { 40.82.188.95 . tcp . 80 : goto external-SG2JDYAZ-web-demo/container-service/tcp/ } What’s happening here? The output shows that any traffic hitting the External IP 40.82.188.95 on port 80 is immediately told to goto a specific chain. Unlike iptables, which has to evaluate rules one by one, nftables jumps straight to the relevant logic. 2. The Dispatcher Chain Once a packet matches a service in the map, it is handed off to a specific chain. This chain acts as a dispatcher, preparing the packet for the "External" world. A map is a simple way to figure out what to do if a flow is matched, here the idea is if source IP, protocol, and destination port matches with a flow the verdict should arrive by using the external-SG2JDYAZ-web-demo/container-service/tcp/ chain entry. Note that an external facing service doesn’t have access to the internal pods, so now nftables need to NAT the communication and send it to the pod. kubectl exec -itn calico-system ds/calico-node -c calico-node -- nft list chain ip kube-proxy "external-SG2JDYAZ-web-demo/container-service/tcp/" Note: Here, nftables marks the packet for Masquerade (SNAT) to ensure the return traffic flows back through the gateway, and then forwards it to the primary service chain. Output: table ip kube-proxy { chain external-SG2JDYAZ-web-demo/container-service/tcp/ { jump mark-for-masquerade goto service-SG2JDYAZ-web-demo/container-service/tcp/ } } 3. Load Balancing via Verdict Maps This is where the magic happens. Kubernetes needs to decide which Pod should receive the traffic. In nftables, this is handled using a vmap (verdict map) combined with a random number generator. kubectl exec -itn calico-system ds/calico-node -c calico-node -- nft list chain ip kube-proxy "service-SG2JDYAZ-web-demo/container-service/tcp/" Notice the numgen random mod 1. Since we currently have only one replica, the logic is simple: 100% of traffic goes to the single available endpoint. table ip kube-proxy { chain service-SG2JDYAZ-web-demo/container-service/tcp/ { ip daddr 10.0.92.204 tcp dport 80 ip saddr != 10.10.0.0/16 jump mark-for-masquerade numgen random mod 1 vmap { 0 : goto endpoint-KEOBOJ73-web-demo/container-service/tcp/__10.10.219.68/80 } } } 4. The Final Destination: DNAT to Pod IP The last stop in the nftables journey is the endpoint chain. This is where the packet’s destination address is actually changed from the Service IP to the Pod’s internal IP. kubectl exec -itn calico-system ds/calico-node -c calico-node -- nft list chain ip kube-proxy "endpoint-KEOBOJ73-web-demo/container-service/tcp/__10.10.219.68/80" The rule dnat to 10.10.219.68:80 rewritten the packet header. The packet is now ready to be routed directly to the container. Output: table ip kube-proxy { chain endpoint-KEOBOJ73-web-demo/container-service/tcp/__10.10.219.68/80 { ip saddr 10.10.219.68 jump mark-for-masquerade meta l4proto tcp dnat to 10.10.219.68:80 } } Scaling Deployments What happens when we scale? Let’s increase our replicas to 10 to see how nftables updates its load-balancing table dynamically. kubectl patch deployment -n web-demo --type=merge anp-demo-app -p='{"spec":{"replicas":10}}' What changed? The numgen (number generator) will now be mod 10, and the vmap will contain 10 different target chains (one for each Pod). This is significantly more efficient than the "probability" flags used in legacy iptables. You can verify this by re-issuing the command from step 3. Note: The actual journey doesn’t start at the map; it begins at the nat-prerouting chain, moves to the services chain, and is finally evaluated against the service-ips map. If you like to learn how iptables services works take a look at this free course. It’s end of an era As more and more Linux distributions, companies, and projects try to shift from iptables to nftables the cost of maintaining an iptables environment will increase for its users. The shift from iptables to nftables isn't just a minor version bump; it is a fundamental architectural upgrade for the modern cloud-native solution. As we examined, the "hidden tax" of linear rule evaluation in iptables becomes a bottleneck while your Kubernetes clusters scale into thousands of services and endpoints. This is why the industry is looking for solutions that are more tuned for the cloud native environment. By leveraging the new nftables mode in kube-proxy, that is now in a preview mode in Microsoft AKS and added to Kubernetes from v1.33 and Project Calico v3.29+, platform engineers can finally move away from iptables. Footnote - Observability and troubleshooting In this blog, we explored the inner workings of nftables using the command line, a powerful way to understand the underlying mechanics of packet filtering. However, manual terminal work isn't the only way to troubleshoot networking in Kubernetes. In a production environment, you often need a higher-level view to observe traffic flows and policy impacts across hundreds of pods. Several open-source projects provide "browser-based" observability, allowing you to debug network flows visually rather than parsing text logs. Network Observability Tools Calico Whisker: A dedicated observability tool for Calico (v3.30+) that visualizes real-time network flows. It is particularly useful for debugging Staged Network Policies, as it shows you exactly which flows would be denied before you actually enforce the rules. Learn more about Calico Whisker here. kubectl port-forward -n calico-system svc/whisker 8081:8081 Microsoft Retina: A CNI-agnostic, eBPF-based platform that provides deep insights into packet drops, DNS health, and TCP metrics. It works across different cloud providers and operating systems (including Windows). Learn more about Microsoft Retina here.Exploring Traffic Manager Integration for External DNS
When you deploy externally accessible applications into Kubernetes, there is usually a requirement for creating some DNS records pointing to these applications, to allow your users to resolve them. Rather than manually creating these DNS records, there are tools that will do this work for you, one of which is External DNS. External DNS can watch your Kubernetes resource configuration for specific annotations and then use these to create DNS records in your DNS zone. It has integrations with many DNS providers, including Azure DNS. This solution works well and is in use by many customers using AKS and Azure DNS. Where we hit a limitation with External DNS in Azure is in scenarios where we are need to distribute traffic across multiple clusters for load balancing and global distribution. There are a few ways to achieve this global distribution in Azure, one way is to use Azure Traffic Manger. Unlike something like Azure Front Door, Azure Traffic Manager is a DNS based global load balancer. Traffic is directed to your different AKS clusters based on DNS resolution using Traffic Manager. When a user queries your Traffic Manager CNAME, they hit the Traffic Manager DNS servers, which then return a DNS record for a specific cluster based on your load balancing configuration. Traffic Manager can then introduce advanced load balancing scenarios such as: Geographic routing - direct users to the nearest endpoint based on their location Weighted distribution - split traffic percentages (e.g., 80% to one region, 20% to another) Priority-based failover - automatic disaster recovery with primary/backup regions Performance-based routing - direct users to the endpoint with lowest latency So, given Azure Traffic Manager is a essentially a DNS service, it would be good if we could manage it using External DNS. We already use External DNS to create DNS records for each of our individual clusters, so why not use it to also create the Traffic Manager DNS configuration to load balance across them. This would also provides the added benefit of allowing us to change our load balancing strategy or configuration by making changes to our External DNS annotations. Unfortunately, an External DNS integration for Traffic Manager doesn't currently exist. In this post, I'll walk through a proof-of-concept for a provider I built to explore whether this integration is viable, and share what I learned along the way. External DNS Webhook Provider External DNS has two types of integrations. The first, and most common for "official" providers are "In-tree" providers, which are the integrations that have been created by External DNS contributors and sit within the central External DNS repository. This includes the Azure DNS provider. The second type of provider is the WebHook provider, which allows for external contributors to easily create their own providers without the need to submit them to the core External DNS repo and go through that release process. We are going to use the WebHook provider External DNS has begun the process of phasing out "In-tree" providers and replacing with WebHook ones. No new "In-tree" providers will be accepted. By using the WebHook provider mechanism, I was able to create a proof of concept Azure Traffic Manager provider that does the following: Watches Kubernetes Services for Traffic Manager annotations Automatically creates and manages Traffic Manager profiles and endpoints Syncs state between your Kubernetes clusters and Azure Enables annotation-driven configuration - no manual Traffic Manager management needed Handles duplication so that when your second cluster attempts to create a Traffic Manager record it adds an endpoint to the existing instance Works alongside the standard External DNS Azure provider for complete DNS automation Here's what the architecture looks like: Example Configuration: Multi-Region Deployment Let me walk you through a practical example using this provider to see how it works. We will deploy an application across two Azure regions with weighted traffic distribution and automatic failover. This example assumes you have built and deployed the PoC provider as discussed below. Step 1: Deploy Your Application You deploy your application to both East US and West US AKS clusters. Each deployment is a standard Kubernetes Service with LoadBalancer type. We then apply annotations that tell our External DNS provider how to create and configure the Traffic Manger resource. These annotations are defined as part of our plugin. apiVersion: v1 kind: Service metadata: name: my-app-east namespace: production annotations: # Standard External DNS annotation external-dns.alpha.kubernetes.io/hostname: my-app-east.example.com # Enable Traffic Manager integration external-dns.alpha.kubernetes.io/webhook-traffic-manager-enabled: "true" external-dns.alpha.kubernetes.io/webhook-traffic-manager-resource-group: "my-tm-rg" external-dns.alpha.kubernetes.io/webhook-traffic-manager-profile-name: "my-app-global" # Weighted routing configuration external-dns.alpha.kubernetes.io/webhook-traffic-manager-weight: "70" external-dns.alpha.kubernetes.io/webhook-traffic-manager-endpoint-name: "east-us" external-dns.alpha.kubernetes.io/webhook-traffic-manager-endpoint-location: "eastus" # Health check configuration external-dns.alpha.kubernetes.io/webhook-traffic-manager-monitor-path: "/health" external-dns.alpha.kubernetes.io/webhook-traffic-manager-monitor-protocol: "HTTPS" spec: type: LoadBalancer ports: - port: 443 targetPort: 8080 selector: app: my-app The West US deployment has identical annotations, except: Weight: "30" (sending 30% of traffic here initially) Endpoint name: "west-us" Location: "westus" Step 2: Automatic Resource Creation When you deploy these Services, here's what happens automatically: Azure Load Balancer provisions and assigns public IPs to each Service External DNS (Azure provider) creates A records: my-app-east.example.com → East US LB IP my-app-west.example.com → West US LB IP External DNS (Webhook provider) creates a Traffic Manager profile named my-app-global Webhook adds endpoints to the profile: East endpoint (weight: 70, target: my-app-east.example.com) West endpoint (weight: 30, target: my-app-west.example.com) External DNS (Azure provider) creates a CNAME: my-app.example.com → Traffic Manager FQDN Now when users access my-app.example.com, Traffic Manager routes 70% of traffic to East US and 30% to West US, with automatic health checking on both endpoints. Step 3: Gradual Traffic Migration Want to shift more traffic to West US? Just update the annotations: kubectl annotate service my-app-east \ external-dns.alpha.kubernetes.io/webhook-traffic-manager-weight="50" \ --overwrite kubectl annotate service my-app-west \ external-dns.alpha.kubernetes.io/webhook-traffic-manager-weight="50" \ --overwrite Within minutes, traffic distribution automatically adjusts to 50/50. This enables: Blue-green deployments - test new versions with small traffic percentages Canary releases - gradually increase traffic to new deployments Geographic optimisation - adjust weights based on user distribution Step 4: Automatic Failover If the East US cluster becomes unhealthy, Traffic Manager's health checks detect this and automatically fail over 100% of traffic to West US, no manual intervention required. How It Works The webhook implements External DNS's webhook provider protocol: 1. Negotiate Capabilities External DNS queries the webhook to determine supported features and versions. 2. Adjust Endpoints External DNS sends all discovered endpoints to the webhook. The webhook: Filters for Services with Traffic Manager annotations Validates configuration Enriches endpoints with metadata Returns only Traffic Manager-enabled endpoints 3. Record State External DNS queries the webhook for current Traffic Manager state. The webhook: Syncs profiles from Azure Converts to External DNS endpoint format Returns CNAME records pointing to Traffic Manager FQDNs 4. Apply Changes External DNS sends CREATE/UPDATE/DELETE operations. The webhook: Creates Traffic Manager profiles as needed Adds/updates/removes endpoints Configures health monitoring Updates in-memory state cache The webhook uses Azure SDK for Go to interact with the Traffic Manager API and maintains an in-memory cache of profile state to optimise performance and reduce API calls. Proof of Concept 🛑 Important: This is a Proof of Concept This project is provided as example code to demonstrate the integration pattern between External DNS and Traffic Manager. It is not a supported product and comes with no SLAs, warranties, or commitments. The code is published to help you understand how to build this type of integration. If you decide to implement something similar for your production environment, you should treat this as inspiration and build your own solution that you can properly test, secure, and maintain. Think of this as a blueprint, not a finished product. With that caveat out of the way, if you want to experiment with this approach, the PoC is available on GitHub: github.com/sam-cogan/external-dns-traffic-manager. The readme file containers detailed instructions on how to deploy the PoC into a single and multi-cluster environment, along with demo applications to try it out. Use Cases This integration unlocks several powerful scenarios: Multi-Region High Availability - Deploy your application across multiple Azure regions with automatic DNS-based load balancing and health-based failover. No additional load balancers or gateways required. Blue-Green Deployments - Deploy a new version alongside your current version, send 5% of traffic to test, gradually increase, and roll back instantly if issues arise by changing annotations. Geographic Distribution - Route European users to your Europe region and US users to your US region automatically using Traffic Manager's geographic routing with the same annotation-based approach. Disaster Recovery - Configure priority-based routing with your primary region at priority 1 and DR region at priority 2. Traffic automatically fails over when health checks fail. Cost Optimisation - Use weighted routing to balance traffic across regions based on capacity and costs. Send more traffic to regions where you have reserved capacity or lower egress costs. Considerations and Future Work This is a proof of concept and should be thoroughly tested before production use. Some areas for improvement: Current Limitations In-memory state only - no persistent storage (restarts require resync) Basic error handling - needs more robust retry logic Limited observability - could use more metrics and events Manual CRD cleanup - DNSEndpoint CRDs need manual cleanup when switching providers Potential Enhancements Support for more endpoint types - currently focuses on ExternalEndpoints Advanced health check configuration - custom intervals, timeouts, and thresholds Metric-based routing decisions - integrate with Azure Monitor for intelligent routing GitOps integration - Flux/ArgoCD examples and best practices Helm chart - simplified deployment If you try this out or have ideas for improvements, please open an issue or PR on GitHub. Wrapping Up This proof of concept shows that External DNS and Traffic Manager can work together nicely. Since Traffic Manager is really just an advanced DNS service, bringing it into External DNS's annotation-driven workflow makes a lot of sense. You get the same declarative approach for both basic DNS records and sophisticated traffic routing. While this isn't production-ready code (and you shouldn't use it as-is), it demonstrates a viable pattern. If you're dealing with multi-region Kubernetes deployments and need intelligent DNS-based routing, this might give you some ideas for building your own solution. The code is out there for you to learn from, break, and hopefully improve upon. If you build something based on this or have feedback on the approach, I'd be interested to hear about it. Resources GitHub Repository: github.com/sam-cogan/external-dns-traffic-manager External DNS Documentation: kubernetes-sigs.github.io/external-dns Azure Traffic Manager: learn.microsoft.com/azure/traffic-manager Webhook Provider Guide: External DNS Webhook Tutorial
Announcing Conversational Diagnostics (Preview) on Azure Kubernetes Service
We are pleased to announce Conversational Diagnostics (Preview), a new feature in Azure Kubernetes Service (AKS) that leverages the powerful capabilities of OpenAI to take your problem-solving to the next level. With this feature, you can engage in a conversational dialogue to accurately identify the issue with your AKS clusters, and receive a clear set of solutions, documentation, and diagnostics to help resolve the issue. How to access Conversational Diagnostics (Preview) Navigate to your AKS cluster on the Azure Portal. Select Diagnose and Solve Problems. Select AI Powered Diagnostics (preview) to open chat What is the tool capable of? Type a question or select any of the sample questions in the options. Types of questions you can ask include but not limited to: Questions regarding a specific issue that your cluster is experiencing Questions about how-to instructions Questions about best practices The AI-powered Diagnostics runs relevant diagnostic checks and provides a diagnosis of the issue and a recommended solution. Additionally, it can provide documentation addressing your questions. Once you are done with troubleshooting, you can create a summary of your troubleshooting session by responding to the chat with the message that the issue has been resolved. If you want to start a new investigation, select New Chat. How to sign up for access To get started, sign up for early access to the Conversational Diagnostics (Preview) feature in the Diagnose and Solve Problems experience. Your access request may take up to 4 weeks to complete. Once access is granted, Conversational Diagnostics (Preview) will be available for all the Azure Kubernetes Service resources on your subscription. Navigate to your AKS cluster on the Azure Portal. Select Diagnose and Solve Problems. Select Request Access in the announcement banner. (Note: Announcement banner will be available starting on Mar 18th, 2024 PST) Whether you're a seasoned developer or a newcomer to Azure Kubernetes Service, Conversational Diagnostics (Preview) provides an intuitive and efficient way to understand and tackle any issue you may encounter. You can easily access this feature whenever you need it without any extra configuration. Say goodbye to frustrating troubleshooting and hello to a smarter, more efficient way of resolving issues with AKS clusters. Preview Terms Of Use | Microsoft Azure Microsoft Privacy Statement – Microsoft privacyExciting Updates Coming to Conversational Diagnostics (Public Preview)
Last year, at Ignite 2023, we unveiled Conversational Diagnostics (Preview), a revolutionary tool integrated with AI-powered capabilities to enhance problem-solving for Windows Web Apps. This year, we're thrilled to share what’s new and forthcoming for Conversational Diagnostics (Preview). Get ready to experience a broader range of functionalities and expanded support across various Azure Products, making your troubleshooting journey even more seamless and intuitive.Simplifying Image Signing with Notary Project and Artifact Signing (GA)
Securing container images is a foundational part of protecting modern cloud‑native applications. Teams need a reliable way to ensure that the images moving through their pipelines are authentic, untampered, and produced by trusted publishers. We’re excited to share an updated approach that combines the Notary Project, the CNCF standard for signing and verifying OCI artifacts, with Artifact Signing—formerly Trusted Signing—which is now generally available as a managed signing service. The Notary Project provides an open, interoperable framework for signing and verification across container images and other OCI artifacts, while Notary Project tools like Notation and Ratify enable enforcement in CI/CD pipelines and Kubernetes environments. Artifact Signing complements this by removing the operational complexity of certificate management through short‑lived certificates, verified Azure identities, and role‑based access control, without changing the underlying standards. If you previously explored container image signing using Trusted Signing, the core workflows remain unchanged. As Artifact Signing reaches GA, customers will see updated terminology across documentation and tooling, while existing Notary Project–based integrations continue to work without disruption. Together, Notary Project and Artifact Signing make it easier for teams to adopt image signing as a scalable platform capability—helping ensure that only trusted artifacts move from build to deployment with confidence. Get started Sign container images using Notation CLI Sign container images in CI/CD pipelines Verify container images in CI/CD pipelines Verify container images in AKS Extend signing and verification to all OCI artifacts in registries Related content Simplifying Code Signing for Windows Apps: Artifact Signing (GA) Simplify Image Signing and Verification with Notary Project (preview article)Identity Bindings: A Cleaner Model for Multi‑Cluster Identity in AKS
AKS has supported assigning Azure Managed Identities to pods for some time, first through Pod Identity and then later through Workload Identity. Using these tools it is possible to give a pod an Azure Identity that it can use to interact with other Azure services - pull secrets from Key Vault, read a file from Blob Storage or write to a database. Workload Identity is the latest incarnation of this feature and significantly simplified this feature, removing the need to run additional management pods in the cluster and to have the identity injected in every node however it does have some issues of it's own. These issues are particularly evident when operating at scale and wanting to share the same Managed Identity across multiple workloads in the same cluster, or across multiple clusters. Workload Identity relies on creating a Federated Identity Credential (FIC) in Azure that defines the trust relationship between the AKS OIDC issuer and Entra ID. Each combination of Service Account and Namespace that uses the same Managed Identity requires a separate FIC, as do services running in different clusters. As your scale grows, this can start to become a problem. Each managed identity can only support up to 20 FICs. Once you hit that limit, your only option is to create another Managed Identity. This leads to the proliferation of Managed Identities that have the same permissions and do the same job, but only exist to work around this problem. In addition to the 20 FIC limit, there are some other issues with Workload Identity: Creation of the FIC is an Azure Resource creation that often needs to occur alongside Kubernetes resource creation and makes automation of app deployment harder There can be a cyclic dependency issue where the service account in Kubernetes needs to know the Identity details before the pod is created, but the FIC needs the service account and namespace details to create the OIDC binding Additional outbound rules are required to allow the AKS cluster to access the Entra ID (login.microsoftonline.com) endpoints Introducing Identity Bindings Identity Bindings are currently in preview. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. Identity Bindings introduce a cleaner, RBAC-driven approach to identity management in AKS. Instead of juggling multiple federated credentials and namespace scoping, you define bindings that link Kubernetes RBAC roles to Azure identities. Pods then request tokens via an AKS-hosted proxy, no external egress required. Key benefits: Centralised Access Control: Authorisation flows through Kubernetes RBAC. Cross-Cluster Identity Reuse: Federated credentials can be shared across namespaces and clusters. Reduced Networking Requirements: No outbound calls for token acquisition; everything stays within the cluster. Simplified IaC: Eliminates the “chicken-and-egg” problem when deploying identities alongside workloads. Identity Bindings act as the link between applications running in AKS and the Azure managed identities they need to use. Instead of every cluster or namespace requiring its own federated identity configuration, each application is authorised through an Identity Binding defined inside the cluster. The binding expresses which workloads (via service accounts and RBAC) are allowed to request tokens for a given identity. When a pod needs a token, AKS validates the request against the binding, and if it matches, the request is routed through the cluster’s identity proxy to the single Federated Identity Credential (FIC) associated with the managed identity. The FIC then exchanges the pod’s OIDC token for an Azure access token. This pattern allows multiple clusters or namespaces to share one managed identity cleanly, while keeping all workload‑level authorisation decisions inside Kubernetes. With Workload Identity, every workload using a managed identity required its own Federated Identity Credential (FIC) tied to a specific namespace and service account, and you had to repeat that for every cluster. Hitting the 20‑FIC limit often forced teams to create duplicate managed identities, and deployments had to be carefully ordered to avoid cyclic dependencies. You also needed outbound access to Entra ID for token requests. Identity Bindings significantly simplifies this. You create a single binding per cluster–identity pair, authorise workloads through RBAC, and let AKS handle token exchange internally with no external egress. There is no FIC sprawl, no need for identity duplication and less complexity in your automation. Using Identity Bindings To get started with using Identity Bindings, you need an AKS cluster and a Managed Identity created. Your Managed Identity should be granted permissions to access the Azure resources you require. The first thing you need to do is ensure the feature is registered. az feature register --namespace Microsoft.ContainerService --name IdentityBindingPreview Next, we need to do is create the identity binding. This is a one-to-one mapping between AKS cluster and Managed Identity, so only needs to be created once for each clusters/identity mapping. You provide the name of the cluster you want the binding mapped to, the full resource ID of the Managed Identity resources, and the name you want to give this binding, and this maps the two resources together. This only needs to be created once per cluster and all further administration is done via Kubernetes. az aks identity-binding create -g "<resource group name>" --cluster-name "<cluster name>" -n "<binding name>" --managed-identity-resource-id "<managed identity Azure Resource ID>" Once this has been created, we need to configure access to it inside Kubernetes. To do this we create a ClusterRole which references the Managed Identity ID. Note that this must be a ClusterRole, it cannot just be a Role. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: identity-binding-user rules: - verbs: ["use-managed-identity"] apiGroups: ["cid.wi.aks.azure.com"] resources: ["<MI_CLIENT_ID>"] Once this ClusterRole is created, we can assign it to any Namespace and Service Account combination we require, using a ClusterRoleBinding. Indentity Bindings are accessible to all Pods that use that Service Account, in that Namespace. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: <clusterrole name> roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: identity-binding-user subjects: - kind: ServiceAccount name: <service account name> namespace: <namespace of service account> Now all that is left to do is configure the Pod to use the Identity Binding, there are a two steps to this. First we need to apply the required labels and annotations to the pod to enable Identity Binding support: metadata: labels: azure.workload.identity/use: "true" annotations: azure.workload.identity/use-identity-binding: "true" Then, we need to ensure that the Pod is running using the Service Account we granted permission to use the Identity Binding. spec: serviceAccountName: <service account name> Below is an example deployment that uses Identity Bindings. apiVersion: apps/v1 kind: Deployment metadata: name: keyvault-demo namespace: identity-binding-demo spec: replicas: 1 selector: matchLabels: app: keyvault-demo template: metadata: labels: app: keyvault-demo azure.workload.identity/use: "true" annotations: azure.workload.identity/use-identity-binding: "true" spec: serviceAccountName: keyvault-demo-sa containers: - name: keyvault-demo ... Once this Pod has been created, the Identity Binding should be attached and you should then be able to use it within your application using your SDK and language of choice. You can see an example of consuming an Identity Binding in GO here . Demo App If you want to deploy a demo workload to test out your bindings, you can use the Pod definition below. This requires you to deploy a Key Vault, and grant your managed identity the "Key Vault Secret User" role on that Key Vault. You will also need to update the service principle and namespace to match your environment. apiVersion: v1 kind: Pod metadata: name: demo namespace: demo labels: azure.workload.identity/use: "true" annotations: azure.workload.identity/use-identity-binding: "true" spec: serviceAccount: demo containers: - name: azure-sdk # source code: https://github.com/Azure/azure-workload-identity/blob/feature/custom-token-endpoint/examples/identitybinding-msal-go/main.go image: ghcr.io/bahe-msft/azure-workload-identity/identitybinding-msal-go:latest-linux-amd64 env: - name: KEYVAULT_URL value: ${KEYVAULT_URL} - name: SECRET_NAME value: ${KEYVAULT_SECRET_NAME} restartPolicy: Never Once deployed, if you look at the logs, you should see that it is able to read the secret from Key Vault. kubectl logs demo -n demo I1107 20:03:42.865180 1 main.go:77] "successfully got secret" secret="Hello!" Conclusion Identity Bindings offer a much cleaner model for managing workload identities in AKS, especially once you start operating at scale. By moving authorisation decisions into Kubernetes and relying on a single Federated Identity Credential per managed identity, they avoid the FIC sprawl, cyclic dependencies, and networking requirements that made Workload Identity harder to operate in larger environments. The end result is a simpler, more predictable way to let pods acquire Azure tokens. If you’re already using Workload Identity today, Identity Bindings are a natural evolution that reduces operational friction while keeping the security properties you expect. Further Reading Identity Bindings Overview Setup Identity Bindings
Find the Alerts You Didn't Know You Were Missing with Azure SRE Agent
I had 6 alert rules. CPU. Memory. Pod restarts. Container errors. OOMKilled. Job failures. I thought I was covered. Then my app went down. I kept refreshing the Azure portal, waiting for an alert. Nothing. That's when it hit me: my alerts were working perfectly. They just weren't designed for this failure mode. Sound familiar? The Problem Every Developer Knows If you're a developer or DevOps engineer, you've been here: a customer reports an issue, you scramble to check your monitoring, and then you realize you don't have the right alerts set up. By the time you find out, it's already too late. You set up what seems like reasonable alerting and assume you're covered. But real-world failures are sneaky. They slip through the cracks of your carefully planned thresholds. My Setup: AKS with Redis I love to vibe code apps using GitHub Copilot Agent mode with Claude Opus 4.5. It's fast, it understands context, and it lets me focus on building rather than boilerplate. For this project, I built a simple journal entry app: AKS cluster hosting the web API Azure Cache for Redis storing journal data Azure Monitor alerts for CPU, memory, pod restarts, container errors, OOMKilled, and job failures Seemed solid. What could go wrong? The Scenario: Redis Password Rotation Here's something that happens constantly in enterprise environments: the security team rotates passwords. It's best practice. It's in the compliance checklist. And it breaks things when apps don't pick up the new credentials. I simulated exactly this. The pods came back up. But they couldn't connect to Redis (as expected). The readiness probes started failing. The LoadBalancer had no healthy backends. The endpoint timed out. And not a single alert fired. Using SRE Agent to Find the Alert Gaps Instead of manually auditing every alert rule and trying to figure out what I missed, I turned to Azure SRE Agent. I asked it a simple question: "My endpoint is timing out. What alerts do I have, and why didn't any of them fire?" Within minutes, it had diagnosed the problem. Here's what it found: My Existing Alerts Why They Didn't Fire High CPU/Memory No resource pressure,just auth failures Pod Restarts Pods weren't restarting, just unhealthy Container Errors App logs weren't being written OOMKilled No memory issues Job Failures No K8s jobs involved The gaps SRE Agent identified: ❌ No synthetic URL availability test ❌ No readiness/liveness probe failure alerts ❌ No "pods not ready" alerts scoped to my namespace ❌ No Redis connection error detection ❌ No ingress 5xx/timeout spike alerts ❌ No per-pod resource alerts (only node-level) SRE Agent didn't just tell me what was wrong, it created a GitHub issue with : KQL queries to detect each failure type Bicep code snippets for new alert rules Remediation suggestions for the app code Exact file paths in my repo to update Check it out: GitHub Issue How I Built It: Step by Step Let me walk you through exactly how I set this up inside SRE Agent. Step 1: Create an SRE Agent I created a new SRE Agent in the Azure portal. Since this workflow analyzes alerts across my subscription (not just one resource group), I didn't configure any specific resource groups. Instead, I gave the agent's managed identity Reader permissions on my entire subscription. This lets it discover resources, list alert rules, and query Log Analytics across all my resource groups. Step 2: Connect GitHub to SRE Agent via MCP I added a GitHub MCP server to give the agent access to my source code repository.MCP (Model Context Protocol) lets you bring any API into the agent. If your tool has an API, you can connect it. I use GitHub for both source code and tracking dev tickets, but you can connect to wherever your code lives (GitLab, Azure DevOps) or your ticketing system (Jira, ServiceNow, PagerDuty). Step 3: Create a Subagent inside SRE Agent for managing Azure Monitor Alerts I created a focused subagent with a specific job and only the tools it needs: Azure Monitor Alerts Expert Prompt: " You are expert in managing operations related to azure monitor alerts on azure resources including discovering alert rules configured on azure resources, creating new alert rules (with user approval and authorization only), processing the alerts fired on azure resources and identifying gaps in the alert rules. You can get the resource details from azure monitor alert if triggered via alert. If not, you need to ask user for the specific resource to perform analysis on. You can use az cli tool to diagnose logs, check the app health metrics. You must use the app code and infra code (bicep files) files you have access to in the github repo <insert your repo> to further understand the possible diagnoses and suggest remediations. Once analysis is done, you must create a github issue with details of analysis and suggested remediation to the source code files in the same repo." Tools enabled: az cli – List resources, alert rules, action groups Log Analytics workspace querying – Run KQL queries for diagnostics GitHub MCP – Search repositories, read file contents, create issues Step 4: Ask the Subagent About Alert Gaps I gave the agent context and asked a simple question: "@AzureAlertExpert: My API endpoint http://132.196.167.102/api/journals/john is timing out. What alerts do I have configured in rg-aks-journal, and why didn't any of them fire? The agent did the analysis autonomously and summarized findings with suggestions to add new alert rules in a GitHub issue. Here's the agentic workflow to perform azure monitor alert operations Why This Matters Faster response times. Issues get diagnosed in minutes, not hours of manual investigation. Consistent analysis. No more "I thought we had an alert for that" moments. The agent systematically checks what's covered and what's not. Proactive coverage. You don't have to wait for an incident to find gaps. Ask the agent to review your alerts before something breaks. The Bottom Line Your alerts have gaps. You just don't know it until something slips through. I had 6 alert rules and still missed a basic failure. My pods weren't restarting, they were just unhealthy. My CPU wasn't spiking, the app was just returning errors. None of my alerts were designed for this. You don't need to audit every alert rule manually. Give SRE Agent your environment, describe the failure, and let it tell you what's missing. Stop discovering alert gaps from customer complaints. Start finding them before they matter. A Few Tips Give the agent Reader access at subscription level so it can discover all resources Use a focused subagent prompt, don't try to do everything in one agent Test your MCP connections before running workflows What Alert Gaps Have Burned You? What's the alert you wish you had set up before an incident? Credential rotation? Certificate expiry? DNS failures? Let us know in the comments.Secure, Seamless Access using Managed Identities with Azure Files SMB
As organizations evolve their application and storage environments, whether on‑premises, hybrid, or cloud, secure access is top of mind. Organizations are vigilant about protecting sensitive data while enabling agile application access across distributed environments. SMB shares are commonly used for persistent storage in applications like AKS for container workloads, web applications, and App Services. Traditional models that rely on credentials like storage account keys do not meet the demands of a Zero Trust architecture, where every access request must be verified explicitly, granted with least privilege, and designed to assume malicious access from bad actors. We are excited to announce the Public Preview of Managed Identities support with Azure Files SMB. This capability provides a secure, identity-driven approach for customer applications that eliminates credentials-based access and integrates seamlessly with MS Entra ID. Azure virtual machines, containers, and applications running in Azure can now authenticate to Azure Files using their own managed identity, and mount shares using short lived OAuth tokens over Kerberos. This unlocks secure file share access for both first party and customer applications, including Azure Kubernetes Service (AKS), Azure Functions, App Services, and other cloud native services By leveraging Managed Identities, customers gain: Zero Trust Alignment–Identity tied to a specific resource, token refreshes every hour, and no passwords or keys to manage or rotate with Azure handling end-to-end identity management Role Based Access Control – Built-in RBAC for least-privilege enforcement Compliance Mandate Resolution – Compliant with FIPS, removing need for NTLMv2 Multi-Client Support – Works with Windows and Linux clients over SMB This capability brings a secure, simple, and scalable access model that helps organizations meet industry standard security requirements while inheriting Microsoft Entra ID’s enterprise grade identity, governance, and security capabilities for file shares. Securing Real World Applications To illustrate how Managed Identities strengthen security, the following example workloads highlight where customers will benefit from this capability. Eliminate Secret Sprawl for Continuous Integration, Continuous Deployment (CI/CD) workloads Azure Files SMB provides a centralized location for storing software development artifacts generated during CI/CD pipelines. CI/CD workloads span far beyond application code, covering infrastructure updates, data engineering workflows, ML pipelines, and compliance automation, making them foundational to modern DevOps practices. Build agents in Azure DevOps or other CI/CD systems often run on both Linux and Windows, requiring a common storage backend for binaries and configuration files. Historically, these agents authenticated to Azure Files using storage account keys. With Managed Identities, build agents can now authenticate using their own identity from Microsoft Entra ID, with authorization governed through Azure RBAC. This enhances security, removes static credentials, and simplifies compliance. “Managed Identities support with SMB shares will enable us to remove dependencies on storage account keys to run our CI/CD pipelines, enabling stronger security and alignment with Zero-Trust principles." Alex Garcia, Staff Dev Ops Engineer, Unity Technologies. Secure Persistent Files Storage with Azure Kubernetes Service (AKS) Stateful AKS workloads rely on persistent volumes for configuration, logs, and application data. Previously, mounting Azure Files required storing account keys or secrets in Kubernetes. Organizations requested exceptions from their security organizations to continue using shared keys until a secure managed identities-based solution was available. With this feature, AKS clusters can authenticate directly to Azure Files SMB without storage account keys. This enables secure, token‑based access for persistent volume mounts, improving security posture and eliminating the need for exceptions to use access tied to storage account keys. Learn more in the Azure Files AKS CSI documentation. Get Started with Managed Identities with SMB Azure Files Start using Managed Identities with Azure Files today at no additional cost. This feature is supported on HDD and SSD SMB shares across all billing models. Refer to our documentation for complete set-up guidance. Whether provisioning new storage or enhancing existing deployments, this capability provides secure, enterprise‑grade access with a streamlined configuration experience. Secure your workloads today! For any questions, reach out to the team at azurefiles@microsoft.comExpanding the Public Preview of the Azure SRE Agent
We are excited to share that the Azure SRE Agent is now available in public preview for everyone instantly – no sign up required. A big thank you to all our preview customers who provided feedback and helped shape this release! Watching teams put the SRE Agent to work taught us a ton, and we’ve baked those lessons into a smarter, more resilient, and enterprise-ready experience. You can now find Azure SRE Agent directly in the Azure Portal and get started, or use the link below. 📖 Learn more about SRE Agent. 👉 Create your first SRE Agent (Azure login required) What’s New in Azure SRE Agent - October Update The Azure SRE Agent now delivers secure-by-default governance, deeper diagnostics, and extensible automation—built for scale. It can even resolve incidents autonomously by following your team’s runbooks. With native integrations across Azure Monitor, GitHub, ServiceNow, and PagerDuty, it supports root cause analysis using both source code and historical patterns. And since September 1, billing and reporting are available via Azure Agent Units (AAUs). Please visit product documentation for the latest updates. Here are a few highlights for this month: Prioritizing enterprise governance and security: By default, the Azure SRE Agent operates with least-privilege access and never executes write actions on Azure resources without explicit human approval. Additionally, it uses role-based access control (RBAC) so organizations can assign read-only or approver roles, providing clear oversight and traceability from day one. This allows teams to choose their desired level of autonomy from read-only insights to approval-gated actions to full automation without compromising control. Covering the breadth and depth of Azure: The Azure SRE Agent helps teams manage and understand their entire Azure footprint. With built-in support for AZ CLI and kubectl, it works across all Azure services. But it doesn’t stop there—diagnostics are enhanced for platforms like PostgreSQL, API Management, Azure Functions, AKS, Azure Container Apps, and Azure App Service. Whether you're running microservices or managing monoliths, the agent delivers consistent automation and deep insights across your cloud environment. Automating Incident Management: The Azure SRE Agent now plugs directly into Azure Monitor, PagerDuty, and ServiceNow to streamline incident detection and resolution. These integrations let the Agent ingest alerts and trigger workflows that match your team’s existing tools—so you can respond faster, with less manual effort. Engineered for extensibility: The Azure SRE Agent incident management approach lets teams reuse existing runbooks and customize response plans to fit their unique workflows. Whether you want to keep a human in the loop or empower the Agent to autonomously mitigate and resolve issues, the choice is yours. This flexibility gives teams the freedom to evolve—from guided actions to trusted autonomy—without ever giving up control. Root cause, meet source code: The Azure SRE Agent now supports code-aware root cause analysis (RCA) by linking diagnostics directly to source context in GitHub and Azure DevOps. This tight integration helps teams trace incidents back to the exact code changes that triggered them—accelerating resolution and boosting confidence in automated responses. By bridging operational signals with engineering workflows, the agent makes RCA faster, clearer, and more actionable. Close the loop with DevOps: The Azure SRE Agent now generates incident summary reports directly in GitHub and Azure DevOps—complete with diagnostic context. These reports can be assigned to a GitHub Copilot coding agent, which automatically creates pull requests and merges validated fixes. Every incident becomes an actionable code change, driving permanent resolution instead of temporary mitigation. Getting Started Start here: Create a new SRE Agent in the Azure portal (Azure login required) Blog: Announcing a flexible, predictable billing model for Azure SRE Agent Blog: Enterprise-ready and extensible – Update on the Azure SRE Agent preview Product documentation Product home page Community & Support We’d love to hear from you! Please use our GitHub repo to file issues, request features, or share feedback with the team
