Secure Native Access to Azure Kubernetes Service (AKS) Private Clusters with Azure Bastion
Written in Collaboration with AvirupChat ShabazShaik Mohit_Kumar YuriDiogenes Introduction: As organizations move toward containerized workloads such as Azure Kubernetes Service (AKS), secure access to cluster resources becomes critical. Making the Kubernetes API server private, therefore, significantly reduces the attack surface. However, it also changes how engineers perform routine management tasks. Operations as simple as running kubectl logs, describing resources, or troubleshooting workloads require connectivity to the virtual network hosting the cluster. This challenge becomes even more apparent during day-to-day operations. An engineer may have the necessary Kubernetes credentials and Azure RBAC permissions yet still be unable to access the cluster because the API server is reachable only from within the private network. Establishing VPN connectivity, using jump hosts, or deploying dedicated management workstations often becomes part of the operational workflow, adding complexity to what should be straightforward administrative tasks. This balance between maintaining strong network isolation and enabling efficient cluster management is exactly what Azure Bastion's native client tunneling support for private AKS clusters is designed to address. Many Azure customers already use Azure Bastion to securely access virtual machines without exposing them to the public internet. Native client tunneling now brings that same secure access model to AKS private clusters. This capability is in public preview currently. Before discussing how it works, it is worth understanding the operational challenges it is designed to solve and how it's done. Bastion for Azure Kubernetes Service (AKS) private clusters: At its core, Azure Bastion is designed to simplify secure access to private Azure resources. Since its launch, it has provided browser-based and native client RDP and SSH connectivity to Azure VMs without exposing management ports to the internet. No public IPs on your VMs. No inbound NSG rules for port 22 or 3389. Just TLS over port 443, routed through a fully managed service that Microsoft patches, scales, and secures on your behalf. When you establish a Bastion tunnel to your private AKS cluster, you are not signing in to an intermediary machine and running kubectl from there. You are opening an encrypted tunnel from your local machine, through Bastion, directly to the private API server endpoint. Your kubeconfig points to localhost on a dynamically assigned port, so local kubectl, helm, and scripts continue to work as they would against a public cluster. You get the developer experience of a public cluster with the security posture of a private one as shown in Figure 1. The high-level flow has three parts: The engineer authenticates to Azure, retrieves cluster credentials, and uses Azure Bastion to establish a managed tunnel into the private network. Local Kubernetes tools then use that tunnel to reach the private API server. This keeps the workflow familiar while removing the need for jump hosts or VPN-based access paths. However, this raises an important question - Does making access easier also make it easier for an attacker to connect? The short answer is no. In fact, Bastion tunneling strengthens the security posture in ways that are worth unpacking. First, the API server itself remains private. There is no public endpoint. There is no public IP address discoverable by scanners. The only network path to the API server runs through Azure's managed infrastructure. Authentication and authorization then depend on the AKS cluster configuration. Second, Bastion eliminates an entire class of infrastructure that itself becomes a security target. Jump box VMs, when not meticulously maintained, accumulate credentials, kubeconfig files, and browser sessions. They are machines that admins log into, which means they are machines that can be compromised. Bastion is not a machine you log into. It is an agentless, managed tunnel you pass through. And for public clusters, there is a related but distinct benefit. Many teams use API server authorized IP ranges to restrict which source IPs can reach their public API endpoint. This is a good practice, but it breaks down quickly when your team works remotely, uses dynamic IPs, or includes contractors. Adding Bastion's stable public IP to the authorized range gives you a consistent, managed access path without the operational toil of constantly updating IP allow lists. With the network path established through Bastion, the next question is how users are authenticated and authorized once they reach the AKS API server. That distinction matters: Bastion provides secure connectivity, while AKS access is governed by the authentication and authorization model configured on the cluster. Authentication and authorization options: When creating an AKS cluster, you can choose from three authentication and authorization modes, as shown in Figure below. That choice shapes everything about how access is granted, audited, and governed over the lifetime of the cluster. ⚠️Local accounts with Kubernetes RBAC is the default if you change nothing. It uses a static certificate that never expires, is shared across all administrators, and has no connection to your corporate identity provider. There is no MFA. No Conditional Access. Note: For most production environments, Microsoft recommends using Microsoft Entra ID integrated authentication rather than local accounts to enable centralized identity, MFA, and auditing. ☑️ Microsoft Entra ID with Kubernetes RBAC moves authentication to Entra ID, which means users sign in with their corporate identity, MFA is enforced, and Conditional Access policies apply. Authorization is handled through native Kubernetes Role and ClusterRole bindings, which reference Entra ID users and groups as subjects. This works well for teams that manage cluster configuration through GitOps, because RBAC manifests live alongside other cluster YAML. The limitation is that these permissions are invisible in Azure IAM — they live only inside the cluster. ✅ Microsoft Entra ID with Azure RBAC is the model we recommend for most production environments. Authentication still flows through Entra ID with full MFA and Conditional Access support. But authorization is handled by Azure RBAC role assignments on the AKS resource itself. Permissions are visible in Azure IAM, participate in access reviews, and integrate with Privileged Identity Management for just-in-time elevation. You can assign the built-in Azure Kubernetes Service RBAC Cluster Admin, Admin, Writer, or Reader roles at either cluster scope or namespace scope. A single subscription-level role assignment can grant access to every cluster in the subscription. The real value is the combination. Bastion provides the encrypted network tunnel. Entra ID provides the identity, MFA, and Conditional Access. Azure RBAC provides centralized, auditable authorization that your security team can review alongside every other Azure resource. For the exact steps to configure Entra ID authentication and Azure RBAC on your AKS cluster, see the AKS identity documentation. End-to-end flow for connectivity to AKS via Bastion: az account set --subscription <subscription ID> Retrieve credentials to your AKS private cluster using the commands below: az aks get-credentials --name <AKSClusterName> --resource-group <ResourceGroupName> Open the tunnel to your target AKS Cluster with the following command: az aks bastion --name <aksClusterName> --resource-group <aksClusterResourceGroup> --bastion <bastionResourceId> Now the default authentication method is Device code authentication i.e., this authentication method prompts the device code for the user to sign in from a browser session. If you want CLI only authentication, you can run the following command next. kubelogin convert-kubeconfig -l azurecli Then go on with your AKS connectivity: kubectl get nodes How to Try It: If you are already running private AKS clusters, getting started is straightforward. You need a Standard or Premium Azure Bastion host deployed in the same VNet as your cluster (or in a peered VNet), with native client support enabled. The aks-preview and bastion CLI extensions handle the rest. The detailed connection steps are documented on Microsoft Learn. We Want Your Feedback Write a comment in this blog or open an issue on the AKS GitHub repository or leave feedback directly on the Microsoft Learn documentation page. We read everything, and it genuinely influences what we prioritize.Lessons Learned #551: Azure SQL Connection Timeouts: Three Things to Check
An application starts reporting intermittent timeouts when connecting to Azure SQL Database. Some requests succeed, others fail, and a test from a developer’s laptop works perfectly. The database appears online, no recent deployment seems related, and the natural reaction is to ask: Is Azure SQL unavailable? Is the firewall blocking the connection? Should we increase the connection timeout? Should we change the driver or scale the database? Those are reasonable questions, but they may lead the investigation in the wrong direction. The most important lesson is simple: A timeout tells us how long the application waited. It does not tell us what the application was waiting for. Not every “SQL timeout” happens inside Azure SQL From the application’s point of view, opening a database connection may involve several operations: Resolving the server name. Reaching the SQL endpoint. Obtaining a Microsoft Entra access token. Waiting for an available pooled connection. Completing the SQL login. Executing the first command. When all these operations are reported through the same application method or log entry, it can look as though Azure SQL took thirty seconds to accept the connection. In reality, only part of that time may have been spent connecting to the database. In one anonymized support scenario, the application experienced problems mainly on its first connection. Network tests were successful and no corresponding SQL connection failure was identified. The investigation eventually showed that access-token acquisition was consuming a significant part of the available time. Increasing the SQL timeout or changing the firewall would not have addressed the real delay. Check 1: Capture the complete error and the exact time A screenshot containing only “Connection Timeout Expired” is rarely enough. Capture: The complete exception and inner exception. The operation being performed. The driver and version. The authentication method. The exact timestamp in UTC. Whether the issue affects every connection or only some of them. The wording around the timeout matters. For example, a timeout while obtaining a connection from the pool points toward the application’s pooling and concurrency behavior. A pre-login or TLS error belongs to a different investigation. A command timeout after the connection was established is usually a query-performance problem rather than a connection problem. Check 2: Measure the application timeline The application should record important operations separately. A simple timeline can completely change the investigation: 10:14:20.100 Token acquisition started 10:14:28.400 Token acquired 10:14:28.405 SQL connection started 10:14:29.050 SQL connection established The complete operation took almost nine seconds, but Azure SQL connection establishment took less than one second. Useful measurements include: Token-acquisition duration. Time waiting for a pooled connection. SQL connection-open duration. SQL command duration. Number of retry attempts. Applications using Microsoft Entra authentication must obtain an access token before authenticating to Azure SQL. Measuring that operation separately helps distinguish an identity delay from a database connectivity problem. This is particularly useful when the issue appears: On the first connection after startup. After a token expires. Only with Managed Identity or Workload Identity. Intermittently, while SQL authentication connections remain unaffected. Check 3: Test from the application environment A successful connection from a laptop does not validate the path used by an application running in: Azure App Service. Azure Functions. Azure Kubernetes Service. A virtual machine. An on-premises application server. A container or integration runtime. The laptop and the application may use different DNS servers, routes, firewalls, proxies and identities. Connectivity and DNS tests should therefore be performed from the environment that is actually failing. This becomes especially important when Private Endpoint is used. The application should continue connecting with: <server>.database.windows.net It should not use the Private Endpoint IP address or the privatelink.database.windows.net hostname directly. Direct login attempts using the private IP or the private-link FQDN fail; the normal logical-server FQDN must remain in the connection string. From the affected environment, confirm that: The expected DNS server answers the request. The server FQDN resolves to the expected private IP. The Private Endpoint connection is approved. The Private DNS zone is linked correctly. The resolved address is reachable through the intended route. A test from an unrelated machine is still useful for comparison, but it does not prove that the application path is healthy. Observed symptom Likely investigation area Timeout while obtaining a connection from the pool Application connection pooling Server name cannot be resolved DNS TCP connection to the endpoint cannot be established Network path, firewall or routing Error during the pre-login handshake TLS, driver, network interruption or pre-login processing Authentication or access-token error Microsoft Entra authentication, identity or token acquisition Timeout during the post-login phase Login completion, session initialization or server-side processing Execution or command timeout after connecting Query execution and database performance Avoid changing several things at once During a production incident, it is tempting to: Increase the timeout. Add firewall rules. Change the connection policy. Upgrade the driver. Restart the application. Clear connection pools. Applying several changes together makes it difficult to determine which one helped, and some may only hide the symptom. A better approach is to define one hypothesis: We believe DNS in the application environment is resolving the public endpoint instead of the Private Endpoint. Then define: The evidence supporting the hypothesis. One controlled change. The expected result. How the result will be measured. How the change will be reverted. Azure SQL supports Proxy and Redirect connection policies, which determine how traffic flows after reaching the Azure SQL gateway. The policy is configured for the logical server, so it should be verified before making firewall assumptions or changes. What should we collect before opening a support request? A small but precise evidence package can avoid several rounds of questions: Complete error and inner exception. Exact UTC timestamps. Application platform and location. Public or Private Endpoint. Server FQDN used by the application. Driver and version. Authentication method. Token, pool, connection and command durations. DNS result from the affected environment. Whether the issue is constant, intermittent or limited to the first connection. Recent application, network, identity or configuration changes.
Announcing Public Preview - Azure Private Link over IPv6
1. Overview Private Link over IPv6 (PL IPv6) enables customers to securely access Azure PaaS services over IPv6-based connectivity. This capability is critical for: IPv6 based PE connectivity to PaaS resources Enabling IPv6 in On-prem environments This document is intended to serve as guide to setup and test the On-prem connectivity from IPv6 customer address to Azure PaaS resources over Express Route via Private link. (IPv6 PE connectivity) Note: This feature is currently in public preview and is not recommended for production workloads. 2. Supported Scenarios Scenario A: Native Azure (Azure VM → PaaS via IPv6 Private Endpoint) IPv6 client VM in VNet → IPv6 Private Endpoint → Azure PaaS Scenario B: OnPrem Access via ExpressRoute On-prem IPv6 client → ExpressRoute → VNet Routing Appliance (VNRA) à IPv6 Private Endpoint → PaaS Supported via ER Circuits 3. Prerequisites 3.1 Supported Regions (Preview Scope) Limited preview regions West Central US East Asia UK South US Central North Europe 3.2 Supported Services (Preview) Azure Storage Azure SQL Azure Key Vault Azure Data Explorer 3.3. Subscription registration An Azure account with an active subscription. Create an account for free. Your subscription must be registered for the Private Link over IPv6 preview. Registration is mandatory before you configure any resources. You can self-register the subscription by running the following commands: az feature register --namespace Microsoft.Network --name SupportIPv6PrivateEndpoint --subscription <subscription-id> az provider register --namespace Microsoft.Network 4. Configurations 4.1 Azure VNET level configurations Please note that you will need to create: Dual-stack IPv6-enabled Vnet with a VM that is assigned both IPv4 and IPv6 address. For on-prem connectivity, this is the same Vnet where ER Gateway is created. Refer: Create an Azure virtual machine with a dual-stack network - Azure Virtual Network | Microsoft Learn PL scale enabled on VNET via configuration flags (refer config flags mentioned further in this section and bolded flags in the code snippet) Dual Stack (IPv4+ IPv6) Subnet Private Endpoints with IPv6 Refer these flags for enabling VNET with PL scale: Vnet level : "privateEndpointVNetPolicies": "Basic", Subnet level: “privateEndpointNetworkPolicies": "RouteTableEnabled" Refer Azure documentation for VNET creation: Quickstart: Create an Azure Virtual Network | Microsoft Learn 4.3 Private endpoint configuration: With the Private Link IPv6 support, we have introduced a new parameter in PE creation API/CLI, ‘ip VersionType’. Please ensure that this is set to ‘IPv6’ for enabling PL IPv6 traffic. Refer Azure documentation for PE creation: Quickstart: Create a private endpoint - Azure portal - Azure Private Link | Microsoft Learn Reference CLI: az network private-endpoint create \ --name <private-endpoint-name> \ --resource-group <resource-group-name> \ --vnet-name <vnet-name> \ --subnet <subnet-name> \ --private-connection-resource-id <resource-id-of-target-service> \ --group-id <group-id> \ --connection-name <connection-name> \ --location <region> \ --ip-version-type <Ipv4|Ipv6 > 4.4 DNS Configuration Ensure these points for DNS configurations: Create Private DNS Zone for each Service type Attach the respective DNS zone to the Private Endpoints. Ensure to check that after above steps completed PaaS FQDN resolves to PEIPv6 address. Refer Azure documentation for On-prem access via Private Link: Azure Private Endpoint DNS Integration Scenarios | Microsoft Learn Native Azure Connectivity (Azure VM → PaaS via IPv6) Validation: Ensure the Storage Account FQDN resolves to the IPv6 Private Endpoint address. Access the Storage Account using the standard service FQDN. From the dual-stack VM: nslookup <storageaccount>.blob.core.windows.net Expected Result: <storageaccount>.privatelink.blob.core.windows.net AAAA: <Private Endpoint IPv6 Address> Connectivity validation: curl https://<storageaccount>.blob.core.windows.net Test-NetConnection <storageaccount>.blob.core.windows.net -Port 443 Configurations specific to On-prem connectivity via ER circuit and Virtual Network Routing Appliance On-premises IPv6 clients access IPv6 private endpoints over ExpressRoute through a Virtual Network Routing Appliance (VNRA). ExpressRoute forwards traffic from on-premises IPv6 clients to the VNRA, which then routes the traffic to the target IPv6 private endpoint that's hosted by the Azure PaaS service. This connectivity requires the following components: An ExpressRoute circuit with the appropriate gateway SKU for your connectivity type (FastPath or standard). For more information, see Create and modify an ExpressRoute circuit. 4.5 Virtual Network Routing Appliance (VNRA) configurations To know more about VNRA, refer: Overview of Routing Appliances - Azure Virtual Network | Microsoft Learn We are leveraging VNRA to facilitate PL IPv6 ER traffic forwarding. User would be required to create a VNRA in the VNet. To facilitate the forwarding of PL IPv6 traffic via VNRA, a UDR would be required to be added in the gateway subnet with next hop as VNRA IPv6 address. Below are the guided steps to achieve all of this. Step 1: Create a VNRA in the VNet Create a VNRA in your resource group via Azure Preview portal. Search for Azure Virtual Network routing appliance on the Azure portal search Click on create: In the above creation page, choose your subscription and resource group, enter name, region, capacity (10-200 Gbps) & the VNet. Review and Create the VNRA. Step 2: Create User Defined Route (UDR) to VNRA The UDR will ensure ER PLIPv6 traffic is forwarded to VNRA which will further process & forward the traffic to the IPv6 Private Endpoint. 1.Create a new route table on Azure portal: 2. Choose the subscription, resource group and set ‘Gateway propagation’ as default (true) 3. Add a route in this route table to forward the on-prem PLIPv6 traffic to VNRA (for this please add route as shown in reference below) as per below details: Enter ‘Destination type’ as ‘IP Address’ Enter ‘Destination IP addresses/CIDR Ranges’ as Private Endpoint IPv6 subnet range - This is the subnet on which your Private Endpoint resides to which the traffic would be sent Enter ‘Next hop’ as “Virtual Appliance”, Give the ‘Next hop address’ as the IPv6 address of VNRA 4.Attach this route table to the Gateway subnet This completes the VNRA setup. 5. Validate Connectivity After establishing configuration and deploying your setup, you can run following validations: From OnPrem VM: Perform DNS lookup on PaaS FQDN → confirm IPv6 PE resolution Connect using PaaS FQDN Validation Checks VM → PE connectivity over IPv6 Data plane traffic successful 6. Preview Considerations Limited regional availability (Check preview regions above) Destination PaaS resource must be in the same region as the Private Endppint. Cross region connectivity is not supported in this release. Limited PaaS onboarding (Azure Storage, Azure SQL, Azure Key Vault, Azure Data Explorer) Current on-premises connectivity support is limited to ExpressRoute-based scenarios and does not currently include VPN, Virtual WAN (vWAN), or Network Virtual Appliances (NVAs). In Private Link IPv6 scenarios, the original client IPv6 address is not preserved in downstream service logs. Due to implicit NAT translation, logs will display the VNet-side translated source IP address instead. Link to Azure documentation: Configure Azure Private Link over IPv6 (Preview) - Azure Private Link | Microsoft LearnAzure Virtual Network routing appliance is now generally available
Modern cloud networks are evolving faster than ever. Organizations are building larger AI platforms, connecting more services through private connectivity, adopting IPv6, and expanding applications across regions and business units. As these environments grow, the network becomes a critical foundation for delivering performance, resiliency, and operational simplicity. Today, we're excited to announce the general availability of Azure Virtual Network routing appliance, a managed, platform-native routing service designed to provide high-performance connectivity across Azure virtual networks at cloud scale. Virtual Network routing appliance brings together Azure-native operations, specialized networking infrastructure, built-in resiliency, and high-bandwidth forwarding to help organizations build the next generation of cloud network architectures. Built for the era of AI infrastructure AI is changing the scale at which networks operate. Training clusters, inference services, analytics platforms, data processing pipelines, and distributed application environments generate unprecedented volumes of east-west traffic. These workloads require high-performance connectivity between services, networks, and regions while maintaining operational simplicity. Virtual Network routing appliance provides a managed routing foundation for these environments, enabling organizations to scale network connectivity alongside their AI investments. Instead of building and operating custom routing infrastructure, teams can focus on accelerating innovation, deploying new services, and delivering business outcomes. Scale hub-and-spoke architectures Hub-and-spoke remains one of the most widely adopted network architectures in Azure because it provides centralized governance, simplified operations, and efficient connectivity. As organizations expand, however, these architectures often grow from a handful of virtual networks into hundreds or even thousands of connected environments. Virtual Network routing appliance enables customers to scale these architectures while maintaining a consistent operational model. By providing a dedicated routing layer within the hub, Virtual Network routing appliance simplifies connectivity between applications, shared services, and business units while supporting the scale required by modern enterprise environments. The result is a network architecture that remains manageable even as organizational growth accelerates. Unlock large-scale private connectivity Private connectivity has become the default connectivity model for modern cloud deployments. Applications, databases, platforms, shared services, and partner solutions increasingly depend on private communication patterns across Azure environments. Virtual Network routing appliance provides a centralized routing foundation that helps customers build and scale these architectures while maintaining a consistent private networking experience across their environments. Virtual Network routing appliance can also help scale Private Endpoint connectivity beyond the current 20,000-endpoint HSPE boundary. Looking ahead, it establishes a foundation for further accelerating private connectivity to on-premises environments, without introducing additional architectural specifics. As organizations continue consolidating services onto private connectivity models, Virtual Network routing appliance provides the performance and scale needed to support long-term growth. Accelerate your IPv6 journey IPv6 adoption continues to grow across enterprise, telecommunications, and cloud environments. Organizations increasingly need network architectures capable of supporting IPv4, IPv6, and dual-stack deployments while maintaining operational consistency. Virtual Network routing appliance supports IPv4, IPv6, and dual-stack virtual networks, enabling customers to modernize network architectures and expand address space without introducing new operational complexity. Whether organizations are beginning their IPv6 transition or building IPv6-first architectures, Virtual Network routing appliance provides a consistent routing foundation across both address families. Simplify multi-region architectures Modern applications rarely live within a single region. Organizations increasingly deploy workloads globally to improve performance, resiliency, business continuity, and regulatory compliance. These architectures require a networking foundation capable of supporting connectivity across regions while remaining simple to operate and govern. Virtual Network routing appliance helps customers build scalable multi-region network architectures by providing a centralized, high-performance routing layer that integrates naturally into Azure networking designs. This allows teams to focus on application architecture and customer experience rather than operational management of routing infrastructure. Built for enterprise scale As organizations continue to grow, networking teams face a common challenge: supporting increasing scale without increasing operational complexity. Virtual Network routing appliance was designed to meet this challenge by combining: High-performance routing using specialized Azure networking infrastructure Built-in resiliency and availability zone support Native Azure management and governance integration Support for IPv4, IPv6, and dual-stack deployments Integrated monitoring and observability through Azure Monitor metrics available Configurable bandwidth tiers for production workloads Future support for scaling Private Endpoints beyond 20,000 These capabilities allow customers to build large-scale networking architectures while maintaining a familiar Azure-native operational experience. Learn more Azure Virtual Network routing appliance is more than a new networking resource. It is a foundational building block for the next generation of Azure networking. Organizations are continuing to build larger AI platforms, expand private connectivity, increase multi-region deployments, and modernize network architectures. These transformations require a routing foundation that can scale alongside them. Virtual Network routing appliance provides that foundation, delivering the performance, scale, resiliency, and operational simplicity required for modern cloud networks. Whether you're building an AI platform, expanding a hub-and-spoke architecture, scaling private connectivity, enabling IPv6, or designing a global application footprint, Azure Virtual Network routing appliance helps simplify networking so you can focus on what matters most: delivering innovation faster. Overview of Routing Appliances - Azure Virtual Network | Microsoft Learn
Simplify secure, zone-resilient outbound connectivity with Azure Firewall and StandardV2 NAT Gateway
As organizations modernize their applications in Azure, secure and resilient outbound connectivity has become just as critical as inbound security. Workloads need reliable access to external APIs, SaaS services, operating system updates, and partner endpoints, while still meeting strong security controls, predictable egress IPs, and high availability. Achieving all of this consistently requires using the right networking services together. To make this easier, we’ve updated the Azure Firewall create experience in the Azure portal to include StandardV2 NAT Gateway directly in the deployment flow. This new experience makes it quick and seamless to adopt a secure, scalable, and zone‑resilient outbound architecture from day one by using Azure Firewall and Azure NAT Gateway together. In this post, we’ll cover: Why pairing Azure Firewall with StandardV2 NAT Gateway is a recommended design How this combination simplifies secure and resilient outbound connectivity What’s new in the Azure Firewall portal experience and how to get started Why Azure Firewall and StandardV2 NAT Gateway? Azure Firewall and Azure NAT Gateway are designed to complement each other, each focusing on what they do best: Azure Firewall provides centralized traffic inspection and policy enforcement, including IP address and FQDN filtering, threat intelligence, and logging. StandardV2 NAT Gateway delivers high‑scale outbound SNAT, static egress IPs, and built‑in zone redundancy. StandardV2 NAT Gateway is zone‑redundant by default, automatically spanning availability zones within a region. This means outbound connectivity remains available even during a zonal failure without requiring multiple zonal NAT gateways or additional routing configurations. Together, this pairing cleanly separates: Security policy and inspection handled by Azure Firewall Outbound scale, resiliency, and IP predictability handled by NAT Gateway This separation is key for modern, large‑scale cloud workloads. A recommended outbound architecture In a typical hub‑and‑spoke design: Workloads in spoke virtual networks route outbound traffic to Azure Firewall in the hub Firewall policies inspect and allow the traffic StandardV2 NAT Gateway is attached to the AzureFirewallSubnet in the Hub Approved traffic flows through StandardV2 NAT Gateway for SNAT Traffic exits Azure using static, predictable public IPs This approach provides several important benefits: Scalable SNAT capacity for high‑connection workloads Static outbound IPs for partner allow‑listing and compliance Zone‑resilient outbound connectivity by default For step-by-step architectural guidance, see Integrate NAT gateway with Azure Firewall in a hub and spoke architecture. Built for secure and resilient Azure environments As customers increasingly adopt availability zones, large‑scale VMSS or AKS deployments, and zero‑trust network models, outbound connectivity must be secure, predictable, and resilient. By pairing Azure Firewall with StandardV2 NAT Gateway—and now surfacing this pairing directly in the portal create experience—customers can start with a production‑ready outbound architecture that scales with their environment. What's new in the Azure Firewall create experience in the portal When creating a new Azure Firewall in the portal, customers can now: Select and associate a StandardV2 NAT Gateway during firewall deployment. Reduce post‑deployment configuration and manual touches. Start with a recommended zone-resilient outbound architecture by default. By bringing NAT Gateway directly into the Firewall create flow, the portal helps guide customers toward a more secure and scalable outbound setup—without requiring them to stitch services together after the fact. Get started You can try the updated experience today by creating a new Azure Firewall in the Azure portal and selecting StandardV2 NAT Gateway during deployment. With just a couple clicks of a button: In the Basics tab, configure your Firewall settings (ex., SKU, policy, virtual network). In the *new* Advanced tab, create a new or add an existing StandardV2 NAT gateway and associate StandardV2 public IP addresses or prefixes. The StandardV2 NAT gateway is automatically attached to the Firewall subnet—no additional routing or configuration required. Review and Create. Note: StandardV2 NAT Gateway is not yet available in all regions. If your selected region does not support StandardV2 NAT Gateway, the option to enable StandardV2 NAT gateway will not appear during Firewall creation. Refer to StandardV2 NAT Gateway limitations for more information. For more details, see: Integrate StandardV2 NAT Gateway with Azure Firewall Integrate NAT Gateway with Azure Firewall in a hub‑and‑spoke network Azure NAT Gateway SKUs
Simplify Virtual WAN Spoke Connectivity at Scale with Azure Virtual Network Manager
With Azure Virtual Network Manager (AVNM) integration, organizations using Virtual WAN for transitive connectivity can simplify spoke connectivity and policy management across large-scale hub-and-spoke deployments. By using a Virtual WAN hub as the hub in an AVNM hub-and-spoke topology, organizations can define connectivity and routing intent once at the network group level and apply it consistently across large numbers of spoke VNets. This reduces repetitive per-spoke connection and routing configuration, helps maintain operational consistency as deployments expand, and makes it easier to manage hub-and-spoke environments at scale. Together, AVNM’s centralized, group-based orchestration and Virtual WAN’s managed routing, security integration, and hybrid connectivity provide a more streamlined way to simplify operations and scale with confidence. What is Azure Virtual Network Manager? Azure Virtual Network Manager is a management service that lets you group, configure, and deploy network connectivity and security policies across virtual networks at scale. Instead of configuring VNet peering and access rules on each virtual network individually, you define network groups — logical collections of virtual networks based on static selection or dynamic Azure Policy conditions — and apply connectivity configurations and security admin rules to those groups. Key capabilities include: Hub-and-spoke and mesh topologies — Define how virtual networks in a network group connect to a central hub or to each other. Network groups — Group VNets statically or dynamically (using tags, subscriptions, resource group names, or other Azure Policy conditions). Security admin rules — Author and enforce access control lists across all VNets in a network group, providing a centralized layer of defense that complements NSGs and firewalls. Region-scoped deployment — Deploy configurations to specific Azure regions, enabling incremental rollout and controlled blast radius. AVNM operates as an overlay management layer — it orchestrates VNet peering, connectivity, and security rules without replacing the underlying networking primitives. What is Azure Virtual WAN? Azure Virtual WAN as a service brings together routing, security, VPN, ExpressRoute, and transitive connectivity in a hub-and-spoke architecture. A Virtual WAN hub is a managed regional resource that acts as a central transit point for branch connectivity, remote users, private enterprise connectivity, spoke virtual networks, and private traffic routing through security services. Site-to-site VPN connectivity (branch offices, SD-WAN devices) Point-to-site VPN connectivity (remote users) ExpressRoute private connectivity (on-premises datacenters) VNet-to-VNet transitive connectivity (spoke virtual networks) Routing, firewall, and encryption for private traffic All hubs in a Standard Virtual WAN are connected in a full mesh over the Microsoft backbone, enabling any-to-any connectivity between spokes, branches, and remote users across regions. Virtual WAN removes the need to manually manage complex route tables and transit VNets — routing is handled by the hub's built-in router. What this integration enables When you select a Virtual WAN hub as the hub in an AVNM connectivity configuration, AVNM handles the spoke-to-hub wiring for you. For each virtual network in your selected network groups: If the VNet is not yet connected to the Virtual WAN hub, AVNM creates the Virtual Network connection to Virtual WAN hub and applies a consistent routing configuration with Virtual WAN connection policy. If the VNet is already connected, AVNM updates the existing Virtual Network connection to utilize the routing properties in the Virtual WAN connection policy. A connection policy is a hub-level Virtual WAN resource that defines shared routing behavior for the virtual network connections it governs, including route table association and propagation, route maps, internet security settings, and propagated labels. Because the policy applies these settings consistently across governed connections, it helps standardize routing and overrides conflicting settings configured directly on individual connections. How it works The setup follows AVNM's standard workflow: Create a network group. Add virtual networks as members — either statically (by selecting specific VNets) or dynamically (using Azure Policy conditions such as tags or resource group names). Create a connectivity configuration. Choose hub-and-spoke topology, select your Virtual WAN hub as the hub, and select or create a connection policy. Deploy. Commit the configuration to your target regions. AVNM connects all VNets in the network groups to the Virtual WAN hub and applies the connection policy in parallel. You can also enable direct connectivity within a spoke network group. When enabled, VNet-to-VNet traffic within that group routes directly between virtual networks instead of transiting the Virtual WAN hub — useful for latency-sensitive or high-throughput east-west workloads. By default, direct connectivity is regional; enable global mesh to extend it across Azure regions. Key use cases Bulk spoke onboarding Connect many virtual networks to a Virtual WAN hub in one operation. All connections are orchestrated in parallel by AVNM, and the pre-defined routing configuration is automatically applied. Policy-based dynamic onboarding Use Azure Policy to define network group membership conditions. When a new virtual network matches those conditions—for example, a VNet tagged env:prod—it is automatically added to the network group. On the next deployment, AVNM connects it to the Virtual WAN hub with the correct routing configuration, reducing manual onboarding effort. Batch routing configuration updates Push routing changes to all virtual networks in a network group as a single, fully parallelized operation. This significantly reduces maintenance window duration for network-wide changes and makes rollback straightforward. Incremental deployment Segment your network into precise update domains by creating separate network groups — for example, by environment (staging, dev, production) or by region. Deploy connection policies to each group or region independently. This lets you test changes on a smaller subset before applying them broadly, minimizing blast radius. Mesh for selective inspection bypass If you use routing intent to send all private traffic through a firewall in the Virtual WAN hub, certain high-throughput or latency-sensitive flows (such as database replication) may benefit from bypassing that inspection. Enable direct connectivity in AVNM to create a mesh between selected spokes, allowing VNet-to-VNet traffic to route directly while all other traffic continues through the hub firewall. Security admin rules at scale Define network groups for your Virtual WAN spokes, then use AVNM security admin rules to author and deploy access control lists across those spokes. This provides an additional layer of defense alongside next-generation firewalls in the Virtual WAN hub. Getting started Prerequisites: An existing Azure Virtual Network Manager instance An existing Azure Virtual WAN and Virtual WAN hub One or more virtual networks to use as spoke members To configure: Go to your Network Manager instance in the Azure portal. Create a network group and add your spoke VNets. Create a connectivity configuration → select hub-and-spoke → select your Virtual WAN hub → select or create a connection policy → add spoke network groups. Deploy the configuration to your target regions. In your Virtual WAN resource, verify that the expected spoke VNet connections are in a connected state. Review effective routes in the virtual hub to confirm routing behavior matches the selected connection policy. For detailed step-by-step instructions, see Configure Azure Virtual WAN hub for Azure Virtual Network Manager. For more on connection policy, see Connection policy in Azure Virtual WAN. Learn more Azure Virtual Network Manager documentation Virtual WAN and Virtual Network Manager integration overview Azure Virtual WAN documentationAzure Front Door edge actions: programmable compute for a secure, resilient, AI-ready edge
The need for secure edge programmability As modern web applications increasingly move decision-making closer to users, programmable compute at the edge is becoming a foundational capability for delivering low-latency, personalized, and intelligent experiences. Azure Front Door edge actions introduces lightweight customer-defined logic that executes close to users at Microsoft's global edge (https://aka.ms/edgeactionsblog). The engineering challenge extends well beyond moving code closer to the request path. It is about enabling edge programmability while preserving the core guarantees customers expect from a global edge platform: hyperscale performance and acceleration, strong security and tenant isolation, resiliency, and fast, controlled recovery. That sets up a much higher engineering bar than simply bringing a serverless runtime to the edge. Programmability introduces customer code, new execution paths, runtime dependencies, and additional failure modes directly into the critical request path. Architecture therefore must make flexibility a first-class capability without compromising the operational characteristics of a hyperscale edge platform. Preserving performance at hyperscale The first architectural challenge was preserving the performance characteristics of Azure Front Door while introducing programmable execution into the request path. Every additional execution step has the potential to increase latency, amplify failures, or reduce throughput at global scale. Edge actions was therefore designed to add programmability without changing the fundamental performance profile customers already expect from Azure Front Door. At request time, Azure Front Door evaluates the request, determines whether an edge action should be executed based on the associated rule, invokes the edge actions runtime, and applies the result inline. Because the runtime sits directly in the request path, every design decision was guided by a common principle: keep execution local whenever possible, bound latency when dependencies degrade, and ensure optional compute never becomes a platform-wide latency amplifier. Performance design principles Node-local execution keeps request processing on the same machine whenever possible, minimizing cross-node communications and preserving low latency. Minimized inter-node hops keep the common path compact while still enabling cluster-level fallback when local dependencies deteriorate. Connection reuse through Edge Action Agent reduces gRPC invocation overhead and improves hot path efficiency. Lightweight Hyperlight isolation provides strong tenant isolation with an execution model suitable for latency-sensitive edge workloads. Fast-fail and circuit-breaker protects latency by bounding waits on degraded dependencies and preventing cascading pressure. Together, these architectural choices introduce programmable compute without turning the Azure Front Door data plane into a distributed orchestration layer. The hot path remains local, predictable, and bounded, with fallback used only when necessary to preserve performance across the global edge. Security and tenant isolation by design Running customer-defined code on a shared global edge fundamentally changes the security model. Unlike traditional request processing, programmable execution introduces untrusted customer code directly into the request path, making strong isolation a foundational architectural requirement rather than an operational safeguard. For Azure Front Door edge actions, every execution is designed to run within a dedicated Hyperlight micro-VM, providing hardware-enforced isolation between customer workloads, the Azure Front Door data plane, and the underlying host environment. Security design principles Hypervisor-backed isolation ensures customer code executes within dedicated Hyperlight micro-VM boundaries rather than shared execution environments. Data plane separation isolates edge actions execution from Azure Front Door's core traffic-processing path. Minimal host surface area reduces the attack surface and limits privileged interactions. Restricted execution context exposes only the request information required to process a request. Reduced operational blast radius helps contain compromised or misbehaving workloads. These architectural boundaries extend beyond workload isolation. Azure Front Door's data plane remains physically separated from the edge actions orchestration service, while each execution receives only the minimum context required to perform its task. This defense-in-depth approach reduces both security risk and operational blast radius without compromising performance. Hyperlight: Security without sacrificing performance A key differentiator of Azure Front Door edge actions is its use of Hyperlight micro-VMs to provide hardware-backed isolation without introducing the traditional performance penalties associated with virtual machines. Hyperlight was designed to make VM-level protection practical for high-throughput function execution, enabling strong tenant isolation while remaining suitable for latency-sensitive edge workloads. Edge actions builds this foundation through the edge action orchestrator, which maintains a pool of warm Hyperlight sandboxes ready to serve requests. By reusing pre-initialized sandboxes instead of creating a new execution environment for every request, edge actions minimizes initialization overhead, reduces request latency, and sustains higher throughput under load. The result is a security model based on VM isolation that remains compatible with the performance expectations of a hyperscale edge platform. Critically, performance optimizations do not weaken isolation guarantees. After each execution, sandbox state is cleaned before reuse, ensuring that subsequent invocations cannot access data from prior executions while preserving the efficiency benefits of warm sandboxing. In internal benchmarking, lightweight edge actions executed in less than 2 ms inside Hyperlight, with approximately 1.27 ms of total sandbox overhead, demonstrating that strong isolation and high-performance edge execution can coexist. Security enables resiliency Security and resiliency are closely related architectural goals. Isolation helps contain malformed inputs, unexpected behavior, and execution failures, preventing individual workloads from affecting the broader platform. In a multitenant edge service, isolation is not only a security requirement; it is also a key resiliency mechanism. Resiliency built into the platform Strong isolation is not only a security property, but also a foundational resiliency mechanism. By containing malformed inputs, unexpected behavior, and execution failures within dedicated execution boundaries, the platform prevents individual workloads from affecting neighboring tenants or the broader service. At hyperscale, robust isolation is essential for maintaining customer trust and predictable platform reliability. Building on that foundation, Azure Front Door edge actions was designed around a simple operating principle: failures are inevitable, but their impact must be predictable, bounded, and recoverable. Because programmable compute introduces additional execution paths and runtime dependencies into the request path, resiliency must be built into the control points that determine when to execute, stop waiting, or fall back. The platform incorporates lessons learned from operating Azure services at a global scale, with a focus on minimizing blast radius, maintaining service continuity, and enabling controlled recovery when dependencies fail, overload, or time out. Resiliency principles Bound failure impact through isolation and containment. Recover predictably using health-aware routing and fallback paths. Protect customer availability first through graceful degradation. Fail fast rather than fail slowly to avoid latency amplification. Continuously validate assumptions through Game Days and fault injections. These principles translate into request-time behavior through deadlines, circuit breakers, fail-open behavior, and health-based fallback. Together, they ensure that optional programmable execution enhances application capabilities without compromising the stability of Azure Front Door's core request-processing pipeline. Continuous validation of resiliency assumptions Resilient architecture is credible only when validation becomes part of the operating model. For edge actions, Game Days and Fault Injections provide recurring opportunities to verify that architectural assumptions continue to hold under production-like stress. Validation includes chaos and failure injections, timeout and dependency-loss exercises, overload and queue-growth scenarios, mixed-workload testing, and interface fuzzing. These exercises answer practical production questions: Does fail-open behavior protect the request path? Do circuit breakers engage early enough? Does fallback routing preserve service continuity? Do malformed inputs remain contained? Repeated validation also strengthens operations. Detection improves, mitigation becomes more predictable, and recovery evolves from architectural intent into demonstrated operational capability. Built for future intelligent & modern workloads Edge actions is designed for lightweight programmable execution today, but the underlying architecture is intended to support increasingly intelligent decision-making over time. The engineering requirement remains unchanged: future intelligence workloads must operate within the same architectural constraints that govern today's request processing - bounded execution, strong isolation, predictable fallback, and protection of the common request path. Architectural implications for intelligence workloads Real-time AI inferencing for request classification and policy evaluation. Intelligent bot, abuse, and fraud detection closer to users. AI-assisted origin selection and traffic-routing decisions. Application-specific SLM-powered decision making at the edge. In that model, the objective is not simply to introduce more intelligence at the edge, but to ensure that intelligence inherits the same platform guarantees as every other component of the request path. Closing thoughts Programmable edge execution is becoming a foundational capability for modern distributed applications. The engineering challenge, however, extends far beyond running customer code closer to users. It is about preserving the system properties that customers already depend on while introducing a new execution surface into the critical request path. Edge actions demonstrates that edge programmability, performance, security, tenant isolation, and resiliency are not independent design goals - they are a single architectural problem that must be solved together. By keeping the common path protected, failures bounded, tenants strongly isolated, and recovery predictable, Azure Front Door edge actions extends the platform's capabilities without compromising the engineering principles that underpin a global hyperscale edge service. Learn more Introducing Hyperlight Edge actions samples: JavaScript request contextIntroducing Azure Front Door edge actions - Bringing secure, programmable logic to the edge
Modern web applications are expected to feel instant, secure, and personalized, no matter where users connect from or what device they use. Delivering that experience requires more than moving static content closer to users. It increasingly depends on making smart decisions at the edge, from validating requests and selecting the right origin to personalizing responses and stopping suspicious traffic before it reaches the backend. Because of this, over the last decade, compute resources for web applications have steadily moved closer to users. What began with content caching at the edge quickly evolved into dynamic routing, security enforcement, and intelligent application traffic management. Today, enterprises expect even more: programmable compute at the edge! Developers and platform teams want the flexibility to run lightweight logic at the edge without giving up the scale, security, and operational confidence they expect from Azure. Azure Front Door through its rulesets offers a robust, declarative framework for applying common traffic management and security policies, such as redirects, header manipulation, and conditional routing. They are optimized for scenarios where behavior can be defined statically and evaluated efficiently at scale. As customer workloads become more dynamic, and application logic increasingly moves closer to the user, additional capabilities are often required beyond declarative configuration. Meet Azure Front Door edge actions, now in public preview We are excited to announce Azure Front Door edge actions which represent Microsoft’s next step in this evolution. With edge actions, customers can execute custom logic directly at Microsoft’s global edge, enabling new classes of real-time personalization, security, and resiliency scenarios, without pushing complexity back to origins! Edge actions allow customers to author lightweight JavaScript functions that execute as part of Azure Front Door request processing. In the current public preview, edge actions are invoked during the client request phase and are attached to Azure Front Door routes through rulesets. This tight integration means edge actions work seamlessly with existing Azure Front Door capabilities, including Web Application Firewall (WAF), caching, and routing - while extending them with programmable logic. Customers can incrementally adopt edge actions without re-architecting their applications or delivery pipelines. What can you build with edge actions today Edge actions in public preview are optimized for lightweight, latency-sensitive scenarios that benefit from immediate decision-making. Common use cases include: A/B experimentation and canary rollouts: Evaluate request context at the edge and route users to different application experiences or releases without adding origin-side decision logic. Request and response header manipulation: Add, remove, or transform headers to support security controls, experimentation, routing, and application modernization patterns. Request rejection: Stop unwanted, malformed, or unauthorized requests before they consume origin resources. Dynamic origin selection: Choose the best origin based on request attributes, health signals, geography, device type, or business logic. URL rewrite and redirect: Adapt paths or redirect users at the edge to simplify migrations, campaign launches, localization, and application routing. Authentication and authorization scenarios: Validate tokens or request attributes close to the user, helping protect applications before traffic reaches backend services. Please refer to Azure/EdgeActionsSamples to start building edge actions today. Designed for industry leading security-first edge compute Running customer-defined code at the edge introduces unique security challenges. Edge environments now process untrusted user code at massive scales, making strong isolation non-negotiable for Azure. Enter Hyperlight: Microsoft’s foundation for secure execution. With Hyperlight, each edge action code runs in its own lightweight, hardware-backed micro-VM - like a secure apartment, isolated from neighbors. Unlike traditional virtual machines, Hyperlight micro‑VMs are extremely lightweight and do not have a general-purpose guest operating system. Because of their small footprint, they provide a much more reduced attack surface with a hardware-backed isolation boundary between every instance of customer code, the host infrastructure, and other tenants. This model allows Azure Front Door to combine the safety properties of hardware virtualization with the performance characteristics required for edge compute. Customer code is isolated by design to reduce blast radius to a single instance, helping protect both the platform and neighboring workloads. How edge actions fit into the request path When a client request reaches Azure Front Door, it follows Azure Front Door’s standard routing pipeline. The request is first evaluated against any configured WAF policies and routing rules. If a rule is configured to invoke an edge action, execution is handed off to the edge actions runtime (Hyperlight) at the edge POP. At execution time, Azure Front Door provides the edge action with a constrained, immutable context that includes request metadata, server variables, origin health information, and geo or device signals. The edge action can then modify the request, generate a response, or influence routing decisions, all within strict execution and resource limits designed to preserve platform stability. Ship safely with versions and execution filters Edge actions are built for operational confidence. Each edge action supports multiple versions of code, and customers can control which version executes using execution filters. This enables header-based routing, canary deployments, and A/B testing scenarios without downtime. Teams can gradually introduce new logic, validate behavior with real traffic, and roll back instantly if needed, bringing modern DevOps practices directly to the edge. Observe and operate with confidence Edge actions integrate with Azure’s existing observability stack. Customers can emit logs from their edge action code and correlate execution results with Azure Front Door access and routing logs. This unified view simplifies troubleshooting and provides visibility into edge execution behavior in production environments. Pricing Please refer to the pricing document for details regarding Edge actions pricing, including applicable billing meters and usage-based charges on invocations and execution time. What’s next for edge actions The public preview of edge actions represents the foundation of a broader roadmap. Over time, Microsoft plans to expand invocation points, capabilities, and integrations - while continuing to prioritize security, performance, and operational simplicity. Stay tuned for scenarios like response invocations, image/video optimizations, and edge inferencing during general availability. By combining programmable edge logic with Hyperlight-based isolation, Azure Front Door edge actions mark a significant milestone in Microsoft’s edge strategy - enabling customers to build faster, safer, and more adaptive applications on a global scale. Start building with Azure Front Door edge actions today! Explore the documentation and bring your complex edge scenarios to life.Azure Front Door: Resiliency Series – Part 2: Faster recovery (RTO)
Abhishek Tiwari, Vice President of Engineering, Azure Networking Amit Srivastava, Partner Director of PM, Azure Networking Varun Chawla, Partner Director of Engineering, Azure Networking Karthik Uthaman, Principal Engineer, Azure Networking In Part 1 of this blog series, we outlined our four‑pillar strategy for resiliency in Azure Front Door: configuration resiliency, data plane resiliency, tenant isolation, and accelerated Recovery Time Objective (RTO). Together, these pillars help Azure Front Door remain continuously available and resilient at global scale. Part 1 focused on the first two pillars: configuration and data plane resiliency. Our goal is to make configuration propagation safer, so incompatible changes never escape pre‑production environments. We discussed how incompatible configurations are blocked early, and how data plane resiliency ensures the system continues serving traffic from a last‑known‑good (LKG) configuration even if a bad change manages to propagate. We also introduced ‘Food Taster’, a dedicated sacrificial process running in each edge server’s data plane, that pretests every configuration change in isolation, before it ever reaches the live data plane. In this post, we turn to the recovery pillar. We describe how we have made key enhancements to the Azure Front Door recovery path so the system can return to full operation in a predictable and bounded timeframe. For a global service like Azure Front Door, serving hundreds of thousands of tenants across 210+ edge sites worldwide, we set an explicit target: to be able to recover any edge site – or all edge sites – within approximately 10 minutes, even in worst‑case scenarios. In typical data plane crash scenarios, we expect recovery in under a second. Repair status The first blog post in this series mentioned the two Azure Front Door incidents from October 2025 – learn more by watching our Azure Incident Retrospective session recordings for the October 9 th incident and/or the October 29 th incident. Before diving into our platform investments for improving our Recovery Time Objectives (RTO), we wanted to provide a quick update on the overall repair items from these incidents. We are pleased to report that the work on configuration propagation and data plane resiliency is now complete and fully deployed across the platform (in the table below, “Completed” means broadly deployed in production). With this, we have reduced configuration propagation latency from ~45 minutes to ~20 minutes. We anticipate reducing this even further – to ~15 minutes by the end of April 2026, while ensuring that platform stability remains our top priority. Learning category Goal Repairs Status Safe customer configuration deployment Incompatible configuration never propagates beyond ‘EUAP or canary regions’ Control plane and data plane defect fixes Forced synchronous configuration processing Additional stages with extended bake time Early detection of crash state Completed Data plane resiliency Configuration processing cannot impact data plane availability Manage data-plane lifecycle to prevent outages caused by configuration-processing defects. Completed Isolated work-process in every data plane server to process and load the configuration. Completed 100% Azure Front Door resiliency posture for Microsoft internal services Microsoft operates an isolated, independent Active/Active fleet with automatic failover for critical Azure services Phase 1: Onboarded critical services batch impacted on Oct 29 th outage running on a day old configuration Completed Phase 2: Automation & hardening of operations, auto-failover and self-management of Azure Front Door onboarding for additional services March 2026 Recovery improvements Data plane crash recovery in under 10 minutes Data plane boot-up time optimized via local cache (~1 hour) Completed Accelerate recovery time < 10 minutes April 2026 Tenant isolation No configuration or traffic regression can impact other tenants Micro cellular Azure Front Door with ingress layered shards June 2026 Why recovery at edge scale is deceptively hard To understand why recovery took as long as it did, it helps to first understand how the Azure Front Door data plane processes configuration. Azure Front Door operates in 210+ edge sites with multiple servers per site. The data plane of each edge server hosts multiple processes. A master process orchestrates the lifecycle of multiple worker processes, that serve customer traffic. A separate configuration translator process runs alongside the data plane processes, and is responsible for converting customer configuration bundles from the control plane into optimized binary FlatBuffer files. This translation step, covering hundreds of thousands of tenants, represents hours of cumulative computation. A per edge server cache is kept locally at each server level – to enable a fast recovery of the data plane, if needed. Once the configuration translator process produces these FlatBuffer files, each worker processes them independently and memory-maps them for zero-copy access. Configuration updates flow through a two-phase commit: new FlatBuffers are first loaded into a staging area and validated, then atomically swapped into production maps. In-flight requests continue using the old configuration, until the last request referencing them completes. The data process recovery is designed to be resilient to different failure modes. A failure or crash at the worker process level has a typical recovery time of less than one second. Since each server has multiple such worker processes which serve customer traffic, this type of crash has no impact on the data plane. In the case of a master process crash, the system automatically tries to recover using the local cache. When the local cache is reused, the system is able to recover quickly – in approximately 60 minutes – since most of the configurations in the cache were already loaded into the data plane before the crash. However, in certain cases if the cache becomes unavailable or must be invalidated because of corruption, the recovery time increases significantly. During the October 29 th incident, a data plane crash triggered a complete recovery sequence that took approximately 4.5 hours. This was not because restarting a process is slow, it is because a defect in the recovery process invalidated the local cache, which meant that “restart” meant rebuilding everything from scratch. The configuration translator process then had to re-fetch and re-translate every one of the hundreds of thousands of customer configurations, before workers could memory-map them and begin serving traffic. This experience has crystallized three fundamental learnings related to our recovery path: Expensive rework: A subset of crashes discarded all previously translated FlatBuffer artifacts, forcing the configuration translator process to repeat hours of conversion work that had already been validated and stored. High restart costs: Every worker on every node had to wait for the configuration translator process to complete the full translation, before it could memory-map any configuration and begin serving requests. Unbounded recovery time: Recovery time grew linearly with total tenant footprint rather than with active traffic, creating a ‘scale penalty’ as more tenants onboarded to the system. Separately and together, the insight was clear: recovery must stop being proportional to the total configuration size. Persisting ‘validated configurations’ across restarts One of the key recovery improvements was strengthening how validated customer configurations are cached and reused across failures, rather than rebuilding configuration states from scratch during recovery. Azure Front Door already cached customer configurations on host‑mounted storage prior to the October incident. The platform enhancements post outage focused on making the local configuration cache resilient to crashes, partial failures, and bad tenant inputs. Our goal was to ensure that recovery behavior is dominated by serving traffic safely, not by reconstructing configuration state. This led us to two explicit design goals… Design goals No category of crash should invalidate the configuration cache: Configuration cache invalidation must never be the default response to failures. Whether the failure is a worker crash, master crash, data plane restart, or coordinated recovery action, previously validated customer configurations should remain usable—unless there is a proven reason to discard it. Bad tenant configuration must not poison the entire cache: A single faulty or incompatible tenant configuration should result in targeted eviction of that tenant’s configuration only—not wholesale cache invalidation across all tenants. Platform enhancements Previously, customer configurations persisted to host‑mounted storage, but certain failure paths treated the cache as unsafe and invalidated it entirely. In those cases, recovery implicitly meant reloading and reprocessing configuration for hundreds of thousands of tenants before traffic could resume, even though the vast majority of cached data was still valid. We changed the recovery model to avoid invalidating customer configurations, with strict scoping around when and how cached entries are discarded: Cached configurations are no longer invalidated based on crash type. Failures are assumed to be orthogonal to configuration correctness unless explicitly proven otherwise. Cache eviction is granular and tenant‑scoped. If a cached configuration fails validation or load checks, only that tenant’s configuration is discarded and reloaded. All other tenant configurations remain available. This ensures that recovery does not regress into a fleet‑wide rebuild due to localized or unrelated faults. Safety and correctness Durability is paired with strong correctness controls, to prevent unsafe configurations from being served: Per‑tenant validation on load: Each cached tenant configuration is validated during the ‘load and verification’ phase, before being promoted for traffic serving. Therefore, failures are contained to that tenant. Targeted re‑translation: When validation fails, only the affected tenant’s configuration is reloaded or reprocessed. Therefore, the cache for other tenants is left untouched. Operational escape hatch: Operators retain the ability to explicitly instruct a clean rebuild of the configuration cache (with proper authorization), preserving control without compromising the default fast‑recovery path. Resulting behavior With these changes, recovery behavior now aligns with real‑world traffic patterns - configuration defects impact tenants locally and predictably, rather than globally. The system now prefers isolated tenant impact, and continued service using last-known-good over aggressive invalidation, both of which are critical for predictable recovery at the scale of Azure Front Door. Making recovery scale with active traffic, not total tenants Reusing configuration cache solves the problem of rebuilding configuration in its entirety, but even with a warm cache, the original startup path had a second bottleneck: eagerly loading a large volume of tenant configurations into memory before serving any traffic. At our scale, memory-mapping, parsing hundreds of thousands of FlatBuffers, constructing internal lookup maps, adding Transport Layer Security (TLS) certificates and configuration blocks for each tenant, collectively added almost an hour to startup time. This was the case even when a majority of those tenants had no active traffic at that moment. We addressed this by fundamentally changing when configuration is loaded into workers. Rather than eagerly loading most of the tenants at startup across all edge locations, Azure Front Door now uses an Machine Learning (ML)-optimized lazy loading model. In the new architecture, instead of loading a large number of tenant configurations, we only load a small subset of tenants that are known to be historically active in a given site, we call this the “warm tenants” list. The warm tenants list per edge site is created through a sophisticated traffic analysis pipeline that leverages ML. However, loading the warm tenants is not good enough, because when a request arrives and we don’t have the configuration in memory, we need to know two things. Firstly, is this a request from a real Azure Front Door tenant – and, if it is, where can I find the configuration? To answer these questions, each worker maintains a hostmap that tracks the state of each tenant’s configuration. This hostmap is constructed during startup, as we process each tenant configuration – if the tenant is in the warm list, we will process and load their configuration fully; if not, then we will just add an entry into the hostmap where all their domain names are mapped to the configuration path location. When a request arrives for one of these tenants, the worker loads and validates that tenant’s configuration on demand, and immediately begins serving traffic. This allows a node to start serving its busiest tenants within a few minutes of startup, while additional tenants are loaded incrementally only when traffic actually arrives—allowing the system to progressively absorb cold tenants as demand increases. The effect on recovery is transformative. Instead of recovery time scaling with the total number of tenants configured on a server, it scales with the number of tenants actively receiving traffic. In practice, even at our busiest edge sites, the active tenant set is a small fraction of the total. Just as importantly, this modified form of lazy loading provides a natural failure isolation boundary. Most Edge sites won’t ever load a faulty configuration of an inactive tenant. When a request for an inactive tenant with an incompatible configuration arrives, impact is contained to a single worker. The configuration load architecture now prefers serving as many customers as quickly as possible, rather than waiting until everything is ready before serving anyone. The above changes are slated to complete in April 2026 and will bring our RTO from the current ~1 hour to under 10 minutes – for complete recovery from a worst case scenario. Continuous validation through Game Days A critical element of our recovery confidence comes from GameDay fault-injection testing. We don’t simply design recovery mechanisms and assume they work—we break the system deliberately and observe how it responds. Since late 2025, we have conducted recurring GameDay drills that simulate the exact failure scenarios we are defending against: Food Taster crash scenarios: Injecting deliberately faulty tenant configurations, to verify that they are caught and isolated with zero impact on live traffic. In our January 2026 GameDay, the Food Taster process crashed as expected, the system halted the update within approximately 5 seconds, and no customer traffic was affected. Master process crash scenarios: Triggering master process crashes across test environments to verify that workers continue serving traffic, that the Local Config Shield engages within 10 seconds, and that the coordinated recovery tool restores full operation within the expected timeframe. Multi-region failure drills: Simulating simultaneous failures across multiple regions to validate that global Config Shield mechanisms engage correctly, and that recovery procedures scale without requiring manual per-region intervention. Fallback test drills for critical Azure services running behind Azure Front Door: In our February 2026 GameDay, we simulated the complete unavailability of Azure Front Door, and successfully validated failover for critical Azure services with no impact to traffic. These drills have both surfaced corner cases and built operational confidence. They have transformed recovery from a theoretical plan into tested, repeatable muscle memory. As we noted in an internal communication to our team: “Game day testing is a deliberate shift from assuming resilience to actively proving it—turning reliability into an observed and repeatable outcome.” Closing Part 1 of this series emphasized preventing unsafe configurations from reaching the data plane, and data plane resiliency in case an incompatible configuration reaches production. This post has shown that prevention alone is not enough—when failures do occur, recovery must be fast, predictable, and bounded. By ensuring that the FlatBuffer cache is never invalidated, by loading only active tenants, and by building safe coordinated recovery tooling, we have transformed failure handling from a fleet-wide crisis into a controlled operation. These recovery investments work in concert with the prevention mechanisms described in Part 1. Together, they ensure that the path from incident detection to full service restoration is measured in minutes, with customer traffic protected at every step. In the next post of this series, we will cover the third pillar of our resiliency strategy: tenant isolation—how micro-cellular architecture and ingress-layered sharding can reduce the blast radius of any failure to a small subset, ensuring that one customer’s configuration or traffic anomaly never becomes everyone’s problem. We deeply value our customers’ trust in Azure Front Door. We are committed to transparently sharing our progress on these resiliency investments, and to exceed expectations for safety, reliability, and operational readiness.Azure Front Door: Implementing lessons learned following October outages
Abhishek Tiwari, Vice President of Engineering, Azure Networking Amit Srivastava, Principal PM Manager, Azure Networking Varun Chawla, Partner Director of Engineering Link to Part 2 - Azure Front Door Resiliency Series Link to Part 3 - Azure Front Door Resiliency Series Introduction Azure Front Door is Microsoft's advanced edge delivery platform encompassing Content Delivery Network (CDN), global security and traffic distribution into a single unified offering. By using Microsoft's extensive global edge network, Azure Front Door ensures efficient content delivery and advanced security through 210+ global and local points of presence (PoPs) strategically positioned closely to both end users and applications. As the central global entry point from the internet onto customer applications, we power mission critical customer applications as well as many of Microsoft’s internal services. We have a highly distributed resilient architecture, which protects against failures at the server, rack, site and even at the regional level. This resiliency is achieved by the use of our intelligent traffic management layer which monitors failures and load balances traffic at server, rack or edge sites level within the primary ring, supplemented by a secondary-fallback ring which accepts traffic in case of primary traffic overflow or broad regional failures. We also deploy a traffic shield as a terminal safety net to ensure that in the event of a managed or unmanaged edge site going offline, end user traffic continues to flow to the next available edge site. Like any large-scale CDN, we deploy each customer configuration across a globally distributed edge fleet, densely shared with thousands of other tenants. While this architecture enables global scale, it carries the risk that certain incompatible configurations, if not contained, can propagate broadly and quickly which can result in a large blast radius of impact. Here we describe how the two recent service incidents impacting Azure Front Door have reinforced the need to accelerate ongoing investments in hardening our resiliency, and tenant isolation strategy to mitigate likelihood and the scale of impact from this class of risk. October incidents: recap and key learnings Azure Front Door experienced two service incidents; on October 9 th and October 29 th , both with customer-impacting service degradation. On October 9 th : A manual cleanup of stuck tenant metadata bypassed our configuration protection layer, allowing incompatible metadata to propagate beyond our canary edge sites. This metadata was created on October 7 th , from a control-plane defect triggered by a customer configuration change. While the protection system initially blocked the propagation, the manual override operation bypassed our safeguards. This incompatible configuration reached the next stage and activated a latent data-plane defect in a subset of edge sites, causing availability impact primarily across Europe (~6%) and Africa (~16%). You can learn more about this issue in detail at https://aka.ms/AIR/QNBQ-5W8 On October 29 th : A different sequence of configuration changes across two control-plane versions produced incompatible metadata. Because the failure mode in the data-plane was asynchronous, the health checks validations embedded in our protection systems were all passed during the rollout. The incompatible customer configuration metadata successfully propagated globally through a staged rollout and also updated the “last known good” (LKG) snapshot. Following this global rollout, the asynchronous process in data-plane exposed another defect which caused crashes. This impacted connectivity and DNS resolutions for all applications onboarded to our platform. Extended recovery time amplified impact on customer applications and Microsoft services. You can learn more about this issue in detail at https://aka.ms/AIR/YKYN-BWZ We took away a number of clear and actionable lessons from these incidents, which are applicable not just to our service, but to any multi-tenant, high-density, globally distributed system. Configuration resiliency – Valid configuration updates should propagate safely, consistently, and predictably across our global edge, while ensuring that incompatible or erroneous configuration never propagate beyond canary environments. Data plane resiliency - Additionally, configuration processing in the data plane must not cause availability impact to any customer. Tenant isolation – Traditional isolation techniques such as hardware partitioning and virtualization are impractical at edge sites. This requires innovative sharding techniques to ensure single tenant-level isolation – a must-have to reduce potential blast radius. Accelerated and automated recovery time objective (RTO) – System should be able to automatically revert to last known good configuration in an acceptable RTO. In case of a service like Azure Front Door, we deem ~10 mins to be a practical RTO for our hundreds of thousands of customers at every edge site. Post outage, given the severity of impact which allowed an incompatible configuration to propagate globally, we made the difficult decision to temporarily block configuration changes in order to expedite rollout of additional safeguards. Between October 29 th to November 5 th , we prioritized and deployed immediate hardening steps before opening up the configuration change. We are confident that the system is stable, and we are continuing to invest in additional safeguards to further strengthen the platform's resiliency. Learning category Goal Repairs Status Safe customer configuration deployment Incompatible configuration never propagates beyond Canary · Control plane and data plane defect fixes · Forced synchronous configuration processing · Additional stages with extended bake time · Early detection of crash state Completed Data plane resiliency Configuration processing cannot impact data plane availability Manage data-plane lifecycle to prevent outages caused by configuration-processing defects. Completed Isolated work-process in every data plane server to process and load the configuration. January 2026 100% Azure Front Door resiliency posture for Microsoft internal services Microsoft operates an isolated, independent Active/Active fleet with automatic failover for critical Azure services Phase 1: Onboarded critical services batch impacted on Oct 29 th outage running on a day old configuration Completed Phase 2: Automation & hardening of operations, auto-failover and self-management of Azure Front Door onboarding for additional services March 2026 Recovery improvements Data plane crash recovery in under 10 minutes Data plane boot-up time optimized via local cache (~1 hour) Completed Accelerate recovery time < 10 minutes March 2026 Tenant isolation No configuration or traffic regression can impact other tenants Micro cellular Azure Front Door with ingress layered shards June 2026 This blog is the first in a multi-part series on Azure Front Door resiliency. In this blog, we will focus on configuration resiliency—how we are making the configuration pipeline safer and more robust. Subsequent blogs will cover tenant isolation and recovery improvements. How our configuration propagation works Azure Front Door configuration changes can be broadly classified into three distinct categories. Service code & data – these include all aspects of Azure Front Door service like management plane, control plane, data plane, configuration propagation system. Azure Front Door follows a safe deployment practice (SDP) process to roll out newer versions of management, control or data plane over a period of approximately 2-3 weeks. This ensures that any regression in software does not have a global impact. However, latent bugs that escape pre-validation and SDP rollout can remain undetected until a specific combination of customer traffic patterns or configuration changes trigger the issue. Web Application Firewall (WAF) & L7 DDoS platform data – These datasets are used by Azure Front Door to deliver security and load-balancing capabilities. Examples include GeoIP data, malicious attack signatures, and IP reputation signatures. Updates to these datasets occur daily through multiple SDP stages with an extended bake time of over 12 hours to minimize the risk of global impact during rollout. This dataset is shared across all customers and the platform, and it is validated immediately since it does not depend on variations in customer traffic or configuration steps. Customer configuration data – Examples of these are any customer configuration change—whether a routing rule update, backend pool modification, WAF rule change, or security policy change. Due to the nature of these changes, it is expected across the edge delivery / CDN industry to propagate these changes globally in 5-10 mins. Both outages stemmed from issues within this category. All configuration changes, including customer configuration data, are processed through a multi-stage pipeline designed to ensure correctness before global rollout across Azure Front Door’s 200+ edge locations. At a high level, Azure Front Door’s configuration propagation system has two distinct components - Control plane – Accepts customer API/portal changes (create/update/delete for profiles, routes, WAF policies, origins, etc.) and translates them into internal configuration metadata which the data plane can understand. Data plane – Globally distributed edge servers that terminate client traffic, apply routing/WAF logic, and proxy to origins using the configuration produced by the control plane. Between these two halves sits a multi-stage configuration rollout pipeline with a dedicated protection system (known as ConfigShield): Changes flow through multiple stages (pre-canary, canary, expanding waves to production) rather than going global at once. Each stage is health-gated: the data plane must remain within strict error and latency thresholds before proceeding. Each stage’s health check also rechecks previous stage’s health for any regressions. A successfully completed rollout updates a last known good (LKG) snapshot used for automated rollback. Historically, rollout targeted global completion in roughly 5–10 minutes, in line with industry standards. Customer configuration processing in Azure Front Door data plane stack Customer configuration changes in Azure Front Door traverse multiple layers—from the control plane through the deployment system—before being converted into FlatBuffers at each Azure Front Door node. These FlatBuffers are then loaded by the Azure Front Door data plane stack, which runs as Kubernetes pods on every node. FlatBuffer Composition: Each FlatBuffer references several sub-resources such as WAF and Rules Engine schematic files, SSL certificate objects, and URL signing secrets. Data plane architecture: o Master process: Accepts configuration changes (memory-mapped files with references) and manages the lifecycle of worker processes. o Workers: L7 proxy processes that serve customer traffic using the applied configuration. Processing flow for each configuration update: Load and apply in master: The transformed configuration is loaded and applied in the master process. Cleanup of unused references occurs synchronously except for certain categories à October 9 outage occurred during this step due to a crash triggered by incompatible metadata. Apply to workers: Configuration is applied to all worker processes without memory overhead (FlatBuffers are memory-mapped). Serve traffic: Workers start consuming new FlatBuffers for new requests; in-flight requests continue using old buffers. Old buffers are queued for cleanup post-completion. Feedback to deployment service: Positive feedback signals readiness for rollout.Cleanup: FlatBuffers are freed asynchronously by the master process after all workers load updates à October 29 outage occurred during this step due to a latent bug in reference counting logic. The October incidents showed we needed to strengthen key aspects of configuration validation, propagation safeguards, and runtime behavior. During the Azure Front Door incident on October 9 th , that protection system worked as intended but was later bypassed by our engineering team during a manual cleanup operation. During this Azure Front Door incident on October 29 th , the incompatible customer configuration metadata progressed through the protection system, before the delayed asynchronous processing task resulted in the crash. Configuration propagation safeguards Based on learnings from the incidents, we are implementing a comprehensive set of configuration resiliency improvements. These changes aim to guarantee that any sequence of configuration changes cannot trigger instability in the data plane, and to ensure quicker recovery in the event of anomalies. Strengthening configuration generation safety This improvement pivots on a ‘shift-left’ strategy where we want to ensure that we catch regression early before they propagate to production. It also includes fixing the latent defects which were the proximate cause of the outage. Fixing outage specific defects - We have fixed the control-plane defects that could generate incompatible tenant metadata under specific operation sequences. We have also remediated the associated data-plane defects. Stronger cross-version validation - We are expanding our test and validation suite to account for changes across multiple control plane build versions. This is expected to be fully completed by February 2026. Fuzz testing - Automated fuzzing and testing of metadata generation contract between the control plane and the data plane. This allows us to generate an expanded set of invalid/unexpected configuration combinations which might not be achievable by traditional test cases alone. This is expected to be fully completed by February 2026. Preventing incompatible configurations from being propagated This segment of the resiliency strategy strives to ensure that a potentially dangerous configuration change never propagates beyond canary stage. Protection system is “always-on” - Enhancements to operational procedures and tooling prevent bypass in all scenarios (including internal cleanup/maintenance), and any cleanup must flow through the same guarded stages and health checks as standard configuration changes. This is completed. Making rollout behavior more predictable and conservative - Configuration processing in the data plane is now fully synchronous. Every data plane issue due to incompatible meta data can be detected withing 10 seconds at every stage. This is completed. Enhancement to deployment pipeline - Additional stages during roll-out and extended bake time between stages serve as an additional safeguard during configuration propagation. This is completed. Recovery tool improvements now make it easier to revert to any previous version of LKG with a single click. This is completed. These changes significantly improve system safety. Post-outage we have increased the configuration propagation time to approximately 45 minutes. We are working towards reducing configuration propagation time closer to pre-incident levels once additional safeguards covered in the Data plane resiliency section below are completed by mid-January, 2026. Data plane resiliency The data plane recovery was the toughest part of recovery efforts during the October incidents. We must ensure fast recovery as well as resilience to configuration processing related issues for the data plane. To address this, we implemented changes that decouple the data plane from incompatible configuration changes. With these enhancements, the data plane continues operating on the last known good configuration—even if the configuration pipeline safeguards fail to protect as intended. Decoupling data plane from configuration changes Each server’s data plane consists of a master process which accepts configuration changes and manages lifecycle of multiple worker processes which serve customer traffic. One of the critical reasons for the prolonged outage in October was that due to latent defects in the data plane, when presented with a bad configuration the master process crashed. The master is a critical command-and-control process and when it crashes it takes down the entire data plane, in that node. Recovery of the master process involves reloading hundreds of thousands of configurations from scratch and took approximately 4.5 hours. We have since made changes to the system to ensure that even in the event of the master process crash due to any reason - including incompatible configuration data being presented - the workers remain healthy and able to serve traffic. During such an event, the workers would not be able to accept new configuration changes but will continue to serve customer traffic using the last known good configuration. This work is completed. Introducing Food Taster: strengthening config propagation resiliency In our efforts to further strengthen Azure Front Door’s configuration propagation system, we are introducing an additional configuration safeguard known internally as Food Taster which protects the master and worker processes from any configuration change related incidents, thereby ensuring data plane resiliency. The principle is simple: every data-plane server will have a redundant and isolated process – the Food Taster – whose only job is to ingest and process new configuration metadata first and then pass validated configuration changes to active data plane. This redundant worker does not accept any customer traffic. All configuration processing in this Food Taster is fully synchronous. That means we do all parsing, validation, and any expensive or risky work up front, and we do not move on until the Food Taster has either proven the configuration is safe or rejected it. Only when the Food Taster successfully loads the configuration and returns “Config OK” does the master process proceed to load the same config and then instruct the worker processes to do the same. If anything goes wrong in the Food Taster, the failure is contained to that isolated worker; the master and traffic-serving workers never see that invalid configuration. We expect this safeguard to reach production globally in January 2026 timeframe. Introduction of this component will also allow us to return closer to pre-incident level of configuration propagation while ensuring data plane safety. Closing This is the first in a series of planned blogs on Azure Front Door resiliency enhancements. We are continuously improving platform safety and reliability and will transparently share updates through this series. Upcoming posts will cover advancements in tenant isolation and improvements to recovery time objectives (RTO). We deeply value our customers’ trust in Azure Front Door. The October incidents reinforced how critical configuration resiliency is, and we are committed to exceeding industry expectations for safety, reliability, and transparency. By hardening our configuration pipeline, strengthening safety gates, and reinforcing isolation boundaries, we’re making Azure Front Door even more resilient so your applications can be too.
