Secure Native Access to Azure Kubernetes Service (AKS) Private Clusters with Azure Bastion
Written in Collaboration with AvirupChat ShabazShaik Mohit_Kumar YuriDiogenes Introduction: As organizations move toward containerized workloads such as Azure Kubernetes Service (AKS), secure access to cluster resources becomes critical. Making the Kubernetes API server private, therefore, significantly reduces the attack surface. However, it also changes how engineers perform routine management tasks. Operations as simple as running kubectl logs, describing resources, or troubleshooting workloads require connectivity to the virtual network hosting the cluster. This challenge becomes even more apparent during day-to-day operations. An engineer may have the necessary Kubernetes credentials and Azure RBAC permissions yet still be unable to access the cluster because the API server is reachable only from within the private network. Establishing VPN connectivity, using jump hosts, or deploying dedicated management workstations often becomes part of the operational workflow, adding complexity to what should be straightforward administrative tasks. This balance between maintaining strong network isolation and enabling efficient cluster management is exactly what Azure Bastion's native client tunneling support for private AKS clusters is designed to address. Many Azure customers already use Azure Bastion to securely access virtual machines without exposing them to the public internet. Native client tunneling now brings that same secure access model to AKS private clusters. This capability is in public preview today. Before discussing how it works, it is worth understanding the operational challenges it is designed to solve and how it's done. Bastion for Azure Kubernetes Service (AKS) private clusters: At its core, Azure Bastion is designed to simplify secure access to private Azure resources. Since its launch, it has provided browser-based and native client RDP and SSH connectivity to Azure VMs without exposing management ports to the internet. No public IPs on your VMs. No inbound NSG rules for port 22 or 3389. Just TLS over port 443, routed through a fully managed service that Microsoft patches, scales, and secures on your behalf. When you establish a Bastion tunnel to your private AKS cluster, you are not signing in to an intermediary machine and running kubectl from there. You are opening an encrypted tunnel from your local machine, through Bastion, directly to the private API server endpoint. Your kubeconfig points to localhost on a dynamically assigned port, so local kubectl, helm, and scripts continue to work as they would against a public cluster. You get the developer experience of a public cluster with the security posture of a private one as shown in Figure 1. The high-level flow has three parts: The engineer authenticates to Azure, retrieves cluster credentials, and uses Azure Bastion to establish a managed tunnel into the private network. Local Kubernetes tools then use that tunnel to reach the private API server. This keeps the workflow familiar while removing the need for jump hosts or VPN-based access paths. However, this raises an important question - Does making access easier also make it easier for an attacker to connect? The short answer is no. In fact, Bastion tunneling strengthens the security posture in ways that are worth unpacking. First, the API server itself remains private. There is no public endpoint. There is no public IP address discoverable by scanners. The only network path to the API server runs through Azure's managed infrastructure. Authentication and authorization then depend on the AKS cluster configuration. Second, Bastion eliminates an entire class of infrastructure that itself becomes a security target. Jump box VMs, when not meticulously maintained, accumulate credentials, kubeconfig files, and browser sessions. They are machines that admins log into, which means they are machines that can be compromised. Bastion is not a machine you log into. It is an agentless, managed tunnel you pass through. And for public clusters, there is a related but distinct benefit. Many teams use API server authorized IP ranges to restrict which source IPs can reach their public API endpoint. This is a good practice, but it breaks down quickly when your team works remotely, uses dynamic IPs, or includes contractors. Adding Bastion's stable public IP to the authorized range gives you a consistent, managed access path without the operational toil of constantly updating IP allow lists. With the network path established through Bastion, the next question is how users are authenticated and authorized once they reach the AKS API server. That distinction matters: Bastion provides secure connectivity, while AKS access is governed by the authentication and authorization model configured on the cluster. Authentication and authorization options: When creating an AKS cluster, you can choose from three authentication and authorization modes, as shown in Figure below. That choice shapes everything about how access is granted, audited, and governed over the lifetime of the cluster. ⚠️Local accounts with Kubernetes RBAC is the default if you change nothing. It uses a static certificate that never expires, is shared across all administrators, and has no connection to your corporate identity provider. There is no MFA. No Conditional Access. Note: For most production environments, Microsoft recommends using Microsoft Entra ID integrated authentication rather than local accounts to enable centralized identity, MFA, and auditing. ☑️ Microsoft Entra ID with Kubernetes RBAC moves authentication to Entra ID, which means users sign in with their corporate identity, MFA is enforced, and Conditional Access policies apply. Authorization is handled through native Kubernetes Role and ClusterRole bindings, which reference Entra ID users and groups as subjects. This works well for teams that manage cluster configuration through GitOps, because RBAC manifests live alongside other cluster YAML. The limitation is that these permissions are invisible in Azure IAM — they live only inside the cluster. ✅ Microsoft Entra ID with Azure RBAC is the model we recommend for most production environments. Authentication still flows through Entra ID with full MFA and Conditional Access support. But authorization is handled by Azure RBAC role assignments on the AKS resource itself. Permissions are visible in Azure IAM, participate in access reviews, and integrate with Privileged Identity Management for just-in-time elevation. You can assign the built-in Azure Kubernetes Service RBAC Cluster Admin, Admin, Writer, or Reader roles at either cluster scope or namespace scope. A single subscription-level role assignment can grant access to every cluster in the subscription. The real value is the combination. Bastion provides the encrypted network tunnel. Entra ID provides the identity, MFA, and Conditional Access. Azure RBAC provides centralized, auditable authorization that your security team can review alongside every other Azure resource. For the exact steps to configure Entra ID authentication and Azure RBAC on your AKS cluster, see the AKS identity documentation. End-to-end flow for connectivity to AKS via Bastion: az account set --subscription <subscription ID> Retrieve credentials to your AKS private cluster using the commands below: az aks get-credentials --name <AKSClusterName> --resource-group <ResourceGroupName> Open the tunnel to your target AKS Cluster with the following command: az aks bastion --name <aksClusterName> --resource-group <aksClusterResourceGroup> --bastion <bastionResourceId> Now the default authentication method is Device code authentication i.e., this authentication method prompts the device code for the user to sign in from a browser session. If you want CLI only authentication, you can run the following command next. kubelogin convert-kubeconfig -l azurecli Then go on with your AKS connectivity: kubectl get nodes How to Try It: If you are already running private AKS clusters, getting started is straightforward. You need a Standard or Premium Azure Bastion host deployed in the same VNet as your cluster (or in a peered VNet), with native client support enabled. The aks-preview and bastion CLI extensions handle the rest. The detailed connection steps are documented on Microsoft Learn. We Want Your Feedback Write a comment in this blog or open an issue on the AKS GitHub repository or leave feedback directly on the Microsoft Learn documentation page. We read everything, and it genuinely influences what we prioritize.Azure Front Door: Resiliency Series – Part 3: Tenant isolation
Abhishek Tiwari, Vice President of Engineering, Azure Networking Amit Srivastava, Partner Director of PM, Azure Networking Varun Chawla, Partner Director of Engineering, Azure Networking Azure Front Door serves hundreds of thousands of tenants from hundreds of edge locations, densely sharing the globally distributed edge fleet. That density is exactly what allows us to deliver global scale, performance, and cost efficiency. It also means that, without strong isolation, a single tenant’s incompatible configuration or anomalous traffic can, in the worst case, affect many other tenants. The October 2025 incidents reinforced how important it is to contain this class of risk. Our goal for this tenant isolation is simple to state and hard to achieve: no single tenant’s configuration or traffic should be able to impact any other tenant. In Part 1 of our three part mini blog series, we outlined our four‑pillar strategy for improving the resiliency of Azure Front Door: configuration resiliency, data plane resiliency, tenant isolation, and accelerated Recovery Time Objective (RTO). Part 1 detailed how we would make configuration propagation safer and how the data plane keeps serving from a ‘last‑known‑good’ (LKG) configuration, even if an incompatible configuration change is propagated to the data plane. Part 2 turned to recovery, showing how we bring the system back to full operation in an accelerated, predictable, and in a bounded timeframe. In this final part, we turn to the tenant isolation pillar that ensures that any single tenant configuration or traffic issues are limited in scope to that tenant alone and do not impact other tenants. We will also show how Azure Front Door achieves single-tenant containment through configuration isolation, lazy loading, and a micro-cellular layered ingress-sharding architecture. Repair status: all outstanding items now complete Before we dive into tenant isolation, here is a final update on the overall repair items from the two October 2025 incidents (you can review the details in our Azure Incident Retrospective sessions for the October 9th and October 29th incidents). We are pleased to report that all outstanding work across every pillar is now complete and fully deployed in production – including the tenant isolation work described in this post. With these safeguards in place, we have also returned configuration propagation latency to pre‑incident levels while keeping platform stability our top priority. In the table below, “Completed” means broadly deployed in production. Learning category Goal Repairs Status Safe customer configuration deployment Incompatible configuration never propagates beyond ‘EUAP or canary regions’ Control plane and data plane defect fixes; forced synchronous configuration processing; additional stages with extended bake time; early detection of crash state Completed Data plane resiliency Configuration processing cannot impact data plane availability Manage data‑plane lifecycle to prevent outages caused by configuration‑processing defects; isolated work‑process in every data plane server to process and load the configuration Completed 100% Azure Front Door resiliency posture for Microsoft internal services Microsoft operates an isolated, independent Active/Active fleet with automatic failover for critical Azure services Phase 1: onboarded critical services batch impacted on Oct 29th outage running on a day‑old configuration; Phase 2: automation & hardening of operations, auto‑failover and self‑management of onboarding for additional services Completed Recovery improvements Data plane crash recovery in under 10 minutes Data plane boot‑up time optimized via local cache; recovery time accelerated to under 10 minutes Completed Tenant isolation No configuration or traffic regression can impact other tenants Micro‑cellular Azure Front Door with ingress layered shards Completed Why isolation at edge scale is deceptively hard Traditional isolation techniques such as dedicating separate hardware to each tenant or running every tenant inside its own virtual machine are impractical at the edge. Edge sites are constrained on space, power, and capacity. The entire premise of a modern, multi-tenant application delivery platform is that any tenant can be served from any site closest to the user. We cannot simply partition hundreds of thousands of tenants onto dedicated machines without giving up either proximity, scale, or the efficiency that make the edge fast and cost efficient. Isolation therefore, must be achieved in software, inside a multi-tenant fleet. What we already do today Azure Front Door already includes several layers of tenant isolation and partitioning. However, the incidents in October clearly highlighted that these techniques were not enough. Prior to the incidents, our protection mechanisms included: Infrastructure partitioning. Edge sites were organized into physically isolated primary and fallback traffic rings. Noisy‑neighbor protection. Fair‑share resource allocation, rate limiting, and anomaly-based load protection kept any single tenant from monopolizing shared resources such as CPU, memory, or network bandwidth on an Azure Front Door server. Circuit breakers. Circuit breakers shed costly work first and can disable a risky per‑tenant feature before it exhausts shared resources on a server. Real‑time crash protection. A crash‑analysis system correlates crash signatures across machines and can pinpoint and block crash patterns caused by tenant IPs or traffic patterns. While these protections are valuable, many of them are reactive and proved insufficient during the October incidents. The next generation of isolation makes single‑tenant containment a fundamental part of the platform which governs how configurations are loaded, and how traffic is served. Configuration isolation: loading only what is needed Part 2 introduced ‘lazy loading’ as a recovery optimization technique. It is also an important configuration‑isolation mechanism. Historically, every worker on every edge server had to be ready to serve any tenant, which meant each worker loaded a large set of tenant configurations. A single incompatible configuration could therefore ripple across many workers. With lazy loading, a worker loads a tenant’s configuration and its TLS certificates only when it actually receives traffic for that tenant. The practical consequence for isolation is powerful: a faulty configuration can only affect the workers that have loaded that specific tenant, never the entire server or fleet. Combined with per‑tenant validation on load, and the Food Taster safeguard from Part 1 (a sacrificial process that pretests every configuration change in isolation), configuration problems are caught early and contained to the smallest possible footprint. Figure 1: With lazy loading, an incompatible configuration is contained to the workers serving that tenant, instead of poisoning the whole server. Tenant isolation: a micro-cellular, layered ingress sharding architecture Configuration isolation limits the blast radius of an incompatible configuration. Traffic isolation addresses the other half of the problem: a tenant whose anomalous traffic incident, like a sudden surge, a pathological request pattern, or malicious activity, could degrade a shared worker. Our approach is a micro‑cellular architecture that combines multiple concepts working together. Worker‑process isolation. Each edge server already runs many independent worker processes. Instead of letting every worker serve every tenant, we assign tenants to specific groups of workers. Those worker group (shards) become the unit of isolation: if a tenant destabilizes its shard, the impact is contained to that shard’s workers while the rest of the server keeps serving normally. Ingress sharding. Rather than a handful of fixed shards, we compose shards from overlapping subsets of a server’s workers. Even a modest number of workers can be combined into an enormous number of distinct, overlapping shards – giving us a very large number of fine‑grained fault domains without dedicating hardware to every tenant. Figure 2: Tenants are randomly assigned to different shards on each server (layer). Even when a good tenant shares the noisy tenant’s shard on one layer, routing steers its traffic to healthy shards on the others. Multi‑layer ingress sharding. This is where ‘layered’ comes in. Each edge server is treated as an independent layer, and each tenant is assigned to a different, randomly chosen shard on every server. Because assignments are independent from one server to the next, two tenants that happen to share a shard on one server are extremely unlikely to share a shard again on another server. The chance of any good tenant repeatedly colliding with a noisy tenant across many servers becomes vanishingly small. Intelligent ingress routing. Tying it together is a routing layer that terminates each incoming connection, identifies the tenant, and steers the request to that tenant’s assigned, healthy shard. If a shard is unhealthy or saturated, traffic is directed to the tenant’s healthy shards on other layers. Figure 3: Intelligent Ingress Routing The combined effect is that when a noisy tenant overwhelms or crashes its shard on one server, only that shard is affected. Because every other tenant is spread across a different, randomized set of shards, they continue to find healthy paths, and the routing layer moves their traffic accordingly. A worst-case availability problem is downgraded to, at most, a small and redistributable capacity problem, that the routing layer smooths over. An in-depth technical analysis of layered ingress sharding is available here for reference. Shrinking the blast radius Taken together, these mechanisms fundamentally change the shape of failure. In a uniform, fully shared fleet, an incompatible tenant can, in the worst case, affect a large share of the tenants on a machine, in an edge location, or beyond. With configuration isolation and layered ingress sharding, the same failure is confined to a subset of workers serving the offending tenant. Our target for this tenant isolation pillar is effectively single‑tenant containment: a configuration or traffic anomaly caused by one tenant should never cause issues to any other tenants. Figure 4: From a shared fleet where a single tenant can affect many, to micro‑cellular shards that confine impact to the offending tenant. Validating isolation in practice As with our recovery work, we don’t simply design these boundaries and assume they hold, we test them. Through deliberate fault‑injections, we have pushed noisy and faulty tenants into the system and confirmed that impact stayed contained to the offending shard, that healthy tenants kept serving, and that the routing layer steered around unhealthy shards as intended. This turns isolation from a design claim into a well-drilled, and repeatable outcome. Closing This post concludes our three-part mini blog series on Azure Front Door resiliency. We have shared how we are making configuration propagation safer (Part 1), recovering faster when failures do occur (Part 2), and containing the blast radius stemming from any single tenant through configuration isolation and a micro‑cellular, layered‑sharding architecture (Part 3). Resiliency, however, is not a project with an end date. It is an ongoing commitment. While this series concludes the blog series of our response to the October 2025 incidents, our investments in Azure Front Door’s resiliency, isolation, and recovery will continue. As we make further improvements, we will keep sharing them with you. We deeply value our customers’ trust in Azure Front Door. We remain committed to exceeding expectations for security, reliability, and transparency.
Advanced Container Apps Networking: VNet Integration and Centralized Firewall Traffic Logging
Azure community, I recently documented a networking scenario relevant to Azure Container Apps environments where you need to control and inspect application traffic using a third-party network virtual appliance. The article walks through a practical deployment pattern: • Integrate your Azure Container Apps environment with a Virtual Network. • Configure user-defined routes (UDRs) so that traffic from your container workloads is directed toward a firewall appliance before reaching external networks or backend services. • Verify actual traffic paths using firewall logs to confirm that routing policies are effective. This pattern is helpful for organizations that must enforce advanced filtering, logging, or compliance checks on container egress/ingress traffic, going beyond what native Azure networking controls provide. It also complements Azure Firewall and NSG controls by introducing a dedicated next-generation firewall within your VNet. If you’re working with network control, security perimeters, or hybrid network architectures involving containerized workloads on Azure, you might find it useful. Read the full article on my blogInter-Hub Connectivity Using Azure Route Server
By Mays_Algebary shruthi_nair As your Azure footprint grows with a hub-and-spoke topology, managing User-Defined Routes (UDRs) for inter-hub connectivity can quickly become complex and error-prone. In this article, we’ll explore how Azure Route Server (ARS) can help streamline inter-hub routing by dynamically learning and advertising routes between hubs, reducing manual overhead and improving scalability. Baseline Architecture The baseline architecture includes two Hub VNets, each peered with their respective local spoke VNets as well as with the other Hub VNet for inter-hub connectivity. Both hubs are connected to local and remote ExpressRoute circuits in a bowtie configuration to ensure high availability and redundancy, with Weight used to prefer the local ExpressRoute circuit over the remote one. To maintain predictable routing behavior, the VNet-to-VNet configuration on the ExpressRoute Gateway should be disabled. Note: Adding ARS to an existing Hub with Virtual Network Gateway will cause downtime that expect to last 10 minutes. Scenario 1: ARS and NVA Coexist in the Hub Option A: Full Traffic Inspection ARS and NVA Coexist in the Hub In this scenario, ARS is deployed in each Hub VNet, alongside the Network Virtual Appliances (NVAs). NVA1 in Region1 establishes BGP peering with both the local ARS (ARS1) and the remote ARS (ARS2). Similarly, NVA2 in Region2 peers with both ARS2 (local) and ARS1 (remote). Let’s break down what each BGP peering relationship accomplishes. For clarity, we’ll focus on Region1, though the same logic applies to Region2: NVA1 Peering with Local ARS1 Through BGP peering with ARS1, NVA1 dynamically learns the prefixes of Spoke1 and Spoke2 at the OS level, eliminating the need to manually configure these routes. The same applies for NVA2 learning Spoke3 and Spoke4 prefixes via its BGP peering with ARS2. NVA1 Peering with Remote ARS2 When NVA1 peers with ARS2, the Spoke1 and Spoke2 prefixes are propagated to ARS2. ARS2 then injects these prefixes into NVA2 at both the NIC level with NVA1 as the next hop, and at the OS level. This mechanism removes the need for UDRs on the NVA subnets to enable inter-hub routing. Additionally, ARS2 advertises the Spoke1 and Spoke2 prefixes to both ExpressRoute circuits (EXR2 and EXR1 due to bowtie configuration) via GW2, making them reachable from on-premises through either EXR1 or EXR2. 👉Important: To ensure that ARS2 accepts and propagates Spoke1/Spoke2 prefixes received via NVA1, AS-Override must be enabled. Without AS-Override, BGP loop prevention will block these routes at ARS2, since both ARS1 and ARS2 use the default ASN 65515, and ARS2 will consider the route as already originated locally. The same principle applies in reverse for Spoke3 and Spoke4 prefixes being advertised from NVA2 to ARS1. Traffic Flow Inter-Hub Traffic: Spoke VNets are configured with UDRs that contain only a default route (0.0.0.0/0) pointing to the local NVA as the next hop. Additionally, the “Propagate Gateway Routes” setting should be set to False to ensure all traffic, whether East-West (intra-hub/inter-hub) or North-South (to/from internet), is forced through the local NVA for inspection. Local NVAs will have the next hop to the other region spokes injected at the NIC level by local ARS, pointing to the other region NVA, for example NVA2 will have next hop to Spoke1 and Spoke2 as NVA1 (10.0.1.4) and vice versa. Why are UDRs still needed on spokes if ARS handles dynamic routing? Even with ARS in place, UDRs are required to maintain control of the next hop for traffic inspection. For instance, if Spoke1 and Spoke2 do not have UDRs, they will learn the remote spoke prefixes (e.g., Spoke3/Spoke4) injected via ARS1, which received them from NVA2. This results in Spoke1/Spoke2 attempting to route traffic directly to NVA2, a path that is invalid, since the spokes don’t have the path to NVA2. The UDR ensures traffic correctly routes through NVA1 instead. On-Premises Traffic: To explain the on-premises traffic flow, we'll break it down into two directions: Azure to on-premises, and on-premises to Azure. Azure to On-Premises Traffic Flow: As previously noted, Spokes send all traffic, including traffic to on-premises, via NVA1 due to the default route in the UDR. NVA1 then routes traffic to the local ExpressRoute circuit, using Weight to prefer the local path over the remote. Note: While NVA1 learns on-premises prefixes from both local and remote ARSs at the OS level, this doesn’t affect routing decisions. The actual NIC-level route injection determines the next hop, ensuring traffic is sent via the correct path—even if the OS selects a different “best” route internally. The screenshot below from NVA1 shows four next hops to the on-premises network 10.2.0.0/16. These include the local ARS (ARS1: 10.0.2.5 and 10.0.2.4) and the remote ARS (ARS2: 10.1.2.5 and 10.1.2.4). On-Premises to Azure Traffic Flow In a bowtie ExpressRoute configuration, Azure VNet prefixes are advertised to on-premises through both local and remote ExpressRoute circuits. Because of this dual advertisement, the on-premises network must ensure optimal path selection when routing traffic to Azure. From Azure side, to maintain traffic symmetry, add UDRs at the GatewaySubnet (GW1 and GW2) with specific routes to the local Spoke VNets, using the local NVA as the next hop. This ensures return traffic flows back through the same path it entered. 👉How Does the ExpressRoute Edge Router Select the Optimal Path? You might ask: If Spoke prefixes are advertised by both GW1 and GW2, how does the ExpressRoute edge router choose the best path? (e.g., diagram below shows EXR1 learns Region1 prefixes from GW1 and GW2) Here’s how: Edge routers (like EXR1) receive the same Spoke prefixes from both gateways. However, these routes have different AS-Path lengths: - Routes from the local gateway (GW1) have a shorter AS-Path. - Routes from the remote gateway (GW2) have a longer AS-Path because NVA1’s ASN (e.g., 65001) is prepended twice as part of the AS-Override mechanism. As a result, the edge router (EXR1) will prefer the local path from GW1, ensuring efficient and predictable routing. For example: EXR1 receives Spoke1, Spoke2, and Hub1-VNet prefixes from both GW1 and GW2. But because the path via GW1 has a shorter AS-Path, EXR1 will select that as the best route. (Refer to the diagram below for a visual of the AS-Path difference). Final Traffic Flow: Option-A Insights: This design simplifies UDR configuration for inter-hub routing, especially useful when dealing with non-contiguous prefixes or operating across multiple hubs. For simplicity, we used a single NVA in each Hub-VNet while explaining the setup and traffic flow throughout this article. However, a high available (HA) NVA deployment is recommended. To maintain traffic symmetry in an HA setup, you’ll need to enable the next-hop IP feature when peering with Azure Route Server (ARS). When on-premises traffic inspection is required, the UDR setup in the GatewaySubnet becomes more complex as the number of Spokes increases. As your Azure network scales, keep in mind that Azure Route Server supports a maximum of 16 BGP peers per instance (as of the time writing this article). This limit can impact architectures involving multiple NVAs or hubs. Option B: Bypass On-Premises Inspection If on-premises traffic inspection is not required, NVAs can advertise a supernet prefix summarizing the local Spoke VNets to the remote ARS. This approach provides granular control over which traffic is routed through the NVA and eliminates the need for BGP peering between the local NVA and local ARS. All other aspects of the architecture remain the same as described in Option A. For example, NVA2 can advertise the supernet 192.168.2.0/23 (supernet of Spoke3 and Spoke4) to ARS1. As a result, Spoke1 and Spoke2 will learn this route with NVA2 as the next hop. To ensure proper routing (as discussed earlier) and inter-hub inspection, you need apply a UDR in Spoke1 and Spoke2 that overrides this exact supernet prefix, redirecting traffic to NVA1 as the next hop. At the same time, traffic destined for on-premises will follow the system route through the local ExpressRoute gateway, bypassing NVA1 altogether. In this setup: UDRs on the Spokes should have "Propagate Gateway Routes" set to True. No UDRs are needed in the GatewaySubnet. 👉Can NVA2 Still Advertise Specific Spoke Prefixes? You might wonder: Can NVA2 still advertise specific prefixes (e.g., Spoke3 and Spoke4) learned from ARS2 to ARS1 instead of a supernet? Yes, this is technically possible, but it requires maintaining BGP peering between NVA2 and ARS2. However, this introduces UDR complexity in Spoke1 and Spoke2, as you'd need to manually override each specific prefix. This also defeats the purpose of using ARS for simplified route propagation, undermining the efficiency and scalability of the design. Bypass On-Premises Inspection Final Traffic Flow: Option B: Bypass on-premises inspection traffic flow Option-B Insights: This approach reduces the number of BGP peerings per ARS. Instead of maintaining two BGP sessions (local NVA and remote NVA) per Hub, you can limit it to just one, preserving capacity within ARS’s 8-peer limit for additional inter-hub NVA peerings. Each NVA should advertise a supernet prefix to the remote ARS. This can be challenging if your Spokes don’t use contiguous IP address spaces, as described in Option B. Scenario 2: ARS in the Hub and the NVA in Transit VNet In Scenario 1, we highlighted that when on-premises inspection is required, managing UDRs at the GatewaySubnet becomes increasingly complex as the number of Spoke VNets grows. This is due to the need for UDRs to include specific prefixes for each Spoke VNet. In this scenario, we eliminate the need to apply UDRs at the GatewaySubnet altogether. In this design, the NVA will be deployed in Transit VNet, where: Transit-VNet will be peered with local Spoke VNets and with the local Hub-VNet to enable intra-Hub and on-premises connectivity. Transit-VNet also peered with remote Transit VNets (e.g., Transit-VNet1 peered with Transit-VNet2) to handle inter-Hub connectivity through the NVAs. Additionally, Transit-VNets are peered with remote Hub-VNets, to establish BGP peering with the remote ARS. NVAs OS will need to add static routes for the local Spoke VNets prefixes, it can be specific or it can supernet prefix, which will later be advertised to ARSs over BGP Peering, then ARS will advertise it to on-premises via ExpressRoute. NVAs will BGP peer with local ARS and also with the remote ARS. To understand the reasoning behind this design, let’s take a closer look at the setup in Region1, focusing on how ARS and NVA are configured to connect to Region2. This will help illustrate both inter-hub and on-premises connectivity. The same concept applies in reverse from Region2 to Region1. Inetr-Hub: To enable NVA1 in Region1 to learn prefixes from Region2, NVA2 will configure static routes at the OS level for Spoke3 and Spoke4 (or their supernet prefix) and advertise them to ARS1 via remote BGP peering. As a result, these prefixes will be received by NVA1, both at the NIC level, with NVA2 as the next hop, and at the OS level for proper routing. Spoke1 and Spoke2 will have a UDR with a default route pointing to NVA1 as the next hop. For instance, when Spoke1 needs to communicate with Spoke3, the traffic will first route through NVA1. NVA1 will then forward the traffic to NVA2 using VNet peering between the two Hubs. A similar configuration will be applied in Region2, where NVA1 will configure static routes at the OS level for Spoke1 and Spoke2 (or their supernet prefix) and advertise them to ARS2 via remote BGP peering, as a result, these prefixes will be received by NVA2, both at the NIC level (injected by ARS2), with NVA1 as the next hop, and at the OS level for proper routing. Note: At the OS level, NVA1 learns Spoke3 and Spoke4 prefixes from both local and remote ARSs. However, the NIC-level route injection determines the actual next hop, so even if the OS selects a different best route, it won’t affect forwarding behavior. same applies to NVA2. On-Premises Traffic: To explain the on-premises traffic flow, we'll break it down into two directions: Azure to on-premises, and on-premises to Azure. Azure to On-Premises Traffic Flow: Spokes in Region1 route all traffic through NVA1 via a default route defined in their UDRs. Because of BGP peering between NVA1 and ARS1, ARS1 advertises the Spoke1 and Spoke2 (or their supernet prefix) to on-premises through ExpressRoute (EXR1). The Transit-VNet1 (hosting NVA1) is peered with Hub1-VNet, with “Use Remote Gateway” enabled. This allows NVA1 to learn on-premises prefixes from the local ExpressRoute gateway (GW1), and traffic to on-premises is routed through the local ExpressRoute circuit (EXR1) due to higher BGP Weight configuration. Note: At the OS level, NVA1 learns on-prem prefixes from both local and remote ARSs. However, the NIC-level route injection determines the actual next hop, so even if the OS selects a different best route, it won’t affect forwarding behavior. same applies to NVA2. On-Premises to Azure Traffic Flow: Through BGP peering with ARS1, NVA1 enables ARS1 to advertise Spoke1 and Spoke2 (or their supernet prefix) to both EXR1 and EXR2 circuits (due to the ExpressRoute bowtie setup). Additionally, due to BGP peering between NVA1 and ARS2, ARS2 also advertises Spoke1 and Spoke2 (or their supernet prefix) to EXR2 and EXR1 circuits. As a result, both ExpressRoute edge routers in Region1 and Region2 learn the same Spoke prefixes (or their supernet prefix) from both GW1 and GW2, with identical AS-Path lengths, as shown below. EXR1 learns Region1 Spokes's supernet prefixes from GW1 and GW2 This causes non-optimal inbound routing, where traffic from on-premises destined to Region1 Spokes may first land in Region2’s Hub2-VNet before traversing to NVA1 in Region1. However, return traffic from Spoke1 and Spoke2 will always exit through Hub1-VNet. To prevent suboptimal routing, configure NVA1 to prepend the AS path for Spoke1 and Spoke2 (or their supernet prefix) when advertising them to the remote ARS2. Likewise, ensure NVA2 prepends the AS path for Spoke3 and Spoke4 (or their supernet prefix) when advertising to ARS1. This approach helps maintain optimal routing under normal conditions and during ExpressRoute failover scenarios. Below diagram shows NVA1 is setting AS-Prepend for Spoke1 and Spoke2 supernet prefix when BGP peer with remote ARS (ARS1), same will apply for NVA2 when advertising Spoke3 and Spoke4 prefixes to ARS1. Final Traffic Flow: Full Inspection: Traffic flow when NVA in Transit-VNet Insights: This solution is ideal when full traffic inspection is required. Unlike Scenario 1 - Option A, it eliminates the need for UDRs in the GatewaySubnet. When ARS is deployed in a VNet (typically in Hub VNets), the VNet will be limited to 500 VNet peerings (as of the time writing this article). However, in this design, Spokes peer with the Transit-VNet instead of directly with the ARS VNet, allowing you to scale beyond the 500-peer limit by leveraging Azure Virtual Network Manager (AVNM) or submitting a support request. Some enterprise customers may encounter the 1,000-route advertisement limit on the ExpressRoute circuit from the ExpressRoute gateway. In traditional hub-and-Spoke designs, there's no native control over what is advertised to ExpressRoute. With this architecture, NVAs provide greater control over route advertisement to the circuit. For simplicity, we used a single NVA in each Hub-VNet while explaining the setup and traffic flow throughout this article. However, a high available (HA) NVA deployment is recommended. To maintain traffic symmetry in an HA setup, you’ll need to enable the next-hop IP feature when peering with Azure Route Server (ARS). This design does require additional VNet peerings, including: Between Transit-VNets (inter-region), Between Transit-VNets and local Spokes, and Between Transit-VNets and both local and remote Hub-VNets.
Designing Cloud Landing Zones by Traffic Flow: A Defence‑in‑Depth, DMZ‑First Architecture
As enterprises adopt Microsoft Azure for large‑scale and regulated workloads, security architecture must be driven by traffic trust boundaries and inspection intent, not connectivity alone. Regulatory frameworks consistently require a clear separation between Internet‑untrusted and private enterprise traffic, enforced through defence‑in‑depth controls aligned to the OSI model. This drives the adoption of a DMZ‑first landing zone architecture, where volumetric protection, application‑layer inspection, and perimeter controls are enforced at well‑defined trust boundaries. Security enforcement is prioritised before access, rather than being an afterthought of connectivity. A distributed hub architecture enables this model at scale, delivering consistent controls while improving resiliency, fault isolation, and operational clarity. Traffic Classification in an Enterprise Landing Zone Enterprise traffic flows are classified based on origin, destination, and trust level, which directly dictates inspection requirements: Internet Inbound (North–South): Traffic from the public Internet to Azure‑hosted applications Internet Outbound (South–North): Egress traffic from private workloads to the Internet East–West: Traffic between virtual networks within or across regions Hybrid Connectivity: Traffic between Azure, on‑premises, and multi‑cloud environments Each flow presents a distinct risk profile, making a uniform connectivity and inspection model unsuitable for enterprise and regulated environments. Hub‑and‑Spoke as the Foundation for Centralised Security Enterprises commonly adopt a Hub‑and‑Spoke topology using VNET peering or Azure Virtual WAN, extending hybrid connectivity via Site‑to‑Site VPN or ExpressRoute. The hub provides a centralised, datapath‑centric layer for shared networking and security services, while spoke VNETs host application workloads and remain private, typically without public IPs. Ingress and egress are handled through shared, centrally managed endpoints. This model consolidates critical controls—Azure Firewall, Azure WAF, and DDoS protection—at controlled entry and exit points, significantly reducing the attack surface. Integration with Microsoft Sentinel enables cross‑layer visibility, threat hunting, and detection of risks such as unauthorised access and data exfiltration. Why a Single Hub Is Not Sufficient Using a single hub to process all traffic types introduces operational and security challenges at scale by forcing Internet‑untrusted and private traffic through the same inspection tier. Key limitations include: Coupled Internet and private security policies Rapid firewall rule sprawl and management overhead A single blast radius across all traffic types Throughput and SNAT scalability constraints Increased difficulty meeting regulatory separation requirements These issues become more pronounced as environments scale across regions and workloads. Multi‑Hub as an Enterprise‑Grade Evolution A multi‑hub architecture resolves these challenges by separating inspection responsibilities across dedicated hubs: DMZ Hub VNET: Internet‑facing traffic and perimeter security enforcement Core Hub VNET: Outbound Internet access, East‑West, and hybrid traffic inspection This separation aligns security controls with traffic intent, reduces policy complexity, limits blast radius, and simplifies operations. In the next section, we explore multi‑hub architecture patterns and examine how each traffic flow is inspected and governed in practice. Scenario 1 – Third‑Party Firewall–Centric Multi‑Hub Design In this scenario, third‑party firewalls provide inspection for all traffic flows—Internet inbound, Internet outbound, East‑West, and hybrid connectivity—while Azure WAF and Azure DDoS Protection are used to defend against volumetric attacks, application‑layer threats, and malicious bot traffic. Architecture Overview The design uses two dedicated hub virtual networks, with spoke VNets peered to both hubs to maintain traffic separation based on trust and inspection requirements: DMZ Hub VNET (Internet Ingress Hub) Core Hub VNET (Egress, East‑West, and Hybrid Hub) DMZ Hub VNET – Internet Inbound (North–South) Traffic The DMZ hub establishes a dedicated perimeter for Internet‑facing applications. Internet traffic lands on a public Application Gateway with Azure WAF, protected by an Azure DDoS Protection Plan applied to the public frontend IP. Third‑party firewalls are deployed behind the Application Gateway to perform deep packet inspection. Firewalls operate in active‑active mode as standalone appliances, using three dedicated network interfaces (Untrust, Trust, and Management), each placed in separate subnets. Application Gateway provides load balancing and health probing, ensuring firewall availability and resiliency. Core Hub VNET – Egress, East‑West, and Hybrid Traffic The core hub handles all non‑Internet‑inbound traffic flows, including private inter‑VNET communication and hybrid connectivity. Third‑party firewalls are deployed behind an internal Azure Load Balancer, which maintains high availability using health probes. Firewalls follow the same active‑active, three‑NIC design for consistent policy enforcement and operational symmetry. Azure NAT Gateway is associated with the firewall Untrust subnet to scale outbound SNAT and simplify Internet egress with deterministic public IP. Site‑to‑site VPN or ExpressRoute terminate in this hub for on‑premises and multi‑cloud connectivity, ensuring private and regulated traffic bypasses Internet‑facing controls. Architecture Patterns - Traffic Flow 1.1 North–South (Internet Inbound) Traffic Flow This flow applies when applications hosted on Azure (VMs, VM Scale Sets, or services behind Internal Load Balancer / Internal Application Gateway) are published to the Internet. As shown in the diagram, Internet-bound client requests enter Azure through a dedicated DMZ Hub, where perimeter security controls are enforced before traffic reaches private application spokes. Inbound Request Flow (Blue Path) Client → Application Gateway (WAF) A client request (for example, app1.example.com) is resolved via DNS and lands on the public endpoint of Azure Application Gateway. Azure WAF performs Layer‑7 inspection, bot protection, and rule evaluation. Application Gateway → Firewall (DMZ Hub) Based on hostname and routing rules, Application Gateway forwards traffic to the firewalls while preserving the client’s real IP using the X‑Forwarded‑For (XFF) header. Firewall → Application (Spoke VNET) The firewall performs DNAT to the internal destination (Internal Application Gateway or Internal Load Balancer) and SNAT to its trust‑interface IP to ensure symmetric return traffic. Traffic is then routed over VNET peering to the private application workload. Response Flow (Green Path) Application → Firewall The application responds using its private IP, with the firewall trust interface as the destination. Firewall → Application Gateway The firewall forwards the response based on its session state. Application Gateway → Client Application Gateway returns the response to the Internet client. DDoS Protection Azure DDoS Protection continuously monitors the Application Gateway public IP and mitigates volumetric attacks before they reach the application stack. Key Design Considerations Application Gateway routing - Use a multi‑site listener with hostname‑based rules to steer traffic to the appropriate firewall backend. Firewall backend mapping - Register firewall VMs in a single backend pool and differentiate applications using distinct backend settings (same protocol, different ports). Application steering at firewall - Firewalls perform port‑based DNAT (or private FQDN‑based DNAT where supported) to forward traffic to the correct application private IP in spoke VNets. Traffic symmetry - Source NAT is applied on the firewall trust interface to maintain symmetric return paths. Protocol support - The same pattern applies to TCP/UDP workloads, using an External Azure Load Balancer instead of Application Gateway where WAF is not required. Client IP visibility Application Gateway enables WAF inspection using the real client IP, preserved downstream via X‑Forwarded‑For. With Azure Load Balancer, client IP is retained up to the firewall for IP‑based enforcement, with Azure DDoS Protection safeguarding the public frontend. 1.2 Hybrid Connectivity Traffic Flow Hybrid connectivity represents communication between Azure workloads and on‑premises environments using Site‑to‑Site VPN or ExpressRoute. Traffic Flow Request (Blue: 1–4): Traffic originates from applications in spoke VNETs and is forwarded via UDRs to the Internal Load Balancer, through active‑active firewalls, and then to on‑premises via VPN or ExpressRoute. Response (Green: 5–8): Return traffic follows the same path in reverse. Key Design Considerations Source and destination IP addresses remain unchanged end‑to‑end, despite firewalls operating behind an Internal Load Balancer. UDRs steer both outbound and inbound hybrid traffic to the Internal Load Balancer as the next hop. Spoke VNET subnets should be associated with custom route tables with Propagate Gateway Routes set to No, preventing gateway-learned routes from bypassing the intended security inspection path configured via UDR. However, VPN Gateway Subnet Route table should have route table with Propagate Gateway Routes set to Yes. Azure Virtual Network Manager can be used to manage and scale UDR deployment centrally across Hub and spoke networks. 1.3 East-West Traffic flow This traffic flow represents the connectivity between resources across VNETs through Firewall. Considerations remain same as “Hybrid Connectivity Traffic flow” mentioned above. 1.4 Egress/Outbound Connectivity Traffic flow This traffic flow represents the connectivity from the resources hosted inside VNETs to Internet through Firewalls deployed behind Internal Load Balancer in Core Hub VNET. Egress/Outbound Connectivity Traffic flow Key Considerations: A default route (0.0.0.0/0) UDR directs Internet-bound traffic to the Internal Load Balancer for firewall inspection. Firewall performs SNAT to its untrusted interface, after which Azure NAT Gateway translates traffic to a fixed public IP. Alternatively, a public IP can be assigned to the firewall untrusted interface to perform SNAT directly, removing the need for NAT Gateway and reducing cost. Spoke VNET subnets should be associated with custom route tables with Propagate Gateway Routes set to No, preventing gateway-learned routes from bypassing the intended security inspection path configured via UDR. However, VPN Gateway Subnet Route table should have route table with Propagate Gateway Routes set to Yes. However, with horizontal firewall scaling, outbound public IPs become non-deterministic. Scenario 2: Third‑Party Firewalls for Ingress, Azure Firewall for Private Flows In this scenario, third‑party firewalls secure Internet‑facing traffic, while Azure Firewall handles Egress, East–West, and Hybrid flows. This separation enables organisations to retain existing perimeter security investments while adopting a fully managed, cloud‑native control and data plane for internal traffic flows. As private traffic patterns scale, particularly for chatty East–West communication, Azure Firewall helps simplify operations and improves scalability by offloading traffic inspection to a managed service. This approach reduces operational overhead and provides consistent policy enforcement across internal and hybrid flows. Setup DMZ Hub design remains unchanged from Scenario 1. Core Hub VNET use Azure Firewall for better scalability, built‑in resilience, and simplified operations. Azure Firewall eliminates the need for load balancers and reduces complexity for chatty east‑west traffic. It requires a single subnet (unless force tunnelling is enabled), optimising IP consumption. Outbound traffic is SNATed by Azure Firewall and further translated via Azure NAT Gateway. Firewall‑based SNAT to public IP can be used to avoid NAT Gateway costs when deterministic egress IPs or higher SNAT scale are not required. Traffic Flow North–South (Internet Inbound) Traffic Flow represented with Numbers. Request flow represented with Blue: 1–3 Response flow represented with Brown: 4–6 Egress/Outbound Connectivity Traffic flow represented with alphabets. Request flow represented with Green: A–C Response flow represented with Blue: D–F Hybrid flow represented with purple Architecture Diagram: Third‑Party Firewalls for Ingress, Azure Firewall for Private Flows Further if you want to use Azure Firewall for the Internet Ingress flow, you can use it as well. Below is the architecture diagram explaining the same pattern. Azure Firewalls for Ingress and Private Flows Key Considerations: Application Gateway behaviour remains the same as Scenario 1; however, backend pools use IP‑based targets. Internal applications should be exposed via internal Application Gateway or internal load balancer to provide stable backend IPs. To optimise traffic flow and avoid unnecessary network address translation, the private IP address of the Application Gateway or internal Load Balancer should be configured in the backend pool of External Load Balancer in DMZ Hub VNET. This design does not need Azure Firewall to perform both source NAT (SNAT) and destination NAT (DNAT) operations. Further, implementing User Defined Routes (UDRs) in the Application Gateway subnet—specifically for spoke VNet CIDR ranges steers traffic routing through the Firewall. For non-web (TCP/UDP) workloads, applications can be directly exposed through the firewall public IP, with Azure DDoS Protection applied to mitigate volumetric attacks. Spoke VNET subnets should be associated with custom route tables with Propagate Gateway Routes set to No, preventing gateway-learned routes from bypassing the intended security inspection path configured via UDR. However, VPN Gateway Subnet Route table should have route table with Propagate Gateway Routes set to Yes. Scenario 3 - Overcoming Peering Complexity at Scale with Azure Virtual WAN In environments where each application is hosted in its own dedicated VNET for isolation and stronger security, a traditional hub‑and‑spoke model quickly becomes complex. With a single hub, "N" spoke VNETs require "N" peering's; introducing multiple hubs increases this to "2N", significantly amplifying operational overhead—especially across multiple regions. Azure Virtual WAN addresses this challenge by eliminating the need for extensive VNET peering. As a global, managed service, it simplifies large‑scale and multi‑region architectures while providing built‑in scalability and operational consistency. Setup DMZ Hub design and components remain unchanged from Scenario 1 keeping defense in depth principles intact for Internet Inbound or North-South traffic. Core Hub VNET is replaced with Azure Virtual WAN Secure Hub which has Azure Firewall inside the managed Secure Hub VNETs. This further eliminates the need for creating Azure Firewall Subnet, Azure NAT Gateway resources for East-West, Egress and Hybrid connectivity and provides same level of security protection as mentioned in Scenario 2. Secure Hub design using Virtual WAN removes the routing and inspection complexity which was unless required in Scenario 2 by removing UDR configurations in Spoke VNETs, Gateway Subnet and Azure Firewall Subnet. This provides simplified design and operations with built-in resiliency by leveraging the managed platform. Azure Firewall eliminates the need for load balancers and reduces complexity for chatty east‑west traffic. East-West, Egress, and Hybrid connectivity happens via VPN Gateway and Express Route Gateway present in Virtual Hub. All private and Internet Egress traffic is inspected by Azure Firewall present in secure Hub. Outbound traffic is SNATed to Public IP by Azure Firewall thus avoiding the need of Azure NAT Gateway. Firewall‑based SNAT to public IP can be used to avoid NAT Gateway costs when deterministic egress IPs or higher SNAT scale are not required. Traffic Flow North–South (Internet Inbound) Traffic Flow represented with Numbers. Request flow represented with Blue: 1–5 Response flow represented with Green: 6–10 Egress/Outbound Connectivity Traffic flow represented with Red Hybrid flow represented with Green East-West connectivity flow represented with black Architecture Diagram: Key Considerations Virtual WAN has additional cost Refer to architecture diagram above for different traffic flows. The DMZ VNET components consideration remains same as per scenario 1 You can use your own managed public IP on Azure Firewall for applying standard IP/Network DDoS protection plan. Benefits of multi-Hub approach over single Hub You can use this table as decision matrix to choose the best option for your application requirements. Dimensions Multi-Hub Design Resiliency · Independent scaling of DMZ and Core firewalls · Failure in Internet inspection does not impact East‑West traffic · Easier multi‑region active‑active designs Security & Compliance Clear trust boundaries Auditable inspection points Easier alignment with Zero Trust & regulatory frameworks Operational Simplicity Smaller, purpose‑built firewall policies Clear ownership (Perimeter vs Core team) Faster troubleshooting due to deterministic flows Cost Optimisation Avoids over‑inspection of East‑West traffic by DMZ firewalls Smaller firewall SKUs per hub Better scale‑unit consumption Performance Efficiency Reduced latency for private traffic No unnecessary hair‑pinning Optimised SNAT and connection tracking Key Takeaways Security‑first landing zones should not be designed as one‑size‑fits‑all networks. By designing hubs based on traffic trust level and inspection intent, rather than convenience, organisations gain: o Stronger security boundaries and better fault isolation o Predictable operations at scale o Lower long‑term cost and complexity o Design for traffic flows, not just connectivity o Use DMZ hubs exclusively for Internet‑untrusted traffic o Use Core hubs for East‑West, outbound, and hybrid connectivity o Map security controls to OSI layers o Avoid single‑hub designs for large, regulated, or multi‑region environments This distributed hub‑and‑DMZ landing zone provides a clean, scalable, and secure foundation for enterprise workloads in any Hyperscaler. For regulated, internet‑exposed, or multi‑region environments, a distributed hub with a dedicated DMZ is no longer optional — it is a foundational architecture decision.
Migrating from MSEE Hairpin Routing to AVNM Mesh for Large-Scale VNet-to-VNet Connectivity
Introduction A common pattern in large Azure deployments is to route VNet-to-VNet traffic through Microsoft Enterprise Edge (MSEE) routers. This happens when spoke VNets in a hub-and-spoke topology communicate with each other by hairpinning through the ExpressRoute circuit: traffic exits the Azure data center, traverses MSEE, and re-enters the data center to reach the destination VNet. This pattern works, but it was not designed as the long-term connectivity model for east-west traffic. With Azure Virtual Network Manager (AVNM) mesh connectivity and recent scale improvements — including high-scale mesh up to 5,000 VNets and High-Scale Private Endpoints (HSPE) up to 20,000 Private Endpoints across connected VNets — enterprises can migrate to a direct, in-datacenter routing model that removes MSEE dependency for VNet-to-VNet traffic. This article explains why the migration is useful, what the new scale limits are, how to enable the required features, and how to execute the migration with minimal disruption. Who This Is For This migration is most relevant if you have 50 or more spoke VNets communicating east-west through ExpressRoute hairpin routing, are approaching VNet peering limits, want to reduce ExpressRoute utilization from internal traffic, or need a simpler centrally managed connectivity model. Even if most east-west flows must continue through a hub firewall for inspection, you can still simplify connectivity management for the flows allowed to go direct. Why MSEE Hairpin Routing for VNet-to-VNet Is Not Recommended When spoke VNets communicate through MSEE, traffic follows a suboptimal path: Spoke A → Hub VNet → ExpressRoute Gateway → MSEE → ExpressRoute Gateway → Hub VNet → Spoke B Single point of failure. MSEE becomes a shared dependency for east-west traffic. An MSEE outage or capacity constraint can affect every VNet pair in the topology. Lower-latency path. Traffic no longer needs to leave the data center and return for spoke-to-spoke communication. Direct mesh keeps east-west traffic on an in-datacenter path, which is generally lower latency than hairpinning through MSEE. Bandwidth constraints. MSEE circuits have finite bandwidth. Routing east-west traffic through them competes with north-south on-premises traffic and can saturate the circuit. Operational risk at scale. Large deployments place significant load on MSEE infrastructure, creating scalability, reliability, and operational concerns for environments with thousands of VNets. Why Manual Peering Was Not a Practical Alternative The obvious alternative — creating direct VNet peerings between every spoke pair — solves the MSEE dependency but introduces its own operational complexity: Combinatorial growth. Connecting N spokes requires N × (N - 1) / 2 peering relationships. For 100 spokes, that is 4,950 peerings. For 1,000 spokes, it is nearly 500,000. Management overhead. Each peering must be individually provisioned, monitored, and maintained. This increases drift, audit, and operational overhead. How AVNM Mesh Solves This AVNM mesh provides group-based connectivity. You define a set of VNets as a network group, apply a mesh connectivity configuration, and AVNM establishes bi-directional connectivity across all members. Traffic between meshed VNets stays within the Azure data center: no MSEE traversal, no hub hop, and no manual peering management. Define once, connect all. A single mesh configuration connects every VNet in the group to every other VNet. Centralized management. Add or remove VNets from the group; AVNM reconciles connectivity automatically. Direct spoke-to-spoke paths. Traffic flows directly between VNets, bypassing both the hub and MSEE. Dynamic membership. Use Azure Policy to auto-enroll new VNets based on tags or resource group conditions. How to Migrate Large Scale Topology High-scale mesh lets user migrate larger topology — up to 5,000 VNets and 20,000 Private Endpoints in a mesh. Scale — Standard Mesh vs. High-Scale Mesh Dimension Standard Mesh High-Scale Mesh VNets per mesh Up to 250 (soft limit, can request increase) Up to 5,000 Private Endpoints per mesh Up to 2,000 Up to 20,000 (with HSPE enabled) Private Endpoints per VNet Up to 1,000 Up to 5,000 (with HSPE enabled) Enabling High-Scale Private Endpoints (HSPE) As a mesh footprint grows, so does the number of Private Endpoints deployed across the connected VNets. The default platform limits — 1,000 Private Endpoints per VNet and 2,000 across connected VNets and mesh — can be reached quickly in large environments. Enabling HSPE raises these limits to 5,000 and 20,000, respectively. For large-scale mesh migrations, enable HSPE proactively if the environment is expected to grow toward standard Private Endpoint limits by following the steps below. Step 1 — Prepare Each VNet Ensure Private Endpoint Network Policies are set to Enabled or RouteTableEnabled on all subnets containing Private Endpoints. This is a prerequisite. Set the VNet-level property PrivateEndpointVNetPolicies to Basic. This activates HSPE on the VNet. Step 2 — Enable HSPE on the Mesh Configuration In the AVNM mesh connectivity configuration, enable the option for high-scale private endpoints. AVNM validates that all VNets in the mesh are HSPE-enabled. If any VNet is missing the configuration, the deployment is blocked with a clear error. Step 3 — Deploy Deploy the connectivity configuration. AVNM applies HSPE across the mesh. Behavior Changes When Enabling HSPE Brief connection reset. Enabling or disabling HSPE triggers a one-time, approximately 1-second connection reset for existing Private Endpoint connections in the VNet. Plan this during a maintenance window. Per-PE Bytes In / Out monitoring is no longer available. HSPE treats each Private Endpoint IP like any other IP in the VNet, which removes per-PE traffic counters. If you depend on per-PE metrics, evaluate alternatives before enabling. On-premises PE traffic billing changes. PE traffic originating from on-premises appears as an aggregate bill on the gateway VNet, not on the individual Private Endpoint resource. The total bill does not change. How to Avoid Downtime Mesh coexists with existing peerings. AVNM does not delete manually created peerings unless explicitly configured to do so. Traffic shifts automatically. Once mesh is deployed, spoke-to-spoke traffic routes directly. The MSEE hairpin path remains available if mesh is removed. No reconfiguration of hub components. Firewalls, gateways, and NVAs in the hub continue to function. North-south on-premises traffic still flows through the hub gateway. Rollback is simple while the legacy path remains. Keep the MSEE hairpin path in place during validation so affected VNets can fall back by removing them from the network group or undeploying the mesh configuration. Security and East-West Traffic Inspection A common concern is whether direct mesh connectivity bypasses hub firewalls or network virtual appliances. The answer depends on how inspection is enforced today. Mesh provides connectivity, not routing policy. If spoke subnets have UDRs that direct traffic through a hub NVA or firewall, those UDRs continue to apply and can keep inspected flows on the firewall path. Security Admin Rules provide centralized segmentation. For flows that do not require firewall inspection, AVNM Security Admin Rules can enforce network-level allow or deny policies across network groups. Use both where appropriate. Mesh can provide direct connectivity for approved flows while Security Admin Rules enforce segmentation boundaries where required. Recommendation: Before migrating, inventory which spoke-to-spoke flows currently traverses the firewall. Decide per flow whether to maintain inspection by keeping UDRs in place or allowing a direct mesh path by removing the UDR for that flow pair. Migration Order and Process The migration from MSEE hairpin routing to AVNM mesh is non-disruptive by design. Mesh connectivity overlays on top of existing peerings and takes routing precedence for east-west traffic. You do not need to tear down the existing hub-and-spoke topology first. Recommended Steps Design the mesh topology. Group VNets by region into mesh groups. If you expect more than 250 VNets per mesh, register the AllowHighScaleConnectedGroup feature in advance. Create a Network Manager and define Network Groups. Ensure that the AVNM scope covers all relevant subscriptions. Use static membership for initial migration or dynamic membership through Azure Policy for ongoing enrollment. Enable HSPE on every VNet in the mesh. Follow the HSPE enablement steps if you need to have more than 2,000 Pes in a mesh. Schedule the change during a maintenance window to account for the brief connection reset. Create the mesh connectivity configuration in AVNM. Select the network groups, enable mesh topology, enable high-scale private endpoints, and enable global mesh if cross-region connectivity is required. Deploy incrementally. Start with a pilot region or non-critical environment. Validate effective routes, spoke-to-spoke connectivity, spoke-to-hub-to-on-premises connectivity, Private Endpoint reachability, and expected VNet flow log patterns before expanding to production regions. Route Behavior During and After Migration During migration, mesh and MSEE can coexist. Mesh-connected VNets receive direct routes for mesh-connected destinations, while existing ExpressRoute gateway routes continue to serve on-premises destinations. UDRs still override system routes, so forced-tunneling and firewall inspection patterns remain in effect when UDRs are present. Mesh destinations. Traffic between mesh-connected VNets goes directly instead of hairpinning through MSEE when no UDR overrides the route. On-premises destinations. ExpressRoute continues to provide north-south connectivity to on-premises networks. Gateway transit. Spokes can continue to reach on-premises through the hub gateway when the design uses gateway transit. Infrastructure-as-Code Considerations If VNet peerings are managed through Terraform, Bicep, or ARM templates, treat AVNM mesh as the new source of truth only after validation. Deploy mesh first. AVNM mesh can coexist with existing peerings, so do not remove peering resources from IaC before validating the mesh path. Validate the traffic path. Use effective routes, Connection Monitor, and flow logs to confirm traffic is using mesh where expected. Guard against drift. Review pipeline state and lifecycle settings before decommissioning old peerings, especially in environments where multiple teams manage network resources. Codify AVNM. Manage the Network Manager, network groups, configuration, and deployment through IaC so mesh becomes the governed connectivity model. DNS Resolution Across Mesh Mesh connectivity does not change DNS resolution behavior by itself. If spoke VNets are already linked to Private DNS Zones hosted or managed through the hub, those links continue to determine name resolution. If spokes use custom DNS servers in the hub, verify that any UDR changes made during migration do not unintentionally alter the DNS traffic path. Migration Example — Two Hub-and-Spoke Topologies into a Single Mesh This example shows how an enterprise can migrate two regional hub-and-spoke environments into one centrally managed AVNM mesh while preserving the existing MSEE path during validation. Current State — Two Hub-and-Spoke Topologies with MSEE Hairpin Contoso Corp operates a large Azure environment in the East US region with two hub-and-spoke topologies: Topology A Topology B Hub VNet Hub-A Hub-B Spoke VNets 500 500 ExpressRoute Gateway ER-GW-A in Hub-A ER-GW-B in Hub-B ExpressRoute Circuit Shared circuit, connected to both gateways Same shared circuit Avg. Private Endpoints per spoke ~8 (4,000 total) ~12 (6,000 total) Total Private Endpoints 10,000 across both topologies How traffic flows today: Spoke-to-spoke within Topology A: Spoke-A-01 → Hub-A → ER-GW-A → MSEE → ER-GW-A → Hub-A → Spoke-A-02 Spoke-to-spoke across topologies: Spoke-A-01 → Hub-A → ER-GW-A → MSEE → ER-GW-B → Hub-B → Spoke-B-01 Every spoke-to-spoke packet — whether within the same topology or across topologies — exits the data center, traverses MSEE, and re-enters. With 1,000 spokes generating east-west traffic, MSEE becomes a shared single point of failure and adds latency to every flow. Target State — A Single AVNM Mesh with 1,000 VNets Contoso's goal is to consolidate all 1,000 spoke VNets into a single AVNM mesh, removing MSEE from the east-west traffic path. Spoke-to-spoke traffic, any pair: Spoke-A-01 → directly→ Spoke-B-01. Traffic stays in the data center and uses the direct mesh path. MSEE role: MSEE carries north-south on-premises traffic. East-west load is removed from the ExpressRoute hairpin path. Migration Execution Phase 0 — Pre-Work Register the feature. Since the mesh contains 1,000 VNets, above the 250 standard limit, Contoso registers the AllowHighScaleConnectedGroup feature flag on the subscription. This enables high-scale mesh support for up to 5,000 VNets. Inventory Private Endpoints. With 10,000 Private Endpoints across 1,000 VNets, Contoso exceeds the standard mesh Private Endpoint limit of 2,000. HSPE must be enabled. Phase 1 — Enable HSPE on All 1,000 Spoke VNets Batch 1: Enable HSPE on all 500 spoke VNets in Topology A during the maintenance window by following the instructions. Batch 2: Apply the same configuration to all 500 spokes in Topology B. Expected impact: Each VNet may experience a brief connection reset for existing Private Endpoint connections when HSPE is enabled. Schedule the change during a maintenance window. Phase 2 — Create the AVNM Mesh Create a Network Manager scoped to the management group containing all 1,000 spoke VNets. Define a single network group called eastus-mesh-all-spokes with dynamic membership using an Azure Policy and the tags. Create a mesh connectivity configuration. Set topology to Mesh, network group to eastus-mesh-all-spokes, high-scale private endpoints to Enabled, and global mesh to Not needed because all VNets are in the same region. Save the configuration as a draft and do not deploy yet. Phase 3 — Incremental Deployment Wave 1 — Pilot: Contoso deploys the mesh configuration to 50 dev/test spokes, either by using a temporary network group or by tagging only those VNets initially. Validation includes effective routes showing ConnectedGroup as the next-hop type for meshed spoke prefixes, spoke-to-spoke connectivity through the direct mesh path, Private Endpoint reachability across meshed spokes, unchanged on-premises connectivity through the hub and MSEE, and VNet flow logs that confirm expected direct spoke-to-spoke flows. Wave 2 — Light production traffic VNets: After a successful pilot, Contoso tags light production traffic VNets. The dynamic network group picks them up automatically. Contoso redeploys the connectivity configuration, runs the same validation checklist, and monitors traffic to confirm that east-west traffic from these VNets is moving away from the ExpressRoute hairpin path. Wave 3 — All remaining production VNets: Contoso tags all remaining spokes and redeploys the configuration. At this point, all 1,000 spokes are in the mesh. No downtime migration: During Waves 2 and 3, existing MSEE hairpin routing remains functional. VNets not yet in the mesh continue to communicate through MSEE. VNets already in the mesh communicate directly through the mesh. This avoids a planned connectivity gap during migration. Phase 4 — Post-Migration Validation After deployment, confirm that mesh is active and traffic is taking the expected path before decommissioning the legacy route. Effective routes. Verify spoke subnets show direct routes to peer VNet prefixes instead of routing through the gateway or MSEE. Connection Monitor. Track representative spoke-to-spoke flows and compare latency and reachability before and after migration. VNet flow logs. Confirm east-west traffic matches the expected mesh path and is not still traversing the ExpressRoute gateway path. Network Watcher topology. Visualize the resulting connectivity model and identify any VNets not enrolled in the target network group. If traffic is still hairpinning after mesh deployment, check for UDRs overriding system routes, spokes missing from the network group, deployment not committed to the target region, or IaC pipelines recreating legacy peerings. Rollback Quick rollback while MSEE remains in place. Remove affected VNets from the network group or undeploy the mesh connectivity configuration. AVNM removes only the connectivity it created, and traffic can fall back to the existing MSEE hairpin path. Rollback after decommissioning legacy paths. If old peerings or route dependencies have already been removed, rollback may require reprovisioning those resources and should be treated as a longer change window. Recommendation. Keep the MSEE hairpin path available for at least two weeks after mesh deployment, monitor traffic patterns, and only then remove the legacy path. Before and After Summary Metric Before (MSEE Hairpin) After (AVNM HSPE Mesh) Spoke-to-spoke latency in the same region Higher due to the MSEE hairpin path Lower-latency direct in-datacenter path; actual latency depends on workload, region, and network conditions Traffic path for east-west Spoke → Hub → MSEE → Hub → Spoke Spoke → Spoke directly through the Mesh MSEE dependency for east-west Yes, shared dependency No MSEE dependency for east-west traffic Manual peerings required 0 when using hairpin routing, but 499,500 if built manually for 1,000 spokes 0 manual spoke-to-spoke peerings; AVNM manages connectivity Private Endpoints supported 2,000 per mesh under standard limits 20,000 per mesh with HSPE Rollback complexity Not applicable to the current hairpin model Remove VNets from the network group or undeploy the connectivity configuration Migration downtime Not applicable Designed for no planned downtime when deployed incrementally and validated carefully Closing Notes Migrating to AVNM mesh does not require tearing down your existing network. The hub gateway, firewalls, and NVAs continue to function as they do today. What changes is that east-west spoke-to-spoke traffic stops leaving the data center unnecessarily. MSEE is not the right tool for the east-west fabric. Removing internal traffic from the ExpressRoute circuit is a reliability and capacity improvement. AVNM mesh replaces combinatorial complexity with group-based intent. The operational model scales with the number of groups, not the number of VNets. High-scale mesh and HSPE remove the ceiling — up to 5,000 VNets and 20,000 Private Endpoints per mesh. The migration is incremental and reversible. Mesh coexists with existing paths, and you can validate wave by wave before decommissioning the legacy route. Start with a pilot mesh in a non-critical environment, validate the traffic shift, and expand from there. Resources Connectivity configurations in Azure Virtual Network Manager Increase Private Endpoint virtual network limits Create a mesh topology with Azure Virtual Network Manager Security admin rules in Azure Virtual Network Manager Create a mesh network topology with Azure Virtual Network Manager using Terraform
ICMP Support for Azure StandardV2 NAT Gateway
Our customers, across all industries, depend on consistent outbound connectivity and observability to operate and troubleshoot their cloud workloads at scale. As Azure environments grow in size and complexity, teams increasingly rely on standard network diagnostic tools such as ping to quickly validate reachability and identify connectivity issues. When customers route outbound traffic through Azure NAT Gateway, they benefit from centralized and scalable outbound connectivity designed to prevent SNAT port exhaustion, while simplifying management by removing the need for per‑resource public IPs. Historically, outbound connectivity through Azure NAT Gateway supported TCP and UDP traffic only for Azure workloads. We are pleased to announce that Azure StandardV2 NAT Gateway now supports outbound ping through ICMP Echo Request and Echo Reply traffic. Workloads that rely on NAT Gateway for internet egress can now use ping to validate reachability, improving operational visibility and simplifying network troubleshooting while preserving the scalability and resiliency characteristics of NAT Gateway. What’s now supported? Azure StandardV2 NAT Gateway supports outbound ICMP Echo Request and Echo Reply traffic (ping) for IPv4 and IPv6. Other ICMP message types are not currently supported. With this update, ICMP traffic sent from workloads behind a NAT Gateway can be translated to a public IP address and reliably receive responses from external endpoints. This brings ICMP behavior in line with existing TCP and UDP outbound flows, enabling consistent network diagnostics across all common transport types. This capability is available automatically when workloads egress through a StandardV2 NAT Gateway - no additional configuration or rule changes are required. Scenarios enabled by ICMP support Support for ICMP through StandardV2 NAT Gateway unlocks several common operational scenarios: Basic reachability testing – Operators can use ping from virtual machines, VM scale sets, or AKS nodes behind NAT Gateway to verify outbound connectivity to internet endpoints. Health validation and troubleshooting – Teams can perform quick checks during incident triage without deploying jump hosts, public IPs per VM, or temporary firewall exceptions. These scenarios are especially valuable in large‑scale environments where workloads rely on centralized outbound connectivity via NAT Gateway. How ICMP works through StandardV2 NAT Gateway Unlike TCP or UDP, ICMP does not use ports to differentiate flows. Instead, ICMP messages include an identifier field, which is used by diagnostic tools to correlate requests and responses. A common troubleshooting scenario is when workloads deployed behind an Azure StandardV2 NAT Gateway are unable to reach external endpoints. Since these workloads typically do not have public IP addresses, it can be difficult to quickly determine whether the issue is related to basic network reachability or higher-layer configuration. In such cases, ICMP‑based diagnostics such as ping can be used to validate outbound connectivity and help isolate the source of the issue. When a workload sends an outbound ICMP packet through a NAT Gateway: NAT Gateway translates the source private IP to one of its associated public IP addresses. The platform tracks the ICMP flow using the ICMP identifier and other packet attributes. Return ICMP responses from the external destination are matched to the originating flow. The response is translated back to the original private IP and delivered to the workload. This stateful handling allows replies to be routed correctly even when multiple workloads are sending ICMP traffic concurrently through the same NAT Gateway. Using ICMP for end-to-end outbound troubleshooting To validate outbound connectivity using ICMP from workloads deployed behind StandardV2 NAT Gateway: Go to your virtual network in the Azure portal and select the subnet hosting your workload. Verify that a StandardV2 NAT Gateway is associated with the subnet. The workload does not require a public IP address. Connect to the workload (for example, a virtual machine) running in the subnet. Run an ICMP ping to an external destination to validate outbound reachability. Successful ping from a workload behind an Azure StandardV2 NAT Gateway demonstrating outbound connectivity and ICMP Echo reply. Review the results. Successful ICMP echo replies confirm that outbound connectivity through StandardV2 NAT Gateway is functioning and that ICMP traffic is translated and returned correctly Summary With ICMP Echo support, Azure StandardV2 NAT Gateway provides a more complete outbound networking experience by enabling essential diagnostic tools to function reliably through NAT. Customers can validate reachability, perform basic troubleshooting, and diagnose connectivity issues while continuing to benefit from the simplicity and scale of centralized outbound connectivity. Learn More Azure NAT Gateway overview Troubleshoot Azure NAT GatewayNetwork security perimeter for Azure Service Bus & also now available in Azure Gov. Regions
TL; DR Network security perimeter support for Azure Service Bus is now Generally Available. With this, you can now place your Service Bus namespace inside a central security boundary and apply perimeter-based governance for inbound/outbound network access—while keeping key PaaS-to-PaaS scenarios secure and auditable. Introduction We’re excited to announce that network security perimeter support for Azure Service Bus is now Generally Available (GA). This milestone brings one of Azure’s most widely used messaging services into the network security perimeter ecosystem, enabling customers to define a centralized security boundary across messaging and data services. Alongside this, we are also expanding network security perimeter’s reach — network security perimeter is now available in Azure Government regions, including: Texas Arizona Virginia DoD East DoD Central This ensures that customers operating in regulated, sovereign, and mission-critical environments can adopt network security perimeter while meeting their compliance and regional requirements. Why this matters? Modern applications rely heavily on messaging layers like Service Bus. These systems often connect microservices, data platforms, key management systems and external integrations. As architectures scale, managing network access individually becomes complex and error prone. Network security perimeter changes this by introducing a perimeter-based access model, where: Communication is restricted by default Access must be explicitly allowed Governance is applied consistently across services With Service Bus now onboarded and network security perimeter extending into Azure Gov regions, customers can apply this model across both commercial and sovereign environments. What you can do with Service Bus + network security perimeter Confine communication within a security boundary Service Bus namespaces communicate only with resources inside the perimeter by default—blocking unintended access. Secure PaaS-to-PaaS communication Enable secure interactions between Service Bus, Azure Key Vault (for CMK scenarios) and other network security perimeter-enabled services (What is a network security perimeter? - Azure Private Link | Microsoft Learn) Define explicit access controls Inbound rules → IP ranges and subscriptions Outbound rules → FQDN-based filtering Enable audit and compliance visibility Diagnostic logs capture all access attempts, supporting compliance and investigation workflows. Use Private Link seamlessly Private endpoint traffic continues to work without additional configuration inside the perimeter. Azure Government Availability With this update, network security perimeter is now available in key Azure Government regions (Texas Arizona Virginia DoD East DoD Central), enabling: Consistent security across clouds Apply the same network security perimeter model across public Azure regions and Azure Government environments. Support for regulated workloads Customers in federal, defence, and highly regulated industries can now: Enforce perimeter-based governance Reduce exposure risks Meet compliance requirements Enable secure cross-service patterns in Gov clouds Azure Government boundaries now support scenarios like: CMK with Key Vault Service-to-service messaging Controlled external access More details of onboarded PaaS services are detailed in What is a network security perimeter? - Azure Private Link | Microsoft Learn What’s next Service Bus GA further strengthens network security perimeter’s growing coverage across Azure PaaS services. We will continue to: Expand PaaS service onboarding Improve access rule capabilities (e.g. Service tag-based access, identity-based access)
Understanding and building an Azure Hybrid Meshed Hub-Spoke Topology
A meshed hybrid hub-spoke topology Azure offers two main approaches to build network architectures. This article focuses on traditional networking (using VNets, peering, route tables, etc.), rather than Azure Virtual WAN. Why a hub-spoke topology? A hub‑spoke topology is the only way to control traffic flows while maintaining scalability, because it enforces a central point of connectivity and policy enforcement: Centralized traffic control / inspection: All connectivity (to on‑premises, the internet, and between spokes) is anchored through the hub. The hub hosts shared services such as firewalls or NVAs, providing a single control point where traffic is inspected, filtered, and governed consistently. Avoids uncontrolled lateral communication: Spokes do not connect arbitrarily to each other. All connectivity is routed through the hub, preventing uncontrolled east‑west communication and ensuring traffic follows defined security and routing policies. Inherent scalability by design: New workloads are added by introducing additional spokes. The core network design remains unchanged, enabling linear scaling without the complexity of full-mesh connectivity. In summary, the hub‑spoke model provides centralized control combined with scalable, decoupled workload networks—something that flat or full-mesh designs struggle to achieve. From hub-spoke to meshed multi-region In a hub‑spoke topology, it’s important to keep in mind that the hub is implemented as an Azure Virtual Network (VNet) and VNets are scoped to a single region. This means that in a multi‑region setup, you’ll always need at least one hub per region. Each of these hubs hosts shared services like firewalls, NVAs, and DNS, acting as the central point for connectivity and traffic control. Extending dependencies across regions—for example by connecting spokes to a hub in another region—is generally not recommended. It creates tight coupling between regions, which goes against the goal of keeping regions independent. A well-designed multi‑region architecture aims for regional self‑containment to improve resilience and fault isolation. Relying on a remote hub can lead to issues like failure propagation between regions, higher latency for inspected traffic and more complex routing and operations. It can also introduce organizational challenges when different regions are managed by separate teams, reducing agility and increasing operational risk. For this reason, meshed hub‑spoke architectures should use hubs that are deployed within each region. Connectivity between regions should be established directly between the hubs, not through spokes. In a meshed design, hubs are typically connected in a full‑mesh peering model, allowing for controlled and predictable inter‑region communication while still maintaining regional independence. Within a single region, it can also make sense to deploy multiple hubs to create isolated environments. This is especially useful when you need to separate workloads based on security requirements, regulatory needs, or organizational boundaries. Each hub can then have its own dedicated set of connectivity and inspection services. Finally, each spoke VNet connects to just one hub. This keeps routing simple and predictable, ensures that all traffic passes through the correct inspection and policy enforcement layers, and reinforces the hub’s role as the central control point for network traffic within the region. Integrating hybrid connectivity In most enterprise scenarios, Azure doesn’t operate in isolation—it needs to connect to external networks such as on‑premises datacenters or other cloud environments. This hybrid connectivity is typically set up using services like Azure ExpressRoute, Azure VPN Gateway or third‑party SD‑WAN solutions. In a (meshed) hub‑spoke topology, these connectivity components are best deployed in the hub VNet, since the hub acts as the central point where all inbound and outbound traffic comes together. By centralizing external connectivity in the hub, all traffic—whether entering or leaving Azure—can be routed, inspected and governed in a consistent way using shared services like firewalls or NVAs. It also avoids the need to duplicate gateways and connectivity components across multiple spokes, which helps reduce cost and operational overhead. This approach also simplifies routing and policy management. Spokes can rely on the hub’s shared connectivity instead of maintaining their own connections to external networks. Overall, this reinforces the hub’s role as the single, controlled integration point between Azure and the broader network landscape. Implementation fundamentals With the overall architecture in place, the next step is to understand how Azure actually handles routing and traffic control in this kind of design. When working with a hub‑spoke topology in Azure, it’s important to realize that a virtual network (VNet) doesn’t behave like a traditional router. While you can associate Azure Route Tables with subnets, those routes only apply to traffic originating from within that subnet. Traffic entering the VNet from outside isn’t automatically re‑routed. This is also why VNet peering is non‑transitive by design: peered VNets can communicate directly, but they won’t forward traffic for other networks. To enable controlled routing between spokes—and between Azure and external networks such as ExpressRoute or VPN—you need a component in the hub that can actively receive and forward traffic. In most cases, this is handled by an Azure Firewall or a network virtual appliance (NVA) deployed in the hub. These components act as an explicit routing hop: they receive traffic, inspect or process it based on defined policies and then send it back into the virtual network so Azure’s routing engine can continue forwarding it. In a secure hub‑spoke design, the firewall plays a dual role. It not only provides centralized traffic inspection and enforces security policies, but also acts as the mechanism that enables transitive communication between spokes and external networks. This combination of control and connectivity is a key part of the architecture. Of course, this only works as intended if the firewall is configured with the right rules to allow or block traffic according to your security requirements. While it’s technically possible to implement routing using a basic virtual machine or even a Virtual Network Gateway, these approaches don’t meet typical enterprise requirements. They lack built‑in capabilities like advanced traffic inspection, high availability, autoscaling and centralized policy management. Purpose‑built solutions such as Azure Firewall or mature third‑party NVAs are designed to provide not just routing, but also integrated security, consistency, and scalability. For that reason, they’re generally the only realistic choice for production‑grade hub‑spoke environments where both control and resilience matter. Design principles for building the topology The diagram below shows the topology for a hybrid meshed hub-spoke, with 2 hubs and an Azure Firewall (any other 3rd party Firewall could be used as well). Ensuring correct connectivity in a hub-and-spoke topology may initially appear complex, but in practice it comes down to understanding and correctly applying four key design principles: controlled routing in the GatewaySubnet controlled routing in each spoke proper peering of spokes to the hub meshing the hubs. Before looking at these in detail, it is important to understand a fundamental behavior of Azure Virtual Network (VNet) peering. When two VNets are peered, Azure automatically exchanges their address spaces (CIDR ranges) and injects these prefixes as system routes into the effective route tables of all subnets. As a result, resources in one VNet can communicate directly with resources in the other using private IP addressing, without any additional routing configuration. This built-in route propagation is what makes VNet peering an efficient and low-latency connectivity mechanism in Azure. However, this default behavior is not always aligned with the requirements of a hub-and-spoke topology. In this model, network services such as firewalls, inspection and routing control are typically centralized in the hub VNet. If communication between spokes is allowed to follow the automatically injected system routes, traffic could bypass these centralized controls, which would undermine design objectives such as inspection, segmentation and governance. For this reason, although VNet peering provides seamless connectivity by default, additional configuration is required in a hub-and-spoke architecture. This is usually achieved through Azure Route Tables, network virtual appliances (NVAs) or Azure Firewall, ensuring that traffic between spokes is routed through the hub as intended. This approach enables a controlled routing model that is essential for maintaining security and architectural consistency in enterprise-scale Azure environments. Design principle 1: Controlled routing in the GatewaySubnet In hybrid connectivity scenarios, traffic originating from on-premises environments over VPN or ExpressRoute is first terminated by the Azure Virtual Network Gateway. From there, the traffic is injected into the Azure network using the routing context of the GatewaySubnet. By default, this process relies on system routes that are automatically populated through VNet peering. As a result, when the destination resides in a spoke VNet, the traffic is forwarded directly to that spoke, since its address space has already been learned and installed as a system route. While this behavior is efficient, it also means that traffic will bypass centralized security controls in the hub, such as Azure Firewall. To ensure that all incoming traffic is properly inspected, this default routing behavior needs to be adjusted. This is done by associating a custom Azure Route Table with the GatewaySubnet and defining user-defined routes for each spoke address range. These routes should point to the private IP address of the firewall as the next hop, effectively overriding the system routes created by VNet peering. Because Azure gives precedence to user-defined routes over system routes, traffic that would normally go directly to the spoke is instead redirected through the firewall before reaching its destination. It is important that these user-defined routes precisely match the CIDR ranges defined for the spoke VNets! Any mismatch, such as using broader or more specific prefixes, can lead to unexpected routing behavior and may introduce issues such as asymmetric traffic flows or packet loss. For instance, if a spoke uses address spaces like 10.10.10.0/24 and 192.168.10.0/24, these exact prefixes must be reflected in the route table. Only by aligning the custom routes with the advertised address ranges can you ensure predictable routing and consistent inspection through the firewall. If the hub VNet hosts additional resources beyond an Azure Firewall or third-party network virtual appliance that also require traffic inspection, the corresponding CIDR ranges—either for the specific subnets or for the entire hub VNet—should be included as routes in the route table associated with the GatewaySubnet. These routes should be configured in the same way as those for spoke VNets, ensuring that traffic destined for these resources is directed through the intended inspection point. A typical example is Azure DNS Private Resolver, which can include both inbound and outbound endpoints deployed in dedicated subnets. When such endpoints are present in the hub, their associated subnet address ranges must also be added to the route table for the GatewaySubnet. This ensures that traffic to and from these endpoints is routed through the designated inspection path, maintaining consistent enforcement of security controls. Design principle 2: Controlled routing in every spoke In a hub-and-spoke architecture, traffic flows should follow the intended security model. Workloads within the same spoke VNet are usually treated as part of the same trust boundary, so traffic between resources in that spoke can flow directly over the Azure backbone without needing to pass through centralized controls. Network Security Groups (NSGs) should still be used at the subnet level to provide granular, stateful filtering, but routing this traffic through a central firewall is typically not required. The situation changes when traffic leaves the local VNet. As soon as traffic is destined for another spoke, the hub, or on-premises networks, it crosses a trust boundary and needs to be inspected centrally. To enforce this, Azure’s default routing behavior must be overridden by associating an Azure Route Table with each subnet in the spoke VNets. In most cases, this route table can be kept simple by defining a single default route that sends all outbound, non-local traffic to the firewall in the hub: Destination: 0.0.0.0/0 Next hop: Private IP address of the hub firewall (Virtual Appliance) With this configuration in place, all traffic that is not local to the spoke is forced through the hub, ensuring that communication between VNets and towards external networks is inspected and controlled. From a management perspective, the same route table can often be reused across multiple subnets or even multiple VNets within the same subscription, which helps keep the design consistent and easy to maintain. It’s worth noting, however, that Azure requires route tables and the subnets they’re associated with to be in the same subscription, as this association is enforced by the platform. There is one additional setting that is often overlooked but plays an important role in getting routing right in a hub-and-spoke design. Azure route tables include an option called “Propagate gateway routes”, which controls whether routes learned by a Virtual Network Gateway are added to the effective routes of the associated subnets. By default, routes learned via BGP (for example from ExpressRoute or VPN) or defined through a Local Network Gateway are propagated not only within the hub VNet, but also across VNet peerings. This means that spoke VNets can automatically learn routes to on-premises or external networks and may send traffic directly to the gateway, bypassing the firewall in the hub. To avoid this and keep traffic flowing through the centralized security controls, this setting should be disabled on the route tables used by the spoke subnets. When “Propagate gateway routes” is set to No, routes learned by the gateway are no longer injected into the spokes. As a result, traffic to those destinations cannot take a direct path and instead follows the user-defined default route (0.0.0.0/0) toward the hub firewall, where it can be properly inspected. When combined with the default route to the firewall, this setup ensures that traffic—whether it is going to other VNets, on-premises environments, or external networks—always follows a controlled and predictable path through the hub. This helps maintain consistent security enforcement and avoids unexpected routing behavior in larger or hybrid deployments. Design principle 3: Peering the spokes to the hub Virtual Network (VNet) peering in Azure is often seen as a simple, single configuration, but in reality it is directional by design. To fully connect two VNets, you need two separate peering configurations—one in each direction—and both must be configured correctly to ensure not only connectivity, but also proper routing behavior. Each peering exposes four key settings and getting these right is especially important in a hub-and-spoke architecture. For basic connectivity, the first two settings—“allow virtual network access” and “allow forwarded traffic”—should be enabled on both peerings. These ensure that traffic can flow between VNets and support scenarios where traffic is routed through a central component, such as a firewall in the hub. The other two settings depend on the direction of the peering. In a typical hub-and-spoke setup, the Virtual Network Gateway (or Azure Route Server) is deployed in the hub. This means the peering from the spoke to the hub must enable “use remote gateways”, while the peering from the hub to the spoke must enable “allow gateway transit.” At first, this might seem to contradict the idea that spokes should not directly use the gateway. However, these settings influence control plane behavior and don't enable unrestricted traffic flow. They are required so the gateway can learn and advertise spoke address ranges via BGP to external networks, such as those connected over VPN or ExpressRoute. Whether those routes are actually used in the spokes is still controlled through the “propagate gateway routes” setting on the route tables, allowing you to enforce routing through the firewall as intended. Even if you are not currently using BGP—for example, in environments relying on static routing—it is still a good practice to configure peerings this way. Doing so makes the design future-proof, allowing you to introduce dynamic routing later without changes to the peering model. This approach keeps the architecture consistent and avoids unnecessary rework as the environment evolves. Design principle 4: Meshing the hubs When you extend a hub-and-spoke design across multiple regions, you typically introduce multiple hubs, each managing its own regional spokes. In this setup, it becomes important to connect the hubs to each other, which is done by fully meshing the hub VNets using VNet peering. At the same time, a key principle remains unchanged: each spoke should connect to only one hub in the same region. This keeps the architecture simple, scalable and easier to reason about from a routing perspective. When configuring connectivity between hubs, it’s important to note that VNet peering settings differ from the typical hub–spoke configuration. For inter-hub peerings, only “allow virtual network access” and “allow forwarded traffic” should be enabled. The remaining options—“allow gateway transit” and “use remote gateways”—should be left disabled, as gateway sharing is not required between hubs and would even be blocked in the configuration. Just connecting the hubs with peering is not enough to guarantee correct traffic flow. To ensure traffic moves between regions in a controlled and secure way, you need additional routing logic. Each hub should have an Azure Route Table assigned to its FirewallSubnet (or the subnet hosting the 3rd party NVAs) defining how traffic towards other hub-and-spoke environments is handled. This ensures that inter-region traffic is always routed through the appropriate hub firewall, instead of flowing directly across the Azure backbone. At this point, IP address planning becomes critical. Without a clear addressing strategy, routing quickly becomes complex and hard to maintain. A common best practice is to assign a single “master” CIDR range per region, and then allocate all VNets in that region—both hub and spokes—from that range. This creates a clean, hierarchical addressing model that simplifies routing decisions. With this approach in place, route tables can remain relatively simple. Instead of adding routes for every individual spoke, you only need one route per remote region. The destination is the master CIDR range of that region and the next hop is the private IP of the firewall in the corresponding hub. Because all hubs are peered with each other, these address ranges and firewall endpoints are automatically known through peering, allowing for consistent and predictable routing. Overall, this design keeps routing logic straightforward while ensuring that all inter-region traffic is inspected in the correct hub, preserving the security model and making it easy to scale as new regions are added. Conclusion When the four design principles described in this article are applied consistently, a hub-and-spoke architecture becomes a strong, scalable and easy-to-operate foundation for your network. By combining controlled routing, centralized inspection and clear traffic flows, the model delivers both solid security and predictable behavior, even in complex environments. More importantly, the concepts covered here go beyond just one specific design. They represent the key building blocks of Azure networking, including routing, peering and traffic control. Understanding these fundamentals not only helps you implement hub-and-spoke topologies correctly, but also gives you a solid base for designing and running reliable, enterprise-grade network architectures in Azure. To make this easier to apply in practice, the table below summarizes the main concepts from this article and how they translate into actual configuration. It can be useful both when setting up a hub-and-spoke topology and when troubleshooting existing environments. Area Configuration Key Setting / Value Purpose Hub VNet Deploy shared services Azure Firewall or NVA in hub Central inspection + routing Deploy connectivity VPN Gateway / ExpressRoute in hub Centralize hybrid connectivity GatewaySubnet Associate Route Table UDRs for each spoke CIDR → Firewall IP Force inbound traffic through firewall Spoke Subnets Associate Route Table 0.0.0.0/0 → Firewall (Virtual Appliance) Force all outbound traffic via hub Route Table setting Propagate gateway routes = Disabled Prevent bypass of firewall via gateway VNet Peering (Spoke → Hub) Setting Allow VNet access = Yes Basic connectivity Setting Allow forwarded traffic = Yes Support transitive routing via firewall Setting Allow gateway transit = Yes Allow spoke to leverage hub gateway Setting Use remote gateways = No - VNet Peering (Hub → Spoke) Setting Allow VNet access = Yes Basic connectivity Setting Allow forwarded traffic = Yes Support routing through firewall Setting Allow gateway transit = No - Setting Use remote gateways = Yes Advertise spoke prefixes via hub gateway VNet Peering (Hub→ Hub) Setting Allow VNet access = Yes Basic connectivity Setting Allow forwarded traffic = Yes Support transitive routing via firewall Setting Allow gateway transit = No - Setting Use remote gateways = No - Hub FirewallSubnet Associate Route Table Route remote region CIDR → remote hub firewall IP Ensure inter-region/hub routing Addressing strategy CIDR planning Assign master CIDR per region Simplify routing and reduce UDR complexity Spoke design rule Peering constraint Each spoke connected to one hub only Prevent routing ambiguityAzure Front Door: Implementing lessons learned following October outages
Abhishek Tiwari, Vice President of Engineering, Azure Networking Amit Srivastava, Principal PM Manager, Azure Networking Varun Chawla, Partner Director of Engineering Introduction Azure Front Door is Microsoft's advanced edge delivery platform encompassing Content Delivery Network (CDN), global security and traffic distribution into a single unified offering. By using Microsoft's extensive global edge network, Azure Front Door ensures efficient content delivery and advanced security through 210+ global and local points of presence (PoPs) strategically positioned closely to both end users and applications. As the central global entry point from the internet onto customer applications, we power mission critical customer applications as well as many of Microsoft’s internal services. We have a highly distributed resilient architecture, which protects against failures at the server, rack, site and even at the regional level. This resiliency is achieved by the use of our intelligent traffic management layer which monitors failures and load balances traffic at server, rack or edge sites level within the primary ring, supplemented by a secondary-fallback ring which accepts traffic in case of primary traffic overflow or broad regional failures. We also deploy a traffic shield as a terminal safety net to ensure that in the event of a managed or unmanaged edge site going offline, end user traffic continues to flow to the next available edge site. Like any large-scale CDN, we deploy each customer configuration across a globally distributed edge fleet, densely shared with thousands of other tenants. While this architecture enables global scale, it carries the risk that certain incompatible configurations, if not contained, can propagate broadly and quickly which can result in a large blast radius of impact. Here we describe how the two recent service incidents impacting Azure Front Door have reinforced the need to accelerate ongoing investments in hardening our resiliency, and tenant isolation strategy to mitigate likelihood and the scale of impact from this class of risk. October incidents: recap and key learnings Azure Front Door experienced two service incidents; on October 9 th and October 29 th , both with customer-impacting service degradation. On October 9 th : A manual cleanup of stuck tenant metadata bypassed our configuration protection layer, allowing incompatible metadata to propagate beyond our canary edge sites. This metadata was created on October 7 th , from a control-plane defect triggered by a customer configuration change. While the protection system initially blocked the propagation, the manual override operation bypassed our safeguards. This incompatible configuration reached the next stage and activated a latent data-plane defect in a subset of edge sites, causing availability impact primarily across Europe (~6%) and Africa (~16%). You can learn more about this issue in detail at https://aka.ms/AIR/QNBQ-5W8 On October 29 th : A different sequence of configuration changes across two control-plane versions produced incompatible metadata. Because the failure mode in the data-plane was asynchronous, the health checks validations embedded in our protection systems were all passed during the rollout. The incompatible customer configuration metadata successfully propagated globally through a staged rollout and also updated the “last known good” (LKG) snapshot. Following this global rollout, the asynchronous process in data-plane exposed another defect which caused crashes. This impacted connectivity and DNS resolutions for all applications onboarded to our platform. Extended recovery time amplified impact on customer applications and Microsoft services. You can learn more about this issue in detail at https://aka.ms/AIR/YKYN-BWZ We took away a number of clear and actionable lessons from these incidents, which are applicable not just to our service, but to any multi-tenant, high-density, globally distributed system. Configuration resiliency – Valid configuration updates should propagate safely, consistently, and predictably across our global edge, while ensuring that incompatible or erroneous configuration never propagate beyond canary environments. Data plane resiliency - Additionally, configuration processing in the data plane must not cause availability impact to any customer. Tenant isolation – Traditional isolation techniques such as hardware partitioning and virtualization are impractical at edge sites. This requires innovative sharding techniques to ensure single tenant-level isolation – a must-have to reduce potential blast radius. Accelerated and automated recovery time objective (RTO) – System should be able to automatically revert to last known good configuration in an acceptable RTO. In case of a service like Azure Front Door, we deem ~10 mins to be a practical RTO for our hundreds of thousands of customers at every edge site. Post outage, given the severity of impact which allowed an incompatible configuration to propagate globally, we made the difficult decision to temporarily block configuration changes in order to expedite rollout of additional safeguards. Between October 29 th to November 5 th , we prioritized and deployed immediate hardening steps before opening up the configuration change. We are confident that the system is stable, and we are continuing to invest in additional safeguards to further strengthen the platform's resiliency. Learning category Goal Repairs Status Safe customer configuration deployment Incompatible configuration never propagates beyond Canary · Control plane and data plane defect fixes · Forced synchronous configuration processing · Additional stages with extended bake time · Early detection of crash state Completed Data plane resiliency Configuration processing cannot impact data plane availability Manage data-plane lifecycle to prevent outages caused by configuration-processing defects. Completed Isolated work-process in every data plane server to process and load the configuration. January 2026 100% Azure Front Door resiliency posture for Microsoft internal services Microsoft operates an isolated, independent Active/Active fleet with automatic failover for critical Azure services Phase 1: Onboarded critical services batch impacted on Oct 29 th outage running on a day old configuration Completed Phase 2: Automation & hardening of operations, auto-failover and self-management of Azure Front Door onboarding for additional services March 2026 Recovery improvements Data plane crash recovery in under 10 minutes Data plane boot-up time optimized via local cache (~1 hour) Completed Accelerate recovery time < 10 minutes March 2026 Tenant isolation No configuration or traffic regression can impact other tenants Micro cellular Azure Front Door with ingress layered shards June 2026 This blog is the first in a multi-part series on Azure Front Door resiliency. In this blog, we will focus on configuration resiliency—how we are making the configuration pipeline safer and more robust. Subsequent blogs will cover tenant isolation and recovery improvements. How our configuration propagation works Azure Front Door configuration changes can be broadly classified into three distinct categories. Service code & data – these include all aspects of Azure Front Door service like management plane, control plane, data plane, configuration propagation system. Azure Front Door follows a safe deployment practice (SDP) process to roll out newer versions of management, control or data plane over a period of approximately 2-3 weeks. This ensures that any regression in software does not have a global impact. However, latent bugs that escape pre-validation and SDP rollout can remain undetected until a specific combination of customer traffic patterns or configuration changes trigger the issue. Web Application Firewall (WAF) & L7 DDoS platform data – These datasets are used by Azure Front Door to deliver security and load-balancing capabilities. Examples include GeoIP data, malicious attack signatures, and IP reputation signatures. Updates to these datasets occur daily through multiple SDP stages with an extended bake time of over 12 hours to minimize the risk of global impact during rollout. This dataset is shared across all customers and the platform, and it is validated immediately since it does not depend on variations in customer traffic or configuration steps. Customer configuration data – Examples of these are any customer configuration change—whether a routing rule update, backend pool modification, WAF rule change, or security policy change. Due to the nature of these changes, it is expected across the edge delivery / CDN industry to propagate these changes globally in 5-10 mins. Both outages stemmed from issues within this category. All configuration changes, including customer configuration data, are processed through a multi-stage pipeline designed to ensure correctness before global rollout across Azure Front Door’s 200+ edge locations. At a high level, Azure Front Door’s configuration propagation system has two distinct components - Control plane – Accepts customer API/portal changes (create/update/delete for profiles, routes, WAF policies, origins, etc.) and translates them into internal configuration metadata which the data plane can understand. Data plane – Globally distributed edge servers that terminate client traffic, apply routing/WAF logic, and proxy to origins using the configuration produced by the control plane. Between these two halves sits a multi-stage configuration rollout pipeline with a dedicated protection system (known as ConfigShield): Changes flow through multiple stages (pre-canary, canary, expanding waves to production) rather than going global at once. Each stage is health-gated: the data plane must remain within strict error and latency thresholds before proceeding. Each stage’s health check also rechecks previous stage’s health for any regressions. A successfully completed rollout updates a last known good (LKG) snapshot used for automated rollback. Historically, rollout targeted global completion in roughly 5–10 minutes, in line with industry standards. Customer configuration processing in Azure Front Door data plane stack Customer configuration changes in Azure Front Door traverse multiple layers—from the control plane through the deployment system—before being converted into FlatBuffers at each Azure Front Door node. These FlatBuffers are then loaded by the Azure Front Door data plane stack, which runs as Kubernetes pods on every node. FlatBuffer Composition: Each FlatBuffer references several sub-resources such as WAF and Rules Engine schematic files, SSL certificate objects, and URL signing secrets. Data plane architecture: o Master process: Accepts configuration changes (memory-mapped files with references) and manages the lifecycle of worker processes. o Workers: L7 proxy processes that serve customer traffic using the applied configuration. Processing flow for each configuration update: Load and apply in master: The transformed configuration is loaded and applied in the master process. Cleanup of unused references occurs synchronously except for certain categories à October 9 outage occurred during this step due to a crash triggered by incompatible metadata. Apply to workers: Configuration is applied to all worker processes without memory overhead (FlatBuffers are memory-mapped). Serve traffic: Workers start consuming new FlatBuffers for new requests; in-flight requests continue using old buffers. Old buffers are queued for cleanup post-completion. Feedback to deployment service: Positive feedback signals readiness for rollout.Cleanup: FlatBuffers are freed asynchronously by the master process after all workers load updates à October 29 outage occurred during this step due to a latent bug in reference counting logic. The October incidents showed we needed to strengthen key aspects of configuration validation, propagation safeguards, and runtime behavior. During the Azure Front Door incident on October 9 th , that protection system worked as intended but was later bypassed by our engineering team during a manual cleanup operation. During this Azure Front Door incident on October 29 th , the incompatible customer configuration metadata progressed through the protection system, before the delayed asynchronous processing task resulted in the crash. Configuration propagation safeguards Based on learnings from the incidents, we are implementing a comprehensive set of configuration resiliency improvements. These changes aim to guarantee that any sequence of configuration changes cannot trigger instability in the data plane, and to ensure quicker recovery in the event of anomalies. Strengthening configuration generation safety This improvement pivots on a ‘shift-left’ strategy where we want to ensure that we catch regression early before they propagate to production. It also includes fixing the latent defects which were the proximate cause of the outage. Fixing outage specific defects - We have fixed the control-plane defects that could generate incompatible tenant metadata under specific operation sequences. We have also remediated the associated data-plane defects. Stronger cross-version validation - We are expanding our test and validation suite to account for changes across multiple control plane build versions. This is expected to be fully completed by February 2026. Fuzz testing - Automated fuzzing and testing of metadata generation contract between the control plane and the data plane. This allows us to generate an expanded set of invalid/unexpected configuration combinations which might not be achievable by traditional test cases alone. This is expected to be fully completed by February 2026. Preventing incompatible configurations from being propagated This segment of the resiliency strategy strives to ensure that a potentially dangerous configuration change never propagates beyond canary stage. Protection system is “always-on” - Enhancements to operational procedures and tooling prevent bypass in all scenarios (including internal cleanup/maintenance), and any cleanup must flow through the same guarded stages and health checks as standard configuration changes. This is completed. Making rollout behavior more predictable and conservative - Configuration processing in the data plane is now fully synchronous. Every data plane issue due to incompatible meta data can be detected withing 10 seconds at every stage. This is completed. Enhancement to deployment pipeline - Additional stages during roll-out and extended bake time between stages serve as an additional safeguard during configuration propagation. This is completed. Recovery tool improvements now make it easier to revert to any previous version of LKG with a single click. This is completed. These changes significantly improve system safety. Post-outage we have increased the configuration propagation time to approximately 45 minutes. We are working towards reducing configuration propagation time closer to pre-incident levels once additional safeguards covered in the Data plane resiliency section below are completed by mid-January, 2026. Data plane resiliency The data plane recovery was the toughest part of recovery efforts during the October incidents. We must ensure fast recovery as well as resilience to configuration processing related issues for the data plane. To address this, we implemented changes that decouple the data plane from incompatible configuration changes. With these enhancements, the data plane continues operating on the last known good configuration—even if the configuration pipeline safeguards fail to protect as intended. Decoupling data plane from configuration changes Each server’s data plane consists of a master process which accepts configuration changes and manages lifecycle of multiple worker processes which serve customer traffic. One of the critical reasons for the prolonged outage in October was that due to latent defects in the data plane, when presented with a bad configuration the master process crashed. The master is a critical command-and-control process and when it crashes it takes down the entire data plane, in that node. Recovery of the master process involves reloading hundreds of thousands of configurations from scratch and took approximately 4.5 hours. We have since made changes to the system to ensure that even in the event of the master process crash due to any reason - including incompatible configuration data being presented - the workers remain healthy and able to serve traffic. During such an event, the workers would not be able to accept new configuration changes but will continue to serve customer traffic using the last known good configuration. This work is completed. Introducing Food Taster: strengthening config propagation resiliency In our efforts to further strengthen Azure Front Door’s configuration propagation system, we are introducing an additional configuration safeguard known internally as Food Taster which protects the master and worker processes from any configuration change related incidents, thereby ensuring data plane resiliency. The principle is simple: every data-plane server will have a redundant and isolated process – the Food Taster – whose only job is to ingest and process new configuration metadata first and then pass validated configuration changes to active data plane. This redundant worker does not accept any customer traffic. All configuration processing in this Food Taster is fully synchronous. That means we do all parsing, validation, and any expensive or risky work up front, and we do not move on until the Food Taster has either proven the configuration is safe or rejected it. Only when the Food Taster successfully loads the configuration and returns “Config OK” does the master process proceed to load the same config and then instruct the worker processes to do the same. If anything goes wrong in the Food Taster, the failure is contained to that isolated worker; the master and traffic-serving workers never see that invalid configuration. We expect this safeguard to reach production globally in January 2026 timeframe. Introduction of this component will also allow us to return closer to pre-incident level of configuration propagation while ensuring data plane safety. Closing This is the first in a series of planned blogs on Azure Front Door resiliency enhancements. We are continuously improving platform safety and reliability and will transparently share updates through this series. Upcoming posts will cover advancements in tenant isolation and improvements to recovery time objectives (RTO). We deeply value our customers’ trust in Azure Front Door. The October incidents reinforced how critical configuration resiliency is, and we are committed to exceeding industry expectations for safety, reliability, and transparency. By hardening our configuration pipeline, strengthening safety gates, and reinforcing isolation boundaries, we’re making Azure Front Door even more resilient so your applications can be too.
