Introducing Azure Front Door edge actions - Bringing secure, programmable logic to the edge
Modern web applications are expected to feel instant, secure, and personalized, no matter where users connect from or what device they use. Delivering that experience requires more than moving static content closer to users. It increasingly depends on making smart decisions at the edge, from validating requests and selecting the right origin to personalizing responses and stopping suspicious traffic before it reaches the backend. Because of this, over the last decade, compute resources for web applications have steadily moved closer to users. What began with content caching at the edge quickly evolved into dynamic routing, security enforcement, and intelligent application traffic management. Today, enterprises expect even more: programmable compute at the edge! Developers and platform teams want the flexibility to run lightweight logic at the edge without giving up the scale, security, and operational confidence they expect from Azure. Azure Front Door through its rulesets offers a robust, declarative framework for applying common traffic management and security policies, such as redirects, header manipulation, and conditional routing. They are optimized for scenarios where behavior can be defined statically and evaluated efficiently at scale. As customer workloads become more dynamic, and application logic increasingly moves closer to the user, additional capabilities are often required beyond declarative configuration. Meet Azure Front Door edge actions, now in public preview We are excited to announce Azure Front Door edge actions which represent Microsoft’s next step in this evolution. With edge actions, customers can execute custom logic directly at Microsoft’s global edge, enabling new classes of real-time personalization, security, and resiliency scenarios, without pushing complexity back to origins! Edge actions allow customers to author lightweight JavaScript functions that execute as part of Azure Front Door request processing. In the current public preview, edge actions are invoked during the client request phase and are attached to Azure Front Door routes through rulesets. This tight integration means edge actions work seamlessly with existing Azure Front Door capabilities, including Web Application Firewall (WAF), caching, and routing - while extending them with programmable logic. Customers can incrementally adopt edge actions without re-architecting their applications or delivery pipelines. What can you build with edge actions today Edge actions in public preview are optimized for lightweight, latency-sensitive scenarios that benefit from immediate decision-making. Common use cases include: A/B experimentation and canary rollouts: Evaluate request context at the edge and route users to different application experiences or releases without adding origin-side decision logic. Request and response header manipulation: Add, remove, or transform headers to support security controls, experimentation, routing, and application modernization patterns. Request rejection: Stop unwanted, malformed, or unauthorized requests before they consume origin resources. Dynamic origin selection: Choose the best origin based on request attributes, health signals, geography, device type, or business logic. URL rewrite and redirect: Adapt paths or redirect users at the edge to simplify migrations, campaign launches, localization, and application routing. Authentication and authorization scenarios: Validate tokens or request attributes close to the user, helping protect applications before traffic reaches backend services. Please refer to Azure/EdgeActionsSamples to start building edge actions today. Designed for industry leading security-first edge compute Running customer-defined code at the edge introduces unique security challenges. Edge environments now process untrusted user code at massive scales, making strong isolation non-negotiable for Azure. Enter Hyperlight: Microsoft’s foundation for secure execution. With Hyperlight, each edge action code runs in its own lightweight, hardware-backed micro-VM - like a secure apartment, isolated from neighbors. Unlike traditional virtual machines, Hyperlight micro‑VMs are extremely lightweight and do not have a general-purpose guest operating system. Because of their small footprint, they provide a much more reduced attack surface with a hardware-backed isolation boundary between every instance of customer code, the host infrastructure, and other tenants. This model allows Azure Front Door to combine the safety properties of hardware virtualization with the performance characteristics required for edge compute. Customer code is isolated by design to reduce blast radius to a single instance, helping protect both the platform and neighboring workloads. How edge actions fit into the request path When a client request reaches Azure Front Door, it follows Azure Front Door’s standard routing pipeline. The request is first evaluated against any configured WAF policies and routing rules. If a rule is configured to invoke an edge action, execution is handed off to the edge actions runtime (Hyperlight) at the edge POP. At execution time, Azure Front Door provides the edge action with a constrained, immutable context that includes request metadata, server variables, origin health information, and geo or device signals. The edge action can then modify the request, generate a response, or influence routing decisions, all within strict execution and resource limits designed to preserve platform stability. Ship safely with versions and execution filters Edge actions are built for operational confidence. Each edge action supports multiple versions of code, and customers can control which version executes using execution filters. This enables header-based routing, canary deployments, and A/B testing scenarios without downtime. Teams can gradually introduce new logic, validate behavior with real traffic, and roll back instantly if needed, bringing modern DevOps practices directly to the edge. Observe and operate with confidence Edge actions integrate with Azure’s existing observability stack. Customers can emit logs from their edge action code and correlate execution results with Azure Front Door access and routing logs. This unified view simplifies troubleshooting and provides visibility into edge execution behavior in production environments. Pricing Please refer to the pricing document for details regarding Edge actions pricing, including applicable billing meters and usage-based charges on invocations and execution time. What’s next for edge actions The public preview of edge actions represents the foundation of a broader roadmap. Over time, Microsoft plans to expand invocation points, capabilities, and integrations - while continuing to prioritize security, performance, and operational simplicity. Stay tuned for scenarios like response invocations, image/video optimizations, and edge inferencing during general availability. By combining programmable edge logic with Hyperlight-based isolation, Azure Front Door edge actions mark a significant milestone in Microsoft’s edge strategy - enabling customers to build faster, safer, and more adaptive applications on a global scale. Start building with Azure Front Door edge actions today! Explore the documentation and bring your complex edge scenarios to life.Azure Front Door: Resiliency Series – Part 2: Faster recovery (RTO)
Abhishek Tiwari, Vice President of Engineering, Azure Networking Amit Srivastava, Partner Director of PM, Azure Networking Varun Chawla, Partner Director of Engineering, Azure Networking Karthik Uthaman, Principal Engineer, Azure Networking In Part 1 of this blog series, we outlined our four‑pillar strategy for resiliency in Azure Front Door: configuration resiliency, data plane resiliency, tenant isolation, and accelerated Recovery Time Objective (RTO). Together, these pillars help Azure Front Door remain continuously available and resilient at global scale. Part 1 focused on the first two pillars: configuration and data plane resiliency. Our goal is to make configuration propagation safer, so incompatible changes never escape pre‑production environments. We discussed how incompatible configurations are blocked early, and how data plane resiliency ensures the system continues serving traffic from a last‑known‑good (LKG) configuration even if a bad change manages to propagate. We also introduced ‘Food Taster’, a dedicated sacrificial process running in each edge server’s data plane, that pretests every configuration change in isolation, before it ever reaches the live data plane. In this post, we turn to the recovery pillar. We describe how we have made key enhancements to the Azure Front Door recovery path so the system can return to full operation in a predictable and bounded timeframe. For a global service like Azure Front Door, serving hundreds of thousands of tenants across 210+ edge sites worldwide, we set an explicit target: to be able to recover any edge site – or all edge sites – within approximately 10 minutes, even in worst‑case scenarios. In typical data plane crash scenarios, we expect recovery in under a second. Repair status The first blog post in this series mentioned the two Azure Front Door incidents from October 2025 – learn more by watching our Azure Incident Retrospective session recordings for the October 9 th incident and/or the October 29 th incident. Before diving into our platform investments for improving our Recovery Time Objectives (RTO), we wanted to provide a quick update on the overall repair items from these incidents. We are pleased to report that the work on configuration propagation and data plane resiliency is now complete and fully deployed across the platform (in the table below, “Completed” means broadly deployed in production). With this, we have reduced configuration propagation latency from ~45 minutes to ~20 minutes. We anticipate reducing this even further – to ~15 minutes by the end of April 2026, while ensuring that platform stability remains our top priority. Learning category Goal Repairs Status Safe customer configuration deployment Incompatible configuration never propagates beyond ‘EUAP or canary regions’ Control plane and data plane defect fixes Forced synchronous configuration processing Additional stages with extended bake time Early detection of crash state Completed Data plane resiliency Configuration processing cannot impact data plane availability Manage data-plane lifecycle to prevent outages caused by configuration-processing defects. Completed Isolated work-process in every data plane server to process and load the configuration. Completed 100% Azure Front Door resiliency posture for Microsoft internal services Microsoft operates an isolated, independent Active/Active fleet with automatic failover for critical Azure services Phase 1: Onboarded critical services batch impacted on Oct 29 th outage running on a day old configuration Completed Phase 2: Automation & hardening of operations, auto-failover and self-management of Azure Front Door onboarding for additional services March 2026 Recovery improvements Data plane crash recovery in under 10 minutes Data plane boot-up time optimized via local cache (~1 hour) Completed Accelerate recovery time < 10 minutes April 2026 Tenant isolation No configuration or traffic regression can impact other tenants Micro cellular Azure Front Door with ingress layered shards June 2026 Why recovery at edge scale is deceptively hard To understand why recovery took as long as it did, it helps to first understand how the Azure Front Door data plane processes configuration. Azure Front Door operates in 210+ edge sites with multiple servers per site. The data plane of each edge server hosts multiple processes. A master process orchestrates the lifecycle of multiple worker processes, that serve customer traffic. A separate configuration translator process runs alongside the data plane processes, and is responsible for converting customer configuration bundles from the control plane into optimized binary FlatBuffer files. This translation step, covering hundreds of thousands of tenants, represents hours of cumulative computation. A per edge server cache is kept locally at each server level – to enable a fast recovery of the data plane, if needed. Once the configuration translator process produces these FlatBuffer files, each worker processes them independently and memory-maps them for zero-copy access. Configuration updates flow through a two-phase commit: new FlatBuffers are first loaded into a staging area and validated, then atomically swapped into production maps. In-flight requests continue using the old configuration, until the last request referencing them completes. The data process recovery is designed to be resilient to different failure modes. A failure or crash at the worker process level has a typical recovery time of less than one second. Since each server has multiple such worker processes which serve customer traffic, this type of crash has no impact on the data plane. In the case of a master process crash, the system automatically tries to recover using the local cache. When the local cache is reused, the system is able to recover quickly – in approximately 60 minutes – since most of the configurations in the cache were already loaded into the data plane before the crash. However, in certain cases if the cache becomes unavailable or must be invalidated because of corruption, the recovery time increases significantly. During the October 29 th incident, a data plane crash triggered a complete recovery sequence that took approximately 4.5 hours. This was not because restarting a process is slow, it is because a defect in the recovery process invalidated the local cache, which meant that “restart” meant rebuilding everything from scratch. The configuration translator process then had to re-fetch and re-translate every one of the hundreds of thousands of customer configurations, before workers could memory-map them and begin serving traffic. This experience has crystallized three fundamental learnings related to our recovery path: Expensive rework: A subset of crashes discarded all previously translated FlatBuffer artifacts, forcing the configuration translator process to repeat hours of conversion work that had already been validated and stored. High restart costs: Every worker on every node had to wait for the configuration translator process to complete the full translation, before it could memory-map any configuration and begin serving requests. Unbounded recovery time: Recovery time grew linearly with total tenant footprint rather than with active traffic, creating a ‘scale penalty’ as more tenants onboarded to the system. Separately and together, the insight was clear: recovery must stop being proportional to the total configuration size. Persisting ‘validated configurations’ across restarts One of the key recovery improvements was strengthening how validated customer configurations are cached and reused across failures, rather than rebuilding configuration states from scratch during recovery. Azure Front Door already cached customer configurations on host‑mounted storage prior to the October incident. The platform enhancements post outage focused on making the local configuration cache resilient to crashes, partial failures, and bad tenant inputs. Our goal was to ensure that recovery behavior is dominated by serving traffic safely, not by reconstructing configuration state. This led us to two explicit design goals… Design goals No category of crash should invalidate the configuration cache: Configuration cache invalidation must never be the default response to failures. Whether the failure is a worker crash, master crash, data plane restart, or coordinated recovery action, previously validated customer configurations should remain usable—unless there is a proven reason to discard it. Bad tenant configuration must not poison the entire cache: A single faulty or incompatible tenant configuration should result in targeted eviction of that tenant’s configuration only—not wholesale cache invalidation across all tenants. Platform enhancements Previously, customer configurations persisted to host‑mounted storage, but certain failure paths treated the cache as unsafe and invalidated it entirely. In those cases, recovery implicitly meant reloading and reprocessing configuration for hundreds of thousands of tenants before traffic could resume, even though the vast majority of cached data was still valid. We changed the recovery model to avoid invalidating customer configurations, with strict scoping around when and how cached entries are discarded: Cached configurations are no longer invalidated based on crash type. Failures are assumed to be orthogonal to configuration correctness unless explicitly proven otherwise. Cache eviction is granular and tenant‑scoped. If a cached configuration fails validation or load checks, only that tenant’s configuration is discarded and reloaded. All other tenant configurations remain available. This ensures that recovery does not regress into a fleet‑wide rebuild due to localized or unrelated faults. Safety and correctness Durability is paired with strong correctness controls, to prevent unsafe configurations from being served: Per‑tenant validation on load: Each cached tenant configuration is validated during the ‘load and verification’ phase, before being promoted for traffic serving. Therefore, failures are contained to that tenant. Targeted re‑translation: When validation fails, only the affected tenant’s configuration is reloaded or reprocessed. Therefore, the cache for other tenants is left untouched. Operational escape hatch: Operators retain the ability to explicitly instruct a clean rebuild of the configuration cache (with proper authorization), preserving control without compromising the default fast‑recovery path. Resulting behavior With these changes, recovery behavior now aligns with real‑world traffic patterns - configuration defects impact tenants locally and predictably, rather than globally. The system now prefers isolated tenant impact, and continued service using last-known-good over aggressive invalidation, both of which are critical for predictable recovery at the scale of Azure Front Door. Making recovery scale with active traffic, not total tenants Reusing configuration cache solves the problem of rebuilding configuration in its entirety, but even with a warm cache, the original startup path had a second bottleneck: eagerly loading a large volume of tenant configurations into memory before serving any traffic. At our scale, memory-mapping, parsing hundreds of thousands of FlatBuffers, constructing internal lookup maps, adding Transport Layer Security (TLS) certificates and configuration blocks for each tenant, collectively added almost an hour to startup time. This was the case even when a majority of those tenants had no active traffic at that moment. We addressed this by fundamentally changing when configuration is loaded into workers. Rather than eagerly loading most of the tenants at startup across all edge locations, Azure Front Door now uses an Machine Learning (ML)-optimized lazy loading model. In the new architecture, instead of loading a large number of tenant configurations, we only load a small subset of tenants that are known to be historically active in a given site, we call this the “warm tenants” list. The warm tenants list per edge site is created through a sophisticated traffic analysis pipeline that leverages ML. However, loading the warm tenants is not good enough, because when a request arrives and we don’t have the configuration in memory, we need to know two things. Firstly, is this a request from a real Azure Front Door tenant – and, if it is, where can I find the configuration? To answer these questions, each worker maintains a hostmap that tracks the state of each tenant’s configuration. This hostmap is constructed during startup, as we process each tenant configuration – if the tenant is in the warm list, we will process and load their configuration fully; if not, then we will just add an entry into the hostmap where all their domain names are mapped to the configuration path location. When a request arrives for one of these tenants, the worker loads and validates that tenant’s configuration on demand, and immediately begins serving traffic. This allows a node to start serving its busiest tenants within a few minutes of startup, while additional tenants are loaded incrementally only when traffic actually arrives—allowing the system to progressively absorb cold tenants as demand increases. The effect on recovery is transformative. Instead of recovery time scaling with the total number of tenants configured on a server, it scales with the number of tenants actively receiving traffic. In practice, even at our busiest edge sites, the active tenant set is a small fraction of the total. Just as importantly, this modified form of lazy loading provides a natural failure isolation boundary. Most Edge sites won’t ever load a faulty configuration of an inactive tenant. When a request for an inactive tenant with an incompatible configuration arrives, impact is contained to a single worker. The configuration load architecture now prefers serving as many customers as quickly as possible, rather than waiting until everything is ready before serving anyone. The above changes are slated to complete in April 2026 and will bring our RTO from the current ~1 hour to under 10 minutes – for complete recovery from a worst case scenario. Continuous validation through Game Days A critical element of our recovery confidence comes from GameDay fault-injection testing. We don’t simply design recovery mechanisms and assume they work—we break the system deliberately and observe how it responds. Since late 2025, we have conducted recurring GameDay drills that simulate the exact failure scenarios we are defending against: Food Taster crash scenarios: Injecting deliberately faulty tenant configurations, to verify that they are caught and isolated with zero impact on live traffic. In our January 2026 GameDay, the Food Taster process crashed as expected, the system halted the update within approximately 5 seconds, and no customer traffic was affected. Master process crash scenarios: Triggering master process crashes across test environments to verify that workers continue serving traffic, that the Local Config Shield engages within 10 seconds, and that the coordinated recovery tool restores full operation within the expected timeframe. Multi-region failure drills: Simulating simultaneous failures across multiple regions to validate that global Config Shield mechanisms engage correctly, and that recovery procedures scale without requiring manual per-region intervention. Fallback test drills for critical Azure services running behind Azure Front Door: In our February 2026 GameDay, we simulated the complete unavailability of Azure Front Door, and successfully validated failover for critical Azure services with no impact to traffic. These drills have both surfaced corner cases and built operational confidence. They have transformed recovery from a theoretical plan into tested, repeatable muscle memory. As we noted in an internal communication to our team: “Game day testing is a deliberate shift from assuming resilience to actively proving it—turning reliability into an observed and repeatable outcome.” Closing Part 1 of this series emphasized preventing unsafe configurations from reaching the data plane, and data plane resiliency in case an incompatible configuration reaches production. This post has shown that prevention alone is not enough—when failures do occur, recovery must be fast, predictable, and bounded. By ensuring that the FlatBuffer cache is never invalidated, by loading only active tenants, and by building safe coordinated recovery tooling, we have transformed failure handling from a fleet-wide crisis into a controlled operation. These recovery investments work in concert with the prevention mechanisms described in Part 1. Together, they ensure that the path from incident detection to full service restoration is measured in minutes, with customer traffic protected at every step. In the next post of this series, we will cover the third pillar of our resiliency strategy: tenant isolation—how micro-cellular architecture and ingress-layered sharding can reduce the blast radius of any failure to a small subset, ensuring that one customer’s configuration or traffic anomaly never becomes everyone’s problem. We deeply value our customers’ trust in Azure Front Door. We are committed to transparently sharing our progress on these resiliency investments, and to exceed expectations for safety, reliability, and operational readiness.Azure Front Door: Implementing lessons learned following October outages
Abhishek Tiwari, Vice President of Engineering, Azure Networking Amit Srivastava, Principal PM Manager, Azure Networking Varun Chawla, Partner Director of Engineering Link to Part 2 - Azure Front Door Resiliency Series Link to Part 3 - Azure Front Door Resiliency Series Introduction Azure Front Door is Microsoft's advanced edge delivery platform encompassing Content Delivery Network (CDN), global security and traffic distribution into a single unified offering. By using Microsoft's extensive global edge network, Azure Front Door ensures efficient content delivery and advanced security through 210+ global and local points of presence (PoPs) strategically positioned closely to both end users and applications. As the central global entry point from the internet onto customer applications, we power mission critical customer applications as well as many of Microsoft’s internal services. We have a highly distributed resilient architecture, which protects against failures at the server, rack, site and even at the regional level. This resiliency is achieved by the use of our intelligent traffic management layer which monitors failures and load balances traffic at server, rack or edge sites level within the primary ring, supplemented by a secondary-fallback ring which accepts traffic in case of primary traffic overflow or broad regional failures. We also deploy a traffic shield as a terminal safety net to ensure that in the event of a managed or unmanaged edge site going offline, end user traffic continues to flow to the next available edge site. Like any large-scale CDN, we deploy each customer configuration across a globally distributed edge fleet, densely shared with thousands of other tenants. While this architecture enables global scale, it carries the risk that certain incompatible configurations, if not contained, can propagate broadly and quickly which can result in a large blast radius of impact. Here we describe how the two recent service incidents impacting Azure Front Door have reinforced the need to accelerate ongoing investments in hardening our resiliency, and tenant isolation strategy to mitigate likelihood and the scale of impact from this class of risk. October incidents: recap and key learnings Azure Front Door experienced two service incidents; on October 9 th and October 29 th , both with customer-impacting service degradation. On October 9 th : A manual cleanup of stuck tenant metadata bypassed our configuration protection layer, allowing incompatible metadata to propagate beyond our canary edge sites. This metadata was created on October 7 th , from a control-plane defect triggered by a customer configuration change. While the protection system initially blocked the propagation, the manual override operation bypassed our safeguards. This incompatible configuration reached the next stage and activated a latent data-plane defect in a subset of edge sites, causing availability impact primarily across Europe (~6%) and Africa (~16%). You can learn more about this issue in detail at https://aka.ms/AIR/QNBQ-5W8 On October 29 th : A different sequence of configuration changes across two control-plane versions produced incompatible metadata. Because the failure mode in the data-plane was asynchronous, the health checks validations embedded in our protection systems were all passed during the rollout. The incompatible customer configuration metadata successfully propagated globally through a staged rollout and also updated the “last known good” (LKG) snapshot. Following this global rollout, the asynchronous process in data-plane exposed another defect which caused crashes. This impacted connectivity and DNS resolutions for all applications onboarded to our platform. Extended recovery time amplified impact on customer applications and Microsoft services. You can learn more about this issue in detail at https://aka.ms/AIR/YKYN-BWZ We took away a number of clear and actionable lessons from these incidents, which are applicable not just to our service, but to any multi-tenant, high-density, globally distributed system. Configuration resiliency – Valid configuration updates should propagate safely, consistently, and predictably across our global edge, while ensuring that incompatible or erroneous configuration never propagate beyond canary environments. Data plane resiliency - Additionally, configuration processing in the data plane must not cause availability impact to any customer. Tenant isolation – Traditional isolation techniques such as hardware partitioning and virtualization are impractical at edge sites. This requires innovative sharding techniques to ensure single tenant-level isolation – a must-have to reduce potential blast radius. Accelerated and automated recovery time objective (RTO) – System should be able to automatically revert to last known good configuration in an acceptable RTO. In case of a service like Azure Front Door, we deem ~10 mins to be a practical RTO for our hundreds of thousands of customers at every edge site. Post outage, given the severity of impact which allowed an incompatible configuration to propagate globally, we made the difficult decision to temporarily block configuration changes in order to expedite rollout of additional safeguards. Between October 29 th to November 5 th , we prioritized and deployed immediate hardening steps before opening up the configuration change. We are confident that the system is stable, and we are continuing to invest in additional safeguards to further strengthen the platform's resiliency. Learning category Goal Repairs Status Safe customer configuration deployment Incompatible configuration never propagates beyond Canary · Control plane and data plane defect fixes · Forced synchronous configuration processing · Additional stages with extended bake time · Early detection of crash state Completed Data plane resiliency Configuration processing cannot impact data plane availability Manage data-plane lifecycle to prevent outages caused by configuration-processing defects. Completed Isolated work-process in every data plane server to process and load the configuration. January 2026 100% Azure Front Door resiliency posture for Microsoft internal services Microsoft operates an isolated, independent Active/Active fleet with automatic failover for critical Azure services Phase 1: Onboarded critical services batch impacted on Oct 29 th outage running on a day old configuration Completed Phase 2: Automation & hardening of operations, auto-failover and self-management of Azure Front Door onboarding for additional services March 2026 Recovery improvements Data plane crash recovery in under 10 minutes Data plane boot-up time optimized via local cache (~1 hour) Completed Accelerate recovery time < 10 minutes March 2026 Tenant isolation No configuration or traffic regression can impact other tenants Micro cellular Azure Front Door with ingress layered shards June 2026 This blog is the first in a multi-part series on Azure Front Door resiliency. In this blog, we will focus on configuration resiliency—how we are making the configuration pipeline safer and more robust. Subsequent blogs will cover tenant isolation and recovery improvements. How our configuration propagation works Azure Front Door configuration changes can be broadly classified into three distinct categories. Service code & data – these include all aspects of Azure Front Door service like management plane, control plane, data plane, configuration propagation system. Azure Front Door follows a safe deployment practice (SDP) process to roll out newer versions of management, control or data plane over a period of approximately 2-3 weeks. This ensures that any regression in software does not have a global impact. However, latent bugs that escape pre-validation and SDP rollout can remain undetected until a specific combination of customer traffic patterns or configuration changes trigger the issue. Web Application Firewall (WAF) & L7 DDoS platform data – These datasets are used by Azure Front Door to deliver security and load-balancing capabilities. Examples include GeoIP data, malicious attack signatures, and IP reputation signatures. Updates to these datasets occur daily through multiple SDP stages with an extended bake time of over 12 hours to minimize the risk of global impact during rollout. This dataset is shared across all customers and the platform, and it is validated immediately since it does not depend on variations in customer traffic or configuration steps. Customer configuration data – Examples of these are any customer configuration change—whether a routing rule update, backend pool modification, WAF rule change, or security policy change. Due to the nature of these changes, it is expected across the edge delivery / CDN industry to propagate these changes globally in 5-10 mins. Both outages stemmed from issues within this category. All configuration changes, including customer configuration data, are processed through a multi-stage pipeline designed to ensure correctness before global rollout across Azure Front Door’s 200+ edge locations. At a high level, Azure Front Door’s configuration propagation system has two distinct components - Control plane – Accepts customer API/portal changes (create/update/delete for profiles, routes, WAF policies, origins, etc.) and translates them into internal configuration metadata which the data plane can understand. Data plane – Globally distributed edge servers that terminate client traffic, apply routing/WAF logic, and proxy to origins using the configuration produced by the control plane. Between these two halves sits a multi-stage configuration rollout pipeline with a dedicated protection system (known as ConfigShield): Changes flow through multiple stages (pre-canary, canary, expanding waves to production) rather than going global at once. Each stage is health-gated: the data plane must remain within strict error and latency thresholds before proceeding. Each stage’s health check also rechecks previous stage’s health for any regressions. A successfully completed rollout updates a last known good (LKG) snapshot used for automated rollback. Historically, rollout targeted global completion in roughly 5–10 minutes, in line with industry standards. Customer configuration processing in Azure Front Door data plane stack Customer configuration changes in Azure Front Door traverse multiple layers—from the control plane through the deployment system—before being converted into FlatBuffers at each Azure Front Door node. These FlatBuffers are then loaded by the Azure Front Door data plane stack, which runs as Kubernetes pods on every node. FlatBuffer Composition: Each FlatBuffer references several sub-resources such as WAF and Rules Engine schematic files, SSL certificate objects, and URL signing secrets. Data plane architecture: o Master process: Accepts configuration changes (memory-mapped files with references) and manages the lifecycle of worker processes. o Workers: L7 proxy processes that serve customer traffic using the applied configuration. Processing flow for each configuration update: Load and apply in master: The transformed configuration is loaded and applied in the master process. Cleanup of unused references occurs synchronously except for certain categories à October 9 outage occurred during this step due to a crash triggered by incompatible metadata. Apply to workers: Configuration is applied to all worker processes without memory overhead (FlatBuffers are memory-mapped). Serve traffic: Workers start consuming new FlatBuffers for new requests; in-flight requests continue using old buffers. Old buffers are queued for cleanup post-completion. Feedback to deployment service: Positive feedback signals readiness for rollout.Cleanup: FlatBuffers are freed asynchronously by the master process after all workers load updates à October 29 outage occurred during this step due to a latent bug in reference counting logic. The October incidents showed we needed to strengthen key aspects of configuration validation, propagation safeguards, and runtime behavior. During the Azure Front Door incident on October 9 th , that protection system worked as intended but was later bypassed by our engineering team during a manual cleanup operation. During this Azure Front Door incident on October 29 th , the incompatible customer configuration metadata progressed through the protection system, before the delayed asynchronous processing task resulted in the crash. Configuration propagation safeguards Based on learnings from the incidents, we are implementing a comprehensive set of configuration resiliency improvements. These changes aim to guarantee that any sequence of configuration changes cannot trigger instability in the data plane, and to ensure quicker recovery in the event of anomalies. Strengthening configuration generation safety This improvement pivots on a ‘shift-left’ strategy where we want to ensure that we catch regression early before they propagate to production. It also includes fixing the latent defects which were the proximate cause of the outage. Fixing outage specific defects - We have fixed the control-plane defects that could generate incompatible tenant metadata under specific operation sequences. We have also remediated the associated data-plane defects. Stronger cross-version validation - We are expanding our test and validation suite to account for changes across multiple control plane build versions. This is expected to be fully completed by February 2026. Fuzz testing - Automated fuzzing and testing of metadata generation contract between the control plane and the data plane. This allows us to generate an expanded set of invalid/unexpected configuration combinations which might not be achievable by traditional test cases alone. This is expected to be fully completed by February 2026. Preventing incompatible configurations from being propagated This segment of the resiliency strategy strives to ensure that a potentially dangerous configuration change never propagates beyond canary stage. Protection system is “always-on” - Enhancements to operational procedures and tooling prevent bypass in all scenarios (including internal cleanup/maintenance), and any cleanup must flow through the same guarded stages and health checks as standard configuration changes. This is completed. Making rollout behavior more predictable and conservative - Configuration processing in the data plane is now fully synchronous. Every data plane issue due to incompatible meta data can be detected withing 10 seconds at every stage. This is completed. Enhancement to deployment pipeline - Additional stages during roll-out and extended bake time between stages serve as an additional safeguard during configuration propagation. This is completed. Recovery tool improvements now make it easier to revert to any previous version of LKG with a single click. This is completed. These changes significantly improve system safety. Post-outage we have increased the configuration propagation time to approximately 45 minutes. We are working towards reducing configuration propagation time closer to pre-incident levels once additional safeguards covered in the Data plane resiliency section below are completed by mid-January, 2026. Data plane resiliency The data plane recovery was the toughest part of recovery efforts during the October incidents. We must ensure fast recovery as well as resilience to configuration processing related issues for the data plane. To address this, we implemented changes that decouple the data plane from incompatible configuration changes. With these enhancements, the data plane continues operating on the last known good configuration—even if the configuration pipeline safeguards fail to protect as intended. Decoupling data plane from configuration changes Each server’s data plane consists of a master process which accepts configuration changes and manages lifecycle of multiple worker processes which serve customer traffic. One of the critical reasons for the prolonged outage in October was that due to latent defects in the data plane, when presented with a bad configuration the master process crashed. The master is a critical command-and-control process and when it crashes it takes down the entire data plane, in that node. Recovery of the master process involves reloading hundreds of thousands of configurations from scratch and took approximately 4.5 hours. We have since made changes to the system to ensure that even in the event of the master process crash due to any reason - including incompatible configuration data being presented - the workers remain healthy and able to serve traffic. During such an event, the workers would not be able to accept new configuration changes but will continue to serve customer traffic using the last known good configuration. This work is completed. Introducing Food Taster: strengthening config propagation resiliency In our efforts to further strengthen Azure Front Door’s configuration propagation system, we are introducing an additional configuration safeguard known internally as Food Taster which protects the master and worker processes from any configuration change related incidents, thereby ensuring data plane resiliency. The principle is simple: every data-plane server will have a redundant and isolated process – the Food Taster – whose only job is to ingest and process new configuration metadata first and then pass validated configuration changes to active data plane. This redundant worker does not accept any customer traffic. All configuration processing in this Food Taster is fully synchronous. That means we do all parsing, validation, and any expensive or risky work up front, and we do not move on until the Food Taster has either proven the configuration is safe or rejected it. Only when the Food Taster successfully loads the configuration and returns “Config OK” does the master process proceed to load the same config and then instruct the worker processes to do the same. If anything goes wrong in the Food Taster, the failure is contained to that isolated worker; the master and traffic-serving workers never see that invalid configuration. We expect this safeguard to reach production globally in January 2026 timeframe. Introduction of this component will also allow us to return closer to pre-incident level of configuration propagation while ensuring data plane safety. Closing This is the first in a series of planned blogs on Azure Front Door resiliency enhancements. We are continuously improving platform safety and reliability and will transparently share updates through this series. Upcoming posts will cover advancements in tenant isolation and improvements to recovery time objectives (RTO). We deeply value our customers’ trust in Azure Front Door. The October incidents reinforced how critical configuration resiliency is, and we are committed to exceeding industry expectations for safety, reliability, and transparency. By hardening our configuration pipeline, strengthening safety gates, and reinforcing isolation boundaries, we’re making Azure Front Door even more resilient so your applications can be too.
Secure Native Access to Azure Kubernetes Service (AKS) Private Clusters with Azure Bastion
Written in Collaboration with AvirupChat ShabazShaik Mohit_Kumar YuriDiogenes Introduction: As organizations move toward containerized workloads such as Azure Kubernetes Service (AKS), secure access to cluster resources becomes critical. Making the Kubernetes API server private, therefore, significantly reduces the attack surface. However, it also changes how engineers perform routine management tasks. Operations as simple as running kubectl logs, describing resources, or troubleshooting workloads require connectivity to the virtual network hosting the cluster. This challenge becomes even more apparent during day-to-day operations. An engineer may have the necessary Kubernetes credentials and Azure RBAC permissions yet still be unable to access the cluster because the API server is reachable only from within the private network. Establishing VPN connectivity, using jump hosts, or deploying dedicated management workstations often becomes part of the operational workflow, adding complexity to what should be straightforward administrative tasks. This balance between maintaining strong network isolation and enabling efficient cluster management is exactly what Azure Bastion's native client tunneling support for private AKS clusters is designed to address. Many Azure customers already use Azure Bastion to securely access virtual machines without exposing them to the public internet. Native client tunneling now brings that same secure access model to AKS private clusters. This capability is in public preview currently. Before discussing how it works, it is worth understanding the operational challenges it is designed to solve and how it's done. Bastion for Azure Kubernetes Service (AKS) private clusters: At its core, Azure Bastion is designed to simplify secure access to private Azure resources. Since its launch, it has provided browser-based and native client RDP and SSH connectivity to Azure VMs without exposing management ports to the internet. No public IPs on your VMs. No inbound NSG rules for port 22 or 3389. Just TLS over port 443, routed through a fully managed service that Microsoft patches, scales, and secures on your behalf. When you establish a Bastion tunnel to your private AKS cluster, you are not signing in to an intermediary machine and running kubectl from there. You are opening an encrypted tunnel from your local machine, through Bastion, directly to the private API server endpoint. Your kubeconfig points to localhost on a dynamically assigned port, so local kubectl, helm, and scripts continue to work as they would against a public cluster. You get the developer experience of a public cluster with the security posture of a private one as shown in Figure 1. The high-level flow has three parts: The engineer authenticates to Azure, retrieves cluster credentials, and uses Azure Bastion to establish a managed tunnel into the private network. Local Kubernetes tools then use that tunnel to reach the private API server. This keeps the workflow familiar while removing the need for jump hosts or VPN-based access paths. However, this raises an important question - Does making access easier also make it easier for an attacker to connect? The short answer is no. In fact, Bastion tunneling strengthens the security posture in ways that are worth unpacking. First, the API server itself remains private. There is no public endpoint. There is no public IP address discoverable by scanners. The only network path to the API server runs through Azure's managed infrastructure. Authentication and authorization then depend on the AKS cluster configuration. Second, Bastion eliminates an entire class of infrastructure that itself becomes a security target. Jump box VMs, when not meticulously maintained, accumulate credentials, kubeconfig files, and browser sessions. They are machines that admins log into, which means they are machines that can be compromised. Bastion is not a machine you log into. It is an agentless, managed tunnel you pass through. And for public clusters, there is a related but distinct benefit. Many teams use API server authorized IP ranges to restrict which source IPs can reach their public API endpoint. This is a good practice, but it breaks down quickly when your team works remotely, uses dynamic IPs, or includes contractors. Adding Bastion's stable public IP to the authorized range gives you a consistent, managed access path without the operational toil of constantly updating IP allow lists. With the network path established through Bastion, the next question is how users are authenticated and authorized once they reach the AKS API server. That distinction matters: Bastion provides secure connectivity, while AKS access is governed by the authentication and authorization model configured on the cluster. Authentication and authorization options: When creating an AKS cluster, you can choose from three authentication and authorization modes, as shown in Figure below. That choice shapes everything about how access is granted, audited, and governed over the lifetime of the cluster. ⚠️Local accounts with Kubernetes RBAC is the default if you change nothing. It uses a static certificate that never expires, is shared across all administrators, and has no connection to your corporate identity provider. There is no MFA. No Conditional Access. Note: For most production environments, Microsoft recommends using Microsoft Entra ID integrated authentication rather than local accounts to enable centralized identity, MFA, and auditing. ☑️ Microsoft Entra ID with Kubernetes RBAC moves authentication to Entra ID, which means users sign in with their corporate identity, MFA is enforced, and Conditional Access policies apply. Authorization is handled through native Kubernetes Role and ClusterRole bindings, which reference Entra ID users and groups as subjects. This works well for teams that manage cluster configuration through GitOps, because RBAC manifests live alongside other cluster YAML. The limitation is that these permissions are invisible in Azure IAM — they live only inside the cluster. ✅ Microsoft Entra ID with Azure RBAC is the model we recommend for most production environments. Authentication still flows through Entra ID with full MFA and Conditional Access support. But authorization is handled by Azure RBAC role assignments on the AKS resource itself. Permissions are visible in Azure IAM, participate in access reviews, and integrate with Privileged Identity Management for just-in-time elevation. You can assign the built-in Azure Kubernetes Service RBAC Cluster Admin, Admin, Writer, or Reader roles at either cluster scope or namespace scope. A single subscription-level role assignment can grant access to every cluster in the subscription. The real value is the combination. Bastion provides the encrypted network tunnel. Entra ID provides the identity, MFA, and Conditional Access. Azure RBAC provides centralized, auditable authorization that your security team can review alongside every other Azure resource. For the exact steps to configure Entra ID authentication and Azure RBAC on your AKS cluster, see the AKS identity documentation. End-to-end flow for connectivity to AKS via Bastion: az account set --subscription <subscription ID> Retrieve credentials to your AKS private cluster using the commands below: az aks get-credentials --name <AKSClusterName> --resource-group <ResourceGroupName> Open the tunnel to your target AKS Cluster with the following command: az aks bastion --name <aksClusterName> --resource-group <aksClusterResourceGroup> --bastion <bastionResourceId> Now the default authentication method is Device code authentication i.e., this authentication method prompts the device code for the user to sign in from a browser session. If you want CLI only authentication, you can run the following command next. kubelogin convert-kubeconfig -l azurecli Then go on with your AKS connectivity: kubectl get nodes How to Try It: If you are already running private AKS clusters, getting started is straightforward. You need a Standard or Premium Azure Bastion host deployed in the same VNet as your cluster (or in a peered VNet), with native client support enabled. The aks-preview and bastion CLI extensions handle the rest. The detailed connection steps are documented on Microsoft Learn. We Want Your Feedback Write a comment in this blog or open an issue on the AKS GitHub repository or leave feedback directly on the Microsoft Learn documentation page. We read everything, and it genuinely influences what we prioritize.Azure Front Door: Resiliency Series – Part 3: Tenant isolation
Abhishek Tiwari, Vice President of Engineering, Azure Networking Amit Srivastava, Partner Director of PM, Azure Networking Varun Chawla, Partner Director of Engineering, Azure Networking Azure Front Door serves hundreds of thousands of tenants from hundreds of edge locations, densely sharing the globally distributed edge fleet. That density is exactly what allows us to deliver global scale, performance, and cost efficiency. It also means that, without strong isolation, a single tenant’s incompatible configuration or anomalous traffic can, in the worst case, affect many other tenants. The October 2025 incidents reinforced how important it is to contain this class of risk. Our goal for this tenant isolation is simple to state and hard to achieve: no single tenant’s configuration or traffic should be able to impact any other tenant. In Part 1 of our three part mini blog series, we outlined our four‑pillar strategy for improving the resiliency of Azure Front Door: configuration resiliency, data plane resiliency, tenant isolation, and accelerated Recovery Time Objective (RTO). Part 1 detailed how we would make configuration propagation safer and how the data plane keeps serving from a ‘last‑known‑good’ (LKG) configuration, even if an incompatible configuration change is propagated to the data plane. Part 2 turned to recovery, showing how we bring the system back to full operation in an accelerated, predictable, and in a bounded timeframe. In this final part, we turn to the tenant isolation pillar that ensures that any single tenant configuration or traffic issues are limited in scope to that tenant alone and do not impact other tenants. We will also show how Azure Front Door achieves single-tenant containment through configuration isolation, lazy loading, and a micro-cellular layered ingress-sharding architecture. Repair status: all outstanding items now complete Before we dive into tenant isolation, here is a final update on the overall repair items from the two October 2025 incidents (you can review the details in our Azure Incident Retrospective sessions for the October 9th and October 29th incidents). We are pleased to report that all outstanding work across every pillar is now complete and fully deployed in production – including the tenant isolation work described in this post. With these safeguards in place, we have also returned configuration propagation latency to pre‑incident levels while keeping platform stability our top priority. In the table below, “Completed” means broadly deployed in production. Learning category Goal Repairs Status Safe customer configuration deployment Incompatible configuration never propagates beyond ‘EUAP or canary regions’ Control plane and data plane defect fixes; forced synchronous configuration processing; additional stages with extended bake time; early detection of crash state Completed Data plane resiliency Configuration processing cannot impact data plane availability Manage data‑plane lifecycle to prevent outages caused by configuration‑processing defects; isolated work‑process in every data plane server to process and load the configuration Completed 100% Azure Front Door resiliency posture for Microsoft internal services Microsoft operates an isolated, independent Active/Active fleet with automatic failover for critical Azure services Phase 1: onboarded critical services batch impacted on Oct 29th outage running on a day‑old configuration; Phase 2: automation & hardening of operations, auto‑failover and self‑management of onboarding for additional services Completed Recovery improvements Data plane crash recovery in under 10 minutes Data plane boot‑up time optimized via local cache; recovery time accelerated to under 10 minutes Completed Tenant isolation No configuration or traffic regression can impact other tenants Micro‑cellular Azure Front Door with ingress layered shards Completed Why isolation at edge scale is deceptively hard Traditional isolation techniques such as dedicating separate hardware to each tenant or running every tenant inside its own virtual machine are impractical at the edge. Edge sites are constrained on space, power, and capacity. The entire premise of a modern, multi-tenant application delivery platform is that any tenant can be served from any site closest to the user. We cannot simply partition hundreds of thousands of tenants onto dedicated machines without giving up either proximity, scale, or the efficiency that make the edge fast and cost efficient. Isolation therefore, must be achieved in software, inside a multi-tenant fleet. What we already do today Azure Front Door already includes several layers of tenant isolation and partitioning. However, the incidents in October clearly highlighted that these techniques were not enough. Prior to the incidents, our protection mechanisms included: Infrastructure partitioning. Edge sites were organized into physically isolated primary and fallback traffic rings. Noisy‑neighbor protection. Fair‑share resource allocation, rate limiting, and anomaly-based load protection kept any single tenant from monopolizing shared resources such as CPU, memory, or network bandwidth on an Azure Front Door server. Circuit breakers. Circuit breakers shed costly work first and can disable a risky per‑tenant feature before it exhausts shared resources on a server. Real‑time crash protection. A crash‑analysis system correlates crash signatures across machines and can pinpoint and block crash patterns caused by tenant IPs or traffic patterns. While these protections are valuable, many of them are reactive and proved insufficient during the October incidents. The next generation of isolation makes single‑tenant containment a fundamental part of the platform which governs how configurations are loaded, and how traffic is served. Configuration isolation: loading only what is needed Part 2 introduced ‘lazy loading’ as a recovery optimization technique. It is also an important configuration‑isolation mechanism. Historically, every worker on every edge server had to be ready to serve any tenant, which meant each worker loaded a large set of tenant configurations. A single incompatible configuration could therefore ripple across many workers. With lazy loading, a worker loads a tenant’s configuration and its TLS certificates only when it actually receives traffic for that tenant. The practical consequence for isolation is powerful: a faulty configuration can only affect the workers that have loaded that specific tenant, never the entire server or fleet. Combined with per‑tenant validation on load, and the Food Taster safeguard from Part 1 (a sacrificial process that pretests every configuration change in isolation), configuration problems are caught early and contained to the smallest possible footprint. Figure 1: With lazy loading, an incompatible configuration is contained to the workers serving that tenant, instead of poisoning the whole server. Tenant isolation: a micro-cellular, layered ingress sharding architecture Configuration isolation limits the blast radius of an incompatible configuration. Traffic isolation addresses the other half of the problem: a tenant whose anomalous traffic incident, like a sudden surge, a pathological request pattern, or malicious activity, could degrade a shared worker. Our approach is a micro‑cellular architecture that combines multiple concepts working together. Worker‑process isolation. Each edge server already runs many independent worker processes. Instead of letting every worker serve every tenant, we assign tenants to specific groups of workers. Those worker group (shards) become the unit of isolation: if a tenant destabilizes its shard, the impact is contained to that shard’s workers while the rest of the server keeps serving normally. Ingress sharding. Rather than a handful of fixed shards, we compose shards from overlapping subsets of a server’s workers. Even a modest number of workers can be combined into an enormous number of distinct, overlapping shards – giving us a very large number of fine‑grained fault domains without dedicating hardware to every tenant. Figure 2: Tenants are randomly assigned to different shards on each server (layer). Even when a good tenant shares the noisy tenant’s shard on one layer, routing steers its traffic to healthy shards on the others. Multi‑layer ingress sharding. This is where ‘layered’ comes in. Each edge server is treated as an independent layer, and each tenant is assigned to a different, randomly chosen shard on every server. Because assignments are independent from one server to the next, two tenants that happen to share a shard on one server are extremely unlikely to share a shard again on another server. The chance of any good tenant repeatedly colliding with a noisy tenant across many servers becomes vanishingly small. Intelligent ingress routing. Tying it together is a routing layer that terminates each incoming connection, identifies the tenant, and steers the request to that tenant’s assigned, healthy shard. If a shard is unhealthy or saturated, traffic is directed to the tenant’s healthy shards on other layers. Figure 3: Intelligent Ingress Routing The combined effect is that when a noisy tenant overwhelms or crashes its shard on one server, only that shard is affected. Because every other tenant is spread across a different, randomized set of shards, they continue to find healthy paths, and the routing layer moves their traffic accordingly. A worst-case availability problem is downgraded to, at most, a small and redistributable capacity problem, that the routing layer smooths over. An in-depth technical analysis of layered ingress sharding is available here for reference. Shrinking the blast radius Taken together, these mechanisms fundamentally change the shape of failure. In a uniform, fully shared fleet, an incompatible tenant can, in the worst case, affect a large share of the tenants on a machine, in an edge location, or beyond. With configuration isolation and layered ingress sharding, the same failure is confined to a subset of workers serving the offending tenant. Our target for this tenant isolation pillar is effectively single‑tenant containment: a configuration or traffic anomaly caused by one tenant should never cause issues to any other tenants. Figure 4: From a shared fleet where a single tenant can affect many, to micro‑cellular shards that confine impact to the offending tenant. Validating isolation in practice As with our recovery work, we don’t simply design these boundaries and assume they hold, we test them. Through deliberate fault‑injections, we have pushed noisy and faulty tenants into the system and confirmed that impact stayed contained to the offending shard, that healthy tenants kept serving, and that the routing layer steered around unhealthy shards as intended. This turns isolation from a design claim into a well-drilled, and repeatable outcome. Closing This post concludes our three-part mini blog series on Azure Front Door resiliency. We have shared how we are making configuration propagation safer (Part 1), recovering faster when failures do occur (Part 2), and containing the blast radius stemming from any single tenant through configuration isolation and a micro‑cellular, layered‑sharding architecture (Part 3). Resiliency, however, is not a project with an end date. It is an ongoing commitment. While this series concludes the blog series of our response to the October 2025 incidents, our investments in Azure Front Door’s resiliency, isolation, and recovery will continue. As we make further improvements, we will keep sharing them with you. We deeply value our customers’ trust in Azure Front Door. We remain committed to exceeding expectations for security, reliability, and transparency.
Advanced Container Apps Networking: VNet Integration and Centralized Firewall Traffic Logging
Azure community, I recently documented a networking scenario relevant to Azure Container Apps environments where you need to control and inspect application traffic using a third-party network virtual appliance. The article walks through a practical deployment pattern: • Integrate your Azure Container Apps environment with a Virtual Network. • Configure user-defined routes (UDRs) so that traffic from your container workloads is directed toward a firewall appliance before reaching external networks or backend services. • Verify actual traffic paths using firewall logs to confirm that routing policies are effective. This pattern is helpful for organizations that must enforce advanced filtering, logging, or compliance checks on container egress/ingress traffic, going beyond what native Azure networking controls provide. It also complements Azure Firewall and NSG controls by introducing a dedicated next-generation firewall within your VNet. If you’re working with network control, security perimeters, or hybrid network architectures involving containerized workloads on Azure, you might find it useful. Read the full article on my blogInter-Hub Connectivity Using Azure Route Server
By Mays_Algebary shruthi_nair As your Azure footprint grows with a hub-and-spoke topology, managing User-Defined Routes (UDRs) for inter-hub connectivity can quickly become complex and error-prone. In this article, we’ll explore how Azure Route Server (ARS) can help streamline inter-hub routing by dynamically learning and advertising routes between hubs, reducing manual overhead and improving scalability. Baseline Architecture The baseline architecture includes two Hub VNets, each peered with their respective local spoke VNets as well as with the other Hub VNet for inter-hub connectivity. Both hubs are connected to local and remote ExpressRoute circuits in a bowtie configuration to ensure high availability and redundancy, with Weight used to prefer the local ExpressRoute circuit over the remote one. To maintain predictable routing behavior, the VNet-to-VNet configuration on the ExpressRoute Gateway should be disabled. Note: Adding ARS to an existing Hub with Virtual Network Gateway will cause downtime that expect to last 10 minutes. Scenario 1: ARS and NVA Coexist in the Hub Option A: Full Traffic Inspection ARS and NVA Coexist in the Hub In this scenario, ARS is deployed in each Hub VNet, alongside the Network Virtual Appliances (NVAs). NVA1 in Region1 establishes BGP peering with both the local ARS (ARS1) and the remote ARS (ARS2). Similarly, NVA2 in Region2 peers with both ARS2 (local) and ARS1 (remote). Let’s break down what each BGP peering relationship accomplishes. For clarity, we’ll focus on Region1, though the same logic applies to Region2: NVA1 Peering with Local ARS1 Through BGP peering with ARS1, NVA1 dynamically learns the prefixes of Spoke1 and Spoke2 at the OS level, eliminating the need to manually configure these routes. The same applies for NVA2 learning Spoke3 and Spoke4 prefixes via its BGP peering with ARS2. NVA1 Peering with Remote ARS2 When NVA1 peers with ARS2, the Spoke1 and Spoke2 prefixes are propagated to ARS2. ARS2 then injects these prefixes into NVA2 at both the NIC level with NVA1 as the next hop, and at the OS level. This mechanism removes the need for UDRs on the NVA subnets to enable inter-hub routing. Additionally, ARS2 advertises the Spoke1 and Spoke2 prefixes to both ExpressRoute circuits (EXR2 and EXR1 due to bowtie configuration) via GW2, making them reachable from on-premises through either EXR1 or EXR2. 👉Important: To ensure that ARS2 accepts and propagates Spoke1/Spoke2 prefixes received via NVA1, AS-Override must be enabled. Without AS-Override, BGP loop prevention will block these routes at ARS2, since both ARS1 and ARS2 use the default ASN 65515, and ARS2 will consider the route as already originated locally. The same principle applies in reverse for Spoke3 and Spoke4 prefixes being advertised from NVA2 to ARS1. Traffic Flow Inter-Hub Traffic: Spoke VNets are configured with UDRs that contain only a default route (0.0.0.0/0) pointing to the local NVA as the next hop. Additionally, the “Propagate Gateway Routes” setting should be set to False to ensure all traffic, whether East-West (intra-hub/inter-hub) or North-South (to/from internet), is forced through the local NVA for inspection. Local NVAs will have the next hop to the other region spokes injected at the NIC level by local ARS, pointing to the other region NVA, for example NVA2 will have next hop to Spoke1 and Spoke2 as NVA1 (10.0.1.4) and vice versa. Why are UDRs still needed on spokes if ARS handles dynamic routing? Even with ARS in place, UDRs are required to maintain control of the next hop for traffic inspection. For instance, if Spoke1 and Spoke2 do not have UDRs, they will learn the remote spoke prefixes (e.g., Spoke3/Spoke4) injected via ARS1, which received them from NVA2. This results in Spoke1/Spoke2 attempting to route traffic directly to NVA2, a path that is invalid, since the spokes don’t have the path to NVA2. The UDR ensures traffic correctly routes through NVA1 instead. On-Premises Traffic: To explain the on-premises traffic flow, we'll break it down into two directions: Azure to on-premises, and on-premises to Azure. Azure to On-Premises Traffic Flow: As previously noted, Spokes send all traffic, including traffic to on-premises, via NVA1 due to the default route in the UDR. NVA1 then routes traffic to the local ExpressRoute circuit, using Weight to prefer the local path over the remote. Note: While NVA1 learns on-premises prefixes from both local and remote ARSs at the OS level, this doesn’t affect routing decisions. The actual NIC-level route injection determines the next hop, ensuring traffic is sent via the correct path—even if the OS selects a different “best” route internally. The screenshot below from NVA1 shows four next hops to the on-premises network 10.2.0.0/16. These include the local ARS (ARS1: 10.0.2.5 and 10.0.2.4) and the remote ARS (ARS2: 10.1.2.5 and 10.1.2.4). On-Premises to Azure Traffic Flow In a bowtie ExpressRoute configuration, Azure VNet prefixes are advertised to on-premises through both local and remote ExpressRoute circuits. Because of this dual advertisement, the on-premises network must ensure optimal path selection when routing traffic to Azure. From Azure side, to maintain traffic symmetry, add UDRs at the GatewaySubnet (GW1 and GW2) with specific routes to the local Spoke VNets, using the local NVA as the next hop. This ensures return traffic flows back through the same path it entered. 👉How Does the ExpressRoute Edge Router Select the Optimal Path? You might ask: If Spoke prefixes are advertised by both GW1 and GW2, how does the ExpressRoute edge router choose the best path? (e.g., diagram below shows EXR1 learns Region1 prefixes from GW1 and GW2) Here’s how: Edge routers (like EXR1) receive the same Spoke prefixes from both gateways. However, these routes have different AS-Path lengths: - Routes from the local gateway (GW1) have a shorter AS-Path. - Routes from the remote gateway (GW2) have a longer AS-Path because NVA1’s ASN (e.g., 65001) is prepended twice as part of the AS-Override mechanism. As a result, the edge router (EXR1) will prefer the local path from GW1, ensuring efficient and predictable routing. For example: EXR1 receives Spoke1, Spoke2, and Hub1-VNet prefixes from both GW1 and GW2. But because the path via GW1 has a shorter AS-Path, EXR1 will select that as the best route. (Refer to the diagram below for a visual of the AS-Path difference). Final Traffic Flow: Option-A Insights: This design simplifies UDR configuration for inter-hub routing, especially useful when dealing with non-contiguous prefixes or operating across multiple hubs. For simplicity, we used a single NVA in each Hub-VNet while explaining the setup and traffic flow throughout this article. However, a high available (HA) NVA deployment is recommended. To maintain traffic symmetry in an HA setup, you’ll need to enable the next-hop IP feature when peering with Azure Route Server (ARS). When on-premises traffic inspection is required, the UDR setup in the GatewaySubnet becomes more complex as the number of Spokes increases. As your Azure network scales, keep in mind that Azure Route Server supports a maximum of 16 BGP peers per instance (as of the time writing this article). This limit can impact architectures involving multiple NVAs or hubs. Option B: Bypass On-Premises Inspection If on-premises traffic inspection is not required, NVAs can advertise a supernet prefix summarizing the local Spoke VNets to the remote ARS. This approach provides granular control over which traffic is routed through the NVA and eliminates the need for BGP peering between the local NVA and local ARS. All other aspects of the architecture remain the same as described in Option A. For example, NVA2 can advertise the supernet 192.168.2.0/23 (supernet of Spoke3 and Spoke4) to ARS1. As a result, Spoke1 and Spoke2 will learn this route with NVA2 as the next hop. To ensure proper routing (as discussed earlier) and inter-hub inspection, you need apply a UDR in Spoke1 and Spoke2 that overrides this exact supernet prefix, redirecting traffic to NVA1 as the next hop. At the same time, traffic destined for on-premises will follow the system route through the local ExpressRoute gateway, bypassing NVA1 altogether. In this setup: UDRs on the Spokes should have "Propagate Gateway Routes" set to True. No UDRs are needed in the GatewaySubnet. 👉Can NVA2 Still Advertise Specific Spoke Prefixes? You might wonder: Can NVA2 still advertise specific prefixes (e.g., Spoke3 and Spoke4) learned from ARS2 to ARS1 instead of a supernet? Yes, this is technically possible, but it requires maintaining BGP peering between NVA2 and ARS2. However, this introduces UDR complexity in Spoke1 and Spoke2, as you'd need to manually override each specific prefix. This also defeats the purpose of using ARS for simplified route propagation, undermining the efficiency and scalability of the design. Bypass On-Premises Inspection Final Traffic Flow: Option B: Bypass on-premises inspection traffic flow Option-B Insights: This approach reduces the number of BGP peerings per ARS. Instead of maintaining two BGP sessions (local NVA and remote NVA) per Hub, you can limit it to just one, preserving capacity within ARS’s 8-peer limit for additional inter-hub NVA peerings. Each NVA should advertise a supernet prefix to the remote ARS. This can be challenging if your Spokes don’t use contiguous IP address spaces, as described in Option B. Scenario 2: ARS in the Hub and the NVA in Transit VNet In Scenario 1, we highlighted that when on-premises inspection is required, managing UDRs at the GatewaySubnet becomes increasingly complex as the number of Spoke VNets grows. This is due to the need for UDRs to include specific prefixes for each Spoke VNet. In this scenario, we eliminate the need to apply UDRs at the GatewaySubnet altogether. In this design, the NVA will be deployed in Transit VNet, where: Transit-VNet will be peered with local Spoke VNets and with the local Hub-VNet to enable intra-Hub and on-premises connectivity. Transit-VNet also peered with remote Transit VNets (e.g., Transit-VNet1 peered with Transit-VNet2) to handle inter-Hub connectivity through the NVAs. Additionally, Transit-VNets are peered with remote Hub-VNets, to establish BGP peering with the remote ARS. NVAs OS will need to add static routes for the local Spoke VNets prefixes, it can be specific or it can supernet prefix, which will later be advertised to ARSs over BGP Peering, then ARS will advertise it to on-premises via ExpressRoute. NVAs will BGP peer with local ARS and also with the remote ARS. To understand the reasoning behind this design, let’s take a closer look at the setup in Region1, focusing on how ARS and NVA are configured to connect to Region2. This will help illustrate both inter-hub and on-premises connectivity. The same concept applies in reverse from Region2 to Region1. Inetr-Hub: To enable NVA1 in Region1 to learn prefixes from Region2, NVA2 will configure static routes at the OS level for Spoke3 and Spoke4 (or their supernet prefix) and advertise them to ARS1 via remote BGP peering. As a result, these prefixes will be received by NVA1, both at the NIC level, with NVA2 as the next hop, and at the OS level for proper routing. Spoke1 and Spoke2 will have a UDR with a default route pointing to NVA1 as the next hop. For instance, when Spoke1 needs to communicate with Spoke3, the traffic will first route through NVA1. NVA1 will then forward the traffic to NVA2 using VNet peering between the two Hubs. A similar configuration will be applied in Region2, where NVA1 will configure static routes at the OS level for Spoke1 and Spoke2 (or their supernet prefix) and advertise them to ARS2 via remote BGP peering, as a result, these prefixes will be received by NVA2, both at the NIC level (injected by ARS2), with NVA1 as the next hop, and at the OS level for proper routing. Note: At the OS level, NVA1 learns Spoke3 and Spoke4 prefixes from both local and remote ARSs. However, the NIC-level route injection determines the actual next hop, so even if the OS selects a different best route, it won’t affect forwarding behavior. same applies to NVA2. On-Premises Traffic: To explain the on-premises traffic flow, we'll break it down into two directions: Azure to on-premises, and on-premises to Azure. Azure to On-Premises Traffic Flow: Spokes in Region1 route all traffic through NVA1 via a default route defined in their UDRs. Because of BGP peering between NVA1 and ARS1, ARS1 advertises the Spoke1 and Spoke2 (or their supernet prefix) to on-premises through ExpressRoute (EXR1). The Transit-VNet1 (hosting NVA1) is peered with Hub1-VNet, with “Use Remote Gateway” enabled. This allows NVA1 to learn on-premises prefixes from the local ExpressRoute gateway (GW1), and traffic to on-premises is routed through the local ExpressRoute circuit (EXR1) due to higher BGP Weight configuration. Note: At the OS level, NVA1 learns on-prem prefixes from both local and remote ARSs. However, the NIC-level route injection determines the actual next hop, so even if the OS selects a different best route, it won’t affect forwarding behavior. same applies to NVA2. On-Premises to Azure Traffic Flow: Through BGP peering with ARS1, NVA1 enables ARS1 to advertise Spoke1 and Spoke2 (or their supernet prefix) to both EXR1 and EXR2 circuits (due to the ExpressRoute bowtie setup). Additionally, due to BGP peering between NVA1 and ARS2, ARS2 also advertises Spoke1 and Spoke2 (or their supernet prefix) to EXR2 and EXR1 circuits. As a result, both ExpressRoute edge routers in Region1 and Region2 learn the same Spoke prefixes (or their supernet prefix) from both GW1 and GW2, with identical AS-Path lengths, as shown below. EXR1 learns Region1 Spokes's supernet prefixes from GW1 and GW2 This causes non-optimal inbound routing, where traffic from on-premises destined to Region1 Spokes may first land in Region2’s Hub2-VNet before traversing to NVA1 in Region1. However, return traffic from Spoke1 and Spoke2 will always exit through Hub1-VNet. To prevent suboptimal routing, configure NVA1 to prepend the AS path for Spoke1 and Spoke2 (or their supernet prefix) when advertising them to the remote ARS2. Likewise, ensure NVA2 prepends the AS path for Spoke3 and Spoke4 (or their supernet prefix) when advertising to ARS1. This approach helps maintain optimal routing under normal conditions and during ExpressRoute failover scenarios. Below diagram shows NVA1 is setting AS-Prepend for Spoke1 and Spoke2 supernet prefix when BGP peer with remote ARS (ARS1), same will apply for NVA2 when advertising Spoke3 and Spoke4 prefixes to ARS1. Final Traffic Flow: Full Inspection: Traffic flow when NVA in Transit-VNet Insights: This solution is ideal when full traffic inspection is required. Unlike Scenario 1 - Option A, it eliminates the need for UDRs in the GatewaySubnet. When ARS is deployed in a VNet (typically in Hub VNets), the VNet will be limited to 500 VNet peerings (as of the time writing this article). However, in this design, Spokes peer with the Transit-VNet instead of directly with the ARS VNet, allowing you to scale beyond the 500-peer limit by leveraging Azure Virtual Network Manager (AVNM) or submitting a support request. Some enterprise customers may encounter the 1,000-route advertisement limit on the ExpressRoute circuit from the ExpressRoute gateway. In traditional hub-and-Spoke designs, there's no native control over what is advertised to ExpressRoute. With this architecture, NVAs provide greater control over route advertisement to the circuit. For simplicity, we used a single NVA in each Hub-VNet while explaining the setup and traffic flow throughout this article. However, a high available (HA) NVA deployment is recommended. To maintain traffic symmetry in an HA setup, you’ll need to enable the next-hop IP feature when peering with Azure Route Server (ARS). This design does require additional VNet peerings, including: Between Transit-VNets (inter-region), Between Transit-VNets and local Spokes, and Between Transit-VNets and both local and remote Hub-VNets.
Designing Cloud Landing Zones by Traffic Flow: A Defence‑in‑Depth, DMZ‑First Architecture
As enterprises adopt Microsoft Azure for large‑scale and regulated workloads, security architecture must be driven by traffic trust boundaries and inspection intent, not connectivity alone. Regulatory frameworks consistently require a clear separation between Internet‑untrusted and private enterprise traffic, enforced through defence‑in‑depth controls aligned to the OSI model. This drives the adoption of a DMZ‑first landing zone architecture, where volumetric protection, application‑layer inspection, and perimeter controls are enforced at well‑defined trust boundaries. Security enforcement is prioritised before access, rather than being an afterthought of connectivity. A distributed hub architecture enables this model at scale, delivering consistent controls while improving resiliency, fault isolation, and operational clarity. Traffic Classification in an Enterprise Landing Zone Enterprise traffic flows are classified based on origin, destination, and trust level, which directly dictates inspection requirements: Internet Inbound (North–South): Traffic from the public Internet to Azure‑hosted applications Internet Outbound (South–North): Egress traffic from private workloads to the Internet East–West: Traffic between virtual networks within or across regions Hybrid Connectivity: Traffic between Azure, on‑premises, and multi‑cloud environments Each flow presents a distinct risk profile, making a uniform connectivity and inspection model unsuitable for enterprise and regulated environments. Hub‑and‑Spoke as the Foundation for Centralised Security Enterprises commonly adopt a Hub‑and‑Spoke topology using VNET peering or Azure Virtual WAN, extending hybrid connectivity via Site‑to‑Site VPN or ExpressRoute. The hub provides a centralised, datapath‑centric layer for shared networking and security services, while spoke VNETs host application workloads and remain private, typically without public IPs. Ingress and egress are handled through shared, centrally managed endpoints. This model consolidates critical controls—Azure Firewall, Azure WAF, and DDoS protection—at controlled entry and exit points, significantly reducing the attack surface. Integration with Microsoft Sentinel enables cross‑layer visibility, threat hunting, and detection of risks such as unauthorised access and data exfiltration. Why a Single Hub Is Not Sufficient Using a single hub to process all traffic types introduces operational and security challenges at scale by forcing Internet‑untrusted and private traffic through the same inspection tier. Key limitations include: Coupled Internet and private security policies Rapid firewall rule sprawl and management overhead A single blast radius across all traffic types Throughput and SNAT scalability constraints Increased difficulty meeting regulatory separation requirements These issues become more pronounced as environments scale across regions and workloads. Multi‑Hub as an Enterprise‑Grade Evolution A multi‑hub architecture resolves these challenges by separating inspection responsibilities across dedicated hubs: DMZ Hub VNET: Internet‑facing traffic and perimeter security enforcement Core Hub VNET: Outbound Internet access, East‑West, and hybrid traffic inspection This separation aligns security controls with traffic intent, reduces policy complexity, limits blast radius, and simplifies operations. In the next section, we explore multi‑hub architecture patterns and examine how each traffic flow is inspected and governed in practice. Scenario 1 – Third‑Party Firewall–Centric Multi‑Hub Design In this scenario, third‑party firewalls provide inspection for all traffic flows—Internet inbound, Internet outbound, East‑West, and hybrid connectivity—while Azure WAF and Azure DDoS Protection are used to defend against volumetric attacks, application‑layer threats, and malicious bot traffic. Architecture Overview The design uses two dedicated hub virtual networks, with spoke VNets peered to both hubs to maintain traffic separation based on trust and inspection requirements: DMZ Hub VNET (Internet Ingress Hub) Core Hub VNET (Egress, East‑West, and Hybrid Hub) DMZ Hub VNET – Internet Inbound (North–South) Traffic The DMZ hub establishes a dedicated perimeter for Internet‑facing applications. Internet traffic lands on a public Application Gateway with Azure WAF, protected by an Azure DDoS Protection Plan applied to the public frontend IP. Third‑party firewalls are deployed behind the Application Gateway to perform deep packet inspection. Firewalls operate in active‑active mode as standalone appliances, using three dedicated network interfaces (Untrust, Trust, and Management), each placed in separate subnets. Application Gateway provides load balancing and health probing, ensuring firewall availability and resiliency. Core Hub VNET – Egress, East‑West, and Hybrid Traffic The core hub handles all non‑Internet‑inbound traffic flows, including private inter‑VNET communication and hybrid connectivity. Third‑party firewalls are deployed behind an internal Azure Load Balancer, which maintains high availability using health probes. Firewalls follow the same active‑active, three‑NIC design for consistent policy enforcement and operational symmetry. Azure NAT Gateway is associated with the firewall Untrust subnet to scale outbound SNAT and simplify Internet egress with deterministic public IP. Site‑to‑site VPN or ExpressRoute terminate in this hub for on‑premises and multi‑cloud connectivity, ensuring private and regulated traffic bypasses Internet‑facing controls. Architecture Patterns - Traffic Flow 1.1 North–South (Internet Inbound) Traffic Flow This flow applies when applications hosted on Azure (VMs, VM Scale Sets, or services behind Internal Load Balancer / Internal Application Gateway) are published to the Internet. As shown in the diagram, Internet-bound client requests enter Azure through a dedicated DMZ Hub, where perimeter security controls are enforced before traffic reaches private application spokes. Inbound Request Flow (Blue Path) Client → Application Gateway (WAF) A client request (for example, app1.example.com) is resolved via DNS and lands on the public endpoint of Azure Application Gateway. Azure WAF performs Layer‑7 inspection, bot protection, and rule evaluation. Application Gateway → Firewall (DMZ Hub) Based on hostname and routing rules, Application Gateway forwards traffic to the firewalls while preserving the client’s real IP using the X‑Forwarded‑For (XFF) header. Firewall → Application (Spoke VNET) The firewall performs DNAT to the internal destination (Internal Application Gateway or Internal Load Balancer) and SNAT to its trust‑interface IP to ensure symmetric return traffic. Traffic is then routed over VNET peering to the private application workload. Response Flow (Green Path) Application → Firewall The application responds using its private IP, with the firewall trust interface as the destination. Firewall → Application Gateway The firewall forwards the response based on its session state. Application Gateway → Client Application Gateway returns the response to the Internet client. DDoS Protection Azure DDoS Protection continuously monitors the Application Gateway public IP and mitigates volumetric attacks before they reach the application stack. Key Design Considerations Application Gateway routing - Use a multi‑site listener with hostname‑based rules to steer traffic to the appropriate firewall backend. Firewall backend mapping - Register firewall VMs in a single backend pool and differentiate applications using distinct backend settings (same protocol, different ports). Application steering at firewall - Firewalls perform port‑based DNAT (or private FQDN‑based DNAT where supported) to forward traffic to the correct application private IP in spoke VNets. Traffic symmetry - Source NAT is applied on the firewall trust interface to maintain symmetric return paths. Protocol support - The same pattern applies to TCP/UDP workloads, using an External Azure Load Balancer instead of Application Gateway where WAF is not required. Client IP visibility Application Gateway enables WAF inspection using the real client IP, preserved downstream via X‑Forwarded‑For. With Azure Load Balancer, client IP is retained up to the firewall for IP‑based enforcement, with Azure DDoS Protection safeguarding the public frontend. 1.2 Hybrid Connectivity Traffic Flow Hybrid connectivity represents communication between Azure workloads and on‑premises environments using Site‑to‑Site VPN or ExpressRoute. Traffic Flow Request (Blue: 1–4): Traffic originates from applications in spoke VNETs and is forwarded via UDRs to the Internal Load Balancer, through active‑active firewalls, and then to on‑premises via VPN or ExpressRoute. Response (Green: 5–8): Return traffic follows the same path in reverse. Key Design Considerations Source and destination IP addresses remain unchanged end‑to‑end, despite firewalls operating behind an Internal Load Balancer. UDRs steer both outbound and inbound hybrid traffic to the Internal Load Balancer as the next hop. Spoke VNET subnets should be associated with custom route tables with Propagate Gateway Routes set to No, preventing gateway-learned routes from bypassing the intended security inspection path configured via UDR. However, VPN Gateway Subnet Route table should have route table with Propagate Gateway Routes set to Yes. Azure Virtual Network Manager can be used to manage and scale UDR deployment centrally across Hub and spoke networks. 1.3 East-West Traffic flow This traffic flow represents the connectivity between resources across VNETs through Firewall. Considerations remain same as “Hybrid Connectivity Traffic flow” mentioned above. 1.4 Egress/Outbound Connectivity Traffic flow This traffic flow represents the connectivity from the resources hosted inside VNETs to Internet through Firewalls deployed behind Internal Load Balancer in Core Hub VNET. Egress/Outbound Connectivity Traffic flow Key Considerations: A default route (0.0.0.0/0) UDR directs Internet-bound traffic to the Internal Load Balancer for firewall inspection. Firewall performs SNAT to its untrusted interface, after which Azure NAT Gateway translates traffic to a fixed public IP. Alternatively, a public IP can be assigned to the firewall untrusted interface to perform SNAT directly, removing the need for NAT Gateway and reducing cost. Spoke VNET subnets should be associated with custom route tables with Propagate Gateway Routes set to No, preventing gateway-learned routes from bypassing the intended security inspection path configured via UDR. However, VPN Gateway Subnet Route table should have route table with Propagate Gateway Routes set to Yes. However, with horizontal firewall scaling, outbound public IPs become non-deterministic. Scenario 2: Third‑Party Firewalls for Ingress, Azure Firewall for Private Flows In this scenario, third‑party firewalls secure Internet‑facing traffic, while Azure Firewall handles Egress, East–West, and Hybrid flows. This separation enables organisations to retain existing perimeter security investments while adopting a fully managed, cloud‑native control and data plane for internal traffic flows. As private traffic patterns scale, particularly for chatty East–West communication, Azure Firewall helps simplify operations and improves scalability by offloading traffic inspection to a managed service. This approach reduces operational overhead and provides consistent policy enforcement across internal and hybrid flows. Setup DMZ Hub design remains unchanged from Scenario 1. Core Hub VNET use Azure Firewall for better scalability, built‑in resilience, and simplified operations. Azure Firewall eliminates the need for load balancers and reduces complexity for chatty east‑west traffic. It requires a single subnet (unless force tunnelling is enabled), optimising IP consumption. Outbound traffic is SNATed by Azure Firewall and further translated via Azure NAT Gateway. Firewall‑based SNAT to public IP can be used to avoid NAT Gateway costs when deterministic egress IPs or higher SNAT scale are not required. Traffic Flow North–South (Internet Inbound) Traffic Flow represented with Numbers. Request flow represented with Blue: 1–3 Response flow represented with Brown: 4–6 Egress/Outbound Connectivity Traffic flow represented with alphabets. Request flow represented with Green: A–C Response flow represented with Blue: D–F Hybrid flow represented with purple Architecture Diagram: Third‑Party Firewalls for Ingress, Azure Firewall for Private Flows Further if you want to use Azure Firewall for the Internet Ingress flow, you can use it as well. Below is the architecture diagram explaining the same pattern. Azure Firewalls for Ingress and Private Flows Key Considerations: Application Gateway behaviour remains the same as Scenario 1; however, backend pools use IP‑based targets. Internal applications should be exposed via internal Application Gateway or internal load balancer to provide stable backend IPs. To optimise traffic flow and avoid unnecessary network address translation, the private IP address of the Application Gateway or internal Load Balancer should be configured in the backend pool of External Load Balancer in DMZ Hub VNET. This design does not need Azure Firewall to perform both source NAT (SNAT) and destination NAT (DNAT) operations. Further, implementing User Defined Routes (UDRs) in the Application Gateway subnet—specifically for spoke VNet CIDR ranges steers traffic routing through the Firewall. For non-web (TCP/UDP) workloads, applications can be directly exposed through the firewall public IP, with Azure DDoS Protection applied to mitigate volumetric attacks. Spoke VNET subnets should be associated with custom route tables with Propagate Gateway Routes set to No, preventing gateway-learned routes from bypassing the intended security inspection path configured via UDR. However, VPN Gateway Subnet Route table should have route table with Propagate Gateway Routes set to Yes. Scenario 3 - Overcoming Peering Complexity at Scale with Azure Virtual WAN In environments where each application is hosted in its own dedicated VNET for isolation and stronger security, a traditional hub‑and‑spoke model quickly becomes complex. With a single hub, "N" spoke VNETs require "N" peering's; introducing multiple hubs increases this to "2N", significantly amplifying operational overhead—especially across multiple regions. Azure Virtual WAN addresses this challenge by eliminating the need for extensive VNET peering. As a global, managed service, it simplifies large‑scale and multi‑region architectures while providing built‑in scalability and operational consistency. Setup DMZ Hub design and components remain unchanged from Scenario 1 keeping defense in depth principles intact for Internet Inbound or North-South traffic. Core Hub VNET is replaced with Azure Virtual WAN Secure Hub which has Azure Firewall inside the managed Secure Hub VNETs. This further eliminates the need for creating Azure Firewall Subnet, Azure NAT Gateway resources for East-West, Egress and Hybrid connectivity and provides same level of security protection as mentioned in Scenario 2. Secure Hub design using Virtual WAN removes the routing and inspection complexity which was unless required in Scenario 2 by removing UDR configurations in Spoke VNETs, Gateway Subnet and Azure Firewall Subnet. This provides simplified design and operations with built-in resiliency by leveraging the managed platform. Azure Firewall eliminates the need for load balancers and reduces complexity for chatty east‑west traffic. East-West, Egress, and Hybrid connectivity happens via VPN Gateway and Express Route Gateway present in Virtual Hub. All private and Internet Egress traffic is inspected by Azure Firewall present in secure Hub. Outbound traffic is SNATed to Public IP by Azure Firewall thus avoiding the need of Azure NAT Gateway. Firewall‑based SNAT to public IP can be used to avoid NAT Gateway costs when deterministic egress IPs or higher SNAT scale are not required. Traffic Flow North–South (Internet Inbound) Traffic Flow represented with Numbers. Request flow represented with Blue: 1–5 Response flow represented with Green: 6–10 Egress/Outbound Connectivity Traffic flow represented with Red Hybrid flow represented with Green East-West connectivity flow represented with black Architecture Diagram: Key Considerations Virtual WAN has additional cost Refer to architecture diagram above for different traffic flows. The DMZ VNET components consideration remains same as per scenario 1 You can use your own managed public IP on Azure Firewall for applying standard IP/Network DDoS protection plan. Benefits of multi-Hub approach over single Hub You can use this table as decision matrix to choose the best option for your application requirements. Dimensions Multi-Hub Design Resiliency · Independent scaling of DMZ and Core firewalls · Failure in Internet inspection does not impact East‑West traffic · Easier multi‑region active‑active designs Security & Compliance Clear trust boundaries Auditable inspection points Easier alignment with Zero Trust & regulatory frameworks Operational Simplicity Smaller, purpose‑built firewall policies Clear ownership (Perimeter vs Core team) Faster troubleshooting due to deterministic flows Cost Optimisation Avoids over‑inspection of East‑West traffic by DMZ firewalls Smaller firewall SKUs per hub Better scale‑unit consumption Performance Efficiency Reduced latency for private traffic No unnecessary hair‑pinning Optimised SNAT and connection tracking Key Takeaways Security‑first landing zones should not be designed as one‑size‑fits‑all networks. By designing hubs based on traffic trust level and inspection intent, rather than convenience, organisations gain: o Stronger security boundaries and better fault isolation o Predictable operations at scale o Lower long‑term cost and complexity o Design for traffic flows, not just connectivity o Use DMZ hubs exclusively for Internet‑untrusted traffic o Use Core hubs for East‑West, outbound, and hybrid connectivity o Map security controls to OSI layers o Avoid single‑hub designs for large, regulated, or multi‑region environments This distributed hub‑and‑DMZ landing zone provides a clean, scalable, and secure foundation for enterprise workloads in any Hyperscaler. For regulated, internet‑exposed, or multi‑region environments, a distributed hub with a dedicated DMZ is no longer optional — it is a foundational architecture decision.
Migrating from MSEE Hairpin Routing to AVNM Mesh for Large-Scale VNet-to-VNet Connectivity
Introduction A common pattern in large Azure deployments is to route VNet-to-VNet traffic through Microsoft Enterprise Edge (MSEE) routers. This happens when spoke VNets in a hub-and-spoke topology communicate with each other by hairpinning through the ExpressRoute circuit: traffic exits the Azure data center, traverses MSEE, and re-enters the data center to reach the destination VNet. This pattern works, but it was not designed as the long-term connectivity model for east-west traffic. With Azure Virtual Network Manager (AVNM) mesh connectivity and recent scale improvements — including high-scale mesh up to 5,000 VNets and High-Scale Private Endpoints (HSPE) up to 20,000 Private Endpoints across connected VNets — enterprises can migrate to a direct, in-datacenter routing model that removes MSEE dependency for VNet-to-VNet traffic. This article explains why the migration is useful, what the new scale limits are, how to enable the required features, and how to execute the migration with minimal disruption. Who This Is For This migration is most relevant if you have 50 or more spoke VNets communicating east-west through ExpressRoute hairpin routing, are approaching VNet peering limits, want to reduce ExpressRoute utilization from internal traffic, or need a simpler centrally managed connectivity model. Even if most east-west flows must continue through a hub firewall for inspection, you can still simplify connectivity management for the flows allowed to go direct. Why MSEE Hairpin Routing for VNet-to-VNet Is Not Recommended When spoke VNets communicate through MSEE, traffic follows a suboptimal path: Spoke A → Hub VNet → ExpressRoute Gateway → MSEE → ExpressRoute Gateway → Hub VNet → Spoke B Single point of failure. MSEE becomes a shared dependency for east-west traffic. An MSEE outage or capacity constraint can affect every VNet pair in the topology. Lower-latency path. Traffic no longer needs to leave the data center and return for spoke-to-spoke communication. Direct mesh keeps east-west traffic on an in-datacenter path, which is generally lower latency than hairpinning through MSEE. Bandwidth constraints. MSEE circuits have finite bandwidth. Routing east-west traffic through them competes with north-south on-premises traffic and can saturate the circuit. Operational risk at scale. Large deployments place significant load on MSEE infrastructure, creating scalability, reliability, and operational concerns for environments with thousands of VNets. Why Manual Peering Was Not a Practical Alternative The obvious alternative — creating direct VNet peerings between every spoke pair — solves the MSEE dependency but introduces its own operational complexity: Combinatorial growth. Connecting N spokes requires N × (N - 1) / 2 peering relationships. For 100 spokes, that is 4,950 peerings. For 1,000 spokes, it is nearly 500,000. Management overhead. Each peering must be individually provisioned, monitored, and maintained. This increases drift, audit, and operational overhead. How AVNM Mesh Solves This AVNM mesh provides group-based connectivity. You define a set of VNets as a network group, apply a mesh connectivity configuration, and AVNM establishes bi-directional connectivity across all members. Traffic between meshed VNets stays within the Azure data center: no MSEE traversal, no hub hop, and no manual peering management. Define once, connect all. A single mesh configuration connects every VNet in the group to every other VNet. Centralized management. Add or remove VNets from the group; AVNM reconciles connectivity automatically. Direct spoke-to-spoke paths. Traffic flows directly between VNets, bypassing both the hub and MSEE. Dynamic membership. Use Azure Policy to auto-enroll new VNets based on tags or resource group conditions. How to Migrate Large Scale Topology High-scale mesh lets user migrate larger topology — up to 5,000 VNets and 20,000 Private Endpoints in a mesh. Scale — Standard Mesh vs. High-Scale Mesh Dimension Standard Mesh High-Scale Mesh VNets per mesh Up to 250 (soft limit, can request increase) Up to 5,000 Private Endpoints per mesh Up to 2,000 Up to 20,000 (with HSPE enabled) Private Endpoints per VNet Up to 1,000 Up to 5,000 (with HSPE enabled) Enabling High-Scale Private Endpoints (HSPE) As a mesh footprint grows, so does the number of Private Endpoints deployed across the connected VNets. The default platform limits — 1,000 Private Endpoints per VNet and 2,000 across connected VNets and mesh — can be reached quickly in large environments. Enabling HSPE raises these limits to 5,000 and 20,000, respectively. For large-scale mesh migrations, enable HSPE proactively if the environment is expected to grow toward standard Private Endpoint limits by following the steps below. Step 1 — Prepare Each VNet Ensure Private Endpoint Network Policies are set to Enabled or RouteTableEnabled on all subnets containing Private Endpoints. This is a prerequisite. Set the VNet-level property PrivateEndpointVNetPolicies to Basic. This activates HSPE on the VNet. Step 2 — Enable HSPE on the Mesh Configuration In the AVNM mesh connectivity configuration, enable the option for high-scale private endpoints. AVNM validates that all VNets in the mesh are HSPE-enabled. If any VNet is missing the configuration, the deployment is blocked with a clear error. Step 3 — Deploy Deploy the connectivity configuration. AVNM applies HSPE across the mesh. Behavior Changes When Enabling HSPE Brief connection reset. Enabling or disabling HSPE triggers a one-time, approximately 1-second connection reset for existing Private Endpoint connections in the VNet. Plan this during a maintenance window. Per-PE Bytes In / Out monitoring is no longer available. HSPE treats each Private Endpoint IP like any other IP in the VNet, which removes per-PE traffic counters. If you depend on per-PE metrics, evaluate alternatives before enabling. On-premises PE traffic billing changes. PE traffic originating from on-premises appears as an aggregate bill on the gateway VNet, not on the individual Private Endpoint resource. The total bill does not change. How to Avoid Downtime Mesh coexists with existing peerings. AVNM does not delete manually created peerings unless explicitly configured to do so. Traffic shifts automatically. Once mesh is deployed, spoke-to-spoke traffic routes directly. The MSEE hairpin path remains available if mesh is removed. No reconfiguration of hub components. Firewalls, gateways, and NVAs in the hub continue to function. North-south on-premises traffic still flows through the hub gateway. Rollback is simple while the legacy path remains. Keep the MSEE hairpin path in place during validation so affected VNets can fall back by removing them from the network group or undeploying the mesh configuration. Security and East-West Traffic Inspection A common concern is whether direct mesh connectivity bypasses hub firewalls or network virtual appliances. The answer depends on how inspection is enforced today. Mesh provides connectivity, not routing policy. If spoke subnets have UDRs that direct traffic through a hub NVA or firewall, those UDRs continue to apply and can keep inspected flows on the firewall path. Security Admin Rules provide centralized segmentation. For flows that do not require firewall inspection, AVNM Security Admin Rules can enforce network-level allow or deny policies across network groups. Use both where appropriate. Mesh can provide direct connectivity for approved flows while Security Admin Rules enforce segmentation boundaries where required. Recommendation: Before migrating, inventory which spoke-to-spoke flows currently traverses the firewall. Decide per flow whether to maintain inspection by keeping UDRs in place or allowing a direct mesh path by removing the UDR for that flow pair. Migration Order and Process The migration from MSEE hairpin routing to AVNM mesh is non-disruptive by design. Mesh connectivity overlays on top of existing peerings and takes routing precedence for east-west traffic. You do not need to tear down the existing hub-and-spoke topology first. Recommended Steps Design the mesh topology. Group VNets by region into mesh groups. If you expect more than 250 VNets per mesh, register the AllowHighScaleConnectedGroup feature in advance. Create a Network Manager and define Network Groups. Ensure that the AVNM scope covers all relevant subscriptions. Use static membership for initial migration or dynamic membership through Azure Policy for ongoing enrollment. Enable HSPE on every VNet in the mesh. Follow the HSPE enablement steps if you need to have more than 2,000 Pes in a mesh. Schedule the change during a maintenance window to account for the brief connection reset. Create the mesh connectivity configuration in AVNM. Select the network groups, enable mesh topology, enable high-scale private endpoints, and enable global mesh if cross-region connectivity is required. Deploy incrementally. Start with a pilot region or non-critical environment. Validate effective routes, spoke-to-spoke connectivity, spoke-to-hub-to-on-premises connectivity, Private Endpoint reachability, and expected VNet flow log patterns before expanding to production regions. Route Behavior During and After Migration During migration, mesh and MSEE can coexist. Mesh-connected VNets receive direct routes for mesh-connected destinations, while existing ExpressRoute gateway routes continue to serve on-premises destinations. UDRs still override system routes, so forced-tunneling and firewall inspection patterns remain in effect when UDRs are present. Mesh destinations. Traffic between mesh-connected VNets goes directly instead of hairpinning through MSEE when no UDR overrides the route. On-premises destinations. ExpressRoute continues to provide north-south connectivity to on-premises networks. Gateway transit. Spokes can continue to reach on-premises through the hub gateway when the design uses gateway transit. Infrastructure-as-Code Considerations If VNet peerings are managed through Terraform, Bicep, or ARM templates, treat AVNM mesh as the new source of truth only after validation. Deploy mesh first. AVNM mesh can coexist with existing peerings, so do not remove peering resources from IaC before validating the mesh path. Validate the traffic path. Use effective routes, Connection Monitor, and flow logs to confirm traffic is using mesh where expected. Guard against drift. Review pipeline state and lifecycle settings before decommissioning old peerings, especially in environments where multiple teams manage network resources. Codify AVNM. Manage the Network Manager, network groups, configuration, and deployment through IaC so mesh becomes the governed connectivity model. DNS Resolution Across Mesh Mesh connectivity does not change DNS resolution behavior by itself. If spoke VNets are already linked to Private DNS Zones hosted or managed through the hub, those links continue to determine name resolution. If spokes use custom DNS servers in the hub, verify that any UDR changes made during migration do not unintentionally alter the DNS traffic path. Migration Example — Two Hub-and-Spoke Topologies into a Single Mesh This example shows how an enterprise can migrate two regional hub-and-spoke environments into one centrally managed AVNM mesh while preserving the existing MSEE path during validation. Current State — Two Hub-and-Spoke Topologies with MSEE Hairpin Contoso Corp operates a large Azure environment in the East US region with two hub-and-spoke topologies: Topology A Topology B Hub VNet Hub-A Hub-B Spoke VNets 500 500 ExpressRoute Gateway ER-GW-A in Hub-A ER-GW-B in Hub-B ExpressRoute Circuit Shared circuit, connected to both gateways Same shared circuit Avg. Private Endpoints per spoke ~8 (4,000 total) ~12 (6,000 total) Total Private Endpoints 10,000 across both topologies How traffic flows today: Spoke-to-spoke within Topology A: Spoke-A-01 → Hub-A → ER-GW-A → MSEE → ER-GW-A → Hub-A → Spoke-A-02 Spoke-to-spoke across topologies: Spoke-A-01 → Hub-A → ER-GW-A → MSEE → ER-GW-B → Hub-B → Spoke-B-01 Every spoke-to-spoke packet — whether within the same topology or across topologies — exits the data center, traverses MSEE, and re-enters. With 1,000 spokes generating east-west traffic, MSEE becomes a shared single point of failure and adds latency to every flow. Target State — A Single AVNM Mesh with 1,000 VNets Contoso's goal is to consolidate all 1,000 spoke VNets into a single AVNM mesh, removing MSEE from the east-west traffic path. Spoke-to-spoke traffic, any pair: Spoke-A-01 → directly→ Spoke-B-01. Traffic stays in the data center and uses the direct mesh path. MSEE role: MSEE carries north-south on-premises traffic. East-west load is removed from the ExpressRoute hairpin path. Migration Execution Phase 0 — Pre-Work Register the feature. Since the mesh contains 1,000 VNets, above the 250 standard limit, Contoso registers the AllowHighScaleConnectedGroup feature flag on the subscription. This enables high-scale mesh support for up to 5,000 VNets. Inventory Private Endpoints. With 10,000 Private Endpoints across 1,000 VNets, Contoso exceeds the standard mesh Private Endpoint limit of 2,000. HSPE must be enabled. Phase 1 — Enable HSPE on All 1,000 Spoke VNets Batch 1: Enable HSPE on all 500 spoke VNets in Topology A during the maintenance window by following the instructions. Batch 2: Apply the same configuration to all 500 spokes in Topology B. Expected impact: Each VNet may experience a brief connection reset for existing Private Endpoint connections when HSPE is enabled. Schedule the change during a maintenance window. Phase 2 — Create the AVNM Mesh Create a Network Manager scoped to the management group containing all 1,000 spoke VNets. Define a single network group called eastus-mesh-all-spokes with dynamic membership using an Azure Policy and the tags. Create a mesh connectivity configuration. Set topology to Mesh, network group to eastus-mesh-all-spokes, high-scale private endpoints to Enabled, and global mesh to Not needed because all VNets are in the same region. Save the configuration as a draft and do not deploy yet. Phase 3 — Incremental Deployment Wave 1 — Pilot: Contoso deploys the mesh configuration to 50 dev/test spokes, either by using a temporary network group or by tagging only those VNets initially. Validation includes effective routes showing ConnectedGroup as the next-hop type for meshed spoke prefixes, spoke-to-spoke connectivity through the direct mesh path, Private Endpoint reachability across meshed spokes, unchanged on-premises connectivity through the hub and MSEE, and VNet flow logs that confirm expected direct spoke-to-spoke flows. Wave 2 — Light production traffic VNets: After a successful pilot, Contoso tags light production traffic VNets. The dynamic network group picks them up automatically. Contoso redeploys the connectivity configuration, runs the same validation checklist, and monitors traffic to confirm that east-west traffic from these VNets is moving away from the ExpressRoute hairpin path. Wave 3 — All remaining production VNets: Contoso tags all remaining spokes and redeploys the configuration. At this point, all 1,000 spokes are in the mesh. No downtime migration: During Waves 2 and 3, existing MSEE hairpin routing remains functional. VNets not yet in the mesh continue to communicate through MSEE. VNets already in the mesh communicate directly through the mesh. This avoids a planned connectivity gap during migration. Phase 4 — Post-Migration Validation After deployment, confirm that mesh is active and traffic is taking the expected path before decommissioning the legacy route. Effective routes. Verify spoke subnets show direct routes to peer VNet prefixes instead of routing through the gateway or MSEE. Connection Monitor. Track representative spoke-to-spoke flows and compare latency and reachability before and after migration. VNet flow logs. Confirm east-west traffic matches the expected mesh path and is not still traversing the ExpressRoute gateway path. Network Watcher topology. Visualize the resulting connectivity model and identify any VNets not enrolled in the target network group. If traffic is still hairpinning after mesh deployment, check for UDRs overriding system routes, spokes missing from the network group, deployment not committed to the target region, or IaC pipelines recreating legacy peerings. Rollback Quick rollback while MSEE remains in place. Remove affected VNets from the network group or undeploy the mesh connectivity configuration. AVNM removes only the connectivity it created, and traffic can fall back to the existing MSEE hairpin path. Rollback after decommissioning legacy paths. If old peerings or route dependencies have already been removed, rollback may require reprovisioning those resources and should be treated as a longer change window. Recommendation. Keep the MSEE hairpin path available for at least two weeks after mesh deployment, monitor traffic patterns, and only then remove the legacy path. Before and After Summary Metric Before (MSEE Hairpin) After (AVNM HSPE Mesh) Spoke-to-spoke latency in the same region Higher due to the MSEE hairpin path Lower-latency direct in-datacenter path; actual latency depends on workload, region, and network conditions Traffic path for east-west Spoke → Hub → MSEE → Hub → Spoke Spoke → Spoke directly through the Mesh MSEE dependency for east-west Yes, shared dependency No MSEE dependency for east-west traffic Manual peerings required 0 when using hairpin routing, but 499,500 if built manually for 1,000 spokes 0 manual spoke-to-spoke peerings; AVNM manages connectivity Private Endpoints supported 2,000 per mesh under standard limits 20,000 per mesh with HSPE Rollback complexity Not applicable to the current hairpin model Remove VNets from the network group or undeploy the connectivity configuration Migration downtime Not applicable Designed for no planned downtime when deployed incrementally and validated carefully Closing Notes Migrating to AVNM mesh does not require tearing down your existing network. The hub gateway, firewalls, and NVAs continue to function as they do today. What changes is that east-west spoke-to-spoke traffic stops leaving the data center unnecessarily. MSEE is not the right tool for the east-west fabric. Removing internal traffic from the ExpressRoute circuit is a reliability and capacity improvement. AVNM mesh replaces combinatorial complexity with group-based intent. The operational model scales with the number of groups, not the number of VNets. High-scale mesh and HSPE remove the ceiling — up to 5,000 VNets and 20,000 Private Endpoints per mesh. The migration is incremental and reversible. Mesh coexists with existing paths, and you can validate wave by wave before decommissioning the legacy route. Start with a pilot mesh in a non-critical environment, validate the traffic shift, and expand from there. Resources Connectivity configurations in Azure Virtual Network Manager Increase Private Endpoint virtual network limits Create a mesh topology with Azure Virtual Network Manager Security admin rules in Azure Virtual Network Manager Create a mesh network topology with Azure Virtual Network Manager using Terraform
ICMP Support for Azure StandardV2 NAT Gateway
Our customers, across all industries, depend on consistent outbound connectivity and observability to operate and troubleshoot their cloud workloads at scale. As Azure environments grow in size and complexity, teams increasingly rely on standard network diagnostic tools such as ping to quickly validate reachability and identify connectivity issues. When customers route outbound traffic through Azure NAT Gateway, they benefit from centralized and scalable outbound connectivity designed to prevent SNAT port exhaustion, while simplifying management by removing the need for per‑resource public IPs. Historically, outbound connectivity through Azure NAT Gateway supported TCP and UDP traffic only for Azure workloads. We are pleased to announce that Azure StandardV2 NAT Gateway now supports outbound ping through ICMP Echo Request and Echo Reply traffic. Workloads that rely on NAT Gateway for internet egress can now use ping to validate reachability, improving operational visibility and simplifying network troubleshooting while preserving the scalability and resiliency characteristics of NAT Gateway. What’s now supported? Azure StandardV2 NAT Gateway supports outbound ICMP Echo Request and Echo Reply traffic (ping) for IPv4 and IPv6. Other ICMP message types are not currently supported. With this update, ICMP traffic sent from workloads behind a NAT Gateway can be translated to a public IP address and reliably receive responses from external endpoints. This brings ICMP behavior in line with existing TCP and UDP outbound flows, enabling consistent network diagnostics across all common transport types. This capability is available automatically when workloads egress through a StandardV2 NAT Gateway - no additional configuration or rule changes are required. Scenarios enabled by ICMP support Support for ICMP through StandardV2 NAT Gateway unlocks several common operational scenarios: Basic reachability testing – Operators can use ping from virtual machines, VM scale sets, or AKS nodes behind NAT Gateway to verify outbound connectivity to internet endpoints. Health validation and troubleshooting – Teams can perform quick checks during incident triage without deploying jump hosts, public IPs per VM, or temporary firewall exceptions. These scenarios are especially valuable in large‑scale environments where workloads rely on centralized outbound connectivity via NAT Gateway. How ICMP works through StandardV2 NAT Gateway Unlike TCP or UDP, ICMP does not use ports to differentiate flows. Instead, ICMP messages include an identifier field, which is used by diagnostic tools to correlate requests and responses. A common troubleshooting scenario is when workloads deployed behind an Azure StandardV2 NAT Gateway are unable to reach external endpoints. Since these workloads typically do not have public IP addresses, it can be difficult to quickly determine whether the issue is related to basic network reachability or higher-layer configuration. In such cases, ICMP‑based diagnostics such as ping can be used to validate outbound connectivity and help isolate the source of the issue. When a workload sends an outbound ICMP packet through a NAT Gateway: NAT Gateway translates the source private IP to one of its associated public IP addresses. The platform tracks the ICMP flow using the ICMP identifier and other packet attributes. Return ICMP responses from the external destination are matched to the originating flow. The response is translated back to the original private IP and delivered to the workload. This stateful handling allows replies to be routed correctly even when multiple workloads are sending ICMP traffic concurrently through the same NAT Gateway. Using ICMP for end-to-end outbound troubleshooting To validate outbound connectivity using ICMP from workloads deployed behind StandardV2 NAT Gateway: Go to your virtual network in the Azure portal and select the subnet hosting your workload. Verify that a StandardV2 NAT Gateway is associated with the subnet. The workload does not require a public IP address. Connect to the workload (for example, a virtual machine) running in the subnet. Run an ICMP ping to an external destination to validate outbound reachability. Successful ping from a workload behind an Azure StandardV2 NAT Gateway demonstrating outbound connectivity and ICMP Echo reply. Review the results. Successful ICMP echo replies confirm that outbound connectivity through StandardV2 NAT Gateway is functioning and that ICMP traffic is translated and returned correctly Summary With ICMP Echo support, Azure StandardV2 NAT Gateway provides a more complete outbound networking experience by enabling essential diagnostic tools to function reliably through NAT. Customers can validate reachability, perform basic troubleshooting, and diagnose connectivity issues while continuing to benefit from the simplicity and scale of centralized outbound connectivity. Learn More Azure NAT Gateway overview Troubleshoot Azure NAT Gateway
