ExpressRoute FastPath for UDRs and VNet peering

This article examines the data flow and performance benefits of Microsoft Azure's ExpressRoute and ExpressRoute FastPath features in hub & spoke environments. It outlines the default asymmetric data routing and the enhancements achieved through FastPath. Key updates and constraints for FastPath, as well as IP address limits and monitoring metrics, are also discussed.

Data flow with ExpressRoute

The data flow between On-Premises and Azure using ExpressRoute is asymmetric by design.

The traffic from on-Premises to Azure transits via the ExpressRoute Gateway, but the return traffic (Azure to on-Premises) bypasses the ExpressRoute Gateway and is forwarded directly to the MSEEs.

If multiple ExpressRoute Circuits are advertising identical on-Premises routes and are connected to the same ExpressRoute Gateway, Azure to on-Premises traffic is distributed (ECMP) across the different ExpressRoute Circuits available unless traffic engineering is configured to prioritise one path over the others.

Data flow with ExpressRoute FastPath enabled

Enabling ExpressRoute FastPath allows on-Premises to Azure traffic to bypass the ExpressRoute Gateway, providing improved data path performances.

FastPath on ExpressRoute Direct circuit connections now also honors:

• UDRs on the Gateway subnet, configured to route traffic through an AzFW in the hub (or a 3P FW NVA) for inspection: traffic coming from on-Premises and matching a UDR on the Gateway subnet will be sent directly to the AzFW or 3P NVA, bypassing the ExpressRoute Gateway.
• VNet peering within the same region: traffic coming from on-Premises will be sent directly to VMs in spoke VNets, bypassing the ExpressRoute Gateway (global VNet peering is not supported).

ExpressRoute FastPath is configured per connection.

This results in reduced latency and the capability to exceed the ExpressRoute Gateway's maximum throughput limit of 10 Gbps with the UltraPerformance or ErGW3AZ Gateway SKU, or the 40 Gbps limit of the upcoming ExpressRoute Scalable Gateway SKU.

Current constraints and limitations:

• IP address limit: see section below.
• Private Endpoint/Private Link: limited GA support, for 100 Gbps ExpressRoute Direct circuits only. Not supported with ExpressRoute partner circuits.
• (temporary) Internal Load Balancers in a spoke VNet: if traffic destined to an ILB in a spoke VNet, traffic will go via the ERGW instead. Likewise, UDRs whose next hop is an ILB in a spoke VNet will not be honored. However, for an ILB in the hub, FastPath will work. 
• vWAN: FastPath is automatically enabled for vWAN hubs with ER Gateways of 5+ scale units that are connected to ER Direct Circuits. FastPath will work for flows destined for spoke VNets directly connected to hub, but won't work for NVA/FW in the datapath. 

VNet peering/UDR support vs PL support with FastPath

When network policies for UDRs are enabled for Private Endpoint (PEP), the /32 route propagated by the PEP becomes invalid. A UDR of the PEP VNet range configured on the Gateway subnet is enough to direct the PEP traffic through a firewall in the hub. With this UDR in place, onboarding to the Private Link FastPath feature for early GA is not necessary. The traffic will follow the path: ExpressRoute MSEE > hub firewall > PEP in spoke, leveraging UDR/VNet peering support for FastPath, which is already GA.

About the FastPath IP address limit

As per documentation, the following limits apply for the number of FastPath IPs. The limit is applied per ExpressRoute provider circuit (in the "Service Provider model") or per ExpressRoute Direct port (when using the "Direct model"):

• When FastPath is enabled, this limit represents the maximum number of Azure endpoints in the VNet environment that can bypass the ExpressRoute Gateway. Whenever a new IP address is added or removed, the FastPath IPs are automatically reprogrammed to reflect that change, keeping the FastPath IP list up to date.
• The limit applies to customer private IPs ; for example, a VM with 3 NICs requires 3 FastPath routes, each counting towards the limit (note: FastPath is configured per connection, not per circuit).
• If the FastPath IP limit is exceeded, traffic will not get dropped but will instead be routed according to the default behaviour via the ExpressRoute Gateway.

It is important to note that when calculating the limit for FastPath IPs, only endpoints with assigned private IP addresses within a subnet or VNet range are counted. The entire address range won't be included in the calculation of consumed IPs.

Impact of UDR support with FastPath on the IP address limit:

When FastPath honors the Gateway subnet UDRs, individual /32 IPs within a UDR range will no longer be programmed by FastPath. For example, if there is a VM with IP address 10.0.0.4 and a 10.0.0.0/16 UDR on the Gateway subnet with the next-hop set to the hub firewall, only the /16 UDR will be programmed by Fastpath.

Azure Monitor offers metrics for ExpressRoute Direct resources, which includes the ability to track the number of configured FastPath routes at the port level.

Key takeaway: FastPath is configured per connection, while the FastPath IP limit is per Service Provider circuit or per ExpressRoute Direct port (= an overall limit for all the FastPath-enabled connections terminating on that port).