Scenarios for AVS connectivity utilizing Private VPN Gateways and ExpressRoute for Azure Virtual WAN
Published May 25 2023 08:36 AM 3,394 Views
Microsoft

Introduction and Purpose: 

 

Existing example reference architectures to implement network connectivity between Azure and Azure VMware Solutions (AVS) recommend the use of ExpressRoute Global Reach where available. However, there are also scenarios where either Global Reach is not available (such as Brazil South), or customers simply might have different requirements and need an alternate solution that substitutes Global Reach entirely.   

 

Generally speaking, one viable solution is to implement a public Site-to-Site VPN with Border Gateway Protocol (BGP) while enabling transit capabilities between an ExpressRoute Gateway and a VPN gateway. The goal here is to advertise AVS or other non-overlapping IP address spaces dynamically via BGP. This is commonly referred to as Overlay. For a Hub and Spoke Topology this can be accomplished by enabling Branch to Branch along with Azure Route Server. For Azure Virtual WAN, this would happen once Branch to Branch is enabled (usually it is enabled by default).  

 

Scope 

 

Another solution that is being adopted more frequently and is equally effective is to utilize ExpressRoute as an underlay network to establish a Site-to-Site VPN, using private IPs. Other reasons besides not having access to Global Reach include preventing AVS traffic from going through the public Internet, and most importantly, the ability to advertise AVS IP Prefixes or other IP address spaces dynamically via BGP. In this manner connectivity between AVS and Azure is accomplished on the overlay, in other words AVS prefixes are injected into the Azure platform via BGP over IPSec. 

 

This article will describe 2 scenarios and the related solutions where customers choose to advertise an underlay private prefix via ExpressRoute and establish a BGP Over IPSec tunnel using Private IP address space utilizing Azure Virtual WAN as part of their Landing Zone topology. This scenario presents some challenges when utilizing Azure vWAN, however it does work flawlessly when utilizing a Hub and Spoke topology.  

 

For all these test scenarios, 1.1.1.1/32 will represent our Overlay Network. 

 

General VWAN VPN setup recommendations. 

 

To successfully install a VPN tunnel using private IPs with BGP, it must ensure that the IPsec peer IP of the on-premises device and the BGP peer IP have different IPs, as explained at the bottom of this section. These can be contiguous or noncontiguous IP ranges. Additionally, since these configurations leverage the use of BGP, the IP address space on the VPN configuration section must be empty as described in this article   

 

Scenario 1. ExpressRoute advertises the on-premises address RFC 1918 space.  

 

Mauricio_Rojas_0-1685026705162.png

 

In this scenario, the on-premises address space is 172.16.100.0/24 

 

Mauricio_Rojas_1-1685026735105.png

 

It can be observed on the Virtual Hub Route table that the overlay route is learned via the VPN tunnel with  AS-Path of 65103, and the underlay has the on-premises IP prefix, RFC 1918 and is learned through the ExR circuit.  

 

Effective routes on the spoke NIC show the overlay prefix being received.  

 

 

 

 

maroja@Azure:~$ az network nic show-effective-route-table -g spoke1 -o table -n spoke11 

Source                 State    Address Prefix    Next Hop Type          Next Hop IP 

---------------------  -------  ----------------  ---------------------  ------------- 

Default                Active   10.5.0.0/16       VnetLocal 

VirtualNetworkGateway  Active   172.16.100.0/24   VirtualNetworkGateway  10.20.89.20 

VirtualNetworkGateway  Active   1.1.1.1/32        VirtualNetworkGateway  172.16.1.13 

 

 

 

Important: There is the possibility that the on-premises device re-advertises the on-premises address space through the BGP Over IPSec tunnel with a shorter AS path. That will override the announcement of the ExpressRoute prefix which has a longer AS Path (12076, 65001, ASN Y), therefore creating BGP flaps on the VPN tunnel. Please ensure the use of Route Maps or policies to prevent these re-advertisements.  

 

Scenario 2. ExpressRoute advertises a default 0.0.0.0/0 Route 

 

Mauricio_Rojas_2-1685027808555.png

 

Although in this scenario we can see the default route being learned from on-premises, the IPsec tunnel will flap when the Route Preference is set to ExpressRoute, which is the default for VWAN. In consequence, BGP will also establish intermittently. This is because the Route Service that manages the routing on the vWAN side does not know that the packet destined over the VPN tunnel peer must traverse the Microsoft Enterprise Edge Routers (MSEE). This behavior is particular to vWAN and it differs to the Hub and Spoke VPN Gateway, where the VPN Gateway will know how to route all the packets back to the MSEE.  

 

For this scenario to work, there are two options:  

 

Option 1: Change the route preference to VPN. You can make the change by following these instructions Configure virtual hub routing preference: Azure portal - Azure Virtual WAN | Microsoft Learn 

 

We can see the results of the Effective Route tables on the Virtual Hub below as expected. The overlay route is being advertised successfully through the S2S VPN with its correspondent ASN. 

 

Mauricio_Rojas_3-1685028160168.png

 

We can also see the results propagated to the NIC on a spoke VM. The NIC effective routes on the spokes show the VPN gateway BGP IP peer for the overlay route, as well as the underlay routes being learned via the MSEEs. 

 

 

 

maroja@Azure:~$ az network nic show-effective-route-table -g spoke1 -o table -n spoke11 

Source                 State    Address Prefix    Next Hop Type          Next Hop IP 

---------------------  -------  ----------------  ---------------------  ------------- 

Default                Active   10.5.0.0/16       VnetLocal 

VirtualNetworkGateway  Active   1.1.1.1/32        VirtualNetworkGateway  172.16.1.13 

VirtualNetworkGateway  Active   0.0.0.0/0         VirtualNetworkGateway  10.20.89.20 

Default                Active   172.16.1.0/24     VNetGlobalPeering 

 

 

 

 

Option 2: If changing the routing preference is not attainable due to specific requirements, another solution is to advertise the IPsec private IP of the on-premises device as explained here. The following image depicts the advertisement of the VPN IPsec /32.  

 

Mauricio_Rojas_0-1685028468081.png

 

Effective Routes on the virtual hub 

 

Mauricio_Rojas_1-1685028502174.png

 

In this scenario it can be observed that the 0/0 route is learned through the ExpressRoute Gateway as well as the /32 prefix of the on-premises peer IP. The overlay route is being advertised successfully through the S2S VPN with its correspondent ASN. 

 

The NIC effective routes on the spokes show the VPN gateway BGP IP peer for the overlay route as well as the underlay routes being learned via the MSEEs. 

 

 

 

maroja@Azure:~$ az network nic show-effective-route-table -g spoke1 -o table -n spoke11 

Source                 State    Address Prefix    Next Hop Type          Next Hop IP 

---------------------  -------  ----------------  ---------------------  ------------- 

Default                Active   10.5.0.0/16       VnetLocal 

VirtualNetworkGateway  Active   172.16.100.3/32   VirtualNetworkGateway  10.20.89.20 

VirtualNetworkGateway  Active   0.0.0.0/0         VirtualNetworkGateway  10.20.89.20 

VirtualNetworkGateway  Active   1.1.1.1/32        VirtualNetworkGateway  172.16.1.13 

Default                Active   172.16.1.0/24     VNetGlobalPeering 

 

 

Conclusion 

 

In conclusion, this article explores the implementation of network connectivity between Azure and Azure VMware Solutions (AVS) using Private VPN Gateways and ExpressRoute within the context of Azure Virtual WAN.  

 

 It is important to consider specific constraints of each scenario when implementing AVS connectivity utilizing Private VPN Gateways and ExpressRoute for Azure Virtual WAN. By leveraging these solutions, organizations can establish secure and efficient connectivity between AVS and Azure, ensuring seamless communication and the dynamic advertisement of IP prefixes. 

Co-Authors
Version history
Last update:
‎May 25 2023 08:49 AM
Updated by: