Introduction and Purpose:
Existing example reference architectures to implement network connectivity between Azure and Azure VMware Solutions (AVS) recommend the use of ExpressRoute Global Reach where available. However, there are also scenarios where either Global Reach is not available (such as Brazil South), or customers simply might have different requirements and need an alternate solution that substitutes Global Reach entirely.
Generally speaking, one viable solution is to implement a public Site-to-Site VPN with Border Gateway Protocol (BGP) while enabling transit capabilities between an ExpressRoute Gateway and a VPN gateway. The goal here is to advertise AVS or other non-overlapping IP address spaces dynamically via BGP. This is commonly referred to as Overlay. For a Hub and Spoke Topology this can be accomplished by enabling Branch to Branch along with Azure Route Server. For Azure Virtual WAN, this would happen once Branch to Branch is enabled (usually it is enabled by default).
Scope
Another solution that is being adopted more frequently and is equally effective is to utilize ExpressRoute as an underlay network to establish a Site-to-Site VPN, using private IPs. Other reasons besides not having access to Global Reach include preventing AVS traffic from going through the public Internet, and most importantly, the ability to advertise AVS IP Prefixes or other IP address spaces dynamically via BGP. In this manner connectivity between AVS and Azure is accomplished on the overlay, in other words AVS prefixes are injected into the Azure platform via BGP over IPSec.
This article will describe 2 scenarios and the related solutions where customers choose to advertise an underlay private prefix via ExpressRoute and establish a BGP Over IPSec tunnel using Private IP address space utilizing Azure Virtual WAN as part of their Landing Zone topology. This scenario presents some challenges when utilizing Azure vWAN, however it does work flawlessly when utilizing a Hub and Spoke topology.
For all these test scenarios, 1.1.1.1/32 will represent our Overlay Network.
General VWAN VPN setup recommendations.
To successfully install a VPN tunnel using private IPs with BGP, it must ensure that the IPsec peer IP of the on-premises device and the BGP peer IP have different IPs, as explained at the bottom of this section. These can be contiguous or noncontiguous IP ranges. Additionally, since these configurations leverage the use of BGP, the IP address space on the VPN configuration section must be empty as described in this article
Scenario 1. ExpressRoute advertises the on-premises address RFC 1918 space.
In this scenario, the on-premises address space is 172.16.100.0/24
It can be observed on the Virtual Hub Route table that the overlay route is learned via the VPN tunnel with AS-Path of 65103, and the underlay has the on-premises IP prefix, RFC 1918 and is learned through the ExR circuit.
Effective routes on the spoke NIC show the overlay prefix being received.
maroja@Azure:~$ az network nic show-effective-route-table -g spoke1 -o table -n spoke11
Source State Address Prefix Next Hop Type Next Hop IP
--------------------- ------- ---------------- --------------------- -------------
Default Active 10.5.0.0/16 VnetLocal
VirtualNetworkGateway Active 172.16.100.0/24 VirtualNetworkGateway 10.20.89.20
VirtualNetworkGateway Active 1.1.1.1/32 VirtualNetworkGateway 172.16.1.13
Important: There is the possibility that the on-premises device re-advertises the on-premises address space through the BGP Over IPSec tunnel with a shorter AS path. That will override the announcement of the ExpressRoute prefix which has a longer AS Path (12076, 65001, ASN Y), therefore creating BGP flaps on the VPN tunnel. Please ensure the use of Route Maps or policies to prevent these re-advertisements.
Scenario 2. ExpressRoute advertises a default 0.0.0.0/0 Route
Although in this scenario we can see the default route being learned from on-premises, the IPsec tunnel will flap when the Route Preference is set to ExpressRoute, which is the default for VWAN. In consequence, BGP will also establish intermittently. This is because the Route Service that manages the routing on the vWAN side does not know that the packet destined over the VPN tunnel peer must traverse the Microsoft Enterprise Edge Routers (MSEE). This behavior is particular to vWAN and it differs to the Hub and Spoke VPN Gateway, where the VPN Gateway will know how to route all the packets back to the MSEE.
For this scenario to work, there are two options:
Option 1: Change the route preference to VPN. You can make the change by following these instructions Configure virtual hub routing preference: Azure portal - Azure Virtual WAN | Microsoft Learn
We can see the results of the Effective Route tables on the Virtual Hub below as expected. The overlay route is being advertised successfully through the S2S VPN with its correspondent ASN.
We can also see the results propagated to the NIC on a spoke VM. The NIC effective routes on the spokes show the VPN gateway BGP IP peer for the overlay route, as well as the underlay routes being learned via the MSEEs.
maroja@Azure:~$ az network nic show-effective-route-table -g spoke1 -o table -n spoke11
Source State Address Prefix Next Hop Type Next Hop IP
--------------------- ------- ---------------- --------------------- -------------
Default Active 10.5.0.0/16 VnetLocal
VirtualNetworkGateway Active 1.1.1.1/32 VirtualNetworkGateway 172.16.1.13
VirtualNetworkGateway Active 0.0.0.0/0 VirtualNetworkGateway 10.20.89.20
Default Active 172.16.1.0/24 VNetGlobalPeering
Option 2: If changing the routing preference is not attainable due to specific requirements, another solution is to advertise the IPsec private IP of the on-premises device as explained here. The following image depicts the advertisement of the VPN IPsec /32.
Effective Routes on the virtual hub
In this scenario it can be observed that the 0/0 route is learned through the ExpressRoute Gateway as well as the /32 prefix of the on-premises peer IP. The overlay route is being advertised successfully through the S2S VPN with its correspondent ASN.
The NIC effective routes on the spokes show the VPN gateway BGP IP peer for the overlay route as well as the underlay routes being learned via the MSEEs.
maroja@Azure:~$ az network nic show-effective-route-table -g spoke1 -o table -n spoke11
Source State Address Prefix Next Hop Type Next Hop IP
--------------------- ------- ---------------- --------------------- -------------
Default Active 10.5.0.0/16 VnetLocal
VirtualNetworkGateway Active 172.16.100.3/32 VirtualNetworkGateway 10.20.89.20
VirtualNetworkGateway Active 0.0.0.0/0 VirtualNetworkGateway 10.20.89.20
VirtualNetworkGateway Active 1.1.1.1/32 VirtualNetworkGateway 172.16.1.13
Default Active 172.16.1.0/24 VNetGlobalPeering
Conclusion
In conclusion, this article explores the implementation of network connectivity between Azure and Azure VMware Solutions (AVS) using Private VPN Gateways and ExpressRoute within the context of Azure Virtual WAN.
It is important to consider specific constraints of each scenario when implementing AVS connectivity utilizing Private VPN Gateways and ExpressRoute for Azure Virtual WAN. By leveraging these solutions, organizations can establish secure and efficient connectivity between AVS and Azure, ensuring seamless communication and the dynamic advertisement of IP prefixes.