VMware HCX Design with Azure VMware Solution

Overview

VMware HCX is one of the Azure VMware Solution components that generates a large number of service requests from our customers. The Azure VMware Solution product group has worked to cover the most common design considerations that you should know about when using VMware HCX with the Azure VMware Solution.

Azure VMware Solution is a VMware validated first party Azure service from Microsoft that provides private clouds containing VMware vSphere clusters built from dedicated bare-metal Azure infrastructure. It enables customers to leverage their existing investments in VMware skills and tools, allowing them to focus on developing and running their VMware-based workloads on Azure.

VMware HCX is the mobility and migration software used by the Azure VMware Solution to connect remote VMware vSphere environments to the Azure VMware Solution. These remote VMware vSphere environments can be on-premises, co-location or cloud-based instances.

Figure 1 – Azure VMware Solution with VMware HCX Service Mesh

In the next section, I will introduce the architectural components of the Azure VMware Solution.

Architectural Components

The diagram below describes the architectural components of the Azure VMware Solution.

Figure 2 – Azure VMware Solution Architectural Components

Each Azure VMware Solution architectural component has the following function:

• Azure Subscription: Used to provide controlled access, budget and quota management for the Azure VMware Solution.
• Azure Region: Physical locations around the world where we group data centers into Availability Zones (AZs) and then group AZs into regions.
• Azure Resource Group: Container used to place Azure services and resources into logical groups.
• Azure VMware Solution Private Cloud: Uses VMware software, including vCenter Server, NSX software-defined networking, vSAN software-defined storage, and Azure bare-metal ESXi hosts to provide compute, networking, and storage resources. Azure NetApp Files, Azure Elastic SAN, and Pure Cloud Block Store are also supported.
• Azure VMware Solution Resource Cluster: Uses VMware software, including vSAN software-defined storage, and Azure bare-metal ESXi hosts to provide compute, networking, and storage resources for customer workloads by scaling out the Azure VMware Solution private cloud. Azure NetApp Files, Azure Elastic SAN, and Pure Cloud Block Store are also supported.
• VMware HCX: Provides mobility, migration, and network extension services.
• VMware Site Recovery: Provides Disaster Recovery automation, and storage replication services with VMware vSphere Replication. Third party Disaster Recovery solutions Zerto DR and JetStream DR are also supported.
• Dedicated Microsoft Enterprise Edge (D-MSEE): Router that provides connectivity between Azure cloud and the Azure VMware Solution private cloud instance.
• Azure Virtual Network (VNet): Private network used to connect Azure services and resources together.
• Azure Route Server: Enables network appliances to exchange dynamic route information with Azure networks.
• Azure Virtual Network Gateway: Cross premises gateway for connecting Azure services and resources to other private networks using IPSec VPN, ExpressRoute, and VNet to VNet.
• Azure ExpressRoute: Provides high-speed private connections between Azure data centers and on-premises or colocation infrastructure.
• Azure Virtual WAN (vWAN): Aggregates networking, security, and routing functions together into a single unified Wide Area Network (WAN).

In the next section, I will describe the design considerations for VMware HCX when used with the Azure VMware Solution.

Design Considerations

When deploying VMware HCX be sure to consider the following design considerations for a successful migration to the Azure VMware Solution.

Design Consideration 1: Multi-Site Network Extension Topology. Multi-site Network Extension topologies are supported by VMware HCX. These should be used in specific cases and discussed in-depth with your architecture team before adoption.

VMware HCX Network Extension appliances do have the capability to extend the same on-prem network to multiple Azure VMware Solution destinations. It is important to note that a common broadcast domain will be used across all environments. Below is a summary of the supported types of multi-site network extensions.

Topology 1: One-to-Many/”V” Architecture: In a One-to-Many topology, a source network can be extended to multiple (up to 3) Azure VMware Solution environments. Here are the design implications when utilizing a one-to-many network extension deployment.

VMware HCX uses point-to-point appliance pairs (NE appliances cannot connect to multiple destinations). It is important to note that a common broadcast domain will be used across all connected sites.

Figure 3 – One-to-Many Network Extension Topology

Extending the same source network to three different Azure VMware Solution is also supported under the one-to-many network extension topology.

Figure 4 – One-to-Many Network Extension Topology

Topology 2: Daisy Chain/”L” Architecture: Daisy chaining or the “L” shaped topology network extension is a supported architecture with VMware HCX. It can be used when you are looking to extend the same network across multiple sites.

In a Daisy chain or “L” shaped topology, the same network can be extended up to 3 environments. This will utilize a common broadcast domain across all connected sites. Please note in this configuration the gateway remains on-premises, additional latency will be incurred.

Figure 5 – Daisy Chain/ “L” Network Extension Topology

Topology 3: Any-to-Any Architecture: For an any-to-any network extension to be supported, the network extension can only be extended between two destinations. Please note, it is supported to have VMware HCX migrations between three sites in a closed loop architecture.

Figure 6 – Any-to-Any Network Extension Topology

Design Consideration 2: MTU requirements for Network Profiles. When configuring your Network profiles in VMware HCX it is important to take into consideration the MTU size of each profile. Be sure to validate the required MTU, as requirements change depending on how connectivity to VMware HCX will be established from on-premises (IPSec VPN, Azure ExpressRoute or VMware NSX Public IP).

Use this guide of recommended MTU sizes for the Network Profiles when connecting to Azure VMware Solution:

 

Table 1 – VMware HCX Network Profile MTU Sizes

Design Consideration 3: Limitations of number of VMware HCX Mobility Optimized Networking (MON) enabled VMs. When Deploying VMware HCX in Azure VMware Solution, the default HCX Manager size is set to 4 vCPU and 12 GB of memory.

With this default configuration you will have the following limitations when it comes to VMs with MON enabled:

• 250 VMs with MON enabled
• 100 Network Extension with MON enabled
• 100 concurrent Migration to MON enabled networks

Within Azure VMware Solution, the option to increase the vCPU and memory configuration of the HCX Manager is possible through a Run Command. The HCX Manager will be increased to 8 vCPU and 24 GB of memory.

With a scaled-up HCX Manager the MON limitations are increased:

• 900 VMs with MON enabled
• 100 Network Extensions with MON enabled
• 100 concurrent Migrations to MON enabled networks.

Figure 7 – Azure VMware Solution Run Command for VMware HCX

Design Consideration 4: DHCP Server on a MON Network Extension. When using DHCP on a MON enabled network, be sure that the default gateway IP and DHCP server IP are not the same. Having the IP address of the default gateway and DHCP server the same, can lead to network disruptions on a MON enabled network. The Network Gateway for the extended segment can provide DHCP services but must have a unique IP address for the DHCP server.

Design Consideration 5 – Anti-Patterns: Try to avoid using these anti-patterns in your recoverability design. The following Multi-Site Network Extension Topologies are not supported with VMware HCX.

Topology 1: One-to-Many/”V” Architecture: Extending an on-prem site to a fourth Azure VMware Solution private cloud is not supported through VMware HCX.

Figure 8 – Unsupported One-to-Many Network Extension Topology

Topology 2: Daisy Chain/”L” Architecture: Extending an on-prem site to a fourth Azure VMware Solution private cloud is not supported in a Daisy Chain/ “L” shape architecture. This will exceed the hop-limit of VMware HCX.

Figure 9 – Unsupported Daisy Chain/ “L” Network Extension Topology

Topology 3: Any-to-Any Architecture: The any-to-any multi-site network extension is supported by VMware. VMware HCX does not support a closed loop layer 2 extension in this type of design. VMware network extension appliances do not detect or mitigate loops that may occur in a closed loop setup.

Figure 10 – Unsupported Any-to-Any Network Extension Topology

In the following section, I will describe the next steps that would need to be made to progress this high-level design estimate towards a validated detailed design.

Next Steps

The Azure VMware Solution sizing estimate should be assessed using Azure Migrate. With large enterprise solutions for strategic and major customers, an Azure VMware Solution Solutions Architect from Azure, VMware, or a VMware Partner should be engaged to ensure the solution is correctly sized to deliver business value with the minimum of risk. This should also include an application dependency assessment to understand the mapping between application groups and identify areas of data gravity, application network traffic flows, and network latency dependencies.

Summary

In this post, we described helpful design considerations when using VMware HCX with the Azure VMware Solution.

In this post, we took a closer look at the architectural building blocks of Azure VMware Solution, and the design considerations of using VMware HCX with the Azure VMware Solution. We also discussed the next steps to continue an Azure VMware Solution design.

If you are interested in the Azure VMware Solution, please use these resources to learn more about the service:

• Homepage: Azure VMware Solution
• Documentation: Azure VMware Solution
• SLA: SLA for Azure VMware Solution
• Azure Regions: Azure Products by Region
• Design: Availability Design Considerations
• Design: Recoverability Design Considerations
• Design: Performance Design Considerations
• Design: Security Design Considerations
• VMware Ports and Protocols for HCX VMware HCX - VMware Ports and Protocols
• VMware Interoperability Matrix Product Interoperability Matrix (vmware.com)
• VMware HCX: Configuration & Best Practices
• Troubleshooting: VMware HCX Troubleshooting with Azure VMware Solution
• GitHub repository: Azure/azure-vmware-solution
• Well-Architected Framework: Azure VMware Solution workloads
• Cloud Adoption Framework: Introduction to the Azure VMware Solution adoption scenario
• Network connectivity scenarios: Enterprise-scale network topology and connectivity for Azure VMware Solution
• Enterprise Scale Landing Zone: Enterprise-scale for Microsoft Azure VMware Solution
• Enterprise Scale GitHub repository: Azure/Enterprise-Scale-for-AVS
• Azure CLI: Azure Command-Line Interface (CLI) Overview
• PowerShell module: Az.VMware Module
• Azure Resource Manager: Microsoft.AVS/privateClouds
• REST API: Azure VMware Solution REST API
• Terraform provider: azurerm_vmware_private_cloud Terraform Registry

Author Bios

Ricky Perez is a Senior Technical Program Manager in the Azure VMware Solution product group at Microsoft. His background is in solution architecture with experience in public cloud and core infrastructure services.

Jason Trammell is a Senior Software Engineer in the Azure VMware Solution engineering group at Microsoft.

Kenyon Hensler is a Principal Technical Program Manager in the Azure VMware Solution product group at Microsoft. His background is in system engineering with experience across all facets of enterprise networking and compute stacks.

René van den Bedem is a Principal Technical Program Manager in the Azure VMware Solution product group at Microsoft. His background is in enterprise architecture with extensive experience across all facets of the enterprise, public cloud & service provider spaces, including digital transformation and the business, enterprise, and technology architecture stacks. René works backwards from the problem to be solved and designs solutions that deliver business value with the minimum of risk. In addition to being the first quadruple VMware Certified Design Expert (VCDX), he is also a Dell Technologies Certified Master Enterprise Architect, a Nutanix Platform Expert (NPX), and a VMware vExpert.

Link to PPTX Diagrams: azure-vmware-solution/azure-vmware-master-diagrams