Azure VMware Solution Security Design Considerations

[rvandenbedem]

Azure VMware Solution Design Series

• Availability Design Considerations
• Recoverability Design Considerations
• Performance Design Considerations
• Security Design Considerations

 

Overview

A global enterprise wants to migrate thousands of VMware vSphere virtual machines (VMs) to Microsoft Azure as part of their application modernization strategy. The first step is to exit their on-premises data centers and rapidly relocate their legacy application VMs to the Azure VMware Solution as a staging area for the first phase of their modernization strategy. What should the Azure VMware Solution look like?

Azure VMware Solution is a VMware validated first party Azure service from Microsoft that provides private clouds containing VMware vSphere clusters built from dedicated bare-metal Azure infrastructure. It enables customers to leverage their existing investments in VMware skills and tools, allowing them to focus on developing and running their VMware-based workloads on Azure.

In this post, I will introduce the typical customer workload security requirements, describe the Azure VMware Solution architectural components, describe the zero trust security model, and describe the security design considerations for Azure VMware Solution private clouds.

In the next section, I will introduce the typical security requirements of a customer’s workload.

Customer Workload Requirements

A typical customer has multiple application tiers that have specific Service Level Agreement (SLA) requirements that need to be met. These SLAs are normally named by a tiering system such as Platinum, Gold, Silver, and Bronze or Mission-Critical, Business-Critical, Production, and Test/Dev. Each SLA will have different availability, recoverability, performance, manageability, and security requirements that need to be met.

For the security design quality, customers will normally have governance, regulatory, and compliance requirements. This is normally documented for each application and then aggregated into the information security policy requirements for each SLA or line of business. For example:

Line of Business	Governance, Regulatory, & Compliance
Finance	ISO/IEC 27001:2022, GLBA, PCI DSS 3.2.1
US Government	NIST Cybersecurity Framework, FISMA, FedRAMP
Health Care	ISO/IEC 27001:2022, HIPAA, PCI DSS 3.2.1

Table 1 – Typical Customer requirements for Information Security

The security concepts introduced in Table 1 have the following definitions:

• Governance: The process of making and enforcing decisions within an IT organization or system. It encompasses decision-making, rule-setting, and enforcement mechanisms to guide the functioning of IT in alignment with the business goals and strategies.
• Regulatory: The set of laws, regulations, and guidelines that govern the collection, storage, processing, and sharing of sensitive information. These regulations are designed to ensure that organizations protect sensitive information from unauthorized access, use, disclosure, and destruction. Compliance with these regulations is mandatory and non-compliance can result in legal penalties, fines, and reputational damage.
• Compliance: The act or process of following the rules, regulations, and standards that apply to the IT organization or system. It can also refer to the ability of an IT system to adapt to changing requirements or demands.

A typical legacy business-critical application will have the following application architecture:

• Load Balancer layer: Uses load balancers to distribute traffic across multiple web servers in the web layer to improve application availability.
• Web layer: Uses web servers to process client requests made via the secure Hypertext Transfer Protocol (HTTPS). Receives traffic from the load balancer layer and forwards to the application layer.
• Application layer: Uses application servers to run software that delivers a business application through a communication protocol. Receives traffic from the web layer and uses the database layer to access stored data.
• Database layer: Uses a relational database management service (RDMS) cluster to store data and provide database services to the application layer.

Depending upon the security requirements for each service, infrastructure design could be a mix of technologies used to meet the different security policies with cost efficiency.

Figure 1 – Typical Legacy Business-Critical Application Architecture

In the next section, I will introduce the architectural components of the Azure VMware Solution.

Architectural Components

The diagram below describes the architectural components of the Azure VMware Solution.

Figure 2 – Azure VMware Solution Architectural Components

Each Azure VMware Solution architectural component has the following function:

• Azure Subscription: Used to provide controlled access, budget, and quota management for the Azure VMware Solution.
• Azure Region: Physical locations around the world where we group data centers into Availability Zones (AZs) and then group AZs into regions.
• Azure Resource Group: Container used to place Azure services and resources into logical groups.
• Azure VMware Solution Private Cloud: Uses VMware software, including vCenter Server, NSX software-defined networking, vSAN software-defined storage, and Azure bare-metal ESXi hosts to provide compute, networking, and storage resources. Azure NetApp Files, Azure Elastic SAN, and Pure Cloud Block Store are also supported.
• Azure VMware Solution Resource Cluster: Uses VMware software, including vSAN software-defined storage, and Azure bare-metal ESXi hosts to provide compute, networking, and storage resources for customer workloads by scaling out the Azure VMware Solution private cloud. Azure NetApp Files, Azure Elastic SAN, and Pure Cloud Block Store are also supported.
• VMware HCX: Provides mobility, migration, and network extension services.
• VMware Site Recovery: Provides Disaster Recovery automation and storage replication services with VMware vSphere Replication. Third party Disaster Recovery solutions Zerto Disaster Recovery and JetStream Software Disaster Recovery are also supported.
• Dedicated Microsoft Enterprise Edge (D-MSEE): Router that provides connectivity between Azure cloud and the Azure VMware Solution private cloud instance.
• Azure Virtual Network (VNet): Private network used to connect Azure services and resources together.
• Azure Route Server: Enables network appliances to exchange dynamic route information with Azure networks.
• Azure Virtual Network Gateway: Cross premises gateway for connecting Azure services and resources to other private networks using IPSec VPN, ExpressRoute, and VNet to VNet.
• Azure ExpressRoute: Provides high-speed private connections between Azure data centers and on-premises or colocation infrastructure.
• Azure Virtual WAN (vWAN): Aggregates networking, security, and routing functions together into a single unified Wide Area Network (WAN).

In the next section, I will introduce the Zero Trust Security Model which should be used as the framework for securing your Azure cloud resources.

Zero Trust Security Model

A holistic approach to Zero Trust should extend to your entire digital estate: inclusive of identities, endpoints, network, data, apps, and infrastructure. Zero Trust architecture serves as a comprehensive end-to-end strategy and requires integration across the elements.

The foundation of Zero Trust security is identities. Both human and non-human identities need strong authorization, connecting from either personal or corporate endpoints with compliant devices, requesting access based on strong policies grounded in Zero Trust principles of explicit verification, least-privilege access, and assumed breach.

As a unified policy enforcement, the Zero Trust policy intercepts the request, explicitly verifies signals from all six foundational elements based upon policy configuration and enforces least-privilege access. Signals include the role of the user, location, device compliance, data sensitivity, and application sensitivity. In addition to telemetry and state information, the risk assessment from threat protection feeds into the policy engine to automatically respond to threats in real time. Policy is enforced at the time of access and continuously evaluated throughout the session.

This policy is further enhanced by policy optimization. Governance and compliance are critical to a strong Zero Trust implementation. Security posture assessment and productivity optimization are necessary to measure the telemetry throughout the services and systems.

Telemetry and analytics feeds into the threat protection system. Large amounts of telemetry and analytics enriched by threat intelligence generate high-quality risk assessments that can be either manually investigated or automated. Attacks happen at cloud speed and, because humans can’t react quickly enough or sift through all the risks, your defense systems must also act at cloud speed. The risk assessment feeds into the policy engine for real-time automated threat protection and additional manual investigation if needed.

Traffic filtering and segmentation is applied to the evaluation and enforcement from the Zero Trust policy before access is granted to any public or private network.

Data classification, labeling, and encryption should be applied to emails, documents, and structured data. Access to apps should be adaptive, whether SaaS or on-premises. Runtime control is applied to infrastructure with serverless, containers, IaaS, PaaS, and internal sites with just-in-time (JIT) and version controls actively engaged.

Finally, telemetry, analytics, and assessment from the network, data, apps, and infrastructure are fed back into the policy optimization and threat protection systems.

Figure 3 – Zero Trust Security Model

Many legacy monolithic application stacks may use the older defense in depth security model, however, we introduced the zero trust security model here for consideration.

For more information, refer to the Microsoft Security Adoption Framework, and the Microsoft Cybersecurity Reference Architectures.

In the next section, I will describe the security design considerations for the Azure VMware Solution.

Security Design Considerations

The architectural design process takes the business problem to be solved and the business goals to be achieved and distills these into customer requirements, design constraints and assumptions. Design constraints can be characterized by the following three categories:

• Laws of the Land – data and application sovereignty, governance, regulatory, compliance, etc.
• Laws of Physics – data and machine gravity, network latency, etc.
• Laws of Economics – owning versus renting, total cost of ownership (TCO), return on investment (ROI), capital expenditure, operational expenditure, earnings before interest, taxes, depreciation, and amortization (EBITDA), etc.

Each design consideration will be a trade-off between availability, recoverability, performance, manageability, and security design qualities. The desired result is to deliver business value with the minimum of risk by working backwards from the customer problem.

Design Consideration 1 – Governance, Regulatory, & Compliance: Learn how Microsoft cloud services protect your data, and how you can manage cloud data security and compliance for your organization with these certifications, regulations, and standards. Also included are reports, whitepapers, artifacts, and industry/regional resources.

You can also use the Microsoft cloud security benchmark to address the common challenges customers have when securing their Azure infrastructure.

For more information visit Security, governance, and compliance disciplines for Azure VMware Solution.

Design Consideration 2 – Azure Region: Select the relevant Azure Regions that meet your governance, regulatory, and compliance requirements. Azure VMware Solution is available in 30 Azure Regions around the world. Azure VMware Solution is also available in two Azure Government regions and in scope for FedRAMP HIGH authorization. Azure VMware Solution in Azure Government is currently pending authorization for DoD IL4/IL5.

Design Consideration 3 – Identity & Access Management: Use role-based access control to provide secure access to the Azure VMware Solution.

Use Microsoft Entra Privileged Identity Management to allow time-bound access to the Azure portal and control pane operations. Use privileged identity management audit history to track operations that highly privileged accounts perform.

For more information refer to Security considerations for Azure VMware Solution workloads and Enterprise-scale identity and access management.

Design Consideration 4 – LDAPS Integration: Select an external identity source for your Azure VMware Solution.

The local cloud administrator user accounts for vCenter Server and NSX Manager should be used as emergency access accounts for "break glass" scenarios in your private cloud. It's not intended to be used for daily administrative activities or for integration with other services. Use an external identity source to give your administrators and operators access to the Azure VMware Solution. This allows them to use their corporate credentials when accessing the Azure VMware Solution.

For more information refer to Identity and access. This MTC blog post also provides a detailed procedure to Configure LDAPS within Azure VMware Solution.

Design Consideration 5 – SIEM Logging: Select the Azure VMware Solution Diagnostic setting to stream platform logs and metrics to your Security Information & Event Management (SIEM) solution.

Azure VMware Solution supports Azure Log Analytics, Azure Event Hub, and Azure Blob Storage as logging targets. The log stream contains the following:

• vCenter Server logs
• ESXi logs
• vSAN logs
• NSX Manager logs
• NSX Data Center Distributed Firewall logs
• NSX Data Center Gateway Firewall logs
• NSX Data Center Edge Appliance logs

The Azure VMware Solution does not currently support Syslog Forwarding to a customer Syslog server, however you can use this Syslog Forwarding function instead.

For more information refer to Configure VMware syslogs for Azure VMware Solution.

Figure 4 – Azure VMware Solution Logging Options

Design Consideration 6 – VMware vSAN Encryption: Use a Customer Managed Key to augment the Azure VMware Solution vSAN encryption process.

Customer-managed keys give you control over the encrypted vSAN data on Azure VMware Solution. You can use Azure Key Vault to generate customer managed keys and centralize the key management process.

For more information refer to Configure customer-managed key encryption at rest in Azure VMware Solution.

Figure 5 – Customer Managed Keys with Azure VMware Solution

Design Consideration 7 – VM Security: Use Trusted Launch with Azure VMware Solution to increase the security of your Virtual Machines.

Trusted Launch comprises of Secure Boot, Virtual Trusted Platform Module (vTPM), and Virtualization-based Security (VBS) to provide a formidable defense against modern cyber threats. Trusted Launch is a requirement for Windows 11 compatibility.

For more information refer to Trusted Launch for Azure VMware Solution virtual machines.

Design Consideration 8 – Firewall Placement: Select the placement locations of network security firewalls within the Azure VMware Solution or Azure native services, including the zones of trust topology to meet your traffic flow requirements.

Zones of trust refer to the concept of segmenting a network into different areas based upon the level of trust assigned to the devices and users within that area. This is used as part of the Zero Trust security model, where access to resources is granted on a need-to-know basis and is strictly enforced through continuous authentication and authorization.

Traffic flows refer to the movement of data between different zones of trust in a network. Understanding traffic flows is important for network design, capacity planning, and information security.

For Azure VMware Solution, the zones of trust need to account for the Azure VMware Solution private clouds, Azure native services, the internet, and other non-Azure locations.

Figure 6 – Zones of Trust

There are four options in the Azure Cloud Adoption Framework that describe different ways to secure and manage the network traffic between the Azure VMware Solution and other Azure resources, on-premises, and the internet:

1. Option 1: Use a Secured Virtual WAN hub with default route propagation to route all traffic through an Azure Firewall or a third-party security provider.
2. Option 2: Use Network Virtual Appliances in Azure Virtual Network to inspect all network traffic and apply firewall rules or policies.
3. Option 3: Egress from Azure VMware Solution with or without NSX or NVA to control the outbound traffic from Azure VMware Solution to other destinations.
4. Option 4: Use third-party firewall solutions in a hub virtual network with Azure Route Server to enable dynamic routing between Azure VMware Solution and the firewall appliances.

Each option has its own benefits and trade-offs depending upon your requirements and preferences.

In addition, we have VMware NSX features that can be used to secure north-south and east-west traffic, including the optional use of host-based endpoint solutions.

Figure 7 – Azure Firewall and 3^rd party NVA options

Option 1 – Secured Virtual WAN hub: Use a Secured Virtual WAN hub with default route propagation to route all traffic through an Azure Firewall or a third-party security provider.

This solution doesn't work for on-premises filtering and Global Reach bypasses the Virtual WAN hubs.

For more information refer to Azure Cloud Adoption Framework.

Figure 8 – Option 1: Secured Virtual WAN hub

Option 2 – Third-party Firewalls for all traffic: Use Network Virtual Appliances in Azure Virtual Network to inspect all network traffic and apply firewall rules or policies.

Choose this option if you want to use your existing NVA and centralize all traffic inspection in your hub virtual network.

For more information refer to Azure Cloud Adoption Framework.

Figure 9 – Option 2: Third-party Firewalls for all traffic

The diagram below provides an example of how FortiNet FortiGate NVAs can be used to build Scenario 2.

For more information refer to Azure routing and network interfaces.

Figure 10 – Example design of Option 2 with FortiNet FortiGate NVAs

Option 3 – Egress from Azure VMware Solution: Egress from Azure VMware Solution with or without NSX or an NVA to control the outbound traffic from Azure VMware Solution to other destinations.

Choose this option if you need to inspect traffic from two or more Azure VMware Solution private clouds. This option lets you use NSX native features. You can also combine this option with NVAs running on Azure VMware Solution between Tier-0 and Tier-1 Gateways.

For more information refer to Azure Cloud Adoption Framework.

Figure 11 – Option 3: Egress from Azure VMware Solution

Option 4 – Third-party Firewall for internet traffic: Use a third-party firewall solution in a hub virtual network with Azure Route Server to enable dynamic routing between Azure VMware Solution and the firewall appliances.

Choose this option to advertise the default route from an NVA in your Azure hub virtual network to an Azure VMware Solution.

For more information refer to Azure Cloud Adoption Framework.

Figure 12 – Option 4: Third-party Firewall for internet traffic

Option 5 – NSX Gateways: Use NSX Gateways to secure North-South network traffic.

You can use the network filtering capabilities of the Tier-0 and Tier-1 Gateways in NSX to provide North-South traffic filtering.

For more information refer to Azure VMware Solution Network Security.

Figure 13 – VMware NSX Gateway Firewall policies for North-South traffic

Option 6 – Micro segmentation with DFW: Use NSX Distributed Firewall to secure East-West network traffic.

You can use the Distributed Firewall in NSX to provide East-West micro segmentation of traffic flows.

For more information refer to Azure VMware Solution Network Security.

Figure 14 – VMware NSX Micro-segmentation (DFW) for East-West traffic

Option 7 – Microsoft Defender for Cloud: Use a Microsoft Defender for Cloud or a third-party host-based security solution to secure your traffic from the guest operating system.

Microsoft Defender for Cloud provides advanced threat protection across your Azure VMware Solution and on-premises virtual machines (VMs). It assesses the vulnerability of Azure VMware Solution VMs and raises alerts as needed. These security alerts can be forwarded to Azure Monitor for resolution. You can define security policies in Microsoft Defender for Cloud.

For more information refer to Integrate Microsoft Defender for Cloud with Azure VMware Solution.

Figure 15 – Microsoft Defender for Cloud with Azure VMware Solution

Design Consideration 9 – Internet Access: Azure VMware Solution has three options within each private cloud instance that can be configured. These are explicitly described separate from the firewall scenarios in Design Consideration 8 even though they overlap.

Option 1 – Internet Disabled: Use Azure native networking to provide internet access.

There are multiple ways to generate a default route in Azure and send it towards your Azure VMware Solution private cloud or on-premises. The options are as follows:

• An Azure firewall in a Virtual WAN Hub.
• A third-party Network Virtual Appliance in a Virtual WAN Hub Spoke Virtual Network.
• A third-party Network Virtual Appliance in a Native Azure Virtual Network using Azure Route Server.
• A default route from on-premises transferred to Azure VMware Solution over Global Reach.

For more information refer to Internet connectivity design considerations.

Figure 16 – Azure VMware Solution Internet Disabled

Option 2 – Managed SNAT: Managed SNAT service provides a simple method for outbound internet access from an Azure VMware Solution private cloud.

Features of Managed SNAT include the following:

• Easily enabled.
• No control over SNAT rules, all sources that reach the SNAT service are allowed.
• No visibility of connection logs.
• Two Public IPs are used and rotated to support up to 128k simultaneous outbound connections.
• No inbound DNAT capability is available.

For more information refer to Internet connectivity design considerations.

Figure 17 – Azure VMware Solution Managed SNAT Internet Access

Option 3 – Public IP Address (PIP): Use an allocated Azure Public IPv4 address directly with the NSX Edge for consumption.

PIP allows the Azure VMware Solution private cloud to directly consume and apply public network addresses in NSX as required. These addresses are used for the following types of connections:

• Outbound SNAT
• Inbound DNAT
• Load balancing using VMware NSX Advanced Load Balancer and other third-party Network Virtual Appliances
• Applications directly connected to a workload VM interface.

This option also lets you configure the public address on a third-party Network Virtual Appliance to create a DMZ within the Azure VMware Solution private cloud. The Azure Public IP addresses are customer owned and are not actively scanned by Microsoft. Microsoft Defender for Cloud or a suitable 3^rd party solution should be used fully secure the internet connection. These options are discussed in Design Consideration 8.

For more information refer to Internet connectivity design considerations.

Figure 18 – Azure VMware Solution VMware NSX Public IP Address (PIP)

The use of a “T1 Sandwich” with a third-party NVA, allows you to scale beyond the 10 vNIC limitation for VMware vSphere virtual machine hardware. The result is an increase in the number of networks you can protect with an NVA. The diagram below provides an example of how a CheckPoint NVA can be used with a T1 Sandwich topology.

Figure 19 – VMware NSX T1 Gateway Sandwich with CheckPoint

In the following section, I will describe the next steps that need to be made to progress this high-level design estimate towards a validated detailed design.

Next Steps

The Azure VMware Solution sizing estimate should be assessed using Azure Migrate. With large enterprise solutions for strategic and major customers, an Azure VMware Solution Solutions Architect from Azure, VMware, or a trusted VMware Partner should be engaged to ensure the solution is correctly sized to deliver business value with the minimum of risk. This should also include an application dependency assessment to understand the mapping between application groups and identify areas of data gravity, application network traffic flows, and network latency dependencies.

Summary

In this post, we took a closer look at the typical security requirements of a customer workload, the architectural building blocks, the zero trust security model, and the security design considerations for the Azure VMware Solution. We also discussed the next steps to continue an Azure VMware Solution design.

If you are interested in the Azure VMware Solution, please use these resources to learn more about the service:

• Homepage: Azure VMware Solution
• Documentation: Azure VMware Solution
• SLA: SLA for Azure VMware Solution
• Azure Regions: Azure Products by Region
• Security Fundamentals: Azure security fundamentals
• Zero Trust Model: Zero Trust Model - Modern Security Architecture
• Zero Trust Security Framework: Zero Trust security in Azure
• Microsoft Cybersecurity: Microsoft Cybersecurity Reference Architectures
• Adoption Framework: Microsoft Security Adoption Framework
• Microsoft cloud: Learn how Microsoft cloud services protect your data
• Benchmark: Microsoft cloud security benchmark
• Azure VMware Solution: Security, governance, and compliance disciplines
• Vulnerabilities: Concepts - How Azure VMware Solution Addresses Vulnerabilities
• Security Recommendations: Concepts - Security recommendations for Azure VMware Solution
• Security Baseline: Azure security baseline for Azure VMware Solution
• Defense in Depth: Microsoft Azure's defense in depth approach to cloud vulnerabilities
• Azure Compliance: Azure compliance documentation
• WAF: Security considerations for Azure VMware Solution workloads
• Identity & Access Management: Enterprise-scale identity and access management
• Configure LDAPS: Configure LDAPS within Azure VMware Solution
• Syslog: Configure VMware syslogs for Azure VMware Solution
• Syslog Forwarding: Syslog Forwarding function
• Customer-managed keys: Configure customer-managed key encryption at rest
• Trusted Launch: Trusted Launch for Azure VMware Solution virtual machines
• Network connectivity scenarios: Enterprise-scale network topology and connectivity for Azure VMware Solution
• Network Security: Azure VMware Solution Network Security
• Defender for Cloud: Integrate Microsoft Defender for Cloud with Azure VMware Solution
• Internet Connectivity: Internet connectivity design considerations
• Service Limits: Azure VMware Solution subscription limits and quotas
• GitHub repository: Azure/azure-vmware-solution
• Well-Architected Framework: Azure VMware Solution workloads
• Cloud Adoption Framework: Introduction to the Azure VMware Solution adoption scenario
• Enterprise Scale Landing Zone: Enterprise-scale for Microsoft Azure VMware Solution
• Enterprise Scale GitHub repository: Azure/Enterprise-Scale-for-AVS
• Azure CLI: Azure Command-Line Interface (CLI) Overview
• PowerShell module: Az.VMware Module
• Azure Resource Manager: Microsoft.AVS/privateClouds
• REST API: Azure VMware Solution REST API
• Terraform provider: azurerm_vmware_private_cloud Terraform Registry

Author Bio

René van den Bedem is a Principal Technical Program Manager in the Azure VMware Solution product group at Microsoft. His background is in enterprise architecture with extensive experience across all facets of the enterprise, public cloud & service provider spaces, including digital transformation and the business, enterprise, and technology architecture stacks. René works backwards from the problem to be solved and designs solutions that deliver business value with the minimum of risk. In addition to being the first quadruple VMware Certified Design Expert (VCDX), he is also a Dell Technologies Certified Master Enterprise Architect, a Nutanix Platform Expert (NPX), and a VMware vExpert.