Nested virtualization on Azure : a step-by-step guide

For many developers, working on a virtual machine (VM) offers several benefits. It allows them to select the configuration that best meets their requirements without investing significantly in physical hardware. VMs provide flexibility and scalability, which can be particularly advantageous for developers, machine learning engineers, and data scientists. They often need to use Docker for their projects, a process known as Nested Virtualization.

Nested virtualization is a feature that enables running Hyper-V inside a Hyper-V virtual machine (VM). Over time, hardware has improved, and the use cases for Nested Virtualization have increased. For instance, Nested Virtualization can be useful for:

• Running applications or emulators in a nested VM
• Testing software releases on VMs
• Reducing deployment times for training environments
• Using Hyper-V isolation for containers

There are also Azure AI services, available as disconnected containers, and these services sometimes have significant hardware requirements that may even surpass those of some gaming PCs. In this case, Azure VMs might be suitable for ML engineers to utilize VMs for launching disconnected containers.

Modern processors include hardware features that enhance virtualization speed and security. Hyper-V relies on these processor extensions, such as Intel VT-x and AMD-V, to run virtual machines. Nested virtualization makes this hardware support available to guest virtual machines.

The following diagram shows Hyper-V without nesting. The Hyper-V hypervisor fully controls the hardware virtualization capabilities (orange arrow), and does not expose them to the guest operating system.

While there are official instructions available, I believe that my step-by-step guide, enriched with personal tips and tricks not covered in the official documentation, will be useful. Up we go!

Step 1: Choose a VM

It is important to consider that not all VM architectures support nested virtualization, so it should be planned in advance. Microsoft provides an official page with an ACU comparison, which serves as a valuable resource. The Azure Compute Unit (ACU) allows comparison of compute (CPU) performance across different Azure SKUs. ACU standardizes the performance measurement using a Small (Standard_A1) virtual machine as a baseline score of 100, enabling users to identify the SKU most likely to meet their performance requirements. A table on this page indicates that three asterisks (***) denote hyper-threading and capability for nested virtualization

However, Azure has ceased publishing ACUs as the metric has limited ability to predict performance across various attributes of a virtual machine. For accurate performance verification, Azure recommends running specific workloads on the target virtual machine. As a result, the page will no longer be maintained and is scheduled for archival on December 16, 2024. Currently, the list of VMs supporting nested virtualization includes:

• D_v3
• Ds_v3
• Dv4
• Dsv4
• Ddv4
• Ddsv4
• E_v3
• Es_v3
• Ev4
• Esv4
• Edv4
• Edsv4
• F2s_v2 - F72s_v2
• FX4 - FX48
• M

You may select any option; however, please be aware that if you intend to use Docker Desktop, your machine must meet the specified system requirements.

• WSL version 1.1.3.0 or later.
• Windows 11 64-bit: Home or Pro version 21H2 or higher, or Enterprise or Education version 21H2 or higher.

• Windows 10 64-bit (Recommended): Home or Pro 22H2 (build 19045) or higher, or Enterprise or Education 22H2 (build 19045) or higher. (Minimum): Home or Pro 21H2 (build 19044) or higher, or Enterprise or Education 21H2 (build 19044) or higher. Update Windows
• 64-bit processor with Second Level Address Translation (SLAT).
• 4GB system RAM.

For example, below is the configuration of my virtual machine.

Another critical step is selecting the appropriate image. Depending on the usage scenario, for data scientists, machine learning engineers, or full-stack developers working with Azure disconnected containers, I recommend utilizing the Data Science Virtual Machine (DSVM). DSVMs are pre-installed, configured, and tested Azure Virtual Machine images equipped with numerous popular tools frequently used for data analytics, machine learning, and AI training.

When creating a virtual machine on Azure, you typically search for "VM" in the Azure marketplace, select the VM size and image, and among the available options, you will find the DSVM.

However, it is advisable to search directly for DSVM. This approach eliminates the need to configure the private network and other settings, as everything is pre-configured. Although this method saves only about five minutes, it significantly simplifies the process, making it worth considering.

The steps discussed in this article apply to Windows and thus are applicable to DSVM Windows Server 2019 and 2022. It is recommended to select a Windows 2022 image.

Step 2: Configure VM

After configuring your virtual machine (VM) by selecting the size, image, login, and password, you can connect to your VM using the public IP address. Upon connection, you will be prompted to decide whether you want your VM to be visible to other machines on the local network. Select "yes" and allow some time for the automatic PowerShell scripts to complete their execution. These scripts will configure the Azure ML extension and other parameters. This process typically takes a few minutes.

Now launch the Power Shell as administrator and run the following command

Set-ExecutionPolicy bypass -force

This will change the execution policy to successfully run a PowerShell script to set up nested virtualization.

Now run:

Invoke-WebRequest 'https://aka.ms/azlabs/scripts/hyperV-powershell' -Outfile SetupForNestedVirtualization.ps1 .\SetupForNestedVirtualization.ps1

This procedure will download and execute the script to enable the Hyper-V feature and associated tools. When prompted, type [A] to set the response as [Yes to All], thereby eliminating the need to enter [Y] for each confirmation.

The script will return an error message indicating that a VM restart is required. You can restart the VM either from within the VM itself by using the restart button in the Windows menu, or through the Azure portal by stopping or restarting the VM. The latter option is preferable as it allows you to monitor when the VM is active and ready for connection.

After restarting the VM, connect to it and reset the execution policy by opening PowerShell as an Administrator.

Set-ExecutionPolicy default -force

The VM is now configured to support nested virtualization, allowing you to create VMs within it. Open your browser and search for "Download Docker Desktop." Select the installation file for Windows ARM 64 and execute it. This will download Docker Desktop along with all necessary dependencies, including WSL.

Docker is a tool used to create, deploy, and run applications using containers. Containers allow developers to package an application with all its required components (libraries, frameworks, dependencies, etc.) into one package. Using a container ensures that the application will run consistently, regardless of any customized settings or previously installed libraries on the computer running it, which may differ from the machine used to write and test the application's code.

Windows Subsystem for Linux (WSL) is a Windows feature that allows you to run a Linux environment on your Windows machine without needing a separate virtual machine or dual-boot setup. WSL is designed to provide a seamless and productive experience for developers who want to use both Windows and Linux simultaneously.

Once Docker is installed, you can test pulling some images. For example, pull an image from the Microsoft Container Registry for AI services, such as text summarization. Open CMD or PowerShell and run the following command.

docker pull mcr.microsoft.com/azure-cognitive-services/textanalytics/summarization:cpu

You have successfully enabled nested virtualization on your Azure VM, congratulations! This allows you to experiment with containers for various purposes, such as web development, data science, LLM finetuning, and developing GenAI applications, even if you have a modest PC.

Step 3: resolving common issues

Enabling RDP

After installing Docker, some users cannot access their VMs from PCs because the RDP port is not opened in the network security group (NSG), all Internet traffic is blocked by default.

To enable the RDP port in an NSG:

• Sign in to the Azure portal.
• In Virtual Machines, select the VM with the issue.
• Go to Settings > Networking.
• In Inbound port rules, ensure the RDP port is configured correctly:
• Priority: 110
• Name: Port_3389
• Port(Destination): 3389
• Protocol: TCP
• Source: Any
• Destinations: Any
• Action: Allow

If you specify the source IP address, ensure your computer's IP falls within the range specified.

Use Windows VM from Mac

To connect to the VM by using RDP, use the Microsoft Remote Desktop app.

To install the Microsoft Remote Desktop app:

1. Open the App Store on your Mac, and search for Microsoft Remote Desktop
2. Select Install to install the latest version of Microsoft Remote Desktop.

Please note that the standard commands function differently (for instance, in VSCode, instead of ⌘+ Shift + P, use ⌃ + Shift + P, etc. However, ⌘+ C and ⌘ + V work in the same manner), as these are distinct keyboard shortcuts. However, you will be able to adapt easily.

Insufficient Disk Size

At times, you may encounter the issue of insufficient disk space, particularly when working with SLMs locally. Although the D: drive is available for use, please remember that it does not offer persistent storage and will be erased upon shutdown. When a new virtual machine (VM) is created within a resource group by deploying an image from Azure Marketplace, the default operating system (OS) disk is typically 127 GiB (with some images having smaller OS disk sizes by default). Additional data disks can be added to your VM, depending on the VM SKU selected, and it is recommended to install applications and CPU-intensive workloads on these data disks.

To resize your managed disk:

1. In the Azure portal, go to the virtual machine in which you want to expand the disk. Select Stop to deallocate the VM.
2. In the left menu under Settings, select Disks.

3. Under Disk name, select the disk you want to expand.

4. In the left menu under Settings, select Size + performance.

5. In Size + performance, select the disk size you want.

6. Select Resize at the bottom of the page.

Now you will need to expand the disk volume in the operating system.

On you VM

1. Start a remote desktop session with the VM
2. Open Disk Management.

3. Right-click on existing C: drive partition -> Extend Volume.

4. Follow the steps you should be able to see the disk with updated capacity:

Install Docker on other versions of Windows Server

When running Docker Desktop on Windows Server 2019 Datacenter Eva, the installation process fails.

Docker only supports Docker Desktop on versions of Windows that are within Microsoft’s servicing timeline. Docker Desktop is not supported on such server versions of Windows. For running containers on those servers, please follow the steps in this documentation: Get started: Prep Windows for containers

 

In this article, you have learned how to enable and configure nested virtualization on Azure, providing the ability to run virtual machines within virtual machines. This guide has covered the selection of appropriate VM sizes and images, configuration steps, and troubleshooting common issues. Leveraging VMs on Azure offers significant flexibility and scalability, making them a powerful tool for developers, data scientists, and machine learning engineers, especially for those who do not have access to high-performance physical hardware. By following these steps, you can efficiently utilize Azure's capabilities to enhance your computational tasks and development workflows.