Nested Virtualization Tips

[planetmaher]

Azure Lab Services provides the ability to create a template machines with nested Hyper-V virtual machines.  See Enable nested virtualization on a template virtual machine in Azure Lab Services documentation for further details.  This is useful for classes where you need a couple machines to talk to one another, like the Ethical Hacking class described in the class type documentation.

Let’s talk about some tips when you need to create your own template machine with nested Hyper-V virtual machines.

1. Consider which Hyper-V VMs are needed for the class, not the department. 

There is sometimes a tendency to want to create one template for all classes in a department.  Taking the time to consider what is needed for only a specific class can save money by allowing for a smaller template machine size to be used. Save the image for reuse in the future.

2. Use VHDX for virtual disks. 

The OS disk size for Lab Services VMs depends on the selected image and for most of the images its 128 GB. The size of a VHD disk for a Hyper-V VM matches the maximum size you want that virtual disk to be.  The VHDX format allows for the actual size of the file to be reduced and to later be increased up to the maximum disk size.  Using the VHDX file format for your virtual disk could potentially allow for more Hyper-V VMs on a lab template machine.  This is dependent on the number and size of files stored on each virtual disk, though.

The ability to shrink virtual disks only works with the VHDX file format.  If you have an existing VHD file, use the Convert-VHD PowerShell cmdlet to convert to a VHD file to VHDX.  Use Resize-VHD PowerShell cmdlet to shrink the virtual disk size and store more virtual disks on the template machine. 

3. Use dynamic memory when creating the Hyper-V virtual machine. 

Users can set the minimum and maximum RAM for the Hyper-V virtual machine.  This allows other Hyper-V virtual machines or the host machine to use the memory when it is not needed.  This can be useful in cases where a Hyper-V VM only needs to be used in bursts rather than continuously.

We hope you find these tips useful.  Please comment below if you know of more tips or have a question.

Thanks,

The Lab Services Team

[mamoreau]

I have been working very hard to build a proper lab of nested VMs using Azure Lab Services, but the 128GB hard disk is just incredibly limiting. I have used a lot of tricks not discussed in this blog post to cut down on the size, so I truly wonder how people really do it without running out of disk space. The truly essential part was to disable all checkpoints (even though they are useful, they're a luxury we cannot afford here) and I created sysprepped images + used Hyper-V differencing disks. Without the differencing disks, there is no way I would have been able to create 5 VMs in the large instance (8 cores, 32GB RAM, nested virtualization). It all works well *except* for the suffocating storage space. It's a shame because this really is the only problem to which I cannot find a good solution, I like everything else about the product.

I'd be more than happy to have a discussion with someone from the Azure Lab Services team, I'd like to get the word out about this wonderful product that I didn't know existed until recently. I really wish I could just pay extra for VM instances that have a decent storage space included.

[Jim Ratsch]

I ran into similar limitations, specifically when using nested virtualization. You may want to consider checking out Azure DevTest labs, as it has a lot more flexibility and you can use more storage, additional drives, upload your own, etc. 

I've heard that more flexibility may be coming to ALS, but in the meantime, DevTest Labs may work out well for you.

Hope this helps!

[mamoreau]

I haven't checked the Azure DevTest Labs yet, but I suspect it doesn't have the same ease and lab shareability as Azure Lab Services, does it? I really like the fact that you just edit a template then easily duplicate it to a large number of people in a cost-efficient way. The ability to invite people outside of my organization and control the number of hours they can use their own test VMs is really a killer feature of Azure Lab Services.

Do you know if expanded storage is coming soon? There's a feature request with a few other people running into the same issues I do, but it's unclear if this limitation is being prioritized or not. The fact that a blog post with tips to save on disk space at least shows that the need is understood, but at the same thing labs using nested VMs cannot realistically fit inside 128GB. I sacrificed one VM to go back down to 4 VMs, but a lot of features in a Windows domain require a lot more VMs to correctly demonstrate.

https://feedback.azure.com/forums/320373-lab-services/suggestions/36495472-be-able-to-add-additional...

[Jim Ratsch]

I totally agree, the cost tracking and ease of inviting users is tough to beat, and I'm watching closely because my customers also want this capability. Unfortunately, I haven't heard anything definitive on it yet. 

For DevTest Labs, you might consider checking out shared image galleries. Set up a gallery, image, etc. and then you can share it out. You might find it easier to create the gallery in regular Azure first and then share it out to use in DevTest labs. Inviting other users is not as simple as with ALS, but it might fit your needs. I'm hoping the two will either merge or the capabilities will overlap eventually.

Check out this article on sharing out the galleries: 

Share VM images with Shared Galleries - Azure Virtual Machines | Microsoft Docs

[mamoreau]

You seem to have been following ALS for much longer than me, do you know where the key people involved hang out? You're actually the first person to respond so far, the feedback site and UserVoice only appear to have sporadic responses, and I couldn't find anybody on twitter. I would love to have a more direct way to talk to the team, as this product definitely fits the bill for me aside from the low storage space.

I was thinking of writing a blog post giving all the tips and tricks I learned along the way, but with the storage space issue, my final recommendation would be that is that it can only be used for a lab of 4 VMs maximum, if you used all the tricks in the book to save space.

As for the rest, you have probably gone through the same process I did, and figured out ways to transfer large files onto the template VM *other* than over RDP which is way too slow. Because I didn't know that you had to create the virtual network *before* the lab to attach a lab to a virtual network, I pretty much got to where I wanted after the 3rd lab created from scratch. You definitely need at last one "lab controller VM" to transfer files more easily to the template VM.

The thing is that even though the workflow ALS enforced appears relatively simple, I don't have the time to make a DIY equivalent, especially for cost control, easy duplication and sharing of the lab VMs. Have you found the equivalent Azure VM SKUs for the instances offered through ALS? I'm particularly interested in the 8 core, 32GB RAM + nested virtualization instance.

This being said, due to my strong background in RDP through the FreeRDP project and what I'm working on these days at Devolutions with Wayk Bastion, there are a few things I could offer to the ALS team to improve secure external RDP connectivity. We have an open source Remote Desktop Gateway replacement (Devolutions Gateway) that does relaying at the TCP-level and accepts a JWT instead of doing the heavyweight RD Gateway protocol, it would be a whole lot better than using the current RDP dynamic range of ports. We are also about to adapt it for SSH access, so it would be usable for that as well in the future.

Are you aware of anyone building their own Azure Lab Services replacement with a similar type of workflow, with a single template VM efficiently duplicated for a large number of students, with good enough cost control to reduce waste to a minimum? Azure VMs have an auto-shutdown feature, but that's once per day, for instances that cost 1$ per hour that's just too much waste if students forget it for a couple of hours. If you've got links to blog posts that cover parts of the puzzle, like how to deal with the disk duplication easily for Azure VMs, I'm all ears.

To conclude, I'd be curious to learn more about your use case, as we may be able to share tips.

[jieun-msft]

hi @mamoreau thanks so much for your interest in AzLabs! The extra disk space is definitely a feedback we hear from other customers as well. We're currently working on a bunch of infrastructure improvements to improve performance and reliability and other features such as the ability to connect to VMs through the browser (via Azure Bastion) and expanded network configurability for the May/June timeframe, so we expect we'll be able to pick up investigating the work to increase disk size post-June. We'll post more updates here as we get closer to that timeframe. 

 The forum (https://aka.ms/azlabs-forum) would be the best place to hold discussions. We're trying to consolidate our channels a bit more, so we expect the forum to play a bigger role for conversations going forward! For reporting any product issues, please log a support ticket from the product.