Windows Server Summit 2024
Mar 26 2024 08:00 AM - Mar 28 2024 04:30 PM (PDT)
Microsoft Tech Community
LIVE
AMD Nested Virtualization Support
Published Jun 10 2020 07:56 AM 118K Views
Microsoft

AMD Nested Support showing a VM running on a VM on AMD HardwareAMD Nested Support showing a VM running on a VM on AMD Hardware

Nested Virtualization is not a new idea. In fact, we announced our first preview of Nested Virtualization running on Windows way back in 2015.  From that Windows Insider preview to now, Nested Virtualization has been used in a variety of offerings in a variety of ways.  Today, you can find Nested Virtualization support in Azure that gives the Azure users flexibility in how they want to setup their environments.  An example of Nested Virtualization being used to support our developer community is to accelerate Microsoft’s Android Emulation.  Nested Virtualization is being used by  IT Pros to set up a home labs. And we can’t forget containers! If you want to use a Hyper-V Containers inside a VM, you guessed it: this is enabled with Nested Virtualization.  You can start to see why Nested Virtualization is such a useful technology.

 

There is one group of users that was unable to take advantage of Nested Virtualization on Windows. These were our users with AMD hardware.  Not a week goes by where the team doesn’t get a request for Nested Virtualization support for AMD from our community or from within Microsoft.  In fact, it is the number 1 ask on Windows Server’s uservoice page. At the time of this blog post, it was almost 5x more than the next feedback item.

 

I am happy to announce that the community has been heard and starting with Windows Build 19636, you will be able to try out Nested Virtualization on AMD processors! If you’re on the Windows Insider Fast ring then you can try this out today.

 

As this is a preview release of Nested Virtualization on AMD, there are some guidance and limitations to keep in mind if you want to try this out.

  • Ensure your OS build number is 19636 or greater
  • Right now, this has been tested on AMD’s first generation Ryzen/Epyc or newer processors
  • For maximum stability and performance use a Windows guest with an OS version that is greater than or equal to the host OS version (19636) for now.  Linux KVM guest support will be coming in the future
  • Create a version 9.3 VM. Here’s an example PowerShell command to ensure a version 9.3 VM is being used:  New-Vm -VMName “L1 Guest” -Version 9.3
  • Follow the rest of the steps in our public documentation

 

June 12, 2020 edit: changed wording around Guest OS recommendation.

100 Comments

Awesome news! Hyper-V is my favorite Hypervisor and the most powerful one.

Copper Contributor

Where do we go to find, or monitor progress, with Hyper-V Default vSwitch enhancements or fixes?  It's still causing many bandwidth issues with certain wifi choosers, so it seems. Our team needs a fix. 

Brass Contributor

This is a wonderful news. However "Use a Windows guest with an OS version that is greater than or equal to the host OS version (19636) for now. " is not a correct statement in my opinion. I was able to successfully use Windows 10 1909 in the guest which is the version of Windows far below the 19636. From my own testing every 64 bit Windows operating system is working as a guest and it does not have to be newer than 19636.

Copper Contributor

Kind of way overdue, but thank you Microsoft, this is a great day for large (and fast growing) segment of your users. Looking forward to this landing in stable builds...

 

@proboszcz  - I am guessing that is the usual "just in case" to cover all bases, plus with all the work on mixed hypervisors and Linux community, that soon we will have support for all weird mixes like VMware, KVM, and any old weird OS we can think of. But thanks for confirming it that quickly!

Microsoft

@proboszcz LpAdm is absolutely right.  We haven't validated the scenarios and performance on earlier versions of Windows.  In order to ensure a high performance, we have to have some enlightenments in the guest OSs.  This is also why we didn't bring up Linux support yet.  We're still working on some enlightenments there.  Once we have that, I'll make another post to share with the community at large.  Keep trying things out!

 

Chris

Copper Contributor

Now please enable me to use nested virtualization in the root domain (in Windows). This is the last puzzle piece that keeps me from upgrading to WSL2. 

Brass Contributor

@LpAdm @chuybregts , I totally agree with you. However I think it would be better to change that statement to something like this : "For maximum stability and performance use a Windows guest with an OS version that is greater than or equal to the host OS version (19636) for now" which will be more precise, because the current one suggests that using older versions of Windows will not work at all, which is not the case.

Iron Contributor

Please enable this also for older AMD Processors!

Brass Contributor

@chuybregts Can you tell us if this feature is planned to be included in Windows 10 20H2 final release? Also what about Windows Server 2019 - is it planned to be included to it or have we wait till the next big Windows Server 2022 version?

You also told something about enlightenments that should be included in level1 guest os for full performance. Will those enlightenments be propagated to earlier versions of Windows like Windows 10 1909, Windows Server 2019, etc.?

 

Is there a dedicated place where we can share our experience with this feature (report bugs, compatibility, etc) or should we do this here in the comments and Feedback Hub?

Deleted
Not applicable

Thank you for adding nested virtualization for AMD!!  Now if I could only get my android emulator to work in nested mode... but that's another story.

 

REFS heads up:  During the upgrade your writable REFS volumes will be upgraded to REFS version 3.5 and will not be readable prior to any build before 19536 (which includes 2004/19041 and earlier builds).  When reverting your build, you'll lose access to your REFS volumes.  REFS volume upgrade behavior is noted here: https://gist.github.com/0xbadfca11/da0598e47dd643d933dc#mountability

Microsoft

@Michael Kiesel How old are we talking about here?  The biggest problem is there are some required CPU features to enable this support.  Depending on the age, this could prevent nested from working.

 

@proboszcz You asked a few questions, let me see if I can address each one.

When will this feature be released: Our plan is to release it in future versions of Windows & Server but I'm not going to give a committed plan just yet as we need to get feedback and telemetry with the current implementation to see how much more work needs to be done.  Said another way, as of right now, yes, but, it's not a 100% yes.

Re: earlier versions - We're looking to backport this to Windows Server 2019 but don't have a committed date on that yet.

As for enlightenments, let me check with the team. I'll post an update here when I get an answer.  

Using the feedback hub is the best place for bugs as there are options for people to share logs there and it goes right into our bug tracking database.  For everything else, you can post here, uservoice, or feedback hub.  Note, I'll likely be checking here the most so take that into account.  

 

@Deleted Some folks over on Github got the android emulator to work. Also the emulator team is aware of the feature reaching this level of maturity.  They haven't shared when their support gets into an Insider build but I can say they're excited as this is a highly desired feature for them.

Brass Contributor

@chuybregts 

Thank you very much for your answers!

I will then be waiting for more information about the planned release and backporting of this feature from you then. 

 

From my own testing I can tell that for now everything is working just fine except the 32bit Windows when launched as a Level 2 guest from 64 bit level 1 guest. All my attempts ends up with a BSOD with TRAP_CAUSE_UNKNOWN code. Except for that this feature seems to work quite stable - good work guys :)

I was even able to run old 64bit Windows XP as a level 2 guest :)

Iron Contributor

@chuybregts 

Thanks for your response. Here is a post on Twitter where i made two screenshosts of the older Processor:

https://twitter.com/excelsi84/status/1273128417429512193?s=20 

 

Microsoft

@kwinz We've spent a lot of time working on the best way to offer compatibility with 3rd party virtualization software running directly in the root partition.  Our solution is the Windows Hypervisor Platform API.  See: https://docs.microsoft.com/en-us/virtualization/api/ and https://docs.microsoft.com/en-us/virtualization/api/hypervisor-platform/hypervisor-platform.  This powerful interface has proven to be very flexible and has already been adopted by major software vendors including VMware.  See: https://blogs.vmware.com/workstation/2020/05/vmware-workstation-now-supports-hyper-v-mode.html

Copper Contributor

How do I upgrade a Hyper-V VM from 9.0 to 9.3 so I can finally leverage this fix? It appears that this is the last step before I can finally successfully run 

Set-VMProcessor -VMName <VMName> -ExposeVirtualizationExtensions $true

on my vm so that I can leverage nested virtualization (Finally be able to use Docker inside my vm) 

 

I am using an AMD Ryzen 5 3600 CPU and currently running a Windows 10 Pro Hyper-V VM inside my Windows 10 Pro OS. Both are on preview build 20150 and the VM  is version 9.0 Gen 2

 

Hopefully someone can point me in the right direction.

Copper Contributor
Brass Contributor

@chuybregts   How do I get this onto current windows 10 pro-2004?

 

I don't want to upgrade the entire base operating system to an insiders build.

 

It is an update to the underlying hyper-v hypervisor; There most be a way of merely patching the hypervisor without moving the entire OS to an insiders build?

Come on MS, we are hurting for this feature and want it delivered via a windows update. Not an insiders build update that will take 1-2 years to hit the mainstream.

Just want to patch the HV on my windows pro 2004, don't even need to update the GUI on HV management tools, because as far as I can tell, the expose CPU extensions is done via PowerShell on the VM itself. HV just needs to support SVM passthrough on the underlying hypervisor.

Why is this update an insider build patch????

 

is there anyway, official or unofficial, I can update my underlying hyper-v hypervisor, that's sit's underneath my windows 10 pro 2004 install, to get Nested under AMD to pass through, without moving the whole system to a windows insiders build??

Copper Contributor

Just updated my gear to AMD from Intel and few VMs arent working because of this :( , for now i turned off nested virtualization because I dont want to update my main rig to fast track.

Copper Contributor

lack of nested Virt was the sole reason holding me back from getting a Ryzen 3950X.  It is defo good news that this is now supported, although within the insider arena at this stage. 

Brass Contributor

@roozbeh18 you don't have to update you main rig to insider builds - you can use Native VHD Boot feature to have a separate OS with insider builds for that purpose. Thanks to that you will have your main OS untouched and can switch between the main OS and the insider one during the boot time.

Brass Contributor

@proboszcz Thanks for the information, I have set up a double boot with native vhd boot and an Insider track installed on VHD + my C drive as my main OS.

I have found that after using it a few times to test it out, it works OK, but is far far from ideal. I feel separated from my Main OS when I am in a Dual Boot environment.

 

It is a bit disingenuous to state that you are not switching your main rig over, Right?

I mean, it does keep your "Main OS" Intact, but otherwise, it's a complete change that introduces additional problems.

 

  • what about all your files, your settings, everything being remapped to D:, all your program install's, and an ever-expanding VHD taking up all your precious space on your SSD? 
  • And dual-booting between OS's means you will never be able to pick up where you just left off?
  • Problems with hibernate/suspend/resume?
  • Permissions HELL where there are unknown users with permissions in either OS?
  • Windows Insiders cannot be updated inside a VHD Boot. For some unknown reason.
  • One needs to hack with Hyperv VM on the Main OS side, boot up the Hyper-V VHD in a VM on the Main OS Side, to update.
  • This introduces activation and Licence Hell, as windows belives the hardware has changed.
  • And on... and on... and on... lots of niggly issues with this approach.
  • It really is not an ideal solution.

 

Amateur Opinion time:

In My Very amateur and humble opinion, my understanding is the following:

Hyper-V is a HyperVisor that sits UNDER Windows 10. Windows 10 isn't even aware of it. The Hyper-V management tools such as Hyper-V manager connect to the underlying hypervisor via a socket on the localhost network interface port.

 

Windows has nothing to do with this Hypervisor management. Hyper-V is its own things that sits underneath.

 

Again, I am by no means a super expert on these things, but I just can't see why I cannot only update the Hyper-V underneath without moving to an insider build. Even the management tools do not need an update, as nested virtualisation & the VM Version is managed over flags and commands inside PowerShell.

 

Why can't I JUST update the Hyper-V???

Manually, separately, like I do with the WSL2 Linux Kernel?

Can anyone explain?

 

Thanks very much.

Brass Contributor

@Ameeno I totally agree with you that this approach with Native VHD Boot is not ideal. But it is more nice than installing a second OS on a separate partition I think. Using such VHD you can very easily go from one PC to another with the same OS and its settings. However that comes at a cost you described.

 

@chuybregts can you share with us an estimation when this feature will be added at least to a slow (Beta) ring? Forcing our main development rigs' OS into a fast (dev) ring is a very uncomfortable situation.

Copper Contributor

I'm also desperate for information on when this feature will be available on the bi-annual or at least beta build.  I've been dual-booting to a dev build but like others cannot devote my rig solely to that, and switching between installs is cumbersome and just not viable.  

 

Yes - silly stupid me for not checking AMD compatibility before spending thousands on a Threadripper rig but this is painful!  Please please let us have some information soon!

Copper Contributor

Hi,

There is a support for threadripper 2970wx, on Windows 10 Build19636 for nested virtualization?

 

Regards

Copper Contributor

@brucesherwin @chuybregts 

 

Windows Hypervisor Platform API strikes me as a workaround that forces 3rd party virtual machine software to abandon large parts of well tested virtualization technology and instead using Hyper-V technology for virtualization in the back end. Instead of directly using nested VT-x/AMD-V in the root domain. Of course Hyper-V would still emulate those instructions the same way it already can outside the root domain.

 

What keeps you from offering this more generic solution that does not requite complete architecture of virtual machine software and adoption of this proprietary API?

Copper Contributor

Hello all

I've seen a lot of contradictory information regarding the actual status of this feature. So does it actually work with Ryzen CPUs on current 20H2?

I'm about to upgrade big time and if this doesn't work with the current production Windows 10 version (and not some preview) on Zen, this is an absolute no and I'll go Intel.

Thanks

Copper Contributor

Hi @GuillaumeIT64, the feature hasn't arrived in the consumer release versions yet.  Only preview builds from 19636 have this and Windows 20H2 is 19042.  The current dev build is something like 20257 but it's far from reliable enough for everyday use. 

 

Will be more optimistic when the beta channel gets past 19636 (in fact would probably take a chance and install this) but there has been little if any talk from Microsoft since this blog post was published.  I wish they would just tell us something.

 

So - if you're adamant that this is a deal breaker, you're either buying an Intel chip or going with VMWare, which already supports AMD nested virtualisation. I've had a play with this and it appeared to work (at least booting nested Hyper-V VMs) but I rely on Hyper-V for work and it really didn't like nested VMWare and needed disabled before anything would work.

Copper Contributor

Hi @philwest1 

 

Thanks for this answer.

Same for me, I rely a lot on Hyper-V.

So this indeed means I have to go Intel or hope that it quickly makes it to a stable beta channel release. I find it quite disturbing as well that after this promising step in the right direction, we haven't heard more since then.

Cheers

Copper Contributor

Bringing nested virtualization on AMD CPU is more than welcome, it could finally bring more professional guys, especially developers to choose AMD over Intel. So we are all waiting for this to be backported to current supported Windows 10 versions.

A backport to Windows Server 2019 (based on 1809) was mentioned earlier.

WSL2 has been backported to 1903 and 1909. Also a welcome move since 1909 is one of the mostly used version in enterprise (which are very often more conservative and have ~ 1 year delays with the consumers deployment).

So can we hope of a backport of the support of nested virtualization for AMD cpu for 20H2 and maybe 1909 versions too?

Copper Contributor

Has there been any news on support for nested virtualization in KVM? I can't seem to find anything online.

Copper Contributor

Any more news on this? When will it be available for the outside of the Windows Insider ring? My development environment heavily depends on nested virtualization and I need to update my rig. This will be the main criteria for me to go with AMD or Intel. Has anybody tried it? I wondering whether it's stable or not.

Copper Contributor

Hello,

 

We need this on Windows Server 2019. Is there any news on the backport or an implementation for Windows Server 2022? It would seem this is one of the primary reasons why someone would choose Windows Server over other operating systems. Can you please give an update on this?

Copper Contributor

Are Linux KVM guest(s) supported now?

Copper Contributor

What I need is to run KVM in my WSL 2.0 and generally nested virtualization. I have too much time invested in setting up this Windows machine (dev+ test rig + occasional gaming rig) and now, to switch to post 2020H2 release on an unstable branch - is not possible. If i decide to do that I will jump to all linux first option. I have 2+ VMs + WSL.2.0 + Docker running 24/7.

I understand it is my fault, as I have not checked if this would fully work - It was unconceivable to me, that there would be issues, where AMD historically supported virtualization in all their CPUs since forever. 

Could you please let us know if 20h2 has any near future plans to receive these features?
Running Win 10 PRO @ Ryzen 5950X 128GB ram and I still cannot utilize the machine to its full potential.

Brass Contributor

Hiya, still waiting for this to hit live. otherwise need to use proxmox with vfio as a base OS.

Brass Contributor

@chuybregts can someone tell us what is going on? I have checked today and the 21H1 Windows 10 version will still not have this functionality. Is Intel paying you guys to delay that?? I am asking this because this is very fishy that such an important feature that holds back a lot of people of using one specific CPU vendor is constantly being delayed and not even back ported or giving any ETA.

 

It is almost a year from introducing that feature and still nothing. I think that we as customers will have to inform some legal authorities to investigate this, because this looks to be really anti-consumer.

Copper Contributor

I've waited half a year in the hope 21H1 will merge this feature...

Now I have no choice but to use insider DEV insider preview - in the current version 21313

 

So far it seems to be stable enough. I hope it stays that way.

 

 

 

Brass Contributor

yes. microsoft this is making me sad. dev builds are getting worse and buggy. I don’t want to deal with insiders bugs, I just want nested and in my wsl2. why cant it be backported?

 

it could be an advanced feature or a flag (powershell is already required for VM version 9.3) and wsl2 required a custom kernel build. You do not expect this to be a mainline feature if you are not fully happy with it yet.

 

 

but to force a whole army of developers using and cpu's to run a buggy insiders build (which you recently broke) is not fair.

 

also the changes to insiders mean dev branch is more often broken then not, what even is the point of the other two branches, why not merge the other two insiders branches?

 

 

anyway that is a point for the future.

Copper Contributor

I still have to use insider builds for my development purposes. @chuybregts when It will be available on RTM build?

Copper Contributor

Hey

Has anyone gotten the chance to test this on the latest Windows Server 2022 release?

Brass Contributor

Still no update. This was posted nearly 1 year ago and no update since about rtm/ regular windows support!!!

Copper Contributor

@Ameeno as I understand it will be in 21H2.

Copper Contributor

@proboszcz  @GuillaumeIT64 im running server 2022 in a hyper-v with windows enterprise 20H2 currently and no it does not work will update to the insider fast track and see if that works but out of the box and in my case no not working will update though.

Copper Contributor

Will Windows 10 client get nested virtualization support for AMD processors? @chuybregts 

Copper Contributor

**Update:**

I was able to get nested virutalization working on server 2022 and 2019 build 20344  (both from enterprise 20344)  so its in progress ... however i could not get nanoserver docker image to load from server  hyper-v. (also insider)

Copper Contributor

@rayiik 

 

That's really great news, thanks for letting us know. That means that probably by September / October, we'll have plenty of choices for labs, like having semi-annual channels 2019 or 2022 as the main OS, or LTSC 2022 with or without Desktop Experience, or even Hyper-V 2022; this allows us to have other things than virtualization on the main OS, without having to sacrifice nested virtualization using AMD platform.

Copper Contributor

@GuillaumeIT64docker isnt an issue either (form linux here) was unaware that docker-desktop defaults to Linux containers and you have to switch it over to windows ( which explains the no linux image for nanoserver error)

Copper Contributor

I am so close to drop windows completely. just go full linux and KVM with GPU passthrough and issue-less Docker. Not only Docker networking is borked with WSL 2.0 (can't bridge), but also this nested virtualization was not shipped yet to beta channel. 

 

Oh Microsoft where are you headed? When do we get this Nested thingie to beta channel?

Copper Contributor

@rayiik thanks for this information as well :)

@CivFan I understand this can be frustrating for some usecases, where Linux might do the job, sometimes better. But there are usecases where Hyper-V nested is a non-negotiable requirement. Anyway, all in all, things are improving overall, and it's a great time for labs.

Copper Contributor

oh yea so to add to that i disabled wsl 2 because i couldn't update the kernel at the time so i dont know if the issue with wsl and docker is fixed i thought that was my problem originally so i disabled it then when i found the issue i never re enabled it so cant speak to that X) i main arch linux but doing 70-740 ... series as part of our college right now and i got given an almost impossible assignement thanks to my amd (with no warning from the college or comment on it lol)

Version history
Last update:
‎Jun 12 2020 05:26 PM
Updated by: