OpenHCL: the new, open source paravisor
Intro From the beginning of the cloud computing era, virtualization technology has enabled compute workloads to run as virtual machines (VMs) in a server environment. As hardware has evolved, and new functionality has become available, the software stack has kept VMs running seamlessly, thanks to sophisticated advances in the hypervisor and virtualization software. Confidential Computing is a powerful technology for significantly improving the security of VMs running in the cloud. However, the trust boundary of a Confidential VM imposes a barrier that prevents the hypervisor from offering the rich virtualization services that VMs normally expect. Customers desiring the benefits of Confidential VMs are therefore forced to update the operating systems of their VMs to newer versions, which must be continually revised as Confidential VM technology maintains its path of rapid evolution. Microsoft has embraced a different approach that offers much more flexibility to customers through the use of a “paravisor”. A paravisor executes within the confidential trust boundary and provides the virtualization and device services needed by a general-purpose operating system (OS), enabling existing VM workloads to execute securely without requiring continual update of the OS to take advantage of innovative advances in Confidential Computing. As Confidential Computing evolves, the software stack can keep VMs running seamlessly thanks to the paravisor, in much the same way other advances in virtualization software kept VMs running seamlessly on ever evolving hardware. Microsoft developed the first paravisor in the industry, and for years, we have been enhancing the paravisor offered to Azure customers. This effort now culminates in the release of a new, open source paravisor, called OpenHCL. We plan to develop OpenHCL in the open here: microsoft/openvmm: Home of OpenVMM and OpenHCL (github.com). OpenHCL Capabilities A paravisor is essentially an execution environment that runs within the guest VM - at a higher privilege level than the guest OS - and provides various services to the guest. A paravisor can run in both confidential environments and traditional (non-confidential) environments. We use VBS to run a paravisor on Microsoft’s virtualization stack. When running in a confidential environment, these privilege levels must be enforced by the confidential hardware platform, and our architecture allows VBS to be appropriately enforced in a hardware platform-agnostic manner. Today, OpenHCL can run on both x86-64 and ARM64 platforms, and it has support for Intel TDX and AMD SEV-SNP confidential hardware platforms. OpenHCL runs in the L1 VMM of a TDX Confidential VM and in the VMPL0 of an SEV-SNP Confidential VM. See the OpenHCL user guide for step-by-step instructions to use it. OpenHCL offers a rich set of powerful services to both Confidential and traditional VMs alike: Device emulation via standard device interfaces, essentially offering a set of emulated devices. Device translation via standard device interfaces, such as NVMe to para-virtualized SCSI, allowing assignment of hardware devices directly to VMs (accelerated IO) without requiring guest OS changes - enabling VMs to take advantage of the performance of cutting-edge devices. Diagnostics support (particularly useful for debugging Confidential VMs where it is difficult to use traditional debugging methods). (To Confidential VMs specifically) Support for guests that are not fully enlightened - such as Windows and older versions of Linux - to run on confidential hardware platforms via standard architectural interfaces. For Confidential VMs, even though OpenHCL provides value to guests that are not fully enlightened (by enabling them), OpenHCL can also provide a lot of value to fully enlightened guests by offering them any of its other services as different scenarios require it. OpenHCL is used in Azure in new Azure Boost SKUs, and it will be used in future Azure Confidential VM SKUs. In the past month alone, over 1.5 Million VMs were running with OpenHCL in Azure [1] . OpenHCL Architecture OpenHCL is composed of several open source components, the most important one being OpenVMM, the modular, cross-platform, virtual machine monitor (VMM) written in Rust. This VMM runs several user mode processes that power OpenHCL. Running a VMM inside OpenHCL allows us to support guests with assigned devices and provide device translation support. Additionally, it allows us to share confidential and non-confidential architecture. We run the same VMM in the same environment for both Confidential and traditional VMs, and the VMM provides the same services tailored to their requirements. This avoids fragmented virtualization solutions among Confidential and traditional VMs. The other components of OpenHCL are a boot loader and a small, customized Linux kernel built to support the VMM, with min. Kconfig to minimize binary size and runtime RAM usage. Running a kernel to support our environment allows the VMM code to be mostly standard Rust, making the VMM much more powerful by enabling it to use the broadly supported and stable Rust toolchains and crate ecosystem. The Two Approaches to Running Confidential VMs There are two approaches to running a guest OS inside a Confidential VM: either the guest must be fully enlightened (modified to understand and manage all aspects of running as a Confidential VM), or it can rely on a paravisor to implement the Confidential Computing enlightenments on its behalf. In Azure, all IaaS Confidential VMs leverage a paravisor today. The paravisor enabled Azure to support the widest variety of guests, including Windows versions released almost a decade ago [2] and Linux versions using kernels as old as the 5.19 kernel [3] (and versions using even older kernels that had a small set of patches backported, such as some Ubuntu and RHEL distro versions). This provides customers with an easier lift-and-shift as well as the flexibility to gain future Confidential Computing advances without needing to upgrade their workloads. Customers’ legacy solutions are safe with Azure because of the approach we embraced. Why is Windows not fully enlightened? When we developed support for the first Confidential VM in Azure on the confidential hardware platforms available at the time, it was not possible to fully enlighten Windows for those platforms because Windows required APIC emulation to be done in a paravisor. APIC emulation, traditionally done by the hypervisor, must be done by another entity for Confidential VMs, where the hypervisor is outside the trust boundary. It can be done by the paravisor or by the hardware platform if it supports APIC virtualization. Early confidential hardware platforms (like 3rd Gen AMD EPYC™ processors) did not support APIC virtualization, so APIC emulation had to be done in a paravisor for Windows guests. This had to be so for Windows guests, but not necessarily for Linux guests, because the architecture of Windows relies directly on the APIC for interrupt management whereas the architecture of Linux does not. Some aspects of Windows interrupt management do not flow through the kernel and are inlined in drivers, so Windows drivers rely on the interrupt management behavior offered by the APIC. Linux offers kernel service routines for handling interrupt state, so Linux drivers rely on these routines. In addition to this, Windows relies on the presence of a TPM for security features, and one cannot implement a vTPM for a Confidential VM with enlightenments alone. We chose to implement a vTPM in a paravisor. Given all the functionality we have built into the paravisor, our plan is not to fully enlighten Windows and continue supporting Windows guests via a paravisor in Azure. For future versions of Linux, we’re evaluating both approaches – fully enlightened and relying on a paravisor – and we will aim to do what is best for customers. OpenHCL and COCONUT-SVSM An SVSM like COCONUT-SVSM plays a very valuable role for Confidential Computing; it can store secrets and provide virtualization services to improve the usability of fully enlightened guests. OpenHCL solves a different problem than COCONUT-SVSM. COCONUT-SVSM aims to provide services to Confidential VMs with fully enlightened guests using new interfaces. OpenHCL aims to provide services to Confidential VMs using existing standard architectural interfaces. COCONUT-SVSM provides device emulation, but OpenHCL uniquely provides this via existing standard interfaces. When running with an SVSM (like COCONUT-SVSM), the guest must establish a specific relationship with the SVSM by discovering its presence and then interact with the SVSM using a custom calling convention. Essentially, a guest needs to be specifically modified to be able to take advantage of SVSM services, including devices. With OpenHCL, existing device interfaces just work, so the guest does not need any custom calling contract modifications to consume them. OpenHCL enables devices to be discovered over standard enumeration mechanisms, like PCI virtualization or existing vTPM device contracts. COCONUT-SVSM could potentially be leveraged by OpenHCL in the future. OpenHCL's VMM component is Rust based, which has strong memory safety properties; evolving OpenHCL's kernel component to also be Rust based would improve the memory safety of OpenHCL. During the development of OpenHCL, we chose the Linux kernel because it was a familiar OS platform for contributors and provided the capabilities needed. Now that Rust-based COCONUT-SVSM exists, we are interested in moving to that in the future and building OpenHCL support for it if it gains the features that OpenHCL needs. Open for Collaboration In this blog we described the existing capabilities and the value of OpenHCL for the future of computing, but we have much more we plan to do with OpenHCL. We track the roadmap of OpenHCL in the open; below are some of its future milestones. As we develop new functionality in the open, we would love to collaborate with you. Please reach out to us if you have ideas you’d like to add to the roadmap or any other feedback; you can open a GitHub issue and even contribute to the project! You can learn more about this project on: https://openvmm.dev. Milestone: OpenHCL support for Intel TDX (Trust Domain Extensions) in Azure Intel and Microsoft collaborated on and co-developed the TDX partitioning architecture so that it could be leveraged by a paravisor. The first ever TDX module with TD partitioning was an amazing co-engineering project between Intel and Microsoft. Intel released TD partitioning as part of the TDX Module that accompanied the general availability of 5th Generation Xeon, and this has also been backported to 4th Generation Xeon. Using this TDX module, Azure launched the first generation of Azure TDX Confidential VMs, being the first cloud service provider to offer TDX in public preview as well as the first cloud service provider to offer Windows guest support for TDX. Intel has been contributing to OpenHCL for the past 6+ months, and we’re close to feature completeness in OpenHCL for the next generation of Azure TDX Confidential VMs! Milestone: OpenHCL support for AMD SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging) in Azure Microsoft and AMD have a long history of collaboration with multiple Azure Confidential VM offerings, powered by AMD Infinity Guard's SEV-SNP. We have been engaging with AMD to ensure OpenHCL and AMD’s platform can best work together to provide great performance and security for customers in the future. Virtual Machine Privilege Levels (VMPLs) provide hardware isolated abstraction layers within a VM for additional security controls, as well as assistance with managing communication with the hypervisor. Microsoft used AMD’s VMPLs to build the first paravisor for Confidential VMs in Azure, and we continue to develop support in OpenHCL to reach feature completeness for future generations of AMD Azure SEV-SNP Confidential VMs. Milestone: OpenHCL support for Arm CCA (Confidential Compute Architecture) Arm Confidential Compute Architecture (CCA) comprises a collection of open source software, firmware, specifications, and hardware support to bring Confidential Computing to the Arm architecture. CCA provides protected environments called Realms that can be used to host Confidential VMs. Microsoft started engaging with Arm almost two years ago to make sure CCA is well equipped to support paravisor stacks like OpenHCL, and our collaboration lead to the creation of the Planes feature, which enables multiple of levels of privilege to coexist inside a Realm. Planes provide the ability to host a paravisor and a guest VM in the same Realm, with the paravisor providing security and compatibility services to the guest. We are excited to collaborate further with Arm to build OpenHCL support for CCA in the open. Milestone: OpenHCL support for KVM as host Today OpenHCL runs only on MSHV. We are looking forward to developing OpenHCL support for KVM as host in collaboration with other cloud service providers and the Linux and KVM community to enable others to leverage OpenHCL in their virtualization stacks. We recently began engaging with Red Hat to discuss the value of OpenHCL and how it has the potential to open the door for significant cross-OS interoperability in the confidential virtualization world. We are excited to collaborate with Red Hat to build an open and collaborative confidential computing ecosystem. - Core OS Team. [1] This number is from when this blog was published, Oct 2024, but this number keeps growing every month. [2] Specifically, older versions include Windows Client 10 (released almost a decade ago) and Windows Server 2019. [3] Specifically, older versions include Linux versions using the kernel 5.19 (for SNP) and 6.6 (for both SNP and TDX).The OpenVMM Project
What is OpenVMM? A virtualization platform comprises 3 primary general components: Firstly, there is the Hypervisor - privileged software that interfaces with the underlying hardware to provide low-level primitives for executing virtual machines (VMs). Secondly, there is the Virtual Machine Manager (VMM) -software that leverages hypervisor interfaces to provide the general capabilities of running VMs and managing their lifecycle. Lastly, there is the host OS on which the VMM runs, which provides networking and storage access. OpenVMM is an open-source, cross-platform VMM project written in Rust. The project is MIT Licensed and developed openly at microsoft/openvmm on GitHub. OpenVMM is used in the OpenHCL paravisor (an execution environment that runs in the guest VM), and it can also be used as a traditional VMM on a host OS. How has OpenVMM grown since it was open sourced? In late 2024, we announced the open sourcing of the OpenVMM project. Since then, our active developer community has grown to almost 100 contributors, with over 200 forks. The OpenVMM project has benefited from strong ongoing contributions from industry partners like Intel and Arm, which have helped the project evolve and mature in several aspects, including: Cross-platform virtualization Improvements include virt_mshv aarch64 support and virt_kvm correctness fixes. Device model and device emulation New support includes emulated SMMUv3 for aarch64 VMs, CXL emulation, PCIe NUMA and ACS configuration exposure, support for the modern VFIO cdev + iommufd device assignment interface (alongside the existing legacy group/container + Type1v2 path), and full netvsp VLAN support. Test infrastructure, CI, build and release engineering Improvements to the testing framework petri based on community feedback include physical NVMe test plumbing, CXL coverage, transparent-huge-page enablement, and richer halt-reason reporting for easier debugging. Improvements in CI and build engineering and new build support for Azure Linux. Confidential computing Thanks to contributions from Intel, we developed General Availability paravisor support for Intel TDX Confidential VMs in Azure in the open (over 50 GitHub issues), and then strongly focused on changes that have led to real, substantial performance improvements (such as reducing CVM performance overhead from about 3% to now 1% in CPU/memory intensive workloads). Thanks to contributions from Arm, support for an entirely new confidential architecture, Arm CCA Realm is underway; initial Arm CCA enablement for OpenHCL and TMK test infrastructure is complete! Thanks to the partnership with AMD, we continue enhancing paravisor performance and debugging for AMD SEV-SNP Confidential VMs with changes like Virtual NMI and openhcl_boot serial logging for SEV-SNP. We continue growing paravisor attestation and sealing capabilities such as adding support for stateless Confidential VM mode with hardware sealing policy. OpenVMM has also shown signs of broader ecosystem/product integration. Community contributors are exploring using OpenVMM code and artifacts in nearby efforts like WSL (WSL is exploring using OpenVMM as VMM backend option for WSL VMs), and some contributions are oriented toward downstream consumption, not just isolated repo improvements. As we develop new OpenVMM functionality in the open, we would love your feedback. You can learn more about this project on and how to use it on https://openvmm.dev. Why OpenVMM? The motivation behind building OpenVMM comes from three of its core characteristics; cross-platform, distributed and flexible, and its modern security architecture. Cross-platform There is a need for a VMM that works consistently across Windows and Linux and across multiple hypervisors. Broad cross-platform support is one of the design principles of, as well as one of the key motivations for developing OpenVMM. OpenVMM supports a variety of host operating systems including Linux, Windows, and macOS, and a variety of virtualization backends including MSHV, WHP, KVM and hypervisor.framework. It also supports both x64 and AArch64 architectures. Distributed and flexible OpenVMM is designed to be a collection of VMM components that can be combined to build specialized VMMs custom tailored to power specific scenarios. Although OpenVMM can run as a traditional VMM on the host, it is designed so that its runtime functionality can be split across multiple environments, like running in a paravisor inside a guest VM. In fact, our goal of running OpenVMM inside a paravisor environment, and this scenario’s unique threading model requirements, was one of the reasons OpenVMM was created. The ability to use OpenVMM in a variety of environments simultaneously is enabled by OpenVMM’s modular design and internal message-passing architecture. Components can be reused in different scenarios without being tied to the broader “block diagram”. OpenVMM is also flexible in terms of the wide range of scenarios in can support. It can optimize modern workloads while also supporting legacy workloads. For instance, today OpenVMM supports different boot modes: UEFI, BIOS and Linux Direct Boot, and it has wide device support across direct assigned devices, para-virtualized devices (both Virtio and VMBus), and emulated devices: vTPM, NVMe, serial UARTs and legacy x86 (i440BX + PIIX4 chipset, IDE HDD/Optical, Floppy, PCI, VGA graphics). In fact, Azure compatibility has also been a goal, which is why OpenVMM has support for the Microsoft Hypervisor ABI and all Hyper-V emulated and para-virtualized devices. Going back to OpenVMM’s modular design, one can choose a subset of the Hyper-V guest interface, or an alternate interface altogether, for new scenarios with different requirements. Modern security architecture OpenVMM aims to have a modern, secure design. It does this, firstly, by implementing as much virtualization functionality (emulation, instruction completion, paravirtualized devices, and memory management) in user mode as possible. This can improve security via hardware memory protection and enhanced exploit mitigations, and it also makes it easier to service the code, diagnose issues, and mitigate problems in the field, enabling faster patching of security vulnerabilities. Secondly, OpenVMM uses powerful process sandboxing, allowing unrelated VMM functionality to run in separate processes. On Windows, this sandboxing is backed by least-privileged app containers (LPAC), while on Linux it’s supported by kernel namespaces and (in the future) SeLinux. Lastly, like with many newer VMMs, OpenVMM is written in Rust, which reduces the opportunity for programming error leading to guest escape. With Safe Rust, programmers do not have to worry about type-safety or memory-safety... about the possibility of a dangling pointer, a use-after-free, or other kinds of Undefined Behaviors (which became known as nasal demons in 1992 for good reason). The OpenVMM community has been actively involved in Rust since the early days, driving the compiler and libraries forward. OpenVMM was one of the five case studies used by the Rust async working group to evaluate the utility of the stabilized MVP version of async functions in traits (OpenVMM is the Microsoft tool cited in Rust’s blog Stabilizing async in traits in 2023)! There are many other great security initiatives in Rust that OpenVMM benefits from: ongoing efforts are regularly creating new open-source security tools for Rust developers, conducting audits and threat models and developing a collaborative team of Rust security experts. Conclusion Today, we covered why the OpenVMM project started and how it has grown since it was open sourced. Microsoft is committed to working across open-source projects and supporting multiple VMMs for customers, alongside OpenVMM. Our goal is to give customers choices to find the solution that best meets their requirements based on their workloads and provide the performance and strong isolation capabilities of the Microsoft Hypervisor - which is used in Azure, on-prem, and Client scenarios - to a wide range of VMMs in various virtualization topologies and operating systems. For example, we are working on Cloud-hypervisor and Hyperlight which power Azure workloads, and we are also investing in broad compatibility, getting a wide range of VMMs including QEMU to run with the Microsoft Hypervisor to support diverse Linux workloads in Azure. - Core OS Team.
Announcing Trusted Launch for Virtual Machines for Windows Server Insiders
Trusted Launch for virtual machines We are excited to announce Trusted Launch for virtual machines (TVMs) in Windows Server Insider Preview. Trusted Launch is a security feature you can enable when creating Hyper-V Generation 2 VMs. It enables Secure Boot, installs a virtual Trusted Platform Module (vTPM), protects vTPM state at rest, and supports boot integrity verification (ability to verify if the VM started in a well-known good state). Further, when the VM runs in a Failover Cluster, the vTPM state is automatically made available when the VM live migrates or fails over to other nodes in the cluster – this ensures the VM remains available after migration or failover. This is unlike a Generation 2 VM with a vTPM, which will not start after migration or failover to another node in the cluster – the TPM state protection key needs to be moved to the destination node manually so the VM can start. With boot integrity verification, the entire boot path is measured and boot integrity is verified by Microsoft Azure Attestation service. This helps detect any alterations to the boot path or boot components. Such alterations, e.g. implanting malware in the boot path, can be detected by boot integrity verification. Increasingly attackers prefer implanting malware in the boot path for a variety of reasons: the OS layer is usually well protected, while firmware – as highly privileged code – can be used to alter what gets loaded (boot loader and drivers). Such alterations are not easily detectable by anti-virus software running at the OS layer. Boot integrity verification helps detect such alterations so a relying party (such as an app or service) can take suitable remediation actions, e.g. shutting down the VM. Boot integrity verification is an important part of establishing trust by verifying that the virtual machine started in a well-known good state. Insider preview TVMs are available for preview starting with Windows Server Insider preview build 29621. This initial preview only supports some of the Trusted Launch capabilities: Secure boot, vTPM, and vTPM state protection (at rest). You can create and manage TVMs using PowerShell. Guest state protection: The guest state (including the vTPM state) for each TVM is protected using a unique key that is stored in a KSP (Key Storage Provider) local to the server. Without this key, the VM will not start. Moving the VM to another server is not supported in this release. Not supported in this release: Moving TVMs to another server. TVMs in Failover Clusters or Hyper-V Replica. Boot integrity verification. Support for TVMs in Windows Admin Center (WAC). Instructions At a high-level, the steps involve: Install Windows Server Insider preview build on your server Enable Hyper-V Enable Trusted Launch feature Verify guest state protection 1. Install Windows Server Insider preview build Trusted Launch for virtual machines is available starting with the Windows Insider preview build number 29621. Install this build or a later build on your server. (Join Windows Server Insiders if you haven’t already!) 2. Enable Hyper-V (if it is not already enabled) Install-WindowsFeature -Name Hyper-V -IncludeManagementTools -Restart After enabling Hyper-V, the server needs to be restarted. 3. Set registry key property value (required to enable the Trusted Launch feature) New-Item -Path "HKLM:\SOFTWARE\Microsoft\AszIgvmAgent" -Force New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\AszIgvmAgent" -Name "TvmWinServer" -Value 1 -PropertyType DWord -Force Setting the above regkey informs relevant system components that the Trusted Launch feature is being used in a Windows Server environment. 4. Enable Trusted Launch feature Enable-WindowsOptionalFeature -Online -FeatureName "IsolatedGuestVm" -NoRestart 5. Verify if IgvmAgent is running IgvmAgent (Isolated Guest Virtual Machine agent) is a system-level service that helps support Trusted Launch capabilities. Get-Service -Name "IGVmAgent" The output should show Status as Running. If the Status is Running, you can skip to next step. If the Status is not Running, please report the issue. Add the following event logs to the report: Event Viewer: Applications and Services Logs => Microsoft => Windows => IGVmAgent => Operational Event Viewer: Applications and Services Logs => Microsoft => Windows => IGVmSystem => Operational You can open Event Viewer via the Run dialog: Press Win + R → type "eventvwr.msc" → press Enter 6. Create an external virtual switch (if you don't already have one you can use) Create and configure a virtual switch with Hyper-V | Microsoft Learn To see available external virtual switches: (Get-VMSwitch | Where-Object { $_.SwitchType -eq "External" }).Name 7. Create TVM If you already have a virtual hard disk (VHD or VHDX) for a Gen 2 VM with an installed guest OS, run: New-VM -Name <VMName> -Generation 2 -GuestStateIsolationType TrustedLaunch -SwitchName <virtual switch name> -VHDPath <path to vhdx> -Path <path to where VM config files will be stored> Else, run: New-VM -Name <VMName> -SwitchName <virtual switch name> -NewVHDPath <path to where new VHD will be stored> -NewVHDSizeBytes 40GB -Generation 2 -GuestStateIsolationType TrustedLaunch -Path <path to where VM config files will be stored> Then, add to the VM a virtual DVD drive containing an ISO image for the guest OS (Windows or Linux OS-compatible with Hyper-V Gen 2 virtual machine). Add-VMDvdDrive -VMName <VMName> -Path <Guest OS ISO image path> Note: The guest OS will be installed when the VM starts up. When connecting to the VM you will be prompted to install the guest OS from the DVD drive. Make sure that the DVD drive is at the top of the boot order specified in the firmware so the VM will boot from the DVD drive. For more information, see New-VM. 8. Verify VM guest state isolation type (Get-VM -name <VMName>).GuestStateIsolationType should return "TrustedLaunch". 9. Verify guest state protection To verify guest state protection, stop IGVmAgent service and restart the VM. Without IGVmAgent in Running state, a TVM with guest state protection will not start. Call to action Trusted Launch brings foundational VM security — Secure Boot, a vTPM, and protected guest state — to Windows Server, helping safeguard your VMs against boot-level and firmware threats. Please try out TVMs and provide your feedback via the Windows Server Insiders Forum. — Christina Curlette and Ram Jeyaraman (and the Windows Server team)
Feature Request: More Cursor/Pointer Roles
We need a feature that makes us customize the cursor roles (below)! Because Linux has more cursor roles, we cannot customize them in Windows. Cursor Roles Suggested: No drop - arrow with a no sign overlay Column Resize - horizontal resize but for grid resizing Row Resize - vertical resize but for grid resizing Vertical text - I-beam rotated 90 degrees Grab - open hand Grabbing - closed hand Cell - bold plus sign Copy - arrow with a plus overlay Alias - arrow with an iron chain overlay Zoom In - magnifying glass with plus sign overlay Zoom Out - magnifying glass with minus sign overlay Tags/Keywords: pointer, cursor, mouse, customizable, custom
