Intro

Microsoft's virtualization stack, which powers the Microsoft Azure Cloud, consists of the Microsoft hypervisor (MSHV) and the Azure Host OS. Security is foundational for the Azure Cloud, and Microsoft is committed to the highest levels of trust, transparency, and regulatory compliance (see Azure Security, Hypervisor Security). Confidential VMs are a new security offering that allow customers to protect their most sensitive data in use and during computation in the Azure Cloud. In this blog, we share how Microsoft built support for Confidential VMs in Azure. We give an overview of our goals and our design approach, detailing the steps we took to enable Confidential VMs to protect their memory, leverage Secure Boot, have their own dedicated vTPM, protect their runtime state, securely use emulated devices and verify their environment through remote attestation.

What is Confidential VM?

The Confidential Computing Consortium defines Confidential Computing as “the protection of data in use by performing computation in a hardware-based, attested Trusted Execution Environment (TEE)”, with three primary attributes for what constitutes a TEE: data integrity, data confidentiality, and code integrity¹. A Confidential VM is a VM that is executed inside such a TEE so “code and data within the entire VM image is protected from the hypervisor and the host OS”¹. As crazy as this sounds – a VM that runs protected from the virtualization stack, the underlying software that makes its very existence possible – a growing community is building a set of technologies to make this possible. Confidential VMs leverage confidential hardware platforms - the combination of the hardware and architecture specific firmware/software supplied by the hardware vendor.

For confidentiality, a Confidential VM requires CPU state protection and private memory to hold contents that cannot be seen in clear text by the virtualization stack. To achieve this, the hardware platform protects the VM’s CPU state and encrypts its private memory with a key unique to that VM. The platform further ensures that the VM’s encryption key remains a secret by storing it in a special register that is inaccessible to the virtualization stack. Finally, the platform ensures that VM private memory is never in clear text outside of the CPU complex, preventing certain physical attacks such as memory bus snooping, cold-boot attacks, etc. 

For integrity, a Confidential VM’s requires memory integrity protection to ensure its memory can only be modified by that VM. To achieve this, the hardware platform protects the VM’s memory contents against software-based integrity attacks, and it also verifies address translation to ensure that the address space layout (i.e., the memory view) of the VM can only be changed with the cooperation and agreement of the VM.

An overview of our approach to Confidential VMs

As a type 1 hypervisor, the MSHV runs directly on the hardware and all operating systems, including the host OS, run on top of it. The hypervisor virtualizes resources for guest VMs and controls capabilities that manage memory translations. The host OS provides functionality for VM virtualization including memory management (i.e., providing guests interfaces for accessing memory), and device virtualization (i.e., providing guests virtual devices) to run and manage guest VMs.

Since they take care of virtualizing and assigning resources to guest VMs, all virtualization stacks up until recently assumed full access to VM state, but we evolved Microsoft's virtualization stack to break those assumptions with Confidential VMs. We set a boundary to protect the guest from our virtualization stack, and we leverage different confidential hardware platforms to enforce this boundary and help guarantee the Confidential VM’s attributes. Azure Confidential VMs today leverage the following platforms: AMD SEV-SNP (generally available) and Intel TDX (in preview).

Due to the additional requirements the Confidential VM architecture introduces on this guest-host boundary, the software running inside of a Confidential VM (i.e., the guest OS) sees different interfaces for CPU, memory, and I/O virtualization. This means that guest operating systems must be modified, adopting a set of enlightenments to be able to run inside Confidential VMs. Because we wanted to enable customers to lift-and-shift workloads with little or no effort, one of our key design goals was to support running mostly unmodified guest operating systems inside Confidential VMs. Existing operating systems were not designed with confidentiality in mind, so we created the guest paravisor to bridge between the classic OS architecture and the need for confidential computing. The guest paravisor implements the TEE enlightenments on behalf of the guest OS, acting as a compatibility layer so the guest OS can run mostly unmodified, even across hardware platforms. One can think about it as a firmware layer that acts like a nested hypervisor. Note that an OS that is fully enlightened (modified) to run inside a Confidential VM can run without a guest paravisor in Azure, but that is out of scope for this blog.

The paravisor is a key concept in our overall design. To run the paravisor in a more privileged mode than the guest OS in a hardware platform-agnostic manner, we extended VBS - which lets a VM have multiple layers of software at different privilege levels - to have it leverage the hardware platform's confidential guest privilege levels. 

A Confidential VM turns the threat model for a VM upside-down: a core pillar of any virtualization stack is protecting the host from the guest, but with Confidential VMs, there is a new focus on protecting the guest from the host too. This means that guest-host interfaces must evolve to break many assumptions. This was another of the design principles for the paravisor, as it allowed us to move logic from the host into the guest to simplify the guest-host interfaces to be better able to protect the guest. As bugs are found, our paravisor is easier to update than N version of the Windows/Linux kernels.

Protecting Memory

Confidential VMs use two types of memory, private memory, where computation is done by default, and shared memory, which serves to communicate with the virtualization stack for any purpose, such as device IO. Private memory is integrity protected and encrypted, and this is guaranteed by the hardware. Confidential VM workloads must be configured appropriately to ensure that sensitive data that will go in shared memory (i.e., leave the Confidential VM) is protected as needed (via TLS, BitLocker, dm-crypt, disk integrity protection, etc.). 

Any page of memory can be private or shared, and we call this its page visibility. Since all accesses use private pages by default, when a guest wants to use a shared page, the guest must explicitly do this by managing page visibility. We evolved guest OS device drivers, both hardening them as well as enabling them to use our mechanism to manage page visibility. The mostly unmodified guest operating systems we support inside Confidential VMs have these changes which are mostly limited to device drivers and could be easily backported.

Our mechanism to manage page visibility has the guest maintain a pool of shared memory. When the guest wants to send data to the host, for example to send a data packet over the network, it allocates a bounce buffer from that pool and then copies the data from its private memory into the bounce buffer. The I/O operation is initiated against the bounce buffer, and the virtual device on the host reads the data in shared memory. Importantly, the size of the pool of shared memory is not fixed; our memory conversion model is dynamic. The memory used by the guest can be converted between private and shared as needed to support the IO flow so that the guest can adapt to the needs of the workloads. The confidential computing platform prevents the hypervisor or anything other than the code inside the Confidential VM from converting memory to make pages shared. Converting memory is therefore always initiated by the guest, and it kicks off both guest actions and host actions so that our virtualization stack can coordinate with the hardware platform to grant or deny the host access to the page.

In terms of hardening device drivers, we analyzed and updated them to take a defensive posture that examines and validates data received from the host (e.g., ring buffer messages) before consuming it. We also reduced the attack surface inside the guests by restricting the channels that are offered to the guest so that only the hardened channels are exposed to the guest, restricting protocol version(s) that will be negotiated with the host to only the latest version(s) that support confidential VM isolation and enabling the guest to control which guest pages are accessible to the host as read-only or read-write. For Linux guest operating systems, we collaborated with the Linux kernel community and with Linux distros such as Ubuntu and SUSE Linux. We continue to further evolve these interfaces to continuously improve the defensive model of Confidential VMs.

Secure Boot

A guest VM traditionally relies on the host OS to store and provide its firmware state and attributes (UEFI variables) as well as to implement authenticated UEFI variables to ensure that secure UEFI variables are isolated from the guest OS and cannot be modified without detection. We created a completely new schema for Confidential VMs. Firstly, we created a new VM Guest State (VMGS) file that allows a Confidential VM to persistently store and access guest state and guest secrets that are inaccessible to the Host OS, and we evolved guest firmware to get trusted UEFI attributes from this file (instead of from the host). This file is packaged as a VHD and can be encrypted before VM deployment, so the host only ever interacts with the encrypted VHD. 

Precisely because we run the guest paravisor in a more privileged mode than the guest OS (using the hardware platform's confidential guest privilege levels), both the guest OS and UEFI do not have access to secrets held in guest paravisor memory. This means the paravisor can provide Confidential VMs a critical privilege-separated feature: Secure Boot. To provide a Confidential VM authenticated UEFI variables that are secure from the host as well as from the guest OS, the guest paravisor (running in the guest VM but isolated from the guest OS), manages authenticated UEFI variables. When a Confidential VM in Azure uses UEFI runtime services to write a variable, the guest paravisor processes the authenticated variable rights and persists that data in the VMGS file. 

Isolating Secrets

Our paravisor model also allows us to provide Confidential VMs another critical privilege-separated feature: secrets isolation from a guest OS, keeping VM secrets isolated from the guest OS inside the VM and providing a mechanism by which the guest OS can interact with those secrets. There are two important considerations for secrets isolation in a Confidential VM: how the VM can securely communicate with the entity that is managing the secrets and multi-tenancy. Regarding the former, the Confidential VM can use its private memory and private register state to communicate with the paravisor (the communication channel with the paravisor is inherently trusted). Regarding the latter, there is a single instance of the paravisor for the Confidential VM, meaning there is a 1:1 mapping between the Confidential VM and the paravisor secrets management entity. A great example of secrets isolation is providing a vTPM to a guest OS; the paravisor provides a Confidential VM its own dedicated vTPM 2.0. The Confidential VM can then also use the VMGS file to persist those secrets (vTPM state). In conclusion, our model allows a Confidential VM to access and persistently store VM guest state and guest secrets (e.g., UEFI state and vTPM state) that are kept secure from the host as well as the guest OS.

Protecting Runtime State

Traditionally, the hypervisor emulates the interrupt controller (APIC) to generate and inject interrupts into the guest. A Confidential VM requires assurances about the integrity of its execution state, so the confidential hardware platform provides mechanisms to protect a Confidential VM from being vulnerable to unexpected interrupts or exceptions during its execution, including a new exception type that is generated only by the hardware and must be handled by the guest rather than the virtualization stack. The guest paravisor performs exception handling for the new confidential exception type and uses these mechanisms to perform interrupt proxying on behalf of the guest OS, validating interrupts coming from the hypervisor before re-injecting them into the guest OS.

Protecting Emulated Devices

For compatibility, the Azure Host OS traditionally provides emulated devices to closely mirror hardware, and our hypervisor reads and modifies a guest VM’s CPU state to emulate the exact behavior of the device. This allows a guest to use an unmodified device driver to interact with the emulated device. Since the CPU state of a Confidential VM is protected from our virtualization stack, Confidential VMs simply cannot use devices emulated by the host OS. Going back to our overarching goal to enable customers to run mostly unmodified guest operating systems inside Confidential VMs, we did not want to eliminate emulated devices. Therefore, we chose to evolve our virtualization stack to support device emulation operating inside the guest instead of the host OS, and we moved emulated devices to the guest paravisor.

Remote Attestation

An attestation report is composed of hardware and software claims about a system, and these claims can be validated by an attestation service. According to the industry definition of Confidential Computing, a TEE deployment “should provide a mechanism to allow validation of an assertion that it is running in a TEE instance” through “the validation of a hardware signed attestation report of the measurements of the TCB”¹. A Confidential VM cannot solely rely on the host to guarantee that it was launched on a confidential hardware platform and with the right virtual guest firmware, so a remote attestation service must be responsible for attesting to its launch.

A Confidential VM on Azure always validates that it was launched on a confidential hardware platform via remote attestation with a hardware root of trust, and it additionally validates its guest boot configuration thanks to its Secure Boot and vTPM capabilities. Once the partition for a Confidential VM is created, the hardware seals the partition from modification by the host OS, and a measurement about the launch context of the VM is provided by the hardware platform. The guest paravisor then boots first and performs attestation on behalf of the guest OS, requesting an attestation report from the hardware platform and sending it to an attestation service for evaluation. Any failures in this attestation verification process will result in the VM not booting. After this, the paravisor transfers control to UEFI. During this phase, Secure Boot verifies the startup components, checking their signatures before they are loaded. In addition to this, thanks to Measured boot, as these startup components are loaded, UEFI accumulates their measurements into the vTPM. If the guest OS disk is encrypted, the vTPM will only release the key to decrypt it if the VM’s firmware code and configuration, original boot sequence, and boot components are unaltered. 

Call to Action

In this blog we described how, in collaboration with our industry partners, we evolved our entire virtualization stack to empower customers to lift and shift their most sensitive Windows and Linux workloads into Confidential VMs. We also gave a technical overview of how we protect our customers’ workloads in these Confidential VMs. This is an innovative new area, and we want to share that journey with our customers as they move into this new Confidential Computing world. As you use Confidential VMs on Azure, we would love to hear about your usage experiences or any other feedback, especially as you think of other scenarios (in the enterprise or cloud).

- Core OS Team.

References:

¹Common Terminology for Confidential Computing, December 2022, Confidential Computing Consortium.