Windows OS Platform Blog articles

Windows on Arm runs more apps and games with new Prism update

Marc_Sweetgall — Fri, 05 Dec 2025 22:29:08 GMT

Today, Windows on Arm devices, such as those powered by the Snapdragon X series processors, are getting a boost thanks to an update to Prism that increases performance and enables running more apps and games under emulation.

The Prism emulator is a core part of the Windows on Arm platform, ensuring that Windows apps are able to run seamlessly on Windows on Arm even if they were built for traditional x86 processors and have not been updated to run natively on Arm. Prism works by transparently converting an app's x86 instructions to Arm64 code as the app runs.

With today's update, Prism expands its capability by supporting translation of more x86 instruction set architecture extensions, including AVX and AVX2, as well as related extensions BMI, FMA, F16C, and others. These extensions are not guaranteed to be supported by a processor when running Windows, but they are common enough for some apps to depend on their availability. They are especially relied on by creative tools and games that benefit from the efficient parallel processing that these CPU instructions provide.

Features marked above in green are some of the emulated CPU features newly supported in Prism, as viewed withcoreinfo64.

With support for these additional processor features, Prism can now run a number of apps and games that were previously unable to work on Windows on Arm. One example is Ableton Live, which will be coming to Windows on Arm as a native app next year. Attempting to install Ableton Live 12 as an emulated app prior to this update would fail with a system requirements error because it requires AVX support. After the Prism update, the app installs and runs without issue.

Comparison of installing Ableton Live 12 on Windows on Arm before (left) and after (right) the update to Prism.

This Prism update has now rolled out to all Windows on Arm devices running Windows 11, version 24H2 or later. All 64-bit x86 apps will have the updated CPU feature support enabled by default. Any 32-bit apps will have it off by default but can be opted in using compatibility settings.

For more information about Windows on Arm, you can view our docs at Windows on Arm documentation on Microsoft Learn.

We are committed to continuing to enhance Prism emulation in Windows and are excited about the additional apps and games that will be able to run seamlessly on Windows on Arm with today's update.

Mobile Plans moves to the web

HunterM — Thu, 28 Aug 2025 20:17:26 GMT

Windows is retiring the built-in Mobile Plans app to simplify how you connect your PC to mobile data. Instead of using Mobile Plans app to buy or manage cellular plans, you’ll use your web browser and the Windows Settings app going forward. This change means a more integrated experience: no extra app installations, just a direct link between Windows and your mobile operator’s website. In this post, we’ll outline why this change is happening, what the new experience looks like, and how it benefits both consumers and mobile operators.

New experience in Settings to share device IDs with operator websites

A more streamlined web-based experience

Direct purchase on operator websites: Instead of launching an app, you’ll purchase and activate your cellular data plan directly on your mobile operator’s website. This change to a web-centric and operator-driven model better aligns with familiar experiences on other platforms. From Windows, when you want to add a mobile plan, you’ll navigate to your carrier’s web portal in your browser. Each operator will handle their own sign-up and payment flow.
No separate app needed: Windows 11 has new built-in functionality to make this web-based activation seamless, meaning one less app installed on your PC. When you purchase a plan on the site of participating carriers, Windows might prompt you via the Settings app to share your device’s cellular identifiers (like EID, IMEI) with the operator. With your consent, these details are securely passed to the carrier, so they can automatically provision your eSIM without you needing to type in codes or scan QR images. You can then download and start using cellular data right away. By using industry-standard web flows, HTTPS, and confirmation steps, this system remains streamlined and secure.

Confirmation in Settings before downloading a cellular profile

Timeline of the transition

Windows already supports activating eSIM using the web, via QR codes, and manual entry. The new experience to share your device’s cellular identifiers is available for Windows Insiders and will release publicly in the last half of 2025. Mobile operators will be adding support throughout the next year. The

You can continue using the Mobile Plans app until February 27, 2026. After that date, the app will be retired and you may uninstall it, and references to the app in Windows will be removed. If you face issues with this transition, please contact your mobile operator or visit their website to buy and manage eSIM data plans for your PC.

What it means for users

For most Windows users, this change should be convenient: connecting your device to a mobile network should be as easy as buying something online. If you already have an active mobile plan, you don’t need to take any action. Here are the key impacts:

No loss of cellular functionality: Existing cellular features on Windows remain intact. Any eSIM profiles or data plans you’ve already activated on your PC will continue to work normally. Any plans you purchased through Mobile Plans will continue working, but you’ll need to go to the operator’s website to manage them. Other ways of activating eSIM (like scanning a QR code from a carrier or manually entering activation codes) will continue to be supported just as before.
Mobile Plans app will be going away: You will see a message within the app about the end of support date. After that date, the app will be retired and may be uninstalled. The app will be removed from the Microsoft Store, and any links to open the app from within Windows will be removed.
Seamless user experience: If you have a laptop or tablet with LTE/5G and eSIM support, you’ll no longer need the Mobile Plans app. Instead, you can go directly to your carrier’s online sign-up page and then follow the Windows Settings prompts to install the eSIM profile and get connected.
Where to get and manage plans now: After the transition, to sign up for a new cellular plan on your PC, directly visit your mobile operator’s website and look for their section on activating an eSIM for Windows devices. After the transition, documentation will be updated to guide you through the new flow.

What it means for mobile operators

Microsoft has reached out to mobile operators participating in the Mobile Plans app, providing them the necessary details to transition to this new model.

Operator enablement: Carriers are adapting their systems to support eSIM activation for Windows PCs via web. This involves adding an option on their websites to initiate the Windows activation flow and handle the secure sharing of device identifiers and eSIM profiles. Microsoft is providing technical guidance so that each operator’s implementation meets the necessary criteria for a seamless user experience. Information on how to download eSIM profile is available here: Use a QR code or URI link to download an eSIM profile.
Trials and feedback: Starting in June 2025, selected operator partners began trialing the new flow with Microsoft. These trials allow operators to test the end-to-end process (from website to Windows device) and ensure any issues are ironed out before broad launch. All mobile operators with Windows data plan offerings are encouraged to participate in testing so that they’re ready by the time the app is retired. Please reach out to your local Microsoft representative with questions.
Removal from Mobile Operator Portal and COSA profiles: Following the retirement of the app, the Mobile Operator Portal will be updated to remove "Mobile Plans" as an option when creating a new draft. The COSA definition for enabling GetBalance will also be removed from all the provider profiles. Only the two entries “SupportDataMarketPlace” and “MobilePlansIdentifier” will be removed.
Updating “View My Account” links: In the current Windows UI, some carriers integrated with Mobile Plans have a “View my account” link in the network settings or Quick Settings. Those links used to point to the Mobile Plans app. Going forward, those need to point to the carrier’s own account management webpage. Operators should submit updated configurations (via COSA, the provisioning database) to ensure their customers can easily click from Windows UI to the correct web page for account info. More information is available here: Microsoft Mobile Operator Configuration Portal Guide.
Continued collaboration: This change enables the operators to have more control when building and providing a great activation experience. Microsoft will continue to work in partnership with mobile operators to ensure a seamless transition.

Next steps

The retirement of the Mobile Plans app is a move toward a simpler, web-powered, and more streamlined future for Windows connectivity. For users, it means one less app and an easier way to get your device online. For operators, it gives them direct control of the customer purchase experience.

Over the coming months, Microsoft will roll out the necessary Windows updates and work with carriers to finalize the new system. Keep an eye on the official Windows release notes and your carrier’s communications for announcements of support for the new eSIM activation flow.

In the meantime, if you’re a user looking to add cellular service to your Windows PC, you can continue to use the Mobile Plans app until it’s retired or check your operator’s website for information. Many operators already allow eSIM activation via QR code or manual entry, which is what the new flow streamlines.

We’re confident that moving to a web-centric solution will provide a smoother, more consistent connectivity experience for everyone. Thank you for being part of this journey to simplify Windows networking!

Additional information

Microsoft Mobile Operator Configuration Portal Guide

Use a QR code or URI link to download an eSIM profile

Cutting the wire without cutting the audio quality

mikeajax — Wed, 27 Aug 2025 15:21:15 GMT

I’m playing Forza Horizon 5 when I get an Xbox party chat invite from my friend who wants to race in multiplayer. As soon as I join the chat, the audio quality of my Bluetooth headphones drops. The sound gets muffled, and I can no longer hear what direction the sound effects are coming from – what’s going on?

This drop in audio quality happens because of how Bluetooth Classic Audio is designed. There are two modes of operation, or profiles: the Advanced Audio Distribution Profile (A2DP), and the Hands-Free Profile (HFP). A2DP provides high quality audio but doesn’t allow the microphone to be used. HFP allows the microphone to be used but plays back audio in mono with lower fidelity than A2DP. With mono audio, there is only one channel of audio that is played in both ears, compared to stereo where there is a separate channel for each ear.

This means that when using a Bluetooth Classic Audio device’s microphone, you can’t hear high-fidelity stereo playback from that device at the same time. Without stereo audio, you don't hear the spatial effects that provide immersive sound in games, movies and music, and you can't use features like Spatial Audio calling in apps like Microsoft Teams.

Voice limitations with Bluetooth Classic Audio

Why does this limitation exist? It’s right in the name – Bluetooth Classic Audio – while it is a tried-and-true technology, it was designed over two decades ago. When the state-of-the-art was flip phones and bulky earpieces, the design decisions made at the time for HFP were hardly seen as limitations.

HFP has improved over time. For example, the original audio compression algorithm was rudimentary and only supported an 8kHz sample rate. This means that nearly all treble frequencies were missing, leading to muffled audio that sounded a little worse than an AM radio. Windows 11 and most modern Bluetooth headsets now support improved audio compression and double the sample rate for HFP, which is called “wideband” voice.

Even with these improvements to HFP, some early design decisions persisted, most notably the lack of stereo support. So even with wideband voice, the audio quality is still closer to an AM radio than it is to the CD quality audio that Bluetooth provides when not using the microphone.

Introducing super wideband stereo for LE Audio voice

LE Audio is a modern Bluetooth audio streaming architecture that’s built on top of the battery-friendly Bluetooth Low Energy radio. It replaces A2DP and HFP with flexible new profiles that use a single profile for both media playback and voice. For headphones and earbuds, the Telephony and Media Profile (TMAP) is used, and for assistive hearing devices like hearing aids and cochlear implants, the Hearing Access Profile (HAP) is used.

LE Audio uses improved audio compression and TMAP requires headphones and earbuds to support even higher “super wideband” audio fidelity at a 32kHz sample rate while using voice. The new compression algorithm provides higher quality audio, and the higher sample rate covers nearly the entire audible audio range. Goodbye, muffled audio!

Today, we’re excited to announce a new feature in Windows 11 that allows Bluetooth Low Energy (LE) Audio devices to play in high quality stereo while using the microphone, for a drastically improved audio experience in games and calls! Until now, LE Audio on Windows 11 was limited to playing super wideband audio in mono while using voice.

Immersive game chat

When using an LE Audio device with a Windows 11 PC that supports super wideband stereo, the switch into game chat no longer causes an abrupt drop in audio quality. In the Forza Horizon 5 example mentioned at the top of this blog post, the game audio will now stay in stereo and stream at super wideband quality, which is a significant improvement over Bluetooth Classic!

It’s easiest to listen yourself – put on a pair of headphones and listen to the following video comparing wideband mono over Bluetooth Classic Audio to super wideband stereo over Bluetooth LE Audio:

Audio stays crisp and clear while in game chat, and you hear the engine sound effects fading between your left and right ears as the cars zoom around the screen.

This simulated demo video uses additional audio and video compression for online streaming, and is intended to illustrate the compression algorithms used for Bluetooth Classic Audio and Bluetooth LE Audio. It may not perfectly represent the audio quality that is heard from a real PC or audio device.

Super wideband stereo is quite literally a game changer, but we won’t be stopping there. We’re continuing to work with our PC and silicon partners, and with Bluetooth LE Audio device makers, to push the boundaries of audio quality further. In a future update to Windows, we plan to bring CD-quality audio playback to game chat and voice calls, just like when not using the microphone!

Spatial Audio in Teams calls

In addition to gaming, this feature improves calls in apps like Microsoft Teams. Teams Spatial Audio is already available for wired headsets, but it depends on stereo audio, so it hasn’t previously been available over Bluetooth.

Think about when you’re in a crowded environment like a restaurant. Other people nearby might be having their own conversations, but you’re able to easily tune them out and focus on your conversation. This is because the brain is good at ignoring sounds that aren’t coming from where you want to listen – a phenomenon called the Cocktail party effect.

Spatial Audio in Teams adds these spatial cues so the voice of each person comes from where their video is located on your screen, just like when sitting around a real table. It makes calls sound more natural and helps you focus on who you want to hear, even if two people speak at the same time.

Thanks to super wideband stereo support, Spatial Audio will be available over Bluetooth LE Audio for the first time! Look for the Spatial Audio toggle under audio settings in Teams while using an LE Audio headset or assistive hearing device with your compatible Windows 11 PC.

To learn more about Spatial Audio in Microsoft Teams and watch a demo video about it, visit the announcement blog post at Follow conversations with ease using Spatial Audio in Microsoft Teams.

How do I get it?

First, you’ll need a Bluetooth headset, earbuds, or assistive hearing device that supports Bluetooth LE Audio, along with a Windows 11 PC that supports LE Audio. Many new PCs and Bluetooth audio devices support LE Audio, but not all PCs that support Bluetooth LE technology also support LE Audio.

For more information about LE Audio support on Windows 11, visit Check if a Windows 11 device supports Bluetooth Low Energy Audio or consult the manufacturer of your PC or Bluetooth audio device.

To use super wideband stereo during game chat and voice calls, you’ll also need to be running the latest updates to Windows 11, version 24H2, and have an updated Bluetooth audio driver from your PC’s manufacturer. Starting later this year, some existing PC models will receive driver updates from the manufacturer to support the feature. We also expect that most new mobile PCs that launch starting in late 2025 will have support from the factory.

Resources for developers

For developers of Windows PCs and Bluetooth hardware, see Super wideband stereo for voice scenarios. For developers of communication applications wanting to detect support for super wideband stereo audio, see Detect audio format capabilities for communications scenarios.

The Bluetooth® word mark and logos are registered trademarks owned by Bluetooth SIG, Inc. and any use of such marks by Microsoft is under license. Other trademarks and trade names are those of their respective owners.

Protecting linear address translations with Hypervisor-enforced Paging Translation (HVPT)

Hilal_Asmat — Thu, 17 Apr 2025 21:18:31 GMT

Microsoft is committed to continually raising our security bar. Windows 11 and Windows Server 2025 are built with layers of defense, helping ensure we are secure by default and by design. Across the board, we have been innovating and integrating security into the fabric of the features we build, from chip-to-cloud, and it’s paid off: out-of-the-box security features in Windows 11 led to a reported 62% drop in security incidents¹. One of our latest silicon-assisted security features that went live recently is called Hypervisor-enforced Paging Translation (HVPT). HVPT is an innovative new feature that leverages the power of Windows Hyper-V to help protect guest virtual address (GVA) to guest physical address (GPA) translation with the goal of protecting against 2 page-table-targeted attacks, known as aliasing and remapping attacks. HVPT can help protect the root OS as well as guest VMs running atop it. In this blog, we will cover the attacks HVPT is designed to help protect against, and dive into how HVPT helps protects GVA to GPA translations.

Attacks on GVAs

Helping protect GVA to GPA translations provides massive security benefit by making GVA addresses tamper-proof. HVPT allows us to help protect critical system structures that make up some of our key security features like Kernel Data Protection, Shadow Stacks and Control Flow Guard. These features are hardened with HVPT, helping ensure that the protections can’t be bypassed by tampering with guest address translations, as we’ve seen attackers do. As mentioned, HVPT helps protect against remapping and aliasing attacks. Here's a brief explanation of how each of these attacks works.

Here is how a remapping attack can be carried out:

Fig 1: A remapping attack explained. An attacker tampers with the CR3-rooted page tables to remap a GVA to a target GPA where it has placed malicious content.

If the attacker exploits a write-what-where vulnerability and manages to write to memory, the attacker can then simply remap the GVA translation to point to the target GPA, by tampering with the CR3-rooted page tables. This attack can map protected software to point to malicious content, effectively changing the content of a GVA.

Another type of page table attack HVPT protects against is aliasing attacks. Here is what an aliasing attack looks like:

Fig 2: An aliasing attack explained. An attacker tampers with the CR3-rooted page tables to have a its GVA point to a target GPA.

In this attack, the CR3-rooted page tables are tampered with once again, to map a malicious GVA to an already-mapped GPA.

Introducing HVPT

Now that we understand the attacks that arise in the case of an unprotected GVA to GPA translation, let’s dive deeper into HVPT.

HVPT is really a collection of changes to the SLAT and OS, all working together to achieve the end goal of helping protect GVA to GPA translations:

Hypervisor-secured paging structures to disallow unauthorized modifications: HVPT adds brand new, secure-kernel-managed paging structures to the OS. These paging structures are physically mapped as read-only to the NT kernel but read/write to the secure kernel.

Fig 3: A diagram explaining the HVPT structures and how they are read-only to normal kernel mode, bur read-write to secure kernel.

Paging Write (PW) bit in the SLAT to allow processor A/D read/writes: The HVPT table is protected by the SLAT and marked read-only to the OS. However, to allow the CPU to be able to designate these pages as accessed/modified, there is a new bit called the “Paging Write” bit added to the SLAT, and set to 1 for page tables corresponding to HVPT structures. This allows for access/dirty bit writes for these structures.

Guest Paging Verification (GPV) bit in the SLAT to help validate secure page walks: To protect against aliasing attacks, we should enable the path taken to arrive at the final system physical address (SPA) was not tampered with. When the GPV bit is set for a GPA translation, the processor enables the paging structures used for the GVA to GPA translation have the PW bit set, signifying that the translation happened via the HVPT tables in SK. This is supported by Intel’s underlying HLAT feature, and will be supported by Windows in the future.

HLAT Prefix Size: The HLAT Prefix Size is a new field added to the Virtual Machine Control Structure (VMCS) that defines the range of addresses meant to be protected by HVPT. Checking the HLAT Prefix Size is the first step in the translation process. The HLAT Prefix Size acts as a filter to determine whether an address should be translated via HVPT- or CR3-rooted page tables.

HLAT Pointer (HLATP): The address of the HVPT paging structures is stored in a new field in the VMCS called the HLATP. This field is used to begin HVPT-based translations once an address is determined to be HVPT-protected via the PLR.

Restart bit: The HVPT paging structures include a restart bit. When this bit is encountered during a page-walk, the walk is stopped and restarted from the CR3-rooted page tables. This allows for performant translations in the case that an address is not fully translated via HVPT tables.

Let’s put all these pieces together and walk through how an HVPT address is translated and protected.

Fig 4: A diagram explaining how page translations happen with HVPT enabled. First, the address is checked against the PLR to determine if the address is protected with HVPT, in which case the translation starts in the HVPT tables (the address of which is stored in HLATP). Next, the translation of the GVA to GPA happens via the HVPT tables. The PW bit is set for each accessed HVPT PTE in the SLAT. The GPV bit is set on the SLAT PTE corresponding to the leaf-level HVPT PTE. If at any point in the walk a Restart bit is encountered, the walk aborts to restart at the CRW-rooted page tables.

HVPT helps eliminate remapping and aliasing attacks on GVAs. In both attacks, the attacker modifying the CR3 tables is ineffective, because the translation does not utilize those tables for HVPT-protected addresses, and the HVPT tables are protected by the hypervisor.

HVPT is a huge leap forward in securing the OS from exploits. We’re excited to add HVPT to the list of features making Windows secure-by-default.

HVPT is live today and on-by-default on Windows 11 24H2 (host) and Windows Server 2025 (host). HVPT requires Intel Alderlake+ vPro-enabled hardware, and for VBS and HVCI to be enabled. We urge all customers and developers to update to the latest version of Windows and Windows Server to benefit from the enhanced security provided by HVPT.

If you want to learn more about HVPT, you can reference the following:

References

Windows 11 security book - Windows security book introduction | Microsoft Learn

Securing the Present, Innovating for the Future

Security is a shared responsibility. Through collaboration across hardware and software ecosystems, we can build more resilient systems secure by design and by default, from Windows to the cloud, enabling trust at every layer of the digital experience.

The updated Windows Security book is available to help you understand how to stay secure with Windows. Learn more about Windows 11 and Copilot+ PCs.

To learn more about Microsoft Security solutions, visit our website. Bookmark the Security blog to keep up with our expert coverage on security matters. Also, follow us on LinkedIn (Microsoft Security) and X (@MSFTSecurity) for the latest news and updates on cybersecurity.

- The Core OS team

CPU oversubscription and new CPU jitter counters in Windows Server 2025

MarkKilimov — Mon, 24 Mar 2025 21:30:24 GMT

What is CPU oversubscription?

CPU oversubscription occurs when more virtual processors (VPs) are allocated to virtual machines (VMs) than there are logical processors (LPs) available on the host machine. This technique is commonly used in virtualized environments to maximize resource utilization, improve cost efficiency, reduce data centers energy consumption and environmental impact. While this approach maximizes resource efficiency, it also introduces challenges related to balancing of compute resources and ensuring consistent performance for critical workloads which can be understood and mitigated through robust monitoring strategies.

Introducing new CPU jitter counters in Windows Server 2025

To address the challenges posed by CPU oversubscription, Windows Server 2025 introduces new CPU jitter counters. These counters are designed to monitor and quantify the variability in CPU processing times, providing valuable insights into the performance of virtualized environments.

CPU jitter refers to the variation in the time it takes for a CPU to process a given workload. This variability can be caused by several factors, including CPU contention, interrupt handling, and other system activities. The new CPU jitter counters in Windows Server 2025 measure this variability, offering a detailed view of performance fluctuations.

\Hyper-V Hypervisor Virtual Processor(*)\CPU Wake Up Time Per Dispatch

\Hyper-V Hypervisor Root Virtual Processor(*)\CPU Wake Up Time Per Dispatch

Wakeup delay: The average amount of time a given VP had to wait for an idle LP to wake up and start running that VP when there is no contention for that LP. This counter estimated the amount of delay incurred by a VP when the system is underutilized.

\Hyper-V Hypervisor Virtual Processor(*)\CPU Contention Time Per Dispatch

\Hyper-V Hypervisor Root Virtual Processor(*)\CPU Contention Time Per Dispatch

Contention delay: The average amount of time a given VP had to spend waiting for an LP to run on that is attributed to VP contention, e.g. having to wait for other VPs to finish running first. This counter estimated the amount of delay incurred by a VP when the system is fully utilized.

Benefits of using CPU jitter counters

The introduction of CPU jitter counters in Windows Server 2025 offers several key benefits:

Improved performance management: By identifying the sources of CPU jitter, administrators can optimize resource allocation and reduce contention, leading to more stable and predictable performance.
Enhanced troubleshooting: Detailed insights into CPU processing times enable quicker identification and resolution of performance bottlenecks and anomalies.
Proactive monitoring: Visibility to CPU jitter allows for proactive management of resources, ensuring that potential issues are addressed before they affect service quality.

Implementing CPU jitter counters in Windows Server 2025

Implementing CPU jitter counters in Windows Server 2025 is a straightforward process.

Launch Performance Monitor on Windows Server 2025.
Select from broad sets of available counters.

Add selected counters (total, per VP, LP, root VM, etc.) so that they appear under Added counters.

Start data monitoring. There are also options of creating User Defined Data Collection Sets.

Best practices for managing CPU oversubscription

To effectively manage CPU oversubscription and leverage the benefits of CPU jitter counters, consider the following best practices:

Establish baselines: Before diving into optimization, establish performance baselines under normal operating conditions. This will help in identifying deviations and potential oversubscription issues.
Use multiple counters: Relying on a single counter can provide a skewed view of the system's performance. Use a combination of the aforementioned counters to get a comprehensive understanding of CPU utilization and potential oversubscription.
Monitor trends over time: Short-term spikes in CPU usage might not necessarily indicate performance issues. Monitor trends over an extended period to identify persistent issues that require intervention.
Regularly review and optimize resource allocation: Periodically review the resource allocation for VMs and adjust as necessary based on workload demands to minimize contention and improve performance.

Suggested primary CPU oversubscription monitoring counters

New Windows Server 2025 counters:

\Hyper-V Hypervisor Virtual Processor(*)\CPU Wake Up Time Per Dispatch
\Hyper-V Hypervisor Virtual Processor(*)\CPU Contention Time Per Dispatch

CPU jitter performance monitoring counters:

\Hyper-V Hypervisor Virtual Processor(*)\CPU Wait Time Per Dispatch

The average combined amount of time a give VP had to spent waiting to run on an LP without specifying the reason for the wait.

\Hyper-V Hypervisor Virtual Processor(*)\Logical Processor Dispatches/sec

The number of times this VP was scheduled to run on any LP

Overall CPU oversubscription load monitoring counters:

\Hyper-V Hypervisor Virtual Processor(*)\% Total Run Time

The percentage of time spent by the virtual processor in guest and hypervisor mode combined.

\Hyper-V Hypervisor Virtual Processor(*)\% Guest Run Time

The percentage of time spent by the virtual processor in guest mode specifically.

\Hyper-V Hypervisor Logical Processor(*)\% Total Run Time

The percentage of time spent by the physical processor in guest and hypervisor mode combined.

\Hyper-V Hypervisor Logical Processor(*)\Scheduler Local Run List Size

The number of virtual processors that are scheduled to run on a given physical processor.

The newly introduced CPU Contention Time Per Dispatch counter provides great insight on impact of system resource contention to jitter. It measures exactly the time a VP had to spend waiting for an LP to start processing the load. It is an enhancement of the previous CPU Wait Time Per Dispatch which captures multiple factors contributing to the wait time. In combination with the Logical Processor Dispatches/sec we can exactly calculate the system delay due to contention.

Contention delay time = (CPU Contention Time Per Dispatch) x (Logical Processor Dispatches/sec) [ns]

Contention delay time should be strictly monitored and used as a factor to further balance workloads. Knowing the details of the system’s process delays will allow us to achieve higher VM density without compromising system performance and end user experience.

Through Windows Server 2025 performance monitor we can calculate the created contention delay time per VP. Please note that using CPU Contention Time Per Dispatch as a direct comparison may cause misleading results. It must be always monitored together with Logical Processor Dispatches/sec to account for workloads’ specifics.

In a test environment a host with 4CPUs (8LPs) with 2 VMs with 4 VP each, we run a load of 2 threads per VM.

2VMs of 4VPs each; VM1 VP0CPU Contention Time Per Dispatch (red) and Logical Processor Dispatches/sec (green)

It is recommended that each VP to be monitored independently or use the total counter values. In case of total counter values CPU Contention Time Per Dispatch is provided as average of all VPs, and Logical Processor Dispatches/sec is a sum of all dispatches per VPs (it must be divided to number of VPs on the system to align with single metric for a VP).

From the example shown on Fig.1 we have VM1 VP0 average values of:

CPU Contention Time Per Dispatch = 30,617 [ns]

Logical Processor Dispatches/sec = 577 [dispatches/s]

Contention delay time = 30,617 x 577 = 17.67 [ms]

We can clearly increase the load on the system and improve efficiency by doubling the VM density and reaching 100% CPU oversubscription adding 2 additional VMs with 4 VPs each. If we run the same load across all VMs as on the first 2 VMs (each of 2 threads) we can observe the increase of the contention delay.

4VMs of 4VPs each 100% CPU oversubscription; VM1 VP0CPU Contention Time Per Dispatch (red) and Logical Processor Dispatches/sec (green)

From the example shown on Fig.2 we have VM1 VP0 average values of:

CPU Contention Time Per Dispatch = 586,700 [ns]

Logical Processor Dispatches/sec = 430 [dispatches/s]

Contention delay time = 586,700 x 430 = 252.28 [ms]

This may be acceptable for some workloads, or we need to place VMs with diverse workload to reduce contention delays.

If system loads increase per VM from 2 threads load to 4 threads load on all 4 VMs (total of 16 threads, all VMs 100% loaded, 100%CPU oversubscription). Contention delay significantly increases.

4VMs of 4VPs each 100% CPU oversubscription 100% utilization; VM1 VP0CPU Contention Time Per Dispatch (red) and Logical Processor Dispatches/sec (green)

From the example shown on Fig.3 we have VM1 VP0 average values of:

CPU Contention Time Per Dispatch = 938,641 [ns]

Logical Processor Dispatches/sec = 794 [dispatches/s]

Contention delay time = 938,641 x 794 = 745.28 [ms]

Contention delay reaching high values could be an indication that VM(s) needs to be migrated to a different host to address the contention scenario.

Virtual Machine Resource Control

Hyper-V provides advanced tools to control resource allocation and jitter through per-VM CPU resource controls (including a CPU cap, weight, and reserve), isolating VM groups to specific host processors, and minroot.

Using CPU weights allows for more granular control over how CPU resources are allocated among VMs. This ensures maintaining consistent performance levels for critical workloads.

Assigning specific LPs to a VM group ensures that the VMs in that group have exclusive access to these compute resources, eliminating competition with other groups. High-priority VMs can achieve optimal performance as they are not de-scheduled by other VMs. This setup is particularly useful for applications requiring real-time processing or low-latency operations.

The minroot configuration allows Hyper-V administrators to dedicate a subset of processors exclusively to the root partition, effectively segregating the work done in the Hyper-V host from the workloads running in guest virtual machines. This segregation helps in better resource management and enhances system stability.

For more information on implementation and utilization of the functions listed above, see Optimizing your Hyper-V hosts.

Advancing performance management

The introduction of new CPU jitter counters in Windows Server 2025 marks a significant advancement in performance management for virtualized environments. By providing detailed insights into CPU variability, these counters empower administrators to optimize resource allocation, increase VM density, improve performance stability, and enhance the overall reliability of their systems. As we continue to embrace virtualization and cloud computing, tools like CPU jitter counters will play a crucial role in ensuring that our digital infrastructure remains robust and efficient.

Optimizing your Hyper-V hosts

Steven Ekren — Wed, 12 Feb 2025 15:06:41 GMT

The good news is that Hyper-V is designed to provide optimal performance and functionality based on the dynamics of what’s running on the host and the needs of the VMs regarding memory and CPU resources. In most cases, running with the defaults will work great.

For those cases where some tuning to optimize for host CPU or live migration is desired, this blog will cover some opportunities to adjust your Hyper-V systems.

Overview of CPU Scheduling

Before we get into the ways to configure and manage Hyper-V there are a few key concepts that are important to understand. For more details, reference: Manage Hyper-V hypervisor scheduler types | Microsoft Learn

Physical CPUs and Logical Processors

Modern systems allow for multiple physical CPUs, with each CPU having multiple Logical Processors (LPs). LPs are what the host and VMs virtual processors map to for thread execution. Here is an outline of how CPUs, Cores, and Logical Processors relate:

A host has X number of sockets in which a CPU plugs into
Each CPU has X number of cores
Each Core has one or more Logical Processors (LP)

o Simultaneous Multithreading (SMT) allows for 2 LPs per Core, if enabled. If not enabled, there is 1 LP per core.

Diagram 1: Showing the host having a single Socket and the CPU with 4 cores. With SMT enabled, this will show 2 logical processors per core.

Note, the diagram above also shows that each core has 2 LPs, and the virtual processors are mapped to LPs of the Core. A core will not execute threads on its LPs from VPs of different VMs. This is why it’s recommended to configure VMs with an even number of VPs.

The image below shows how this configuration looks in Task Manager. It’s an easy way to confirm that SMT is configured, there are twice as many logical processors as there are cores:

This is the powershell to get the same information:

PS C:\Users\Administrator> (Get-WmiObject -Class Win32_Processor) | fl NumberOfEnabledCore,NumberOfLogicalProcessors

NumberOfEnabledCore : 4

NumberOfLogicalProcessors : 8

Virtual Processors and Scheduling

Part of the configuration of a VM is the number of Virtual Processors (VPs) it has. As the name implies, the operating system in the VM sees these as CPUs to execute threads on. The scheduler then manages execution on an LP.

The sum of the VPs configured for the running VMs can be greater than the available LPs. To manage this, there is a “scheduler” in the hypervisor or host that manages which VP can execute on which LP.

Diagram 2: Showing a configuration where there are more Virtual Processors than Logical Processors. The scheduler will assign which VPs each set of LPs in a core will execute at any moment in time.

Because there are more VP’s than LPs in the above example, when the other VPs have threads waiting to execute it will stop the execution of LPs of some cores switch them to execution for waiting VPs, so another moment in time for the same system might look like this:

Diagram 3: Showing the same configuration as diagram 2, but in a different moment in time where there is a difference in the Virtual Processor to Logical Processor assignment

For the purposes of this blog, most examples will be specific to the “Core Scheduler” with SMT enabled on the system, unless otherwise noted. The Core Scheduler is the default and recommended option for Windows Server and Azure Local. For more information on the different schedulers refer to: https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/manage-hyper-v-scheduler-types#hypervisor-scheduler-types

Optimize Host and VM CPU by Dedicating CPUs to the Host

The host OS is running processes critical to the system, like managing live migrations for VMs, hosting services like the Failover Cluster Service, and managing virtual machines’ resources.

If the host is showing high CPU utilization, it may be that the host is impacted by the CPU demand of the VMs. Dedicating CPUs for the host’s use can alleviate the impact of contention between its processes and the resources that the VMs it’s hosting require. This separation can be beneficial to the VMs also, as it keeps host processes from causing contention with the VMs CPU scheduling.

Using MinRoot

MinRoot is a feature that limits CPUs that are used for the host. This article describes, in detail, how it works and how to configure MinRoot. Essentially, a BCDedit command is used to specify the number of Virtual Processors (VPs) to assign to the Host. It assigns a Logical Processor (LPs) directly to each host VP for the host’s use. The remaining LPs of the system will be available to VMs without the host VPs being scheduled on them. The LPs assigned to the host may share time with VPs from VMs. The MinRoot setting will become effective once the host is restarted.

Diagram 4: Showing a configuration where MinRoot is used to dedicate 2 logical processors to the host. The other 6 logical processors are shared between the VMs

Choosing the number of LPs dedicated to the host is similar to the process of identifying the number of VPs to assign to a VM. It depends on what’s running inside of it (its workloads). Common variables include the CPU load from networking (RDMA enabled networks can help reduce this) and other software that has CPU demands. The CPU utilization from the OS may increase with the increase of the number or type of VMs running on it. Loads are specific to each deployment and often change over time, so occasional monitoring of host CPU utilization, particularly at times of peak loads, is recommended.

This article has some helpful information on how to use Performance Monitor, which is a tool that is built into Windows Server and Azure Local and Windows Client, which is commonly used to analyze CPU utilization. While the article references Microsoft SQL Server, the premise is the same for general CPU utilization monitoring.

Optimizing Live Migration

Live migration (LM) is used to load balance VMs between hosts and to facilitate host maintenance with limited impact on the workloads running inside of the VMs. For example, Failover clusters use LM to automatically move VMs for activities like updating hosts using Cluster Aware Updating (CAU).

More information: Live Migration Overview | Microsoft Learn

Limiting the interruption of network communications and CPU while a VM it is moving from one host to another is often important for workloads that have low “jitter” requirements.

Hyper-V’s Live Migration has been optimized over the years for speed and reliability. The initial phase of a live migration copies the VM state from the source host to the destination host using multiple passes, since the VMs are in use and the memory and processor state are continuously changing.

Once the delta between the source and destination state is minimal, the source VM is paused, and the final memory and state copy is completed. It is at that point that the destination VM is resumed and becomes active. This final phase is often completed in under a second, but the time can depend on the amount of CPU/Memory churn caused by what’s running in the VM, and network bandwidth and latency that is used to transfer the state.

Here are some settings that can optimize live migration. It’s also good to remember that if the hosts are part of a failover cluster, all nodes of the cluster should be configured identically.

Number of Simultaneous Live Migrations

The number of LM sessions that a host can run at the same time, whether source or destination, is controlled by setting the -MaximumVirtualMachineMigrations parameter. Live migration uses host resources like CPU and networking.

If the goal is to reduce the time for individual VMs to be live migrated, then a lower number is better. If the goal is to take a set of VMs and live migrate them all in the shortest amount of time, then a higher number might be better. However, since VMs are constantly changing memory state because they are running and active, simultaneously live migrating more VMs might actually take more overall time to accomplish.

The recommendation is to keep this number low and analyze the effect on the LM time and the system resources when trying higher settings, to find the sweet spot for your system.

To get the value:

Get-VMHost -MaximumVirtualMachineMigrations

To set the value to 2:

Set-VMHost -MaximumVirtualMachineMigrations 2

Reminder: ensure this setting is the same across all nodes of a failover cluster

Host Networking Considerations

Live migrations use a network connection between the source and destination nodes. Optimizing the network configuration can allow the initial and final memory and state transfers to complete faster.

RDMA and SMB

RDMA is a network offload that reduces host CPU overhead for the network traffic, while increasing the throughput. It requires NICs on the hosts and switches that the traffic flows to support RDMA and to be properly configured. When enabled, the same network traffic can have as much as 20% higher throughput (more data per second) and 20% less CPU usage. Therefore, the live migration will be completed faster with less system CPU impact.

RDMA does require the servers to have NICs that have the capability, and the proper configuration for the NICs and switches that the network traffic flows through between the source/destination servers.

RDMA configuration is specific to the NICs and switches in the environment, so details are not in-scope for this blog.

If the source/destination systems for a live migration are configured to allow RDMA networking, live migration needs to be set to use SMB as the transport protocol.

Set live migration to use SMB:

Live migration can transfer data between nodes using 3 different mechanisms.

The default is “TCP/IP” which uses a Hyper-V managed connection between the source and destination systems.
“Compression” is useful to reduce the amount of data that is transmitted over the networking by compressing it. Compression will use more host CPU to do the compression and decompression on each node. This is often the best option for scenarios like doing live migration over a WAN where bandwidth is limited or charged.
“SMB” allows the data to be transmitted using SMB optimizations like SMB Direct, which is what allows the LM traffic to go over RDMA enabled connections. If the nodes have more than one network path with the same characteristics (throughput, RDMA/Non-RDMA), SMB will manage using all of them using SMB Mulitchannel.

To change the setting, use Set-VMHost from a PowerShell Console, opened as Administrator. This is the command to use to change to SMB. Ensure this is set consistently all the nodes of a failover cluster.

Set-VMHost –VirtualMachineMigrationPerformanceOption SMB

Set SMB bandwidth limit for Live Migration

Live migration will use as much of the network bandwidth as possible, which can starve other components that may need to use the same connection, like failover cluster traffic. When the system is set to use SMB for live migration, it is recommended to set a limit for live migration traffic.

The amount of bandwidth to limit live migration depends on the available bandwidth between the source and destination systems.

For example, if there are 2 connections between the host and destination which are RDMA capable and have a throughput of 10Gb (Gigabit), the aggregate would be 20Gb. A general recommendation would be to limit live migration traffic to ~50%, which would be 10Gb. While a good starting point, if live migrations are too slow the limit could be increased. Likewise, if other components of the system, like Failover Cluster and Cluster Shared Volumes, are having issues during live migration, reducing the limit for Live Migration may be needed.

To set the bandwidth limit for live migration traffic over SMB (example value of 10Gb):

Install the SMB bandwidth limit feature

Add-WindowsFeature -Name FS-SMBBW

Then set the limit for LiveMigration

Set-SmbBandwidthLimit -Category LiveMigration -BytesPerSecond 10GB

Resources:

Virtual Machine Resource Controls | Microsoft Learn

Live Migration Overview | Microsoft Learn

Optimizing Hyper-V Live Migrations on an Hyperconverged Infrastructure - Microsoft Community Hub

https://learn.microsoft.com/en-us/troubleshoot/windows-server/virtualization/troubleshoot-live-migration-guidance

Set up hosts for live migration without Failover Clustering | Microsoft Learn

Monitor CPU Usage - SQL Server | Microsoft Learn

OpenHCL: the new, open source paravisor

Caroline_Perezvargas — Sun, 09 Feb 2025 23:03:27 GMT

Intro

From the beginning of the cloud computing era, virtualization technology has enabled compute workloads to run as virtual machines (VMs) in a server environment. As hardware has evolved, and new functionality has become available, the software stack has kept VMs running seamlessly, thanks to sophisticated advances in the hypervisor and virtualization software.

Confidential computing is now a powerful technology for significantly improving the security of VMs running in the cloud. However, the trust boundary of a confidential VM imposes a barrier that prevents the hypervisor from offering the rich virtualization services that VMs normally expect. Customers desiring the benefits of confidential VMs have been forced to update the operating systems of their VMs to newer versions, which must be continually revised as confidential VM technology maintains its path of rapid evolution.

Microsoft has embraced a different approach that offers much more flexibility to customers through the use of a “paravisor”. A paravisor executes within the confidential trust boundary and provides the virtualization and device services needed by a general-purpose operating system (OS), enabling existing VM workloads to execute securely without requiring continual service of the OS to take advantage of innovative advances in confidential computing technology. As confidential computing becomes available on more hardware platforms and evolves, the software stack can keep VMs running seamlessly thanks to the paravisor, in much the same way other advances in virtualization software enabled VMs to run seamlessly on ever evolving hardware.

Microsoft developed the first paravisor in the industry, and for years, we have been enhancing the paravisor offered to Azure customers. This effort now culminates in the release of a new, open source paravisor, called OpenHCL. We plan to develop OpenHCL in the open here: microsoft/openvmm: Home of OpenVMM and OpenHCL (github.com).

OpenHCL capabilities

A paravisor is essentially an execution environment that runs within the guest VM - at a higher privilege level than the guest OS - and provides various services to the guest. A paravisor can run in both confidential environments and non-confidential environments. When running in a confidential environment, these privilege levels must be enforced by the confidential computing hardware platform.

We use virtual secure mode (VSM) to run a paravisor on Microsoft’s virtualization stack. When running in a confidential context, our architecture allows VSM to be appropriately enforced in a hardware platform-agnostic manner.

Today, OpenHCL can run on both x86-64 and ARM64 platforms, and it has support for Intel TDX and AMD SEV-SNP confidential computing platforms. OpenHCL runs in the L1 VMM of a TDX confidential VM and in the VMPL0 of an SEV-SNP confidential VM. See the OpenHCL user guide for step-by-step instructions to use it. OpenHCL offers a rich set of powerful services to both confidential and non-confidential VMs alike:

Device emulation via standard device interfaces, essentially offering a set of emulated devices, such as vTPM and serial.
Device translation via standard device interfaces, such as NVMe to para-virtualized SCSI, allowing assignment of hardware devices directly to VMs (accelerated IO) without requiring guest OS changes - enabling VMs to take advantage of the performance of cutting-edge devices.
Diagnostics support, particularly useful to allow debugging confidential VMs where it is difficult to use traditional methods of debugging.
(To confidential VMs specifically) Support for guests that are not fully enlightened - such as Windows and older versions of Linux - to run on confidential computing platforms via standard architectural interfaces.

For confidential VMs, even though OpenHCL provides amazing value to guests that are not fully enlightened (by enabling them), OpenHCL can also provide a lot of value to fully enlightened guests by offering them any or all its other services as different scenarios require it.

OpenHCL is used in Azure in new Azure Boost SKUs, and it will be used in future Azure confidential VM SKUs. In the past month alone, over 1.5 Million VMs were running with OpenHCL in Azure^[1].

OpenHCL architecture

OpenHCL is composed of several open-source components, the most important one being OpenVMM, the modular, cross-platform, virtual machine monitor (VMM) written in Rust. This VMM runs several user mode processes to power OpenHCL. Running a VMM inside OpenHCL allows us to support guests with assigned devices and provide device translation support. Additionally, it allows us to share confidential and non-confidential architecture. We run the same VMM in the same environment for both confidential and non-confidential guests, and the VMM provides the same services tailored to their requirements. This avoids fragmented virtualization solutions among confidential and non-confidential VMs, moving towards closing the feature gaps of confidential VMs.

The other components of OpenHCL are a boot loader and a small, customized Linux kernel built to support the VMM, with min. Kconfig to minimize binary size and runtime RAM usage. Running a kernel to support our environment allows the VMM code to be mostly standard Rust, making it much more powerful by enabling the VMM to use the broadly supported and stable Rust toolchains and crate ecosystem.

The two approaches to running confidential VMs

There are two approaches to running a guest OS inside a confidential VM: either the guest must be fully enlightened (modified to understand and manage all aspects of running as a confidential VM), or it can rely on a paravisor to implement the confidential computing enlightenments on its behalf. When a guest runs in with a paravisor, it doesn’t seem like a confidential guest precisely because it doesn’t need to act like a confidential guest.

In Azure, we support all IaaS confidential VMs via a paravisor today. The paravisor enabled Azure to support the widest variety of guests, including Windows versions released almost a decade ago^[2] and Linux versions using kernels as old as the 5.19 kernel^[3] (and versions using even older kernels that had a small set of patches backported, such as some Ubuntu and RHEL distro versions). This provides customers with an easier lift as well as the flexibility to gain future confidential computing advances without needing to upgrade their workloads. Customers’ legacy solutions are safe with Azure because of the approach we embraced.

Why is Windows not fully enlightened to run as a confidential guest? I.e., why does Windows rely on a paravisor?

When we developed the first confidential VM in Azure on the confidential computing hardware platforms available at the time, it was not possible to fully enlighten Windows guests for those platforms because Windows required APIC (interrupt controller) emulation to be done in a paravisor. APIC emulation, traditionally done by the hypervisor, must be done by another entity for confidential VMs, where the hypervisor is outside the trust boundary. It can be done by the paravisor or by the hardware platform if it supports APIC virtualization, which early platforms like 3rd Gen AMD EPYC™ processors, didn’t.

On those hardware platforms, APIC emulation had to be done in a paravisor for Windows guests but not necessarily for Linux guests. The architecture of Windows relies directly on the APIC for interrupt management. Some aspects of Windows interrupt management don't flow through the kernel and are inlined in drivers, so Windows drivers rely on the interrupt management behavior offered by the APIC. The architecture of Linux, on the other hand, doesn’t rely directly on the APIC for interrupt management. Linux offers kernel service routines for handling interrupt state, so Linux drivers rely on these routines.

In addition to that, Windows relies on the presence of a TPM for security features, and one cannot implement a vTPM for a confidential VM with enlightenments alone. We chose to implement a vTPM in a paravisor. Given all the functionality we have built into the paravisor, our plan is not to fully enlighten Windows and continue supporting Windows guests via a paravisor in Azure. For future versions of Linux, we’re evaluating both approaches – fully enlightened and relying on a paravisor – and we will aim to do what is best for customers.

OpenHCL and COCONUT-SVSM

An SVSM like COCONUT-SVSM plays a very valuable role for confidential computing. It can store secrets and provide virtualization services to improve the usability of fully enlightened guests. OpenHCL solves a different problem than COCONUT-SVSM. COCONUT-SVSM aims to provide services to confidential VMs with fully enlightened guests using new interfaces. OpenHCL aims to provide services to confidential VMs using existing standard architectural interfaces.

COCONUT-SVSM provides device emulation, but OpenHCL uniquely provides this via existing standard interfaces. When running with an SVSM (like COCONUT-SVSM), the guest must establish a specific relationship with the SVSM by discovering its presence and then interact with the SVSM using a custom calling convention. Essentially, a guest needs to be specifically modified to be able to take advantage of SVSM services, including devices. With OpenHCL, devices are easier to consume because existing device interfaces just work, and the guest does not need any custom calling contract modifications to consume them. OpenHCL enables devices to be discovered over standard enumeration mechanisms, like PCI virtualization or existing vTPM device contracts.

COCONUT-SVSM could potentially be leveraged by OpenHCL in the future. The VMM of component OpenHCL is Rust based, which has strong memory safety properties, and evolving its kernel component to also be Rust based would improve the memory safety of OpenHCL. During the development of OpenHCL, we chose the Linux kernel because it was a familiar OS platform for contributors and provided the capabilities needed. Now that Rust-based COCONUT-SVSM exists, we are interested in moving to that in the future and building OpenHCL support for it if it gains the features that OpenHCL needs.

Open for collaboration

In this blog we described the value of OpenHCL for the future of computing. We still have much more we plan to do with OpenHCL, and as we develop new functionality in the open, we would love to collaborate with you. You can learn more about this project on: https://openvmm.dev. Please reach out to us if you have ideas you’d like to add to the OpenHCL roadmap or any other feedback. You can open a GitHub issue, reach out to us on Zulip , and even contribute to this project! We track the roadmap of OpenHCL in the open; below are some of its future milestones!

OpenHCL support for Intel TDX (Trust Domain Extensions) in Azure

Intel and Microsoft collaborated on and co-developed the TDX partitioning architecture so that it could be leveraged by a paravisor. The first ever TDX module with TD partitioning was an amazing co-engineering project between Intel and Microsoft, and Intel released TD partitioning as part of the TDX Module that accompanied the general availability of 5th Generation Xeon, and this has also been backported to 4th Generation Xeon. Using this TDX module, Azure launched the first generation of Azure TDX confidential VMs with the first paravisor, being the first cloud service provider to offer TDX in public preview as well as the first cloud service provider to offer Windows guest support for TDX. Intel has been contributing to OpenHCL for the past 6+ months, and we’re close to feature completeness in OpenHCL for the next generation of Azure TDX confidential VMs!

OpenHCL support for Arm CCA (Confidential Compute Architecture)

We started engaging with Arm almost two years ago to make sure the Arm Confidential Compute Architecture (CCA) is well equipped to support paravisor stacks like OpenHCL. CCA comprises a collection of open-source software, firmware, specifications, and hardware support to bring confidential computing to the Arm architecture. CCA provides protected environments called Realms, that can be used to host confidential VMs. Our collaboration lead to the creation of the Planes feature, which enables multiple of levels of privilege to coexist inside a Realm. Planes provide the ability to host a paravisor, and a guest VM in the same Realm, with the paravisor providing security and compatibility services to the guest. We are excited to collaborate further and in the open with Arm to build OpenHCL support for Arm CCA.

OpenHCL support for AMD SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging) in Azure

Microsoft and AMD have a long history of collaboration with multiple Azure Confidential VM offerings, powered by AMD Infinity Guard's SEV-SNP. We have been engaging with AMD to ensure OpenHCL and AMD’s platform can best work together to provide great performance and security for customers in the future. Virtual Machine Privilege Levels (VMPLs) provide hardware isolated abstraction layers within a VM for additional security controls, as well as assistance with managing communication with the hypervisor. We used AMD’s VMPLs to build the first paravisor for confidential VMs in Azure. We will continue to develop support in OpenHCL to reach feature completeness for future generations of AMD Azure SNP confidential VMs.

OpenHCL support for KVM as host

Today OpenHCL runs only on the MSFT hypervisor. We are looking forward to developing OpenHCL support for KVM as host in collaboration with other cloud service providers and the Linux and KVM community to enable others to leverage OpenHCL in their virtualization stacks.

And more to come

We also began engaging with Red Hat recently to discuss the value of OpenHCL and how it has the potential to open-up the door for significant cross-OS interoperability in the confidential virtualization world. We are excited to collaborate with Red Hat to build an open and collaborative confidential computing ecosystem.

- the Core OS Platform team.

[1] This number is from when this blog was published, Oct 2024, but this number keeps growing every month.

[2] Specifically, older versions include Windows Client 10 (released almost a decade ago) and Windows Server 2019.

[3] Specifically, older versions include Linux versions using the kernel 5.19 (for SNP) and 6.6 (for both SNP and TDX).

Announcing the open sourcing of OpenHCL

Caroline_Perezvargas — Fri, 22 Nov 2024 01:22:21 GMT

Introducing OpenHCL

Today, OpenHCL can run on both x86-64 and ARM64 platforms, and it has support for Intel TDX and AMD SEV-SNP confidential computing platforms. See the OpenHCL user guide for step-by-step instructions to use it. OpenHCL offers a rich set of powerful services to both confidential and non-confidential VMs alike:

Device emulation via standard device interfaces, such as vTPM and serial.
Device translation via standard device interfaces.
Diagnostics.
Support for guests that are not fully enlightened - such as Windows and older versions of Linux - to run via standard architectural interfaces.

OpenHCL is used in Azure in new Azure Boost SKUs, and it will be used in future confidential VM SKUs. In the past month alone, over 1.5 Million VMs were running with OpenHCL in Azure.

Open for collaboration

We still have much more we plan to do with OpenHCL, and as we develop new functionality in the open, we would love to collaborate with you. You can learn more about OpenHCL in our extended blog OpenHCL: the new, open source paravisor - Microsoft Community Hub as well as on https://openvmm.dev. Please reach out to us if you have ideas you’d like to add to the OpenHCL roadmap or any other feedback. You can open a GitHub issue, reach out to us on Zulip and even contribute to this project!

- the Core OS Platform team.

OpenHCL: Evolving Azure’s virtualization model

Hari_Pulapaka — Wed, 18 Sep 2024 16:45:10 GMT

Azure Boost is a revolutionary accelerator system designed by Microsoft that offloads server virtualization processes traditionally performed by the hypervisor and host OS onto purpose-built software and hardware. This offloading frees up CPU resources for virtual machines, resulting in improved performance and a secure foundation for your cloud workloads.

In this blog, we will talk about some of the advances we’ve made within Azure Host OS that allow us to provide the industry-leading benefits of Azure Boost and improve the security of our customers with other features. Azure Host OS (aka Cloud Host), if you recall, is a purpose-built minimal version of Windows that powers Azure in the data center. These Azure Host advancements in conjunction with Azure Boost have enabled features like Confidential VMs, Trusted Launch, to improve IO performance, harden security, and introduce VM compatibility for seamless feature delivery. These features are powered by a completely new transparent para-virtualized layer that runs within each guest VM instance, named “OpenHCL”. OpenHCL is a para-virtualization layer built from the ground-up in the Rust programming language. Rust is designed with strong memory safety principles, making it ideally suited for the virtualization layer.

Chris Oo from our team has a talk on OpenHCL at the “Linux Plumbers Conference 2024”, which has more technical design and details. The talk titled “OpenHCL: A Linux based paravisor for Confidential VMs” is available [here].

In the upcoming sections, we’ll start by exploring the virtualization landscape and how Azure’s infrastructure has evolved over time to take advantage of the modern hardware architecture. We’ll then talk about the internals of this para-virtualized layer and how it supports some of the core Azure features that our customers depend on.

Virtualization models

Azure Host OS provides core virtualization services for managing compute and memory resources, as well as virtualizing devices for VMs. Under the hood, it partitions physical hardware into logically separated virtual environments, each with their dedicated (virtual) processors, memory, and view of devices (storage, networking).

Traditional device virtualization

In traditional virtualization architecture, the host operating system handles most of the communication between the guest operating system (VM) and the underlying physical hardware (CPU, memory, device IO). For example, if the VM wishes to perform a network or storage operation (i.e. send a packet over the network, read/write data to storage), the guest communicates with the host OS (over a shared channel called VMBus) and the host facilitates the IO operation on the guest’s behalf.

This device virtualization model is referred to as a Para-virtualized IO model [wiki]. The guest OS is “enlightened” or aware that its running virtualized and runs special drivers to communicate with the host. This model is simple, efficient, and widely used across most cloud providers.

One drawback of this mode is that there is significant interaction with the host OS to do IO, which can add latency, affect throughput, or result in noisy neighbor side-effects. The performance of this mode can be significantly improved by allowing the guest VM to directly access the PCIe device instead of relying on the host for communication. Bypassing the host OS data path allows for lower latency, reduced jitter, and improved VM responsiveness. This is typically called “discrete device assignment” in Microsoft documentation or sometime referred to as accelerated device model.

Accelerated Device IO

As explained to achieve higher IO performance, the virtualization stack supports a direct assigned device or accelerated IO mode, where VMs can directly access and communicate with devices without Host intervention. If the VM wishes to perform an IO operation, the guest leverages special drivers that live within its context to communicate directly with the physical device.

In the same example above, if the VM needs to perform a network operation, it can perform it more efficiently by communicating using the direct path to the network device. This VM is considered fully enlightened-- it possesses the right drivers for direct communication with device hardware. The direct data path reduces overhead in comparison to the additional translations found in the para-virtualized IO model. This leads to improved performance and throughput that is comparable to physical devices running without virtualization.

Discrete Device Assignment (DDA) and Single Root I/O Virtualization (SR-IOV) are two types of accelerated device models used in virtualization. DDA assigns an entire device to a VM and is mostly used in GPU assignment scenarios to provide VMs full access to the GPU’s capabilities for workloads such as AI training and inferencing. SR-IOV divides a single physical device’s resources into multiple virtual interfaces for different VMs. SR-IOV is typically used for network and storage IO devices, as it allows multiple virtual machines to share the same physical hardware resources most efficiently

Some examples in the Azure fleet today include, GPU acceleration via Discrete Device Assignment, Accelerated networking via SR-IOV, and NVMe Direct VMs for storage.

In the next section, we will talk about OpenHCL which is another evolution of the device IO virtualization.

OpenHCL: A privileged guest compatibility layer

Building on the advancements of the accelerated model, we introduced OpenHCL, a new virtualization layer that can transparently provide guest VMs with facilities such as accelerated IO and other security features. This lightweight virtualization environment runs privileged within the guest virtual machine and isolated from the guest operating system. Instead of sharing para-virtualized components exposed by host interfaces, each VM runs its own virtualization instance which enhances security isolation and efficiency. As we’ll discuss below, OpenHCL is essential for Azure Boost guest compatibility scenarios, in which VMs require the appropriate drivers and orchestration to leverage performance enhancements from Boost’s NVMe storage and MANA network accelerated device.

This environment consists of two main components: a minimal Linux kernel and a Rust-based VMM that provides device emulation and I/O translation. This layer equips VMs with the necessary software and drivers to light up functionality such as SR-IOV device assignment for Azure Boost network and storage optimized accelerators without needing any change in the guest OS. This is hugely beneficial to our customers who can now use the same VM image while getting the benefits of Azure Boost – continuing to show our customers how much Microsoft invests in application compatibility.

To do this, we leverage Virtual Secure Mode (VSM) technology, a set of Hyper-V capabilities that enable new security boundaries (or “virtual trust levels”) within a VM context. By creating a new isolated Virtual Trust Level (VTL2) within the guest environment, we establish a higher privilege execution environment that can transparently host code in the VM. This allows us to run privileged security functionality like a virtual TPM for Trusted Launch VMs and paravisor for Azure Confidential VMs (we’ll cover these topics in later sections). Within this layer, we can also run device virtualization facilities that enlighten VMs to communicate with Azure Boost hardware.

The VSM isolation model and reduced data path from VM to device adds protective measures by providing more robust multi-tenant isolation and reducing the Trusted Computing Base (TCB) on the Azure Host. By confining the virtualization stack to the tenant’s VM and reducing dependencies on the Host for IO operations, we can eliminate shared host components which narrow down the potential attack surface and enhance security. Shifting the architecture from host providing para-virtualized interface to each VM instance running its own virtualization, additionally allows for greater performance isolation and efficiency. Reiterating this point since its so important with the OpenHCL architecture, each VM receives its own para-virtualized layer and doesn’t share anything with the Host or other VMs. This isolation hugely improves the customer VM experience and isolation.

Zooming into the components that make up this layer, the VTL2 environment is made up of a completely newly written Rust based virtualization stack running on a minimal Linux kernel that provides device emulation and I/O translation. Rust system programming language has emerged as one of the leading memory safe programming languages. Rust’s memory safety & type system features help prevent common vulnerabilities like buffer overflows and dangling pointers. Its concurrency model enhances security in multi-threaded environments by preventing data race conditions. Rust offers robust security benefits making it especially advantageous and critical for sensitive workloads. Together these components make up the para-virtualized VTL2 environment that underpins some of Azure’s key technologies.

In the next section we’ll describe some of the uses of this technology in Azure Boost, Trusted Launch VMs, and Azure Confidential VMs. This virtualization environment was first introduced with Trusted Launch VMs and was later extended to introduce additional capabilities around I/O compatibility and paravisor support for Azure Boost and Azure Confidential VMs respectively.

OpenHCL in Azure Boost

As mentioned, Azure Boost is Microsoft’s hardware acceleration solution that offers industry leading network and storage optimization via Microsoft Azure Network Adapter (MANA) and NVMe storage, by offloading networking and storage operations onto specialized FPGA hardware and software.

Offloading networking and storage tasks onto dedicated Azure Boost hardware frees up CPU for guest VMs and eliminates I/O virtualization bottlenecks. The result is a network capable of 200 Gbps bandwidth via Microsoft’s next generation network interface, Microsoft Azure Network Adapter (MANA), local storage operations reaching 17.3GBps with 3.8 million IOPs, and remote storage operations reaching 12.5 GBps throughput with 650K IOPs. Enhancing Azure's infrastructure by isolating it from hypervisor and host resources boosts performance while reducing latency and jitter.

Using the OpenHCL para-virtualized layer, VMs receive the necessary MANA and NVMe drivers and virtual functions to bootstrap accelerated IO connections. As a result, the guest can begin direct communication with the specialized Azure Boost hardware. On enlightened guest VMs, which come with pre-installed drivers and VMBus support to communicate with the accelerated hardware, the model sets up initial communication and reduces latency and downtime for networking and storage devices as it allows guest VMs to fall back to the software networking path in case of disconnection to the acceleration path.

For unenlightened guest VMs that come with default inbox virtualization drivers, OpenHCL transparently provides the necessary drivers to enable these guest VMs to communicate with the new accelerated hardware without the need to install new images or update the operating system. This allows the existing VM types to get the power of Azure Boost, with no changes to their images. To achieve optimal performance, we recommend adding the appropriate drivers to VM image.

Azure Boost VM SKUs are available today in preview across a variety of VM series to optimize for the demands of varying workloads. To learn more, see Overview of Azure Boost | Microsoft Learn

OpenHCL in Trusted Launch VMs

The OpenHCL virtualization layer also helped launch Trusted Launch for Azure virtual machines on Generation 2 VMs. Trusted Launch VMs introduced virtual Trusted Platform Module (vTPM) and secure boot with guest attestation. Secure Boot establishes a “root of trust” and verifies that only VMs with properly signed OS code can boot, preventing rootkits and boot kits from infecting the OS boot process with malware. A virtual Trusted Platform Module is virtualized hardware that serves as a dedicated storage vault for key and measurements. The vTPM measures and seals the VM’s entire boot chain (UEFI, OS, system, drivers), which allows the guest VM to perform remote guest attestation. Everything from the firmware through the OS drivers are “measured” and chained to a hardware root of trust. The VM can then establish trust with a 3rd party by cryptographically “attesting” or proving its boot integrity and compliance.

Leveraging the privileged OpenHCL VTL2 layer allows us to run a virtualized TPM and execute remote attestation processes directly from within the guest operating system. A virtual TPM cannot run and perform attestation if its running at the same privilege as the rest of the guest operating system as it stores and persists secrets.

OpenHCL in Azure confidential VMs

Confidential VMs provide enhanced security features that allow customers to protect their most sensitive data in use by performing computation inside a hardware based, attested Trusted Execution Environment (TEE). The Trusted Execution Environment is a secure, isolated environment that prevents unauthorized access or modification of applications and data while in use. This increases the security level of organizations that manage sensitive and regulated data.

Azure confidential VMs use the concept of a paravisor to implement enlightenment on behalf of the guest OS so that the guest OS can run mostly unmodified inside a CVM across various hardware providers. With the paravisor, the guest OS does not need to be fully enlightened to run confidential in Azure, meaning that we can support older OS versions on Azure confidential VMs. Without this paravisor, Azure confidential VM support would be limited to specific OS versions with the necessary features. This allows for easier “lift and shift” of sensitive workloads.

OpenHCL is an implementation of the paravisor for Confidential VMs in Azure, that will soon be available in the Azure fleet. Similarly to the virtualized TPM on Trusted Launch VMs, the OpenHCL VTL2 partition is used to securely host the guest paravisor firmware layer for confidential VM support. As mentioned above, OpenHCL will effectively allow guest VMs to run as confidential VMs in Azure, adding support across a wide variety of guest OS’s and confidential hardware providers.

To learn more about Confidential VMs, visit our other blog: Confidential VMs on Azure - Microsoft Community Hub.   

Learn more

In this blog, we've explored the evolution of Azure's virtualization architecture which help power industry-leading technologies like Azure Boost, Trusted Launch VMs, and Azure confidential VMs. We've outlined key benefits of this model across hardware acceleration, security isolation, performance, and seamless feature compatibility. As you read through the blog and the links within, if you have any questions, please feel free to comment below.

Hari and Marysia (on behalf of the entire OpenHCL team)

Windows Server 2025: The upgrade and update experience

DanCuomo — Fri, 16 Aug 2024 15:43:53 GMT

Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!

Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!

The 2024 Windows Server Summit was held in March and brought three days of demos, technical sessions, and Q&A, led by Microsoft engineers, guest experts from Intel®, and our MVP community. For more videos from this year’s Windows Server Summit, please find the full session list here.

This article focuses on the upgrade experience to Windows Server 2025.

Windows Server 2025: The upgrade and update experience

Discover the streamlined upgrade process to Windows Server 2025 in our session. We will cover N-4 media-based upgrades, feature upgrades through Windows Update, and efficient management of feature and quality updates with Windows Server Update Services (WSUS). Gain insights into best practices and tools for a smooth transition, ensuring your infrastructure aligns seamlessly with the latest advancements. Don't miss this opportunity for valuable insights, practical tips, and a roadmap to upgrade your Windows Servers effectively.

Networking improvements in Windows Server 2025

DanCuomo — Fri, 09 Aug 2024 12:00:00 GMT

Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!

Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!

This article focuses on networking improvements in Windows Server 2025.

Host networking at the edge

It's time for a closer look at what's new and exciting in networking for Windows Server! Explore Network ATC, which simplifies deployment and network configuration management for Azure Stack HCI clusters. Get to know Network HUD and how it can perform real-time analysis of network issues and provides prescriptive alerts or auto-remediation of the issue when possible. Learn how to greatly improve networking performance with Accelerated Networking.

Software Defined Networking

Take a whirlwind tour of the most requested and exciting new core features for Software Defined Networking (SDN) in this jam-packed session of demos! We'll share advancements in network security and Azure Kubernetes Service (AKS) integrations! If that isn't enough, we'll share great resources to help you learn and advance your skills fast. This is a session you won't want to miss!

Demo Bytes: Storage Replica, Failover Clustering, and Winget

DanCuomo — Fri, 02 Aug 2024 12:00:00 GMT

Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!

Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!

This article covers some demos of Windows Server 2025.

Demo Bytes: Storage Replica

Demo time! Get an up-close look at the next generation of Storage Replica!

Storage Replica was first released in Windows Server 2016 and has come a long way. See how we've improved performance by enhancing logs and compression. Watch demos where we replace DFSR with this modern replication system that will replicate in-use files and protect your organization from disasters.

Demo bytes: Failover clustering | Installing packages with WinGet

More demos! First, we'll look at the newest capabilities for failover clustering in Windows Server 2025. Find out how your organization can achieve high availability for manufacturing, retail, and AI scenarios. Then we'll switch gears to WinGet, the command-line utility that enables you to install applications and other packages in Windows Server 2025 from the command line.

Active Directory improvements in Windows Server 2025

DanCuomo — Fri, 26 Jul 2024 15:19:51 GMT

Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!

Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!

This article focuses on improvements to Active Directory in Windows Server 2025.

What's new in Active Directory for Windows Server 2025

The AD product group presents and demonstrate some of the new AD capabilities coming in Windows Server 2025, including new functional levels, security enhancements, and improved scalability.

Protecting Active Directory from management plane attacks

Mind the management plane! Whether your organization is running Active Directory on-premises, hybrid, or fully in the cloud, virtualized domain controllers are almost always present. But when is the last time you checked to ensure your privileged access model, aka Tier 0, extended to encompass the management plane?

Explore the common modern deployment scenarios for virtualized domain controllers and examine the relationship with the management plane. Why? Because attackers can exploit a weakly implemented privileged model and use the management plane as an easy back door into Active Directory.

In this session, we explore scenarios where organizations can unknowingly leave the door open to these attacks, diving deep into commonly observed gaps, and walking through a demonstration of using the management plane as a means of pivoting into Active Directory. Learn how to defend yourself and get actionable recommendations your organization can take today to ensure that the management plane does not become an attacker’s new friend.

The evolution of Windows authentication

As the security landscape evolves, Windows must continue to change to protect users and organizations. Foundational to this is user authentication. In Windows Server 2025 and Windows vNext, we have created completely new Kerberos features to minimize use of NTLM in your environments. This session explains and demonstrates IAKerb, Local KDC, IP SPN, and the roadmap to the end of NTLM.

Next-generation SMB file services

DanCuomo — Fri, 19 Jul 2024 12:00:00 GMT

Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!

Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!

This article focuses on improvements in SMB file services coming in Windows Server 2025.

Next-generation SMB file services

Radical changes are coming to the Server Message Block (SMB) and file services in Windows Server 2025 and Windows 11. Learn about new security behaviors, new functionality, and new scenarios available to your organization.

Supercharge your datacenters with Hyper-V and virtualized GPUs

DanCuomo — Fri, 12 Jul 2024 12:00:00 GMT

Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!

Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!

This article talks about virtualized GPUs with Hyper-V in Windows Server 2025.

Supercharge your datacenters with Hyper-V and virtualized GPUs

This session will dive into new GPU features on Windows Server and Azure Stack HCI, including GPU failover clustering and GPU partitioning (GPU-P). GPU-P, or GPU virtualization, is a new feature that allows users to share a physical GPU-P device with multiple virtual machines. With GPU-P, each virtual machine (VM) gets a dedicated portion of the GPU capacity instead of the entire GPU. This translates into more cost efficiency for your organization as you can assign GPU resources where needed. Join us as we share more on this new feature and run through multiple exciting demos.

New storage features in Windows Server 2025

DanCuomo — Fri, 05 Jul 2024 12:00:00 GMT

Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!

Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!

This article focuses on what’s new and what’s coming in Windows Server 2025.

New storage features in Windows Server 2025

Explore storage features in Windows Server 2025. We'll showcase exciting new capabilities in the ReFS file system, software-defined storage with Storage Spaces, innovation in Storage-spaces Direct (S2D), and new enhancements for Storage Area Networks (SANs) such as NVMe over Fabrics (NVMeoF). We will also show some of the incredible storage performance enhancements that make upgrading to Windows Server 2025 a must. Come for an overview of everything new, then dive into demos and a look "under the hood" of storage in Windows Server.

Securely design your applications and protect your sensitive data with VBS enclaves

Hari_Pulapaka — Wed, 13 Nov 2024 22:51:14 GMT

In November 2023, Microsoft’s Brad Smith and Charlie Bell announced the Secure Future Initiative (SFI), a new initiative to pursue our next generation of cybersecurity protection. At Microsoft and Windows, we have a unique responsibility and leading role to play in securing the future for our customers and our community. As you saw in David Weston’s blog post on Windows 11 security, we have a longstanding commitment to security in Windows. We introduced the Secured-core PC to help secure from chip to cloud and that critical layer of computing. In Windows 11, hardware and software work together to help shrink the attack surface, protect system integrity, and shield valuable data. Windows 11 comes with several security advances that protect our customers from Credential and identity theft, such as Local Security Authority protection, Advanced key protection using VBS, Windows Hello hardening etc.

Along with credential protection, we also prioritized helping app developers better protect people from phishing attacks and malware. One of the key advances we have made in this area is a feature called VBS enclaves. With Windows 11* and Windows Server 2025, VBS enclaves are now available to third-party application developers.

The next sections of this blog post will describe VBS Enclaves, its internals and how to use VBS Enclaves as a developer.

VBS enclaves

A VBS enclave is a software-based trusted execution environment (TEE) inside a host application. This is a revolutionary change in our security model for the application, allowing an app to protect its secrets using the power of VBS, from admin-level attacks.

Some background: Virtualization Based Security (VBS) is the core feature of Windows used to the high value secrets stored within Windows (e.g., Credential Guard). VBS utilizes the Hyper-V hypervisor to create an environment that is higher privileged than the rest of the system kernel. Like VM isolation, the hypervisor sets memory protections in the second level address tables and IOMMU tables to isolate this environment from the rest of the system kernel. The secure kernel (part of VBS) can also provide memory integrity protection to the system kernel, ensuring the system loads only signed drivers that are not tampered with.

We are now extending the isolated user mode in the VBS environment to allow developers to protect portions of application data in a software-based trusted execution environment (TEE) known as a VBS enclave.

As mentioned, a VBS enclave is a software-based TEE inside the address space of a host application. It is a Dynamic Link Library (DLL) loaded by a standard Windows application. VBS enclaves can help secure secrets and sensitive operations in memory. The basic premise is that a VBS enclave can isolate a portion of your application that you want to secure while it is in memory – for example, to securely decrypt and process sensitive information. To understand how a VBS enclave isolates secrets, you’ll need to understand the underlying technology it leverages, VBS. As mentioned earlier, VBS uses the Windows Hyper-V hypervisor to create an isolated, privileged virtual environment known as Virtual Trust Level 1 (or VTL1) that becomes the root of trust of the OS. The traditional Windows environment is called VTL0. VTL1 is further split into isolated user mode and the secure kernel. Windows uses VTL1 to host many of its security features. The hypervisor uses the second level address tables to maintain access and privileges for these virtual trust levels. The higher the number, the higher the privilege level. This means everything in a higher VTL is isolated from everything in a lower VTL.

The isolation provided by VBS is the core technology that allows a VBS enclave to isolate a portion of an application in higher-privilege VTL1, inaccessible to VTL0. Let’s look at what an application hosting a VBS enclave looks like:

The enclave hosting application lives in VTL0 and calls into the enclave when it needs to perform sensitive operations. Control is transferred to the VBS enclave, and the CPU register state is cleaned (except for specific parameter and result registers). Note that code and data inside a VBS enclave is inaccessible to VTL0 (including its own host application) and to other processes in VTL1. Code and data inside of a VBS enclave is visible only to the enclave itself, the VTL1 secure kernel, and the hypervisor.

At this point, you might be thinking “What’s stopping an attacker from exploiting the enclave in VTL1?” Good question! Though there is usually a process boundary between a non-Enclave application and a malicious actor in VTL0, by moving part of the application into VTL1, we add an additional boundary for the attacker to cross. This additional boundary is enforced by the hypervisor and is designed to be much more rigid. Think of VBS enclaves as a way to further harden your applications. VTL1 is a privileged space, and we don’t allow code other than Microsoft-signed code to run in the VTL1 secure kernel. The boundary between VTL0 and VTL1 is much more rigid to ensure that we can maintain this high bar.

This strong boundary doesn’t come for free. Accessing VTL1 is – at least more expensive than accessing VTL0 (we’re talking fractions of a millisecond here). Additionally, unlike VTL0 which has a lower barrier to entry, VTL1 requires all code to be signed. So, although VTL1 is a higher-privileged space, this privilege is maintained by requiring additional efforts by developers who wish to leverage VTL1. The nature of these efforts can be illustrated through some of the tenets VBS enclaves were designed with:

Limited API Surface

This is by design and serves a couple of purposes. Firstly, the smaller the range of functionality is within an enclave, the smaller the attack surface becomes. This ensures we can maintain the integrity of VTL1. Secondly, having a small API surface requires that you, the developer, think about how to best design your application so you only isolate what is critical in VTL1. Again, accessing VTL1 is comparatively expensive, so design your application wisely.

Code Integrity

Only code signed by Microsoft using a Trusted Signing VBS enclave certificate profile is permitted to run in an enclave. This includes loaded DLLs into the enclave. When control is handed from the VTL0 host application to the enclave, the VTL1 secure kernel will first verify that all the enclave code and data are authentic and are authorized to run inside of an enclave using image signature verification on the enclave image. This allows us to maintain our high bar for what we allow into VTL1.

This requirement means that developers are required to use Trusted Signing to obtain a certificate to production-sign their enclaves.

Attestation

Together with code integrity, VBS enclaves can generate attestation reports to attest to the state of the host system, the enclave itself, all DLLs that may have been loaded into the enclave, and whether the enclave is executing in debug mode. Note that once an enclave is initialized by the host application, the host can no longer modify the enclave and , to maintain the attestation state of the enclave throughout its lifetime. Attestation ensures that the code running in the enclave is exactly what you expect. Using a VBS enclave-generated attestation report and the MAA attestation service, you can attest to the state of the Enclave.

How do I use a VBS enclave?

You can use VBS enclaves to store secrets, seal data and perform decrypt operations, all in an isolated environment. The first step, as with any security feature, is thinking about secure design. VBS enclaves limit what you can do to maintain their security guarantees, so it isn’t as simple as moving the bulk of your application to live inside one.

Some considerations as you design an enclave:

A VBS enclave can be loaded by any application, not just the intended host application. Design your enclave without placing trust in the host application.
VBS enclaves operate by isolating from VTL0. To maintain the security promises of a VBS enclave, treat VTL0 as an untrusted environment. This includes not sending sensitive data outside of a VBS enclave. Only trust the enclave itself.
Ensure you understand the APIs available to you from within the enclave. Networking, for example, is not supported.

A sample use case of a VBS enclave is in Always Encrypted with secure enclaves for Azure SQL Database and SQL Server.

Always Encrypted protects the confidentiality of sensitive data from the database engine and administrators, placing trust only in the database clients to whom the data belongs. In the database, the sensitive data exists in an encrypted state. The data is only decrypted on the client side, where operations can be performed on it. In the database engine, operations are limited to equality checks.

Always Encrypted with secure enclaves, however, adds a VBS enclave to the database engine. When the database engine encounters computations on encrypted data, it delegates these computations to a VBS enclave, where the enclave decrypts the data and performs computations on plaintext. Employing a VBS enclave here makes it so database administrators cannot see the data inside the VBS enclave. With the use of VBS enclaves, Always Encrypted with secure enclaves can perform richer confidential queries and in-place cryptographic operations than Always Encrypted without secure enclaves, all without having to place trust in the database administrators.

The database client establishes a secure channel with the database engine containing the VBS enclave and identifies which columns are encrypted and manages encryption of these parameters in both directions. Now, when the database engine encounters operations on encrypted columns, it can delegate these to the VBS enclave. In the enclave the data can securely be decrypted if needed, and the operation can be performed.

Always Encrypted with secure enclaves is a fantastic use case of VBS enclaves:

The VBS enclave never reveals the decrypted information to the host application – the database engine in this case. It only trusts itself.
The database engine hosting the VBS enclave uses a secure channel (Transport Layer Security or TLS) to communicate with the database client.
By utilizing VBS enclaves, the database client no longer performs all of the operations on sensitive data client-side after decrypting them.

The Always Encrypted with secure enclaves use case illustrates the importance of secure design when thinking about how to employ VBS enclaves in your application. When used effectively, VBS enclaves are a powerful tool.

Now that you understand VBS enclaves and all that they can do, jump into the docs here, or read on for a guide on how to build your first VBS enclave by following the development guide here.

We hope this blog has you as excited as we are to begin developing secure applications with VBS Enclaves on Windows 11* and Windows Server 2025! We look forward to your feedback! Also take a look at the Windows Security Book to learn more about Windows Security technologies.

Hilal Asmat, Akash Trehan, and Hari (on behalf of the enclaves team)

*VBS Enclaves are available on Windows 11 Build 26100.2314 or later and Windows Server 2025 or later

Improving server security and productivity with Hotpatching

DanCuomo — Fri, 28 Jun 2024 12:00:00 GMT

Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!

Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!

This article focuses on Hotpatching coming in Windows Server 2025.

When it comes to installing securing updates, organizations are often concerned about the potential for business disruption and reduced system availability. This is a thing of the past with Hotpatching!

Come see how Hotpatching enables you to apply critical security updates without rebooting your servers, reducing downtime and improving productivity. Hear from the Xbox team, who have successfully adopted Hotpatching for the online gaming platform. Discover what is in store as we expand the service and make it more broadly available.

Microsoft options for VMware migration

DanCuomo — Fri, 21 Jun 2024 12:00:00 GMT

Looking to migrate from VMware to Windows Server 2025? Contact your Microsoft account team!

Windows Server 2025 is the most secure and performant release yet! Download the evaluation now!

Microsoft options for VMware migration

Recent developments in the on-premises virtualization market have unsettled users and prompted a re-evaluation of their organization's strategy. Microsoft provides a robust set of solutions tailored to your specific goals and requirements. During this session, we will delve into these options, emphasizing the long-term advantages of choosing Microsoft & Hyper-V.

Exploring New GPU Virtualization Features: A Closer Look

afiaboakye — Tue, 08 Oct 2024 20:35:12 GMT

In the rapidly evolving landscape of artificial intelligence (AI), the demand for more powerful and efficient computing resources is ever-increasing. Microsoft is at the forefront of this technological revolution, empowering customers to harness the full potential of their AI workloads with their GPUs. GPU virtualization makes the ability to process massive amounts of data quickly and efficiently possible.

With Windows Server 2025 Microsoft is introducing multiple new virtualized GPU advancements, including GPUs with clustered VMs through DDA (Discrete Device Assignment), GPU-P (GPU Partitioning) and Live Migration for GPU-Ps. Using GPUs with clustered VMs through DDA (Discrete Device Assignment) becomes particularly significant in failover clusters, offering direct GPU access. These new features will provide benefits for compute-heavy workloads, including machine learning and virtual desktop workloads such as CAD (Computer Aided Design) or FEA (finite element analysis).

NEW! Move your GPU Partitioned Devices Quickly with Live Migration

GPU-P or GPU partitioning allows users to share a single physical GPU device with multiple virtual machines (VMs) by providing each VM with a dedicated portion of the GPU’s capacity. This allows each VM to have the dedicated resources it needs for its specific workload. With heightened priority on security, GPU-P uses SR-IOV (single root I/O virtualization) to create a hardware-backed security boundary layer for each VM. This prevents unauthorized access from other VMs by ensuring each VM only has access to the specific GPU resources dedicated to that VM.  

Live Migration will now be enabled for GPU-P devices starting with Windows Server 2025 and Azure Stack HCI 24H2 OS releases later this year. Live Migration allows customers to provide maintenance and updates to their VM fleets with minimal workload impact. Live Migration enables the use of cluster-aware updating (CAU) on failover clusters nodes for GPU VMs. CAU allows automated updating of cluster nodes by moving the workload and cluster resources to a new node prior to a patch being applied. This allows the workloads to maintain availability with little to no impact.  With Live Migration, and the use of CAU, customers keep their datacenters fleets secure, updated and running so they can provide the services their customers rely on. 

Figure 1 – Example of GPU partitioning between two VMs  

GPU-P brings virtualization to the modern era by no longer requiring an entire GPU to be given to a single VM. The added feature of Live Migration will ensure customers can maintain their GPU-P workloads without impact while systems are up and running. GPU-P devices and hardware will now natively support virtualization, helping drive AI innovation.  

Live migration scenarios for GPU-P include clustered environments and standalone servers (outside a cluster). Live Migration will be enabled for GPU-P VMs with the Windows Server 2025 and Azure Stack HCI 24H2 OS releases later this year.  GPU-P is coming to Windows Server 2025 and is already enabled and available on Azure Stack HCI since the 22H2 OS release.  

System Requirements for GPU-P 

Supported GPU-P Devices 

NVIDIA A2, L4, A10, A16, A40, L40, L40S

Note:  

Above GPU devices listed are those currently supported for GPU-P and GPU-P Live Migration

This list of GPU devices is expected to expand in the future as IVHs update their GPU product portfolios. Check with IHVs for latest supported devices for GPU partitioning.

CPU Requirements 

AMD EPYC 7002 and later (also known by codename AMD Milan)

5th Generation Intel® Xeon® Scalable Processors and newer (also known by codename Intel Emerald Rapids)

NEW! Use GPUs with Clustered VMs through Direct Device Assignment
Using GPUs with clustered VMs through DDA allows you to assign one or more entire physical GPUs to a single virtual machine (VM). DDA allows virtual machines (VMs) to have direct access to the physical GPUs. This results in reduced latency and full utilization of the GPU’s capabilities, which is crucial for compute-intensive tasks.

Figure 1: This diagram shows users using GPU with clustered VMs via DDA, where full physical GPU are assigned to VMs.

Using GPUs with clustered VMs enables these high-compute workloads to be executed within a failover cluster. A failover cluster is a group of independent nodes that work together to increase the availability of clustered roles. If one or more of the cluster nodes fail, the other nodes begin to provide service, meaning high availability by failover clusters. By integrating GPU with clustered VMs, these clusters can now support high-compute workloads on VMs. Failover clusters use GPU pools, which are managed by the cluster. An administrator creates these GPU pools name and declares a VM’s GPU needs. Pools are created on each node with the same name. Once GPUs and VMs are added to the pools, the cluster then manages VM placement and GPU assignment. Although live migration is not supported, in the event of a server failure, workloads can automatically restart on another node, minimizing downtime and ensuring continuity.

Using GPU with clustered VMs through DDA will be available in Windows Server 2025 Datacenter and was initially enabled in Azure Stack HCI 22H2.

To use GPU with clustered VMs, you are required to have a Failover Cluster that operates on Windows Server 2025 Datacenter edition and ensure the functional level of the cluster is at the Windows Server 2025 level. Each node in the cluster must have the same set up, and same GPUs in order to enable GPU with clustered VMs for failover cluster functionality . DDA does not currently support live migration. DDA is not supported by every GPU. In order to verify if your GPU works with DDA, contact your GPU manufacturer. Ensure you adhere to the setup guidelines provided by the GPU manufacturer, which includes installing the GPU manufacturer specific drivers on each server of the cluster and obtaining manufacturer-specific GPU licensing where applicable.

For more information on using GPU with clustered VMs, GPU Partitioning and GPU-P Live Migration please review our documentation below:

Introducing GPU Innovations with Windows Server 2025 - Microsoft Community Hub 

Partition and share GPUs with virtual machines on Hyper-V | Microsoft Learn 

Partition and assign GPUs to a virtual machine in Hyper-V | Microsoft Learn 

Use GPUs with clustered VMs on Hyper-V | Microsoft Learn

Deploy graphics devices by using Discrete Device Assignment | Microsoft Learn