vNVMe on Hyper-V to unlock PCIe 5.0 NVMe performance

Question

On hosts with NVMe PCIe 5.0 (E3.S/U.2), Hyper-V guests still use virtual SCSI and leave a lot of performance on the table.

We are paying for top-tier storage, yet software becomes the limiter.

A virtual NVMe device that preserves checkpoints/Replica/Live Migration would align guest performance with modern hardware without forcing DDA and its operational trade-offs.

l_youtell_974 · Answer

what did you use to benchmark your virtual disk ?

In my case i use a crystaldiskmark and it work fine.
In my lab Hyper-v server, I have 8 NVME in raid 0 (RAID 0, not RAID 5 because its a lab server) and my VM use scsi connector like always.

This is the result

festuc · Answer

&nbsp;Thanks for the reply and for sharing your CDM settings. I reran everything on my side using the exact same CrystalDiskMark 9.0.1 parameters from your screenshot (5 passes, 1 GiB, R70%/W30%, SEQ1M Q8T64 and RND4K Q32T64). Hardware and topology details below for context.1) Drop in RND4K write after enabling the Hyper-V role (still testing on the host)Host (64 cores) on bare-metal vs same host with the Hyper-V role enabled.Sequential read/write stayed essentially identical (within ~±2%).However, RND4K write (Q32T64) on the host fell by ~44% on both drives after enabling Hyper-V.This was reproducible on:Intel Optane P5800X 800 GB (U.2, PCIe 4.0 x4)Solidigm D7-PS1030 3.2 TB (E3.S, PCIe 5.0 x4)Same OS, power plan = High Performance, same CDM build, same test size/data pattern, AV exclusions in place.In short: just turning on the Hyper-V role didn’t touch sequential, but it did hurt 4K random writes on the host by ~44% for me.2) Performance inside the VMVM configured with 32 vCPU (I do not want to break NUMA; NUMA Spanning = OFF on the host).Sequential (SEQ1M Q8T64) inside the VM is almost identical to the host (within ~1–2%) on both drives.RND4K read (Q32T64) inside the VM drops significantly versus the host:Optane P5800X: about −55% vs hostD7-PS1030: about −74% vs hostRND4K write (Q32T64) inside the VM goes up versus the Hyper-V host run (roughly +30%), but it’s still ~25% below bare-metal.My interpretation: the virtual SCSI + VMBus path and scheduling at T=64 threads inside the VM don’t scale like the host (especially for 4K random reads), while writes may benefit from coalescing/scheduling effects but still can’t match bare-metal.3) Test setup (for completeness)Host: 64 cores.VM: 32 vCPU pinned to a single NUMA node (to keep locality); NUMA Spanning OFF at the host level.Drives under test (one at a time, direct to CPU, no RAID):Intel Optane P5800X 800 GB, U.2, PCIe 4.0 x4Solidigm D7-PS1030 3.2 TB, E3.S, PCIe 5.0 x4CrystalDiskMark 9.0.1 x64, Admin, same settings as in your screenshot:5 passes, 1 GiB, R70/W30, SEQ1M Q8T64, RND4K Q32T64Note: the “RND4K (µs)” row in CDM is the same Q32T64 test expressed in average µs, not QD1 latency.Thanks again!&nbsp;

l_youtell_974 · Answer

Hi,&nbsp;i don't think you can obtain 100% of performance in your VM. Don't forget, you are not on the hard drive but you use an interface to write on a virtual drive so it take a little longer to write the information on the virtual drive.

festuc · Answer

Thanks for the follow-up—totally agree that 100% host-class performance in a VM isn’t realistic due to the virtual I/O path. My goal isn’t zero-overhead; it’s to separate what’s expected from what might be avoidable, especially for SQL-like, small-IO latency.What I’m seeing (CDM 9.0.1, Profile: Default, 5×, 32 GiB, Random; Windows Server 2025):Sequential &amp; higher-queue reads are fine in a VMSEQ 1MiB Q8T1 (Read): ~parity host ↔ VM on both drives.RND4K Q32T1 (Read/Write): ~parity on Optane; small drop on the Solidigm.The big gap is low-queue 4 K latency (Q1T1)Optane P5800X: 8.9 µs → 52.6 µs in VM (~6× slower 😒 ; 111k → 18.9k IOPS, −83%).Solidigm D7-PS1030: 55.1 µs → 119.7 µs in VM (~2.2× slower; 18.1k → 8.3k IOPS, −54%).That pattern feels heavier than “a little slower”: the vSCSI/VMBus/StorVSP path seems OK for throughput and for moderate queues, but penalizes Q1/T1 latency very strongly—precisely the OLTP zone.A couple of clarifications/questions:I’m not using exotic CDM presets (no T=64). This is the Default profile (RND4K Q1T1, RND4K Q32T1, SEQ Q1T1 Write, SEQ Q8T1 Read) to mimic SQL patterns.The host shows another oddity: after merely enabling the Hyper-V role, host-side RND4K write dropped ~44% (no VMs running). Is that expected (e.g., storage filter/stack changes when the role is installed)?If there are recommended tunables (StorVSC/StorVSP queueing, controller settings, best-practice for vSCSI) or an ETA/plan for vNVMe / multi-queue improvements, I’m happy to re-run with whatever parameters you suggest (can also switch to DiskSpd for 8 KiB Q1/T1 with latency percentiles).&nbsp;&nbsp;