Server 2025 Hyper-V deduplication corruption is still there
I can still reproduce this with 100% confidence on two VERY different machines and hosts with my "easy to reproduce" package: - AMD Ryzen 5950 with Windows 11 23H2 (self built) - Dual Xeon 6226R with Server 2019 (HP Proliant DL 380). As I posted quite often now: If you use deduplication for Hyper-V, i.e. where the virtual machines reside, you will get data corruptions, your virtual machines will crash, and you may even get a filesystem corruption on the drive hosting the machines. For more details see here: 26063 deduplication data corruption is still there. - Microsoft Community Hub As for now: If you used deduplication in Hyper-V scenario, like you used it with Server 2012 up to server 2019, don't bother with Server 2022 and Server 2025 until that bug is fixed. My offer still stands for a live demonstration how I reproduce and talk about the differences how Microsoft tried to reproduce this and how I reproduce this. This is the result for "right about now" albeit without "funk soul brother"...Server vNext 26040 and Server 2022 deduplication data corruption
Following scenario gives me, easy to reproduce on various newer comptuer with SSD-based storage, data corruption with deduplication. Server 2019 does not have that issue. Reproduced on several Intel and AMD machines with 16 or more (real, non SMT) cores. Can be reproduced with HDD storage on i7-4960x CPU too, but with less that 50% probability. Host: Server 2019, Server 2022 or Windows 11. 1-st level VM: Server 2022 or Server vNext 25xxx or 26xxx, nested-v active for that VM. Have a drive 😧 for your nested VMs. 2-nd level Guests: More than one, since at least two running concurrently are needed to trigger the issue. OS does not matter, but easiest are Window VMs running Windows Update. As long as deduplication for 1-st level VM is not activated everything is fine. But if you activate deduplication for drive D:, where the nested VMs reside, you will get data corruption, in some cases even so bad that the dedup-chunkstore shows up as filesystem defect. Ready-to-reproduce packages: Those are created on a Server 2019 host, so Server 2019 and ANY later OS should be able to import them. No network connection needed, not even a Hyper-V switch. Needs a computer with 32 GB RAM and 'round 100 GB of free space on a non-system drive. Unpack to non-system-drive, import into Hyper-V, start the VM. Then follow the text file on the desktop. Result as video, here Server 2022: https://www.joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4 Package with Server vNext 26040: https://www.joumxyzptlk.de/tmp/microsoft/Server-vNext-26040-nested-dedup-problem-export-from-S2019-host.7z Package with Server 2022: https://www.joumxyzptlk.de/tmp/microsoft/S2022-nested-2023-09-30-exported-from-S2019-host.7z Package with Server 2019, just as "See, Server 2019 works fine!" counter proof: https://www.joumxyzptlk.de/tmp/microsoft/S2019-nested-2023-10-07-exported-from-S2019-host.7zServer vNext 25987, 26010 (and Server 2022) deduplication corruption in nested-V setups
The deduplication corruption in the case of nested virtualization in Server 2022 and Server 25987 is still there. The package to reproduce for Server 25987: https://joumxyzptlk.de/tmp/microsoft/s-vnext-25987-dedup-problem-2023-11-03-exported-from-S2019.7z Server 2022 variant: https://joumxyzptlk.de/tmp/microsoft/S2022-nested-2023-09-30-exported-from-S2019-host.7z Host OS can be Server 2019, Windows 11 21H2 or Server 2022. CPU must be reasonable fast (i.e. from somewhere 'round 2018), support nested-v together with the Host OS (AMD and Intel CPU are fine). Host should have at least 24 GB RAM. The the "ready to reproduce package" must be unpacked to 😧 or higher drive letter which should be at least SATA SSD storage. Else the problem may not show up as predictable as it does here. These packages were exported from a Server 2019 host, HP DL 380 dual Xeon 6228R with 1 TB ECC RAM, so it is easier to reproduce since Exports from Windows 11 don't import so well on other hosts. Originally the bug showed up on a Ryzen 5950x with 64 GB ECC RAM. Reproduced with less than 50% chance on an i7-4960x 32 GB RAM with HDD storage, more than 50% on SSD storage. Server 2019 variant: Does exist, but there is no data corruption. How to use: Once unpacked to Non-System-Drive SSD storage import the the VM and follow the Textfile on the desktop. In short: Activate Hyper-V in that VM, default settings. Configure deduplication with default settings any profile in that VM for D:, start a dedup run. Check dedup rate. Should be above 50%, usually it is around 55%. Import those two VMs on drive 😧 of that VM: Start them both in parallel, and execute the Offline-Windows-Updates stored on the desktop. You won't make it past the first one.vNext 25951 deduplication profile "VDI/Hyper-V" data corruption (applies to Server 2022 too BTW)
Hello Community! Edit: It seems to be a racing condition. To reproduce it needs a rather modern and fast CPU, else the problem not show up at all. My i7-4960x is at the border, the problem does not always show up. Problem: Using the deduplication profile "VDI", internally called "hyperv" causes data corruption and, in some cases filesystem errors on host-level. Ready to reproduce package with a clean fresh ENGLISH vNext install: https://joumxyzptlk.de/tmp/microsoft/SNEXT-25951.7z Follow the text file on the desktop. If you skip the "configure deduplication as VDI for drive 😧 + Start-Dedupjob" step you won't have issues, it will work fine. What happens is the same from current Server 2022 and vNext 25951: https://joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4 Reproduced on Ryzen 5950x with ECC RAM and i7-4960x, quite different hardware. The insider feedback hub is still broken... Sadly. EDIT: How to reproduce (contents of text file on desktop of that VM): Creation host: Ryzen 5950x, 64 GB ECC RAM, Windows 11 21H2. (Issue reproduced on host: intel i7 4960x, 32 GB RAM (non ECC), Server 2022 on host.) This guest: Standard as you can see except for: "Set-VMProcessor -VMName SNEXT-25931 -ExposeVirtualizationExtensions:$true" and "Allow network spoofing" No second VHDX yet. Server 25931 VM PREPARATION: - Standard install. - Two GPEDIT.MSC: Allow empty passwords, and Lanman Workstation "Enable insecure Guest logons", which makes copying those test VMs easier on my home network. - Be in a network which allow internet access for Windows Updates, including the VMs. - Windows/Microsoft updates right after installation. - Activate Role Deduplication and Hyper-V, do not configure deduplication. Reboot. - Add VHDX for second drive, add folder \Hyper-V. - copy two test VMs, here those two Server 2012 R2 VMs with update state of 6th June 2022, import them in Hyper-V. - Remove Hyper-V role, cause dependent on the Host CPU different virtualization seem to be used up upon activation of that role. - Update 2. Sept 2023: In-Place upgrade to Server vNext 25941. This is the export you see now. These are the steps to reproduce the issue from here on, see screenshots attached / youtube video link: - Activate Hyper-V in this VM. Do not configure Switch during that step. Reboot. - Import the two machines in drive D:\Hyper-V into Hyper-V Manager. - Configure the virtual switch. You may have to untick the the "Hyper-V Extensible Virtual Switch" in the network properties of the adapter, a misbehaviour of the host OS. Connect the nested VMs to that virtual switch. - Activate deduplication for second volume, choose template "VDI", no adjustment of anything. - Run from Powershell: Start-DedupJob -Volume <volume> -Type Optimization -Full -Wait - Wait until it is finished - Start those two Test VMs, and try to run Windows update. This step is important, 'cause the corruption occurs on deduplicated files which then get written to. My test result: - https://joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4 - Both machines may blue screen during Windows Updates or show weird errors. Both may run into a recovery boot loop. - Both have defective filesystems, if they manage to boot instead of getting stuck in boot loop. - After the blue screen desaster: Start-Dedupjob -Volume <volume> -Type Optimization -Full -Wait may not work any more. - Get-DedupVolume | fl still shows valid statistics if Start-Dedupjob fails. - CHKDSK /f may show file system damage. Usually <NUMBER>.ccc and <NUMBER>.cd files of the deduplication chunk store which usually resides in \System Volume Information\Dedup\ChunkStore\{UNIQUE IDENTIFIER}.ddp\Data Counter verification: - Run Windows Updates on those VMs without deduplication. It will work fine. How I discovered that bug: I have a Server 2019-nested-vms testserver with many test-os variants installed there. I upgraded that VM to Server 2022. This is where I ran into the issues with windows updates within the VMs, and then traced it down to Server 2022 corrupting the filesystem on the host level. kind regards, Joachim Otahal GermanyServer vNext 25941 and Server 2022 deduplication profile hyperv/VDI corrupts data
I have described it here too: https://techcommunity.microsoft.com/t5/windows-server-for-it-pro/server-2022-and-server-vnext-build-25931-corrupts-filesystem/m-p/3915647 and I just reproduced it with Server vNext 25941 right now - still the same behavior. "Ready to reproduce" package as VM to import into Windows 11 or Server 2022 - follow the text file on Desktop of the VM: Based on Server vNext 25941 (24 GB): https://joumxyzptlk.de/tmp/microsoft/SNEXT-25941.7z Based on Server 2022 (26 GB): https://joumxyzptlk.de/tmp/microsoft/S2022-Nested.7z The video when I tested with Server 2022, same behavior with 25941 and 25931: https://joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4 Has been reproduced on Windows 11 21H2/AMD 5950x/64 GB ECC RAM and i7-4960x/Server 2022 as host OS for above VM. If anyone else can reproduce it: Tell me. Would be nice. Minor problem with build 25941: Feedback hub does not open.B20292 cpu spikes since b20292 due dedup
Since the upgrade to b20292 the server computer regularly has high cpu use Spikes (hear this due turning up fans) + TM Caused by dedup feature running ever since in this config. There is not much data changes and dedup optimize for VDI / Hyper-V / vhdx Anyone else. I will upgrade to the latest tomorrow and report back. Important to note that the cpu hog in TM is not tied to dedup task but remaining unvisible for the user think so procmon might help. Volumes are either a mirrored Storage Spaces or Single volumes. All with lastest ReFS. Anyone else?
