Forum Discussion
Joachim_Otahal
Feb 26, 2024Iron Contributor
26063 deduplication data corruption is still there.
From Server 2022 up to this newest 26063 build, they all have the same problem, as described here: https://techcommunity.microsoft.com/t5/windows-server-insiders/server-vnext-26040-and-server-2022-de...
Joachim_Otahal
Mar 07, 2024Iron Contributor
Could the engineering reproduce this bug, maybe even as easy as I can? I don't expect a fix so fast, but if they could reproduce it would be a good feedback.
MSBernstein
Microsoft
Mar 07, 2024I checked on the bug report. As far as I can tell, the engineering team is still working to reproduce the issue.
- Joachim_OtahalMay 09, 2024Iron ContributorI can still perfectly reproduce this with build 26212, with 100% reliability, I suppose I have to recommend not using deduplication in Hyper-V on the local volume hosting the virtual machines with both Server 2022 and Server 2025.
https://techcommunity.microsoft.com/t5/windows-server-insiders/server-2025-hyper-v-deduplication-corruption-is-still-there/m-p/4136286- MSBernsteinMay 09, 2024
Microsoft
Hi, Joachim. MSSWRahman had asked you for a full dump and logs - were you able to hand those off? I understand that you can perfectly reproduce the issue, but we have not been able to reproduce this in-house at all.- Joachim_OtahalMay 10, 2024Iron Contributor
The full dump and the logs are requested for a different machine, an i7-4960x running Server 2022, which did no crash again yet to have such a dump - I activated the "full dump please" option. That single crash may be completely independent from that deduplication issue, which I reproduce in machines I have more confidence in (both have ECC RAM, for example). The i7-4960x machine may show its age since it is eleven years old now, so I have to rule it out as reliable test machine for the deduplication issue.
- Joachim_OtahalMar 08, 2024Iron ContributorThe easiest way is to use a Server 2019 or Server 2022 or Windows 11 Hyper-V host (real metal, not virtual). Somewhat current CPU, whether AMD Ryzen 2xxx or newer or intel Gen 8 (or their respective Threadripper / Epyc / Xeon counterpart ), SSD, 32 GB RAM. I could repo this in my old i7-4960x too, but not every time but rather every third time.
Then use these prepared packages, unpack on D:, import into Hyper-V. They are completely self contained repos.
The Server 2022 repo package: https://www.joumxyzptlk.de/tmp/microsoft/S2022-nested-2023-09-30-exported-from-S2019-host.7z
The Server 26040 repo package: https://www.joumxyzptlk.de/tmp/microsoft/Server-vNext-26040-nested-dedup-problem-export-from-S2019-host.7z
The follow the text file on the desktop.
To give you the full information: These are the contents of the text file on the desktop of that VM:
############################################################
Creation host: Ryzen 5950x, 64 GB ECC RAM, Windows 11 21H2.
----------
This guest: Standard as you can see except for:
Set-VMProcessor -VMName <VmName> -ExposeVirtualizationExtensions:$true"
No second VHDX yet.
Server 26040 / Server 2022 VM PREPARATION:
- Standard install, C: = 30720 MB during setup.
As for Server nVext 26063: This is the VHDX download version since the .ISO download does not boot. See here: https://techcommunity.microsoft.com/t5/windows-server-insiders/iso-build-26063-fails-to-boot-in-hyper-v-but-why/m-p/4067708
- GPEDIT.MSC: Allow empty passwords.
- Windows/Microsoft updates right after installation.
- Activate Role Deduplication, do not configure deduplication.
- Add VHDX for second drive, add folder \Hyper-V.
- copy two test VMs, here those two Server 2012 R2 VMs with update state of 6th June 2022.
This is the export you see now.
These are the steps to reproduce the issue from here on, see screenshots attached / youtube video link:
- Activate Hyper-V in this VM. Not virtual switch needed.
- Activate deduplication for second volume, default setting. Template DOES NOT MATTER, all three expose the issue.
IF you started the VMs before activation deduplication you will have to set the "deduplication files older than" to "0 days".
- Run from Powershell: Start-DedupJob -Volume 😧 -Type Optimization -Full -Wait
- Wait until dedupe is finished
- Check with Get-DedupStatus | fl whether deduplication actually did the job. Expected saving is above 40%, usually above 50%.
- Import the two machines from D:\Hyper-V into Hyper-V Manager. The network cards don't need to be connected.
- Start those two Test VMs. Check folder "Updates-for-offlineinstall-test". they contain updates for offline installing.
Or simply run Windows updates, if you connected the network.
- Run the updates on both machines simultanously, not one after the other.
- Corruption occurs on deduplicated .vhdx files which then get written to.
My test result (here Server 2022 with VDI template):
- https://joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4
- Both machines may blue screen during Windows Updates or show weird errors. Both may run into a recovery boot loop.
- Both have defective filesystems, if they manage to boot instead of getting stuck in boot loop.
- After the blue screen desaster: Start-Dedupjob -Volume <volume> -Type Optimization -Full -Wait may not work any more.
- Get-DedupVolume | fl still shows valid statistics if Start-Dedupjob fails.
- CHKDSK /f may show file system damage. Usually <NUMBER>.ccc and <NUMBER>.cd files of the deduplication
chunk store which usually resides in \System Volume Information\Dedup\ChunkStore\{UNIQUE IDENTIFIER}.ddp\Data
Counter verification:
- Run Windows Updates on those VMs without deduplication. It will work fine.
I tried to replicate the issue using powershell/.NET methods [System.IO.File]::* to create and modify
30+ GB testfiles, but that did not provoke the issue. The checksums were always correct.
How I discovered that bug:
I have a Server 2019-nested-vms testserver with many test-os variants installed there. I upgraded that VM to Server 2022. This is where I ran into the issues with windows updates within the nested VMs, and then traced it down to Server 2022 corrupting the filesystem.