Forum Discussion
26063 deduplication data corruption is still there.
From Server 2022 up to this newest 26063 build, they all have the same problem, as described here: https://techcommunity.microsoft.com/t5/windows-server-insiders/server-vnext-26040-and-server-2022-de...
MSBernstein
Microsoft
MicrosoftI checked on the bug report. As far as I can tell, the engineering team is still working to reproduce the issue.
The easiest way is to use a Server 2019 or Server 2022 or Windows 11 Hyper-V host (real metal, not virtual). Somewhat current CPU, whether AMD Ryzen 2xxx or newer or intel Gen 8 (or their respective Threadripper / Epyc / Xeon counterpart ), SSD, 32 GB RAM. I could repo this in my old i7-4960x too, but not every time but rather every third time.
Then use these prepared packages, unpack on D:, import into Hyper-V. They are completely self contained repos.
The Server 2022 repo package: https://www.joumxyzptlk.de/tmp/microsoft/S2022-nested-2023-09-30-exported-from-S2019-host.7z
The Server 26040 repo package: https://www.joumxyzptlk.de/tmp/microsoft/Server-vNext-26040-nested-dedup-problem-export-from-S2019-host.7z
The follow the text file on the desktop.
To give you the full information: These are the contents of the text file on the desktop of that VM:
############################################################
Creation host: Ryzen 5950x, 64 GB ECC RAM, Windows 11 21H2.
----------
This guest: Standard as you can see except for:
Set-VMProcessor -VMName <VmName> -ExposeVirtualizationExtensions:$true"
No second VHDX yet.
Server 26040 / Server 2022 VM PREPARATION:
- Standard install, C: = 30720 MB during setup.
As for Server nVext 26063: This is the VHDX download version since the .ISO download does not boot. See here: https://techcommunity.microsoft.com/t5/windows-server-insiders/iso-build-26063-fails-to-boot-in-hyper-v-but-why/m-p/4067708
- GPEDIT.MSC: Allow empty passwords.
- Windows/Microsoft updates right after installation.
- Activate Role Deduplication, do not configure deduplication.
- Add VHDX for second drive, add folder \Hyper-V.
- copy two test VMs, here those two Server 2012 R2 VMs with update state of 6th June 2022.
This is the export you see now.
These are the steps to reproduce the issue from here on, see screenshots attached / youtube video link:
- Activate Hyper-V in this VM. Not virtual switch needed.
- Activate deduplication for second volume, default setting. Template DOES NOT MATTER, all three expose the issue.
IF you started the VMs before activation deduplication you will have to set the "deduplication files older than" to "0 days".
- Run from Powershell: Start-DedupJob -Volume 😧 -Type Optimization -Full -Wait
- Wait until dedupe is finished
- Check with Get-DedupStatus | fl whether deduplication actually did the job. Expected saving is above 40%, usually above 50%.
- Import the two machines from D:\Hyper-V into Hyper-V Manager. The network cards don't need to be connected.
- Start those two Test VMs. Check folder "Updates-for-offlineinstall-test". they contain updates for offline installing.
Or simply run Windows updates, if you connected the network.
- Run the updates on both machines simultanously, not one after the other.
- Corruption occurs on deduplicated .vhdx files which then get written to.
My test result (here Server 2022 with VDI template):
- https://joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4
- Both machines may blue screen during Windows Updates or show weird errors. Both may run into a recovery boot loop.
- Both have defective filesystems, if they manage to boot instead of getting stuck in boot loop.
- After the blue screen desaster: Start-Dedupjob -Volume <volume> -Type Optimization -Full -Wait may not work any more.
- Get-DedupVolume | fl still shows valid statistics if Start-Dedupjob fails.
- CHKDSK /f may show file system damage. Usually <NUMBER>.ccc and <NUMBER>.cd files of the deduplication
chunk store which usually resides in \System Volume Information\Dedup\ChunkStore\{UNIQUE IDENTIFIER}.ddp\Data
Counter verification:
- Run Windows Updates on those VMs without deduplication. It will work fine.
I tried to replicate the issue using powershell/.NET methods [System.IO.File]::* to create and modify
30+ GB testfiles, but that did not provoke the issue. The checksums were always correct.
How I discovered that bug:
I have a Server 2019-nested-vms testserver with many test-os variants installed there. I upgraded that VM to Server 2022. This is where I ran into the issues with windows updates within the nested VMs, and then traced it down to Server 2022 corrupting the filesystem.
Then use these prepared packages, unpack on D:, import into Hyper-V. They are completely self contained repos.
The Server 2022 repo package: https://www.joumxyzptlk.de/tmp/microsoft/S2022-nested-2023-09-30-exported-from-S2019-host.7z
The Server 26040 repo package: https://www.joumxyzptlk.de/tmp/microsoft/Server-vNext-26040-nested-dedup-problem-export-from-S2019-host.7z
The follow the text file on the desktop.
To give you the full information: These are the contents of the text file on the desktop of that VM:
############################################################
Creation host: Ryzen 5950x, 64 GB ECC RAM, Windows 11 21H2.
----------
This guest: Standard as you can see except for:
Set-VMProcessor -VMName <VmName> -ExposeVirtualizationExtensions:$true"
No second VHDX yet.
Server 26040 / Server 2022 VM PREPARATION:
- Standard install, C: = 30720 MB during setup.
As for Server nVext 26063: This is the VHDX download version since the .ISO download does not boot. See here: https://techcommunity.microsoft.com/t5/windows-server-insiders/iso-build-26063-fails-to-boot-in-hyper-v-but-why/m-p/4067708
- GPEDIT.MSC: Allow empty passwords.
- Windows/Microsoft updates right after installation.
- Activate Role Deduplication, do not configure deduplication.
- Add VHDX for second drive, add folder \Hyper-V.
- copy two test VMs, here those two Server 2012 R2 VMs with update state of 6th June 2022.
This is the export you see now.
These are the steps to reproduce the issue from here on, see screenshots attached / youtube video link:
- Activate Hyper-V in this VM. Not virtual switch needed.
- Activate deduplication for second volume, default setting. Template DOES NOT MATTER, all three expose the issue.
IF you started the VMs before activation deduplication you will have to set the "deduplication files older than" to "0 days".
- Run from Powershell: Start-DedupJob -Volume 😧 -Type Optimization -Full -Wait
- Wait until dedupe is finished
- Check with Get-DedupStatus | fl whether deduplication actually did the job. Expected saving is above 40%, usually above 50%.
- Import the two machines from D:\Hyper-V into Hyper-V Manager. The network cards don't need to be connected.
- Start those two Test VMs. Check folder "Updates-for-offlineinstall-test". they contain updates for offline installing.
Or simply run Windows updates, if you connected the network.
- Run the updates on both machines simultanously, not one after the other.
- Corruption occurs on deduplicated .vhdx files which then get written to.
My test result (here Server 2022 with VDI template):
- https://joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4
- Both machines may blue screen during Windows Updates or show weird errors. Both may run into a recovery boot loop.
- Both have defective filesystems, if they manage to boot instead of getting stuck in boot loop.
- After the blue screen desaster: Start-Dedupjob -Volume <volume> -Type Optimization -Full -Wait may not work any more.
- Get-DedupVolume | fl still shows valid statistics if Start-Dedupjob fails.
- CHKDSK /f may show file system damage. Usually <NUMBER>.ccc and <NUMBER>.cd files of the deduplication
chunk store which usually resides in \System Volume Information\Dedup\ChunkStore\{UNIQUE IDENTIFIER}.ddp\Data
Counter verification:
- Run Windows Updates on those VMs without deduplication. It will work fine.
I tried to replicate the issue using powershell/.NET methods [System.IO.File]::* to create and modify
30+ GB testfiles, but that did not provoke the issue. The checksums were always correct.
How I discovered that bug:
I have a Server 2019-nested-vms testserver with many test-os variants installed there. I upgraded that VM to Server 2022. This is where I ran into the issues with windows updates within the nested VMs, and then traced it down to Server 2022 corrupting the filesystem.
