B20292 cpu spikes since b20292 due dedup
Since the upgrade to b20292 the server computer regularly has high cpu use Spikes (hear this due turning up fans) + TM Caused by dedup feature running ever since in this config. There is not much data changes and dedup optimize for VDI / Hyper-V / vhdx Anyone else. I will upgrade to the latest tomorrow and report back. Important to note that the cpu hog in TM is not tied to dedup task but remaining unvisible for the user think so procmon might help. Volumes are either a mirrored Storage Spaces or Single volumes. All with lastest ReFS. Anyone else?
Server 2022 and Server vNext build 25931 corrupts filesystem with deduplication profile HyperV/VDI
Actually I should double post this in "Windows Server for IT Pro" and "Windows Server Insider", but would this be rude? Both Server 2022 and the newest Server vNext build 25931 have following behavior: The deduplication profile HyperV, called VDI in GUI, can corrupt data, can corrupt the filesystem. Server 2019 show no such behavior. (Edit: Server 2016 and 2012 R2 are fine too) How to reproduce as test: Create a VM with nested-V, add a second virtual drive where you copy some Hyper-V VMs. You can try on a physical host. I would have done that if I'd have a spare machine for that. Activate deduplication with profile "VDI" on that second drive. Start-DedupJob -Volume <the volume> -Type Optimization -Full -Wait Start those nested VMs, do some actions within those nested VMs, like running Windows Updates. Shut them down, if they make it this far. Don't be surprised if they blue screen during Windows Updates. Right after that run chkdsk /f again. It MAY show file system damage. Usually <NUMBER>.ccc and <NUMBER>.cd files of the deduplication chunk store which usually resides in \System Volume Information\Dedup\ChunkStore\{UNIQUE IDENTIFIER}.ddp\Data I have "ready to reproduce" VM-packages there which use the freely available EVAL versions of Windows Server. Unpack, import into Windows 11 Hyper-V, in my case a Ryzen 5950x (with 64 GB ECC RAM). I reproduced that issue on an i7-4960x/Server2022 host as well. Read the prepared information file on the desktop. Server 2022 as base (26 GB): https://joumxyzptlk.de/tmp/microsoft/S2022-Nested.7z Server vNext build 25931 as base (22 GB): https://joumxyzptlk.de/tmp/microsoft/SNEXT-25931.7z Video from July what is happening here: https://joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4 Any advice here? I've been battling that for Server 2022 for three month now, it first appeared when I upgraded the Server 2019 VM "for nested VMs only for testing" to Server 2022 in June and took me a while to pin down what was going on.vNext 25951 deduplication profile "VDI/Hyper-V" data corruption (applies to Server 2022 too BTW)
Hello Community! Edit: It seems to be a racing condition. To reproduce it needs a rather modern and fast CPU, else the problem not show up at all. My i7-4960x is at the border, the problem does not always show up. Problem: Using the deduplication profile "VDI", internally called "hyperv" causes data corruption and, in some cases filesystem errors on host-level. Ready to reproduce package with a clean fresh ENGLISH vNext install: https://joumxyzptlk.de/tmp/microsoft/SNEXT-25951.7z Follow the text file on the desktop. If you skip the "configure deduplication as VDI for drive 😧 + Start-Dedupjob" step you won't have issues, it will work fine. What happens is the same from current Server 2022 and vNext 25951: https://joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4 Reproduced on Ryzen 5950x with ECC RAM and i7-4960x, quite different hardware. The insider feedback hub is still broken... Sadly. EDIT: How to reproduce (contents of text file on desktop of that VM): Creation host: Ryzen 5950x, 64 GB ECC RAM, Windows 11 21H2. (Issue reproduced on host: intel i7 4960x, 32 GB RAM (non ECC), Server 2022 on host.) This guest: Standard as you can see except for: "Set-VMProcessor -VMName SNEXT-25931 -ExposeVirtualizationExtensions:$true" and "Allow network spoofing" No second VHDX yet. Server 25931 VM PREPARATION: - Standard install. - Two GPEDIT.MSC: Allow empty passwords, and Lanman Workstation "Enable insecure Guest logons", which makes copying those test VMs easier on my home network. - Be in a network which allow internet access for Windows Updates, including the VMs. - Windows/Microsoft updates right after installation. - Activate Role Deduplication and Hyper-V, do not configure deduplication. Reboot. - Add VHDX for second drive, add folder \Hyper-V. - copy two test VMs, here those two Server 2012 R2 VMs with update state of 6th June 2022, import them in Hyper-V. - Remove Hyper-V role, cause dependent on the Host CPU different virtualization seem to be used up upon activation of that role. - Update 2. Sept 2023: In-Place upgrade to Server vNext 25941. This is the export you see now. These are the steps to reproduce the issue from here on, see screenshots attached / youtube video link: - Activate Hyper-V in this VM. Do not configure Switch during that step. Reboot. - Import the two machines in drive D:\Hyper-V into Hyper-V Manager. - Configure the virtual switch. You may have to untick the the "Hyper-V Extensible Virtual Switch" in the network properties of the adapter, a misbehaviour of the host OS. Connect the nested VMs to that virtual switch. - Activate deduplication for second volume, choose template "VDI", no adjustment of anything. - Run from Powershell: Start-DedupJob -Volume <volume> -Type Optimization -Full -Wait - Wait until it is finished - Start those two Test VMs, and try to run Windows update. This step is important, 'cause the corruption occurs on deduplicated files which then get written to. My test result: - https://joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4 - Both machines may blue screen during Windows Updates or show weird errors. Both may run into a recovery boot loop. - Both have defective filesystems, if they manage to boot instead of getting stuck in boot loop. - After the blue screen desaster: Start-Dedupjob -Volume <volume> -Type Optimization -Full -Wait may not work any more. - Get-DedupVolume | fl still shows valid statistics if Start-Dedupjob fails. - CHKDSK /f may show file system damage. Usually <NUMBER>.ccc and <NUMBER>.cd files of the deduplication chunk store which usually resides in \System Volume Information\Dedup\ChunkStore\{UNIQUE IDENTIFIER}.ddp\Data Counter verification: - Run Windows Updates on those VMs without deduplication. It will work fine. How I discovered that bug: I have a Server 2019-nested-vms testserver with many test-os variants installed there. I upgraded that VM to Server 2022. This is where I ran into the issues with windows updates within the VMs, and then traced it down to Server 2022 corrupting the filesystem on the host level. kind regards, Joachim Otahal GermanyServer vNext 26040 and Server 2022 deduplication data corruption
Following scenario gives me, easy to reproduce on various newer comptuer with SSD-based storage, data corruption with deduplication. Server 2019 does not have that issue. Reproduced on several Intel and AMD machines with 16 or more (real, non SMT) cores. Can be reproduced with HDD storage on i7-4960x CPU too, but with less that 50% probability. Host: Server 2019, Server 2022 or Windows 11. 1-st level VM: Server 2022 or Server vNext 25xxx or 26xxx, nested-v active for that VM. Have a drive 😧 for your nested VMs. 2-nd level Guests: More than one, since at least two running concurrently are needed to trigger the issue. OS does not matter, but easiest are Window VMs running Windows Update. As long as deduplication for 1-st level VM is not activated everything is fine. But if you activate deduplication for drive D:, where the nested VMs reside, you will get data corruption, in some cases even so bad that the dedup-chunkstore shows up as filesystem defect. Ready-to-reproduce packages: Those are created on a Server 2019 host, so Server 2019 and ANY later OS should be able to import them. No network connection needed, not even a Hyper-V switch. Needs a computer with 32 GB RAM and 'round 100 GB of free space on a non-system drive. Unpack to non-system-drive, import into Hyper-V, start the VM. Then follow the text file on the desktop. Result as video, here Server 2022: https://www.joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4 Package with Server vNext 26040: https://www.joumxyzptlk.de/tmp/microsoft/Server-vNext-26040-nested-dedup-problem-export-from-S2019-host.7z Package with Server 2022: https://www.joumxyzptlk.de/tmp/microsoft/S2022-nested-2023-09-30-exported-from-S2019-host.7z Package with Server 2019, just as "See, Server 2019 works fine!" counter proof: https://www.joumxyzptlk.de/tmp/microsoft/S2019-nested-2023-10-07-exported-from-S2019-host.7zServer vNext 25987, 26010 (and Server 2022) deduplication corruption in nested-V setups
The deduplication corruption in the case of nested virtualization in Server 2022 and Server 25987 is still there. The package to reproduce for Server 25987: https://joumxyzptlk.de/tmp/microsoft/s-vnext-25987-dedup-problem-2023-11-03-exported-from-S2019.7z Server 2022 variant: https://joumxyzptlk.de/tmp/microsoft/S2022-nested-2023-09-30-exported-from-S2019-host.7z Host OS can be Server 2019, Windows 11 21H2 or Server 2022. CPU must be reasonable fast (i.e. from somewhere 'round 2018), support nested-v together with the Host OS (AMD and Intel CPU are fine). Host should have at least 24 GB RAM. The the "ready to reproduce package" must be unpacked to 😧 or higher drive letter which should be at least SATA SSD storage. Else the problem may not show up as predictable as it does here. These packages were exported from a Server 2019 host, HP DL 380 dual Xeon 6228R with 1 TB ECC RAM, so it is easier to reproduce since Exports from Windows 11 don't import so well on other hosts. Originally the bug showed up on a Ryzen 5950x with 64 GB ECC RAM. Reproduced with less than 50% chance on an i7-4960x 32 GB RAM with HDD storage, more than 50% on SSD storage. Server 2019 variant: Does exist, but there is no data corruption. How to use: Once unpacked to Non-System-Drive SSD storage import the the VM and follow the Textfile on the desktop. In short: Activate Hyper-V in that VM, default settings. Configure deduplication with default settings any profile in that VM for D:, start a dedup run. Check dedup rate. Should be above 50%, usually it is around 55%. Import those two VMs on drive 😧 of that VM: Start them both in parallel, and execute the Offline-Windows-Updates stored on the desktop. You won't make it past the first one.
