User Profile
Joachim_Otahal
Iron Contributor
Joined Jan 15, 2021
User Widgets
Recent Discussions
Re: 26063 deduplication data corruption is still there.
Al little update on this: I am in contact with others to reproduce this, and we came across the "write cache "setting of the drive. But even with the checkmar removed it still happens. We are trying to narrow it down, for example minimum CPU GHz speed and similar things. Sadly I cannot do such a test in the Xeon 6226R on bare metal instead of nested-V, that other machine is not mine.Re: 26063 deduplication data corruption is still there.
Sorry for my late answer: Yes. Base is still the AMD Ryzen 5950x with 64 GB ECC RAM (real ECC, DDR4), but as I previously tested: Happens on Intel too. Requirement: Somewhat fast storage and fast CPU, should be > 3 GHz class. The Volume where the deduptest is made on MUST be on a different physical disk than the System. If it is on the same disk as the System it won't reproduce. SSD is recommended, here a Samsung SATA 860 EVO 4 TB is fast enough, 'cause reproduding it on slower storage takes several attempts. The volume is formatted as NTFS. Drivers are default-MS supplied, not swapped or updated against OEM drivers. The drive which I run the repo on is the fresh formatted H: Volume, but Volume F: has a lot of other VMs which all show the same result, down to the Windows NT 3.51 VM, even though it is a storage-space JBOD which uses part of the System drive. (Those VMs are backup up of course, so no real harm done). Here as text to the screenshots (msinfo32 output is available on private contact): Step 0 DedupTest Shot hardware overwiew.PNG Step 1 DedupTest Volume H fresh formatted, NOT A SYSTEM DISK, in this case Samsung SATA SSD. ANY non system disk volume exposes this.PNG Step 2 DedupTest Server2025 26100.1742.240906-0331 VMs fresh installed, PW set to empty, Updates ready on desktop, VMs exported, about to enable dedup.PNG Step 3 DedupTest Dedup done.PNG Step 4 DedupTest before Start updates.PNG Step 5 DedupTest UNEXCPECTED Server 2025 core crashed even bore starting updates.PNG Step 6 DedupTest Trying to update the two GUI installs too, crash or strange error.PNG Step 7 DedupTest Counter-test. Reformatted volume H, imported VMs from Step 2, before starting to update.PNG Step 8 DedupTest Counter-test. Updates started.PNG Step 9 DedupTest Counter-test. 15 minutes later - all three updating in parallel.PNG Step 10 DedupTest Counter-Test. Reboot after update.PNG Step 11 DedupTest Counter-Test. Reboot after update.PNG Step 12 DedupTest Counter-Test. Everything fine after reboot when having non dedup active.PNG The "Ready to import Eval-Server2025 VMs, with updates prepared on Desktop" are here (still uploading, upload will be finished 1200 hours UTC): https://joumxyzptlk.de/tmp/microsoft/Deduptest-Server2025-VMs-ready-to-import-to-save-time.7z kind regards, Joachim Otahal From here on: Counter-Test WITHOUT dedup, fresh formatted H: drive.Re: 26063 deduplication data corruption is still there.
If you are still reading this: I changed my main machine installation from Windows 11 24h2 to Server 2025 in september 2025. Retesting this deduplication corruption showed: I can reliably reproduce this WITHOUT Nested-V. I know the "Data Deduplication isn't supported in a nested virtualization scenario." was ONLY added because of me, and now I am back, reproducing this on metal without nested-v... https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/interopRe: Nested-V Dedup corruption 26100.1742 and Insider 26296.5001 (and Server 2022) is still there.
MSSWRahman I redid it fresh. Host is Windows 11 24H2, Ryzen 5950x ECC RAM. L1 VM fresh Server 26100.2033, and the two L2 VMs fresh too, though I accidently used build 26296 there :D. The L2 guest VMs are identical, the second is a clone of the first. One big difference: Instead of running Windows Update a powershell script "Server2025-and-Server2022-dedup-corruption-repo-test.ps1", used in the L2 guests, is enough to trigger it. Script is at https://github.com/Joachim-Otahal/PowerShell . Has a little menu, create a 4 GB file, modify it, check whether the CRC is the expected value. In some cases running Step 1 in one of the L2 VMs alone is enough o crash both, on other cases the "step 1, step 2, step 3" had to be repeated a few times in both L2 VMs. There is no memory.DMP, it did not seem to survive the chkdsk.exe run. Even cmd.exe, dir c:\*.dmp /b /s /a shows no .DMP file anywhere. The chkdsk output and events of L2 guests, as far as I could capture them: https://joumxyzptlk.de/tmp/microsoft/Server2025-dedup-corruption-Events-CHKDSK-etc.7z The whole package, L1 guest and the L2 guests BEFORE activating deduplication. I had it on the second D : 😧 SSD, not the boot SSD. (Can anybody tell me how to turn of that over-eager auto smiley stuff? Feels like over-eager MS office autocorrect, correcting even when turned off...) https://joumxyzptlk.de/tmp/microsoft/Server2025-dedup-corruption-BEFORE-actviating-dedup.7z The whole package, L1 guest and the L2 guests AFTER they crashed, which should include "everything". https://joumxyzptlk.de/tmp/microsoft/Server2025-dedup-corruption-AFTER-the-crash.7z If you import those in Server 2025 or Windows 11 24H2 on an Intel machine: You may have to deactivate and reactivate Hyper-V on the L1 guest, else it may not work. There seems to be some differences on that regard, which are only taken into account when activating Hyper-V. The culprit, from my point of view, is still the L1 guest, either Server 2022 or Server 2025, not the host OS below or the second level guests. Using Server 2019 as L1 guest with the nested VMs inside works fine, whether the real hardware host is Server 2019, Windows 10, Server 2022, Windows 11 or Server 2025 does not make a difference. If I could get access to .ISOs from vNext between Server 2019 and Server 2022 I could narrow it down at bit when that bug was introduced. For the "fun": A few screenshots. This last one is with the chkdsk run in both VMs when booted from Server 2022 ISO into recovery "repair computer" "cmd.exe" mode.If only MS would take more care of details...
Currently setting up the "deduplication-corruption repo with newest pre-release Server 2025 as L1 and L2 VMs". And then I see this: It is still the problem a lot of "Client only, never needed or wanted on a server by default" stuff creeps into the Server UI. Another example of today is this bad default setting: Microsoft could do so much better if it would take more care of details, reduce the Marketing/Public-Relations (previously known as Propaganda Departement) budget and invest more in actual quality. Please Dave Cutler, you have to rescue Windows - AGAIN, like you did when XP was released (i.e. when it was SP0, I remember how bad it was at first)...Re: Nested-V Dedup corruption 26100.1742 and Insider 26296.5001 (and Server 2022) is still there.
MSSWRahman Thank you for your reply! Which profile: Does not matter, happens with all three (as described in previous. postings..). "Deduplicacte older than 0 days" is only needed if the L2 VMs are fresh set up, before starting any Windows Update. If you import existing VMs without starting them, which have not been touched for > 3 days, you don't need "0 days". How to import the L2 VMs: Attach the VHDX, open Hyper-V GUI, import "register directly". Exactly as in the video. As for sharing the crashed: I will follow your additional preparation instructions, and then provide you with the complete VM package "right before dedup", "right before the final repo step", and "after L2 VMs died", which includes the L1 OS and the (crashed) L2 VMs. Host will be Win11 24h2, Ryzen 5950x, ECC RAM, which is important since the L2 VMs does not work on Intel hosts in my tests unless you deactivate and reactivate Hyper-V in the L1 guest. Do you have a special request regarding what OS the 1st or 2nd level guests have to be? Else I'll use "Server 2025 with updates" for L1 guest, and "Server 2012 R2 German and English" for L2 since the latter are already there. regards, Joachim OtahalNested-V Dedup corruption 26100.1742 and Insider 26296.5001 (and Server 2022) is still there.
Nested-V, host can be Server 2019, Server 2022, Windows 10, Windows 11. Can be Intel or AMD. SSD recommended, 32 GB minimum. 1st level guest is Server 2022, 2025 or the newest vNext, dedup on drive D : active where the nested VMs reside. And they experience data corruption, and in some rare cases even on the D : drive of the 1st level guest. Does not happen if 1st level guest is Server 2019, rock solid then. (I have to use D : with a space, because that over-eagerness of office products to force auto-guess and auto-correct things crept into this forum too, converting it to a smiley!) The video, in that case Server 2022 on Win11 as host: https://www.joumxyzptlk.de/tmp/microsoft/S2022-Nested_Deduplication_VDI-Hyperv_profile_kills_filesystem_v2.mp4 See here my previous posts: https://techcommunity.microsoft.com/t5/windows-server-insiders/server-vnext-26040-and-server-2022-deduplication-data-corruption/m-p/4047321 Here: https://techcommunity.microsoft.com/t5/windows-server-insiders/server-2025-hyper-v-deduplication-corruption-is-still-there/m-p/4136286 Here: https://techcommunity.microsoft.com/t5/windows-server-insiders/26063-deduplication-data-corruption-is-still-there/m-p/4067827 Still perfectly reproducible, in this case the host was just upgraded to Win11 24h2. What is new and different? I know you (i.e. Microsoft) not only does have a bug entry, it got reproduced, and in one case I even know the CPU type to be an E5-1650 where this bug got reproduced (though with ~50% chance there since that CPU is old and slow). I even know it got sorted into "existing features fixed" category this bug belongs since deduplication is not a new feature. Here the two screens from todays re-test: Happy fixing! I am lucky I never used deduplication on a CSV in a Hyper-V Cluster - I was just about to do in 2023, but then this happened. kind regards, Joachim OtahalRe: Active Directory Advanced Threat Hunting - Tracing the cause of account lockouts and password errors
In my experience with large security logs adding "ProviderName" speeds up the query significantly: -FilterHashtable @{ProviderName="Microsoft-Windows-Security-Auditing";LogName="Security";ID=4625} If only failed logins are needed I add "Keywords=4503599627370496" aka "Audit Fail": -FilterHashtable @{ProviderName="Microsoft-Windows-Security-Auditing";LogName="Security";ID=4625;Keywords=4503599627370496} Only -FilterXPath syntax is still faster, and the "[Provider[@Name='Microsoft-Windows-Security-Auditing']]" helps speeding that up too. But I had not yet the need to use -FilterXPath for those 4740/4625 security events, only for NTLM logging where the amount of log entries is huge.Other things to improve (to add to the survey)
If you open an mmc console, like lusrmgr.msc gpedit.msc gpmc.msc dsa.msc etc, every administrator, since the year 2000, moves the divider between the tree pane and the list pane to the right. If you look at every linkedin video you can find, every administrator does that move. Every time. Between three and six seconds wasted. For the last ~24 years. And the divider is too thin to grab on slow remote sessions, costing more time. Updating that default in mmc.exe, to make the tree pane at least double the width and that divider two pixels bigger, would be great. My suggestion would be 250% the current width. For Edge on Windows Server, it should skip the enforced "do you want to use your data?" at the start. We administrators log on to many MANY servers, often after weeks or month of not logging on, and about every time that wizard comes up and costs us about 20+ seconds. The default start page should be an empty page for data protection reasons and not Bing, let alone that many servers are not allowed to go on the internet anyway. (The same applies to clients as well, every time wasting time with that Edge questions, and no way around, even if we only log in once and never again for the whole life time of that client). All cloud features (OneDrive, including the notification to backup to OneDrive, Azure/Entra tool etc) should be not installed, but available as feature/role/capability without needing internet to install those.
