SOLVED

ISSUE - FIXED - Hyper-V VM In-Place Upgrade (IPU) WS 2022 to WS 2025 b26063 fails

MVP

Host: WS b26052
VM: WS 2022 20348.2227 

Target OS: WS 2025 26063.1000

IPU, dynamic update enabled
Flightsigining: on
Error: Rollback after upgrade online phase on first reboot. 

SetupDiag: processing rule: FindRollbackFailure.
.....
Error: SetupDiag reports rollback failure found.
Last Phase = Finalize
Last Operation = Cleanup external drivers after installation
Error = 0xC1900101-0x20017
LogEntry:

Refer to "https://docs.microsoft.com/en-us/windows/desktop/Debug/system-error-codes" for error information.

SetupDiag found 1 matching issue.

Any ideas? Quite unusual for Hyper-V VMs.

17 Replies

installed a fresh VM and will try to reproduce it. 

So far when continuing the setup after enable flighting. setup comes up with the "Ready to Install Windows 11"

 

 

Karl_WesterEbbinghaus_1-1709585085015.png

 

Same result. Does WS 2025 require (v)TPM? From techcommunity answer I understood that's only on later HW but not for VMs.

 

Karl_WesterEbbinghaus_0-1709585889145.png

 

So I have now:

  • enabled vTPM and VBS / Core Isolation on the VM. This didn't change a thing.
  • The procedure is a as follows:
    • online phase of the setup runs well until VM restart
    • Then there's a short shutdown (about 3 seconds) -
    • NO applying phase during shutdown and no applying phase during startup
    • The old OS get's immediately booted on first reboot. You do not even see the "rollback" to the old version
    • bcdedit will have a leftover entry from setup


This behaviour is very strange, I am open for further diags if you like. Thanks for your time!

This one seems to be very reproducible.
The host OS is now WS b26063 but no change from b26053 when I first posted it.

This is creepy - this is way too much "client creep into server OS"...
Hi, Karl,

This is quite odd - we do Server 2022 -> Server 2025 upgrades often in our automation. So, I'd like to figure out what is different.

A few clarifying questions:
1. What generation is the VM?
2. How many GB of RAM? (Doubtful it matters, just checking)
3. Where are you getting the WS2022 image?
4. And from where are you getting the WS2025 image to upgrade to?

I'd like to avoid coming back and saying, "well, I couldn't reproduce it."

Oh, please also include which edition of Server you are using.

 

I tried the most basic Server 2022 -> Server 2025 upgrade scenario on Hyper-V.  I used a Gen 2 VM with 4GB of RAM.  I started with a vanilla image of Datacenter with Desktop Experience, Windows Server 2022, with February patches, and then attached an ISO of WS 2025 26063 and ran Setup.exe.  It worked fine.

 

MSBernstein_0-1710200240844.png

 

So, let's figure out what's different in your setup.

 

@MSBernstein I am thankful you are investigating this.

If you like we can have a remote session to figure this out. 

 

I am using mslab (github) with Windows Server 2025 on the metal / host OS and Windows Server 2022 guest VMs. The ISO Origins from 2025 Insider portal and WS 2022 from latest Visual Studio subscription. So no VL / EVAL.

You can contact me via Teams if you like, for a remote session, also if you have questions on mslab. 

@MSBernstein could this note from 26080 be related to the issue we're discussing here?

"If you are validating upgrades from Windows Server 2019 or 2022, we do not recommend that you use this build as intermittent upgrade failures have been identified for this build."

Perhaps, but you saw the same issue on 26053.

I looked up what MSLab is, which I had not been familiar with. I had been running the repro by hand. I'll plan to retry it with MSLab a little later in the week. I do want to get to the bottom of this.

I'm working through the process of doing this in MSLab. Based on the pictures you documented, I assume it is the 'DC' VM that I need to update - but I am getting a strange complaint about an additional step I need to take for the domain controller. Did you see this as well? Is there a reasonable way to address this?  (I can see that the link documents a set of steps, but this seems rather far off the track of a simple upgrade.)

MSBernstein_0-1710370337362.png

 

@MSBernstein thank you for going the extra mile with mslab. as there is even a session on this on WS summit supposed it is still supported.

When running Deploy.ps1 with the default labconfig it will create the base, virtual disks templates with WS2022 core and GUI, and a DC template VM

1. please rename the existing labconfig file
2. and create a new one with code below
3. run deploy.ps1

this will create the "lab" with the DC and 2 VMs used for Management (GUI) and WAC (Core)
plain and naked and AD join them.
As we do not need the azure stack hci hosts for this lab these are # out.

We can play the upgrade repro on these.
DC IPU is possible but in fact you need a cmdline command for forest prep executed from the ISO tools folder. This is by design.

At the time of writing I am updating WS 2022 to b26080. Online phase was always ok. 

This is the management VM config (vTPM could be optional) does not cause it. Just adopting it from what I learned these days about new security features.

 

 

 

 

#basic config for Windows Server 2022, that creates VMs for S2D Hyperconverged scenario https://github.com/Microsoft/MSLab/tree/master/Scenarios/S2D%20Hyperconverged

$LabConfig=@{AllowedVLANs="1-10,711-719" ; DomainAdminName='LabAdmin'; AdminPassword='LS1setup!' ; DCEdition='4'; Internet=$true ; TelemetryLevel='Full' ; TelemetryNickname='' ; AdditionalNetworksConfig=@() ; SecureBoot=$true ; VMs=@() ; CustomDnsForwarders=@("1.1.1.1","1.0.0.1")}



#Azure Stack HCI 22H2
#1..2 | ForEach-Object {$LABConfig.VMs += @{ VMName="AzSHCI$_" ; Configuration='S2D' ; ParentVHD='AzSHCI22H2_G2.vhdx' ; HDDNumber=4 ; HDDSize=1TB ; MemoryStartupBytes=1GB; VMProcessorCount=4 ; vTPM=$true}}

#Azure Stack HCI 22H2 with nested virtualization
#1..2 | ForEach-Object {$LABConfig.VMs += @{ VMName="AzSHCI$_" ; Configuration='S2D' ; ParentVHD='AzSHCI22H2_G2.vhdx' ; HDDNumber=4 ; HDDSize=1TB ; MemoryStartupBytes=32GB ; VMProcessorCount=8 ; vTPM=$true ; NestedVirt=$true}}

#Windows Admin Center in GW mode
$LabConfig.VMs += @{ VMName = 'WACGW' ; ParentVHD = 'Win2022Core_G2.vhdx'; MGMTNICs=1}

#Management machine
$LabConfig.VMs += @{ VMName = 'Management' ; ParentVHD = 'Win2022_G2.vhdx'; MGMTNICs=1}

 

 

 

 



Karl_WesterEbbinghaus_0-1710377565992.png

 

very unfortunately no change for Windows Server 2022 Datacenter GUI to 26080
C1900101 - 0x20017

Host OS is now 26080
Guest OS is still 2022 > 26080

Next steps:
trying IPU on WS 2022 Core (below)
trying a fresh VM, . just to make sure it is not an RSAT tool or other dependency on this box.

tested Windows Server 2022 Datacenter Core to 26080
Tried uprade the core upgrade but this fails even earlier after confirming to keep files and settings in the wizard, could be related to the known issues in release notes.

Error when IPU 2022 Datacenter to WS 2025 b26080 Core

SetupDiag: processing rule: FindAbruptDownlevelFailure.
.....
Error: SetupDiag reports abrupt down-level failure.
Last Operation:
Error: 0xC1900220 - 0x4000D
LogEntry: 2024-03-14 02:34:32, Error MOUPG CSetupResponseTemplate<class ICompatActionResponse>::InitializeRoutine(84): Result = 0xC1900220[gle=0x00000002]

Karl_WesterEbbinghaus_0-1710380130861.png

 



new VM 23048.2159
@MSBernstein last update for this tonight, same behaviour
So opposite to you, (unfortunately) I can replicate the problem.
Hope you'll have success to repro it, too. Here it appears to be absolutely reproducible.

One to note, after setup completed to 100% in the online phase, now with 26080 recognize a small difference vs 26063.

The first boot after IPU takes several minutes showing (old) circling balls, where as I would be expecting to see a brief "preparing drivers" + Apply phase with "You are x% there yet - do not shut down..." But this is not showing up just very long boot phase with 26080 now.

With 26063 the reboot was quick (about 2 seconds) and it directly booted into the "wrong" bcdedit as it appears.

With 26080 it seems it is picking the setup bcdentry

in both cases each IPU attempt would generate a new bcdentry. So maybe there is the culprit.
It does not initiate a real visible rollback and the setup bcd remains.

Karl_WesterEbbinghaus_1-1710381839010.png

 

Hyper-V Host settings all default, except on host HCVI / VBS is enabled via Defender Security Center (Core Isolation). And off the bare defaults also vTPM is activated on the Gen2 Secure Boot guest VM.

best response confirmed by Karl_Wester-Ebbinghaus (MVP)
Solution
Update on this investigation:

The Windows Boot team looked over logs from my repro and noticed that the upgrade sequence depends on creating a large (500MB) RAM disk and that ramdisk creation is failing due to not enough memory. Since the documented recommendation for Windows Server with Desktop Experience is 2GB, I turned off Dynamic Memory and set the RAM to 2048 GB.

This resolved the issue for me - in-place upgrade now succeeds.

Karl, can you please see if that same mitigation works for you?

If so, I can follow up with the MSLab owners to see if they can change the memory configuration.

Thanks,
Michael

@Karl_Wester-Ebbinghaus 

 

Good morning,
I managed to do an in-place migration from 2022 to 2025 by modifying the boot store:
Bcdedit /set flightsigning on
Preview certificates are not the same as RTMs.
Sincerely

Hi Luc, the flight signing is enabled. We are looking at a different issue as Michael outlined in the posting above.

Hi @MSBernstein this did the trick.

But only for GUI. Core again fails way before as outlined in a previous reply.

for GUI we seem to have a new issue when dynamic memory is enabled. 
The extra mile would be if this is 2025 specific or even happened before. It is very common to have dynamic memory enabled. And it is set to 512 min / 512 startup mem here. 
It would be great if setup could check this when checking the requirements or we have a more transparence error message when it fails. 

edit: another test with dynamic memory on, showed that dynamic memory itself is not the issue. 
Using dynamic memory with 1 GB startup memory, 512 GB min memory works fine with the upgrade, so it is certainly the startup memory that should be 1 GB or more.


1 best response

Accepted Solutions
best response confirmed by Karl_Wester-Ebbinghaus (MVP)
Solution
Update on this investigation:

The Windows Boot team looked over logs from my repro and noticed that the upgrade sequence depends on creating a large (500MB) RAM disk and that ramdisk creation is failing due to not enough memory. Since the documented recommendation for Windows Server with Desktop Experience is 2GB, I turned off Dynamic Memory and set the RAM to 2048 GB.

This resolved the issue for me - in-place upgrade now succeeds.

Karl, can you please see if that same mitigation works for you?

If so, I can follow up with the MSLab owners to see if they can change the memory configuration.

Thanks,
Michael

View solution in original post