SOLVED

b26085 - Windows Server 2025 Domain Controller fail to deploy via DSC

MVP

Hi there,
I am trying to deploy mslab using Windows Server 2025 preview VM using Windows Server 2025 OS as a Hyper-V platform.


I believe Jaromir as maintainer of mslab opened an SR. In result he updated the code, plus PS DSC Modules. edit: this issue is unrelated to the running SR.

Still the deployment of the domain controller template will fail during first boot.
This is not specific to b26085, this one is troubling for since WS2025 insider, so it never worked with this branch.

 

Repro steps

  • Install Windows Server 2025 b26085.1 on a physical machine and enable Hyper-V
  • Download and extract mslab
    https://github.com/microsoft/MSLab/releases/tag/v24.04.1
  • run the mslab scripts in order
    1_Prereq
    2_CreateParentDisks
    > select Windows Server 2025 ISO b26085 (WS 2022 ISO will work)
    > skip the MSU dialogue


Domain Controller VM gets created from parent disk image
First boot will fail with following error

 

 

         Creating DC VM                                                                                                          Applying Unattend and copying Powershell DSC Modules                                                                                                                                                                                   PSPath            : Microsoft.PowerShell.Core\FileSystem::J:\mslab_v24.04.1\Temp\MountDir                               PSParentPath      : Microsoft.PowerShell.Core\FileSystem::J:\mslab_v24.04.1\Temp                                        PSChildName       : MountDir
PSDrive           : J
PSProvider        : Microsoft.PowerShell.Core\FileSystem
PSIsContainer     : True
Name              : MountDir
FullName          : J:\mslab_v24.04.1\Temp\MountDir
Parent            : Temp
Exists            : True
Root              : J:\
Extension         :
CreationTime      : 21.04.2024 13:20:16
CreationTimeUtc   : 21.04.2024 11:20:16
LastAccessTime    : 21.04.2024 13:20:16
LastAccessTimeUtc : 21.04.2024 11:20:16
LastWriteTime     : 21.04.2024 13:20:16
LastWriteTimeUtc  : 21.04.2024 11:20:16
Attributes        : Directory
Mode              : d-----
BaseName          : MountDir
Target            : {}
LinkType          :


Path   : J:\mslab_v24.04.1\Temp\MountDir
Online : False


Path   : J:\mslab_v24.04.1\Temp\MountDir
Online : False


PSPath            : Microsoft.PowerShell.Core\FileSystem::J:\mslab_v24.04.1\Temp\MountDir\Windows\Panther
PSParentPath      : Microsoft.PowerShell.Core\FileSystem::J:\mslab_v24.04.1\Temp\MountDir\Windows
PSChildName       : Panther
PSDrive           : J
PSProvider        : Microsoft.PowerShell.Core\FileSystem
PSIsContainer     : True
Name              : Panther
FullName          : J:\mslab_v24.04.1\Temp\MountDir\Windows\Panther
Parent            : Windows
Exists            : True
Root              : J:\                                                                                                 Extension         :                                                                                                     CreationTime      : 21.04.2024 13:20:22                                                                                 CreationTimeUtc   : 21.04.2024 11:20:22                                                                                 LastAccessTime    : 21.04.2024 13:20:22                                                                                 LastAccessTimeUtc : 21.04.2024 11:20:22
LastWriteTime     : 21.04.2024 13:20:22                                                                                 LastWriteTimeUtc  : 21.04.2024 11:20:22                                                                                 Attributes        : Directory                                                                                           Mode              : d-----                                                                                              BaseName          : Panther                                                                                             Target            : {J:\Windows\Panther}
LinkType          :

         Creating DSC Configs for DC

LastWriteTime : 21.04.2024 13:20:23
Length        : 1206
Name          : DC.meta.mof


LastWriteTime : 21.04.2024 13:20:25
Length        : 32600
Name          : DC.mof

         Copying DSC configurations (pending.mof and metaconfig.mof)
         Applying changes to VHD

LogPath          : C:\WINDOWS\Logs\DISM\dism.log
ScratchDirectory :
LogLevel         : WarningsInfo

         Starting DC
         Configuring DC using DSC takes a while.
                 Initial configuration in progress. Sleeping 250 seconds

 

 

Karl_WesterEbbinghaus_0-1713693973428.png

 



Question: 
What could be reasons it keeps failing?
We have no clue / pointers and imho it would be expected and necessary that mslab works with WS2025, despite being preview, too.

Who at Microsoft could afford time to repro and looking into that?

As mslab is also used by Microsoft, I hope anyone could spare time to investigate (with Jaromir and/or me). I personally can spare time for that investigation.

Thank you very much in advance!

12 Replies

repeating the same with a different release like Windows Server 2022, the initial DSC configuration won't cause any issues.

When this issue was reported back in the days, the problem was a bit different.
Back then using WS2025 ISO, the DSC config worked and then the DC failed to boot when being imported as Hyper-V VM from the DC template VHDX. So clearly, now this process fails much earlier, so I cannot rate if the initially reported issue still exist.

Example of DC creation with WS2022 ISO instead of WS2025

 

 

LastWriteTime : 21.04.2024 13:20:25
Length        : 32600
Name          : DC.mof

         Copying DSC configurations (pending.mof and metaconfig.mof)
         Applying changes to VHD

LogPath          : C:\WINDOWS\Logs\DISM\dism.log
ScratchDirectory :
LogLevel         : WarningsInfo

         Starting DC
         Configuring DC using DSC takes a while.
                 Initial configuration in progress. Sleeping 250 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds
                 Configuration in Progress. Sleeping 10 seconds

 

 

 

 

After further investigations, what makes the difference here, is the Hyper-V host OS.
If this is WS2025 then the DSC config will fail to apply for the domain controller VM.
Hi, Karl,
Have you logged an issue with MSLab on this?

And did you disable dyanmic memory, per the discussion in https://github.com/microsoft/MSLab/issues/580?

Also, if you are getting the error, "Windows could not configure one or more system components", we should collect these folders and figure out how to share them to me:
I think we can start with these three folders:
\$WINDOWS.~BT\Sources\Panther
\$WINDOWS.~BT\Sources\Rollback
\WINDOWS\Panther

They will have useful logging information.

I am attempting to reproduce this on my side, but I'd rather get the diagnostic data you have already.

Thanks,
Michael

@MSBernstein Thank you for taking the time looking into this.

The DC VM created by "2_CreateParentDisks.ps1 will have the dynamic memory settings enabled and they are matching the requirements now of #580
Startup: 2048 MB
Min: 2048 MB

I have done the following today: 
- clean install of WS 2025 b26085 for Host OS
- Install-WindowsFeature Hyper-v -IncludeManagementTools -Restart
- started with an new mslab folder
- The error message still appears some seconds after the first reboot of VM DC (WS2025 b26085).

How can I connect to the folders of the VM to collect the logs?
Enter-PSSession (PowerShell Direct) to the VM will fail to authenticate via WinRM even when setting trusted host.

Will repeat and mounting the vhdx to the Host OS.

What's notable: If I do not connect to the VM but watch the process via the Hyper-V preview window, I am seeing the error message and at the same time notice that the CPU load on the VM changes and after a while ~ 4-5 mins the process continues despite the message.


Karl_WesterEbbinghaus_0-1714818246822.png

 

At this point
- turned off the VM, as the configuration process appears to stall
- mounted the VHDX 
- the $Windows.~BT folders don't exist (hidden files enabled)
- Attached Windows\Panther

panther.zip, as requested

Repeated the process and immediately stopped the VM when the error message appeared after the first reboot. again no ~BT folders.
Attached
windows\logs\dism
dism.zip, this might provide some insights what the DSC was about to configure. 

windows\panther
panther_early_turnoff.zip, as requested.

If required, I could also provide the exported VMs. They won't boot by-design as we have terminated the setup phase, so just for accessing more files / logs.

Hope this is helpful.

filed an Issue on gitub as requested:
https://github.com/microsoft/MSLab/issues/588

I have attached Logs_2022 which is the same, then successful procedure using Windows Server 2022 ISO.

Maybe comparing both could give a clue.
windows\logs

windows\panther
Logs_2022.zip

@Karl-WE Thank you!  The panther.zip and panther_early_turnoff.zip files have some interesting content on them. 

 

Particularly:

2024-05-04 03:34:15, Error [0x030149] CMI Callback_CBS_Do_OnlineInstall:Calling startup processing failed 0x80070643

 

I'll get these over to the Deployment team to see what they can glean from them.

 

 

The Deployment team and I went through several iterations on this, but we could not unearth anything from the logs. They mostly say that deployment was interrupted, but they do not have details on which process hung or why.

I retried this scenario with the Windows Server 2025 Public Preview ISO from the Eval Center:

MSLab version 24.04.1
Host OS - en-us_windows_server_2025_preview_x64_dvd_ce9eb1a5.iso
Guest OS - en-us_windows_server_2025_preview_x64_dvd_ce9eb1a5.iso

I was able to get all the way through the scenario and did not see any error messages.

@MSBernstein 

Tested Scenarios:

Test 1 - FAILED:
MSLab version 24.04.1
Host OS: Windows Server 2025 build 26085
Guest OS - en-us_windows_server_2025_preview_x64_dvd_ce9eb1a5.iso WS 2025 build 26100
Result: 2_CreateParentDisks.ps1,
OK  - VHDX Creation 
FAILED - DC creation

Test 2- FAILED:
MSLab version 24.04.1
Host OS: Windows Server 2025 build 26085
Guest OS - en-us_Windows_InsiderPreview_Server_vNext_en-us_26227 WS 2025 build 26227
Result: 2_CreateParentDisks.ps1,
OK - VHDX Creation 
FAILED - DC creation

Test 3 - FAILED:
MSLab version 24.04.1
Host OS: , en-us_windows_server_2025_preview_x64_dvd_ce9eb1a5.iso, Windows Server 2025 build 26100
Guest OS - en-us_windows_server_2025_preview_x64_dvd_ce9eb1a5.iso WS 2025 build 26100
Result: 2_CreateParentDisks.ps1,
OK - VHDX Creation 
FAILED - DC creation and boot

Test 4 - OK:
MSLab version 24.04.1
Host OS: en-us_windows_server_2025_preview_x64_dvd_ce9eb1a5.iso, Windows Server 2025 build 26100, 
Guest OS - en-us_Windows_InsiderPreview_Server_vNext_en-us_26227 WS 2025 build 26227
Result: 2_CreateParentDisks.ps1,
OK - VHDX Creation 
OK - DC creation

@MSBernstein unfortunately I cannot exactly repro the results with the newer release.
Good news though that 26277 as guest on 26100 works. Afterall it seems that the Host OS plays a role, too.

Repeated Test 3 using MSlab 24.05.01 doesn't change anything here.


Test 5 - FAILED:
MSLab version 24.04.1
Host OS: , Windows Server 2025 build 26277
Guest OS - en-us_windows_server_2025_preview_x64_dvd_ce9eb1a5.iso WS 2025 build 26100
Result: 2_CreateParentDisks.ps1,
OK - VHDX Creation 
FAILED - DC creation and boot

Test 4 - OK:
MSLab version 24.04.1
Host OS: Windows Server 2025 build 26277 
Guest OS - en-us_Windows_InsiderPreview_Server_vNext_en-us_26227 WS 2025 build 26227
Result: 2_CreateParentDisks.ps1,
OK - VHDX Creation 
OK - DC creation

On the MSLab side, I put together a script change that can help workaorund the issue, and I'm working with the maintainer to see about incorporating the change into the main build.

We're continuing to investigate this on the OS side.
best response confirmed by Karl-WE (MVP)
Solution
I wanted to update this discussion. We have found the issue that we believe is the root cause here and are developing the fix. The issue is triggered when a Domain Controller VM is 'dehydrated' and then re-imported. This is uncommon in production environments but is important to how MSLab works. Thus, we're working on a fix. It takes us some time to get a fix out to the world with appropriate testing, so please be patient while we roll it out. Thanks.
Cannot thank you enough for all the efforts. It will give mslab a rewind when it's becoming fully compatible with Windows Server 2025.
Is this fix in the works for 26100 and later or limited to this build?

@Karl-WE It should be a servicing fix to 26100, which will come out in one of the Patch Tuesday releases.  Assuming all goes well, etc.  :)  

I appreciate the compliment - it is very motivating!

Michael

1 best response

Accepted Solutions
best response confirmed by Karl-WE (MVP)
Solution
I wanted to update this discussion. We have found the issue that we believe is the root cause here and are developing the fix. The issue is triggered when a Domain Controller VM is 'dehydrated' and then re-imported. This is uncommon in production environments but is important to how MSLab works. Thus, we're working on a fix. It takes us some time to get a fix out to the world with appropriate testing, so please be patient while we roll it out. Thanks.

View solution in original post