Starting VMs may fail after installation of 10-2023 CU when Veeam RCT is being used -SOLVED-

Dear community,

Please consider reading this article https://learn.microsoft.com/en-us/answers/questions/1390624/virtual-machines-failed-to-start-after-installing?page=2#answers

in case you are directly affected by the issue, that Hyper-V VMs fail to start, in the certain scenario described below.

The issue is not related to VMs not starting on VMware ESXi after applying current CUs. Find remediation instructions for https://kb.vmware.com/s/article/90947

Affected Hosts:
Hyper-V Hosts
OS:
Windows Server 2019 (by community)
Windows Server 2022 (by community)
Azure Stack HCI 22H2 (experienced myself)

Scenario:

Gen1 / Gen2 VMs backed up by Veeam using RCT (Change Block Tracking)
Not completely confirmed but it seems that unclustered VMs are affected, and clustered VMs are not affected. In a case this morning this caused the unclustered Domain Controllers dedicated for the Azure Stack HCI cluster to fail.

edit: 19. Oct. 2023: clustered and unclustered VMs are affected.

Reasons:

Microsoft has made changes, fixing a long-standing performance issue with Veeam RCT feature, but the fix this can cause a side-effect. The fix is not mentioned in the change log (Windows Update history), despite the criticality of the improvement.

https://forums.veeam.com/memberlist.php?mode=viewprofile&u=55718 wrote: https://forums.veeam.com/post500560.html#p500560Oct 09, 2023 10:15 pmThe long awaited 'RCT fix', for the RCT side of this ongoing issue, should be released in the October 2023 Windows update, ie tomorrow!"

example of perf. improvements during Veeam backups using RCT from the community:

Root cause:
directly related to Veeam RCT and CU 10-2023, yet unknown why it happens.

Reproducible:

yes, 100% in certain scenarios.

The https://aka.ms/hci-slack is voluntarily working on the reproducibility in more scenarios to understand the dependencies.

Remediation:
a. rename or delete MCT / RCT files (quick and dirty). Keep the files if you want Microsoft Support / Veeam Support to engage on this issue.
b. uninstall the CU 10-2023 from your cluster (I cannot recommend that for several reasons), using manual suspending of nodes, WUSA and monitor migration and storage replication jobs to be finished before and after restarting nodes.

Solution:

Deploy Cumulative Update 11-2023, instead of 10-2023 on affected Hyper-V hosts.

kudos: ErnieCostaCVLT Jaromir Kaspar Darryl van der Peijl