Server 2k19 Hyper-V VMs no longer start

[samw5875]

I'm having some issues with VMs no longer starting. This started May 11th 2022. 
The last updates that I see installed around that timeframe are KB5001384 and the May 10 KB5013941.

Tried to uninstall both to no avail. The server has been patched to the latest update since.

Server is running on a Dell R630.

Server is domain joined.

VM services are started but when trying to launch the VM I am getting a generic error stating that the VM(s) failed to start. VMs were shutdown gracefully (there are 4 currently, 3 windows, 1 linux server).

Errors in the logs that I am seeing:

- Microsoft-Windows-Hyper-V-VMMS Event 15130

- Microsoft-Windows-Hyper-V-Worker Event 3062

Tried running DISM.exe /Online /Cleanup-Image /Restorehealth and SFC /scannow but still not allowing me to start them.

Using Multiplexor switch and I'm able to view/edit the Vswitch w/o issues.

Services for Hyper-V are present and running. 

I did try removing the Hyper-V role (grayed our in server manager) and getting an error via PowerShell: The referenced assembly could not be found. Error: 0x80073701

I'm at a complete loss right now and if I had sufficient storage to migrate and rebuild the server I certainly would consider it... 

Any suggestions would be welcome.

[#2]

@samw5875 

Not being able to replicate the errors, I ran a search and hit the article below, which in turn jogged my memory that I have had to re-create virtual switches in the past to resolve unexpected issues at times.

I'd recommend creating a new virtual switch(es) with the same configuration as the current and try moving one of the impacted guests across to it. That provides you an easy, non-destructive way back to the old one should it be required.

Windows 2012 R2 Hyper-V event id's 20144 and 15130 (microsoft.com)

Edited:

Since I'm in guessing mode based on prior experience, I may as well quickly add this extra point.

When I've moved virtual machines from one host to another during hypervisor replacements, I've had to, on occasion, remove and re-create just the NIC(s) within the guest. I didn't look into it since it was a quick fix but I expect this probably addressed a bad GUID reference or something similar.

Cheers,

Lain

[#3]

@LainRobertson No luck here.

I removed 2 NICs from the NIC team (all 4 NICs were added). 

Created a new virtual switch, reassigned the NIC on the VM and tried starting it.

Same errors. 3062 for the V-Worker and 15130 for V-WMMS. 

I also did recreate the vSwitch using the Multiplexor Driver as it was previously configured... 
Still not go.

Last thing I tried was set the NIC of one of the VM to "Not Connected" and attempted to start... same error.

[#4]

@samw5875 

Yeah, fair enough.

As an aside, setting the NIC to disconnected won't do anything. You'd want to remove the NIC from the configuration and create a new one.

That's how I've resolved issued when using a plain old copy process to migrate guests from an old Hyper-V host to a new.

Again, it may not help in your case, but that's what I was referring to, not simply disconnecting the existing NIC.

For testing purposes, you could even try creating a new internal switch and hooking up a guest to that. If nothing else, that would serve to confirm there's nothing wrong with your multiplexor switch (i.e. if you still get the same error on the internal switch.)

Cheers,

Lain

[#5]

@samw5875 

Hello samw5875,

The error 0x80073701 is ERROR_SXS_ASSEMBLY_MISSING, meaning some system files are missing, causing the Windows update installation failure. The 0x80073701 error appears due to missing system files. System files get corrupted or deleted due to various reasons.

However, you can follow the steps mentioned below to troubleshoot  Hyper-V VMs no longer start issues,

Run Windows Built-in Troubleshooter:

The Error 0x80073701 can be solved using the built-in Windows Update Troubleshooter,

• Firstly, right-click the Start button and click Settings
• Then, In the left pane, select Troubleshoot.
• Now, In the right pane, click Additional troubleshooters.
• Lastly, select Windows Update and click Run the troubleshooter.

The Update troubleshooter should provide fixes and apply them automatically.

Flush the SoftwareDistribution Folder:

• Firstly, press the Windows logo + R keys to open the Run dialog
• Then, type in CMD and hit Enter to open the Command Prompt.
• Now, type the following command and press Enter,

net stop wuauserv 

• Next, type in rd /s /q %systemroot%\SoftwareDistribution and press Enter to clear the Update cache folder.
• Lastly, type in net start wuauserv and press Enter.

Execute a System File Checker (SFC) Scan:

It replaces the corrupt, missing files and downloads them, causing a 0x80073701 error. 

• Firstly, press the Windows logo + R keys to open the Run dialog
• Then, type in CMD and hit Enter to open the Command Prompt.
• Now, type the following command and press Enter,

SFC /Scannow

• Lastly, restart your server.

Execute DISM scan:

DISM is a command-line tool to fix corrupt system files,

• Firstly, press the Windows logo + R keys to open the Run dialog
• Then, type in CMD and hit Enter to open the Command Prompt.
• Now, type the following command and press Enter,

DISM /Online /Cleanup-image /StartComponentCleanup

• Then, type the below and hit Enter,

DISM /Online /Cleanup-Image /ScanHealth

• Next, type the following command and press Enter,

DISM /Online /Cleanup-Image /RestoreHealth

• Lastly, restart your server and check for the error.

Reset Windows Update Components:

• Firstly, press the Windows logo + R keys to open the Run dialog
• Then, type in CMD and hit Enter to open the Command Prompt.
• Now, type the following commands and press Enter on each command,

net stop wuauserv

net stop cryptSvc

net stop bits

net stop msiserver

• Then, rename the SoftwareDistribution and Catroot2 folders, type the following command, and press Enter,

ren C:\Windows\SoftwareDistribution SoftwareDistribution.old

• Then, type the below command and hit Enter,

ren C:\Windows\SoftwareDistribution SoftwareDistribution.old

• Lastly, re-type the following commands and hit Enter on each command to restart stopped services.

net start wuauserv

net start cryptSvc

net start bits

net start msiserver

After restarting Windows update components, you must check whether the issue persists.

I hope the above fixes help you to resolve the Hyper-V VMs no longer start on Server 2019 with error code 0x80073701 without facing any problem.

-----------------------

Regards,

Austin M

[#6]

@Austin_M 

Will give this a try and report.

[#7]

Hi Austin_M
I have this exact same issue with a newly installed Hyper-V host and newly created VM. Did you find a solution?

[#8]

@DarrenM-DMCS 

nothing that worked. Vms on that box weren't critical so the next step was going to export them to another host but haven't gone around this.
I tried tons of stuff including remove NICs, DSIM reports, reinstall roles nothing seemed to work 😞