KB5041578 causing Windows Server 2019 to be unresponsive

[MNichols]

In testing the August 2024 security update for Windows Server 2019 (KB5041578) we are finding the update causes some Windows Server 2019 servers to become unresponsive. 

There is also a thread on Reddit of many admins experiencing this same behavior. 

A workaround has been to stop the "Cryptographic Services" service, purge the contents of C:\windows\system32\catroot2 

Reboot, then install KB5041578 and the issue does not occur. 

Performing these steps on a "broken" server that was patched before performing these steps also fixes the issue, however you can't log on to the VM when this is occurring due to the performance issue, it must be done remotely. 

[#2]

Good post, hopefully MS will fix this on short notice or at least issue statement that something is broken.

[#3]

We are also seeing similar issues with all of our Windows Server 2019 machines on Azure. I am wondering if we would go with the solution above, or wait for some official confirmation from MS with a patch or rollback of the existing patch..

[#4]

@MNichols We also had this issue on 2 2019 servers today. The last time they were patched was back in april. Wondering if this is related to skipping patches as I dont see it on all 2019 servers. 

[#5]

@ausername215   Same problem here.  Any response from Microsoft yet?

[#6]

Here is a response we got from MS which is over the top. We have 1000s of 2019 machines that are not in a domain, so this solution will not work.

I’ve received confirmation from the debugging team to implement the Known Issue Rollback (KIR) policy on the affected devices instead of removing the patch. The instructions below will guide you through applying the fix for the unresponsive issue caused by KB5041578.

Action Plan:
Apply KIR to a single device using Group Policy
To use Group Policy to apply a KIR to a single device, follow these steps:
Download the KIR policy definition MSI file https://download.microsoft.com/download/03c7aacb-1f7b-443d-95e8-6d7d301ac831/Windows%2010%201809%20a...
Important: Make sure that the operating system that is listed in the .msi file name matches the operating system of the device that you want to update.
Run the .msi file on the device. This action installs the KIR policy definition in the Administrative Template.
Open the Local Group Policy Editor. To do this, select Start, and then enter gpedit.msc.
Select Local Computer Policy > Computer Configuration > Administrative Templates > KBXXXXXXXX XXXXXX_XXXXX Known Issue Rollback > Windows XX, version XXXX and Windows NNNN XXXX
Path
Computer Configuration -> Administrative Templates -> KB5041578 240816_2150 Known Issue Rollback -> Windows 10, version 1809 and Windows Server 2019
Setting
KB5041578 240816_2150 Known Issue Rollback
Value
Disabled
Reboot Requirements
A reboot is required once the device has applied the KIR GP setting

[#7]

Thanks - my response to Microsoft would be that if I could get logged into the problem system, I would just uninstall the patch myself! How can they expect us to do this when some companies have hundreds of servers that could potentially have this issue. And I don't want to apply this to all systems, because the problem isn't happening on all systems. Ugh. Can't wait to see what September brings!

[#8]

@TammySanders while it took a little bit i had no issues remotely stopping the cypto service, deleting the data in the folder referenced above and rebooting to resolve the performance issues. There was no need to remove the patch. 

speculating here but this has hit a small subset of our systems, specifically systems that were last patched in april 2024 unlike most which was patched the previous month. My thoughts are that systems that have missed previous CU's may be impacted.

[#9]

Makes you wonder if MS even tests any of this. Not the first time a monthy has been bad and won't be the last.

I can remember at least 2 times when RDP fell down after a monthly update.

[#10]

@ausername215 we patch our fleet everymonth and its still the same.

[#11]

@MNichols
I've found a "fix" that appears to do the job, it's not particularly graceful and is manual, but at least pretty quick and doesn't require a reboot or apparent roll-back of the update. You might need patience depending on how badly affected your system(s) are... Additionally if you have any kind of IOP-limited disks on affected servers this will make them even slower; if possible, remove any IOP limit and it should be less painful.

You'll need to launch (as admin) PowerShell or CMD, services.msc, Task Manager, and Windows Explorer. I put each in a corner of the UI for ease of navigation.

• In services.msc find Cryptographic Services and Disable it. You won't be able to stop it conventionally.
• In Task Manager/Services tab look for CryptSvc and note the PID.
• In Explorer, attempt to rename C:\Windows\System32\catroot2 to something else, doesn't matter what. It will fail, because it is in-use. Leave the pop-up open.
• In PowerShell/CMD, type "taskkill /F /PID <PID noted above>.
  
• Quickly flick back to the "Folder in Use" pop-up and hit Try Again, it should rename.
• In services.msc put Cryptographic Services back to Automatic, the process will restart automatically very quickly so you don't need to do that. The catroot2 folder will regenerate also, and apparently not look much/at all different from the old one.

Server should be response again...

[#12]

Another way using posh.

set-service CryptSvc -StartupType Disabled
stop-service Wuauserv -Force
stop-service cryptsvc -Force
stop-service bits -Force
remove-item -Path C:\windows\system32\catroot2 -Recurse -Force
set-service CryptSvc -StartupType Automatic
start-service Wuauserv
start-service cryptsvc
start-service bits

[#13]

Heheh this would work nicely, when I'm lucky enough to be able to do PS.

A strange side-effect which could be cause, effect, or something in-between seen on a lot of affected servers is that no form of PowerShell CLI, ISE will launch. This would appear to be as a result of something in Windows Defender locking-up, and with Tamper Protection via the XDR agent enabled, it's impossible to work-around.

[#14]

This process worked great for me on one server. However, another server is so unresponsive that I cannot use this process. Can you please adapt this process so that the steps can be done via remote CLI to a server? Thank you.

[#15]

@jbruns2019 

stop-service cryptsvc -Force

This step fails due to other dependent processes. What can be done about this?

Thank you.

[#16]

Heya Karen - This may be something of a catch-22 situation; I found scripting it myself was problematic for just such reasons. If you have a device which is borderline unresponsive locally then any commands passed over WinRM/remote PS will equally never complete. I don't even get eventual timed-out, just a locked-up script which would never move on to the next device in a text file fed to it.

For those ones that are really unresponsive I've found the only fix is patience unfortunately. You may also need to taskkill the PID (which will be different every time the process spawns) numerous times before you can rename or delete the catroot2 folder.

[#17]

Thank you, guess I just try and be patient. If I do get this unresponsive server to process the commands you placed here, I will respond with a reply of success.

[#18]

Booting the server in safemode and uninstall the KB5041578 did the trick for me.  Not all 2019 servers we are running were impacted.

[#19]

Success, thank you.

[#20]

OK, that could be a good option too. Thank you.