Forum Discussion

MNichols's avatar
MNichols
Copper Contributor
Aug 15, 2024

KB5041578 causing Windows Server 2019 to be unresponsive

In testing the August 2024 security update for Windows Server 2019 (KB5041578) we are finding the update causes some Windows Server 2019 servers to become unresponsive. 

 

There is also a thread on Reddit of many admins experiencing this same behavior. 

 

A workaround has been to stop the "Cryptographic Services" service, purge the contents of C:\windows\system32\catroot2 

 

Reboot, then install KB5041578 and the issue does not occur. 

 

Performing these steps on a "broken" server that was patched before performing these steps also fixes the issue, however you can't log on to the VM when this is occurring due to the performance issue, it must be done remotely. 

 

 

 

  • Shakedown1979's avatar
    Shakedown1979
    Copper Contributor

    MNichols
    I've found a "fix" that appears to do the job, it's not particularly graceful and is manual, but at least pretty quick and doesn't require a reboot or apparent roll-back of the update. You might need patience depending on how badly affected your system(s) are... Additionally if you have any kind of IOP-limited disks on affected servers this will make them even slower; if possible, remove any IOP limit and it should be less painful.

     

    You'll need to launch (as admin) PowerShell or CMD, services.msc, Task Manager, and Windows Explorer. I put each in a corner of the UI for ease of navigation.

     

    • In services.msc find Cryptographic Services and Disable it. You won't be able to stop it conventionally.
    • In Task Manager/Services tab look for CryptSvc and note the PID.
    • In Explorer, attempt to rename C:\Windows\System32\catroot2 to something else, doesn't matter what. It will fail, because it is in-use. Leave the pop-up open.
    • In PowerShell/CMD, type "taskkill /F /PID <PID noted above>.
    • Quickly flick back to the "Folder in Use" pop-up and hit Try Again, it should rename.
    • In services.msc put Cryptographic Services back to Automatic, the process will restart automatically very quickly so you don't need to do that. The catroot2 folder will regenerate also, and apparently not look much/at all different from the old one.

    Server should be response again...

    • jbruns2019's avatar
      jbruns2019
      Brass Contributor
      Another way using posh.

      set-service CryptSvc -StartupType Disabled
      stop-service Wuauserv -Force
      stop-service cryptsvc -Force
      stop-service bits -Force
      remove-item -Path C:\windows\system32\catroot2 -Recurse -Force
      set-service CryptSvc -StartupType Automatic
      start-service Wuauserv
      start-service cryptsvc
      start-service bits
      • Shakedown1979's avatar
        Shakedown1979
        Copper Contributor
        Heheh this would work nicely, when I'm lucky enough to be able to do PS.

        A strange side-effect which could be cause, effect, or something in-between seen on a lot of affected servers is that no form of PowerShell CLI, ISE will launch. This would appear to be as a result of something in Windows Defender locking-up, and with Tamper Protection via the XDR agent enabled, it's impossible to work-around.
    • Karen8488150's avatar
      Karen8488150
      Copper Contributor
      This process worked great for me on one server. However, another server is so unresponsive that I cannot use this process. Can you please adapt this process so that the steps can be done via remote CLI to a server? Thank you.
      • Shakedown1979's avatar
        Shakedown1979
        Copper Contributor
        Heya Karen - This may be something of a catch-22 situation; I found scripting it myself was problematic for just such reasons. If you have a device which is borderline unresponsive locally then any commands passed over WinRM/remote PS will equally never complete. I don't even get eventual timed-out, just a locked-up script which would never move on to the next device in a text file fed to it.

        For those ones that are really unresponsive I've found the only fix is patience unfortunately. You may also need to taskkill the PID (which will be different every time the process spawns) numerous times before you can rename or delete the catroot2 folder.
  • dwijsman's avatar
    dwijsman
    Copper Contributor
    Good post, hopefully MS will fix this on short notice or at least issue statement that something is broken.
  • ShahzadAnsari's avatar
    ShahzadAnsari
    Copper Contributor
    We are also seeing similar issues with all of our Windows Server 2019 machines on Azure. I am wondering if we would go with the solution above, or wait for some official confirmation from MS with a patch or rollback of the existing patch..
  • ausername215's avatar
    ausername215
    Copper Contributor

    MNichols We also had this issue on 2 2019 servers today. The last time they were patched was back in april. Wondering if this is related to skipping patches as I dont see it on all 2019 servers. 

      • jbruns2019's avatar
        jbruns2019
        Brass Contributor
        Here is a response we got from MS which is over the top. We have 1000s of 2019 machines that are not in a domain, so this solution will not work.

        I’ve received confirmation from the debugging team to implement the Known Issue Rollback (KIR) policy on the affected devices instead of removing the patch. The instructions below will guide you through applying the fix for the unresponsive issue caused by KB5041578.

        Action Plan:
        Apply KIR to a single device using Group Policy
        To use Group Policy to apply a KIR to a single device, follow these steps:
        Download the KIR policy definition MSI file https://download.microsoft.com/download/03c7aacb-1f7b-443d-95e8-6d7d301ac831/Windows%2010%201809%20and%20Windows%20Server%202019%20KB5041578%20240816_21501%20Known%20Issue%20Rollback.msi
        Important: Make sure that the operating system that is listed in the .msi file name matches the operating system of the device that you want to update.
        Run the .msi file on the device. This action installs the KIR policy definition in the Administrative Template.
        Open the Local Group Policy Editor. To do this, select Start, and then enter gpedit.msc.
        Select Local Computer Policy > Computer Configuration > Administrative Templates > KBXXXXXXXX XXXXXX_XXXXX Known Issue Rollback > Windows XX, version XXXX and Windows NNNN XXXX
        Path
        Computer Configuration -> Administrative Templates -> KB5041578 240816_2150 Known Issue Rollback -> Windows 10, version 1809 and Windows Server 2019
        Setting
        KB5041578 240816_2150 Known Issue Rollback
        Value
        Disabled
        Reboot Requirements
        A reboot is required once the device has applied the KIR GP setting

Resources