KB5016623 Issues with AAD App Proxy

[Andrew Emmett]

Hello

We have encountered some issues with KB5016623. The is causing the server, Win 2019 server running IIS, to crash after 5 to 10 minutes and to be unable to use AAD App Proxy connections that are setup to use Windows Authentication on the backend via kerberos.

We have 2 different scenarios:

1. A webserver some legacy windows auth based apps, alongside newer apps that use modern auth. The AAD app proxy connector in also installed on the webserver. The newer apps using modern auth are working fine, but the old windows auth apps are failing to authenticate. Errors are:
   • Microsoft AAD Application Proxy Connector cannot retrieve a Kerberos ticket on behalf of the user because of the following general API error: The handle specified is invalid

     (0x80090301) 
     After about 5-10 minutes, the server seems to crash with this error:
     A critical system process, C:\WINDOWS\system32\lsass.exe, failed with status code c0000005.The machine must now be restarted.
     
     

   • The process wininit.exe has initiated the restart of computer <ServerName> on behalf of user  for the following reason: No title for this reason could be found

      Reason Code: 0x50006

      Shut-down Type: restart

      Comment: The system process 'C:\WINDOWS\system32\lsass.exe' terminated unexpectedly with status code -1073741819.  The system will now shut down and restart.

2. Another server, this one only with AAD app proxy that accesses a separate SSRS Web Server with the same issues as above.

In both examples, uninstalling KB5016623 has resolved the issue. We don't seem to be seeing any issues with other servers e.g. DCs at present. It mainly seems to be the combination of KB5016623 and AAD App Proxy with Kerberos back ends.  Anyone else seeing any similar problems?

Thanks

Andy

[#2 · Accepted solution]

@Andrew Emmett 

Hi Andrew,

We had the same issue today.   Uninstalling KB016623 resolved it as well.  I've logged it with Microsoft.  Will report back when they respond. 

Glen. 

[#3]

Thank you Andrew! This is the only hit on this issue. My company also hit this this morning. Uninstalling 5016623 from the 2019 Proxy Servers fixed the issue. Kudos :thumbs_up:

[#4]

Hi!

We had exactly the same issue , but with 2016 servers and KB5016622.

https://support.microsoft.com/en-us/topic/august-9-2022-kb5016622-os-build-14393-5291-19fa46af-be4b-...

Unistall this KB fixed the issue.

[#5]

Hi,

We have same issue, on 2019 KB016623.
Have support case with MS, but not gotten any breakthrough yet other than uninstalling the patch.

[#6]

Thanks for the heads up Andrew. Had issues with our WebApp Proxy this morning caused by the Windows 2012 R2 security update KB5016681. Uninstalled the update and service is operational again. I expect MS will be looking into this at some point shortly.

[#7]

No one is safe apparently :)

[#8]

Anyone else using "RunAsPPL LSA Protections on the servers?

As part of debug with MS I had to remove the RunAaPPL reg key to be able to trace lsass.
To my surprise the AAD App Proxy started working after removing reg key and reboot server, with KB5016681 installed.

Ref: https://docs.microsoft.com/en-us/windows-server/security/credentials-protection-and-management/confi...

[#9]

I don't use that, but I do use the ASR for LSASS, which is basically the same thing... Good to know I will look at disabling that on the app proxies for now.

[#10]

@GeirF. I was mistaken, I did have RunAaPPL enabled. I also disabled the ASR rule so I don't know if that has any impact. When its safe to try I will reenable the ASR LSASS rule and update the thread for anyone interested. Also, if anyone has any idea how to disable this in Azure it would be greatly appreciated. As per the doc UEFI boxes can tattoo the setting in its UEFI. MS offers a efi file to help remove the setting but requires access to UEFI to accept the setting change. To my knowledge this is not possible in Azure so I just moved the apps to on prem proxies till I rebuild or figure that part out. Thanks again!

[#11]

@GeirF

I have removed the RunAaPPL key from some standby servers (we deploy the key as standard practice) and provisionally I think things are working. My production servers are still running without KB5016623 and I can’t risk the instability at the moment as I work in education and the next few days are the most important of the year.

However, to test, I have routed a few non essential web sites through backup servers running AAD proxy and the latest server patch (Aug 23rd - KB5016690) with RunAaPPL key removed and both Windows Auth and Modern Auth websites are working as expected. So, I feel confident that this might be the answer.

I found that servers crashed more quickly when the server was under load, so will need to see what the nest few days brings. If my fully patched servers last until Monday without crashing & rebooting, I’ll update the production servers again.

Fingers crossed

[#12]

To get access to UEFI on an Azure machine I think you will be able if you use a "Repair VM with nested hyper-v".

Ref the "Repair VM with Nested Hyper-V example":
https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/repair-windows-vm-using-azure-v...

[#13]

Nice info dude thx

[#14]

We have almost similar issue where AAD App Proxy Servers stops authenticating Apps, Local User Logon, RDP via Domain user.
We opened a case with Microsoft and have been told that Its a known issue and would be fixed in November 2022 Security update.

[#15]

Have you or Microsoft come up with an approach to mitigate the issue when it happens?

[#16]

@freddy104 MS has advised not to patch the server until fix is released. They proposed a registry change, however it did not work.

[#17]

Word from Microsoft is that this issue is fixed in the November 8 Patch Tuesday patches. Can anyone confirm if they have tested the 11/8 patches to confirm the issue is resolved?

[#18]

Word from Microsoft is that this issue is fixed in the November 8 Patch Tuesday patches. Can anyone confirm if they have tested the 11/8 patches to confirm the issue is resolved?

[#19]

Installed Nov patch for Server 2016 (kb5019964) and appears to have resoled the issue.

[#20]

Similar patch for Server 2019 is KB5019966:
November 8, 2022—KB5019966 (OS Build 17763.3650) - Microsoft Support