November 2022 Out of Band update released! Take action!

[JR_Nieves]

Greetings from the Windows Directory Services team!

The team wanted to bring to your attention the November 17th, 2022 release of an Out of Band (OOB), non-security update that addresses the Kerberos authentication issues experienced in some environments after installing November 8, 2022 (or later) updates on domain controllers.

After installing Windows Updates released on November 8, 2022 on Windows domain controllers, you might have issues with Kerberos authentication.

This specific failure is identified by the logging of Microsoft-Windows-Kerberos-Key-Distribution-Center Event ID 14 in the System event log of DC role computers with this unique signature in the event message text:

While processing an AS request for target service <service>, the account <account name> did not have a suitable key for generating a Kerberos ticket (the missing key has an ID of 1). The requested etypes : <etype numbers >. The accounts available etypes : <etype numbers>. Changing or resetting the password of <account name> will generate a proper key.

Where

• (a.) “the missing key has an ID 1” and (b.) "4" is not listed in the "requested etypes" or "account available etypes" fields.

Some scenarios that might be affected:

• Domain user sign in might fail. This also might affect Active Directory Federation Services (AD FS) authentication.
• ​Group Managed Service Accounts (gMSA) used for services such as Internet Information Services (IIS Web Server) might fail to authenticate.
• ​Remote Desktop connections using domain users might fail to connect.
• ​You might be unable to access shared folders on workstations and file shares on servers.
• ​Printing that requires domain user authentication might fail

For important details about how to obtain and install the November OOB update, please see the following link on Windows Release Health Message Center at:

• Take action: OOB update to address an issue with sign in and Kerberos authentication

[cgaurav20]

Apparently after installing this out of band update still we are facing issue but with slight change this event 14 is fixed and now we have an issue logged with event id 27 at stage of generating TGS ticket as follows.

While processing a TGS request for the target server ****, the account **** did not have a suitable key for generating a Kerberos ticket (the missing key has an ID of 9). The requested etypes were 23 3 1. The accounts available etypes were 23 18 17.

To fix this I still had to apply the workaround of allowing RC4 encryption, can you suggest please if there is a permanent fix available for this too so I can revert this setting?

[Finalnet]

What helped us:
Uncheck the AES 128 bit and AES 256 bit boxes under Account for the affected users.
Under the attributes, check that msDS-SupportedEncryptionTypes = Not set or 0.

No need to reenable RC4

I'm not 100% sure if related to the problem.

[Alban1998]

@cgaurav20 Hello, did you check if DES encryption is enabled for this account ?

@JR_Nieves Hello, could you confirm if this update will prevent OS older than Vista to authenticate with domain controllers ? There are still customers with Windows Server 2000/2003 out there, and it could be the last push to trigger a migration.

[cgaurav20]

@Alban1998 @Finalnet We already have this security feature enabled for the DC's "Network security: Configure encryption types allowed for Kerberos" which is only restricting Kerberos to AES128/AES256 & Future keys. No DES & RC4 is allowed in our environment. Neither we have any per account setting configured for AES or other types. 

We just have 2 DC in that particular forest, one is patched with November updates (Including 17th Nov OOB patch) other one is not and there are no authentication failures for the unpatched DC with exact same configuration. So, it's definitely the issue post patching.

So, is it really required to change this setting "Network security: Configure encryption types allowed for Kerberos" as not defined? Will it not make RC4 encryption to be allowed for use if some application want it? I don't think our security team would approve to remove these settings. Any other suggestion?

[Finalnet]

@cgaurav20: In our case, only a few users were affected. A new user (Not copied) can be created for testing.
If the new user has no problems, it is probably the AES128 and AES256 bit options under <UserObject>->Account-> Scroll down a little.

We have not modified any policies to solve the problem.

I have testet in a VM once for the user "msDS-SupportedEncryptionTypes" is set the problems appear until option is reverted.

You can check with: 

Get-ADUser -Filter * -Properties msDS-SupportedEncryptionTypes | ft msDS-SupportedEncryptionTypes, name, UserPrincipalName -AutoSize

The eventlog will show the entries as soon as one affected users does anything with kerberos...

If you see Tickets (elevated cmd) "klist –li 0x3e7"

But none (non-elevated cmd) for "klist"

It's likley to be a user problem.

I suspect that the options in the user object from the bitmask still contain the old disabled protocols. However, this is only a guess.

[Alban1998]

@cgaurav20 

I suggest to make sure all you DC have an identical configuration (including patch versioning), as it is really a recommend setting, avoiding unsupported scenarios.

As for your issue, if it's not a bug, then you may have a real issue there - try investigating operating system, apps and accounts related to the error message to see if something is actively using RC4.

[Finalnet]

Please have a look on this KB: KB5021131: How to manage the Kerberos protocol changes related to CVE-2022-37966 - Microsoft Support

It disables DES / RC4 for my understanding.

I found the following table very helpful (Not my page, use at own risk):

KnowledgeBase: You experience errors with Event ID 14 and source Kerberos-Key-Distribution-Center on...

AS KB5021131 deactiviated the first 3 colums, all configuration with green in the first 3 columns should/would break for my understanding.

I'm not 100% sure. Microsoft should clarify or deny: That if the Policy on DC Level is set to deny lower encryption methods, the user setttings shouldn't have any impact.

PS: All DCs should all have the updates.

[cgaurav20]

@Finalnet I have already gone through above articles and that's where I got to know that RC4 is the issue. I have tested with PS scripts, and I do not have any account using DES or RC4. Even as I said earlier our DC's were already enforced for AES encryption, so if DES or RC4 would have been set for the accounts then it would have already stopped working for those accounts before the patch.

I can also confirm the moment I set my GPO for encryption keys as not defined things start to work. So, this patch is fixing half part only for AS authentication requests now it gets block at TGS part.

I have another environment (Different Forest) as well too where I could see similar behavior of authentication failing at TGS part. 

Can I conclude final solution is only to not to set msDS-SupportedEncryptionTypes by any means (GPO or per user settings)?

[Finalnet]

@cgaurav20 Which value do you have configured for msDS-SupportedEncryptionTypes?

[Yannick1285]

Hello everyone,

Does someone has identified a memory leak caused by lsass.exe process (attach to services : netlogon, KDC, Active Directory, etc.)  on Windows Server 2016 with domain controllers role since the KB5019964 installation?

I have some domain controllers on W2k16 with latest November CU patch and they all have a memory leak caused by the lsass.exe process.

The other domain controllers which doesn't have the November CU installed don't have this memory leak. 

I'm currently installing the OOB patch to see if it fix this memory leak.

EDIT 23/11/2022:
The OOB patch didn't fix the memory leak on lsass.exe process so we proceed to uninstall the November CU KB5019964 and OOB KB5021664  on our  Windows Server 2016 Domain Controllers.

EDIT 23/11/2022:
I have open a case and Microsoft confirm this is a known issue on domain controllers W2k12R2, W2K16 and W2K19.

The temporay fix is to configure the reg key  KrbtgtFullPacSignature to 0

Details about this reg key available here:

KB5020805: How to manage Kerberos protocol changes related to CVE-2022-37967 - Microsoft Support

This key will prevent the memory leak on process Lsass.exe but a reboot is required to fully release the memory.

[Alban1998]

@Finalnet Indeed, I will edit my post as it was confusing.

So short story so far :

1) install all updates for all DC (including out-of-band if you encounter the issue) then restart all DC

2) Remove any GPO and/or account configuration settings related to encryption protocols

3) It works

[ColonelHawx1008]

Here is a Powershell script to output msDS-SupportedEncryptionTypes for users.

Get-ADUser -Filter * -Properties msDS-SupportedEncryptionTypes | Select-Object msDS-SupportedEncryptionTypes, name, UserPrincipalName | Export-Csv -Path "C:\Path\Document.csv" -Append -NoTypeInformation

[EmanueleSignorin89]

Hi all,

after the November updates and OOB we are no longer able to access shares on legacy servers (Win server 2003). Has anyone experienced this problem?

[cgaurav20]

@Alban1998 definitely this works but is it really a recommend solution, now we have taken a step back in terms of security earlier all of the machines were enforced to use AES only because of our GPO and now after setting it to not configured for kerberos enforcement we have some machines which now started using RC4. Note we still have latest November patches installed, so it's not like AES is enforced after these updates. We are now Kind of forced now to enable the old encryption as things stop working now if we try to enforce AES.

Even MS is not able to give the solution here on this, we already have a running case.

[RR13]

Hello, exactly the same issue here: we have installed the kb of November and have some authentification issues. We open a case with microsoft and they advise to use the registry key to remediate this situation. It works fine. 
After Microsoft says that an out-of-band is available. We try to install it on 4 dc only and we have again some authentification issues. We rollback and uninstall the out of band, everything works again. So I can confirm that this out-of-band doesn’t work as expect. Moreover, since the installation of the CU of November, we also see some LSASS high CPU usage on a lot of DC.. so they are a lot of issues with the KB and the out-of-band ! We are waiting for a communication from Microsoft for that. Thanks 

[Alban1998]

@cgaurav20 I agree, going back to RC4 is not a pretty thing.

Did you absolutely check for everything which could ask for RC4 on those machines ? Computer accounts, service accounts with SPN, gMSA, keytabs, or other non-Microsoft software with a missing/faulty Kerberos implementation ?

Do you have any msDS-SupportedEncryptionTypes attribute configured manually on them ?

The most secure protocol available is negotiated between clients and DC, so if some of your machines go back to RC4, that's likely because they have no other choices. You need to identify why.

There is a script which checks all possible accounts here : How Do I Know If My AD Environment Is Impacted By The November 8th 2022 Patch? - Microsoft Community...

Not to say you do not have a real issue either, and another hotfix will be required in a near future. Keep us updated when Microsoft Support troubleshoot this.

[Alban1998]

@RR13 What version of Windows Server are you using ?

[RR13]

@Alban1998 all the DC are running under Windows Server 2016. It’s not a fix for me to change the encryption type used by the service account or something else. The KB and the Out-of-Band should be fix to avoid this kind of errors ! It’s my opinion :)

[Alban1998]

@RR13 This is about your high CPU usage. It seems to impact only WS 2016 machines so far.

And you cannot ask a KB to fix misconfigurations and faulty apps.
We could ask for a better QA with such hotfix but that's a thing of the past.

[RR13]

@Alban1998 A missconfiguration of what exactly ? When you have all your gMSA accounts for example configured with only AES128/256 encryption type and the KB / out-of-band caused authentication failures, I don’t think it’s a missconfiguration.. Same for LSASS, Windows Server 2016 is still supported by MS and a stable OS so if the KB causes some high CPU usage, Microsoft has to provide a fix. 

[Alban1998]

Of course they should provide a fix for high CPU usage, I'm just pointing it's an issue on Windows Server 2016, not something affecting all Windows Server OS (at least that's what it seems).

[danielkaliel]

Kerberos in Chrome has stopped working since this update and the OOB install.  I have ensured that msDS-SupportedEncryptionTypes is cleared for all users.  The reg key has been set to enable auditing on the domain controllers.  Anyone else experiencing this?

[BishopstonIT]

does this OOB update still need installing, or is it included within the December updates?

[swisslegit90]

I am confused, i am getting 44 Events, but just from domain controllers, so host and Computer Attribute in the Event is a Domain Controller.

Can I assume that i just move to enforcement mode and all will work, cause dc's are adding full pac signature (all Domain Controllers are patched with the december and january updates)

Example Event:

host	dc1
sourcetype	windows_syslog
Computer	dc1.domain.ch
EventChannel	System
EventDescription	Key Distribution Center (KDC) encountered a ticket that did not contain the full PAC Signature
EventID	44
EventLevel	3
EventProvider	Microsoft-Windows-Kerberos-Key-Distribution-Center

[jt8585]

If we didn't install 11/2022 patch and installed 3/2023, is this OOB still needed?

Once OOB is isntalled, does it automatically add the required registry key on DCs? or nothing needs to be done?

[GreenVomit8]

EmanueleSignorin89 , Yes same for us, workaround for us is to replace hostname with IP address in UNC Path for shares to become accessible.