So, you say your DC’s memory is getting all used up after installing November 2022 security update

Microsoft

Dec 13, 2022

Hello, Chris here from Directory Services support team with part 2 of the series.

After installing the November 2022/Out of Band update on your domain controllers you might experience a memory leak happening within LSASS.exe (Local Security Authority Subsystem Service). This could affect domain controller performance, cause operational failures, and/or reliability issues.

If you have already patched your domain controllers, the December 13, 2022 security update should resolve the known memory leak that is happening within LSASS.exe at this time. See table below, however if you do not currently feel comfortable with doing this please read the below:

OS	Resolving Rollup KB	Resolving Security Only Update
Windows Server 2019	5021237	N/A
Windows Server 2016	5021235	N/A
Windows Server 2012 R2	5021294	5021296
Windows Server 2012	5021285	5021303
Windows Server 2008 R2	5021291	5021288
Windows Server 2008	5021289	5021293

To briefly summarize the below, there is currently a registry key workaround for the memory leak. If you haven’t installed the December update or newer yet, you can use the registry key to avoid this problem. Run the following commands in an elevated command prompt on all of your domain controllers:

reg add "HKLM\System\CurrentControlSet\services\KDC" -v "KrbtgtFullPacSignature" -d 0 -t REG_DWORD

The above registry change will stop the memory leak without stopping and starting the KDC Service. It WILL NOT free up memory that has already been leaked within LSASS. So, it is recommended that a reboot be done of the domain controller when it is feasible to do so.

Note: Once you have installed the patch that resolves this known issue, you should either remove this value or set KrbtgtFullPacSignature to a higher setting depending on what your environment will allow. It is recommended to enable Enforcement mode as soon as your environment is ready. See: KB5020805: How to manage Kerberos protocol changes related to CVE-2022-37967

If you want to see if you're affected by this specific memory issue, check for constant increases of this performance counter within Perfmon.exe to see if it is constantly rising:

\Process(lsass)\Private Bytes

You will want to monitor “Private Bytes” for LSASS over a period of time. If this value just keeps increasing after installation of the November 2022/OOB update, then you are more than likely affected by this issue. Normal behavior should be that this value should go up during higher loads on the DC and then go down when the DC is not being utilized overtime. Please be aware that domain controllers will, by default, attempt to cache as much of the Active Directory database in memory as possible. See the linked section of Memory usage considerations in AD DS performance tuning | Microsoft Learn.

Information about the changes made to Kerberos Privilege Attribute Certificate (PAC) with the November 2022 security update:
KB5020805: How to manage Kerberos protocol changes related to CVE-2022-37967

Links to operating system versions affected by this issue:

https://learn.microsoft.com/en-us/windows/release-health/status-windows-10-1809-and-windows-server-2019#2966msgdesc

https://learn.microsoft.com/en-us/windows/release-health/status-windows-10-1607-and-windows-server-2016#2966msgdesc

https://learn.microsoft.com/en-us/windows/release-health/status-windows-8.1-and-windows-server-2012-r2#2966msgdesc

https://learn.microsoft.com/en-us/windows/release-health/status-windows-server-2012#2966msgdesc
https://learn.microsoft.com/en-us/windows/release-health/status-windows-7-and-windows-server-2008-r2-sp1#2966msgdesc

https://learn.microsoft.com/en-us/windows/release-health/status-windows-server-2008-sp2#2966msgdesc

Introduction to this blog series: https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/having-issues-since-deploying-november-2022-security-updates-to/ba-p/3696512

Part 3 of this blog series: https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/what-happened-to-kerberos-authentication-after-installing-the/ba-p/3696351

Updated Dec 13, 2022

Version 2.0

DavidFisher

Microsoft

Joined June 17, 2020

View Profile

Ask the Directory Services Team

Follow this blog board to get notified when there's new activity

1 Comment

TMChannel

Copper Contributor

Dec 15, 2022

Not to be rude, but is Microsoft going to reinstate proper regression testing of AD-related patches in future? I can't think of one update to AD since November last year that has not resulted in significant issues.

Especially since major security changes are ongoing with the underlying protocols - the scale of these changes should entail more testing, not less, as it seems at present. I'm starting to develop a conspiracy theory that Microsoft trying to stealth-deprecate on-prem AD with problem patches (reminiscent of Exchange CUs)!

In case this helps someone else, we've found that the overall commit charge for the system memory is useful to determine which DCs are struggling. In our environment, we have different hardware specs in various AD sites, so the raw LSASS PrivateBytes counter was a little tricky to compare (other than the figures going up over time, of course). For us, we rebooted the DCs once they got to 70% memory commit, after setting KrbtgtFullPacSignature to 0.

$dcs = ([System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain()).domaincontrollers.name |sort
$DCMemUsage = foreach ($dc in $DCs) {
    write-host "Querying $dc"
    Invoke-command $dc -scriptblock {
        $vmemsize = Get-CIMInstance -class win32_operatingsystem | select totalvirtualmemorysize,freevirtualmemory
        [pscustomobject]@{
        # total commit charge
        "SystemCommit (%)" = [math]::round((($vmemsize.totalvirtualmemorysize - $vmemsize.freevirtualmemory)/$vmemsize.totalvirtualmemorysize * 100),1)
        #Lsass private bytes
        "LsassPB (MB)" = [math]::round((gcim -query "SELECT PrivateBytes FROM Win32_PerfFormattedData_PerfProc_Process WHERE name='lsass'").privatebytes /1Mb)
        }

    }
}
$DCMemUsage |select PSComputerName,"SystemCommit (%)","LsassPB (MB)" |sort PrivateBytes -Descending |ft -AutoSize

----------------
PS > .Get-DCMemoryCommit.ps1
Querying TSTDC001.example.com
Querying TSTDC002.example.com
Querying TSTDC003.example.com
Querying TSTDC006.example.com
Querying TSTDC008.example.com

PSComputerName         SystemCommit (%) LsassPB (MB)
--------------         ---------------- ------------
TSTDC001.example.com           34.8          593
TSTDC002.example.com           66.1         5701
TSTDC003.example.com           34.7          803
TSTDC006.example.com           59.4          390
TSTDC008.example.com             63          205

Blog Post

So, you say your DC’s memory is getting all used up after installing November 2022 security update