Jim and the Directory Services Team here again to alert you to an emerging issue which is an unintended consequence of a recent update released in March 2024.
What is LSASS and why is it important?
The Local Security Authority Subsystem Service (LSASS) is a process that handles user authentication, security policies, and auditing on Windows systems. It is essential for the proper functioning of your computer, as it verifies your identity and facilitates your access to your files and applications. For domain controllers, it has the additional responsibility of hosting the Active Directory related services that provide authentication, replication, database query processing, and other domain functions.
Given the importance of the LSASS process, most Enterprise environments monitor its operation and alert when LSASS is consuming a large amount of CPU or memory resources affecting the system’s performance. This can happen due to assorted reasons, but in this blog post, we will focus on one specific cause that has been recently reported and is currently being addressed by the Microsoft Product Group.
What is the 3B Windows update and how does it affect LSASS?
As of March 18, 2024, customers are experiencing excessive memory consumption by LSASS on Windows Server 2012-2022 DCs that have installed the following Windows Update(s):
KB 5035857: March 12, 2024, KB5035857 (OS Build 20348.2340) Windows Server 2022
KB 5035849: March 12, 2024, KB5035849 (OS Build 17763.5576) Windows Server 2019
KB 5035855: March 14, 2024, KB5035855 (OS Build 14393.5786) Windows Server 2016
KB 5035885: March 12, 2024, KB5035885 Monthly Rollup for Windows Server 2012 R2: March 12, 2024
Affected platforms: Windows Server 2022, Windows Server 2019, Windows Server 2016, Windows Server 2012 R2
Following installation of the March 2024 security updates released March 12, 2024, the Local Security Authority Subsystem Service (LSASS) may experience a memory leak on domain controllers (DCs). This is observed when on-premises and cloud-based Active Directory Domain Controllers service Kerberos authentication service requests (AS-REQ).
SYMPTOMS
Log Name: System
Source: Microsoft-Windows-Resource-Exhaustion-Detector
Event ID: 2004
Task Category: Resource Exhaustion Diagnosis Events
Level: Warning
Keywords: Events related to exhaustion of system commit limit (virtual memory).
User: SYSTEM
Computer: <hostname> Description:
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: lsass.exe (PID) consumed <amount of memory in> bytes, <filename>.<extension> (PID ) consumed <amount of memory in> bytes, and <filename>.<extension> (PID) consumed <amount of memory in> bytes.
Alternatively, if you have other resource monitoring software, you may want to leverage it for restarts to keep in line with organizational requirements and procedures.
LSASS Memory leaks at the rate of 2GB per hour have been observed. Memory exhaustion may cause application or service crashes, including the crashing of LSASS which in turn will trigger a reboot of the underlying OS. In addition, customers who have very busy domain controllers will experience not only the memory leak, but these sorts of heap leaks in LSASS typically also cause a lot of heap fragmentation. This heap fragmentation can cause a surprisingly severe CPU performance penalty in addition to just memory growth. The high CPU usage may be the first performance indicator seen and could be indicative of the underlying memory leak problem.
LSASS Private Bytes increases linearly with system uptime:
For more information, see Use Performance Monitor to Find a User-Mode Memory Leak - Windows drivers | Microsoft Learn.
Task Manager
Task Manager shows LSASS consuming significant percentage of memory:
Lsass.exe Process Exceptions
LSASS crashes and reboots the entire server after LSASS consumes sufficient memory. LSASS crashes and device reboots will occur more often on physical and virtual machines with LESS memory.
Associated event log entries:
Log Name: Application
Source: Application Error
Event ID 1000:
Faulting application name: lsass.exe, version: 6.3.9600.17415, time stamp: 0x545042fe
Faulting module name: kerberos.DLL, version: 6.3.9600.17423, time stamp: 0x545ff681
Exception code: 0xc0000005
Fault offset: 0x00000000000910b7
Faulting process id: 0x448
Faulting application start time: 0x01d029e23a389f2e
Faulting application path: C:\Windows\system32\lsass.exe
Faulting module path: C:\Windows\system32\kerberos.DLL
Log Name: System
Source: User32
Event ID: 1074
User: SYSTEM
Description:
The process wininit.exe has initiated the restart of computer <COMPUTERNAME> on behalf of user for the following reason: No title for this reason could be found
Reason Code: 0x50006
Shutdown Type: restart
Comment: The system process 'C:\WINDOWS\system32\lsass.exe' terminated unexpectedly with status code -1073741819. The system will now shut down and restart.
Log Name: Application
Source: Microsoft-Windows-Wininit
Date: DateTime
Event ID: 1015
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: ComputerName
Description:
A critical system process, C:\WINDOWS\system32\lsass.exe, failed with status code c0000005. The machine must now be restarted.
How to fix high LSASS usage after Windows update?
Updated 4/15/2024
We have seen some questions in the comments around whether the April 2024 Cumulative/Security updates do or do not include the Out of Band release for the March 2024 update. The short answer is yes, the OOB update is superseded by the April cumulative Update, and at this time you can skip the March updates if you have not applied them as of yet and install just the April 2024 update if you would like.
The commenters are correct that the release notes do not specifically list this update. So how do you know that it has been superseded, and more importantly how can windows administrators find out this information without relying on forums or opening a support case? Well, fortunately you can do this by looking at the Microsoft Update catalog at the address https://aka.ms/updatecatalog.
1. Once you get to the page you can type in the KB number in the search bar that you are interested in.
2. Once you get the result back showing you the update that you searched on, click on the “Title” of the update.
3. It will spawn a context browser showing you information about the specific update. We are interested in the Package Details tab on this page.
4. The Red box shows what updates are replaced by this update, while the Green box shows what updates replace this update.
So, by looking at the information about the Windows Server 2022 March 2024 OOB (out of band) update Catalog site we can determined that this update has been superseded by April 2024 Cumulative update.
Background details and previous guidance
How long until your domain controller begins to experience failures after the March update is installed varies based on how much RAM is available to it, and how much authentication traffic is being sent to it. If it is critical to have your DC's reboot before running out of memory, an Event Trigger for Event ID 2004 could be configured to reboot the server when that event is logged if that would help.
However, if your DC's have a large amount of memory, you may just want to perform proactive periodic reboots of your domain controllers before they hit their maximum memory range.
Fortunately, there are workable solutions that you can use to address the high LSASS usage after the 3b Windows update has been installed. See the following FIRST for installation details and methodologies -
https://learn.microsoft.com/en-us/windows/release-health/status-windows-server-2022#march-2024
The root cause has been identified and the current resolution is an Out-of-band update (OOB) available as of NOW!!!
The OOB update is available via the Windows Catalog location links below. The OOB update will NOT be available through the normal Windows update channels.
Server 2022 3OOB: March 22, 2024—KB5037422 (OS Build 20348.2342) Out-of-band - Microsoft Support
Windows Server 2019 3OOB: March 25, 2024—KB5037425 (OS Build 17763.5579) Out-of-band - Microsoft Support
Server 2016 3OOB: March 22, 2024—KB5037423 (OS Build 14393.6799) Out-of-band - Microsoft Support
SupportServer 2012 R2 3OOB: KB5037426: Update to address a known issue that affects LSASS in Windows Server 2012 R2 - Microsoft ...
Download the aforementioned OOB update from the links provided above for your operating system and install.
You do not have to uninstall the 3b update prior to installing the OOB update. If you have not installed the 3b update you can just install the OOB update instead.
Uninstalling the 3b Windows update is not recommended. Although this may seem like the most straightforward and effective way to resolve the issue, your servers will remain vulnerable to multiple bug fixes and other CVEs that ship in the average Monthly Updates.
Jim “looking forward to the next update 
” Tierney and the DS Gang!