High CPU/Memory utilization using WMI to read Security Event log

[m_giusti]

Hi Tech Community,

We have 2 systems that read the Security Event log of our three 2012 R2 DC's, a SIEM (Sentinel) and Netwrix account lockout examiner (these have been operational for many years and no changes have been made to either). Since November last year, the CPU and memory usage of all DC's jumped up from average 40% to 80% and RAM usage increased by 4GB. I know the cause of this high usage is the WMI calls reading the 4GB Security log. Using ProcMon I can see the 2 threads reading the log continuously from beginning to end. I am making an educated guess that prior to November, the remote WMI calls would only read the delta changes to the Event log, which is the how I would expect it to work. Why is it now, the complete 4GB file is read?? I have also used RAMMap and can see that the Security.evtx file is completely loaded into RAM, understandably so, since the file is constantly being read.

The only change made, 12 hours prior to this issue appearing is that we uplifted our DFL and FFL from 2003 to 2012 R2 (DC's have been running on Server 2012 R2 for at least 18 months). I can't see why that would cause this issue. Since then, to rule out DC's, I have run up a 2008 R2 member server, loaded the log with 1 GB of events, and pointed our SIEM to read the log and the same problem occurs (also did the same on a 2016 server, same problem).

I have spent many hours searching the Internet, but have not found any information regarding this issue. As both systems use WMI to read the event log, this is only common factor I can see. I have tried disabling the SIEM to see if running both, concurrently, would mess up the location Netwrix had previously read, but no, the log would continue reading from start to end. If I disable both then CPU/RAM usage would go back normal levels.

The question keeps coming back to me, "did uplifting the functional levels somehow change the way WMI reads event logs???"

Any help would be much appreciated
Thanks
Mark

EDIT:

I forgot to mention that I have also cleared the Security log on a DC, problem still existed. I then recreated the log (stopped service and renamed file), problem still exists. Have also rebooted DC.

[#2]

Funny that the only posts n the internet regarding this New behavior where there is some kind of answers are not on the official Microsoft Forums @m_giusti .

why is Microsoft silent on this matter?

Microsoft should be more transparent when making changes that have huge impact on memory as this.
we log more and more stuff into security.evtx as per cybersecurity recommendation dictate, thus upping the evtx to 4GB to retain some acceptable retention, but this loading of the file in memory is now affecting our users windows machines as well as our servers's memory consumption.

we now face a dillema, where we need some log retention, but also want to mitigate this memory usage issues...

[#3]

Did you get anywhere with this issue? we have the same issue

[#4]

Hi Scotty,

Unfortunately no, we upped the server specs for all our DC's, doubled CPU to 4 and increased memory by 4GB, which isn't much of a big deal in itself, however it adds extra load to our virtual infrastructure. But, our DC days are numbered, as we will be transitioning to cloud over the next 12 months.

Cheers
Mark

[#5]

@scottystunz :

we ended up compromizing with the infrastructure team by dropping the security.evtx to 2gb, they get some ram back, at the expense of loosing a bit of retention.

noted that some of the events in theres are cherry picked to be sent to SIEM.

the only theory of why it work like this is to be able to continue logging events if the system lose access to disk writes. that way, you can scrape the RAM for the latest evtx in forensic situations.