Home

High CPU/Memory utilization using WMI to read Security Event log

%3CLINGO-SUB%20id%3D%22lingo-sub-330639%22%20slang%3D%22en-US%22%3EHigh%20CPU%2FMemory%20utilization%20using%20WMI%20to%20read%20Security%20Event%20log%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-330639%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Tech%20Community%2C%3CBR%20%2F%3E%3CBR%20%2F%3EWe%20have%202%20systems%20that%20read%20the%20Security%20Event%20log%20of%20our%20three%202012%20R2%20DC's%2C%20a%20SIEM%20(Sentinel)%20and%20Netwrix%20account%20lockout%20examiner%20(these%20have%20been%20operational%20for%20many%20years%20and%20no%20changes%20have%20been%20made%20to%20either).%20Since%20November%20last%20year%2C%20the%20CPU%20and%20memory%20usage%20of%20all%20DC's%20jumped%20up%20from%20average%2040%25%20to%2080%25%20and%20RAM%20usage%20increased%20by%204GB.%20I%20know%20the%20cause%20of%20this%20high%20usage%20is%20the%20WMI%20calls%20reading%20the%204GB%20Security%20log.%20Using%20ProcMon%20I%20can%20see%20the%202%20threads%20reading%20the%20log%20continuously%20from%20beginning%20to%20end.%20I%20am%20making%20an%20educated%20guess%20that%20prior%20to%20November%2C%20the%20remote%20WMI%20calls%20would%20only%20read%20the%20delta%20changes%20to%20the%20Event%20log%2C%20which%20is%20the%20how%20I%20would%20expect%20it%20to%20work.%20Why%20is%20it%20now%2C%20the%20complete%204GB%20file%20is%20read%3F%3F%20I%20have%20also%20used%20RAMMap%20and%20can%20see%20that%20the%20Security.evtx%20file%20is%20completely%20loaded%20into%20RAM%2C%20understandably%20so%2C%20since%20the%20file%20is%20constantly%20being%20read.%3CBR%20%2F%3E%3CBR%20%2F%3EThe%20only%20change%20made%2C%2012%20hours%20prior%20to%20this%20issue%20appearing%20is%20that%20we%20uplifted%20our%20DFL%20and%20FFL%20from%202003%20to%202012%20R2%20(DC's%20have%20been%20running%20on%20Server%202012%20R2%20for%20at%20least%2018%20months).%20I%20can't%20see%20why%20that%20would%20cause%20this%20issue.%20Since%20then%2C%20to%20rule%20out%20DC's%2C%20I%20have%20run%20up%20a%202008%20R2%20member%20server%2C%20loaded%20the%20log%20with%201%20GB%20of%20events%2C%20and%20pointed%20our%20SIEM%20to%20read%20the%20log%20and%20the%20same%20problem%20occurs%20(also%20did%20the%20same%20on%20a%202016%20server%2C%20same%20problem).%3CBR%20%2F%3E%3CBR%20%2F%3EI%20have%20spent%20many%20hours%20searching%20the%20Internet%2C%20but%20have%20not%20found%20any%20information%20regarding%20this%20issue.%20As%20both%20systems%20use%20WMI%20to%20read%20the%20event%20log%2C%20this%20is%20only%20common%20factor%20I%20can%20see.%20I%20have%20tried%20disabling%20the%20SIEM%20to%20see%20if%20running%20both%2C%20concurrently%2C%20would%20mess%20up%20the%20location%20Netwrix%20had%20previously%20read%2C%20but%20no%2C%20the%20log%20would%20continue%20reading%20from%20start%20to%20end.%20If%20I%20disable%20both%20then%20CPU%2FRAM%20usage%20would%20go%20back%20normal%20levels.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20question%20keeps%20coming%20back%20to%20me%2C%20%22did%20uplifting%20the%20functional%20levels%20somehow%20change%20the%20way%20WMI%20reads%20event%20logs%3F%3F%3F%22%3CBR%20%2F%3E%3CBR%20%2F%3EAny%20help%20would%20be%20much%20appreciated%3CBR%20%2F%3EThanks%3CBR%20%2F%3EMark%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EEDIT%3A%3C%2FP%3E%3CP%3EI%20forgot%20to%20mention%20that%20I%20have%20also%20cleared%20the%20Security%20log%20on%20a%20DC%2C%20problem%20still%20existed.%20I%20then%20recreated%20the%20log%20(stopped%20service%20and%20renamed%20file)%2C%20problem%20still%20exists.%20Have%20also%20rebooted%20DC.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-330639%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EWindows%20Server%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
m_giusti
Established Member

Hi Tech Community,

We have 2 systems that read the Security Event log of our three 2012 R2 DC's, a SIEM (Sentinel) and Netwrix account lockout examiner (these have been operational for many years and no changes have been made to either). Since November last year, the CPU and memory usage of all DC's jumped up from average 40% to 80% and RAM usage increased by 4GB. I know the cause of this high usage is the WMI calls reading the 4GB Security log. Using ProcMon I can see the 2 threads reading the log continuously from beginning to end. I am making an educated guess that prior to November, the remote WMI calls would only read the delta changes to the Event log, which is the how I would expect it to work. Why is it now, the complete 4GB file is read?? I have also used RAMMap and can see that the Security.evtx file is completely loaded into RAM, understandably so, since the file is constantly being read.

The only change made, 12 hours prior to this issue appearing is that we uplifted our DFL and FFL from 2003 to 2012 R2 (DC's have been running on Server 2012 R2 for at least 18 months). I can't see why that would cause this issue. Since then, to rule out DC's, I have run up a 2008 R2 member server, loaded the log with 1 GB of events, and pointed our SIEM to read the log and the same problem occurs (also did the same on a 2016 server, same problem).

I have spent many hours searching the Internet, but have not found any information regarding this issue. As both systems use WMI to read the event log, this is only common factor I can see. I have tried disabling the SIEM to see if running both, concurrently, would mess up the location Netwrix had previously read, but no, the log would continue reading from start to end. If I disable both then CPU/RAM usage would go back normal levels.

 

The question keeps coming back to me, "did uplifting the functional levels somehow change the way WMI reads event logs???"

Any help would be much appreciated
Thanks
Mark

 

EDIT:

I forgot to mention that I have also cleared the Security log on a DC, problem still existed. I then recreated the log (stopped service and renamed file), problem still exists. Have also rebooted DC.

Related Conversations
Stable version of Edge insider browser
HotCakeX in Discussions on
35 Replies
Tabs and Dark Mode
cjc2112 in Discussions on
30 Replies
flashing a white screen while open new tab
Deleted in Discussions on
14 Replies
Security Community Webinars
Valon_Kolica in Security, Privacy & Compliance on
7 Replies