How many SysLog servers needed

%3CLINGO-SUB%20id%3D%22lingo-sub-1365631%22%20slang%3D%22en-US%22%3EHow%20many%20SysLog%20servers%20needed%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1365631%22%20slang%3D%22en-US%22%3E%3CP%3EIs%20there%20any%20sort%20of%20documentation%20that%20states%20a%20Syslog%20server%20of%20X%20size%20can%20handle%20Y%20amount%20of%20traffic%3F%26nbsp%3B%20I%20thought%20I%20saw%20something%20somewhere%20but%20I%20cannot%20seem%20to%20locate%20the%20document%20again.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1378526%22%20slang%3D%22en-US%22%3ERe%3A%20How%20many%20SysLog%20servers%20needed%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1378526%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F46875%22%20target%3D%22_blank%22%3E%40Gary%20Bushey%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThat's%20a%20good%20question%20and%20I'm%20guessing%20that%20it%20is%20not%20just%20about%20a%20syslog%20server%20in%20general%20but%20also%20about%20their%20ability%20to%20upload%20the%20logging%20data%20to%20Sentinel.%20I%20think%20that%20will%20be%20a%20bottleneck%20when%20large%20volumes%20of%20logs%20are%20involved%20as%20a%20Linux%20syslog%20server%20can%20be%20tweaked%20to%20support%20a%20high%20volume%20of%20events%20per%20second.%20I%20think%20that%20Log%20Analytics%20API%20only%20supports%20chunks%20of%2030%20MB%20upload%20per%20post%20so%20depending%20on%20the%20available%20bandwidth%20one%20can%20do%20some%20math%20on%20how%20many%20collectors%20would%20be%20needed%20for%20a%20specific%20volume%20of%20raw%20logs.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20typically%20start%20with%20a%202%20x%20CPU%2C%208%20GB%20RAM%2C%20minimal%20CentOS%207.7%20VM%20and%20it%20seems%20to%20be%20falling%20asleep%20for%2015%20-%2020%20GB%2Fday%20(using%20standard%20UDP-based%20syslog%20traffic).%20When%20dealing%20with%20more%20stringent%20requirements%2C%20such%20as%20high%20availability%20one%20can%20start%20introducing%20load%20balancers%2C%20maybe%20engage%20in%20real%20devops%20by%20spinning%20syslog%20containers%20through%20Kubernetes%2C%20maybe%20use%20Kafka%20to%20manage%20the%20log%20stream.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20guess%20that%20empirically%2C%20one%20can%20setup%20a%20log%20generator%20and%20flood%20a%20%22standard%22%20syslog%20server%20to%20see%20where%20it%20starts%20to%20fall%20apart.%20I%20will%20probably%20add%20this%20to%20my%20to-do%20list.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAdrian%20Grigorof%3C%2FP%3E%3CP%3E%3CA%20href%3D%22http%3A%2F%2Fwww.managedsentinel.com%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ewww.managedsentinel.com%3C%2FA%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E
Super Contributor

Is there any sort of documentation that states a Syslog server of X size can handle Y amount of traffic?  I thought I saw something somewhere but I cannot seem to locate the document again.

1 Reply

@Gary Bushey 

That's a good question and I'm guessing that it is not just about a syslog server in general but also about their ability to upload the logging data to Sentinel. I think that will be a bottleneck when large volumes of logs are involved as a Linux syslog server can be tweaked to support a high volume of events per second. I think that Log Analytics API only supports chunks of 30 MB upload per post so depending on the available bandwidth one can do some math on how many collectors would be needed for a specific volume of raw logs. 

 

We typically start with a 2 x CPU, 8 GB RAM, minimal CentOS 7.7 VM and it seems to be falling asleep for 15 - 20 GB/day (using standard UDP-based syslog traffic). When dealing with more stringent requirements, such as high availability one can start introducing load balancers, maybe engage in real devops by spinning syslog containers through Kubernetes, maybe use Kafka to manage the log stream.

 

I guess that empirically, one can setup a log generator and flood a "standard" syslog server to see where it starts to fall apart. I will probably add this to my to-do list.

 

Adrian Grigorof

www.managedsentinel.com