This blog post is authored by Nicholas DiCola
In the last few months working on Microsoft Sentinel, many customers have asked me about scaling up syslog CEF collection for getting data into Mic...
Updated Nov 03, 2021
Version 4.0
Blog Post
Nicholas,
do you have any concerns about scaling down a vm scale set without knowing if it's queue is empty? isn't there some risk of lost messages?
I am toying with the idea of adapting my syslog-ng front-ends to push messages into a service bus queue (or multiple queues), and then using a custom input into logstash to peek a message from the queue, process it through logstash, and then deque the message after processing.
That would remove the need for load balancers, and allow the scale sets to expand/contract based on load (or ideally service bus queue sizes), and guarantee delivery of the messages.
thoughts?