Dataloss in transfer between syslog and Azure LogAnalytics

[reda21]

We're experiencing an issue where we lose data between our syslog server data and the results in Sentinels LogAnalytics.

Our configuration writes syslog messages to a file /var/syslog_data. This file is read by the rsyslog imfile module to forward its content to the OMS agents syslog port (25224), to identify this data we gave it the syslog tag "sys_data". We found this issue by comparing the counted lines in the /var/syslog_data file and the data received in the Sentinel workspace per day for the last week. The results show a mismatch of ±80% with peeks up to 90% of dataloss, see attached image for comparison results.

To see if the configuration works properly we restarted the omsagent and added a new line tot the /var/syslog_data file with a copy of a line including some recognisable content. This was shown in Sentinel within 5 minutes. Based on this test we conclude that the configuration is set properly.

rsyslog config:

module(load="imfile")

ruleset(name="syslog_data"){
 action(type="omfwd" Target="127.0.0.1" Port="25224" Protocol="udp")
 stop 
}

input(type="imfile" File="/var/log/syslog_data.log" Tag="sys_data" Severity="warn" Facility="local5" PersistStateInterval="1" ruleset="syslog_data" reopenOnTruncate="on") 

omsagent.d/syslog.conf config:

<source>
  type syslog
  port 25224
  bind 127.0.0.1
  protocol_type udp
  tag oms.syslog
</source>

<filter oms.syslog.**>
  type filter_syslog
</filter>

[#2]

@reda21 That's odd, but I wonder about your format.

What process(es) are writing to "/var/syslog_data"? It may make more sense to configure the connector (OMS Agent) to read /var/syslog_data as a Custom Log versus send it in via syslog? If you do it that way you can see if the issue is with rsyslog forwarding the data, Sentinel assuming everything in /var/syslog_data is in syslog format, and if the problem is between your host and Azure. You can compare the size of the new data table (_CL table) to the number of lines in the /var/syslog_data file and see if any are lost between your site and Sentinel.

[#3]

@JKatzmandu thanks for your reply, could you guid me to any information options or documentation which have info on how to configure the OMS agent to read the file directly?

[#4]

@reda21 Once the agent is installed (which it is) you tell it to read a flat log file. It's basically like this:

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-custom-logs