Urgent !! CEF Syslog duplication Issue

[Deepanshu_Marwah]

Hi All 

I have configured a Fortinet integration with Azure sentinel on local7 facility. My current configuration is ingesting Fortinet logs in both the tables `CommonSecurityLog` and `syslog`. The number of logs are quite large and duplication may cost double cost. Please help to fix this issue 

 

Actions tried:

1. Performed steps as mentioned in the documentation.

Using the same machine to forward both plain Syslog and CEF messages

If you plan to use this log forwarder machine to forward Syslog messages as well as CEF, then in order to avoid the duplication of events to the Syslog and CommonSecurityLog tables:

On each source machine that sends logs to the forwarder in CEF format, you must edit the Syslog configuration file to remove the facilities that are being used to send CEF messages. This way, the facilities that are sent in CEF won't also be sent in Syslog. See Configure Syslog on Linux agent for detailed instructions on how to do this.

You must run the following command on those machines to disable the synchronization of the agent with the Syslog configuration in Azure Sentinel. This ensures that the configuration change you made in the previous step does not get overwritten.
sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable'

2. Tried removing syslog facilities on log analytics workspace => agent configuration => syslog

 

[#2]

For option 2, did you press SAVE after making the change? After that point (as it wont delete already ingested data) are you still getting 'new' duplicates into the Syslog table?

[#3]

Yes. I pressed the save button. What I observed was default /etc/rsyslog.conf contains the syslog facilities as well that adds duplicate values in syslog table. Also, whenever I remove the entries from 95-omsagent.conf for any facility as per docs, it reappears after 5 mins in the conf file.

[#4 · Accepted solution]

Please take a look at https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux-troubleshoot#log-analytics-t... and the note
https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux-troubleshoot#important-confi...

Editing configuration files for performance counters and Syslog is overwritten if the collection is configured from the data menu Log Analytics Advanced Settings in the Azure portal for your workspace. To disable configuration for all agents, disable collection from Log Analytics Advanced Settings or for a single agent run the following: sudo /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable && sudo rm /etc/opt/omi/conf/omsconfig/configuration/Current.mof* /etc/opt/omi/conf/omsconfig/configuration/Pending.mof*