Urgent !! CEF Syslog duplication Issue

Hi All

I have configured a Fortinet integration with Azure sentinel on local7 facility. My current configuration is ingesting Fortinet logs in both the tables `CommonSecurityLog` and `syslog`. The number of logs are quite large and duplication may cost double cost. Please help to fix this issue

Actions tried:

1. Performed steps as mentioned in the documentation.

Using the same machine to forward both plain Syslog and CEF messages

If you plan to use this log forwarder machine to forward Syslog messages as well as CEF, then in order to avoid the duplication of events to the Syslog and CommonSecurityLog tables:

On each source machine that sends logs to the forwarder in CEF format, you must edit the Syslog configuration file to remove the facilities that are being used to send CEF messages. This way, the facilities that are sent in CEF won't also be sent in Syslog. See Configure Syslog on Linux agent for detailed instructions on how to do this.

You must run the following command on those machines to disable the synchronization of the agent with the Syslog configuration in Azure Sentinel. This ensures that the configuration change you made in the previous step does not get overwritten.
sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable'

2. Tried removing syslog facilities on log analytics workspace => agent configuration => syslog

CliveWatson
Jun 28, 2021
Please take a look at https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux-troubleshoot#log-analytics-troubleshooting-tool and the note
https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux-troubleshoot#important-configuration-files

Editing configuration files for performance counters and Syslog is overwritten if the collection is configured from the data menu Log Analytics Advanced Settings in the Azure portal for your workspace. To disable configuration for all agents, disable collection from Log Analytics Advanced Settings or for a single agent run the following: sudo /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable && sudo rm /etc/opt/omi/conf/omsconfig/configuration/Current.mof* /etc/opt/omi/conf/omsconfig/configuration/Pending.mof*

CliveWatson
Microsoft
Jun 23, 2021
For option 2, did you press SAVE after making the change? After that point (as it wont delete already ingested data) are you still getting 'new' duplicates into the Syslog table?
- Deepanshu_Marwah
  Brass Contributor
  Jun 28, 2021
  Yes. I pressed the save button. What I observed was default /etc/rsyslog.conf contains the syslog facilities as well that adds duplicate values in syslog table. Also, whenever I remove the entries from 95-omsagent.conf for any facility as per docs, it reappears after 5 mins in the conf file.
  - CliveWatson
    Microsoft
    Jun 28, 2021
    Please take a look at https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux-troubleshoot#log-analytics-troubleshooting-tool and the note
    https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux-troubleshoot#important-configuration-files
    
    Editing configuration files for performance counters and Syslog is overwritten if the collection is configured from the data menu Log Analytics Advanced Settings in the Azure portal for your workspace. To disable configuration for all agents, disable collection from Log Analytics Advanced Settings or for a single agent run the following: sudo /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable && sudo rm /etc/opt/omi/conf/omsconfig/configuration/Current.mof* /etc/opt/omi/conf/omsconfig/configuration/Pending.mof*

Forum Discussion

Urgent !! CEF Syslog duplication Issue

Share

Resources