Forum Discussion

Deepanshu_Marwah's avatar
Deepanshu_Marwah
Brass Contributor
Jun 23, 2021
Solved

Urgent !! CEF Syslog duplication Issue

Hi All 

I have configured a Fortinet integration with Azure sentinel on local7 facility. My current configuration is ingesting Fortinet logs in both the tables `CommonSecurityLog` and `syslog`. The number of logs are quite large and duplication may cost double cost. Please help to fix this issue 

 

 

Actions tried:

 

1. Performed steps as mentioned in the documentation.

 

 

 

 

Using the same machine to forward both plain Syslog and CEF messages

If you plan to use this log forwarder machine to forward Syslog messages as well as CEF, then in order to avoid the duplication of events to the Syslog and CommonSecurityLog tables:

On each source machine that sends logs to the forwarder in CEF format, you must edit the Syslog configuration file to remove the facilities that are being used to send CEF messages. This way, the facilities that are sent in CEF won't also be sent in Syslog. See Configure Syslog on Linux agent for detailed instructions on how to do this.

You must run the following command on those machines to disable the synchronization of the agent with the Syslog configuration in Azure Sentinel. This ensures that the configuration change you made in the previous step does not get overwritten.
sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable'

 

 

 

 

 

2. Tried removing syslog facilities on log analytics workspace => agent configuration => syslog

 

 

  • CliveWatson's avatar
    CliveWatson
    Jun 28, 2021
    Please take a look at https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux-troubleshoot#log-analytics-troubleshooting-tool and the note
    https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux-troubleshoot#important-configuration-files

    Editing configuration files for performance counters and Syslog is overwritten if the collection is configured from the data menu Log Analytics Advanced Settings in the Azure portal for your workspace. To disable configuration for all agents, disable collection from Log Analytics Advanced Settings or for a single agent run the following: sudo /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable && sudo rm /etc/opt/omi/conf/omsconfig/configuration/Current.mof* /etc/opt/omi/conf/omsconfig/configuration/Pending.mof*

6 Replies

Sort By
  • RogerS's avatar
    RogerS
    Copper Contributor
    Jan 28, 2025

    I have created a new file in /etc/rsyslog.d/ called 10-filter-CEF.conf and I've tried all sorts of combinations to stop the CEF records from Fortigate firewall being duplicated into the syslog without success.
    Some of the combinations I've tried are below; none of which seem to work. Help!!!
    #:programname, contains, "CEF" /var/log/CEF.log
    #& stop
    #:hostname, contains, "prdvmfwall01" /var/log/CEF.log
    #& stop
    #:msg, contains, "prdvmfwall01 CEF" /var/log/CEF.log
    #& stop
    #:msg, contains, "0|Fortinet|Fortigate|" /var/log/CEF.log
    #& stop
    #:rawmsg, contains, "CEF:" /var/log/CEF.log
    #& stop
    #:rawmsg, contains, "CEF:" ~
    #if ($rawmsg contains "CEF:") then stop
    #$rawmsg contains "CEF:" then @@127.0.0.1:25226
    #& stop
    #:msg, contains, "CEF:" ~
    if ($programname == "CEF" or $rawmsg contains "CEF:") then stop
    #if ($rawmsg contains "CEF:") then stop

    • RogerS's avatar
      RogerS
      Copper Contributor
      Jan 30, 2025

      I'm now getting closer.

      I've created a file /etc/rsyslog.d/05-filter-CEF.conf that contains the following:
      if ($programname == "CEF") then @@127.0.0.1:28330 
      & stop

      By doing this I still get CEF records in the syslog.

      As suggested by some other articles I changed the port to 25226 but then don't get CEF records in syslog or CommonSecurityLog.

      Please can anybody help

    • RogerS's avatar
      RogerS
      Copper Contributor
      Jan 29, 2025

      I found the problem, I had to prefix the file with 05 so that it got processed before 10-azuremonitoragent-omfwd.conf. With 05 prefix most of the combinations above would work

  • CliveWatson's avatar
    CliveWatson
    Former Employee
    Jun 23, 2021
    For option 2, did you press SAVE after making the change? After that point (as it wont delete already ingested data) are you still getting 'new' duplicates into the Syslog table?
    • Deepanshu_Marwah's avatar
      Deepanshu_Marwah
      Brass Contributor
      Jun 28, 2021
      Yes. I pressed the save button. What I observed was default /etc/rsyslog.conf contains the syslog facilities as well that adds duplicate values in syslog table. Also, whenever I remove the entries from 95-omsagent.conf for any facility as per docs, it reappears after 5 mins in the conf file.
      • CliveWatson's avatar
        CliveWatson
        Former Employee
        Jun 28, 2021
        Please take a look at https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux-troubleshoot#log-analytics-troubleshooting-tool and the note
        https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux-troubleshoot#important-configuration-files

        Editing configuration files for performance counters and Syslog is overwritten if the collection is configured from the data menu Log Analytics Advanced Settings in the Azure portal for your workspace. To disable configuration for all agents, disable collection from Log Analytics Advanced Settings or for a single agent run the following: sudo /opt/microsoft/omsconfig/Scripts/OMS_MetaConfigHelper.py --disable && sudo rm /etc/opt/omi/conf/omsconfig/configuration/Current.mof* /etc/opt/omi/conf/omsconfig/configuration/Pending.mof*

Resources