Forum Discussion

RogerS's avatar
RogerS
Copper Contributor
Jan 30, 2025

Cannot stop CEF duplication to syslog when both processed by same Linux VM

We have a situation where we are sending CEF records from FortiGate firewall to Microsoft Sentinel via Common Event Format (CEF) via AMA Data connector and we also use Syslog via AMA Data connector (both on the same Ubuntu Linux VM using rsyslog) and result is that we are getting duplicates of the CEF records in the syslog.
I've read a lot of articles about the duplication and possible ways to fix however I've had not success.
My most recent attempt is to create a file /etc/rsyslog.d/05-filter-CEF.conf with the following entries:
if ($programname == "CEF") then @@127.0.0.1:28330 
& stop
Unfortunately we still get duplicates.

One article I read said to use @@127.0.0.1:25226 however then we don't get CEF records in a CommonSecurityLog or Syslog.

Is there anyone that can help?

data collection
siem

3 Replies

Sort By
  • jandrusk's avatar
    jandrusk
    Copper Contributor
    Mar 18, 2025

    I know this isn't an orthodox method, but I ran into the same issue, but my issue was running out of local log space in /var/log as a result of this, so I put together this hack since not even MS support could give me a solution that worked.

    This runs as a daily cron job. 

     

    #!/bin/bash
# Delete CEF messages from /data/var/log/messages so we don't run out of disk space ;)
export SYSLOG_FILE='/var/log/messages'

sed -i '/CEF:'/d $SYSLOG_FILE  # Delete CEF messages
sed -i '/^$/d' $SYSLOG_FILE  # Delete any blank lines from previous operation

service rsyslog restart

     

  • MHenshaw's avatar
    MHenshaw
    Brass Contributor
    Mar 03, 2025

    Hey 

    You need to add this line of code to you Data collection rule for syslog, this will stop the duplicate data 

    so you dont need to mess around with rsyslog : )

    • "transformKql": " source\n | where ProcessName !contains \"CEF\"\n"

    https://learn.microsoft.com/en-us/azure/sentinel/cef-syslog-ama-overview?tabs=single

    Thanks 

    Matt

  • luchete's avatar
    luchete
    Iron Contributor
    Mar 02, 2025

    Hi RogerS,

    The issue is that the rsyslog is forwarding CEF logs to both destinations (AMA Data Connector and Syslog), causing duplication. The key is ensuring that rsyslog only forwards the logs to one destination and stops processing them after that.

    Try updating the filter to make sure it properly stops processing after forwarding the logs. You can modify the configuration like this:

    if ($programname == "CEF") then {
   action(type="omfwd" Target="127.0.0.1" Port="28330" Protocol="tcp")
   stop
}

    This ensures that the CEF records are only forwarded once to the right destination.

    Regards!

Resources