Duplicate logs of CEF with Syslog

[Qusai_Ismail]

Hello,

Is there a way to remove duplication of CommonSecurity and Syslog when Log collector Server is configured to forward CEF and Syslog.

for example F5 WAF firewall sending Syslog with CEF formate in facility Local0, which result to duplication.

We already configured the Log analytic Agent management to fetch the syslog of Local0, bcz there is different sources send with that facility. 

Is there a way to remove duplication when taken into account that we can't change it from the source system(F5 waf)

Thanks.

[#2]

Hello @Qusai_Ismail,

You can try Data collection transformations - Azure Monitor | Microsoft Learn.

[#3]

Thanks, but this need to use Azure Monitor agent, not Log Analytic agent, yes?
Bcz we are using Log analytic agent (OMS agent)

[#4]

@Qusai_Ismail 

I think you can use the data transformation with old Log Analytics agents as well. Because it is done on the Azure level and not on the log forwarder.

Transform or customize data at ingestion time in Microsoft Sentinel (preview) | Microsoft Learn

[#5]

Thank you, we find a workaround and solve it by edit oms configuration (/etc/rsyslog.d/security-config-omsagent.conf) to

if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then {
@@127.0.0.1:25226
stop
}

[#6]

@Qusai_Ismail 

This workaround gets overwritten at some point by the azure sentinel no? 

[#7]

How does it get overwritten?

[#8]

Any issue observed after these changes?

[#9]

Yes it's got overwritten by Sentinel configuration

[#10]

Unfortunately ,it's got overwritten by Sentinel configuration

[#11 · Accepted solution]

i got a solution that worked for me:
i've created a seperate machine used only for CEF logs - on that machine just make an IPTABLES that blocks port 25224.

sudo iptables -A INPUT -p udp --dport 25224 -j DROP
sudo iptables -A OUTPUT -p udp --dport 25224 -j DROP

[#12]

Thank you, i will try this.