Optimizing Microsoft Sentinel: Resolving AMA-Induced Syslog & CEF Duplicates
2) Recommended Solutions When collecting both Syslog and CEF logs from the same Linux collector using the Azure Monitor Agent (AMA) in Microsoft Sentinel, duplicate log entries can occur. These duplicates arise because the same event may be ingested through both the Syslog and CEF pipelines, leading to redundancy in the Log Analytics Workspace (LAW). The following solutions aim to eliminate or reduce duplicate log ingestion, ensuring that: CEF events are parsed correctly and only once. Syslog data remains clean and non-redundant. Storage and analytics efficiency is improved. Alerting and incident investigation are not skewed by duplicate entries. Each option provides a different strategy based on your environment’s flexibility and configuration capabilities—from facility-level separation, to ingestion-time filtering, to daemon-side log routing. Option 1: Facility Separation (Preferred) Configure devices to emit CEF logs on a dedicated facility (for example, 'local4'), and adjust the Data Collection Rules (DCRs) so that the CEF stream includes only that facility, while the Syslog stream excludes it. This ensures CEF events are parsed once into 'CommonSecurityLog' and never land in 'Syslog'. CEF via AMA DCR (include only CEF facility): { "properties": { "dataSources": { "syslog": [ { "streams": ["Microsoft-CommonSecurityLog"], "facilityNames": ["local4"], "logLevels": ["*"], "name": "cefDataSource" } ] }, "dataFlows": [ { "streams": ["Microsoft-CommonSecurityLog"], "destinations": ["laDest"] } ] } } Syslog via AMA DCR (exclude CEF facility): { "properties": { "dataSources": { "syslog": [ { "streams": ["Microsoft-Syslog"], "facilityNames": [ "auth","authpriv","cron","daemon","kern","mail", "syslog","user","local0","local1","local2","local3", "local5","local6","local7" ], "logLevels": ["*"], "name": "syslogDataSource" } ] }, "dataFlows": [ { "streams": ["Microsoft-Syslog"], "destinations": ["laDest"] } ] } } Option 2: Ingest-time Transform (Drop CEF from Syslog) If facility separation is not feasible, apply a transformation to the Syslog stream in the DCR so that any CEF-formatted messages are dropped during ingestion. Syslog stream transformKql: { "properties": { "dataFlows": [ { "streams": ["Microsoft-Syslog"], "transformKql": "source | where not(SyslogMessage startswith 'CEF:')", "destinations": ["laDest"] } ] } } Option 3: Daemon-side Filtering/Rewriting (rsyslog/syslog-ng) Filter or rewrite CEF messages before AMA sees them. For example, route CEF messages to a dedicated facility using syslog-ng and stop further processing: # Match CEF filter f_cef { message("^CEF:"); }; # Send CEF to local5 and stop further processing log { source(s_src); filter(f_cef); rewrite { set_facility(local5); }; destination(d_azure_mdsd); flags(final); } 3) Verification Steps with KQL Queries Detect CEF messages that leaked into Syslog: Syslog | where TimeGenerated > ago(1d) | where SyslogMessage startswith "CEF:" | summarize count() by Computer | order by count_ desc Estimate duplicate count across Syslog and CommonSecurityLog: let sys = Syslog | where TimeGenerated > ago(1d) | where SyslogMessage startswith "CEF:" | extend key = hash_sha256(SyslogMessage); let cef = CommonSecurityLog | where TimeGenerated > ago(1d) | extend key = hash_sha256(RawEvent); cef | join kind=innerunique (sys) on key | summarize duplicates = count() Note : You should identify the RawEvent that might be causing the duplicates. 3.1) Duplicate Detection Query Explained This query helps quantify duplicate ingestion when both Syslog and CEF connectors ingest the same events. It works as follows: Build the Syslog set (sys): Filter the 'Syslog' table for the last day and keep only messages that start with 'CEF:'. Compute a SHA-256 hash of the entire message as a stable join key ("key"). Build the CEF set (cef): Filter the 'CommonSecurityLog' table for the last day and compute a SHA-256 hash of the 'RawEvent' field as the same-style join key. Join on the key: Use 'join kind=innerunique' to find messages that exist in both sets (i.e., duplicates). Summarize: Count the number of matching rows to get a duplicate total. 4) Common Pitfalls - Overlapping DCRs applied to the same collector VM causing overlapping facilities/severities. - CEF and Syslog using the same facility on sources, leading to ingestion on both streams. - rsyslog/syslog-ng filters placed after AMA’s own configuration include (ensure your custom rules run before '10-azuremonitoragent.conf'). 5) References - Microsoft Learn: Ingest syslog and CEF messages to Microsoft Sentinel with AMA (https://learn.microsoft.com/en-us/azure/sentinel/connect-cef-syslog-ama)Ask Microsoft Anything: Security Service Edge (SSE)
Ask Microsoft Anything about securing access to any app or resource, from anywhere. Our panel of experts will answer you questions about Microsoft Entra Internet Access, Microsoft Entra Private Access, and how these products work together as part of Microsoft's Security Service Edge solution. This session is part of the Microsoft Entra Suite Tech Accelerator. Get a head start Watch Zero Trust in the Age of AI to learn how to simplify your Zero Trust strategy with the latest end-to-end security innovations.Ask Microsoft Anything: Discover the Microsoft Entra Suite
Ask Microsoft Anything…about the Microsoft Entra Suite! Join our panel of experts with your questions about unifying Conditional Access policies for identities and networks, and ensuring least privilege access for all users accessing all resources and apps, including AI. Want to improve the user experience for both in-office and remote workers while reducing the complexity and cost of managing security tools? We're here to help with tips! This session is part of the Microsoft Entra Suite Tech Accelerator. Get a head start Watch Zero Trust in the Age of AI to learn how to simplify your Zero Trust strategy with the latest end-to-end security innovations.AMA: Security for AI
Looking for tips on how to prepare your environment for secure AI adoption? What’s the best way to protect your AI stack and sensitive data? Join this Ask Microsoft Anything session to get answers to your questions from the teams adopting security for AI at Microsoft! Big questions or small ones—-we’re here to help you confidently embrace the age of AI with industry-leading cybersecurity and compliance solutions. This session is part of Tech Community Live: Microsoft Security edition.AMA: Microsoft Security Exposure Management
If you are responsible for maintaining a strong security posture in your organization, don’t miss this Ask Microsoft Anything session! Microsoft Security Exposure Management can help you get a unified view of attack surface across your organization, manage and investigate attack paths, manage exposure, and better prioritize and safeguard critical assets. Check out our Tech Community blogs aka.ms/exposuremanagement/blogs and bring your questions about getting ahead of attackers by using and getting the most from Exposure Management. This session is part of Tech Community Live: Microsoft Security edition.AMA: Security Service Edge (SSE)
Join us to explore Microsoft’s SSE (Global Secure Access) partner ecosystem, where we collaborate with top industry leaders to deliver integrated, identity-centric solutions for enhanced security and seamless connectivity. Ask Microsoft Anything and learn how these partnerships are simplifying security and networking, empowering you to support your hybrid workforce effectively. This session is part of Tech Community Live: Microsoft Security edition.AMA: Microsoft Defender for Cloud
Ask Microsoft Anything…about Microsoft Defender for Cloud and get your questions answered! Join our panel of experts to discuss our latest innovations announced at Microsoft Ignite, including Endor Labs integration for supply chain security, multiple posture management enhancements, and new container security features. This session is part of Tech Community Live: Microsoft Security edition.AMA: Microsoft Entra Suite
Join our panel of experts with your questions about the Microsoft Entra Suite. Want to secure access for your employees and extend Conditional Access to all apps – including on-premises? Want to retire old VPNs and automate identity lifecycle workflows? We’re here to share best practices and insights about how our identity and network access products can help. Ask Microsoft Anything! This session is part of Tech Community Live: Microsoft Security edition.AMA: Microsoft Security Copilot
Have questions about how to best use Microsoft Security Copilot to respond to cyberthreats quickly and assess risk exposure in minutes? Ask Microsoft Anything! This session is your opportunity to get answers from the product team to help you configure Microsoft Security Copilot and process signals at machine speed! This session is part of Tech Community Live: Microsoft Security edition.Transform your defense: Microsoft Security Exposure Management
Learn how Exposure Management consolidates risk-based views of the attack surface and provides advanced attack path modeling. Learn how to use these capabilities to reduce your organization’s attack surface and limit an adversary’s opportunity for attack. With an expanding attack surface and adversaries constantly evolving it is critical that defenders have a comprehensive view that supports them to effectively reduce risk across the digital estate. Join this deep dive to get to know Exposure Management in and out. We'll wrap up with an Ask Microsoft Anything (AMA) section so post your questions for our product team!. This session is part of the Microsoft Secure Tech Accelerator. Add it to your calendar, RSVP for event reminders, and post your questions and comments below! This session will also be recorded and available on demand shortly after conclusion of the live event.
