Optimizing Microsoft Sentinel: Resolving AMA-Induced Syslog & CEF Duplicates
2) Recommended Solutions When collecting both Syslog and CEF logs from the same Linux collector using the Azure Monitor Agent (AMA) in Microsoft Sentinel, duplicate log entries can occur. These duplicates arise because the same event may be ingested through both the Syslog and CEF pipelines, leading to redundancy in the Log Analytics Workspace (LAW). The following solutions aim to eliminate or reduce duplicate log ingestion, ensuring that: CEF events are parsed correctly and only once. Syslog data remains clean and non-redundant. Storage and analytics efficiency is improved. Alerting and incident investigation are not skewed by duplicate entries. Each option provides a different strategy based on your environment’s flexibility and configuration capabilities—from facility-level separation, to ingestion-time filtering, to daemon-side log routing. Option 1: Facility Separation (Preferred) Configure devices to emit CEF logs on a dedicated facility (for example, 'local4'), and adjust the Data Collection Rules (DCRs) so that the CEF stream includes only that facility, while the Syslog stream excludes it. This ensures CEF events are parsed once into 'CommonSecurityLog' and never land in 'Syslog'. CEF via AMA DCR (include only CEF facility): { "properties": { "dataSources": { "syslog": [ { "streams": ["Microsoft-CommonSecurityLog"], "facilityNames": ["local4"], "logLevels": ["*"], "name": "cefDataSource" } ] }, "dataFlows": [ { "streams": ["Microsoft-CommonSecurityLog"], "destinations": ["laDest"] } ] } } Syslog via AMA DCR (exclude CEF facility): { "properties": { "dataSources": { "syslog": [ { "streams": ["Microsoft-Syslog"], "facilityNames": [ "auth","authpriv","cron","daemon","kern","mail", "syslog","user","local0","local1","local2","local3", "local5","local6","local7" ], "logLevels": ["*"], "name": "syslogDataSource" } ] }, "dataFlows": [ { "streams": ["Microsoft-Syslog"], "destinations": ["laDest"] } ] } } Option 2: Ingest-time Transform (Drop CEF from Syslog) If facility separation is not feasible, apply a transformation to the Syslog stream in the DCR so that any CEF-formatted messages are dropped during ingestion. Syslog stream transformKql: { "properties": { "dataFlows": [ { "streams": ["Microsoft-Syslog"], "transformKql": "source | where not(SyslogMessage startswith 'CEF:')", "destinations": ["laDest"] } ] } } Option 3: Daemon-side Filtering/Rewriting (rsyslog/syslog-ng) Filter or rewrite CEF messages before AMA sees them. For example, route CEF messages to a dedicated facility using syslog-ng and stop further processing: # Match CEF filter f_cef { message("^CEF:"); }; # Send CEF to local5 and stop further processing log { source(s_src); filter(f_cef); rewrite { set_facility(local5); }; destination(d_azure_mdsd); flags(final); } 3) Verification Steps with KQL Queries Detect CEF messages that leaked into Syslog: Syslog | where TimeGenerated > ago(1d) | where SyslogMessage startswith "CEF:" | summarize count() by Computer | order by count_ desc Estimate duplicate count across Syslog and CommonSecurityLog: let sys = Syslog | where TimeGenerated > ago(1d) | where SyslogMessage startswith "CEF:" | extend key = hash_sha256(SyslogMessage); let cef = CommonSecurityLog | where TimeGenerated > ago(1d) | extend key = hash_sha256(RawEvent); cef | join kind=innerunique (sys) on key | summarize duplicates = count() Note : You should identify the RawEvent that might be causing the duplicates. 3.1) Duplicate Detection Query Explained This query helps quantify duplicate ingestion when both Syslog and CEF connectors ingest the same events. It works as follows: Build the Syslog set (sys): Filter the 'Syslog' table for the last day and keep only messages that start with 'CEF:'. Compute a SHA-256 hash of the entire message as a stable join key ("key"). Build the CEF set (cef): Filter the 'CommonSecurityLog' table for the last day and compute a SHA-256 hash of the 'RawEvent' field as the same-style join key. Join on the key: Use 'join kind=innerunique' to find messages that exist in both sets (i.e., duplicates). Summarize: Count the number of matching rows to get a duplicate total. 4) Common Pitfalls - Overlapping DCRs applied to the same collector VM causing overlapping facilities/severities. - CEF and Syslog using the same facility on sources, leading to ingestion on both streams. - rsyslog/syslog-ng filters placed after AMA’s own configuration include (ensure your custom rules run before '10-azuremonitoragent.conf'). 5) References - Microsoft Learn: Ingest syslog and CEF messages to Microsoft Sentinel with AMA (https://learn.microsoft.com/en-us/azure/sentinel/connect-cef-syslog-ama)Inside Intune: Live AMA with product leaders
Kick off Tech Community Live with updates and insights from Microsoft Intune engineering leaders. They’ll walk you through where Microsoft Intune and the Microsoft Intune Suite are today, discuss trends in feedback from customers and partners, and outline the vision for the Intune roadmap. This event is part of Tech Community Live: Intune edition. I'm in! How do I sign up? Select “Add to calendar” to save the date, then click the “Attend” button to save your spot, receive event reminders, and participate in the Q&A. If you can’t make the live event, don’t worry. You can post your questions in advance and catch up on the answers and insights later in the week. All sessions for this Intune edition of Tech Community Live will be recorded and available on demand immediately after airing. This event will feature AI-generated captions during the live broadcast. Human-generated captions and a recap of the Q&A will be available by the end of the week. Where do I post my questions? Scroll to the bottom of this page, and select “Comment.”AMA: All things Copilot with Microsoft Intune
How can you optimize IT operations and efficiently manage your digital estate at scale with Copilot? How do I get started with Copilot in Intune? How do you control access? What is the plan for agents—and what agents could we provide in Intune to help you mark more tasks off your checklist with ease and speed? Post your questions and feedback below and Ask Microsoft Anything! This event is part of Tech Community Live: Intune edition. I'm in! How do I sign up? Select “Add to calendar” to save the date, then click the “Attend” button to save your spot, receive event reminders, and participate in the Q&A. If you can’t make the live event, don’t worry. You can post your questions in advance and catch up on the answers and insights later in the week. All sessions for this Intune edition of Tech Community Live will be recorded and available on demand immediately after airing. This event will feature AI-generated captions during the live broadcast. Human-generated captions and a recap of the Q&A will be available by the end of the week. Where do I post my questions? Scroll to the bottom of this page, and select “Comment.”AMA: Security policy and endpoint management with Intune
Curious how to stay agile and ready for growth with a secure Zero Trust approach? From baselines and policies to device actions and compliance, our engineering teams will be ready to answer your questions and help you find the right solutions for your ecosystem. Whether you are looking at those first steps of going cloud-native, or are already cloud-based, this is Ask Microsoft Anything (AMA) so post early, post often to get the advice you need to simplify, optimize, and secure. This event is part of Tech Community Live: Intune edition. I'm in! How do I sign up? Select “Add to calendar” to save the date, then click the “Attend” button to save your spot, receive event reminders, and participate in the Q&A. If you can’t make the live event, don’t worry. You can post your questions in advance and catch up on the answers and insights later in the week. All sessions for this Intune edition of Tech Community Live will be recorded and available on demand immediately after airing. This event will feature AI-generated captions during the live broadcast. Human-generated captions and a recap of the Q&A will be available by the end of the week. Where do I post my questions? Scroll to the bottom of this page, and select “Comment.”AMA: Cross-platform management with Microsoft Intune
With Microsoft Intune, you can protect cloud-connected endpoints across Windows, Android, macOS, iOS, and Linux. Looking for tips to help you reduce costs and complexity by unifying the way you manage your endpoints? Have questions about mobile application management for unenrolled devices? Need to support a variety of device types for frontline workers? We’re here to help. Post your questions early and often. Our engineering teams will be here to help! This event is part of Tech Community Live: Intune edition. I'm in! How do I sign up? Select “Add to calendar” to save the date, then click the “Attend” button to save your spot, receive event reminders, and participate in the Q&A. If you can’t make the live event, don’t worry. You can post your questions in advance and catch up on the answers and insights later in the week. All sessions for this Intune edition of Tech Community Live will be recorded and available on demand immediately after airing. This event will feature AI-generated captions during the live broadcast. Human-generated captions and a recap of the Q&A will be available by the end of the week. Where do I post my questions? Scroll to the bottom of this page, and select “Comment.”Join the Fabric Partner Community for an AMA with Arun Ulag!
🚨 Mark your calendars now! 📅 New this year in the Fabric Partner Community, an AMA (Ask Me Anything) call series with members of the Fabric Leadership Team! 🥳 Arun Ulag, CVP of Azure Data, will kick of this new call series Thursday, September 25, from 8-9 am PT. You will not want to miss this opportunity to ask all your questions, including those related to the announcements made at #FabConEurope, provide your feedback, and more! 👏 To join theses calls, you must be a member of the Fabric Partner Community Teams Channel. Not yet part of the Fabric Partner Community? Join now by submitting the form at https://aka.ms/JoinFabricPartnerCommunity.AMA: Windows 365 - August 2025
Catch up on the latest capabilities for Windows 365 and get answers to your questions! Host Christian Montoya and members of the product team provide answers and insights to help you configure, deploy, and manage Windows in the cloud with ease. On the panel: Christian Montoya, Donna Ryan, Andrej Radinger, & Logan Silliman Windows 365 AMAs offer you an opportunity to participate in live Q&A, on a quarterly basis, with the engineering and product teams building the future of Windows in the cloud. Bookmark our Windows 365 AMA series page for access to previous AMAs on demand and a list of future AMA dates!AMA: Windows 365
Catch up on the latest capabilities for Windows 365 and get answers to your questions! Host Christian Montoya and members of the product team provide answers and insights to help you configure, deploy, and manage Windows in the cloud with ease. How do I participate? Registration is not required. Simply add this event to your calendar and select Attend to receive reminders. Post your questions in advance, or any time during the live broadcast. Windows 365 AMAs offer you an opportunity to participate in live Q&A, on a quarterly basis, with the engineering and product teams building the future of Windows in the cloud. Bookmark our Windows 365 AMA series page for access to previous AMAs on demand and a list of future AMA dates!AMA: Windows 365
Catch up on the latest capabilities for Windows 365 and get answers to your questions! Host Christian Montoya and members of the product team provide answers and insights to help you configure, deploy, and manage Windows in the cloud with ease. How do I participate? Registration is not required. Simply add this event to your calendar and select Attend to receive reminders. Post your questions in advance, or any time during the live broadcast. Windows 365 AMAs offer you an opportunity to participate in live Q&A, on a quarterly basis, with the engineering and product teams building the future of Windows in the cloud. Bookmark our Windows 365 AMA series page for access to previous AMAs on demand and a list of future AMA dates!AMA: Windows 365
Catch up on the latest capabilities for Windows 365 and get answers to your questions! Host Christian Montoya and members of the product team provide answers and insights to help you configure, deploy, and manage Windows in the cloud with ease. How do I participate? Registration is not required. Simply add this event to your calendar and select Attend to receive reminders. Post your questions in advance, or any time during the live broadcast. Windows 365 AMAs offer you an opportunity to participate in live Q&A, on a quarterly basis, with the engineering and product teams building the future of Windows in the cloud. Bookmark our Windows 365 AMA series page for access to previous AMAs on demand and a list of future AMA dates!
