Revolutionizing log collection with Azure Monitor Agent

With the deprecation of Log analytics agent (also called MMA or OMS), it’s a great opportunity to discuss its successor – the Azure Monitor Agent or in short - (AMA), and why it is so much better and keeps improving! 

AMA is a lightweight log collection agent, designed to consume as little resources as possible when collecting metrics and logs from your server. It can be installed on various flavors and OS versions of both Linux as well as Windows machines hosted in Azure, on-premises or any other cloud environments. When installed on non-Azure machines, AMA requires the installation of Azure Arc agentry to provide mirroring and centralized cloud management capabilities to your machine. 

Associated with a Microsoft Sentinel workspace, all logs collected form AMA-installed machines, are sent to the various Microsoft Sentinel tables, depending on the source type from which they were collected (Windows DNS, Windows security events, Firewall, IIS, Syslog, CEF, etc.). 

AMA can be controlled using Data Collection Rules (DCR), enabling you to define where to collect the logs from, what data manipulations to perform with KQL transformations (enabling you filtering, parsing, enrichment and more) and where to send the logs to, whether that be a workspace, Eventhubs (for Azure VMS only), Auxiliary tier and so on.  You can group machines by using different DCRs. 

DCRs for AMA can be created in multiple ways: 

• Through the Portals (Azure or Unified Security Operations Platform): This method provides a user-friendly interface for defining data collection scenarios:  

• Configuring an AMA-based connector in Microsoft Sentinel > Configuration > Data collection (in either Azure or the security portals), will create the DCR for you. Through this option data will be directed to Microsoft Sentinel tables, some of which are not accessible when defining the DCR in Azure Monitor portal (e.g CommonSecurityLog, SecurityEvent, WindowsEvent and ASIM tables). 

• Creating and editing the DCRs through Azure Monitor by browsing to Azure portal > Azure Monitor > Settings > Data Collection Rules. Note: using this UI creation of DCR does not enable access to Microsoft Sentinel tables are not accessible and require editing of the DCR’s outputStream to divert the data to Microsoft Sentinel’s tables). 

• Azure Resource Manager (ARM) Templates: ARM templates allow you to define DCRs as code, enabling you to deploy and manage them programmatically. This is useful for automation and consistency across environments. 

• Azure CLI and PowerShell: These command-line tools provide another way to create and manage DCRs programmatically. They are particularly useful for scripting and automation. 

• REST API: The Azure Monitor REST API allows you to create, update, and manage DCRs programmatically. This method offers the most flexibility and integration with other systems. 

• PowerShell: Use Azure cmdlet to create, edit and deploy DCRs.  

So why do we like AMA better? 

Ah! Easy: 

• AMA is more performant than the legacy agent, reaching 25% increase in performance when compared to the Linux OMS, and 500% better in EPS than the MMA for Windows!  

• It’s centrally configured through the cloud, by the DCRs, enabling grouping of machines. Using Azure policies enables scale deployments, upgrades, and configuration changes over time. Any change in the DCR configuration is automatically rolled out to all agents without the need for further action at the client-side. 

• AMA supports multi-homing, sending events to multiple workspaces, cross region and cross-tenant (with Azure Lighthouse).  

• Most importantly, the AMA is more secured connecting to the cloud using Managed Identity and Microsoft Entra tokens. With non-Azure controlled machines, Arc enhances the security of the connection by enforcing and handling authentication and identity tokens  

The greatest thing is that AMA keeps evolving through multiple enhancements and improvements we’re constantly working on! 

Next, we’ll cover a few noticeable changes to connectors. 

Windows Security Events: 
We’ve enhanced the schema of the SecurityEvent table, hosting Windows Security Event, and have added new columns that AMA version 1.28.2 and up will be populating. These enhancements are designated to provide better coverage and visibility of the events collected.

New columns added are: 

• Keywords: A 64 bitmask of keywords defined in the event. Keywords classify types of events (e.g events associated with reading data), with the left most 2 bits representing audit success or failure. 

• Opcode: Operation code identifying the location in the application from where the event was logged, together with Task. 

• EventRecordId: The record number assigned to the event when it was logged 

• SystemThreadId: The thread that generated the event 

• SystemProcessId: The process that generated the event 

• Correlation: Activity identifiers that consumers can use to group related events together 

• Version: The version number of the event’s definition 

• SystemUserId: The ID of the user responsible for the event 

Common Event Format (CEF) and Syslog:  
 We all know how important it is to collect and analyze data from various sources, such as firewalls, routers, switches, servers, DNS and applications. Two of the most common protocols used by many devices to emit their logs are CEF and Syslog.  

With the legacy agent you had to configure a connector for each source separately, which could be tedious and time-consuming. That's why we are excited to announce the updates to the Syslog and CEF data connectors via AMA, which will improve your overall experience with Microsoft Sentinel data connectors. All devices will now depend on either the generic CEF or the generic Syslog connectors, based on the log source used protocol. The relevant generic connector will be deployed as part of the device solution (don't forget to check the box to select it for installation after you click the 'install with dependencies' button!). 

To monitor the ingestion of your logs from the separated device types with the graphs, we’ve added a dedicated workbook, installed with the solution, where device types are aggregated in a single location. You can further filter the view based on device type or connectivity status. 

To help you set the source device to streamline the logs, we’ve included the instructions or relevant referrals for many common CEF appliances or Syslog in our documentation. 

Windows Events: 
What happens if you wish to collect other Windows audit events? You cannot send them to the SecurityEvents table as those events are not from the security channel and do not match that table schema. Instead, the non-security events can be directed to the WindowsEvents table using the Windows Forwarded Events data connector, which can be used to stream both forwarded events collected from a WEC/WEF server, as well as those Windows server, by setting the DCR wizard to Custom option and specifying the XPath expression to point to the desired events. 

Windows Firewall Logs: 
This connector enables the collection of the machine’s Firewall logs. We’ve added a granularity selection of the profile from which to collect and stream logs to the ASimNetworkSessionLogs table.  

Custom Logs: 
Some appliances packaged in Content Hub solutions are streaming data to _CL tables. For those 15 specific devices and to enable a quick setting up of file collection, we’ve added the Custom logs connector. 

 
We hope this post was informative and that you have already upgraded your agents to AMA, or plan to do so shortly. For more information on other connectors agent-based or others, refer to our Data connectors documentation or browse the content hub to locate your source of interest. If you would like more content about using AMA, please let us know in the comments below! 

Lastly, to stay current with the latest updates and announcements stay tuned to our What’s new page.