Azure Sentinel Agent: Collecting from servers and workstations, on-prem and in the cloud

[Ofer_Shezaf]

This is part of a series of blogs on connectors. You might find what you are looking for also here:

• Syslog, CEF, Logstash and other 3rd party connectors grand list
• Collecting logs from Microsoft Services and Applications
• Creating Custom Connectors

My previous blog posts discussed collecting events from Azure PaaS resources and networking and security sources. But what about collecting from servers? Whether deployed in the cloud, on-prem VMs or even physical machines, those are probably still the biggest attack surface and therefore the most common sources of events.

To collect events from servers wherever those are deployed, use the Azure Log Analytics agent (also called "MMA" for Microsoft Monitoring Agent). The agent supports collecting from Windows machines as well as Linux. The agent can be installed manually or provisioned in Azure using Microsoft VM extensions for Windows or Linux.

Azure Sentinel connectors which utilize the agent 

The agent supports the following Sentinel connectors:

• Microsoft DNS servers
• Windows Firewall
  • The Windows firewall writes logs to files which are collected and sent  by the agent when files are rotated. This leads to additional collections latency, which can be controlled by changing the log file size as described here.
• Windows Security Events

Once you enabled them through the Sentinel's Data Connectors, they will be collected by every agent configured to send data to the workspace.

Additional data streams collected by the Agent

However, the agent is not limited to this telemetry, and Azure Sentinel can collect the following additional data streams using the agent:

• When installed on a domain controller, the agent collects AD events.
• You can configure the agents to send any Windows event type, not just security events, such as Sysmon. Some examples:
  • Using Sysmon in Azure Sentinel 
  • Adding MBAM/Bitlocker Logs to Azure Sentinel 
• IIS logs
• Wire Data: sFlow-like data collected by the agent (being replaced by VM Insights below)
• VM Insights: network connections, open ports, processes, and general computer information
  • Schema
  • Sample queries
• Files: Events stored in files on the server. See Custom logs in Azure Monitor.

AKS and Containiners

To collect control and data plane telemtry from containers, including AKS, see Azure Monitor for containers and how to enable it.

Additional on-premises Microsoft telemetry

For completeness, n addition you can collect on-premises telemetry not using the agent for the following sources:

• Windows Defender
• Intune
• Microsoft SQL: Logs to the Windows Event

Agent deployment and management

• Maintenance:
  • Maintain the Log Analytics agent
  • Monitor agents health using the Azure Monitor Agent Health solution
  • Troubleshooting
• Deployment:
  • Scale the agent using a VM scale set
  • You can send telemetry from an agent to multiple destination workspaces.
  • No direct internet access for the agent? Use the Log Analytics gateway Need to scale the gateway? Use Azure load balancer.
• Configuration
  • You can select the data collection tier to control how many Windows Security events are collected.
  • For caching, see below.

Agent general data

• The agent compresses data when sending it to the cloud to reduce the network load.
• Read about agent collection latency.

Agent caching

The agent caches data, which helps prevent data loss in case of communication issues between the agent and the cloud. The following described this mechanism and how it can be controlled.

Linux

Output is controlled by modifying the agent configuration files. Specifically, the output configuration file, /etc/opt/microsoft/omsagent/<workspace id>/conf/omsagent.conf . Modify the section below to control the Agent's caching behavior:

<match oms.blob.**>
  type out_oms_blob
  log_level info
  num_threads 5
  buffer_chunk_limit 10m
  buffer_type file
  buffer_path /var/opt/microsoft/omsagent/state/out_oms_blob*.buffer
  buffer_queue_limit 10
  buffer_queue_full_action drop_oldest_chunk
  flush_interval 20s
  retry_limit 10
  retry_wait 30s
  max_retry_wait 9m
</match>

Note that for custom logs, the section would be different (for example <match oms.api.**>). The buffer parameters are documented here.

Windows

To change the cache size, modify this registry entry

Key: HKLM\SYSTEM\CurrentControlSet\Services\HealthService\Parameters\Management Groups\<Management Group Name>
Value: MaximumQueueSizeKb

Default: 15360
Min Value: 5120
Max Value: 1536000