Blog Post

Microsoft Sentinel Blog
7 MIN READ

Automating the deployment of Sysmon for Linux 🐧 and Azure Sentinel in a lab environment 🧪

Cyb3rWard0g's avatar
Cyb3rWard0g
Icon for Microsoft rankMicrosoft
Oct 14, 2021

Thanks to Kevin Sheldrake, Russell McDonald, Jessen Kurien and Ofer Shezaf for making this blog possible.

 

Today, we celebrate 25 years of Sysinternals, a set of utilities to analyze, troubleshoot and optimize Windows systems and applications. Also, as part of this special anniversary, we are releasing Sysmon for Linux, an open-source system monitor tool developed to collect security events from Linux environments using eBPF (Extended Berkeley Packet Filter) and sending them to Syslog for easy consumption. Sysmon for Linux is built on a library also released today named sysinternalsEBPF which is built on libbpf including a library of eBPF inline functions used as helpers. 

 

In this post, I will show you how to automatically deploy a research lab environment with an Azure Sentinel instance and a few Linux virtual machines with Sysmon for Linux already installed and configured to take it for a drive and explore its coverage.

 

As always, before getting into the technical parts of the main topics, it is important to understand some of the fundamental concepts behind Sysmon for Linux.  

 

What is eBPF?

According to the eBPF Foundation, eBPF is a technology that allows programs to run in a sandbox in an operating system kernel. In other words, eBPF enables programmers to write code which gets executed in kernel space in a more secure and restricted way in order to add additional capabilities to the operating system at runtime. 

 

Some of the use cases for eBPF are: 

  • Security: Combining visibility and better level of control to secure systems. 
  • Tracing and profiling: Powerful and unique insights to troubleshoot system performance. 
  • Networking: A natural fit for all packet processing requirements of networking solutions. 
  • Observability and monitoring: Collection and in-kernel aggregation of custom metrics.

 

Why eBPF for Sysmon for Linux? 

From an event-tracing perspective, eBPF allows us to write event-driven programs and have pre-defined hooks into operations such as system calls, network connections, file write/read, etc. We can then collect those events and use them to understand adversary behavior during research or an investigation. As mentioned before, Sysmon for Linux uses its own library “sysinternalsEBPF” to handle the security events monitoring process.

 

 

You can find more information about the implementation of the new sysinternals EBPF library in the following resources: 

 

Installing Sysmon for Linux 

All the information presented here about the installation is available in its own GitHub repository:

 

Register Microsoft Key and Feed 

Sysmon for Linux requires the following packages during installation: 

  • sysinternalsebpf (.DEB or .RPM) 
  • sysmonforlinux (.DEB or .RPM)

For example, for Ubuntu you can run the following (More examples in the INSTALL documents above):

 

 

wget -q https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb

 

 

Install Packages 

Depending on the Linux distribution and package manager, you can use the apt dependency resolver (Debian based distros) or the RPM package manager (Fedora based distros). 

 

Following the Ubuntu example, you can run the following commands to install sysinternalsEBPF and Sysmon in that order

 

 

sudo apt-get update
sudo apt-get install sysinternalsebpf
sudo apt-get install sysmonforlinux

 

 

Next, you should be able to run the sysmon command:

 

 

sysmon –h 

 

 

 

Run Sysmon as a Service 

Finally, we can use the sysmon binary to install and run Sysmon as a service with a specific Sysmon config (like how one installs Sysmon for Windows).

 

 

sudo sysmon -accepteula -i sysmonconfig.xml

 

 

 

Explore Syslog Events  

You can explore Sysmon events from the Syslog log. Later in this post, I will show you how to use other tools to show Sysmon events in a more user-friendly view 😉

 

 

tail –f /var/log/Syslog

 

 

 

The Sysmon for Linux Configuration 

At the time of writing, the Sysmon schema version is 4.81, check here for latest version. An example of a config to collect all events can be found in the following link (Not recommended to use in a production environment due to the large number of events generated):

 

SysmonForLinux-CollectAll-Config.xml (github.com) 

 
Use the following command to update the Sysmon config:

 

 

sudo sysmon –c newconfig.xml

 

 

Configuration Options 

Option 

Description 

Sample 

FieldSizes 

specify how long you want fields to be so you can avoid the Syslog overrun/broken XML problem 

<FieldSizes>CommandLine:50,Image:50</FieldSizes> 

 

Available Events 

Event ID 

Description 

1 

Logs when a new process is created. 

3 

Logs TCP/UDP connections on the machine 

4

Logs the state of the Sysmon service (started or stopped).

5 

Logs when a process terminates. 

9 

Logs when a process conducts reading operations, from the drive. 

11 

Logs when a file is created or overwritten. 

16 

Logs when the local Sysmon configuration is updated. 

23 

Logs when a file is deleted by a process. 

 

All the events in Sysmon for Linux are already documented as data dictionaries in the open-source project OSSEM. You can access that information in the following link:

 

 https://github.com/OTRF/OSSEM-DD/tree/main/linux/sysmon 
 

The MSTIC Sysmon for Linux Configuration 

The MSTIC R&D team is sharing a few configuration files as part of the release of this project and will be maintaining them as we use them for research and development of detections.

 

MSTIC-Sysmon/linux/configs at main · microsoft/MSTIC-Sysmon (github.com)

How do we automate the installation process? 

The installation of Sysmon for Linux can be automated with the following bash script 

 

https://raw.githubusercontent.com/OTRF/Blacksmith/master/resources/scripts/bash/Install-Sysmon-For-Linux.sh  
 

What about a full lab environment? Enter Azure Sentinel To-go!  

Azure Sentinel2Go is an open-source project developed to expedite the deployment of an Azure Sentinel lab along with other Azure resources to expedite research and the development of detections. 
 
https://github.com/OTRF/Azure-Sentinel2Go

 

Azure Sentinel + Sysmon for Linux Environment 

We have updated our previous Linux environment and we can now deploy everything needed for a small research lab with Sysmon for Linux configured and an Azure monitor agent sending logs to Azure Sentinel:

 

Azure-Sentinel2Go/grocery-list/Linux at master · OTRF/Azure-Sentinel2Go (github.com)

 

We were able to use Azure Resource Manager (ARM) templates and a bash script to automate the whole setup. These are all the resources used for each component of the research lab:  

 

Deploying the Lab Environment  

 

 

  • Fill out the following parameters:
    • Subscription (selected by default)
    • Resource group
    • Region (selected by default) 
    • Admin Username
    • Admin Password
    • Remote Access Mode (AllowPublicIP selected by default. You can also use Azure Bastion Host. You would just need to set the Allowed IP Addresses parameter to *)
    • Allowed IP Addresses (If you use the default access mode AllowPublicIP, use your home or office public IP address to only allow access from secure places. 
  • Click the Review > Create buttons to start the deployment 
  • You can go to your resource group and explore all the resources being deployed 

  

 

  • Wait around 5-10 minutes! You should be good to go! 

 

 

  • You can go to the resource group and see all the resources  

 

 

Validate Deployment 

It is very important to validate if everything was deployed properly before exploring events from Sysmon.

 

Sysmon running as a service 

SSH to all your VMs and run the following commands

 

 

systemctl status sysmon

 

 

 

Explore Syslog Events  

You can explore Sysmon events from the Syslog log.

 

 

tail –f /var/log/Syslog

 

 

 

Explore Sysmon Events via sysmonLogView 

Sysmon also comes with a binary named sysmonLogView to explore sysmon events in a friendly format.

 

 

 

Run the following commands to explore Sysmon event id 1 (ProcessCreate) events locally:

 

 

sudo tail -f /var/log/syslog | sudo /opt/sysmon/sysmonLogView -e 1

 

 

 

Azure Sentinel 

Check if you are getting Syslog events via the Azure Sentinel interface: 

 

 

Next, click on logs and run the following Kusto query to see if all your endpoints are generating events and are being collected by the Azure Log Analytics agent:

 

 

Syslog 
| summarize count() by Computer

 

 

 

Querying Sysmon for Linux 

You can query Sysmon for Linux logs by using the Syslog table with the following Kusto query:

 

 

Syslog 
| extend EventID = parse_xml(SyslogMessage).Event.System.EventID 
| extend EventData = parse_xml(SyslogMessage).Event.EventData.Data 
| mv-expand bagexpansion=array EventData 
| evaluate bag_unpack(EventData) 
| extend Key=tostring(['@Name']), Value=['#text'] 
| evaluate pivot( 
    Key, any(Value), TimeGenerated, TenantId, SourceSystem, 
    EventID, Computer, Facility, SeverityLevel, HostIP, MG, Type, _ResourceId 
) 
| summarize count() by tostring(EventID)

 

 

Additionally, as part of the ASIM (Azure Sentinel Information Model) project, we have created parsers for Sysmon for Linux. The parsers get imported automatically by the template we use to deploy the lab environment. Therefore, you can simply use the parsers available under Functions > Workspace functions:

 

 

vimProcessCreateLinuxSysmon
| limit 10 

 

 

 

That’s it! You are now ready to use Sysmon for Linux in a lab environment for research and development of detections 😉  

 

Sysmon for Linux Resources 

 

References 

Updated Nov 09, 2023
Version 10.0
  • This is so awesome!!! I love it! Thanks for sharing!

     

    Happy Azure Stacking!!!

  • ErikOppedijk's avatar
    ErikOppedijk
    Copper Contributor

    How is this related to the Defender for Linux Agent?

     

    Will this be part of the Defender Agent or a completely seperate product?

  • PeterSmit1974's avatar
    PeterSmit1974
    Copper Contributor

    Thank you for sharing, would this also work on an on-premise Linux Box, with an ARC Agent enabled and the use of the new Azure Monitoring Agent (as a extension for Azure Arc enabled servers) ?

     

    I just do not seem to get it working...

     

    kind regards,

     

    Peter

  • Hi, ErikOppedijk , Sysmon for Linux is a separate project that is just focused on logging and is open source.