Microsoft Sentinel Blog

7 MIN READ

Automating the deployment of Sysmon for Linux 🐧 and Azure Sentinel in a lab environment 🧪

Microsoft

Oct 14, 2021

Thanks to Kevin Sheldrake, Russell McDonald, Jessen Kurien and Ofer Shezaf for making this blog possible.

Today, we celebrate 25 years of Sysinternals, a set of utilities to analyze, troubleshoot and optimize Windows systems and applications. Also, as part of this special anniversary, we are releasing Sysmon for Linux, an open-source system monitor tool developed to collect security events from Linux environments using eBPF (Extended Berkeley Packet Filter) and sending them to Syslog for easy consumption. Sysmon for Linux is built on a library also released today named sysinternalsEBPF which is built on libbpf including a library of eBPF inline functions used as helpers.

In this post, I will show you how to automatically deploy a research lab environment with an Azure Sentinel instance and a few Linux virtual machines with Sysmon for Linux already installed and configured to take it for a drive and explore its coverage.

As always, before getting into the technical parts of the main topics, it is important to understand some of the fundamental concepts behind Sysmon for Linux.

What is eBPF?

According to the eBPF Foundation, eBPF is a technology that allows programs to run in a sandbox in an operating system kernel. In other words, eBPF enables programmers to write code which gets executed in kernel space in a more secure and restricted way in order to add additional capabilities to the operating system at runtime.

Some of the use cases for eBPF are:

Security: Combining visibility and better level of control to secure systems.
Tracing and profiling: Powerful and unique insights to troubleshoot system performance.
Networking: A natural fit for all packet processing requirements of networking solutions.
Observability and monitoring: Collection and in-kernel aggregation of custom metrics.

Why eBPF for Sysmon for Linux?

From an event-tracing perspective, eBPF allows us to write event-driven programs and have pre-defined hooks into operations such as system calls, network connections, file write/read, etc. We can then collect those events and use them to understand adversary behavior during research or an investigation. As mentioned before, Sysmon for Linux uses its own library “sysinternalsEBPF” to handle the security events monitoring process.

You can find more information about the implementation of the new sysinternals EBPF library in the following resources:

EBPF Summit: Auto-discovery of Kernel Struct Offsets without BTF – Kevin Sheldrake, Microsoft

Installing Sysmon for Linux

All the information presented here about the installation is available in its own GitHub repository:

Register Microsoft Key and Feed

Sysmon for Linux requires the following packages during installation:

sysinternalsebpf (.DEB or .RPM)
sysmonforlinux (.DEB or .RPM)

For example, for Ubuntu you can run the following (More examples in the INSTALL documents above):

wget -q https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb

Install Packages

Depending on the Linux distribution and package manager, you can use the apt dependency resolver (Debian based distros) or the RPM package manager (Fedora based distros).

Following the Ubuntu example, you can run the following commands to install sysinternalsEBPF and Sysmon in that order

sudo apt-get update
sudo apt-get install sysinternalsebpf
sudo apt-get install sysmonforlinux

Next, you should be able to run the sysmon command:

sysmon –h

Run Sysmon as a Service

Finally, we can use the sysmon binary to install and run Sysmon as a service with a specific Sysmon config (like how one installs Sysmon for Windows).

sudo sysmon -accepteula -i sysmonconfig.xml

Explore Syslog Events

You can explore Sysmon events from the Syslog log. Later in this post, I will show you how to use other tools to show Sysmon events in a more user-friendly view 😉

tail –f /var/log/Syslog

The Sysmon for Linux Configuration

At the time of writing, the Sysmon schema version is 4.81, check here for latest version. An example of a config to collect all events can be found in the following link (Not recommended to use in a production environment due to the large number of events generated):

SysmonForLinux-CollectAll-Config.xml (github.com)

Use the following command to update the Sysmon config:

sudo sysmon –c newconfig.xml

Configuration Options

Option	Description	Sample
FieldSizes	specify how long you want fields to be so you can avoid the Syslog overrun/broken XML problem	<FieldSizes>CommandLine:50,Image:50</FieldSizes>

Available Events

Event ID	Description
1	Logs when a new process is created.
3	Logs TCP/UDP connections on the machine
4	Logs the state of the Sysmon service (started or stopped).
5	Logs when a process terminates.
9	Logs when a process conducts reading operations, from the drive.
11	Logs when a file is created or overwritten.
16	Logs when the local Sysmon configuration is updated.
23	Logs when a file is deleted by a process.

All the events in Sysmon for Linux are already documented as data dictionaries in the open-source project OSSEM. You can access that information in the following link:

https://github.com/OTRF/OSSEM-DD/tree/main/linux/sysmon

The MSTIC Sysmon for Linux Configuration

The MSTIC R&D team is sharing a few configuration files as part of the release of this project and will be maintaining them as we use them for research and development of detections.

MSTIC-Sysmon/linux/configs at main · microsoft/MSTIC-Sysmon (github.com)

How do we automate the installation process?

The installation of Sysmon for Linux can be automated with the following bash script

https://raw.githubusercontent.com/OTRF/Blacksmith/master/resources/scripts/bash/Install-Sysmon-For-Linux.sh

What about a full lab environment? Enter Azure Sentinel To-go!

Azure Sentinel2Go is an open-source project developed to expedite the deployment of an Azure Sentinel lab along with other Azure resources to expedite research and the development of detections.

https://github.com/OTRF/Azure-Sentinel2Go

Azure Sentinel + Sysmon for Linux Environment

We have updated our previous Linux environment and we can now deploy everything needed for a small research lab with Sysmon for Linux configured and an Azure monitor agent sending logs to Azure Sentinel:

Azure-Sentinel2Go/grocery-list/Linux at master · OTRF/Azure-Sentinel2Go (github.com)

We were able to use Azure Resource Manager (ARM) templates and a bash script to automate the whole setup. These are all the resources used for each component of the research lab:

Azure Sentinel

Linux Virtual Machines
- Linux virtual machines
  - Ubuntu 18.04.6 LTS (Kernel release: 5.4.0-1059-azure)
  - Centos 8.2.2004 (Kernel release: 4.18.0-193.28.1.el8_2.x86_64
  - Red Hat 8.2 (Kernel release: 4.18.0-193.65.2.el8_2.x86_64)
- Azure Log Analytics Agent for Linux
- Sysmon for Linux Installer

Deploying the Lab Environment

Go to: https://github.com/OTRF/Azure-Sentinel2Go/tree/master/grocery-list/Linux/demos/Sysmon-For-Linux
Click on the “Deploy to Azure” Button

Fill out the following parameters:
- Subscription (selected by default)
- Resource group
- Region (selected by default)
- Admin Username
- Admin Password
- Remote Access Mode (AllowPublicIP selected by default. You can also use Azure Bastion Host. You would just need to set the Allowed IP Addresses parameter to *)
- Allowed IP Addresses (If you use the default access mode AllowPublicIP, use your home or office public IP address to only allow access from secure places.
Click the Review > Create buttons to start the deployment
You can go to your resource group and explore all the resources being deployed

Wait around 5-10 minutes! You should be good to go!

You can go to the resource group and see all the resources

Validate Deployment

It is very important to validate if everything was deployed properly before exploring events from Sysmon.

Sysmon running as a service

SSH to all your VMs and run the following commands

systemctl status sysmon

Explore Syslog Events

You can explore Sysmon events from the Syslog log.

tail –f /var/log/Syslog

Explore Sysmon Events via sysmonLogView

Sysmon also comes with a binary named sysmonLogView to explore sysmon events in a friendly format.

Run the following commands to explore Sysmon event id 1 (ProcessCreate) events locally:

sudo tail -f /var/log/syslog | sudo /opt/sysmon/sysmonLogView -e 1

Azure Sentinel

Check if you are getting Syslog events via the Azure Sentinel interface:

https://portal.azure.com/
Search > Azure Sentinel

Next, click on logs and run the following Kusto query to see if all your endpoints are generating events and are being collected by the Azure Log Analytics agent:

Syslog 
| summarize count() by Computer

Querying Sysmon for Linux

You can query Sysmon for Linux logs by using the Syslog table with the following Kusto query:

Syslog 
| extend EventID = parse_xml(SyslogMessage).Event.System.EventID 
| extend EventData = parse_xml(SyslogMessage).Event.EventData.Data 
| mv-expand bagexpansion=array EventData 
| evaluate bag_unpack(EventData) 
| extend Key=tostring(['@Name']), Value=['#text'] 
| evaluate pivot( 
    Key, any(Value), TimeGenerated, TenantId, SourceSystem, 
    EventID, Computer, Facility, SeverityLevel, HostIP, MG, Type, _ResourceId 
) 
| summarize count() by tostring(EventID)

Additionally, as part of the ASIM (Azure Sentinel Information Model) project, we have created parsers for Sysmon for Linux. The parsers get imported automatically by the template we use to deploy the lab environment. Therefore, you can simply use the parsers available under Functions > Workspace functions: