Thanks to Kevin Sheldrake, Russell McDonald, Jessen Kurien and Ofer Shezaf for making this blog possible.
Today, we celebrate 25 years of Sysinternals, a set of utilities to analyze, troubleshoot and optimize Windows systems and applications. Also, as part of this special anniversary, we are releasing Sysmon for Linux, an open-source system monitor tool developed to collect security events from Linux environments using eBPF (Extended Berkeley Packet Filter) and sending them to Syslog for easy consumption. Sysmon for Linux is built on a library also released today named sysinternalsEBPF which is built on libbpf including a library of eBPF inline functions used as helpers.
In this post, I will show you how to automatically deploy a research lab environment with an Azure Sentinel instance and a few Linux virtual machines with Sysmon for Linux already installed and configured to take it for a drive and explore its coverage.
As always, before getting into the technical parts of the main topics, it is important to understand some of the fundamental concepts behind Sysmon for Linux.
What is eBPF?
According to the eBPF Foundation, eBPF is a technology that allows programs to run in a sandbox in an operating system kernel. In other words, eBPF enables programmers to write code which gets executed in kernel space in a more secure and restricted way in order to add additional capabilities to the operating system at runtime.
Some of the use cases for eBPF are:
- Security: Combining visibility and better level of control to secure systems.
- Tracing and profiling: Powerful and unique insights to troubleshoot system performance.
- Networking: A natural fit for all packet processing requirements of networking solutions.
- Observability and monitoring: Collection and in-kernel aggregation of custom metrics.
Why eBPF for Sysmon for Linux?
From an event-tracing perspective, eBPF allows us to write event-driven programs and have pre-defined hooks into operations such as system calls, network connections, file write/read, etc. We can then collect those events and use them to understand adversary behavior during research or an investigation. As mentioned before, Sysmon for Linux uses its own library “sysinternalsEBPF” to handle the security events monitoring process.
You can find more information about the implementation of the new sysinternals EBPF library in the following resources:
Installing Sysmon for Linux
All the information presented here about the installation is available in its own GitHub repository:
- SysinternalsEBPF/INSTALL.md at main · Sysinternals/SysinternalsEBPF (github.com)
- SysmonForLinux/INSTALL.md at main · Sysinternals/SysmonForLinux (github.com)
Register Microsoft Key and Feed
Sysmon for Linux requires the following packages during installation:
- sysinternalsebpf (.DEB or .RPM)
- sysmonforlinux (.DEB or .RPM)
For example, for Ubuntu you can run the following (More examples in the INSTALL documents above):
wget -q https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
Install Packages
Depending on the Linux distribution and package manager, you can use the apt dependency resolver (Debian based distros) or the RPM package manager (Fedora based distros).
Following the Ubuntu example, you can run the following commands to install sysinternalsEBPF and Sysmon in that order
sudo apt-get update
sudo apt-get install sysinternalsebpf
sudo apt-get install sysmonforlinux
Next, you should be able to run the sysmon command:
sysmon –h
Run Sysmon as a Service
Finally, we can use the sysmon binary to install and run Sysmon as a service with a specific Sysmon config (like how one installs Sysmon for Windows).
sudo sysmon -accepteula -i sysmonconfig.xml
Explore Syslog Events
You can explore Sysmon events from the Syslog log. Later in this post, I will show you how to use other tools to show Sysmon events in a more user-friendly view 😉
tail –f /var/log/Syslog
The Sysmon for Linux Configuration
At the time of writing, the Sysmon schema version is 4.81, check here for latest version. An example of a config to collect all events can be found in the following link (Not recommended to use in a production environment due to the large number of events generated):
SysmonForLinux-CollectAll-Config.xml (github.com)
Use the following command to update the Sysmon config:
sudo sysmon –c newconfig.xml
Configuration Options
|
Option |
Description |
Sample |
|
FieldSizes |
specify how long you want fields to be so you can avoid the Syslog overrun/broken XML problem |
<FieldSizes>CommandLine:50,Image:50</FieldSizes> |
Available Events
|
Event ID |
Description |
|
1 |
Logs when a new process is created. |
|
3 |
Logs TCP/UDP connections on the machine |
|
4 |
Logs the state of the Sysmon service (started or stopped). |
|
5 |
Logs when a process terminates. |
|
9 |
Logs when a process conducts reading operations, from the drive. |
|
11 |
Logs when a file is created or overwritten. |
|
16 |
Logs when the local Sysmon configuration is updated. |
|
23 |
Logs when a file is deleted by a process. |
All the events in Sysmon for Linux are already documented as data dictionaries in the open-source project OSSEM. You can access that information in the following link:
https://github.com/OTRF/OSSEM-DD/tree/main/linux/sysmon
The MSTIC Sysmon for Linux Configuration
The MSTIC R&D team is sharing a few configuration files as part of the release of this project and will be maintaining them as we use them for research and development of detections.
MSTIC-Sysmon/linux/configs at main · microsoft/MSTIC-Sysmon (github.com)
How do we automate the installation process?
The installation of Sysmon for Linux can be automated with the following bash script
What about a full lab environment? Enter Azure Sentinel To-go!
Azure Sentinel2Go is an open-source project developed to expedite the deployment of an Azure Sentinel lab along with other Azure resources to expedite research and the development of detections.
https://github.com/OTRF/Azure-Sentinel2Go
Azure Sentinel + Sysmon for Linux Environment
We have updated our previous Linux environment and we can now deploy everything needed for a small research lab with Sysmon for Linux configured and an Azure monitor agent sending logs to Azure Sentinel:
Azure-Sentinel2Go/grocery-list/Linux at master · OTRF/Azure-Sentinel2Go (github.com)
We were able to use Azure Resource Manager (ARM) templates and a bash script to automate the whole setup. These are all the resources used for each component of the research lab:
- Azure Sentinel
- Linux Virtual Machines
- Linux virtual machines
- Ubuntu 18.04.6 LTS (Kernel release: 5.4.0-1059-azure)
- Centos 8.2.2004 (Kernel release: 4.18.0-193.28.1.el8_2.x86_64
- Red Hat 8.2 (Kernel release: 4.18.0-193.65.2.el8_2.x86_64)
- Azure Log Analytics Agent for Linux
- Sysmon for Linux Installer
- Linux virtual machines
Deploying the Lab Environment
- Go to: https://github.com/OTRF/Azure-Sentinel2Go/tree/master/grocery-list/Linux/demos/Sysmon-For-Linux
- Click on the “Deploy to Azure” Button
- Fill out the following parameters:
- Subscription (selected by default)
- Resource group
- Region (selected by default)
- Admin Username
- Admin Password
- Remote Access Mode (AllowPublicIP selected by default. You can also use Azure Bastion Host. You would just need to set the Allowed IP Addresses parameter to *)
- Allowed IP Addresses (If you use the default access mode AllowPublicIP, use your home or office public IP address to only allow access from secure places.
- Click the Review > Create buttons to start the deployment
- You can go to your resource group and explore all the resources being deployed
- Wait around 5-10 minutes! You should be good to go!
- You can go to the resource group and see all the resources
Validate Deployment
It is very important to validate if everything was deployed properly before exploring events from Sysmon.
Sysmon running as a service
SSH to all your VMs and run the following commands
systemctl status sysmon
Explore Syslog Events
You can explore Sysmon events from the Syslog log.
tail –f /var/log/Syslog
Explore Sysmon Events via sysmonLogView
Sysmon also comes with a binary named sysmonLogView to explore sysmon events in a friendly format.
Run the following commands to explore Sysmon event id 1 (ProcessCreate) events locally:
sudo tail -f /var/log/syslog | sudo /opt/sysmon/sysmonLogView -e 1
Azure Sentinel
Check if you are getting Syslog events via the Azure Sentinel interface:
- https://portal.azure.com/
- Search > Azure Sentinel
Next, click on logs and run the following Kusto query to see if all your endpoints are generating events and are being collected by the Azure Log Analytics agent:
Syslog
| summarize count() by Computer
Querying Sysmon for Linux
You can query Sysmon for Linux logs by using the Syslog table with the following Kusto query:
Syslog
| extend EventID = parse_xml(SyslogMessage).Event.System.EventID
| extend EventData = parse_xml(SyslogMessage).Event.EventData.Data
| mv-expand bagexpansion=array EventData
| evaluate bag_unpack(EventData)
| extend Key=tostring(['@Name']), Value=['#text']
| evaluate pivot(
Key, any(Value), TimeGenerated, TenantId, SourceSystem,
EventID, Computer, Facility, SeverityLevel, HostIP, MG, Type, _ResourceId
)
| summarize count() by tostring(EventID)
Additionally, as part of the ASIM (Azure Sentinel Information Model) project, we have created parsers for Sysmon for Linux. The parsers get imported automatically by the template we use to deploy the lab environment. Therefore, you can simply use the parsers available under Functions > Workspace functions:
vimProcessCreateLinuxSysmon
| limit 10
That’s it! You are now ready to use Sysmon for Linux in a lab environment for research and development of detections 😉
Sysmon for Linux Resources
- Azure-Sentinel/Parsers/ASim Sysmon for Linux at master · Azure/Azure-Sentinel (github.com)
- Azure-Sentinel2Go/grocery-list/Linux/demos/Sysmon-For-Linux at master · OTRF/Azure-Sentinel2Go (gith...
- Sysinternals/SysinternalsEBPF: The Linux port of the Sysinternals Sysmon tool. (github.com)
- Sysinternals/SysmonForLinux (github.com)
- MSTIC-Sysmon/linux/configs at main · microsoft/MSTIC-Sysmon (github.com)
References
- eBPF - Introduction, Tutorials & Community Resources
- What is eBPF? An Introduction and Deep Dive into the eBPF Technology
- OTRF/Azure-Sentinel2Go: Azure Sentinel2Go is an open source project developed to expedite the deploy...
- Azure Sentinel To-Go! A Linux :penguin: Lab with AUOMS Set Up to Learn About the OMI Vulnerability �...