Thanks to Kevin Sheldrake, Russell McDonald, Jessen Kurien and Ofer Shezaf for making this blog possible.
Today, we celebrate 25 years of Sysinternals, a set of utilities to analyze, troubleshoot and optimize Windows systems and applications. Also, as part of this special anniversary, we are releasing Sysmon for Linux, an open-source system monitor tool developed to collect security events from Linux environments using eBPF (Extended Berkeley Packet Filter) and sending them to Syslog for easy consumption. Sysmon for Linux is built on a library also released today named sysinternalsEBPF which is built on libbpf including a library of eBPF inline functions used as helpers.
In this post, I will show you how to automatically deploy a research lab environment with an Azure Sentinel instance and a few Linux virtual machines with Sysmon for Linux already installed and configured to take it for a drive and explore its coverage.
As always, before getting into the technical parts of the main topics, it is important to understand some of the fundamental concepts behind Sysmon for Linux.
According to the eBPF Foundation, eBPF is a technology that allows programs to run in a sandbox in an operating system kernel. In other words, eBPF enables programmers to write code which gets executed in kernel space in a more secure and restricted way in order to add additional capabilities to the operating system at runtime.
Some of the use cases for eBPF are:
From an event-tracing perspective, eBPF allows us to write event-driven programs and have pre-defined hooks into operations such as system calls, network connections, file write/read, etc. We can then collect those events and use them to understand adversary behavior during research or an investigation. As mentioned before, Sysmon for Linux uses its own library “sysinternalsEBPF” to handle the security events monitoring process.
You can find more information about the implementation of the new sysinternals EBPF library in the following resources:
All the information presented here about the installation is available in its own GitHub repository:
Sysmon for Linux requires the following packages during installation:
For example, for Ubuntu you can run the following (More examples in the INSTALL documents above):
wget -q https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
Depending on the Linux distribution and package manager, you can use the apt dependency resolver (Debian based distros) or the RPM package manager (Fedora based distros).
Following the Ubuntu example, you can run the following commands to install sysinternalsEBPF and Sysmon in that order
sudo apt-get update
sudo apt-get install sysinternalsebpf
sudo apt-get install sysmonforlinux
Next, you should be able to run the sysmon command:
sysmon –h
Finally, we can use the sysmon binary to install and run Sysmon as a service with a specific Sysmon config (like how one installs Sysmon for Windows).
sudo sysmon -accepteula -i sysmonconfig.xml
You can explore Sysmon events from the Syslog log. Later in this post, I will show you how to use other tools to show Sysmon events in a more user-friendly view 😉
tail –f /var/log/Syslog
At the time of writing, the Sysmon schema version is 4.81, check here for latest version. An example of a config to collect all events can be found in the following link (Not recommended to use in a production environment due to the large number of events generated):
SysmonForLinux-CollectAll-Config.xml (github.com)
Use the following command to update the Sysmon config:
sudo sysmon –c newconfig.xml
Option |
Description |
Sample |
FieldSizes |
specify how long you want fields to be so you can avoid the Syslog overrun/broken XML problem |
<FieldSizes>CommandLine:50,Image:50</FieldSizes> |
Event ID |
Description |
1 |
Logs when a new process is created. |
3 |
Logs TCP/UDP connections on the machine |
4 |
Logs the state of the Sysmon service (started or stopped). |
5 |
Logs when a process terminates. |
9 |
Logs when a process conducts reading operations, from the drive. |
11 |
Logs when a file is created or overwritten. |
16 |
Logs when the local Sysmon configuration is updated. |
23 |
Logs when a file is deleted by a process. |
All the events in Sysmon for Linux are already documented as data dictionaries in the open-source project OSSEM. You can access that information in the following link:
https://github.com/OTRF/OSSEM-DD/tree/main/linux/sysmon
The MSTIC R&D team is sharing a few configuration files as part of the release of this project and will be maintaining them as we use them for research and development of detections.
MSTIC-Sysmon/linux/configs at main · microsoft/MSTIC-Sysmon (github.com)
The installation of Sysmon for Linux can be automated with the following bash script
Azure Sentinel2Go is an open-source project developed to expedite the deployment of an Azure Sentinel lab along with other Azure resources to expedite research and the development of detections.
https://github.com/OTRF/Azure-Sentinel2Go
We have updated our previous Linux environment and we can now deploy everything needed for a small research lab with Sysmon for Linux configured and an Azure monitor agent sending logs to Azure Sentinel:
Azure-Sentinel2Go/grocery-list/Linux at master · OTRF/Azure-Sentinel2Go (github.com)
We were able to use Azure Resource Manager (ARM) templates and a bash script to automate the whole setup. These are all the resources used for each component of the research lab:
It is very important to validate if everything was deployed properly before exploring events from Sysmon.
SSH to all your VMs and run the following commands
systemctl status sysmon
You can explore Sysmon events from the Syslog log.
tail –f /var/log/Syslog
Sysmon also comes with a binary named sysmonLogView to explore sysmon events in a friendly format.
Run the following commands to explore Sysmon event id 1 (ProcessCreate) events locally:
sudo tail -f /var/log/syslog | sudo /opt/sysmon/sysmonLogView -e 1
Check if you are getting Syslog events via the Azure Sentinel interface:
Next, click on logs and run the following Kusto query to see if all your endpoints are generating events and are being collected by the Azure Log Analytics agent:
Syslog
| summarize count() by Computer
You can query Sysmon for Linux logs by using the Syslog table with the following Kusto query:
Syslog
| extend EventID = parse_xml(SyslogMessage).Event.System.EventID
| extend EventData = parse_xml(SyslogMessage).Event.EventData.Data
| mv-expand bagexpansion=array EventData
| evaluate bag_unpack(EventData)
| extend Key=tostring(['@Name']), Value=['#text']
| evaluate pivot(
Key, any(Value), TimeGenerated, TenantId, SourceSystem,
EventID, Computer, Facility, SeverityLevel, HostIP, MG, Type, _ResourceId
)
| summarize count() by tostring(EventID)
Additionally, as part of the ASIM (Azure Sentinel Information Model) project, we have created parsers for Sysmon for Linux. The parsers get imported automatically by the template we use to deploy the lab environment. Therefore, you can simply use the parsers available under Functions > Workspace functions:
vimProcessCreateLinuxSysmon
| limit 10
That’s it! You are now ready to use Sysmon for Linux in a lab environment for research and development of detections 😉
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.