Hunting Threats on Linux with Azure Sentinel

Introduction

All sorts of activity and security data can be collected by Azure Sentinel for storage and mining.  The Syslog data collector is good for collecting data from Linux platforms but needs a helping hand to access information produced by the Linux kernel’s audit subsystem, kaudit, and the optional user-space  daemon, auditd.  These components can be configured to generate event data when syscalls are invoked, such as process creations, file access, and other telemetry that could be used to identify malicious activity.

 

While it is possible to use the audispd daemon to redirect auditd events to syslog, there are a couple of potential problems with this approach.  The first issue is that while kaudit is a standard component on most Linux distributions, the user-space daemon auditd is not, and audispd relies on auditd to work.  The second issue is that audispd simply forwards the auditd event data without any filtering or processing.

 

Filtering events is essential to reduce the noise generated by known system tools that run regularly; these include cron jobs to rotate logs and system tools that ensure software is kept up to date.  There is usually little need to see this data in your SIEM and filtering it at the source reduces bandwidth and storage requirements.

 

Similarly, event processing is important to enrich the data so that it makes more sense when it is mined.  The Linux audit sub-system uses numerical values for a range of identifiers, and these need to be converted into corresponding names for them to make sense.  It’s possible to do this in the SIEM but it is easier if this happens before the events leave the machine that generated them.

 

The Microsoft audit collection tool, AUOMS, includes configurable filtering and processing steps, collects events from either kaudit or auditd/audispd, and outputs them in a range of formats to specified locations and pipes.  An experimental feature of AUOMS can be used to forward events to the syslog, from where they can be collected by Azure Sentinel.  This blog post will describe how to use this feature of AUOMS and how to configure Azure Sentinel to collect the events.

 

In this blog post, we will cover how to:

• Install the OMS (Operations Management Suite) agent that Azure Sentinel will use to collect the syslog
• Configure the syslog feature of AUOMS
• Configure Azure Sentinel to collect the events
• Build useful functions in Azure Sentinel to aid threat hunting

Azure Sentinel also supports the use of Jupyter Notebooks and Ian Hellen has already written a great blog post Jupyter Notebooks in Sentinel which covers their use.

 

Create an Azure Sentinel Workspace

If you don’t already have an Azure Sentinel workspace, then you’ll need to create one.  The Quickstart guide provides details on the prerequisites and steps to create an Azure Sentinel workspace.

 

Install OMS Agent

The Operations Management Suite agent is used by Azure Sentinel to collect the syslog.  Installing it is straight forward and is covered in this related blog post.

 

Configure the Syslog Feature of AUOMS

Configuring the syslog feature is really simple.  Just download this file and save it, as root, as

/etc/opt/microsoft/auoms/outconf.d/syslog.conf

You can edit this file (as root) to change the filtering and processing functions.  Instruct AUOMS to reconfigure itself by sending it a SIGHUP signal:

sudo killall -HUP auoms

If 'killall' doesn't exist on your system, then you can find the process id with 'ps' and issue the SIGHUP signal with 'kill':

$ ps -ef | grep auoms
root      1344     1  0 Oct20 ?        00:08:13 /opt/microsoft/auoms/bin/auoms
...
$ sudo kill -HUP 1344

To configure AUOMS with a set of useful rules, download this file and save it, as root, as

/etc/opt/microsoft/auoms/rules.d/mstic-research.rules

 

Description of Event Types

The mstic-research.rules file (installed into /etc/opt/microsoft/auoms/rules.d) contains the kaudit/auditd rules that generate event records.  These rules initially report on:

• Process creations using execve and execveat
• Kernel module loading and unloading
• Use of the ptrace debugging interface
• Mapping of memory pages with write and execute permissions (excluding just-in-time compilation languages, such as Python)
• Aggressive killing of processes using signal 9 (kill) and pausing of processes with signal 19 (stop)
• Changes to a selection of critical system files
• Changes to the system clock

You may edit this file to change these rules to suit your environment.  The AUOMS agent will automatically notice the change when the file is written and will load the new rules.  You can check which rules are in use with:

 

 

sudo /opt/microsoft/auoms/bin/auomsctl -l

 

 

Description of Process Filters

The syslog.conf file (installed into /etc/opt/microsoft/auoms/outconf.d) contains the filter descriptions that specify which processes and syscalls are excluded from being reported upon.  The initial configuration filters events from the following:

• The Microsoft OMS software
• The connect, openat, unlink, and fchown syscalls for the dpkg and rpm package managers
• The logrotate cron jobs that rotate the OMS logs
• The connect syscall for the systemd-resolved and nscd domain name resolvers

Note that the provided rules aren’t configured to generate events from the connect syscall and that the filters to block it are provided as examples only.

 

Configure Azure Sentinel To Collect Events

Now that AUOMS is generating audit events and sending them to the syslog, we need to configure Azure Sentinel to collect them.

1. Log into the Azure Portal at portal.azure.com
2. Type sentinel into the search box and select Azure Sentinel
3. Click on your Sentinel workspace, then Settings, then Workspace settings, and then Advanced settings
4. Click on Data, then Syslog
5. Check the Apply below configuration to my machines checkbox
6. Type user into the box and click the + button.  Ensure the Info level checkbox is checked
7. Click Save and then OK

 

View Events, Build Functions, and Hunt Threats

The events will start to be collected and may take fifteen minutes to arrive.  The raw events can be viewed as follows:

1. Navigate to your Sentinel workspace, then click Logs
2. If the example queries dialog is displayed, close it by clicking the X in the top right
3. In the query box, type Syslog and click the Run button

 

 

Syslog events will appear in the results window.  Note that the audit event data is contained in the SyslogMessage column.  This column can be parsed to extract the EventType.  Example parsers have been uploaded to the Azure Sentinel GitHub to demonstrate how to parse process creations (execve), syscalls, and user errors such as logon failures.  These can be saved as functions for use in hunting queries.

 

 

Threat Hunting Example

As an example of threat hunting, you can search for some popular crypto currency miners being downloaded using the CryptoCurrencyMiners.yaml hunting query:

 

Conclusion

Hopefully, this article has helped you understand how easy it is to use Azure Sentinel to obtain audit event data from Linux machines and to hunt for threats within it.

 

We will continue to develop the MSTIC-Research branch of the AUOMS agent, adding additional functionality and improving throughput.  We will also update our mstic-research.rules file with extra event types to report, and our syslog.conf file with extra processes and syscalls to filter out.