Last week, on September 14th, 2021, Microsoft released fixes for three Elevation of Privilege (EoP) vulnerabilities CVE-2021-38645, CVE-2021-38649, CVE-2021-38648, and one unauthenticated Remote Code Execution (RCE) vulnerability CVE-2021-38647 .
These vulnerabilities affect the Open Management Infrastructure (OMI), an open-source project to further the development of a production quality implementation of the DMTF CIM/WBEM standards. The OMI Common Information Model Object Manager (CIMOM) is also designed to be portable and highly modular. It is written in C and the code is available in GitHub.
Great Resources to Read First
The following resources have already been shared by Microsoft to provide guidance on updating vulnerable extensions for Cloud and On-Premises deployments, and indicators to detect the exploitation of the vulnerability:
- MSRC: Additional Guidance Regarding OMI Vulnerabilities within Azure VM Management Extensions
- MSTIC: Hunting for OMI Vulnerability Exploitation with Azure Sentinel
- Azure Security Center: Using ASC to find machines affected by OMI vulnerabilities in Azure VM Manage...
What is this about?
In this post, I will show you how to automatically deploy a research lab environment with Azure Sentinel , a few Linux virtual machines and the Microsoft Audit Collection Tool (AUOMS) set up to understand the underlying behavior of the exploitation of the OMI vulnerability.
This is an extension of the amazing work shared by MSTIC through the following resources:
- Kevin Sheldrake - Hunting Threats on Linux with Azure Sentinel
- Jannie Li - Guided Hunting Notebook: Base64-Encoded Linux Commands
Before going through a few concepts and the deployment process, remember that this vulnerability is actively being exploited. Therefore, make sure you do not expose your lab environment to the Internet.
Microsoft Audit Collection Tool (AUOMS)
AUOMS is a Microsoft audit collection tool that can collect events generated by the Linux kernelâs audit subsystem, kaudit, and the optional user-space daemon, auditd. This allows, for example, the collection of syscalls events such as process creations, file access, and other valuable telemetry for research.
AUOMS is part of the installation of the Log Analytics Agent for Linux, also known as the Operations Management Suite (OMS) Agent for Linux, which allows the streaming of events from Linux-based, syslog supporting devices into Azure Sentinel. However, AUOMS is not set up by default as shown below:
My colleague Kevin Sheldrake documented everything that is required to set it up in this blog post Hunting Threats on Linux with Azure Sentinel.
The question is âHow do we automate the whole setup?â
Enter Azure Sentinel To-go!
Azure Sentinel2Go is an open-source project developed to expedite the deployment of an Azure Sentinel lab along with other Azure resources and a data ingestion pipeline to consume pre-recorded datasets for research purposes.
Azure Sentinel + Linux Environment
Currently, we have a Linux environment ready to go and deploy everything needed for a small research lab with AUOMS configured and sending logs to Azure Sentinel:
Azure-Sentinel2Go/grocery-list/Linux at master · OTRF/Azure-Sentinel2Go (github.com)
We were able to use Azure Resource Manager (ARM) templates and a few bash scripts to automate the whole setup. These are all the resources used for each component of the lab:
- Azure Sentinel
- Linux Virtual Machines
What about the OMI Vulnerability?
As we know, older versions of the OMI agent (< 1.6.8.1) are vulnerable. Therefore, we created the following script to install version 1.6.8.0, and open port 5986.
Blacksmith/Install-OMI.sh at master · OTRF/Blacksmith (github.com)
We added that script to the Linux lab templates, and we now have a demo environment that you can also use to learn more about the exploitation of the OMI vulnerability.
Deploying the Lab Environment
Remember that this vulnerability is actively being exploited. Therefore, make sure you do not expose your lab environment to the Internet.
- Go to the following link: Azure-Sentinel2Go/grocery-list/Linux/demos/CVE-2021-38647-OMI at master · OTRF/Azure-Sentinel2Go (gi...
- Click on the âDeploy to Azureâ Button
- Fill out the following parameters:
- Subscription (selected by default)
- Resource group
- Region (selected by default)
- Admin Username
- Admin Password
- Remote Access Mode (AllowPublicIP selected by default. You can also use Azure Bastion Host. You would just need to set the Allowed IP Addresses parameter to *)
- Allowed IP Addresses (If you use the default access mode AllowPublicIP, use your home or office public IP address to only allow access from secure places. Remember that this vulnerability is actively being exploited. Therefore, make sure you do not expose your lab environment to the Internet.)
- Click the Review > Create buttons to start the deployment
- You can go to your resource group and explore all the resources being deployed
- Wait around 5-10 minutes! You should be good to go!
Validate Deployment
It is very important to validate if everything was deployed properly before doing research.
OMI Server
SSH to your virtual machines and check the OMI version to confirm it is 1.6.8-0
/opt/omi/bin/omiserver -v
Check if the OMI service is running
systemctl status omid
Check if port 5986 is open (You might have to update your package manager and install net-tools)
netstat -na | grep :5986
AUOMS Setup
Check if the AUOMS service is running with the following two commands:
sudo /opt/microsoft/auoms/bin/auomsctl status
systemctl status auoms
Check if events are being sent to the OMS Agent:
- Open another SSH session to your virtual machine and in one run the following command:
sudo /opt/microsoft/auoms/bin/auomsctl monitor
- Then, in the other session run whoami. If everything is connected properly, you will be able to see events flowing through your first session as shown below:
You can continue using `sudo /opt/microsoft/auoms/bin/auomsctl monitor` if you want to do research locally. You can have it running while you test the exploitation of the OMI vulnerability.
Azure Sentinel
Check if logs are being sent to your Azure Sentinel instance.
- https://portal.azure.com/
- Azure Sentinel
- Click on `logs` and explore the `Syslog` table
Learning About the OMI vulnerability
After validating that everything was deployed properly, you should be ready to run a few public proofs of concepts to test the OMI vulnerability.
One thing to remember is that there are three ways to execute arbitrary code via OMI. They are all part of the SCX RunAsProvider and their execution context varies a little bit.
Run Public POC (ExecuteShellCommand)
- SSH to machine A
- SSH to machine B
- On machine A, prepare the data that you want to send to machine B
- Request without authorization header
- Set the command you want to execute. For this example we execute âidâ.
- Use the ExecuteShellCommand method in data.
- Send HTTP request
Explore data in Azure Sentinel (ExecuteShellCommand)
You can run the following hunting query to explore the execution context:
Syslog
| parse SyslogMessage with "type=" EventType " audit(" * "): " EventData
| where EventType =~ "AUOMS_EXECVE" and EventData has '/var/opt/microsoft/scx/tmp'
| project TimeGenerated, EventType, Computer, EventData
| parse EventData with * "syscall=" syscall " syscall_r=" * " success=" success " exit=" exit " a0" *
" ppid=" ppid " pid=" pid " audit_user=" audit_user " auid=" auid " user=" user " uid=" uid " group=" group "
gid=" gid "effective_user=" effective_user " euid=" euid " set_user=" set_user " suid=" suid
" filesystem_user=" filesystem_user " fsuid=" fsuid " effective_group=" effective_group " egid=" egid
" set_group=" set_group " sgid=" sgid " filesystem_group=" filesystem_group " fsgid=" fsgid " tty=" tty
" ses=" ses " comm=\"" comm "\" exe=\"" exe "\"" * "cwd=\"" cwd "\"" * "name=\"" name "\"" * "cmdline=\"" cmdline "\"" *
| where uid == '0'
| where cwd == '/var/opt/microsoft/scx/tmp'
| where comm == 'sh'
| extend Timestamp = TimeGenerated, HostCustomEntity = Computer, AccountCustomEntity = user
We observed that the execution was happening from the `current working directory (cwd): /var/opt/microsoft/scx/tmp`. This is an indicator that repeats across the other two methods to execute arbitrary code abusing the OMI vulnerability. Group the results by the command line values to identify initial outliers.
Run Public POC (ExecuteScript)
- SSH to machine A
- SSH to machine B
- On machine A, prepare the data that you want to send to machine B
- Request without authorization header
- Set the script you want to execute:
- Pick a command. Let's say whoami
- Base64 encode the command: d2hvYW1p
- Use the ExecuteScript method in data.
- Send HTTP request
Explore data in Azure Sentinel (ExecuteScript)
You can run the previous hunting query again and explore the results. You will see that the current working directory (cwd) is the same, but the command line or in this case the script is now being hosted at the following directory: /etc/opt/microsoft/scx/conf/tmpdir/. The name of the scripts in that directory has the string âscxâ as a prefix. For example: scxzEPOS4.
Syslog
| parse SyslogMessage with "type=" EventType " audit(" * "): " EventData
| where EventType =~ "AUOMS_EXECVE" and EventData has '/var/opt/microsoft/scx/tmp'
| project TimeGenerated, EventType, Computer, EventData
| parse EventData with * "syscall=" syscall " syscall_r=" * " success=" success " exit=" exit " a0" * "
ppid=" ppid " pid=" pid " audit_user=" audit_user " auid=" auid " user=" user " uid=" uid " group=" group
" gid=" gid "effective_user=" effective_user " euid=" euid " set_user=" set_user " suid=" suid
" filesystem_user=" filesystem_user " fsuid=" fsuid " effective_group=" effective_group " egid=" egid
" set_group=" set_group " sgid=" sgid " filesystem_group=" filesystem_group " fsgid=" fsgid " tty=" tty
" ses=" ses " comm=\"" comm "\" exe=\"" exe "\"" * "cwd=\"" cwd "\"" * "name=\"" name "\"" * "cmdline=\"" cmdline "\"" *
| where uid == '0'
| where cwd == '/var/opt/microsoft/scx/tmp'
| where comm == 'sh'
| extend Timestamp = TimeGenerated, HostCustomEntity = Computer, AccountCustomEntity = user
I was wondering what process had created that file in that directory. I ran the following query to answer that question:
let syscallsList = dynamic(["unlink","openat","chmod"]);
Syslog
| parse SyslogMessage with "type=" EventType " audit(" * "): " EventData
| where EventType =~ "AUOMS_SYSCALL" and EventData contains "/etc/opt/microsoft/scx/conf/tmpdir/"
| project TimeGenerated, EventType, Computer, EventData
| parse EventData with * "syscall=" syscall " syscall_r=" * " success=" success " exit=" exit " a0" *
" ppid=" ppid " pid=" pid " audit_user=" audit_user " auid=" auid " user=" user " uid=" uid " group=" group
" gid=" gid "effective_user=" effective_user " euid=" euid " set_user=" set_user " suid=" suid
" filesystem_user=" filesystem_user " fsuid=" fsuid " effective_group=" effective_group " egid=" egid
" set_group=" set_group " sgid=" sgid " filesystem_group=" filesystem_group " fsgid=" fsgid " tty=" tty
" ses=" ses " comm=\"" comm "\" exe=\"" exe "\"" * "cwd=\"" cwd "\"" * "name=\"" name "\"" *
" path_name=" path_name " path_nametype=" path_nametype " path_mode=" * " proctitle=" cmdline " redactors=" *
| where syscall in (syscallsList)
| extend fileAction = (parse_json(path_nametype))[1]
| where fileAction in ("CREATE","DELETE")
It seems that the omiagent creates and deletes the file. The file is available only during the execution of the script. Once the execution is done, the file gets deleted. After doing some more research and reading some of the SCXCore code in GitHub, this the behavior of the ExecuteScript method:
How can we cover both methods (ExecuteShellCommand and ExecuteScript) and show what the script executed?
As mentioned before, both methods execute from the /var/opt/microsoft/scx/tmp directory. Therefore, all we have to do is create a JOIN query to show the process parent-child relationship to get to the commands executed via the script method:
let scx_execve=(){
Syslog
| parse SyslogMessage with "type=" EventType " audit(" * "): " EventData
| where EventType =~ "AUOMS_EXECVE" and EventData has '/var/opt/microsoft/scx/tmp'
| project TimeGenerated, EventType, Computer, EventData
| parse EventData with * "syscall=" syscall " syscall_r=" * " success=" success " exit=" exit " a0" * " ppid=" ppid
" pid=" pid " audit_user=" audit_user " auid=" auid " user=" user " uid=" uid " group=" group " gid=" gid
"effective_user=" effective_user " euid=" euid " set_user=" set_user " suid=" suid " filesystem_user=" filesystem_user
" fsuid=" fsuid " effective_group=" effective_group " egid=" egid " set_group=" set_group " sgid=" sgid
" filesystem_group=" filesystem_group " fsgid=" fsgid " tty=" tty " ses=" ses " comm=\"" comm "\" exe=\"" exe "\"" *
"cwd=\"" cwd "\"" * "name=\"" name "\"" * "cmdline=" cmdline " redactors=" *
| where uid == '0'
| where cwd == '/var/opt/microsoft/scx/tmp'
| where success == 'yes'
};
scx_execve
| where comm == 'sh' // ExecuteScript cmdline would trigger on /bin/sh /etc/opt/microsoft/scx/conf/tmpdir/scx_
| join kind=leftouter ( scx_execve ) on $left.Computer == $right.Computer, $left.pid == $right.ppid
| project-rename parentEventData=EventData,parentppid=ppid,parentpid=pid,parentcomm=comm,parentexe=exe,
parentname=name,parentcmdline=cmdline,childEventData=EventData1,childppid=ppid1,childpid=pid1,childcomm=comm1
,childexe=exe1,childname=name1,childcmdline=cmdline1
| project TimeGenerated, Computer, user, parentEventData,parentppid,parentpid,parentcomm,parentexe,parentname,parentcmdline,
childEventData,childppid,childpid,childcomm,childexe,childname,childcmdline
| extend Timestamp = TimeGenerated, HostCustomEntity = Computer, AccountCustomEntity = user
This is now the hunting query we share via our Azure Sentinel GitHub repo!
Azure-Sentinel/SCXExecuteRunAsProviders.yml at master · Azure/Azure-Sentinel (github.com)
Thatâs it! I am sure there is more to explore! I hope this lab environment can help you to test a few things in a safer way and experience what it might look like if it happens in your environment.
Once again, we highly recommend upgrading the OMI agent to version 1.6.8-1+, and if possible, controlling access to ports 5986,5985 and 1270. Remember that this vulnerability is actively being exploited. Therefore, make sure you do not expose your lab environment to the Internet.
In addition, remember that the behavior documented in this post is not malicious. The lab was created to help us understand how the execution of commands or a script was being handled by OMI from a data perspective. You must then go through the results of those queries and validate what is legitimate behavior or not depending on your organizationâs baseline.
Also, use this knowledge to map data you collect to every single action documented ;) You might be collecting data from other sources that provide the same or similar visibility.
Resources
- Linux Lab Environment - OMI Vulnerability: Azure-Sentinel2Go/grocery-list/Linux/demos/CVE-2021-38647-OMI at master · OTRF/Azure-Sentinel2Go (gi...
- Install OMS Agent bash script: Blacksmith/Install-OMS-Linux-Agent.sh at master · OTRF/Blacksmith (github.com)
- Install OMS Auditd Pluging bash script: Blacksmith/Install-OMS-Auditd-Plugin.sh at master · OTRF/Blacksmith (github.com)
- Install OMI bash script: Blacksmith/Install-OMI.sh at master · OTRF/Blacksmith (github.com)
- Hunting query: Azure-Sentinel/SCXExecuteRunAsProviders.yml at master · Azure/Azure-Sentinel (github.com)
References
- Hunting for OMI Vulnerability Exploitation with Azure Sentinel - Microsoft Tech Community
- Hunting Threats on Linux with Azure Sentinel - Microsoft Tech Community
- Guided Hunting Notebook: Base64-Encoded Linux Commands - Microsoft Tech Community
- Collect Syslog data sources with Log Analytics agent in Azure Monitor - Azure Monitor | Microsoft Do...
- microsoft/SCXcore: System Center Cross Platform Provider for Operations Manager (github.com)
- OMS-Agent-for-Linux/tools/OMIcheck at master · microsoft/OMS-Agent-for-Linux (github.com)
- 7.6. Understanding Audit Log Files Red Hat Enterprise Linux 7 | Red Hat Customer Portal
- Release v1.6.8-0 · microsoft/omi (github.com)
- microsoft/OMS-Auditd-Plugin: Auditd plugin that forwards audit events to OMS Agent for Linux (github...
- Azure-Sentinel/SCXRunAsProviderExecuteShellCommand.yml at master · Azure/Azure-Sentinel (github.com)