In the previous episode, we described how you can easily use PowerBi to represent Microsoft 365 data in a visual format. In this episode, we will explore another way you can interact with the Microsoft 365 Defender API. We will describe how to automate data analysis and hunting using Jupyter notebook.
Automate your hunting queries
While hunting and conducting investigations on a specific threat or IOC, you may want to use multiple queries to obtain wider optics on the possible threats or IOCs in your network. You may also want to leverage queries that are used by other hunters and use it as a pivot point to perform deep analysis and find anomalous behaviors. You can find a wide variety of examples in our Git repository where various queries related to the same campaign or attack technique are shared.
In scenarios such as this, it is sensible to leverage the power of automation to run the queries rather than running individual queries one-by-one.
This is where Jupyter Notebook is particularly useful. It takes in a JSON file with hunting queries as input and executes all the queries in sequence. The results are saved in a .csv file that you can analyze and share.
Before you begin
JUPYTER NOTEBOOK
If you're not familiar with Jupyter Notebooks, you can start by visiting https://jupyter.org for more information. You can also get an excellent overview on how to use Microsoft 365 APIs with Jupyter Notebook by reading Automating Security Operations Using Windows Defender ATP APIs with Python and Jupyter Notebooks.
VISUAL STUDIO CODE EXTENSION
If you currently use Visual Studio Code, make sure to check out the Jupyter extension.
Figure 1. Visual Studio Code – Jupyter Notebook extension
Another option to use Jupyter Notebook is the Microsoft Azure Machine Learning service.
Microsoft Azure Machine Learning is the best way to share your experiment with others and for collaboration.
Please refer to Azure Machine Learning - ML as a Service | Microsoft Azure for additional details.
Figure 2. Microsoft Azure Machine Learning
In order to create an instance, create a resource group and add the Machine Learning resource. The resource group lets you control all of the resources from a single entry point.
Figure 3. Microsoft Azure Machine Learning - Resource
When you’re done, you can run the same Jupyter Notebook you are running locally on your device.
Figure 4. Microsoft Azure Machine Learning Studio
App Registration
The easy way to access the API programmatically is to register an app in your tenant and assign the required permissions. This way, you can authenticate using the application ID and application secret.
Follow these steps to build your custom application.
- Connect to https://portal.azure.com/
- App registration
Figure 5. App registration
Select "NEW REGISTRATION".
Figure 6. Register an application
Provide the Name of your app, for example, MicrosoftMTP, and select Register.
Once done, select "API Permission".
Figure 7. API Permissions
Select "Add a permission".
Figure 8. Add permission
Select the "APIs my organization uses".
Figure 9. Alert Status
Figure 10. Request API permission
Search for Microsoft Threat Protection and select it.
Figure 11. Microsoft Threat Protection API
Select "Application Permission".
Figure 12. Application Permissions
Then select:
- AdvancedHunting.Read.All
- Incident.Read.All
Figure 13. Microsoft 365 Defender API - Read permission
Once done select "Add permissions".
Figure 14. Microsoft 365 Defender API - Add permission
Get Started
Now that we have the application ready to access the API via code, let’s try to see is any of the Qakbot queries shared in Microsoft 365 Defender Git produce any results.
Figure 15. Microsoft 365 Defender – Hunting Queries
The following queries will be used in this tutorial:
Javascript use by Qakbot malware
Process injection by Qakbot malware
Registry edits by campaigns using Qakbot malware
Self-deletion by Qakbot malware
Outlook email access by campaigns using Qakbot malware
Browser cookie theft by campaigns using Qakbot malware
Detect .jse file creation events
We need to grab the queries that we want to submit and populate a JSON file with this format. Please be sure that you are properly managing the escape character in the JSON file (if you use Visual Studio Code (VSCode) you can find extensions that can make the ESCAPE/UNESCAPE process easiest, just pick your favorite one).
[
{
"Description": "Find Qakbot overwriting its original binary with calc.exe",
"Name": "Replacing Qakbot binary with calc.exe",
"Query": "DeviceProcessEvents | where FileName =~ \"ping.exe\" | where InitiatingProcessFileName =~ \"cmd.exe\" | where InitiatingProcessCommandLine has \"calc.exe\" and InitiatingProcessCommandLine has \"-n 6\" and InitiatingProcessCommandLine has \"127.0.0.1\" | project ProcessCommandLine, InitiatingProcessCommandLine, InitiatingProcessParentFileName, DeviceId, Timestamp",
"Mitre": "T1107 File Deletion",
"Source": "MDE"
}
]
Once you have all your queries properly filled, we must provide the following parameters to the script in order to configure the correct credential, the JSON file, and the output folder.
Figure 16. Jupyter Notebook – Authentication
Because we registered an Azure Application and we used the application secret to receive an access token, the token is valid for 1 hour. Within the code verify if we need to renew this token before submitting the query.
Figure 17. Application Token lifetime validation
When building such flow we should take into consideration Microsoft 365 Defender Advanced hunting API quotas and resources allocation. For more information, see Advanced Hunting API | Microsoft Docs.
Figure 18. API quotas and resources allocation taking into consideration
We run the code by loading the query from the JSON file we defined as input. We then view the progress and the execution status on screen.
Figure 19. Query Execution
The blue message indicates the number of queries that is currently running and its progress.
The green message shows the name of the query that is being run.
The grey message shows the details of the submitted query.
If there are any results you will see the first 5 records, and then all the records will be saved in a .csv file in the output folder you defined.
Figure 20. Query results - First 5 records
Bonus
You can post the summary of the query execution in a Teams channel, you need to add Incoming Webhook in your teams.
Figure 21. Incoming Webhook
Then you need to select which Teams channel you want to add the app.
Figure 22. Incoming Webhook – add to a team
Select “Set up a connector”.
Figure 23. Incoming Webhook – Setup a connector
Specify a name.
Figure 24. Incoming Webhook – Config
Now you need to copy the URL, then paste the URL in the Jupyter Notebook.
Figure 25. Incoming Webhook – teamurl variable
Then remove the comment from the latest line in the code to send the message to Teams.
Figure 26. Incoming Webhook – teamsurl variable
You should receive a similar message like the following in the Teams channel:
Figure 27. Query result summary – Teams Message
Conclusion
In this post, we demonstrated how you can use the Microsoft 365 Defender APIs and Jupyter Notebook to automate execution of hunting queries playbook. We hope you found this helpful!
Appendix
For more information about Microsoft 365 Defender APIs and the features discussed in this article, please read:
- Access the Microsoft 365 Defender APIs | Microsoft Docs
- Advanced hunting APIs - Microsoft 365 security | Microsoft Docs
- Advanced hunting best practices
- Microsoft-365-Defender-Hunting-Queries
- Best practices for leveraging Microsoft 365 Defender API's - Episode Two - Microsoft Tech Community
- Sending messages to Connectors and Webhooks - Teams | Microsoft Docs
- Automating Security Operations Using Windows Defender ATP APIs with Python and Jupyter Notebooks - M...
The sample Notebook discussed in the post is available in the github repository
Microsoft-365-Defender-Hunting-Queries/M365D APIs ep3.ipynb at master · microsoft/Microsoft-365-Defe...
As always, we’d love to know what you think. Leave us feedback directly on Microsoft 365 security center or start a discussion in Microsoft 365 Defender community