Best practices for leveraging Microsoft 365 Defender API's - Episode Three
Published Apr 26 2021 09:15 AM 3,913 Views
Microsoft

In the previous episode, we described how you can easily use PowerBi to represent Microsoft 365 data in a visual format. In this episode, we will explore another way you can interact with the Microsoft 365 Defender API. We will describe how to automate data analysis and hunting using Jupyter notebook.

 

Automate your hunting queries 

While hunting and conducting investigations on a specific threat or IOC, you may want to use multiple queries to obtain wider optics on the possible threats or IOCs in your network. You may also want to leverage queries that are used by other hunters and use it as a pivot point to perform deep analysis and find anomalous behaviors. You can find a wide variety of examples in our Git repository where various queries related to the same campaign or attack technique are shared.  

In scenarios such as this, it is sensible to leverage the power of automation to run the queries rather than running individual queries one-by-one.  

This is where Jupyter Notebook is particularly useful. It takes in a JSON file with hunting queries as input and executes all the queries in sequence. The results are saved in a .csv file that you can analyze and share. 

 

Before you begin 

JUPYTER NOTEBOOK 

If you're not familiar with Jupyter Notebooks, you can start by visiting https://jupyter.org for more information. You can also get an excellent overview on how to use Microsoft 365 APIs with Jupyter Notebook by reading Automating Security Operations Using Windows Defender ATP APIs with Python and Jupyter Notebooks.   

 

VISUAL STUDIO CODE EXTENSION 

If you currently use Visual Studio Code, make sure to check out the Jupyter extension 

msftdario_27-1619422918103.png

Figure 1. Visual Studio Code – Jupyter Notebook extension 

 

Another option to use Jupyter Notebook is the Microsoft Azure Machine Learning service. 

Microsoft Azure Machine Learning is the best way to share your experiment with others and for collaboration. 

Please refer to Azure Machine Learning - ML as a Service | Microsoft Azure for additional details. 

msftdario_28-1619422918116.png

Figure 2. Microsoft Azure Machine Learning 

 

In order to create an instance, create a resource group and add the Machine Learning resource. The resource group lets you control all of the resources from a single entry point. 

msftdario_29-1619422918122.png

Figure 3. Microsoft Azure Machine Learning - Resource 

 

When you’re done, you can run the same Jupyter Notebook you are running locally on your device.  

msftdario_30-1619422918118.png

Figure 4. Microsoft Azure Machine Learning Studio 

 

App Registration 

The easy way to access the API programmatically is to register an app in your tenant and assign the required permissions. This way, you can authenticate using the application ID and application secret. 

Follow these steps to build your custom application. 

msftdario_31-1619422977477.png

Figure 5. App registration 

  

Select "NEW REGISTRATION". 

  

msftdario_32-1619422977481.png

Figure 6. Register an application 

 

Provide the Name of your app, for example, MicrosoftMTP, and select Register. 

Once done, select "API Permission". 

  

msftdario_33-1619422977495.png

Figure 7. API Permissions 

  

Select "Add a permission". 

msftdario_34-1619422977484.png

Figure 8. Add permission 

 

Select the "APIs my organization uses". 

  

msftdario_35-1619422977485.png

  Figure 9. Alert Status 

  

msftdario_36-1619422977486.png

Figure 10. Request API permission 

  

Search for Microsoft Threat Protection and select it. 

msftdario_37-1619422977487.png

Figure 11. Microsoft Threat Protection API 

 

Select "Application Permission". 

msftdario_38-1619422977489.png

Figure 12. Application Permissions 

 

Then select: 

  • AdvancedHunting.Read.All 
  • Incident.Read.All 

 

msftdario_39-1619422977491.png

Figure 13. Microsoft 365 Defender API - Read permission 

 

Once done select "Add permissions". 

msftdario_40-1619422977492.png

Figure 14. Microsoft 365 Defender API - Add permission 

 

Get Started 

Now that we have the application ready to access the API via code, let’s try to see is any of the Qakbot queries shared in Microsoft 365 Defender Git produce any results. 

msftdario_41-1619423114158.png

 

Figure 15. Microsoft 365 Defender – Hunting Queries 

 

The following queries will be used in this tutorial:  

 

Javascript use by Qakbot malware 

Process injection by Qakbot malware 

Registry edits by campaigns using Qakbot malware 

Self-deletion by Qakbot malware 

Outlook email access by campaigns using Qakbot malware 

Browser cookie theft by campaigns using Qakbot malware 

Detect .jse file creation events 

 

We need to grab the queries that we want to submit and populate a JSON file with this formatPlease be sure that you are properly managing the escape character in the JSON file (if you use Visual Studio Code (VSCode) you can find extensions that can make the ESCAPE/UNESCAPE process easiest, just pick your favorite one). 

 

 

 

 

 

[ 
        { 
            "Description": "Find Qakbot overwriting its original binary with calc.exe", 
            "Name": "Replacing Qakbot binary with calc.exe", 
            "Query": "DeviceProcessEvents | where FileName =~ \"ping.exe\" | where InitiatingProcessFileName =~ \"cmd.exe\" | where InitiatingProcessCommandLine has \"calc.exe\" and InitiatingProcessCommandLine has \"-n 6\" and InitiatingProcessCommandLine has \"127.0.0.1\" | project ProcessCommandLine, InitiatingProcessCommandLine, InitiatingProcessParentFileName, DeviceId, Timestamp", 
            "Mitre": "T1107 File Deletion", 
            "Source": "MDE" 
        } 
] 

 

 

 

 

 

Once you have all your queries properly filled, we must provide the following parameters to the script in order to configure the correct credential, the JSON file, and the output folder. 

msftdario_42-1619423295313.png

Figure 16. Jupyter Notebook – Authentication 

 

Because we registered an Azure Application and we used the application secret to receive an access token, the token is valid for 1 hour. Within the code verify if we need to renew this token before submitting the query. 

msftdario_43-1619423295303.png

Figure 17. Application Token lifetime validation 

 

When building such flow we should take into consideration Microsoft 365 Defender Advanced hunting API quotas and resources allocation. For more information, see Advanced Hunting API | Microsoft Docs.  

msftdario_44-1619423295312.png

Figure 18. API quotas and resources allocation taking into consideration 

 

 We run the code by loading the query from the JSON file we defined as input. We then view the progress and the execution status on screen. 

msftdario_45-1619423295315.png

Figure 19. Query Execution 

 

The blue message indicates the number of queries that is currently running and its progress. 

The green message shows the name of the query that is being run. 

The grey message shows the details of the submitted query. 

If there are any results you will see the first 5 records, and then all the records will be saved in a .csv file in the output folder you defined. 

 

msftdario_46-1619423295309.png

Figure 20.  Query results - First 5 records 

 

Bonus 

You can post the summary of the query execution in a Teams channel, you need to add Incoming Webhook in your teams. 

 

msftdario_47-1619423368892.png

Figure 21.  Incoming Webhook 

 

Then you need to select which Teams channel you want to add the app. 

msftdario_48-1619423368929.png

Figure 22.  Incoming Webhook – add to a team 

 

Select “Set up a connector”. 

msftdario_49-1619423368932.png

Figure 23.  Incoming Webhook – Setup a connector 

 

Specify a name. 

msftdario_50-1619423368937.png

Figure 24.  Incoming Webhook – Config 

 

Now you need to copy the URL, then paste the URL in the Jupyter Notebook. 

msftdario_51-1619423368906.png

Figure 25.  Incoming Webhook – teamurl variable 

 

Then remove the comment from the latest line in the code to send the message to Teams. 

msftdario_52-1619423368909.png

Figure 26.  Incoming Webhook – teamsurl variable 

 

You should receive a similar message like the following in the Teams channel: 

msftdario_53-1619423368914.png

Figure 27.  Query result summary – Teams Message 

 

Conclusion 

In this post, we demonstrated how you can use the Microsoft 365 Defender APIs and Jupyter Notebook to automate execution of hunting queries playbook. We hope you found this helpful! 

 

Appendix  

For more information about Microsoft 365 Defender APIs and the features discussed in this article, please read: 

The sample Notebook discussed in the post is available in the github repository
Microsoft-365-Defender-Hunting-Queries/M365D APIs ep3.ipynb at master · microsoft/Microsoft-365-Defe...

 

As always, we’d love to know what you think. Leave us feedback directly on Microsoft 365 security center or start a discussion in Microsoft 365 Defender community

Co-Authors
Version history
Last update:
‎Apr 26 2021 09:15 AM
Updated by: