Special thanks to @Satyajit Dash, Anki Narravula, and Sidhartha Bose for their contributions.
Office 365 Advanced Threat Protection provides several built-in security alerts to help customers detect and mitigate potential threats. In many large enterprises there is often a need to integrate these alerts with a SIEM platform or other case management tools to enable a Security Operations Center (SOC) team to monitor the alerts. The Office 365 Management Activity API provides these SOC teams the ability to integrate O365 ATP alerts with other platforms.
One of the challenges that organizations often face, particularly large enterprises, is the ever-increasing volume of alerts that the SOC needs to monitor. This at times makes it important to integrate and onboard only specific alerts to the monitoring and case management platforms or SIEM.
Let’s take an integration scenario that we worked on recently. As part of effort to deal with phishing related threats, one of our large enterprise customers wanted to fetch and integrate “user-reported phishing alerts”. However, they only wanted to their SOC to get those alerts that have already been processed by an Automated Investigation and Response (AIR) playbook to reduce false positives and focus on real threats.
Our engineering team worked on a solution which efficiently fetches only the relevant alerts using the Office 365 Management API and integrates them with the SIEM and case management platform. Below is the solution and the reference architecture. This could potentially be used to fetch and integrate other relevant alerts from Office 365 ATP.
As mentioned in the introduction, we have used the Azure cloud to set up end-to-end infrastructure for getting O365 audit events and storing the required filtered data for near real-time security monitoring and historical analysis. After evaluating and analyzing various combinations of Azure services, we have decided to use following Azure components.
The architecture diagram below depicts the end-to-end setup.
To access audit data from the O365 Management API, we’ll configure access though an Azure Active Directory (AAD) application. Create an AAD application using the steps below, and get access to this AAD application from the tenant admin. Once you have access, keep a note of the Client Id and secret key of this application, because we will need these details later.
We need to set up a storage account for data storage, which we will use as a staging area. We can retain data here for longer term, to be used for historical data analysis. Below is the step-by-step process for setting up the storage account.
Once we create the container, we can see the container like below when we navigate to “Container” section inside the storage account.
Steps to create Storage Account
We will use Azure File share to store the .dlls needed by Azure Data Explorer for the ingestion process. Follow the step-by-step process as described to create an Azure file share in the storage account. We can create this file share under the same storage account, which we have created in previous step.
Create an Azure Data Explorer cluster and database by following this step-by-step guide or the steps given below. Once the cluster is created, copy the details of cluster and database which we’ll need when inputting the Azure Automation Account variables.
For writing data into Azure Data Explorer we will use “Service Principal / AAD application” access (using the service principal secret). We will need “Admin” permissions during the first run of the script to create a table in the Kusto database. After the table is successfully created, permissions can be reduced to “Ingestor” permissions.
We need to upload Kusto libraries to the Azure Storage Account File Share.
Setup Azure Key Vault and store the required secrets. Key vault name and secret names will be used in PowerShell runbooks.
Make sure that Run-as account of Automation account has access to the Key Vault (Read – Get Keys and Secrets).
We have broken down our PowerShell scripts into 3 different runbooks to achieve the following.
Deploy 3 runbooks into the automation account by following the steps below.
Table 1: Runbook scripts
Name of the runbook (Provide the same as below while importing) |
Type of Runbook |
PowerShell Script to import |
GetO365DataBlobURLs |
PowerShell |
GetO365DataBlobURLs.ps1 |
GetO365GeneralAuditData |
PowerShell |
GetO365GeneralAuditData.ps1 |
ExporttoKusto_O365AuditGeneralData |
PowerShell |
ExporttoKusto_O365AuditGeneralData.ps1 |
Table 2: Variables
Variable Name |
Data Type |
Value description |
AutomationAccountName |
String |
Name of the Automation Account created in chapter 8 |
AutomationAccountResourceGroupName |
String |
Name of Resource Group which Automation Account was created in. |
BlobStorageContainerName |
String |
Name of Container created in chapter 4 |
ClientIDtoAccessO365 |
String |
Azure AD application ID created in chapter 3.1 |
ClientSecrettoAccessO365 |
String (encrypted or from KeyVault) |
Azure AD application secret created in chapter 3.1 |
FileShareNameinStorageAccount |
String |
Name of the file share created in chapter 5 |
KeyVaultName |
String |
Name of the Azure Key Vault created in chapter 9 |
KustoAccessAppId |
String |
Azure Run As Account application ID. Steps to find it are described in chapter 12.1 |
KustoAccessAppKey (optional) |
String (encrypted or from KeyVault) |
Not required if access key is stored in Azure Key Vault in KustoAccessAppKey |
KustoClusterName |
String |
Azure Data Explorer cluster name created in chapter 6 |
KustoDatabaseName |
String |
Azure Data Explorer database name created in chapter 6 |
KustoIngestionURI |
String |
|
KustoTableName |
String |
Name of the Azure Data Explorer table that PowerShell script will create. |
MicrosoftLoginURL |
String |
|
O365ResourceUrl |
String |
|
O365TenantDomain |
String |
Default domain name of the tenant |
O365TenantGUID |
String |
This is the ID of the Office 365 tenant where alerts and investigation will be pulled out from. Follow this article to locate tenant ID. https://docs.microsoft.com/en-us/onedrive/find-your-office-365-tenant-id |
PathforKustoExportDlls |
String |
Name of the folder created in the file share in chapter 5. |
RunbookNameforExportDatatoKusto |
String |
ExporttoKusto_O365AuditGeneralData |
RunbookNameforGetAuditDataBlobURIs |
String |
GetO365DataBlobURLs |
RunbookNameforGetAuditDataFromURIs |
String |
GetO365GeneralAuditData |
StorageAccountforBlobstorage |
String |
Name of the storage account created in chapter 4 |
StorageAccountforBlobstorageAccessKey (optional) |
String (encrypted or from KeyVault) |
Not required if access key is stored in Azure Key Vault in StorageAccountforBlobstorageAccessKey (default configuration) |
StorageAccountforFileShare |
String |
Storage account created in chapter 4 |
StorageAccountforFileShareAccessKey (optional) |
String (encrypted or from KeyVault) |
Not required if access key is stored in Azure Key Vault in StorageAccountforFileShareAccessKey (default configuration) |
TenantIdforKustoAccessApp |
String |
This is the ID of the Azure AD tenant where Azure Run As Account is provisioned. Follow this article to locate tenant ID. https://docs.microsoft.com/en-us/onedrive/find-your-office-365-tenant-id |
You can find the KustoAccessAppId by navigating to your Automation Accounts. On the Automation Accounts page, select your Automation account from the list. In the left pane, select Run As Accounts in the account settings section. Click Azure Run As Account. Copy Application ID and paste it as KustoAccessAppId variable value.
After data is successfully imported by the scripts you can query it using KQL.
In the Azure Portal navigate to Azure Data Explorer Clusters. Click on the cluster name. Click on Query.
Example query to verify that data is ingested:
KustoAuditTable
| extend IngestionTime=ingestion_time()
| order by IngestionTime desc
| project Name,Severity,InvestigationType,InvestigationName,InvestigationId,CreationTime,StartTimeUtc,LastUpdateTimeUtc,EndTimeUtc,Operation,ResultStatus,UserKey,ObjectId,Data,Actions,Source,Comments,Status
And finally, after all this effort an example of the output:
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.