Stream your advanced hunting events to your Azure storage account and control your data with Azure storage lifecycle rules
Oftentimes, organizations require better control over their raw data. Typical scenarios where increased control is needed include:
- Data retention policies.
- Business needs for long term investigations.
- Integration with other security\Big-data products.
To answer this need, Microsoft Defender ATP allows you to stream advanced hunting events to Azure Event Hubs or to an Azure storage account.
In this blog, I am going to demonstrate how you can easily stream your advanced hunting events to Azure storage account and set an Azure blob storage lifecycle rule to move old data to low-cost storage.
Let’s start
The following four simple steps will get you up and running with the required configurations:
- Step 1: Create a storage account in your Azure tenant.
- Step 2: Register to Microsoft.insights provider with your subscription.
- Step 3: Enable raw data streaming in Microsoft Defender ATP Portal.
- Step 4: Set an Azure blob lifecycle rule.
Note:
you can find full documentation for raw data streaming API in this link.
Step 1 - Create a storage account in your Azure tenant:
To create an Azure storage account, follow these steps:
- Sign in to the Azure portal.
- Go to All Services > Storage Account.
- Click Add.
- In the Create storage account form enter the following:
1 – Choose your Azure’s subscription.2 – Choose the Resource Group you want to add the storage account to.
3 – Give your new storage account a name.
Leave all other fields set to their default values, or you can use the tooltip for each configuration to find the meaning of each setting.
- Select Review + Create to review your storage account settings and create the account.
- Select Create.
- Save your new storage account resource ID (you will need it on Step 3)
- Go to Storage account > {your new storage account name} > Properties.
- Copy the value in “storage account resource ID” textbox and save it in Step 3: Enable Raw data streaming in Microsoft Defender ATP Portal.
Done! You have successfully created a new storage account.
Step 2: Create a subscription to Microsoft.insights provider
- Log in to your Azure tenant.
- Go to Subscriptions > {Your subscription name}.
- Go to Resource Providers. Click on Microsoft.insights and select Register.
Done! You have successfully registered to Microsoft.insights provider.
Step 3: Enable Raw data streaming in Microsoft Defender ATP Portal
- Log in to Microsoft Defender ATP portal with a Global Admin role.
- Go to Interoperability > Data export settings> Add data export settings.
- Choose a Name to your new settings.
- Select Forward events to Azure Storage.
- Type your Storage Account Resource ID you saved at the end of Step 1.
- Choose the event type you want to forward.
- Click Save.
Done! You have successfully enabled raw data steaming.
In about 5 minutes, data will start to be written to the blob storage.
You can view your raw data files on Azure portal:
Go to Storage account > {your new storage account name} > overview > Blobs.
You’ll see that new file created for each event type on an hourly basis:
The schema of each row in each file in the blob is the following JSON:
|
{ "time": "" "tenantId": "" "category": "" "properties": { } } |
Step 4: Set Azure blob lifecycle rule
You now have your data stored on your storage blob. Let’s create a rule that set the periods of time for each stage in our data lifecycle.
I will demonstrate how I created a rule for the following lifecycle definition:
- Move blob data from hot storage to cold storage after 30 days from last modification.
- Move blob data from cold storage to archive storage after 90 days from last modification.
- Delete blob data after 365 days from last modification.
- Open Azure Portal.
- Go to Storage account > {your new storage account name} > overview > Blob.
- Under Blob Service, select Lifecycle management -> Add rule.
- Give your new rule a name and check the setting you want to set in the rule.
- Click Review + add
Done! you have successfully created an Azure storage lifecycle rule for your raw Microsoft Defender ATP data.
In the next blog, we will demonstrate how to stream advanced hunting events to Azure Event Hubs.
Thanks,
@Haim Goldshtein, Security software engineer, Microsoft Defender ATP
@Dan Michelson, Program Manager, Microsoft Defender ATP
@Ben Alfasi, Software engineer, Microsoft Defender ATP