One common questions that we get from customers and partners is how to ingest data from Azure Log Analytics / Azure Sentinel to Azure Data Explorer from different reasons: joining data between different ADX clusters, longer retention period, heavy queries that aren't alight with LA limitations and etc.
There are many good articles in the web, here is a step by step guide and code from Sentinel team at GitHub for automation script to integrate Azure Data Explorer for Long term storage option for Azure Sentinel Log Analytics Workspace
https://github.com/Azure/Azure-Sentinel/tree/master/Tools/AzureDataExplorer
- Asking input from the user
- Do you want all the tables from the LA? Yes or No
- If Yes, will get all tables
- If No, User will enter table names (,) separated
- Before creating TableRAW and TableRAWMapping, checking against fully supported tables
https://docs.microsoft.com/en-us/azure/azure-monitor/logs/logs-data-export?tabs=rest#supported-tables - Dividing the tables into Size 10 and then creating Standard EventHub Namespaces programmatically for each 10 tables
- Creating “Data Export” rule programmatically using REST API
- Creating “Data Connection” rule in Azure Data Explorer Database programmatically using REST API
- Creating Log file to verify what went successfully vs wrong
Azure Log Analytics Log Management using Azure Data Explorer by Sreedhar Ande
Additional resources:
- Blog: https://techcommunity.microsoft.com/t5/azure-sentinel/using-azure-data-explorer-for-long-term-retention-of-azure/ba-p/1883947
- Official documentation: https://docs.microsoft.com/en-us/azure/sentinel/store-logs-in-azure-data-explorer?tabs=adx-event-hub
- MSSP Architecture Reference (page 23-24): MSSP Playbook
- Script to provision ADX for Sentinel long-retention: https://github.com/sreedharande/AzureDataExplorer
Updated Sep 12, 2021
Version 7.0
Tzvia
Microsoft
Microsoft
