Application Insights data from Data Lake into Data Explorer

[vukovicd]

Hi!

I have a scenario where Azure-Subscription_A (Department A) is exporting data from multiple Application Insights resources into a Data-Lake-Gen2_A. The data from the Data-Lake-Gen2_A is being shared using Azure-Data-Share_A and received by Azure-Data-Share_B and mapped into a Data-Lake-Gen2_B which are located in Azure-Subscription_B (Department B - different AAD tenant). 

I would like to ingest the data from Data-Lake-Gen2_B into Azure Data Explorer to be able to explore the data, but am not sure how to implement the tables and the ingestion process. I have seen that ADX offers ingestion mappings, but I could not figure out how this should work with the data format of the application insights data stored in the data lake. Also, is using Azure Data Factory a better option for the ingestion into ADX from the Data Lake? 

I would appreciate it if someone could point me in the right direction or link me some guides/samples that would help me out.

Thanks!    

[#2]

Hello @vukovicd, 

Many folks are routing data from Azure Monitor Application Insights to Azure Data Explorer automatically. Data from Application Insights ends up as jsonl - A set of json objects each one in its own line in the file.

Typically they use the EventGrid method to listen on the data lake container and automatically ingest data as it lands. https://docs.microsoft.com/en-us/azure/data-explorer/ingest-data-event-grid

In order to set up the tables, column mappings etc, 1-click ingestion helps you do it based on your data sample:

https://docs.microsoft.com/en-us/azure/data-explorer/ingest-data-one-click

Please let us know how it works for you.

Uri