Does Azure Log Analytics has provision to load data from Azure Data Bricks

[yuraje]

Exploring a way to load data from Azure DataBricks into Azure log analytics. Basically the DataBricks dataframe (Json data)should be loaded into Azure Log Analytics. 

[#2]

@yuraje 

Hi what is your main goal  ? 

Send Azure Databricks application logs or diagnostic logging for the resource itself ? 

 .Logs are  collected  by Azure Monitor Logs so you can do some queries after ingestion . 

I've seen that you can also define custom logs if native doesn't fit your needs

Send Azure Databricks application logs to Azure Monitor - Azure Architecture Center | Microsoft Docs

Diagnostic logging in Azure Databricks - Azure Databricks - Workspace | Microsoft Docs

[#3]

@yuraje Azure log analytics can import data using a custom field format.  If you've stored logs within DataBricks, you first need to export them to .txt files and then ingest them with Log Analytics.

Export your data from DataBricks as .txt files.  First do a sample locally, then to Azure storage.

Then follow the steps in this article to import the logs.  https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-custom-logs#defining-a-cu...

[#4]

Hi @yuraje ,

If you would like to send diagnostic logs from Azure Databricks to Azure Log Analytics then it is possible. Please refer to this link : https://docs.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagno...

[#5]

@cberon & @ibnmbodji  - Thank you for your response.

My goal is to push the data from Azure DataBricks (basically its a application data on which I want to perform some analysis to identify anomaly or threat) into LogAnalytics. 

I achieved it using below method.

I have created a dataframe in ADB with the required dataset and converted into JSON format using inbuild py function. Using POST API method I'm able to push this dataframe into LogAnalytics. Previously I have created a signature with Workspace ID and Key for authentication. 

[#6]

@yuraje 

Awsome ! thanks for this input .May be if you have time you can drop a tutorial so everyone can see how it is implemented .