Does Azure Log Analytics has provision to load data from Azure Data Bricks

Question

Exploring a way to load data from Azure DataBricks into Azure log analytics. Basically the DataBricks dataframe (Json data)should be loaded into Azure Log Analytics.&nbsp;

sagar_lad · Answer

Hi&nbsp;yuraje&nbsp;,&nbsp;If you would like to send diagnostic logs from Azure Databricks to Azure Log Analytics then it is possible. Please refer to this link :&nbsp;https://docs.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/azure-diagnostic-logs&nbsp;&nbsp;

yuraje · Answer

cberon&nbsp;&amp;&nbsp;ibnmbodji&nbsp;&nbsp;- Thank you for your response.
My goal is to push the data from Azure DataBricks (basically its a application data on which I want to perform some analysis to identify anomaly or threat) into LogAnalytics.&nbsp;
&nbsp;
I achieved it using below method.
I have created a dataframe in ADB with the required dataset and converted into JSON format using inbuild py function. Using POST API method I'm able to push this dataframe into LogAnalytics. Previously I have created a signature with Workspace ID and Key for authentication.&nbsp;

ibnmbodji · Answer

yuraje&nbsp;Awsome ! thanks for this input .May be if you have time you can drop a tutorial so everyone can see how it is implemented .&nbsp;

ibnmbodji · Answer

yuraje&nbsp;&nbsp;Hi what is your main goal&nbsp; ?&nbsp;Send Azure Databricks application logs&nbsp;or diagnostic logging for the resource itself ?&nbsp; &nbsp;.Logs are&nbsp; collected&nbsp;&nbsp;by Azure Monitor Logs so you can do some queries after ingestion .&nbsp;I've seen that you can also define custom logs if native doesn't fit your needs&nbsp;Send Azure Databricks application logs to Azure Monitor - Azure Architecture Center | Microsoft DocsDiagnostic logging in Azure Databricks - Azure Databricks - Workspace | Microsoft Docs&nbsp;&nbsp;

cberon · Answer

yuraje&nbsp;Azure log analytics can import data using a custom field format.&nbsp; If you've stored logs within DataBricks, you first need to export them to .txt files and then ingest them with Log Analytics.&nbsp;Export your data from DataBricks as .txt files.&nbsp; First do a sample locally, then to Azure storage.&nbsp;Then follow the steps in this article to import the logs.&nbsp;&nbsp;https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-custom-logs#defining-a-custom-log&nbsp;&nbsp;

Forum Discussion

Does Azure Log Analytics has provision to load data from Azure Data Bricks

5 Replies

Resources