Forum Discussion
Connecting Azure Databricks VMs to external VNet for Log Analytics ingestion
Hello, We run an ADB cluster using cluster pools. I'm looking for a way connect the ADB cluster pool VMs which reside on the automatically generated Resource Group databricks-rg-<xyz> to another ...
It sounds like you want to enable private communication between your ADB cluster pool VMs and your Log Analytics workspace. To do this, you can use Azure Private Link, which allows you to access Azure services over a private endpoint within your own virtual network.
Here's how you can set up Private Link between your ADB cluster pool and Log Analytics workspace:
Create a new virtual network in the same region as your ADB cluster pool. This virtual network will be used to host the Private Link endpoint for Log Analytics.
Create a Private Link service for your Log Analytics workspace. This will allow you to connect to your workspace over a private endpoint within your virtual network.
In your virtual network, create a Private Link endpoint for Log Analytics. This will allow your ADB cluster pool VMs to connect to Log Analytics over a private IP address.
Create a virtual network peering between your ADB cluster pool virtual network and your Log Analytics virtual network. This will allow your ADB cluster pool VMs to communicate with Log Analytics over the Private Link endpoint.
Configure your ADB cluster pool VMs to send their OMS metrics to the Log Analytics Private Link endpoint.
By following these steps, you can ensure that all communication between your ADB cluster pool and Log Analytics workspace is secure and private, and that no data is transmitted over the public internet.
Here's how you can set up Private Link between your ADB cluster pool and Log Analytics workspace:
Create a new virtual network in the same region as your ADB cluster pool. This virtual network will be used to host the Private Link endpoint for Log Analytics.
Create a Private Link service for your Log Analytics workspace. This will allow you to connect to your workspace over a private endpoint within your virtual network.
In your virtual network, create a Private Link endpoint for Log Analytics. This will allow your ADB cluster pool VMs to connect to Log Analytics over a private IP address.
Create a virtual network peering between your ADB cluster pool virtual network and your Log Analytics virtual network. This will allow your ADB cluster pool VMs to communicate with Log Analytics over the Private Link endpoint.
Configure your ADB cluster pool VMs to send their OMS metrics to the Log Analytics Private Link endpoint.
By following these steps, you can ensure that all communication between your ADB cluster pool and Log Analytics workspace is secure and private, and that no data is transmitted over the public internet.