The Challenge: Outbound Control in a Serverless World

Serverless compute resources run in the serverless compute plane, and is managed by Microsoft, for ease of use.  Databricks serverless provides hassle-free compute for running notebooks, jobs, and pipelines and by default, outbound traffic can reach the internet and other networks freely.

One of the most common security requirements for customers in the financial and government sectors is to have the ability to retain network paths within a private network for their users to access their data and for system integration. 

Solution Objective

• Enforce deny-by-default posture: Control outbound access with granular precision by enabling a deny-by-default policy for internet. By default, all outbound access is blocked unless explicitly allowed via Private Endpoint Rules.
• Control outbound connections: by specifying allowed locations, connections, FQDN
• Enforce the traffic to go over the customer network for traffic controls and inspection

Solution Overview

The solution is designed to route Databricks Serverless outbound traffic to the customer customer-managed Policy Enforcement Point (e.g. Azure Firewall), to allow the customer to securely connect to services hosted on the cloud without exposing the data to the public internet. Essentially, it establishes a private, secure connection between Databricks Control Plane to the customer virtual network.

Pre-Requisites

Azure Firewall

• Deploy an Azure Firewall if you don’t already have one.

Virtual Networks and Subnets

• Create a VNET for Databricks and Load Balancer deployment.
• Set up subnets for the Azure Standard Load Balancer frontend (e.g., 10.0.2.0/26) and backend (e.g., 10.0.2.64/26).
• Enable Private Endpoint network policy for Network Security Groups and Route Tables on the backend subnet.

VNET Peering

• Peer the Databricks VNET with your hub VNET to allow secure routing.

Azure Databricks Workspace

• Deploy an Azure Databricks workspace if you don’t have one.

Follow the official Azure Databricks documentation for detailed steps on creating workspaces and private endpoints

Summary of Steps

Deploy Azure Firewall and Networking

• Set up an Azure Firewall and create the necessary virtual networks (VNets) and subnets for your environment.
• Peer the Databricks VNet with your hub VNet to enable secure routing.

Configure the Azure Load Balancer

• Create an internal Standard Load Balancer.
• Set up frontend and backend pools using NICs (not IP addresses).
• Add load balancing rules and configure a health probe (typically HTTP on port 8082).

Create a Private Link Service

• Deploy the Private Link Service behind the load balancer.
• Associate it with the correct frontend and backend subnets.

Set Up Route Tables

• Create route tables to direct backend VM traffic to the Azure Firewall.
• Ensure the route tables are associated with the correct subnets (e.g., backend subnet for the router VM).

Deploy and Configure the Router VM

• Deploy a Linux VM to act as a router.
• Enable IP forwarding on the VM and in Azure settings.
• Configure IPTables for NAT and traffic forwarding.
• Install and configure NGINX to serve as a health probe for the load balancer.

Configure Network Security Groups (NSGs)

• Set up NSGs to allow necessary traffic (SSH, load balancer, HTTP/HTTPS, health probe) to and from the router VM.

Configure Azure Firewall Application Rules

• Define application rules to allow outbound access only to approved FQDNs (e.g., microsoft.com).
• Block all other outbound traffic by default.

Configure Databricks Account Portal

• Enable outbound (serverless) Azure Private Link to customer-managed resources in the Databricks Account Portal.
• Create Network Connectivity Configurations (NCCs) and attach them to your workspaces.
• Add private endpoint rules for each Azure resource you want to allow access to.

Approve Private Endpoints

• In the Azure Portal, approve the private endpoint connections created by Databricks for your resources.

Troubleshooting

• Use tools like netstat, conntrack, and tcpdump on the router VM to diagnose routing issues.
• Double-check route table and NSG associations.
• Validate private endpoint rule configurations in both Databricks and the Azure Portal.

References

Serverless compute plane networking - Azure Databricks | Microsoft Learn

Configure private connectivity to Azure resources - Azure Databricks | Microsoft Learn

Key Takeaway

This solution enforces a deny-by-default posture for outbound traffic from Azure Databricks Serverless, only allowing explicitly approved connections via Private Endpoints. All traffic is routed through your network for inspection and control, helping you meet strict compliance and security requirements.

Ready to Get Started?

Securing your Databricks Serverless environment doesn’t have to be daunting. With Azure Private Link, Azure Firewall, and a smart configuration, you get the best of both worlds: agility and airtight security.

For more details, check out the official Azure Databricks documentation and start building your secure analytics platform today.

Questions or want to share your experience? Drop a comment below or reach out to the Azure Databricks community.