Microsoft Architect's: Aladdin Alchalabi AladdinAlchalabi, Kiran Raja KiranRaja, Peter Lenges PeterLenges, Jessica Reece jareece, Benjamin Coughtry bcoughtry, Anishek Kamal anishekkamal, Tayo Akigbogun takigbogun, Eric Kwashie ekwashie, Peter Lo PeterLo and Rafia Aqil Rafia_Aqil
Peer Reviewed: Ted Kim tedkim and Arvind Periyasamy ArvindPeriyasamy
Purpose and WHY
Azure Databricks has put in place controls to meet the unique compliance needs of highly regulated industries. The requirement for the compliance security profile (CSP) is a joint effort between Microsoft and Databricks for Azure-Databricks workspaces. The value proposition of the compliance security profile is that it provides Customers significantly more hardening and security features.
- Mandatory Deadline: The Compliance Security Profile (CSP) becomes mandatory for processing HIPAA, HITRUST, and IRAP regulated data on Azure-Databricks by September 1, 2026.
- Key Dates: Enable the Compliance Security Profile and select HIPAA by September 1, 2026. Prepare for Azure Virtual Network encryption enforcement beginning June 1, 2027.
Compliance Responsibility: Enabling CSP supports applicable technical controls but does not, by itself, establish HIPAA compliance. Compliance is a shared responsibility among the customer, Microsoft, and Databricks. Customers must evaluate their administrative, physical, and technical safeguards and confirm that an applicable Microsoft Business Associate Agreement is in place.
Enabling CSP on Workspaces
These requirements are checked and enforced on new workspaces today, with enforcement on existing workspaces expected in the future; where prerequisites are missing, clusters may fail to start.
|
Prerequisite |
Requirement |
|
Costs |
There is a 10% cost of the Azure Databricks product spend within each workspace where CSP is enabled. **Review with your account team for any grace period during which the Enhanced Security & Compliance (ESC) add-on is available at no charge. After the grace period ends, a 10% DBU upcharge applies. |
|
Enhanced Security & Compliance add-on |
For existing workspaces: From Azure portal, click the Settings > Security & compliance on an existing Azure Databricks workspace: **Review Note #2 below |
|
Azure VNet encryption |
Azure Virtual Network encryption must be enabled on the Azure Databricks workspace VNet.
**Review Note #4 below |
|
Supported VM instance types |
Use a VM series that supports VNet encryption and verify compatibility before enabling the profile. **This does not apply to serverless compute. |
NOTE:
- Confirm your workspace is using Premium Pricing tier.
- The profile can be enabled when a workspace is created or on an existing workspace, through the Azure portal, the Azure CLI, PowerShell, an ARM template, or Terraform.
- Only the Public Preview, Private Preview, and Beta features listed in this section are supported for workspaces with the compliance security profile enabled: Compliance security profile - Azure Databricks | Microsoft Learn
-
Currently, the compliance security profile checks and enforces only the use of specific VM instance types, not the enablement of Azure Virtual Network encryption. Enforcement of the Azure Virtual Network encryption requirement begins on June 1, 2027, including on workspaces that already have the compliance security profile enabled. This flexibility shall allow customers more time to set up VNET encryption. This has been updated in documentation today (See ‘Important’ box).
- Regarding rollback, CSP can be reversed via a support ticket, if no regulated data has been processed on a particular workspace.
- Plan for possible effects on cluster startup, networking, feature availability, maintenance operations, and cost.
A closer look at VNet encryption
CSP is enabled per Databricks workspace, but VNet encryption is applied at the VNet level. Enabling it for a Databricks workload therefore affects every resource within that VNet, not just the workspace. A common approach in a hub-and-spoke design is to leave the hub VNet unencrypted and encrypt only the spoke VNet. The hub typically holds shared services such as the DNS resolver, while the spoke hosts the Databricks workspaces that require CSP.
What does it mean for a VNet to be encrypted?
An encrypted VNet is a security measure that protects VM-to-VM traffic. Data is encrypted in transit through a DTLS tunnel. This is platform-level encryption, applied automatically to traffic within your VNet and across peered VNets. It requires no changes to your operating system or applications.
What happens to my VM-to-VM traffic?
Qualifying VM-to-VM traffic is encrypted. Traffic involving unqualified instances simply keeps flowing unencrypted. The only enforcement available today is AllowUnencrypted.
Important clarifications
-
- Encrypting the VNet does not guarantee all traffic within it will be encrypted. The only traffic that gets encrypted is VM-to-VM traffic where both the source and destination VMs are (1) on a supported SKU and (2) have Accelerated Networking enabled on the network interface.
- Encrypting the VNet does not drop or break traffic from unsupported SKUs. The only supported GA setting today is to allow unencrypted traffic, so non-qualifying traffic is still permitted; it just isn't encrypted. A future DropUnencrypted setting will drop that traffic instead for further hardening. It isn't available yet, and it's currently unknown whether it will become a required setting for CSP.
Review the following recommended steps
The steps below represent a validated implementation pattern. The exact network design can vary by environment, but the same prerequisite, isolation, and end-to-end validation principles should be applied.
|
Validated implementation step |
Recommended approach and expected outcome |
|
Isolated sandbox workspace |
Enable CSP first in a representative non-production workspace. This avoids irreversible changes to DEV or production while the network topology, dependencies, VM compatibility, and operational behavior are validated. |
|
Enable CSP and select HIPAA |
Enable the Compliance Security Profile and select HIPAA under Settings > Security & compliance before processing PHI after September 1, 2026. |
|
Enable VNet encryption |
Enable VNet encryption. **Review Azure Virtual Network encryption limitations: What is Azure Virtual Network encryption? - Azure Virtual Network | Microsoft Learn |
|
Start a classic cluster |
Confirm that a classic cluster starts successfully after CSP and VNet encryption prerequisites are applied. This validates that the selected compute path and VM types remain operational. |
|
Validate storage connectivity |
Confirm storage connectivity continue to work. |
|
Confirm rollout readiness |
Proceed to DEV and production only after the complete private connectivity path, cluster startup, storage access, DNS resolution, data pipelines, and performance have been validated from end to end. |
Things to Review
Enablement is permanent
Enabling the compliance security profile, or adding a compliance standard, is intended to be a permanent change. You cannot remove the profile or an individual standard from a workspace that has ever processed regulated data; to revert, you must delete the workspace and create a new one. Validate the configuration in an isolated, representative non-production workspace before enabling DEV or production.
Inventory and Assessment
-
-
- Identify Regulated Workspaces: Catalogue all existing Azure-Databricks workspaces. Determine which ones currently process, or are planned to process, data subject to HIPAA, HITRUST, or IRAP.
-
-
-
- Review Data Pipelines: Map out all data ingress and egress points for these identified workspaces, including connections to on-premises data sources, other cloud services, and external APIs. This helps identify potential network impacts.
-
Verify Prerequisites Before Rollout: Confirm that selected VM instance types support VNet encryption and that every required CSP and networking setting is in place, because missing prerequisites can prevent clusters from starting.
-
-
- Enablement Method: Choose the appropriate tooling for enablement of Azure Portal, Azure CLI, PowerShell, ARM templates, or Terraform to ensure consistency and automation.
-
Keep sensitive data out of customer-defined fields
You are solely responsible for ensuring that PHI or other sensitive information is never entered into customer-defined input fields. These include workspace names, compute and resource names, tags, job names, job run names, network names, credential names, storage account names, and Git repository IDs or URLs, all of which may be stored, processed, or accessed outside the compliance boundary.
What Changes After Enabling Compliance Security Profile
On CSP-enabled workspaces, Partner-powered AI features are disabled by default and some assistive features such as Genie Code are also disabled; a workspace admin can re-enable them if required. In addition, only the specific preview features listed in the compliance security profile documentation are supported. No other Public Preview, Private Preview, or Beta feature may be used to process regulated data.
Compliance Security Profile (CSP) enhances the security posture of Azure Databricks by enabling a hardened compute image, enhanced security monitoring, and automatic cluster updates. With automatic cluster updates enabled, classic compute resources are periodically updated and may restart during configured maintenance windows, so production schedules should be planned accordingly. Enhanced security monitoring deploys security monitoring agents on supported compute resources and generates logs that security teams can ingest and analyze. When deploying through ARM templates, CSP, enhancedSecurityMonitoring, and automaticClusterUpdate are configurable security and compliance settings that can be specified as part of the workspace deployment.
Why GPU-accelerated compute may be affected
Azure Databricks supports several GPU families, but Azure VNet encryption currently documents a narrower GPU list. Please review the list here: What is Azure Virtual Network encryption? - Azure Virtual Network | Microsoft Learn.
Note: Please review the important section for any GPU The use of GPU-based VMs that support VNet encryption is required but not yet enforced.
What happens if a customer does not enable Azure Databricks CSP?
Azure Databricks has implemented additional controls to support the security and compliance requirements of highly regulated industries. CSP should not be characterized solely as a Microsoft initiative; it is part of the Azure Databricks security and compliance offering delivered by Microsoft and Databricks. CSP provides additional platform hardening and security capabilities, including a CIS Level 1 hardened compute image, automatic cluster updates, enhanced security monitoring, and TLS 1.2 or higher for relevant communications.
Beginning September 1, 2026, CSP is required for Azure Databricks workspaces processing data subject to applicable standards, including HIPAA, HITRUST, and IRAP. If a customer chooses not to enable CSP while processing data subject to one of these standards, the workspace would not meet the documented Azure Databricks configuration requirements for that regulated workload. This does not, by itself, determine the customer's overall legal or regulatory compliance; customers should assess their obligations with their legal, compliance, and audit teams.
References
- Compliance security profile: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/security-profile
- Configure enhanced security and compliance settings: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/enhanced-security-compliance
- HIPAA, Azure Databricks, Microsoft Learn: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/hipaa
- What is Azure Virtual Network encryption: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-encryption-overview
- Create a Virtual Network with encryption: https://learn.microsoft.com/en-us/azure/virtual-network/how-to-create-encryption?tabs
- Hashicorp azurerm_virtual_network: azurerm_virtual_network | Resources | hashicorp/azurerm | Terraform | Terraform Registry