Navigating the Azure Databricks CSP Mandate

• Mandatory Deadline: The Compliance Security Profile (CSP) becomes mandatory for processing HIPAA, HITRUST, and IRAP regulated data on Azure Databricks by September 1, 2026. 
• Permanent Change: Enabling CSP is an irreversible, workspace-level configuration. Reversion requires deleting and recreating the workspace, emphasizing the need for meticulous pre-enablement testing.
• Architectural Impact: CSP introduces significant changes, notably requiring VNet encryption which can disrupt existing on-premises connections, and disabling partner-powered AI features by default. 

The evolving regulatory landscape demands rigorous security and compliance measures for data processing in the cloud. For organizations leveraging Azure Databricks, the introduction and upcoming mandatory enforcement of the Compliance Security Profile (CSP) represent a pivotal shift. This profile is not merely a feature toggle; it's a foundational architectural commitment designed to harden your Databricks environment against stringent compliance standards. Understanding its implications and preparing proactively is crucial to ensure uninterrupted operations and continued regulatory adherence. 

Why CSP Matters 

The Compliance Security Profile (CSP) is a robust, platform-hardening configuration applied at the workspace level within Azure Databricks. Its primary objective is to facilitate adherence to a broad spectrum of compliance standards by enforcing additional monitoring, utilizing hardened compute images, ensuring inter-node encryption with specific instance types, and implementing stricter controls across Databricks workspaces. This provides a secure baseline for the data plane, incorporating enhanced security monitoring capabilities. 

For organizations handling sensitive data, such as Protected Health Information (PHI) under HIPAA, or data subject to HITRUST and IRAP regulations, CSP is not just beneficial—it is becoming mandatory. Starting September 1, 2026, all Azure Databricks workspaces processing data under HIPAA, HITRUST, or IRAP will be required to have CSP enabled. Failure to comply will impede the ability to process such regulated workloads. 

Conceptual diagram illustrating data protection and complianceonDatabricks.

Key Compliance Standards Supported by CSP 

The compliance security profile supports or is required for processing data under a wide array of global and industry-specific compliance standards. This comprehensive coverage underscores its importance for organizations operating in highly regulated sectors: 

• C5 
• CCCS Medium (Protected B) 
• FedRAMP  
• HIPAA (Health Insurance Portability and Accountability Act) 
• HITRUST (Health Information Trust Alliance) 
• IRAP (Infosec Registered Assessors Program) 
• ISMAP 
• Korean Financial Security Institute (K-FSI) 
• PCI-DSS (Payment Card Industry Data Security Standard) 
• TISAX 
• UK Cyber Essentials Plus 

Understanding CSP's Architectural Implications 

One of the most critical aspects of CSP to grasp is its permanence. Once enabled on a workspace, or once any regulated data has been processed within a CSP-enabled workspace, the profile cannot be simply toggled off. To revert to a non-CSP state, you would need to delete the existing workspace and provision a new one without the CSP activated. This irreversible nature necessitates a thorough planning and validation phase before implementing CSP in production environments. 

Cost Considerations: The Enhanced Security & Compliance Add-on 

Enabling CSP automatically activates the "Enhanced Security and Compliance add-on" billing, which will incur additional costs. Organizations must factor these increased expenditures into their budgeting and financial planning. Reviewing the Azure Databricks pricing page is essential to understand the full financial impact. 

Networking Challenges: VNet Encryption and On-Premises Connectivity 

A significant "gotcha" highlighted in various sources is the impact of CSP on network architecture. CSP enforces stricter network security policies, often requiring VNet encryption. If your Azure Databricks workspaces utilize VNet Injection and connect to on-premises systems via ExpressRoute or VPN, this VNet encryption can potentially disrupt existing connections. This necessitates close collaboration with networking teams to verify compatibility, test connectivity in pre-production environments, and potentially reconfigure network components such as routing tables, security groups, or firewall rules. 

Feature Impact: AI Assist and Public Preview Limitations 

To maintain stringent compliance boundaries, CSP workspaces alter default behaviors and restrict certain features: 

• Partner-powered AI Features: By default, "Partner-powered AI features," which include some AI-powered coding assistants like Genie Code, are disabled in CSP workspaces. While Databricks-hosted models may power Genie Code in CSP workspaces for specific compliance standards (e.g., HIPAA, FedRAMP Moderate, PCI-DSS, HITRUST) in certain regions, organizations relying on these features should assess alternatives or plan for explicit enablement, understanding the potential constraints. 
• Preview Features: Only a limited set of Public Preview features are supported in CSP workspaces. Workloads dependent on unsupported preview features may face limitations or require adjustments. 

Preparing for the CSP Mandate 

To ensure a smooth transition to CSP, a detailed and phased approach is recommended. This involves thorough assessment, strategic planning, rigorous testing, and clear communication across various teams. 

The radar chart above visualizes the current readiness level (Pre-CSP Readiness) versus the desired target state (Post-CSP Target State) across critical preparation areas for the Compliance Security Profile. It underscores the significant effort required in areas like network reconfiguration and thorough testing to achieve full compliance and operational stability.

1. Inventory and Assessment
   • Identify Regulated Workspaces: Catalogue all existing Azure Databricks workspaces. Determine which ones currently process, or are planned to process, data subject to HIPAA, HITRUST, or IRAP.
   • Review Data Pipelines: Map out all data ingress and egress points for these identified workspaces, including connections to on-premises data sources, other cloud services, and external APIs. This helps identify potential network impacts. 

2. Network Architecture Review

• • Evaluate VNet Encryption: If your workspace uses VNet Injection and has private connectivity to an on-premises network (e.g., via ExpressRoute or VPN), verify that these connections will remain functional under CSP's new encryption requirements.
  • Plan for Reconfiguration: Allocate resources for potential network re-architecting, including updating routing tables, security groups, or firewall rules to accommodate encrypted traffic flows. 

3. Feature Impact Analysis

• • AI & Assist Features: Assess the reliance of your development teams on "Partner-powered AI features" like Genie Code. Determine if Databricks-hosted alternatives meet their needs or if explicit enablement (and its associated compliance considerations) is required.
  • Preview Features: Consult official documentation to check for limitations on Public Preview features in CSP workspaces and plan accordingly if your workloads depend on any unsupported features. 

4. Data Handling Responsibilities

• • Sensitive Information Fields: Implement policies to prevent sensitive data from being entered into customer-defined input fields (e.g., workspace names, compute resource names, tags, job names, URLs), as these might be stored or processed outside the compliance boundary.
  • Encryption at Rest: Verify that all data containing PHI or other regulated information is encrypted at rest in any storage location interacting with Databricks, including workspace storage accounts. 

5. Execution and Validation Plan

• • Staging Environment First: Always enable CSP on a clone of your production environment in a staging subscription before rolling out to production.
  • Thorough Testing: Conduct comprehensive end-to-end tests on the staging workspace to validate data pipelines, network connectivity, user workflows, and performance.
  • Enablement Method: Choose the appropriate tooling for enablement—Azure Portal, Azure CLI, PowerShell, ARM templates, or Terraform—to ensure consistency and automation. 

Detailed Enablement Methods 

Enabling CSP can be done for both new and existing workspaces using various Azure management tools. Regardless of the method chosen, it's crucial to select the applicable compliance standards during enablement. 

Enabling CSP on New Workspaces 

For new workspaces, the compliance security profile settings at the account level can control whether it's enabled by default. Account administrators can enable CSP individually during deployment using: 

• Azure Portal 
• Azure CLI (e.g., az databricks workspace create --enable-compliance-security-profile --compliance-standards '["HIPAA"]') 
• PowerShell 
• ARM templates 
• Terraform (using the databricks_compliance_security_profile_workspace_setting resource) 

Enabling CSP on Existing Workspaces 

Existing workspaces can also have CSP enabled through similar methods: 

• Azure Portal: Navigate to Workspace > Settings > Security & compliance > Enable compliance security profile; select standards and save. 
• Azure CLI / PowerShell / ARM templates / Terraform: Utilize these tools to modify the workspace settings and enable CSP, specifying the relevant compliance standards. 

Understanding Key Technical Considerations 

Beyond the general planning, specific technical details must be understood and addressed to ensure a successful CSP implementation. 

Account-Level Requirements 

Before enabling CSP, ensure your Databricks account meets these prerequisites: 

• Your Databricks account must include the Enhanced Security and Compliance add-on. 
• Your Databricks workspace needs to be on the Premium pricing tier. 

Security Configuration Table 

The table below summarizes critical security configuration aspects influenced by CSP: 

To further understand the broader context of security and compliance on Databricks, including how CSP fits into the larger ecosystem, watch this video from Databricks: Databricks Security & Compliance 

References 

• Compliance security profile: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/security-profile  
• Configure enhanced security and compliance settings: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/enhanced-security-compliance  
• HIPAA, Azure Databricks, Microsoft Learn: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/hipaa  
• What is Azure Virtual Network encryption: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-encryption-overview