As customers continue to deploy large scale data and AI projects in Azure Databricks, the complexity of deploying and managing these increases. To help you with this, we’re excited to announce the public preview of Serverless workspaces in Azure Databricks. With Serverless workspaces, creation of workspaces is extremely simplified. This means you can quickly create a workspace with serverless compute and default storage. Your teams can focus on gathering the insights from the data instead of managing workspace infrastructure.
What’s new?
Azure Databricks now has Serverless workspaces in addition to Classic workspaces. Serverless workspaces are managed and run by the Azure Databricks service. You can still use Unity Catalog, identity federation, and your workspace settings but without the effort to build and maintain the underlying infrastructure. This means your teams can get immediate access to the workspace preventing delays in spinning projects up.
On the contrary, when deploying Classic workspaces, you need to make all the decisions from how to set up the Virtual Network (VNet), deploy and manage compute, where data should land, and ensuring inbound and outbound connectivity to the workspace is properly set up. While essential, it is not something that an end user of the workspace should need to worry about.
What are Serverless Workspaces?
A Serverless Workspace is one where the overhead of deploying and managing the infrastructure is removed from the end user. The workspace is ready to use right away once created. Unity Catalog and storage are set up automatically. As a result, the same governance model is in tract without the set up overhead.
Why should I use Serverless Workspaces?
The main reason to use Serverless workspaces is to reduce the configuration and set up needed. These workspaces are managed for you, so these can be created almost instantaneously and given to your teams to start working on projects.
Serverless workspaces also have the following features you can benefit from features in the following areas:
- Storage: You can create managed catalogs, tables, and volumes without needing to bring your own Azure Blob storage account, provide storage credentials, or configure the locations up front since each Serverless Workspace provides fully managed object storage by default. Multi-key projection and no direct access to the object storage ensure only you can access your data.
- Compute: You can run workloads without provisioning a cluster. Azure Databricks automatically deploys and manages the needed serverless compute efficiently. This means end users can focus on analyzing the data without needing managing clusters.
- Network security: You no longer have to deploy NAT Gateways, Firewalls, or Private Link endpoints and instead define the serverless egress policies and serverless Private Link rules that then are applied to all serverless workloads in the workspace serverless egress policies
- Unity Catalog data estate: Because all of your existing governed data is available in the new workspace existing permissions allow access.
- Bring your own data when needed: Similar to Classic Workspaces, you can connect to your existing Azure Blob Storage account through Unity Catalog credentials and external locations. As a result, lineage and permissions are consistent while avoiding duplication
- Cost management: You can set attribution tags via budget policies on workloads to analyze spend without needing end users to tag jobs continually.
When should I use Serverless Workspaces?
Azure Databricks supports both –Serverless and Classic workspaces.
You can choose Serverless when time to get going on your project matters since it is the fastest way to get a governed environment set up with minimal configuration and management.
You can choose Classic when you need your own custom VNet design, specific network patterns, or other functionality not supported by serverless. Your organization also may prefer to continue to manage the underlying Azure resources directly, in which case Classic workspaces are a good choice.
Things to note
- Region availability: Serverless workspaces are only available in all Azure Databricks regions that support serverless compute.
- Feature surface: A serverless workspace inherits serverless compute constraints (e.g., Python/SQL focus, unsupported legacy APIs). Validate critical workloads before migrating your workspace.
- Billing for default storage is not yet enabled in Azure Databricks serverless workspaces. During this time, Azure Databricks will not charge for default storage use in serverless workspaces. Azure will notify customers 30 days before we enable billing for default storage usage
For other considerations on Serverless Workspaces, see Azure documentation
In summary, with Azure Databricks Serverless Workspaces, you can:
- Create new Azure Databricks workspaces quickly without manual set up
- Ensure consistent data governance and security right from set up
- Support your analytics and AI projects with clear cost visibility and built-in budget guardrails
You can learn how to create an Azure Databricks serverless workspace and get started today here.
Microsoft