Azure Databricks articles

Introducing Lakeflow Connect Free Tier, now available on Azure Databricks

AnaviNahar — Wed, 18 Mar 2026 16:00:00 GMT

We're excited to introduce the Lakeflow Connect Free Tier on Azure Databricks, so you can easily bring your enterprise data into your lakehouse to build analytics and AI applications faster.

Modern applications require reliable access to operational data, especially for training analytics and AI agents, but connecting and gathering data across silos can be challenging. With this new release, you can seamlessly ingest all of your enterprise data from SaaS and database sources to unlock data intelligence for your AI agents.

Ingest millions of records per day, per workspace for free

This new Lakeflow Connect Free Tier provides 100 DBUs per day, per workspace, which allows you to ingest approximately 100 million records* from many popular data sources**, including SaaS applications and databases.

Unlock your enterprise data for free with Lakeflow Connect

This new offering provides all the benefits of Lakeflow Connect, eliminating the heavy lifting so your teams can focus on unlocking data insights instead of managing infrastructure.

In the past year, Databricks has continued rolling out several fully managed connectors, supporting popular data sources. The free tier supports popular SaaS applications (Salesforce, ServiceNow, Google Analytics, Workday, Microsoft Dynamics 365), and top-used databases (SQL Server, Oracle, Teradata, PostgreSQL, MySQL, Snowflake, Redshift, Synapse, and BigQuery).

Lakeflow Connect benefits include:

Simple UI: Avoid complex setups and architectural overhead, these fully managed connectors provide a simple UI and API to democratize data access. Automated features also help simplify pipeline maintenance with minimal overhead.
Efficient ingestion: Increase efficiency and accelerate time to value. Optimized incremental reads and writes and data transformation help improve the performance and reliability of your pipelines, reduce bottlenecks, and reduce impact to the source data for scalability.
Unified with the Databricks Platform: Create ingestion pipelines with governance from Unity Catalog, observability from Lakehouse Monitoring, and seamless orchestration with Lakeflow Jobs for analytics, AI and BI.

Availability

The Lakeflow Connect Free Tier is available starting today on Azure Databricks.

If you are at FabCon in Atlanta, Accelerating Data and AI with Azure Databricks on Thursday, March 19th, 8:00–9:00 AM, room C302 to see how these capabilities come together to accelerate performance, simplify architecture, and maximize value on Azure

Getting Started Resources

To learn more about the Lakeflow Connect Free Tier and Lakeflow Connect, review our pricing page, and documentation. Get started ingesting your data today for free, signup with an Azure free account.

Get started with Azure Databricks for free
Product tour: Databricks Lakeflow Connect for Salesforce: Powering Smarter Selling with AI and Analytics
Product tour: Effortless ServiceNow Data Ingestion with Databricks Lakeflow Connect
Product tour: Simplify Data Ingestion with Lakeflow Connect: From Google Analytics to AI
On-demand video: Use Lakeflow Connect for Salesforce to predict customer churn
On-demand video: Databricks Lakeflow Connect for Workday Reports: Connect, Ingest, and Analyze Workday Data Without Complexity
On-demand video: Data Ingestion With Lakeflow Connect

—--

* Your actual ingestion capacity will vary based on specific workload characteristics, record sizes, and source types.

** Excludes Zerobus Ingest, Auto Loader and other self-managed connectors. Customer will continue to incur charges for underlying infrastructure consumption from the cloud vendor.

Near–Real-Time CDC to Delta Lake for BI and ML with Lakeflow on Azure Databricks

AnaviNahar — Wed, 18 Mar 2026 16:00:00 GMT

The Challenge: Too Many Tools, Not Enough Clarity

Modern data teams on Azure often stitch together separate orchestrators, custom streaming consumers, hand-rolled transformation notebooks, and third-party connectors — each with its own monitoring UI, credential system, and failure modes. The result is observability gaps, weeks of work per new data source, disconnected lineage, and governance bolted on as an afterthought.

Lakeflow, Databricks’ unified data engineering solution, solves this by consolidating ingestion, transformation, and orchestration natively inside Azure Databricks — governed end-to-end by Unity Catalog.

Component	What It Does
Lakeflow Connect	Point-and-click connectors for databases (using CDC), SaaS apps, files, streaming, and Zerobus for direct telemetry
Lakeflow Spark Declarative Pipelines	Declarative ETL with AutoCDC, data quality enforcement, and automatic incremental processing
Lakeflow Jobs	Managed orchestration with 99.95% uptime, a visual task DAG, and repair-and-rerun

Architecture

Step 1: Stream Application Telemetry with Zerobus Ingest

Zerobus Ingest, part of Lakeflow Connect, lets your application push events directly to a Delta table over gRPC — no message bus, no Structured Streaming job. Sub-5-second latency, up to 100 MB/sec per connection, immediately queryable in Unity Catalog.

Prerequisites

Azure Databricks workspace with Unity Catalog enabled and serverless compute on
A service principal with write access to the target table

Setup

First, create the target table in a SQL notebook:

CREATE CATALOG IF NOT EXISTS prod; 
CREATE SCHEMA IF NOT EXISTS prod.bronze; 
CREATE TABLE IF NOT EXISTS prod.bronze.telemetry_events ( 
  event_id    STRING, 
  user_id     STRING, 
  event_type  STRING, 
  session_id  STRING, 
  ts          BIGINT, 
  page        STRING, 
  duration_ms INT 
);

1. Go to Settings → Identity and Access → Service Principals → Add service principal

2. Open the service principal → Secrets tab → Generate secret. Save the Client ID and secret.

3. In a SQL notebook, grant access:

GRANT USE CATALOG ON CATALOG prod TO `<client-id>`; 
GRANT USE SCHEMA ON SCHEMA prod.bronze TO `<client-id>`; 
GRANT MODIFY, SELECT ON TABLE prod.bronze.telemetry_events TO `<client-id>`;

4. Derive your Zerobus endpoint from your workspace URL: <workspace-id>.zerobus.<region>.azuredatabricks.net (The workspace ID is the number in your workspace URL, e.g. adb-**1234567890**.12.azuredatabricks.net)

5. Install the SDK: pip install databricks-zerobus-ingest-sdk

6. In your application, open a stream and push records:

from zerobus.sdk.sync import ZerobusSdk 
from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties 
sdk    = ZerobusSdk("<workspace-id>.zerobus.<region>.azuredatabricks.net", 
                    "https://<workspace-url>") 
stream = sdk.create_stream( 
    "<client-id>", "<client-secret>", 
    TableProperties("prod.bronze.telemetry_events"), 
    StreamConfigurationOptions(record_type=RecordType.JSON) 
) 
stream.ingest_record({"event_id": "e1", "user_id": "u42", 
                      "event_type": "page_view", "ts": 1700000000000}) 
stream.close()

7. Verify in Catalog → prod → bronze → telemetry_events → Sample Data

Step 2: Ingest from On-Premises SQL Server via CDC

Lakeflow Connect reads SQL Server's transaction log incrementally — no full table scans, no custom extraction software. Connectivity to your on-prem server is over Azure ExpressRoute.

Prerequisites

SQL Server reachable from Databricks over ExpressRoute (TCP port 1433)
CDC enabled on the source database and tables (see setup below)
A SQL login with CDC read permissions on the source database
Databricks: CREATE CONNECTION privilege on the metastore; USE CATALOG, CREATE TABLE on the destination catalog

Setup

Enable CDC on SQL Server: 
USE YourDatabase; 
EXEC sys.sp_cdc_enable_db; 
EXEC sys.sp_cdc_enable_table 
  @source_schema = N'dbo', @source_name = N'orders',    @role_name = NULL; 
EXEC sys.sp_cdc_enable_table 
  @source_schema = N'dbo', @source_name = N'customers', @role_name = NULL;

Configure the connector in Databricks:

Click Data Ingestion in the sidebar (or + New → Add Data)
Select SQL Server from the connector list
Ingestion Gateway page — enter a gateway name, select staging catalog/schema, click Next
Ingestion Pipeline page — name the pipeline, click Create connection:
Host: your on-prem IP (e.g. 10.0.1.50) · Port: 1433 · Database: YourDatabase
Enter credentials, click Create, then Create pipeline and continue
Source page — expand the database tree, check dbo.orders and dbo.customers; optionally enable History tracking (SCD Type 2) per table. Set Destination table name to orders_raw and customers_raw respectively.
Destination page — set catalog: prod, schema: bronze, click Save and continue
Settings page — set a sync schedule (e.g. every 5 minutes), click Save and run pipeline

Step 3: Transform with Spark Declarative Pipelines

The Lakeflow Pipelines Editor is an IDE built for developing pipelines in Lakeflow Spark Declarative Pipelines (SDP), and lets you define Bronze → Silver → Gold in SQL. SDP then handles incremental execution, schema evolution, and lineage automatically.

Prerequisites

Bronze tables populated (from Steps 1 and 2)
CREATE TABLE and USE SCHEMA privileges on prod.silver and prod.gold

Setup

1. In the sidebar, click Jobs & Pipelines → ETL pipeline → Start with an empty file → SQL

2. Rename the pipeline (click the name at top) to lakeflow-demo-pipeline

3. Paste the following SQL:

-- Silver: latest order state (SCD Type 1) 
CREATE OR REFRESH STREAMING TABLE prod.silver.orders; 
APPLY CHANGES INTO prod.silver.orders 
  FROM STREAM(prod.bronze.orders_raw) 
  KEYS (order_id) SEQUENCE BY updated_at STORED AS SCD TYPE 1; 

-- Silver: full customer history (SCD Type 2) 
CREATE OR REFRESH STREAMING TABLE prod.silver.customers; 
APPLY CHANGES INTO prod.silver.customers 
  FROM STREAM(prod.bronze.customers_raw) 
  KEYS (customer_id) SEQUENCE BY updated_at STORED AS SCD TYPE 2; 

-- Silver: telemetry with data quality check 
CREATE OR REFRESH STREAMING TABLE prod.silver.telemetry_events ( 
  CONSTRAINT valid_event_type 
    EXPECT (event_type IN ('page_view', 'add_to_cart', 'purchase')) 
    ON VIOLATION DROP ROW 
) AS SELECT * FROM STREAM(prod.bronze.telemetry_events); 

-- Gold: materialized view joining all three Silver tables 
CREATE OR REFRESH MATERIALIZED VIEW prod.gold.customer_activity AS 
SELECT 
  o.order_id, o.customer_id, c.customer_name, c.email, 
  o.order_amount, o.order_status, 
  COUNT(e.event_id) AS total_events, 
  SUM(CASE WHEN e.event_type = 'purchase' THEN 1 ELSE 0 END) AS purchase_events 
FROM prod.silver.orders o 
LEFT JOIN prod.silver.customers c ON o.customer_id = c.customer_id 
LEFT JOIN prod.silver.telemetry_events e 
  ON CAST(o.customer_id AS STRING) = e.user_id  -- user_id in telemetry is string 
GROUP BY o.order_id, o.customer_id, c.customer_name, c.email, 
  o.order_amount, o.order_status;

4. Click Settings (gear icon) → set Pipeline mode: Continuous → Target catalog: prod → Save

5. Click Start — the editor switches to the live Graph view

Step 4: Govern with Unity Catalog

All tables from Steps 1–3 are automatically registered in Unity Catalog, Databricks’ built-in governance and security offering, with full lineage. No manual registration needed.

View lineage

Go to Catalog → prod → gold → customer_activity
Click the Lineage tab → See Lineage Graph
Click the expand icon on each upstream node to reveal the full chain: Bronze sources → Silver → Gold

Set Permissions


-- Grant analysts read access to the Gold layer only 
GRANT SELECT ON TABLE prod.gold.customer_activity TO `analysts@contoso.com`; 
-- Mask PII for non-privileged users 
CREATE FUNCTION prod.security.mask_email(email STRING) 
  RETURNS STRING 
  RETURN CASE WHEN is_account_group_member('data-engineers') THEN email 
              ELSE CONCAT(LEFT(email, 2), '***@***.com') END; 
ALTER TABLE prod.silver.customers 
  ALTER COLUMN email SET MASK prod.security.mask_email;

Step 5: Orchestrate and Monitor with Lakeflow Jobs

Wire the Connect pipeline and SDP pipeline into a single job with dependencies, scheduling, and alerting, all from the UI with Lakeflow Jobs.

Prerequisites

Pipelines from Steps 2 and 3 saved in the workspace

Setup

Go to Jobs & Pipelines → Create → Job
Task 1: click the Pipeline tile → name it ingest_sql_server_cdc → select your Lakeflow Connect pipeline → Create task
Task 2: click + Add task → Pipeline → name it transform_bronze_to_gold → select lakeflow-demo-pipeline → set Depends on: ingest_sql_server_cdc → Create task
In the Job details panel on the right: click Add schedule → set frequency → add email notification on failure → Save
Click Run now to trigger a run, then click the run ID to open the Run detail view

For health monitoring across all jobs, query system tables in any notebook or SQL warehouse:

SELECT job_name, result_state, DATEDIFF(second, start_time, end_time) AS duration_sec

FROM system.lakeflow.job_run_timeline

WHERE start_time >= CURRENT_TIMESTAMP - INTERVAL 24 HOURS

ORDER BY start_time DESC;

Step 6: Visualize with AI/BI Dashboards and Genie

AI/BI Dashboard helps you create AI-powered, low-code dashboards.

Click + New → Dashboard
Click Add a visualization, connect to prod.gold.customer_activity, and build charts
Click Publish — viewers see data under their own Unity Catalog permissions automatically

Genie allows you to interact with their data using natural language

1. In the sidebar, click Genie → New

2. On Choose data sources, select prod.gold.customer_activity → Create

3. Add context in the Instructions box (e.g., table relationships, business definitions)

4. Switch to the Chat tab and ask a question:

"Which customers have the highest total events and what were their order amounts?"

5. Genie generates and executes SQL, returning a result table. Click View SQL to inspect the query.

Everything in One Platform

Capability	Lakeflow	Previously Required
Telemetry ingestion	Zerobus Ingest	Message bus + custom consumer
Database CDC	Lakeflow Connect	Custom scripts or 3rd-party tools
Transformation + AutoCDC	Spark Declarative Pipelines	Hand-rolled MERGE logic
Data quality	SDP Expectations	Separate validation tooling
Orchestration	Lakeflow Jobs	External schedulers (Airflow, etc.)
Governance	Unity Catalog	Disconnected ACLs and lineage
Monitoring	Job UI + System Tables	Separate APM tools
BI + NL Query	AI/BI Dashboards + Genie	External BI tools

Customers seeing results on Azure Databricks:

Ahold Delhaize — 4.5x faster deployment and 50% cost reduction running 1,000+ ingestion jobs daily
Porsche Holding — 85% faster ingestion pipeline development vs. a custom-built solution

Next Steps

Azure Databricks Lakebase is now generally available

AnaviNahar — Tue, 03 Mar 2026 03:34:11 GMT

Modern applications are built on real-time, intelligent, and increasingly powered by AI agents that need fast, reliable access to operational data—without sacrificing governance, scale, or simplicity.

To solve for this, Azure Databricks Lakebase introduces a serverless, Postgres database architecture that separates compute from storage and integrates natively with the Databricks Data Intelligence Platform on Azure. Lakebase is now generally available in Azure Databricks enabling you and your team to start building and validating real-time and AI-driven applications directly on your lakehouse foundation.

Why Azure Databricks Lakebase?

Lakebase was created for modern workloads and reduce silos. By decoupling compute from storage, Lakebase treats infrastructure as an on-demand service—scaling automatically with workload needs and scaling to zero when idle.

Key capabilities include:

Serverless Postgres for Production Workloads: Lakebase delivers a managed Postgres experience with predictable performance and built-in reliability features suitable for production applications, while abstracting away infrastructure management.
Instant Branching and Point-in-Time Recovery: Teams can create zero-copy branches of production data in seconds for testing, debugging, or experimentation, and restore databases to precise points in time to recover from errors or incidents.
Unified Governance with Unity Catalog: Operational data in Lakebase can be governed using the same Unity Catalog policies that secure analytics and AI workloads, enabling consistent access control, auditing, and compliance across the platform.
Built for AI and Real-Time Applications: Lakebase is designed to support AI-native patterns such as real-time feature serving, agent memory, and low-latency application state—while keeping data directly connected to the lakehouse for analytics and learning workflows.
Lakebase allows applications to operate directly on governed, lake-backed data—reducing complexity with pipeline synchronization or duplicating storage

On Azure Databricks, this unlocks new scenarios such as:

Real-time applications built on lakehouse data
AI agents with persistent, governed memory
Faster release cycles with safe, isolated database branches
Simplified architectures with fewer moving parts

All while using familiar Postgres interfaces and tools.

Get Started with Azure Databricks Lakebase

Lakebase is integrated into the Azure Databricks experience and can be provisioned directly within Azure Databricks workspaces. For Azure Databricks customers building intelligent, real-time applications, it offers a new foundation—one designed for the pace and complexity of modern data-driven systems. We’re excited to see what you build, get started today!

Serverless Workspaces are generally available in Azure Databricks

AnaviNahar — Mon, 02 Feb 2026 17:15:00 GMT

Recently, we announced that Serverless Workspaces in public preview. Today, we are excited to share that Serverless Workspaces are generally available in Azure Databricks. Azure Databricks now offers two workspace models: Serverless and Classic.

With Serverless, Azure Databricks operates and maintains the entire environment on your behalf. You still configure governance elements like Unity Catalog, identity federation, and workspace-level policies, but the heavy lifting of infrastructure setup disappears. As a result, teams can start building immediately instead of waiting on networking or compute provisioning.

Classic workspaces take the opposite approach. When creating a Classic workspace, you must design and deploy the full networking layout, determine how compute should be managed, establish storage patterns, and configure all inbound and outbound connectivity. These decisions are critical and have benefits in heavily regulated or secure industries, but they may create overhead for teams who simply want to start working with data.

A Serverless Workspace eliminates that overhead entirely. Once created, it’s ready for use—no virtual network design, no storage configuration, and no cluster management. Serverless workspaces use serverless compute and default storage. Unity Catalog is automatically provisioned so that the same governance model applies.

Key capabilities and consideration of Serverless workspaces

Storage:
Each Serverless workspace includes fully managed object storage called default storage. You can build managed catalogs, volumes, and tables without supplying your own storage accounts or credentials. Features like multi-key projection and restricted object-store access ensure that only authorized users can work with the data. Classic compute cannot interact with data assets in default storage.

If you already have an Azure Blob Storage account (likely with hierarchical namespace enabled) or if your organization requires using your own storage due to security or compliance requirements, you can also create a connection between your storage account and Serverless Workspace.

Compute:
Workloads run on automatically provisioned serverless compute—no need to build or maintain clusters. Azure Databricks handles scaling and resource optimization so users can focus purely on data and analytics.

Network:
There’s no requirement to deploy NAT gateways, firewalls, or Private Link endpoints. Instead, you define serverless egress rules and serverless Private Link controls that apply uniformly to all workloads in the workspace.

Unity Catalog access:
Governed data remains accessible from the new workspace with existing permissions intact. Your data estate stays consistent and secure.

Choosing between Serverless and Classic

Azure Databricks supports both workspace types so organizations can select what best matches their needs.

Use Serverless when rapid workspace creation, minimal configuration, and reduced operational overhead are priorities. It’s the fastest path to a fully governed environment.

Use Classic when you require a custom VNet design, specific network topologies, finer grain security controls, or features that are not yet available in the serverless model. Some organizations also simply prefer to manage Azure resources directly, making Classic workspaces a suitable option.

Note that Azure Databricks serverless workspaces are only available in regions that support serverless compute. To learn get started create your Serverless workspace today!

Serverless Workspaces are live in Azure Databricks

AnaviNahar — Tue, 02 Dec 2025 23:08:45 GMT

As customers continue to deploy large scale data and AI projects in Azure Databricks, the complexity of deploying and managing these increases. To help you with this, we’re excited to announce the public preview of Serverless workspaces in Azure Databricks. With Serverless workspaces, creation of workspaces is extremely simplified. This means you can quickly create a workspace with serverless compute and default storage. Your teams can focus on gathering the insights from the data instead of managing workspace infrastructure.

What’s new?

Azure Databricks now has Serverless workspaces in addition to Classic workspaces. Serverless workspaces are managed and run by the Azure Databricks service. You can still use Unity Catalog, identity federation, and your workspace settings but without the effort to build and maintain the underlying infrastructure. This means your teams can get immediate access to the workspace preventing delays in spinning projects up.

On the contrary, when deploying Classic workspaces, you need to make all the decisions from how to set up the Virtual Network (VNet), deploy and manage compute, where data should land, and ensuring inbound and outbound connectivity to the workspace is properly set up. While essential, it is not something that an end user of the workspace should need to worry about.

What are Serverless Workspaces?

A Serverless Workspace is one where the overhead of deploying and managing the infrastructure is removed from the end user. The workspace is ready to use right away once created. Unity Catalog and storage are set up automatically. As a result, the same governance model is in tract without the set up overhead.

Why should I use Serverless Workspaces?

The main reason to use Serverless workspaces is to reduce the configuration and set up needed. These workspaces are managed for you, so these can be created almost instantaneously and given to your teams to start working on projects.

Serverless workspaces also have the following features you can benefit from features in the following areas:

Storage: You can create managed catalogs, tables, and volumes without needing to bring your own Azure Blob storage account, provide storage credentials, or configure the locations up front since each Serverless Workspace provides fully managed object storage by default. Multi-key projection and no direct access to the object storage ensure only you can access your data.
Compute: You can run workloads without provisioning a cluster. Azure Databricks automatically deploys and manages the needed serverless compute efficiently. This means end users can focus on analyzing the data without needing managing clusters.
Network security: You no longer have to deploy NAT Gateways, Firewalls, or Private Link endpoints and instead define the serverless egress policies and serverless Private Link rules that then are applied to all serverless workloads in the workspace serverless egress policies
Unity Catalog data estate: Because all of your existing governed data is available in the new workspace existing permissions allow access.
Bring your own data when needed: Similar to Classic Workspaces, you can connect to your existing Azure Blob Storage account through Unity Catalog credentials and external locations. As a result, lineage and permissions are consistent while avoiding duplication
Cost management: You can set attribution tags via budget policies on workloads to analyze spend without needing end users to tag jobs continually.

When should I use Serverless Workspaces?

Azure Databricks supports both –Serverless and Classic workspaces.

You can choose Serverless when time to get going on your project matters since it is the fastest way to get a governed environment set up with minimal configuration and management.

You can choose Classic when you need your own custom VNet design, specific network patterns, or other functionality not supported by serverless. Your organization also may prefer to continue to manage the underlying Azure resources directly, in which case Classic workspaces are a good choice.

Things to note

Region availability: Serverless workspaces are only available in all Azure Databricks regions that support serverless compute.
Feature surface: A serverless workspace inherits serverless compute constraints (e.g., Python/SQL focus, unsupported legacy APIs). Validate critical workloads before migrating your workspace.
Billing for default storage is not yet enabled in Azure Databricks serverless workspaces. During this time, Azure Databricks will not charge for default storage use in serverless workspaces. Azure will notify customers 30 days before we enable billing for default storage usage

For other considerations on Serverless Workspaces, see Azure documentation

In summary, with Azure Databricks Serverless Workspaces, you can:

Create new Azure Databricks workspaces quickly without manual set up
Ensure consistent data governance and security right from set up
Support your analytics and AI projects with clear cost visibility and built-in budget guardrails

You can learn how to create an Azure Databricks serverless workspace and get started today here.

Azure Databricks Genie integration with Copilot Studio and Microsoft Foundry is now live!

AnaviNahar — Tue, 18 Nov 2025 18:48:18 GMT

This blog was co-authored by Toussaint Webb, Databricks

We’re excited to announce the Public Preview availability of AI/BI Genie in Microsoft Copilot Studio and Microsoft Foundry via MCP. This makes it easier than ever for organizations to unlock and scale the power of their Genie spaces across the Microsoft ecosystem (e.g., Teams), ultimately democratizing trusted data and insights to business users.

AI/BI Genie opens the power of conversational analytics to everyone in the organization. A user can ask a question such as “What is my revenue growth this month?” and Genie interprets the intent, generates the appropriate query, and returns the data insight. Users can also review the underlying logic for transparency. By supporting iterative questioning, Genie enables users to investigate their data directly and build confidence in their understanding without requiring code or specialist intervention.

The Challenge Before: Complex setup via Custom Code

Previously, connecting Genie to the Microsoft ecosystem was challenging. Organizations had to develop custom connections to manage API flows, which added architectural overhead. This complexity limited organizations’ ability to distribute Genie’s trusted insights efficiently across Microsoft platforms.

What’s Now Possible: Unlock the value of Microsoft Ecosystem

The new integrations between Genie and Copilot Studio, as well as Genie and Microsoft Foundry, solve these challenges by providing easy and secure ways to connect each platform. Additionally, by leveraging MCP, updates to the underlying Genie APIs are seamlessly managed for users, eliminating the need to modify your integration.

Genie + Copilot Studio

Connect a Genie space to a Copilot Studio Agent as a tool (over MCP)
Instantly make Genie’s trusted insights available in Teams or M365 Copilot by publishing Genie enabled Copilot Studio agents
Genie’s complete context of a customer’s data estate will enable Copilot Studio agents to deliver richer, more accurate responses

Genie + Microsoft Foundry

Connect a Genie space to Foundry as a tool (over MCP)
Instantly make Genie’s trusted insights available to developers building agents using Foundry

How It Works: Genie + Copilot Studio

Create a connection to your Azure Databricks workspace in Power Apps, using OAuth or a Microsoft Entra ID Service Principal
Open Copilot Studio
Select an existing Copilot Studio agent or create a new one
Open ‘Tools’, click ‘+Add a tool’, and search for “Azure Databricks Genie” or find it within the Model Context Protocol section
Select the Genie space to connect and configure the connection

Note: It’s important to give your Genie space a clear title and description so the Copilot Studio agent can effectively orchestrate requests.

After completing the steps above, you are ready to go and can publish the agent to channels, such as Microsoft Teams, to easily distribute the value of Genie to your organization.

How It Works: Genie + Microsoft Foundry Portal

Open Microsoft Foundry
Go to the tool catalog within the Discover tab
Select ‘Azure Databricks Genie’
Configure the connection your Azure Databricks Genie space and click ‘Connect’
Click ‘Use in an agent’ and select the desired agent to connect your Genie space to

Once completed, Genie is available for use in your Foundry agent!

Try It Out: Get started with Genie in Copilot Studio and Microsoft Foundry

To get started with Genie + Copilot Studio, check out our technical documentation
To get started with Genie + Microsoft Foundry, check out the documentation
To learn more about the Generally Available Azure Databricks Power Platform connector explore this blog

SAP Business Data Cloud Connect with Azure Databricks is now generally available

AnaviNahar — Wed, 15 Oct 2025 16:00:00 GMT

We are excited to share that SAP Business Data Cloud (SAP BDC) Connect for Azure Databricks is generally available. With this announcement, Azure Databricks customers like you, can connect your SAP BDC environment to your existing Azure Databricks instance – without copying the data – to enable bi-directional, live data sharing.

Connecting SAP data with other enterprise data prevents governance risk, compliance gaps, and data silos. In addition, maintenance costs are also reduced and manual building of semantics is no longer needed.

SAP data products can now be shared directly via Delta Sharing into your existing Azure Databricks instances ensuring complete context for your business.

You can now unify your data estate across Azure Databricks and SAP BDC This makes it easier for you to:

Enforce governance

Power analytics, data warehousing, BI and AI

Connecting SAP BDC to Azure Databricks is simple, secure, and fast. The connection is trusted and requires approval from both platforms to enable bi-directional sharing of data products. Once approved, data products in SAP BDC can be directly mounted into Azure Databricks Unity Catalog and are treated like other assets shared using Delta sharing. As a result, your teams can query, analyze, and gather insights on SAP data in addition to your existing business data in one unified way. Instead of spending time gathering the data in once place, your teams can instead focus on unlocking insights from this unified data quickly and securely.

This launch complements SAP Databricks in SAP BDC running on Azure that enables AI, ML, data engineering, and data warehousing capabilities directly inside your SAP environment. We have expanded the list of supported regions for SAP Databricks on SAP BDC running on Azure.

To learn more with SAP BDC Connect with Azure Databricks review documentation and get started today.

General Availability: Automatic Identity Management (AIM) for Entra ID on Azure Databricks

AnaviNahar — Wed, 10 Sep 2025 16:00:00 GMT

In February, we announced that Automatic Identity Management in public preview and loved to hear your overwhelmingly positive feedback. Prior to public preview, you either had to set up an Entra Enterprise Application or involve an Azure Databricks account admin to import the appropriate groups. This required manual steps whether it was adding or removing users with organizational changes, maintaining scripts, or requiring additional Entra or SCIM configuration. Identity management was thus cumbersome and required management overhead.

Today, we are excited to announce that Automatic Identity management (AIM) for Entra ID on Azure Databricks is generally available. This means no manual user setup is needed and you can instantly add users to your workspace(s). Users, groups, and service principals from Microsoft Entra ID are automatically available within Azure Databricks, including support for nested groups and dashboards.

This native integration is one of the many reasons Databricks runs best on Azure.

Here are some addition ways AIM could benefit you and your organization:

Seamlessly share dashboards

You can share AI/BI dashboards with any user, service principal, or group in Microsoft Entra ID immediately as these users are automatically added to the Azure Databricks account upon login. Members of Microsoft Entra ID who do not have access to the workspace are granted access to a view-only copy of a dashboard published with embedded credentials. This enables you to share dashboards with users outside your organization, too. To learn more, see share a dashboard.

Updated defaults for new accounts

All new Azure Databricks accounts have AIM enabled – no opt in or additional configuration required. For existing accounts, you can enable AIM with a single click in the Account Admin Console. Soon, we will also make this the default for existing accounts.

Automation at scale enabled via APIs

You can also register users, groups, or service principles in Microsoft Entra ID via APIs. Being able to do this programmatically enables the enterprise scale most of our customers need. You can also enable automation via scripts leveraging these APIs.

Read the Databricks blog here and get started via documentation today!

Closing the loop: Interactive write-back from Power BI to Azure Databricks

AnaviNahar — Fri, 15 Aug 2025 00:12:37 GMT

This is a collaborative post from Microsoft and Databricks. We thank Toussaint Webb, Product Manager at Databricks, for his contributions.

We're excited to announce that the Azure Databricks connector for Power Platform is now Generally Available. With this integration, organizations can seamlessly build Power Apps, Power Automate flows, and Copilot Studio agents with secure, governed data and no data duplication.

A key functionality unlocked by this connector is the ability to write data back from Power BI to Azure Databricks. Many organizations want to not only analyze data but also act on insights quickly and efficiently. Power BI users, in particular, have been seeking a straightforward way to “close the loop” by writing data back from Power BI into Azure Databricks. This capability is now here - real-time updates and streamlined operational workflows with the new Azure Databricks connector for Power Platform.

With this connector, users can now read from and write to Azure Databricks data warehouses in real time, all from within familiar interfaces — no custom connectors, no data duplication, and no loss of governance.

How It Works: Write-backs from Power BI through Power Apps

Enabling writebacks from Power BI to Azure Databricks is seamless. Follow these steps:

Open Power Apps and create a connection to Azure Databricks (documentation).
In Power BI (desktop or service), add a Power Apps visual to your report (purple Power Apps icon).
Add data to connect to your Power App via the visualization pane.
Create a new Power App directly from the Power BI interface, or choose an existing app to embed.
Start writing records to Azure Databricks!

With this integration, users can make real-time updates directly within Power BI using the embedded Power App, instantly writing changes back to Azure Databricks. Think of all the workflows that this can unlock, such as warehouse managers monitoring performance and flagging issues on the spot, or store owners reviewing and adjusting inventory levels as needed. The seamless connection between Azure Databricks, Power Apps, and Power BI lets you close the loop on critical processes by uniting reporting and action in one place.

Try It Out: Get started with Azure Databricks Power Platform Connector

The Power Platform Connector is now Generally Available for all Azure Databricks customers. Explore more in the deep dive blog here and to get started, check out our technical documentation. Coming soon we will add the ability to execute existing Azure Databricks Jobs via Power Automate.

If your organization is looking for an even more customizable end-to-end solution, check out Databricks Apps in Azure Databricks! No extra services or licenses required.

Announcing the Azure Databricks connector in Power Platform

AnaviNahar — Wed, 11 Jun 2025 13:00:00 GMT

We are ecstatic to announce the public preview of the Azure Databricks Connector for Power Platform. This native connector is specifically for Power Apps, Power Automation, and Copilot Studio within Power Platform and enables seamless, single click connection.

With this connector, your organization can build data-driven, intelligent conversational experiences that leverage the full power of your data within Azure Databricks without any additional custom configuration or scripting – it's all fully built in!

The Azure Databricks connector in power platform enables you to:

Maintain governance: All access controls for data you set up in Azure Databricks are maintained in Power Platform

Prevent data copy: Read and write to your data without data duplication

Secure your connection: Connect Azure Databricks to Power Platform using Microsoft Entra user-based OAuth or service principals

Have real time updates: Read and write data and see updates in Azure Databricks in near real time

Build agents with context: Build agents with Azure Databricks as grounding knowledge with all the context of your data

Instead of spending time copying or moving data and building custom connections which require additional manual maintenance, you can now seamlessly connect and focus on what matters – getting rich insights from your data – without worrying about security or governance.

Let’s see how this connector can be beneficial across Power Apps, Power Automate, and Copilot Studio:

Azure Databricks Connector for Power Apps – You can seamlessly connect to Azure Databricks from Power Apps to enable read/write access to your data directly within canvas apps enabling your organization to build data-driven experiences in real time. For example, our retail customers are using this connector to visualize different placements of items within the store and how they impact revenue.

Azure Databricks Connector for Power Automate – You can execute SQL commands against your data within Azure Databricks with the rich context of your business use case. For example, one of our global retail customers is using automated workflows to track safety incidents, which plays a crucial role in keeping employees safe.

Azure Databricks as a Knowledge Source in Copilot Studio – You can add Azure Databricks as a primary knowledge source for your agents, enabling them to understand, reason over, and respond to user prompts based on data from Azure Databricks.

To get started, all you need to do in Power Apps or Power Automate is add a new connection – that's how simple it is!

Check out our demo here and get started using our documentation today! This connector is available in all public cloud regions. You can also learn more about customer use cases in this blog. You can also review the connector reference here

Announcing the availability of Azure Databricks connector in Azure AI Foundry

AnaviNahar — Mon, 19 May 2025 16:00:00 GMT

At Microsoft, Databricks Data Intelligence Platform is available as a fully managed, native, first party Data and AI solution called Azure Databricks. This makes Azure the optimal cloud for running Databricks workloads. Because of our unique partnership, we can bring you seamless integrations leveraging the power of the entire Microsoft ecosystem to do more with your data.

Azure AI Foundry is an integrated platform for Developers and IT Administrators to design, customize, and manage AI applications and agents.

Today we are excited to announce the public preview of the Azure Databricks connector in Azure AI Foundry. With this launch you can build enterprise-grade AI agents that reason over real-time Azure Databricks data while being governed by Unity Catalog. These agents will also be enriched by the responsible AI capabilities of Azure AI Foundry.

Here are a few ways this can benefit you and your organization:

Native Integration: Connect to Azure Databricks AI/BI Genie from Azure AI Foundry

Contextual Answers: Genie agents provide answers grounded in your unique data

Supports Various LLMs: Secure, authenticated data access

Streamlined Process: Real-time data insights within GenAI apps

Seamless Integration: Simplifies AI agent management with data governance

Multi-Agent workflows: Leverages Azure AI agents and Genie Spaces for faster insights

Enhanced Collaboration: Boosts productivity between business and technical users

To further democratize the use of data to those in your organization who aren't directly interacting with Azure Databricks, you can also take it one step further with Microsoft Teams and AI/BI Genie. AI/BI Genie enables you to get deep insights from your data using your natural language without needing to access Azure Databricks.

Here you see an example of what an agent built in AI Foundry using data from Azure Databricks available in Microsoft Teams looks like

We'd love to hear your feedback as you use the Azure Databricks connector in AI Foundry. Try it out today – to help you get started, we’ve put together some samples here. Read more on the Databricks blog, too.

Announcing the availability of Azure Databricks connector in Azure AI Foundry

AnaviNahar — Mon, 19 May 2025 16:00:00 GMT

Azure AI Foundry is an integrated platform for Developers and IT Administrators to design, customize, and manage AI applications and agents.

Here are a few ways this seamless integration can benefit you and your organization:

Native Integration: Connect to Azure Databricks AI/BI Genie from Azure AI Foundry
Contextual Answers: Genie agents provide answers grounded in your unique data
Supports Various LLMs: Secure, authenticated data access
Streamlined Process: Real-time data insights within GenAI apps
Seamless Integration: Simplifies AI agent management with data governance
Multi-Agent workflows: Leverages Azure AI agents and Genie Spaces for faster insights
Enhanced Collaboration: Boosts productivity between business and technical users

To further democratize the use of data for those in your organization aren't directly interacting with Azure Databricks, you can also take it one step further with Microsoft Teams and AI/BI Genie. AI/BI Genie enables you to get deep insights from your data using your natural language without needing to access Azure Databricks.

Here you see an example of what an agent built in AI Foundry using data from Azure Databricks available in Microsoft Teams looks like

We'd love to hear your feedback as you use the Azure Databricks connector in AI Foundry. Try it out today – to help you get started, we’ve put together some samples here.

Llama 4 is now available in Azure Databricks

AnaviNahar — Mon, 07 Apr 2025 05:18:30 GMT

We are excited to announce the availability of Meta's Llama 4 in Azure Databricks.

As you know, enterprises all over the world already use Llama models in Azure Databricks to power AI enterprise agents, workflows, and applications. Now with Llama 4 and Azure Databricks, you can get higher quality, faster inference, and lower cost than previous models.

Llama 4 Maverick, the highest-quality and largest Llama model from today's announcement, is built for developers building the next generation of AI products that combine multilingual fluency, image understanding precision, and security. With Maverick on Azure Databricks, you can:

Build domain specific AI agents with your data
Run scalable inference with your data pipeline
Fine-tune for accuracy and
Govern AI usage with Mosaic AI Gateway

Azure Databricks Intelligence Platform makes it easy for you to securely connect Llama 4 to your enterprise data using Unity Catalog governed tools to build agents with contextual awareness.

Enterprise data needs enterprise scale, whether it is to summarize documents or analyze support tickets, but without the infrastructure overhead. With Azure Databricks workflows and Llama 4 at scale, you can use SQL/Python to run LLMs at scale without overhead.

You can tune Llama 4 to your custom use case for accuracy and alignment such as assistant behavior or summarization.

All this comes with built in security controls and compliant model usage via Azure Databricks Mosaic AI Gateway with PII detection, logging, and policy guardrails on Azure Databricks.

Llama 4 is available now in Azure Databricks. More models will become available in phases. Llama 4 Scout is coming soon and you'll be able to pick the model that fits your workload best. Learn more about Llama 4 and supported models in Azure Databricks here and get started today.

Power BI & Azure Databricks: Smarter Refreshes, Less Hassle

LindseyAllen — Mon, 31 Mar 2025 20:59:40 GMT

We are excited to extend the deep integration between Azure Databricks and Microsoft Power BI with the Public Preview of the Power BI task type in Azure Databricks Workflows. This new capability allows users to update and refresh Power BI semantic models directly from their Azure Databricks workflows, ensuring real-time data updates for reports and dashboards. By leveraging orchestration and triggers within Azure Databricks Workflows, organizations can improve efficiency, reduce refresh costs, and enhance data accuracy for Power BI users.

Power BI tasks seamlessly integrate with Unity Catalog in Azure Databricks, enabling automated updates to tables, views, materialized views, and streaming tables across multiple schemas and catalogs. With support for Import, DirectQuery, and Dual Storage modes, Power BI tasks provide flexibility in managing performance and security. This direct integration eliminates manual processes, ensuring Power BI models stay synchronized with underlying data without requiring context switching between platforms.

Built into Azure Databricks Lakeflow, Power BI tasks benefit from enterprise-grade orchestration and monitoring, including task dependencies, scheduling, retries, and notifications. This streamlines workflows and improves governance by utilizing Microsoft Entra ID authentication and Unity Catalog suite of security and governance offerings.

We invite you to explore the new Power BI tasks today and experience seamless data integration—get started by visiting the [ADB Power BI task documentation].

Anthropic State-of-the-Art Models Available to Azure Databricks Customers

LindseyAllen — Thu, 27 Mar 2025 02:27:17 GMT

Our customers now have greater model choices with the arrival of Anthropic Claude 3.7 Sonnet in Azure Databricks. Databricks is announcing a partnership with Anthropic to integrate their state-of-the-art models into Databricks Data Intelligence Platform as a native offering, starting with Claude 3.7 Sonnet http://databricks.com/blog/anthropic-claude-37-sonnet-now-natively-available-databricks. With this announcement, Azure customers can use Claude Models directly in Azure Databricks. Foundation model REST API reference - Azure Databricks | Microsoft Learn

With Anthropic models available in Azure Databricks, customers can use the Claude "think" tool with business data optimized promote to guide Claude efficiently perform complex tasks. With Claude models in Azure Databricks, enterprises can deliver domain-specific, high quality AI agents more efficiently. As an integrated component of the Azure Databricks Data Intelligence Platform, Anthropic Claude models benefit from comprehensive end-to-end governance and monitoring throughout the entire data and AI lifecycle with Unity Catalog.

With Claude models, we remain committed to providing customers with model flexibility. Through the Azure Databricks Data Intelligence Platform, customers can securely connect to any model provider and select the most suitable model for their needs. They can further enhance these models with enterprise data to develop domain-specific, high-quality AI agents, supported by built-in custom evaluation governance across both data and models.

Part 2: Performance Configurations for Connecting PBI to a Private Link ADB Workspace

katiecummiskey — Mon, 24 Mar 2025 19:19:58 GMT

This blog was written in conjunction with Leo Furlong, Lead Solutions Architect at Databricks.

In Part 1, we discussed networking options for connecting Power BI to an Azure Databricks workspace with a Public Endpoint protected with a workspace IP Access List.

In Part 2, we continue our discussion and elaborate on private networking options for an Azure Databricks Private Link workspace. When using Azure Databricks Private Link with Allow Public Network Access setting set to Disabled, all connections to the workspace must go through Private Endpoints. For one of the private networking options, we’ll also discuss how to configure your On-Premise Data Gateway VM to get good performance.

Connecting Power BI to a Private Link Azure Databricks Workspaces

As covered in Part 1, Power BI offers two primary methods for secure connections to data sources with private networking:

1. On-premises data gateway: An application that gets installed on a Virtual Machine that has a direct networking connection to a data source. It allows Power BI to connect to data sources that don’t allow public connections. The general flow of this setup entails:

a. Create or leverage a set of Private Endpoints to the Azure Databricks workspace - both sub-resources for databricks_ui_api and browser_authentication are required

b. Create or leverage a Private DNS Zone for privatelink.azuredatabricks.net

c. Deploy an Azure VM into a VNet/subnet

d. The VM’s VNet/subnet should have access to the Private Endpoints (PEs) via either them being in the same VNet or being peered with another VNet where they reside

e. Install and configure the on-premise data gateway software on the VM

f. Create a connection in the Power BI Service via Settings -> Manage Connections and Gateways UIs

g. Configure the Semantic Model to use the connection under the Semantic Model’s settings and gateway and cloud connections sub-section

2. Virtual Network Data Gateway: A fully managed data gateway that gets created and managed by the Power BI service. Connections work by allowing Power BI to delegate into a VNet for secure connectivity to the data source. The general flow of this setup entails:

a. Create or leverage a set of Private Endpoints (PEs) to the Azure Databricks workspace - both sub-resources for databricks_ui_api and browser_authentication are required

b. Create or leverage a Private DNS Zone for privatelink.azuredatabricks.net

c. Create a subnet in a VNet that has access to the Private Endpoints (PEs) via either them being in the same VNet or being peered with another VNet where they reside. Delegate the Subnet to Microsoft.PowerPlatform/vnetaccesslinks

d. Create a virtual network data gateway in the Power BI Service via Settings -> Manage Connections and Gateways UIs

e. Configure the the Semantic Model to use the connection under the Semantic Model’s settings and gateway and cloud connections sub-section

The documentation for both options is fairly extensive, and this blog post will not focus on breaking down the configurations further. Instead, this post is about configuring your private connections to get the best Import performance.

On-Premise Data Gateway Performance Testing

In order to provide configuration guidance, a series of Power BI Import tests were performed using various configurations and a testing dataset.

Testing Data

The testing dataset used was a TPC-DS scale factor 10 dataset (you can create your own using this Repo). A scale factor of 10 in TPC-DS generates about 10 gigabytes (GB) of data. The TPC-DS dataset was loaded into Unity Catalog and the primary and foreign keys were created between the tables. A model was then created in the Power BI Service using the Publish to Power BI capabilities in Unity Catalog; the primary and foreign keys were used to automatically create relationships between the tables in the Power BI semantic model. Here’s an overview of the tables used in this dataset:

Fabric Capacity

An F64 Fabric Capacity was used in the West US region. The F64 was the smallest size available (in terms of RAM) for refreshing the model without getting capacity errors - the compressed Semantic Model size is 5,244 MB.

Azure Databricks SQL Warehouse

An Azure Databricks workspace using Unity Catalog was deployed in the East US 2 and West US regions for the performance tests. A Medium Databricks SQL Warehouse was used. For Imports, generally speaking, the size of the SQL Warehouse isn’t very important. Using an aggressive Auto Stop configuration of 5 minutes is ideal to minimize compute charges (1 minute can be used if the SQL Warehouse is deployed via an API).

Testing Architecture

The following diagram summarizes a simplified Azure networking architecture for the performance tests.

A Power BI Semantic Model is connected to a Power BI On-Premise Data Gateway Connection
The On-Premise Data Gateway Connection connects to the Azure Databricks workspace using Private Endpoints.
Azure Databricks provisions up a Serverless SQL Warehouse in ~5 seconds within the Serverless Data Plane within Azure. SQL queries are executed on the Serverless SQL Warehouse.
Unity Catalog gives the Serverless SQL Warehouse a read-only, down-scoped, and pre-signed URL to ADLS.
Data is fetched from ADLS and placed on the Azure Databricks workspace’s managed storage account via a capability called Cloud Fetch.
Arrow Files are pulled from Cloud Fetch and delivered to the Power BI Service through the Data Gateway.
Data in the Semanic Model is compressed and stored in Vertipaq In-Memory storage.

Testing Results

The following grid outlines the scenarios tested and the results for each test. We’ll review the different configurations tested below in specific sections.

Scenario	Gateway Scenario	Avg Refresh Duration Minutes
A	East US 2, Public Endpoint	17:01
B	West US, Public Endpoint	12:21
C	West US, Public Endpoint via IP Access List	15:19
D	West US, E VM Gateway Base	12:14
E	West US, E VM StreamBeforeRequestCompletes	07:46
F	West US, E VM StreamBeforeRequestCompletes + Logical Partitions	07:31
G	West US, E VM Spooler (D)	12:57
H	West US, E VM Spooler (E)	13:32
I	West US, D VM Gateway Base	16:47
J	West US, D VM StreamBeforeRequestCompletes	12:19
K	West US, PBI Managed Vnet	27:04

Scenario	VM Configuration
D	Standard E8bds v5 (8 vcpus, 64 GiB memory) [NVMe, Accelerated Networking], C Drive default (Premium SSD LRS 127 GiB)
E	Standard E8bds v5 (8 vcpus, 64 GiB memory) [NVMe, Accelerated Networking], C Drive default (Premium SSD LRS 127 GiB)
F	Standard E8bds v5 (8 vcpus, 64 GiB memory) [NVMe, Accelerated Networking], C Drive default (Premium SSD LRS 127 GiB)
G	Standard E8bds v5 (8 vcpus, 64 GiB memory) [NVMe, Accelerated Networking], D drive
H	Standard E8bds v5 (8 vcpus, 64 GiB memory) [NVMe, Accelerated Networking], E Drive (Premium SSD LRS 600 GiB)
I	Standard D8s v3 (8 vcpus, 32 GiB memory), C Drive default (Premium SSD LRS 127 GiB)
J	Standard D8s v3 (8 vcpus, 32 GiB memory), C Drive default (Premium SSD LRS 127 GiB)

Performance Configurations

1. Regional Alignment

Aligning your Power BI Premium/Fabric Capacity to the same region as your Azure Databricks deployment and your On-Premise Data Gateway VM helps reduce the overall network latency and data transfer duration. It should also eliminate cross-region networking charges.

In scenario A, the Azure Databricks deployment was in East US 2 while the Fabric Capacity and On-Premise Data Gateway VM were in West US. The Import processing time when using the public endpoint between the regions was 17:01 minutes. In scenario B, while still using the public endpoint, there is complete regional alignment in the West US region and the Import times averaged 12:21 minutes which is a 27.4% decrease

2. Configure a Gateway Cluster

A Power BI Data Gateway Cluster configuration is highly recommended for Prouduction deployments but this configuration was not performance tested during this experiment. Data Gateway clusters can help with data refresh redundancy and for overall volume / throughput of data transfer. This configuration is highly recommended for Production Power BI environments.

3. VM Family Selection

The Power BI documentation recommends a VM with 8 cores, 8 GB of RAM, and an SSD for the VM used for the On-Premise Data Gateway. Through testing, it can be proven that using a VM with good performance characteristics can provide immense value in the Import times.

In scenario D, data gateway tests were run using a Standard E8bds v5 with 8 cores and 64 GB RAM that also included NVMe, and Accelerated Networking, and a C drive using a Premium SSD. The import times for this scenario averaged 12:14 minutes which was slightly faster than the regionally aligned public endpoint test in scenario B.

In scenario I, data gateway tests were run using a Standard D8s v3 with 8 cores and 32 GB RAM and a C drive using a Premium SSD. The import times for this scenario averaged 16:47 minutes which was noticeably slower than using the regionally aligned public endpoint in cenario B which was a 35.96% performance degradation.

More tests could certainly be done to determine which VM characteristics help the most with Import performance, but it is clear certain features can be helpful like:

And while the better E8bds v5 Azure VM costs ~$820 per month in West US at list and the D8s v3 costs ~$610 per month at list (25% more expensive), this feels like a scenario where you pay the premium to get better performance and optimize through Azure VM reservations.

4. StreamBeforeRequestCompletes

By default, the on-premise data gateway spools data to disk before sending it to Power BI. Enabling the StreamBeforeRequestCompletes setting to True can significantly improve gateway refresh performance as it allows data to be streamed directly to the Power BI Service without first being spooled to disk.

In scenario E, when StreamBeforeRequestCompletes is set to True and restarted, you can see that the average Import times significantly improved to 07:46 minutes which is a 54% improvement compared to scenario A and a 36% improvement over the base VM configuration in scenario D.

5. Spooler Location

As discussed above, when using the default setting for StreamBeforeRequestCompletes as False, Power BI spools the data to the data gateway spool directory before sending it to the Power BI Service.

In scenarios D, G, and H, StreamBeforeRequestCompletes is False and the Spooler directory has been mapped to the C drive, D drive, and E drives respectively which all correspond to an SSD (of varying configuration) on the Azure VMs.

In all scenarios, you can see the times are similar between 12:14, 12:57, and 13:32 minutes, respectively. In all three scenarios the tests were performed with SSDs on the E series VM configured with NVMe. Using this configuration mix, it doesn’t appear that the Spooler directory location provides significant performance improvements. Since the C drive configuration gave the best performance it seems prudent to keep the C drive default configuration. However, it is possible that that the Spooler directory setting might provide more value on a different VM configurations.

6. Logical Partitioning

As outlined in the QuickStart samples guide, logical partitioning can often help with Power BI Import performance as multiple logical partitions in the Semantic Model can be processed at the same time.

In scenario F, logical partitions were created for the inventory and store_sales table to have 5 partitions each. When combined with the StreamBeforeRequestCompletes setting, the benefit from adding Logical Partitions was negligible (15 second improvement) even though the parallelization settings were increased to 30 (Max Parallelism Per Refresh and Data Source Default Max Connections).

While logical partitions are usually a very valuable strategy, combining them with StreamBeforeRequestCompletes, the E series VM configurations, and a Fabric F64 capacity yielded diminishing returns. It is probably worth more testing at some point in the future.

Virtual Network Data Gateway Performance Testing

The configuration and performance of a Virtual Network Data Gateway was briefly tested. A Power BI subnet was created in the same VNet as the Azure Databricks workspace and delegated to the Power BI Service. A virtual network data gateway was created in the UI with 2 gateways (12 queries can run in parallel) and assigned to the Semantic Model.

In scenario K, an Import test was performed through the Virtual Network Data Gateway that took 27:04 minutes. More time was not spent trying to tune the Virtual Network Data Gateway as it was not the primary focus of this blog post.

The Best Configuration

The Best Configuration: Region Alignment + Good VM + StreamBeforeRequestsCompletes

While the Import testing performed for this blog post isn’t definitive, it does provide good directional value in forming an opinion on how you can configure your Power BI On-Premise Data Gateway on an Azure Virtual Machine to get good performance.

When looking at the tests performed for this blog, an Azure Virtual Machine, in the same region as the Azure Databricks Workspace and the Fabric Capacity, with Accelerated networking, an SSD, NVMe, and memory optimized compute provided performance that was faster than just using the public endpoint of the Azure Databricks Workspace alone. Using this configuration, we improved our Import performance from 17:01 to 07:46 minutes which is a 54% performance improvement.

6 critical phases to prepare for a successful Azure Databricks migration

katiecummiskey — Thu, 20 Mar 2025 21:03:46 GMT

As organizations adopt advanced analytics and AI to drive decision-making, moving data applications to Azure Databricks has become a strategic and significant endeavor. This transition requires careful planning and execution to succeed. Based on numerous successful implementations, we’ve identified six critical phases that can help you prepare for a smooth migration.

Phase 1: Infrastructure and workload assessment

Starting with a thorough analysis of your current environment prevents unexpected issues during migration. Many organizations face setbacks by rushing ahead without a complete picture of their data estate.

A comprehensive assessment includes:

Data source and workload cataloging: Use automated assessment tools to create a detailed inventory of your data assets. Track data volumes, update frequencies, and usage patterns.
ETL process analysis: Record the business logic, scheduling dependencies, and performance characteristics of each ETL process. Focus on custom transformations that may need redesign in the Databricks environment.
SQL code dependency mapping: Build a dependency graph of SQL objects, including stored procedures, views, and user-defined functions. This identifies which elements need to migrate together and shows potential improvements.
Application interdependency analysis: Monitor how applications interact with your data systems, including read/write patterns, API dependencies, and real-time processing needs.
Performance baseline: Document current performance metrics and SLA requirements to set a clear performance baseline and identify areas where Databricks can improve efficiency.

Best practice: Engage various tools that can speed up an assessment by automatically mapping your data estate.

Phase 2: Strategic migration planning

With clear insights into your environment, develop an approach that balances risk management with business value. This phase helps secure stakeholder support and set realistic expectations.

Your migration strategy should include:

Workload prioritization framework: Create a scoring system based on business impact, technical complexity, and resource needs. High-value, low-complexity workloads make excellent candidates for initial migration phases.
Timeline development: Build a realistic schedule that considers dependencies, resource availability, and business cycles. Include extra time for addressing challenges and learning new processes.
Success criteria definition: Set specific, measurable KPIs aligned with business goals, such as performance improvements, cost reductions, or new analytical capabilities.
Resource allocation planning: Specify the skills and staff needed for each migration phase, including whether specific components might benefit from external expertise.

Best practice: Start with a pilot project using noncritical workloads to learn and refine processes before moving to business-critical applications.

Phase 3: Technical preparation

Technical preparation creates a foundation for successful migration through proper configuration and security. This phase needs attention to detail and collaboration between infrastructure, security, and development teams.

Key preparation steps include:

Environment configuration: Create separate Azure Databricks environments for development, testing, and production. Configure cluster sizes, runtime versions, and autoscaling policies.
Security implementation: Set up security controls, including network isolation, access management, and data encryption.
Delta Lake implementation: Use Delta Lake format for ACID compliance and features like time travel and schema enforcement to maintain data quality and consistency.
Connectivity setup: Create and test secure connections between Azure Databricks and source systems with sufficient bandwidth and minimal latency.

Best practice: Use Azure Databricks Unity Catalog for precise access control and data governance.

Phase 4: Data and code migration planning

Moving data and code requires careful planning to maintain business operations and data integrity. This phase has two main components:

ETL migration strategy:

Workflow mapping: Map existing ETL processes to Azure Databricks equivalents, using native capabilities to improve efficiency.
Transformation logic conversion: Convert legacy transformation logic to Spark SQL or PySpark to use Databricks’ distributed processing.
Data quality framework: Add automated testing to verify data quality and completeness during migration.
Performance optimization: Create strategies for optimizing workflows through proper partitioning, caching, and resource allocation.

SQL code migration approach:

Code conversion process: Create a systematic method for working with SQL stored procedures, handling vendor-specific SQL syntax.
Query optimization: Apply best practices for Spark SQL performance with proper join strategies and partition pruning.
Version control integration: Implement version control with Git integration for collaborative development and change tracking.

Best practice: Monitor the migration using Azure-native tools (such as Azure Monitoring and Azure Databricks Workflows) to identify and resolve bottlenecks in real-time.

Phase 5: Validation and testing

Complete testing ensures migration success. Create a testing strategy that includes:

Data accuracy validation: Compare migrated data to source systems using automated tools.
Performance validation: Validate performance under various loads to ensure meeting or exceeding SLAs and previously established performance baseline.
Integration testing: Check that all system components work together, including external applications.
User acceptance testing: Verify with business users that migrated systems meet their needs.

Phase 6: Team enablement and governance

Success requires more than technical implementation. Prepare your organization by:

Role-based training: Create specific training programs for each user type, from data engineers to business analysts.
Governance framework: Apply comprehensive governance with Unity Catalog for data classification, access controls, and audit logging.
Support structure: Define support channels and procedures for addressing issues after migration.
Monitoring framework: Add proactive monitoring to identify and fix potential issues before they affect operations.

Best practice: Schedule regular reviews of compliance and security measures to address evolving risks.

Measuring success and future optimization

Success means delivering clear business value. Monitor key metrics:

Query performance improvements
ETL processing time reduction/data freshness improvement
Resource utilization efficiency
Cost savings versus previous systems

After migration, focus on ongoing improvements using Azure Databricks features:

Automated performance optimization
Resource management for cost control
Integration of advanced analytics and AI
Improved real-time processing

A successful Azure Databricks migration requires careful planning across all six phases. This approach minimizes risks while maximizing the benefits of your modernized data platform. The goal extends beyond moving workloads, as it transforms your organization’s data capabilities.

Want more information about planning your migration? Get our detailed e-book for in-depth guidance on strategies, governance, and business impact measurement. See how organizations improve their data infrastructure and prepare for advanced analytics.

Download the e-book.

Part 1: Power BI Service Connections to Azure Databricks with Private Networking

katiecummiskey — Fri, 21 Feb 2025 21:50:07 GMT

This blog was written in conjunction with Leo Furlong, Lead Solutions Architect at Databricks.

Enhancing Security and Connectivity: Azure Databricks SQL, Unity Catalog, and Power BI Integration

The combination of Azure Databricks SQL, Unity Catalog, and Power BI offers an unparalleled set of capabilities for modern data analytics. However, as organizations increasingly prioritize security, many Azure Databricks customers are deploying their Databricks workspace with private networking requirements which requires additional configuration for allowing connections from BI tools like Power BI. This blog post explores the options available for secure Azure Databricks deployments and how to maintain Power BI connectivity in these scenarios.

Private Networking Options for Azure Databricks

When deploying Azure Databricks with enhanced security, customers can choose from three main private networking configurations:

Public Endpoint with an IP Access List for the Workspace: This option exposes a public endpoint for the Azure Databricks workspace but restricts access to specific IP ranges.
Azure Databricks Private Link: Front-end private link provides fully private connectivity, routing all traffic through private endpoints.
Hybrid Deployment: Combines front-end private link with a public endpoint protected by a Workspace IP Access List which is typically used for SaaS service connections.

Connecting Power BI to a Private Azure Databricks Workspaces

While private networking enhances security, it can require additional connection configurations from SaaS services like Power BI. Power BI offers two primary methods for secure connections to data sources with private networking:

On-premises data gateway: an application that gets installed on a Virtual Machine that has a direct networking connection to the data source. It allows Power BI to connect to data sources that don’t allow public connections
Virtual Network Data Gateway: a managed (virtual/serverless) data gateway that gets created and managed by the Power BI service. Connections work by allowing Power BI to delegate into a VNet for secure connectivity to the data source.

While Power BI offers these two options, many customers prefer not to manage additional infrastructure or configurations required for these gateways. In such cases, Power BI can be allowed to access the private Azure Databricks workspace through the IP Access List.

Implementing Power BI Connectivity via IP Access List

To enable the Power BI Service connectivity to a private Azure Databricks workspace using an IP Access List:

Obtain the Power BI Public IPs:
Download the latest Azure IP Ranges and Service Tags file from the Microsoft Download Center. This file is updated weekly and contains IP ranges for various Azure services, including Power BI.
Add Power BI IPs to Azure Databricks Workspace IP Access List:
Extract the Power BI IP ranges from the downloaded file and add them to the Azure Databricks IP Access List using the API or SDK.
Regular Updates:
Since Power BI public IPs can change frequently, it's crucial to update the Workspace IP Access List regularly. This can be automated using a Databricks Job that periodically downloads the latest IP ranges and updates the Workspace IP Access List. The Job will need to be run by a Workspace Admin in order to set the configurations. You can run the Databricks Job as a Service Principal to make the updates. If you use the Databricks SDK from within a notebook in the Databricks Workspace, authentication is handled for you.

The following sample code can be used to turn on your Workspace IP Access List which is more of a one-time operation. The Power BI IPs for IP Access List sample code can be used to refresh your Power BI IPs from a Databricks Workflow.

Conclusion

By leveraging IP Access Lists, organizations can maintain the security benefits of private Azure Databricks deployments while ensuring seamless connections from Power BI. This approach offers a balance between security and functionality with low maintenance overhead.

Workarounds for Maven Json-smart 2.5.2 Release Breaking Azure Databricks Job Dependencies

LindseyAllen — Wed, 12 Feb 2025 14:47:32 GMT

A recent Maven library release corrupted json-smart’s central metadata, ended up removing all previous versions of Maven library other than 2.5.2. This results in DBR ivy resolution failures if customers’ job has transitive dependency on previous version of json-smart. The impact to Azure Databricks customers is jobs failure and in some cases the job clusters could fail to start. More details and updates of the issue can be found here: https://github.com/netplex/json-smart-v2/issues/240.

This also affects google’s maven mirror, which is used by DBR 11+ to resolve maven libraries, and maven central is used as the backup for google’s maven mirror.

To mitigate the issue, you can take the workaround steps to install the customer’s requested library and smart-json separately

Steps 1: Install the customer’s requested library (use azure-eventhubs-spark as an example) but exclude json-smart.

Step 2: Install net.minidev:json-smart:2.3 (or any other version that was needed)

Workaround for run time, if you have a preferred Maven mirror that is not affected by this issue, or if you host a private Maven mirror, you can configure your Databricks environment to use it.

Step:

Set the Spark configuration parameter spark.databricks.driver.preferredMavenCentralMirrorUrl to your preferred mirror repository URL.

Replicating Azure Cosmos DB into Azure Databricks using CDC

katiecummiskey — Thu, 21 Nov 2024 15:11:37 GMT

This blog was written in conjunction with David Poulet, Senior Solutions Architect at Databricks.

Microsoft’s NoSQL database-as-a-service, Azure Cosmos DB, is a key platform in Azure for storing non-relational, transactional data and vectors for applications with high throughput and availability requirements. This data often holds valuable business insights, and the ability to analyze this data at scale with Azure Databricks is a key requirement for many customers. Azure Cosmos DB is optimized for fast reads and writes of individual items. However, in common with other data stores of this type, it is not optimized for analytical workloads and this can create a challenge to analyzing stored data in a performant and cost-effective way.

Microsoft’s solution to this problem is the Analytical Store, which stores a copy of the Azure Cosmos DB data in a columnar format and keeps it up-to-date. However, until recently this feature stored the data in a proprietary format and a hidden location that could not be accessed except via Azure Synapse and was subject to a number of restrictions around the types of data structures and query types that it could handle.

But there is now a flexible and open solution to this problem! Microsoft has a feature in Azure Data Factory that enables users to replicate the Azure Cosmos DB Analytical Store into their lakehouse in Delta format, automatically inserting/updating/deleting records as the source transactional database changes. The incremental nature of this offers significant cost savings vs pulling data directly from the transactional store and dealing with complex incremental ingestion logic in code. In this article, I’ll show how we can leverage this feature to create a simple process to continuously ingest operational data in Azure Cosmos DB into Azure Databricks’ powerful analytics and AI platform.

Architecture Overview

The architecture we’ll discuss in this article will use the CDC capability for Azure Cosmos DB within Azure Data Factory to process changes in a Azure Cosmos DB container and then merge them into a Delta Lake table in the lakehouse. See the diagram below:

Azure Data Factory (ADF) will read a container from Azure Cosmos DB (via the analytical store) and periodically replicate any changes from that container into a Delta Lake table in Azure Databricks. This incremental replication process will operate on a schedule that is defined within ADF.

There are a couple of possibilities for how we ingest these changes into Azure Databricks: we could move the data to a staging area, and ingest into Bronze from there using a workflow or Delta Live Tables, but for simplicity we’ll write directly to a table in the Bronze layer of our medallion architecture from ADF.

Once the data is in our Bronze layer standard Azure Databricks patterns can be used to cleanse and transform the data into Silver/Gold layers.

The bulk of the activities happen in Azure Data Factory, but there are some prerequisites. Before we can create the CDC pipeline it’s assumed the following already exist:

A Azure Cosmos DB for NoSQL container, with Analytical Store enabled.
Azure Data Factory instance in which to create a CDC pipeline.
An ADLS storage container to act as our staging area.

With these in place, we can create the CDC pipeline from ADF.

Setting Up The Pipeline

The feature in ADF that consumes the Azure Cosmos DB changes is in the Data Flows area, so we start by launching the ADF studio and creating a new data flow:

The new data flow needs a Source and a Sink. The source will be our Azure Cosmos DB container and the Sink will be our Delta Table in Bronze.

First we’ll create and configure the Source to consume from our Azure Cosmos DB container. Click to Add Source in the new Dataflow. In the source settings we have to set the Source Type to Inline and the Inline Dataset Type to Azure Cosmos DB for NoSQL. The Store Type should be set to Analytical.

The Linked Service should be set to a linked service for Azure Cosmos DB that has been set up to connect to our source container. For details on how to create an ADF Linked Service see the getting started documentation for Azure Cosmos DB.

In the Source Options for the Data Flow, there are some settings that are important to control the behavior of the reads from the source feed.

The Container name field is where we select the Azure Cosmos DB container we are interested in. In this example we have a container with some simple customer related data in.

The Start from field allows us to synchronize ALL the data in the container from the start of its life, or you can select to only sync changes from now on (or from a given timestamp).

You have the option to capture intermediate updates, if you want to maintain a history of all the changes, but we are just going to capture the latest state so this is unselected. Capture Deletes ensures that deleted items from the source are also deleted in our Bronze table. Capture Transactional store TTLs means that if items are expired from the Azure Cosmos DB transactional store by the Time-To-Live function, they will also be deleted from our copy of the data. This is enabled by default but many people may not want this functionality as TTL is often used to reduce the data size of the transactional store at the cost of losing historical data, but in the analytics world this historical data is often important. We’ll leave it at the default though for now.

Next we’ll add a Sink to publish the change data to. Click the + button next to the source icon and search for the Sink option.

We then need to configure the Sink to point to our Bronze table in the lakehouse.

In the Sink settings we select our incoming source stream (there is only one in this case, the one we just created). We again select Inline for Sink type. And the Inline dataset type is Delta. Once again the Linked service is an ADF linked service which points to a blob container/folder that will store our Bronze table. You can read the documentation for creating an ADF blob linked service (or ADLS, either will work) on this page.

Next, the Settings page for our Sink has some important options to control the behavior of the table we are creating.

First we need to select the correct Folder path for the folder in the blob container that will store our Bronze table data. Here we have a simple folder called customer where ADF will put the Delta Lake files.

We also need to think about the Update method field. In this case we will allow Insert (to put new rows in the table as they are added in the source), Delete (to remove rows in the table as they are deleted in the source) and Update (updating existing rows to match changes in the source). To do this ADF needs a unique field in the source that it can match in the target table - so we select List of columns and put {_rid} in the column field. _rid is a system field in Azure Cosmos DB that uniquely identifies a data item.

At this point we are actually ready to run this Data Flow to start syncing Azure Cosmos DB changes to our Bronze table. To do this we need to create a Pipeline in ADF to run the Data Flow defined above.

In the ADF studio resources section, under Pipelines create a new pipeline, and in that pipeline drag a single action onto the pipeline edit canvas - a Data Flow action.

Once we’ve created a pipeline with a Data Flow action, we will edit the Data Flow action settings to trigger the CDC Data Flow we created above. Here all we need to do is select our data flow in the Data Flow drop-down.

Then, like all ADF pipelines we need a trigger to start the pipeline and we’re ready to start ingesting data. From the pipeline editor menu select Add Trigger and then New/Edit - this will bring up the trigger menu below.

We’ll set our trigger to run on creation and then run every 5 minutes after that. This means that every 5 minutes the pipeline will get the latest changes from Azure Cosmos DB and push them into our Bronze table.

Using The Target Table

With the pipeline running, we should start to see data flowing into our target Delta Lake table. I have created a simple customer data set for this example, with three items in the container. After the pipeline has run these items are pushed into a Delta Lake table in our target ADLS container.

In a notebook in Azure Databricks, we can load that Delta Lake table and see its contents:

We can already access the data in the target Delta table from Azure Databricks. Each time the pipeline in ADF runs, it will update this table with whatever inserts/updates/deletes have happened in the source container.

To really make the best use of this as a Bronze table in Azure Databricks, we’re going to create an external table in Unity Catalog to integrate this data with the rest of our UC resources and in this way make it securely accessible to all our Azure Databricks users.

First in the Catalog view in Azure Databricks we create a new external location:

Then we configure the external location to point to our target ADLS folder.

In the new external location dialog, we give the location a name, we select the storage credential that we’ll use to access the external container (in this case the managed identity that is assigned to my Azure Databricks workspace), and the URL to the actual storage container itself. Note that if you have not already done so you will have to ensure that the managed identity for your Azure Databricks workspace has been assigned the relevant permissions to access the storage container. For more information on configuring external locations in Azure Databricks see this documentation.

Finally we can create an external table over our target storage container location so that we can access the table in UC. Inside an Azure Databricks notebook we can do this very simply:

In the above example this creates the bronze table in the cdc_demo schema of my cdc_catalog catalog. Once this is done we can query this table like any other table in Unity Catalog, and view the data that’s being replicated from Azure Cosmos DB by our ADF pipeline. We can then continue to enrich, clean and merge this data downstream using standard Azure Databricks processes for example as shown in the documentation here.

So we can see that with a simple pipeline in ADF, we have created a robust way of opening up our Azure Cosmos DB transactional data to whatever complex analytical processes we want to use in Azure Databricks without reading the transactional data store itself, thus reducing cost and “noisy neighbor” risks.