analytics
41 TopicsIntroducing the new Azure Databricks on Microsoft Azure Government, now in Beta
Closing the gap between innovation and compliance For years, public sector leaders have faced a difficult choice: adopt the latest data and AI capabilities or meet the strict security and compliance requirements of their mission. As a result, many agencies and their partners have not had access to the full set of Azure Databricks capabilities available in the global Azure cloud. On Azure Databricks, we believe the public sector should not have to make that trade-off. Today, we're announcing the Beta release of Azure Databricks on Microsoft Azure Government. Agencies and the organizations that serve them can now build on the same modern data foundation available in global Azure, including Unity Catalog, Databricks SQL, and serverless compute. For details on Azure Databricks release types, see Azure Databricks preview releases - Azure Databricks | Microsoft Learn. One platform across global Azure and Azure Government On global Azure, Azure Databricks has already achieved FedRAMP Class D authorization. With this Beta, the same modern platform arrives in Microsoft Azure Government, giving U.S. public sector customers a modern Azure Databricks option in both Azure clouds. Azure Government is Microsoft's dedicated Azure cloud for U.S. government agencies and their partners. It is physically isolated from the global Azure cloud, and screened U.S. persons operate it to meet the compliance requirements of federal, state, local and tribal governments. Choosing between the two comes down to the workload. Azure Databricks is available today for federal civilian agencies, state and local governments, and partners whose workloads fit FedRAMP Class D in global Azure regions. The new Azure Government environment, now open in Beta, is for organizations that need a dedicated government cloud. In both clouds, Azure Databricks is a first-party Azure service. Agencies manage it in the Azure portal, rely on its native integration with Microsoft Entra ID and apply usage toward their existing Azure commitments, so adoption happens within the cloud, contracts and identity systems they already trust. Under the hood, the new environment shares the architecture of Azure Databricks in global Azure and is built to take in the full Databricks platform, with every feature released in global Azure reaching Azure Government through the same ongoing releases. Authorization path FedRAMP Class D (formerly FedRAMP High) and DoD Impact Level 5 (IL5) authorizations for the new Azure Databricks on Azure Government environment are pending. Once they are in place, agencies and defense partners will be able to bring Controlled Unclassified Information (CUI) and mission-critical workloads to the modern Azure Databricks platform. Until then, the Beta environment must not be used to process CUI or any other data regulated by these standards, and General Availability depends on completing the applicable government authorization process. What's available in Beta Beta is the first step toward the full platform, and it already delivers the core capabilities that data teams depend on. It is intended for non-production workloads, and feature availability might differ from the generally available Azure Databricks service. Unified governance across data and AI: Unity Catalog comes to Azure Government for the first time. Teams can control access, trace lineage and audit usage for every data and AI asset from one place: write a policy once, enforce it everywhere and answer who accessed what, when and why. Analytics that scale without the overhead: Databricks SQL provides fast analytics at high concurrency. For each workload, engineers and data scientists pick classic clusters or serverless compute, spending less time sizing and patching infrastructure and more time on the mission. Security controls on from day one: The Compliance Security Profile is enabled by default in every workspace. It adds enhanced security monitoring, automatic cluster updates and a hardened compute image, and it enforces VM instance types that support Azure Virtual Network encryption for hardware-implemented encryption between cluster nodes. Front-end and back-end connections both require Azure Private Link, which keeps all traffic on the Microsoft backbone network. Visibility into usage and cost: Administrators can query system tables for audit logs, billing, compute, jobs and pipelines, ML experiments, query history and storage in one place, so responding to audit requests and managing spend uses tools they already know. Data that stays in government regions: Data residency is always enforced: customer content is stored and processed only in the U.S. government Azure regions US Gov Virginia and US Gov Arizona. Cross-Geo processing does not apply to Azure Government workspaces and cannot be used to process data outside those regions. Take the next step Starting fresh in the new environment During Beta, access is enabled per subscription. Register a dedicated Azure Government subscription that has never been used to create an Azure Databricks workspace. In the Azure Government portal, open that subscription's Preview features, find the Azure Databricks new GovCloud environment feature and register it (this requires permission to register subscription features). Once the status reads Registered, each new workspace created in that subscription lands in the new environment. Plan Beta workloads for non-production and non-CUI scenarios. Moving from the existing Azure Government environment Because the new environment is separate, registering for Beta leaves your existing workspaces, notebooks, data and configurations untouched, and you can evaluate the new environment alongside them. Use a different subscription from the one that hosts your existing workspaces, since a subscription cannot be registered for both environments at the same time. Four areas deserve early attention: Unity Catalog: Begin now so your teams have time to design and roll out unified governance. Serverless compute: Serverless is new to government environments, so open your internal security and compliance review early. Network encryption: Because the Compliance Security Profile is enabled on every workspace, clusters need instance types that support Azure Virtual Network encryption. Align your cluster configurations and Terraform/ARM templates accordingly. Your ATO: Engage your security and authorizing officials now about adding Azure Databricks on Azure Government to your system's Authorization to Operate (ATO). Updating system security plans, data flows and boundary documentation during Beta positions you to run CUI production workloads at General Availability. Getting started To get started, visit Azure Databricks on Microsoft Azure Government. For questions or feedback during Beta, contact your Microsoft account team. Learn More Azure Databricks on Microsoft Azure Government The new Azure Databricks on Azure Government is now in Beta | Databricks Blog Databricks Partners with Microsoft to bring our Data + AI Platform to Azure Government | Databricks Blog305Views0likes0CommentsAzure Databricks delivers new integrations for enterprise intelligence to use, build, and govern AI
At Microsoft Ignite 2026, we are announcing new integrations that bring Azure Databricks intelligence into Microsoft 365 Copilot and Copilot Cowork, connect business meaning through Genie Ontology alongside Microsoft IQ, and bring Genie agents into enterprise agent oversight. The goal is to help customers put enterprise data to work through AI, with the business context and governance that data requires. For business users, that means asking questions in familiar Microsoft tools. For data teams, it means extending the reach of the data and business definitions they manage. For agent builders and IT teams, it means connecting those experiences to enterprise intelligence and oversight. These updates build on Azure Databricks, the jointly engineered Microsoft and Databricks service that brings data engineering, analytics, and AI together. They extend that foundation across a Microsoft ecosystem that includes Foundry, Fabric, and Copilot. Bring enterprise data into the task at hand A business question rarely ends with a query result. A sales leader investigating a change in performance needs an answer they can use in a review, a discussion, or a decision. The Genie One plugin, built on Model Context Protocol (MCP), lets Microsoft 365 Copilot and Copilot Cowork call Genie for Databricks data questions as part of a broader task. Copilot plans the work, selects tools, and composes the final response. Genie contributes the intelligence needed for the data question. Copilot sends Genie a scoped question, not the entire user prompt. The user can keep working in Copilot while Genie handles the relevant part of the request. For people who want to explore their data directly, the Genie app in Teams and Microsoft 365 Copilot provides a conversational experience with Genie. Whether the starting point is a direct data question or a larger Copilot task, Azure Databricks intelligence becomes accessible within the tools people already use. Put business meaning to work beyond the Lakehouse Data teams invest in more than pipelines and tables. They establish what a metric means, which sources are authoritative, and who can access the underlying information. Those decisions need to remain relevant when AI becomes another consumer of enterprise data. Genie Ontology contributes the business meaning of Azure Databricks data, including metrics, semantics, and usage, with access gated by Unity Catalog permissions. That intelligence complements Microsoft IQ. Work IQ, Fabric IQ, Foundry IQ, and Web IQ provide shared context about how people work, how the business runs, what the organization knows, and what is happening on the web. Genie Ontology adds the meaning of the organization's Databricks data. For a question about revenue, the intended metric matters as much as the result. Connecting work context with business meaning helps bring the data team's definitions into the AI experience, while Unity Catalog permissions continue to gate access to Azure Databricks data. The benefit is a wider role for the governed data foundation: the work data teams do can support conversations and tasks beyond a traditional analytics interface. Connect agents to governed Databricks intelligence An agent that coordinates work needs a way to ask the right system the right question. The Genie One MCP integration gives Copilot a defined path to Genie for Databricks data questions, rather than treating every request as a generic prompt. For teams building custom agents, these updates build on the previously announced workspace-wide MCP endpoint for Copilot Studio. That endpoint lets an agent work across connected Genie spaces and Unity Catalog datasets through one workspace connection, preserving the curated context in Genie spaces and enforcing data permissions at query time. The distinction matters: the Genie One plugin brings Databricks intelligence into Copilot-coordinated work, while the Copilot Studio integration supports teams authoring their own agents. Both extend the usefulness of the governed Azure Databricks foundation. Microsoft and Databricks are also working to extend agent registry integration to Agent Bricks agents and tools. That is a direction for the joint work, distinct from the Genie agent registry synchronization described below. Bring Genie agents into enterprise IT observability and security As agents become part of everyday work, IT teams need visibility into the agent estate as well as controls over the data those agents use. Genie agents sync into the Microsoft 365 agent registry, helping bring them into the organization's agent inventory. Agent 365 provides capabilities to register, observe, and govern agents across platforms, working with Microsoft Entra, Microsoft Purview, and Microsoft Defender. Enterprise agent oversight complements Databricks governance in Unity Catalog and Unity Gateway. These layers address different responsibilities: understanding and managing agents across platforms, and governing data and AI assets within Databricks. Microsoft Entra provides a shared identity foundation for the Azure Databricks experience. Together, these capabilities connect the user-facing AI experience with the identity, data permissions, and oversight that enterprise teams need to manage. A shared foundation, useful across roles Consider a business review supported by Copilot. A business user asks about performance. The data team's metric definitions give the question meaning. Copilot calls Genie for the relevant Databricks answer. Data permissions govern what can be accessed, while agent inventory and oversight help IT manage the agents involved. That is the unified Azure Databricks story behind these announcements: make governed enterprise intelligence useful to more people, through the Microsoft experiences where their work happens. Learn more at Microsoft Ignite Databricks joins Ignite as a Visionary Sponsor, with six featured sessions, a sponsor booth, expert meet-ups, and dedicated customer conversations. To explore the integrations and their technical context, start with these key sessions: Session Title BRK364 Powering AI with Azure Databricks, Foundry, Fabric, and Copilot THRSP1130 Build governed AI that understands your business context BRKSP617 Scale agentic AI and turn data into autonomous action The broader program also includes BRK453: What's new and what's next in Azure Storage THR909: Seismic shift: Turning subsurface data into decisions LTG1303: Azure Databricks for government cloud: Modern data and AI, delivered Find times, locations, and attendance options in the Microsoft Ignite session catalog. Visit Databricks at booth 5271R and connect with the team through the Azure Expert Meet-Up programming for demos and technical conversations. Dedicated meeting space at The Social at the Hilton Canopy Hotel is available November 17-19. Explore the Databricks Ignite program and meeting options or visit the Azure Databricks documentation to learn more about the platform. Stay in the know with all things Azure Databricks https://aka.ms/adbreleasenotes275Views0likes0CommentsSimplify private connectivity for Azure Databricks Serverless with Private Network Gateway
Recently, we announced the public preview of Private Network Gateway (PNG) for Azure Databricks - a simpler way to connect Azure Databricks serverless workloads to resources accessible from your Azure Virtual Network. With a single managed connection into your network, PNG helps reduce connectivity complexity while letting you control which resources serverless workloads can access. In this post, we explain what PNG means for private connectivity and how to get started. PNG brings the serverless experience together with the Azure networking environment and controls organizations already use, helping make serverless practical for a broader set of enterprise workloads. One connection to your network Azure Databricks serverless compute runs in an Azure Databricks managed network. This removes infrastructure management overhead, but private resources still need an explicit network path from serverless compute. Today, connecting serverless compute to resources you manage such as databases, Kafka clusters, and internal APIs could require separate private endpoints, load balancers, and proxy VMs for individual resources. That complexity grows as workloads need to reach more services across virtual networks, peered networks, and on-premises environments. Private Network Gateway introduces a simpler model: one gateway to your network instead of separately creating connectivity infrastructure for each resource. With Private Network Gateway on Azure Databricks, you can: Simplify private data ingestion. Use Lakeflow Connect on serverless compute to ingest from supported databases across your virtual network and connected on-premises environments. Connect AI agents to private services. An agent built on Azure Databricks can call a private MCP server you host. Use private package repositories. Install Python libraries from a private PyPI, Artifactory, or Nexus repository inside your network. Route egress through your existing firewall. Send traffic through your firewall or content inspection service so egress can be filtered and logged using your own tools. Use your own source IP. SaaS applications such as Salesforce can see a source IP belonging to your organization. Getting started is straightforward Private Network Gateway is designed to simplify the setup needed to bring serverless compute into your network. You delegate one subnet in your Azure Virtual Network to Azure Databricks. Private Network Gateway then establishes a managed, encrypted gateway from serverless compute into that subnet. The gateway is configured within a Network Connectivity Configuration (NCC), the account-level object used for Azure Databricks serverless networking. You can attach the NCC to your workspaces, and supported serverless workloads can then reach the private destinations you allow. And because the gateway is managed for you, there are no gateway VMs for you to deploy, patch, or scale in your network. Keep control of your Azure network Simplifying connectivity doesn’t mean giving up control of your network boundary. Private Network Gateway provides a dedicated path into your network rather than a shared appliance carrying other customers’ traffic. The gateway applies workspace-level access controls before traffic leaves, and isolation is enforced through encapsulation and network security rules. Once traffic reaches your subnet, your existing Azure networking controls continue to determine what serverless workloads can reach. This includes your route tables, network security groups, DNS resolvers, and security appliances. You also control how much traffic travels through the gateway: Specific destinations: Send only the fully qualified domain names you specify through Private Network Gateway. Other traffic retains its normal path. All traffic: Route outbound traffic through your network, except traffic matching a more specific Private Link or service-endpoint path. This can be used when you want broader egress inspection through your firewall or want traffic to leave through your own IP addresses. Private Network Gateway is also additive to your existing connectivity architecture. It sits alongside the private connectivity and egress controls you already use and can be enabled or removed without changing the existing setup. Bringing serverless to more of your Azure Databricks workloads Azure Databricks combines Databricks capabilities with native integration across the Azure ecosystem. Private Network Gateway extends that Azure Databricks experience to private networking for serverless workloads. Instead of building connectivity infrastructure for every resource, you can establish one managed gateway from serverless compute to your network and keep using your existing Azure network controls to decide where traffic goes. The result is a simpler approach to connecting serverless workloads to private enterprise resources while keeping the network boundary in your control. Get started today Private network gateway - Azure Databricks | Microsoft Learn Introducing Private Network Gateway | Databricks Blog768Views1like0CommentsAzure Databricks at Databricks Data + AI Summit 2026: updates and new announcements
Databricks Data + AI Summit brings together the global data and AI community in San Francisco to share product news, technical breakthroughs, and customer stories. This year, as usual, we have a lot of Azure Databricks announcements, a strong presence across the event, and a continued focus on helping customers put their data to work across analytics, AI, and enable business productivity. Find us at Data and AI Summit As a Legend Sponsor and Databricks’ long-standing strategic partner, Microsoft is joining Databricks Data + AI Summit during the keynote, multiple breakout sessions, and at the Expo booth. We're also engaging with customers 1:1 to hear from you. Satya Nadella will join Ali Ghodsi, CEO Databricks, in a pre-recorded keynote conversation on the importance of data in AI implementation and the deep integrations we co-engineer. We encourage you to visit us at the Microsoft Booth (Booth # 103) on the Expo floor to chat with the Azure Databricks team, see demos, and learn more about the recent announcements. Azure Databricks Breakout Sessions Unlocking the Microsoft Data & AI Ecosystem with Azure Databricks: From Insight to Impact Wednesday, June 17 | 1:50 PM – 2:30 PM PDT | Speaker: Anavi Nahar, Head of Product, Azure Data Lake Storage & Azure Databricks, Microsoft In today’s data-driven landscape, organizations need more than analytics—they need a unified platform that turns raw data into actionable intelligence across the Microsoft ecosystem. This session explores how Azure Databricks serves as the backbone of modern data architecture, integrating with core Microsoft cloud services and platforms to accelerate innovation. Learn how to use Azure Databricks for scalable data engineering, advanced analytics, and AI-driven solutions while enabling real-time collaboration and governance. Through practical examples and architectural patterns, we’ll show how to eliminate data silos, optimize performance, and empower teams to deliver insights faster. Zero-Copy Federated Energy Analytics: ADME + Databricks in Action Wednesday, June 17 | 12:40 PM - 1:20 PM PDT | Speaker: Andy Corran, Principal Product Manager, Azure Databricks, Microsoft Oil and gas companies have standardized on Azure Data Manager for Energy (ADME) as their subsurface system of record, but running analytics and AI on that data has meant copying massive datasets into downstream platforms, breaking governance and slowing every workflow that follows. In this jointly developed Microsoft and Databricks session, we introduce a new zero‑copy, federated path that brings Databricks compute directly to data, with native governance and serverless scale. We walk through the architecture, show the solution in action against live ADME, and share how operators across the industry are accelerating subsurface analytics while keeping ADME as the single source of truth. Unity Catalog External Locations: Extending Governance to OneLake and Beyond Wednesday, Jun 17 | 5:20 PM - 5:40 PM PDT | Speaker: Ljubica Vujovic Boskovic, Senior Product Manager, Databricks In this session, we'll show how External Locations provide a consistent, extensible pattern for connecting Databricks to any storage platform — and walk through what it takes to create External Location for Microsoft OneLake. You'll see the architecture, the setup end-to-end, and a demo reading and writing UC-governed assets directly into OneLake storage without needing to setup any ETL pipelines. Latest announcements We recently announced new ways to build AI apps and agents with Azure Databricks, Copilot Studio, and GitHub Copilot, including authoring Copilot Studio agents that reason over an entire Azure Databricks workspace through one MCP connection. At Microsoft Build, PepsiCo also shared its blueprint for agentic AI, illustrating how Azure Databricks can provide the data foundation for agentic apps. This week’s announcements make it easier to use Azure Databricks with the Microsoft tools your teams rely on every day, including Microsoft Teams, M365 Copilot, Excel, SharePoint, Power BI, and OneLake: Genie for Microsoft Teams and M365 Copilot (Beta): You can tag Genie in a Teams thread and get a context-aware answer from your Azure Databricks lakehouse without leaving the conversation. Responses are governed by Unity Catalog, so each answer is scoped to what the user is permitted to see. It’s part of the broader Genie One experience for report generation, reusable agents, low-code apps, and natural-language pipeline design. See it in action in the Databricks + Microsoft co-authored training in AI Skills Navigator Genie in Copilot Cowork (Beta): Available today, Databricks Genie works seamlessly with M365 Copilot Cowork. This integration will allow teams to anchor Cowork’s tasks with the Genie Ontology, bringing trusted data intelligence straight into their workflows Azure Databricks Excel Add-in (Public Preview): This brings governed lakehouse data into Excel without SQL or per-user ODBC setup. Unity Catalog metric views let business logic be defined once and stay consistent across tools, and the add-in supports write-back, so permitted users can push updates from Excel into Databricks. Learn how to set it up. SharePoint Connector (Beta) via Lakeflow Connect. A fully managed connector for point-and-click ingestion pipelines that bring SharePoint content — structured sheets and unstructured PDFs, Word docs, and PowerPoints — into Delta tables, keeping downstream analytics, Genie spaces, and Excel workbooks supplied with current data. Read the documentation here. Azure Databricks OneLake Catalog Federation (Generally Available): The ability to query OneLake data directly from Azure Databricks without pipelines, duplication, or data movement is generally available. This announcement coupled with the Azure Databricks Mirrored Catalog item enable bidirectional READ from Azure Databricks and OneLake. Learn more here Storing Unity Catalog Managed Tables in OneLake (Beta): You can now customers can use OneLake as a storage location option for Unity Catalog tables in addition to Azure Data Lake Storage (ADLS). Read more on how to do this here. CustomerLake: a customer data platform inside the lakehouse Introducing CustomerLake, a Customer Data Platform (CDP) built directly within the lakehouse rather than as a separate application. CustomerLake is now available in Azure Databricks. Two kinds of agents do much of the work: Profile Agents help assemble business-ready Customer 360 profiles from fragmented sources, reducing the manual effort of stitching customer data together. Campaign Agents give marketing teams a workspace to segment audiences, recommend next-best actions, activate across channels, and continuously optimize personalized experiences. Because CustomerLake runs inside your governed storage boundary, customer data, AI models, and governance stay together — avoiding much of the data movement and duplication that come with connecting separate marketing tools. For Azure customers, that means building customer engagement on the same governed lakehouse foundation they already use for analytics and AI, rather than maintaining a parallel stack. “What excites us most about the CustomerLake and the new CDP capability is the ability to bring customer data together in a way that is actionable, timely, and scalable. By creating a more complete view of each customer, we can better understand behaviors, preferences, and needs across channels, which will help us deliver more personalized experiences and more relevant offers. Ultimately, we see this as a powerful step toward stronger engagement, deeper loyalty, and better outcomes for both our business and our customers.” Jay Malepati Global Director of Data Science, Circle K All of these announcements benefit from built in Governance with Azure Databricks Unity Catalog. By connecting governed lakehouse data to the Microsoft tools your teams already use — Teams, M365 Copilot, Excel, SharePoint, OneLake, and Power BI — these updates make it easier to put trusted AI to work on Azure. To learn more, explore the Azure Databricks documentation and try these capabilities in your own workspace.2.1KViews1like0CommentsIntroducing Lakeflow Connect Free Tier, now available on Azure Databricks
We're excited to introduce the Lakeflow Connect Free Tier on Azure Databricks, so you can easily bring your enterprise data into your lakehouse to build analytics and AI applications faster. Modern applications require reliable access to operational data, especially for training analytics and AI agents, but connecting and gathering data across silos can be challenging. With this new release, you can seamlessly ingest all of your enterprise data from SaaS and database sources to unlock data intelligence for your AI agents. Ingest millions of records per day, per workspace for free This new Lakeflow Connect Free Tier provides 100 DBUs per day, per workspace, which allows you to ingest approximately 100 million records* from many popular data sources**, including SaaS applications and databases. Unlock your enterprise data for free with Lakeflow Connect This new offering provides all the benefits of Lakeflow Connect, eliminating the heavy lifting so your teams can focus on unlocking data insights instead of managing infrastructure. In the past year, Databricks has continued rolling out several fully managed connectors, supporting popular data sources. The free tier supports popular SaaS applications (Salesforce, ServiceNow, Google Analytics, Workday, Microsoft Dynamics 365), and top-used databases (SQL Server, Oracle, Teradata, PostgreSQL, MySQL, Snowflake, Redshift, Synapse, and BigQuery). Lakeflow Connect benefits include: Simple UI: Avoid complex setups and architectural overhead, these fully managed connectors provide a simple UI and API to democratize data access. Automated features also help simplify pipeline maintenance with minimal overhead. Efficient ingestion: Increase efficiency and accelerate time to value. Optimized incremental reads and writes and data transformation help improve the performance and reliability of your pipelines, reduce bottlenecks, and reduce impact to the source data for scalability. Unified with the Databricks Platform: Create ingestion pipelines with governance from Unity Catalog, observability from Lakehouse Monitoring, and seamless orchestration with Lakeflow Jobs for analytics, AI and BI. Availability The Lakeflow Connect Free Tier is available starting today on Azure Databricks. If you are at FabCon in Atlanta, Accelerating Data and AI with Azure Databricks on Thursday, March 19th, 8:00–9:00 AM, room C302 to see how these capabilities come together to accelerate performance, simplify architecture, and maximize value on Azure Getting Started Resources To learn more about the Lakeflow Connect Free Tier and Lakeflow Connect, review our pricing page, and documentation. Get started ingesting your data today for free, signup with an Azure free account. Get started with Azure Databricks for free Product tour: Databricks Lakeflow Connect for Salesforce: Powering Smarter Selling with AI and Analytics Product tour: Effortless ServiceNow Data Ingestion with Databricks Lakeflow Connect Product tour: Simplify Data Ingestion with Lakeflow Connect: From Google Analytics to AI On-demand video: Use Lakeflow Connect for Salesforce to predict customer churn On-demand video: Databricks Lakeflow Connect for Workday Reports: Connect, Ingest, and Analyze Workday Data Without Complexity On-demand video: Data Ingestion With Lakeflow Connect —-- * Your actual ingestion capacity will vary based on specific workload characteristics, record sizes, and source types. ** Excludes Zerobus Ingest, Auto Loader and other self-managed connectors. Customer will continue to incur charges for underlying infrastructure consumption from the cloud vendor.5KViews0likes0CommentsNear–Real-Time CDC to Delta Lake for BI and ML with Lakeflow on Azure Databricks
The Challenge: Too Many Tools, Not Enough Clarity Modern data teams on Azure often stitch together separate orchestrators, custom streaming consumers, hand-rolled transformation notebooks, and third-party connectors — each with its own monitoring UI, credential system, and failure modes. The result is observability gaps, weeks of work per new data source, disconnected lineage, and governance bolted on as an afterthought. Lakeflow, Databricks’ unified data engineering solution, solves this by consolidating ingestion, transformation, and orchestration natively inside Azure Databricks — governed end-to-end by Unity Catalog. Component What It Does Lakeflow Connect Point-and-click connectors for databases (using CDC), SaaS apps, files, streaming, and Zerobus for direct telemetry Lakeflow Spark Declarative Pipelines Declarative ETL with AutoCDC, data quality enforcement, and automatic incremental processing Lakeflow Jobs Managed orchestration with 99.95% uptime, a visual task DAG, and repair-and-rerun Architecture Step 1: Stream Application Telemetry with Zerobus Ingest Zerobus Ingest, part of Lakeflow Connect, lets your application push events directly to a Delta table over gRPC — no message bus, no Structured Streaming job. Sub-5-second latency, up to 100 MB/sec per connection, immediately queryable in Unity Catalog. Prerequisites Azure Databricks workspace with Unity Catalog enabled and serverless compute on A service principal with write access to the target table Setup First, create the target table in a SQL notebook: CREATE CATALOG IF NOT EXISTS prod; CREATE SCHEMA IF NOT EXISTS prod.bronze; CREATE TABLE IF NOT EXISTS prod.bronze.telemetry_events ( event_id STRING, user_id STRING, event_type STRING, session_id STRING, ts BIGINT, page STRING, duration_ms INT ); 1. Go to Settings → Identity and Access → Service Principals → Add service principal 2. Open the service principal → Secrets tab → Generate secret. Save the Client ID and secret. 3. In a SQL notebook, grant access: GRANT USE CATALOG ON CATALOG prod TO `<client-id>`; GRANT USE SCHEMA ON SCHEMA prod.bronze TO `<client-id>`; GRANT MODIFY, SELECT ON TABLE prod.bronze.telemetry_events TO `<client-id>`; 4. Derive your Zerobus endpoint from your workspace URL: <workspace-id>.zerobus.<region>.azuredatabricks.net (The workspace ID is the number in your workspace URL, e.g. adb-**1234567890**.12.azuredatabricks.net) 5. Install the SDK: pip install databricks-zerobus-ingest-sdk 6. In your application, open a stream and push records: from zerobus.sdk.sync import ZerobusSdk from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties sdk = ZerobusSdk("<workspace-id>.zerobus.<region>.azuredatabricks.net", "https://<workspace-url>") stream = sdk.create_stream( "<client-id>", "<client-secret>", TableProperties("prod.bronze.telemetry_events"), StreamConfigurationOptions(record_type=RecordType.JSON) ) stream.ingest_record({"event_id": "e1", "user_id": "u42", "event_type": "page_view", "ts": 1700000000000}) stream.close() 7. Verify in Catalog → prod → bronze → telemetry_events → Sample Data Step 2: Ingest from On-Premises SQL Server via CDC Lakeflow Connect reads SQL Server's transaction log incrementally — no full table scans, no custom extraction software. Connectivity to your on-prem server is over Azure ExpressRoute. Prerequisites SQL Server reachable from Databricks over ExpressRoute (TCP port 1433) CDC enabled on the source database and tables (see setup below) A SQL login with CDC read permissions on the source database Databricks: CREATE CONNECTION privilege on the metastore; USE CATALOG, CREATE TABLE on the destination catalog Setup Enable CDC on SQL Server: USE YourDatabase; EXEC sys.sp_cdc_enable_db; EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'orders', @role_name = NULL; EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'customers', @role_name = NULL; Configure the connector in Databricks: Click Data Ingestion in the sidebar (or + New → Add Data) Select SQL Server from the connector list Ingestion Gateway page — enter a gateway name, select staging catalog/schema, click Next Ingestion Pipeline page — name the pipeline, click Create connection: Host: your on-prem IP (e.g. 10.0.1.50) · Port: 1433 · Database: YourDatabase Enter credentials, click Create, then Create pipeline and continue Source page — expand the database tree, check dbo.orders and dbo.customers; optionally enable History tracking (SCD Type 2) per table. Set Destination table name to orders_raw and customers_raw respectively. Destination page — set catalog: prod, schema: bronze, click Save and continue Settings page — set a sync schedule (e.g. every 5 minutes), click Save and run pipeline Step 3: Transform with Spark Declarative Pipelines The Lakeflow Pipelines Editor is an IDE built for developing pipelines in Lakeflow Spark Declarative Pipelines (SDP), and lets you define Bronze → Silver → Gold in SQL. SDP then handles incremental execution, schema evolution, and lineage automatically. Prerequisites Bronze tables populated (from Steps 1 and 2) CREATE TABLE and USE SCHEMA privileges on prod.silver and prod.gold Setup 1. In the sidebar, click Jobs & Pipelines → ETL pipeline → Start with an empty file → SQL 2. Rename the pipeline (click the name at top) to lakeflow-demo-pipeline 3. Paste the following SQL: -- Silver: latest order state (SCD Type 1) CREATE OR REFRESH STREAMING TABLE prod.silver.orders; APPLY CHANGES INTO prod.silver.orders FROM STREAM(prod.bronze.orders_raw) KEYS (order_id) SEQUENCE BY updated_at STORED AS SCD TYPE 1; -- Silver: full customer history (SCD Type 2) CREATE OR REFRESH STREAMING TABLE prod.silver.customers; APPLY CHANGES INTO prod.silver.customers FROM STREAM(prod.bronze.customers_raw) KEYS (customer_id) SEQUENCE BY updated_at STORED AS SCD TYPE 2; -- Silver: telemetry with data quality check CREATE OR REFRESH STREAMING TABLE prod.silver.telemetry_events ( CONSTRAINT valid_event_type EXPECT (event_type IN ('page_view', 'add_to_cart', 'purchase')) ON VIOLATION DROP ROW ) AS SELECT * FROM STREAM(prod.bronze.telemetry_events); -- Gold: materialized view joining all three Silver tables CREATE OR REFRESH MATERIALIZED VIEW prod.gold.customer_activity AS SELECT o.order_id, o.customer_id, c.customer_name, c.email, o.order_amount, o.order_status, COUNT(e.event_id) AS total_events, SUM(CASE WHEN e.event_type = 'purchase' THEN 1 ELSE 0 END) AS purchase_events FROM prod.silver.orders o LEFT JOIN prod.silver.customers c ON o.customer_id = c.customer_id LEFT JOIN prod.silver.telemetry_events e ON CAST(o.customer_id AS STRING) = e.user_id -- user_id in telemetry is string GROUP BY o.order_id, o.customer_id, c.customer_name, c.email, o.order_amount, o.order_status; 4. Click Settings (gear icon) → set Pipeline mode: Continuous → Target catalog: prod → Save 5. Click Start — the editor switches to the live Graph view Step 4: Govern with Unity Catalog All tables from Steps 1–3 are automatically registered in Unity Catalog, Databricks’ built-in governance and security offering, with full lineage. No manual registration needed. View lineage Go to Catalog → prod → gold → customer_activity Click the Lineage tab → See Lineage Graph Click the expand icon on each upstream node to reveal the full chain: Bronze sources → Silver → Gold Set Permissions -- Grant analysts read access to the Gold layer only GRANT SELECT ON TABLE prod.gold.customer_activity TO `analysts@contoso.com`; -- Mask PII for non-privileged users CREATE FUNCTION prod.security.mask_email(email STRING) RETURNS STRING RETURN CASE WHEN is_account_group_member('data-engineers') THEN email ELSE CONCAT(LEFT(email, 2), '***@***.com') END; ALTER TABLE prod.silver.customers ALTER COLUMN email SET MASK prod.security.mask_email; Step 5: Orchestrate and Monitor with Lakeflow Jobs Wire the Connect pipeline and SDP pipeline into a single job with dependencies, scheduling, and alerting, all from the UI with Lakeflow Jobs. Prerequisites Pipelines from Steps 2 and 3 saved in the workspace Setup Go to Jobs & Pipelines → Create → Job Task 1: click the Pipeline tile → name it ingest_sql_server_cdc → select your Lakeflow Connect pipeline → Create task Task 2: click + Add task → Pipeline → name it transform_bronze_to_gold → select lakeflow-demo-pipeline → set Depends on: ingest_sql_server_cdc → Create task In the Job details panel on the right: click Add schedule → set frequency → add email notification on failure → Save Click Run now to trigger a run, then click the run ID to open the Run detail view For health monitoring across all jobs, query system tables in any notebook or SQL warehouse: SELECT job_name, result_state, DATEDIFF(second, start_time, end_time) AS duration_sec FROM system.lakeflow.job_run_timeline WHERE start_time >= CURRENT_TIMESTAMP - INTERVAL 24 HOURS ORDER BY start_time DESC; Step 6: Visualize with AI/BI Dashboards and Genie AI/BI Dashboard helps you create AI-powered, low-code dashboards. Click + New → Dashboard Click Add a visualization, connect to prod.gold.customer_activity, and build charts Click Publish — viewers see data under their own Unity Catalog permissions automatically Genie allows you to interact with their data using natural language 1. In the sidebar, click Genie → New 2. On Choose data sources, select prod.gold.customer_activity → Create 3. Add context in the Instructions box (e.g., table relationships, business definitions) 4. Switch to the Chat tab and ask a question: "Which customers have the highest total events and what were their order amounts?" 5. Genie generates and executes SQL, returning a result table. Click View SQL to inspect the query. Everything in One Platform Capability Lakeflow Previously Required Telemetry ingestion Zerobus Ingest Message bus + custom consumer Database CDC Lakeflow Connect Custom scripts or 3rd-party tools Transformation + AutoCDC Spark Declarative Pipelines Hand-rolled MERGE logic Data quality SDP Expectations Separate validation tooling Orchestration Lakeflow Jobs External schedulers (Airflow, etc.) Governance Unity Catalog Disconnected ACLs and lineage Monitoring Job UI + System Tables Separate APM tools BI + NL Query AI/BI Dashboards + Genie External BI tools Customers seeing results on Azure Databricks: Ahold Delhaize — 4.5x faster deployment and 50% cost reduction running 1,000+ ingestion jobs daily Porsche Holding — 85% faster ingestion pipeline development vs. a custom-built solution Next Steps Lakeflow product page Lakeflow Connect documentation Live demos on Demo Center Get started with Azure Databricks936Views0likes0CommentsAzure Databricks Lakebase is now generally available
Modern applications are built on real-time, intelligent, and increasingly powered by AI agents that need fast, reliable access to operational data—without sacrificing governance, scale, or simplicity. To solve for this, Azure Databricks Lakebase introduces a serverless, Postgres database architecture that separates compute from storage and integrates natively with the Databricks Data Intelligence Platform on Azure. Lakebase is now generally available in Azure Databricks enabling you and your team to start building and validating real-time and AI-driven applications directly on your lakehouse foundation. Why Azure Databricks Lakebase? Lakebase was created for modern workloads and reduce silos. By decoupling compute from storage, Lakebase treats infrastructure as an on-demand service—scaling automatically with workload needs and scaling to zero when idle. Key capabilities include: Serverless Postgres for Production Workloads: Lakebase delivers a managed Postgres experience with predictable performance and built-in reliability features suitable for production applications, while abstracting away infrastructure management. Instant Branching and Point-in-Time Recovery: Teams can create zero-copy branches of production data in seconds for testing, debugging, or experimentation, and restore databases to precise points in time to recover from errors or incidents. Unified Governance with Unity Catalog: Operational data in Lakebase can be governed using the same Unity Catalog policies that secure analytics and AI workloads, enabling consistent access control, auditing, and compliance across the platform. Built for AI and Real-Time Applications: Lakebase is designed to support AI-native patterns such as real-time feature serving, agent memory, and low-latency application state—while keeping data directly connected to the lakehouse for analytics and learning workflows. Lakebase allows applications to operate directly on governed, lake-backed data—reducing complexity with pipeline synchronization or duplicating storage On Azure Databricks, this unlocks new scenarios such as: Real-time applications built on lakehouse data AI agents with persistent, governed memory Faster release cycles with safe, isolated database branches Simplified architectures with fewer moving parts All while using familiar Postgres interfaces and tools. Get Started with Azure Databricks Lakebase Lakebase is integrated into the Azure Databricks experience and can be provisioned directly within Azure Databricks workspaces. For Azure Databricks customers building intelligent, real-time applications, it offers a new foundation—one designed for the pace and complexity of modern data-driven systems. We’re excited to see what you build, get started today!1.7KViews0likes0CommentsServerless Workspaces are generally available in Azure Databricks
Recently, we announced that Serverless Workspaces in public preview. Today, we are excited to share that Serverless Workspaces are generally available in Azure Databricks. Azure Databricks now offers two workspace models: Serverless and Classic. With Serverless, Azure Databricks operates and maintains the entire environment on your behalf. You still configure governance elements like Unity Catalog, identity federation, and workspace-level policies, but the heavy lifting of infrastructure setup disappears. As a result, teams can start building immediately instead of waiting on networking or compute provisioning. Classic workspaces take the opposite approach. When creating a Classic workspace, you must design and deploy the full networking layout, determine how compute should be managed, establish storage patterns, and configure all inbound and outbound connectivity. These decisions are critical and have benefits in heavily regulated or secure industries, but they may create overhead for teams who simply want to start working with data. A Serverless Workspace eliminates that overhead entirely. Once created, it’s ready for use—no virtual network design, no storage configuration, and no cluster management. Serverless workspaces use serverless compute and default storage. Unity Catalog is automatically provisioned so that the same governance model applies. Key capabilities and consideration of Serverless workspaces Storage: Each Serverless workspace includes fully managed object storage called default storage. You can build managed catalogs, volumes, and tables without supplying your own storage accounts or credentials. Features like multi-key projection and restricted object-store access ensure that only authorized users can work with the data. Classic compute cannot interact with data assets in default storage. If you already have an Azure Blob Storage account (likely with hierarchical namespace enabled) or if your organization requires using your own storage due to security or compliance requirements, you can also create a connection between your storage account and Serverless Workspace. Compute: Workloads run on automatically provisioned serverless compute—no need to build or maintain clusters. Azure Databricks handles scaling and resource optimization so users can focus purely on data and analytics. Network: There’s no requirement to deploy NAT gateways, firewalls, or Private Link endpoints. Instead, you define serverless egress rules and serverless Private Link controls that apply uniformly to all workloads in the workspace. Unity Catalog access: Governed data remains accessible from the new workspace with existing permissions intact. Your data estate stays consistent and secure. Choosing between Serverless and Classic Azure Databricks supports both workspace types so organizations can select what best matches their needs. Use Serverless when rapid workspace creation, minimal configuration, and reduced operational overhead are priorities. It’s the fastest path to a fully governed environment. Use Classic when you require a custom VNet design, specific network topologies, finer grain security controls, or features that are not yet available in the serverless model. Some organizations also simply prefer to manage Azure resources directly, making Classic workspaces a suitable option. Note that Azure Databricks serverless workspaces are only available in regions that support serverless compute. To learn get started create your Serverless workspace today!1.1KViews0likes0CommentsGeneral Availability: Automatic Identity Management (AIM) for Entra ID on Azure Databricks
In February, we announced that Automatic Identity Management in public preview and loved to hear your overwhelmingly positive feedback. Prior to public preview, you either had to set up an Entra Enterprise Application or involve an Azure Databricks account admin to import the appropriate groups. This required manual steps whether it was adding or removing users with organizational changes, maintaining scripts, or requiring additional Entra or SCIM configuration. Identity management was thus cumbersome and required management overhead. Today, we are excited to announce that Automatic Identity management (AIM) for Entra ID on Azure Databricks is generally available. This means no manual user setup is needed and you can instantly add users to your workspace(s). Users, groups, and service principals from Microsoft Entra ID are automatically available within Azure Databricks, including support for nested groups and dashboards. This native integration is one of the many reasons Databricks runs best on Azure. Here are some addition ways AIM could benefit you and your organization: Seamlessly share dashboards You can share AI/BI dashboards with any user, service principal, or group in Microsoft Entra ID immediately as these users are automatically added to the Azure Databricks account upon login. Members of Microsoft Entra ID who do not have access to the workspace are granted access to a view-only copy of a dashboard published with embedded credentials. This enables you to share dashboards with users outside your organization, too. To learn more, see share a dashboard. Updated defaults for new accounts All new Azure Databricks accounts have AIM enabled – no opt in or additional configuration required. For existing accounts, you can enable AIM with a single click in the Account Admin Console. Soon, we will also make this the default for existing accounts. Automation at scale enabled via APIs You can also register users, groups, or service principles in Microsoft Entra ID via APIs. Being able to do this programmatically enables the enterprise scale most of our customers need. You can also enable automation via scripts leveraging these APIs. Read the Databricks blog here and get started via documentation today!2.5KViews1like0Comments