Secure Medallion Architecture Pattern on Azure Databricks (Part I)
This article presents a security-first pattern for Azure Databricks: a Medallion Architecture where Bronze, Silver and Gold each run as their Lakeflow Job and cluster, orchestrated by a parent job. Run-as identities are Microsoft Entra service principals; storage access is governed via Unity Catalog External Locations backed by the Access Connector’s managed identity. Least-privilege is enforced with cluster policies and UC grants. Prefer managed tables to unlock Predictive Optimisation, Automatic liquid clustering and Automatic statistics. Secrets live in Azure Key Vault and are read at runtime. Monitor reliability and cost with system tables and Jobs UI. Part II covers more low-level concepts and CI/CD.Securing Azure Databricks Serverless: Practical Guide to Private Link Integration
The Challenge: Outbound Control in a Serverless World Serverless compute resources run in the serverless compute plane, and is managed by Microsoft, for ease of use. Databricks serverless provides hassle-free compute for running notebooks, jobs, and pipelines and by default, outbound traffic can reach the internet and other networks freely. One of the most common security requirements for customers in the financial and government sectors is to have the ability to retain network paths within a private network for their users to access their data and for system integration. Solution Objective Enforce deny-by-default posture: Control outbound access with granular precision by enabling a deny-by-default policy for internet. By default, all outbound access is blocked unless explicitly allowed via Private Endpoint Rules. Control outbound connections: by specifying allowed locations, connections, FQDN Enforce the traffic to go over the customer network for traffic controls and inspection Solution Overview The solution is designed to route Databricks Serverless outbound traffic to the customer customer-managed Policy Enforcement Point (e.g. Azure Firewall), to allow the customer to securely connect to services hosted on the cloud without exposing the data to the public internet. Essentially, it establishes a private, secure connection between Databricks Control Plane to the customer virtual network. Pre-Requisites Azure Firewall Deploy an Azure Firewall if you don’t already have one. Virtual Networks and Subnets Create a VNET for Databricks and Load Balancer deployment. Set up subnets for the Azure Standard Load Balancer frontend (e.g., 10.0.2.0/26) and backend (e.g., 10.0.2.64/26). Enable Private Endpoint network policy for Network Security Groups and Route Tables on the backend subnet. VNET Peering Peer the Databricks VNET with your hub VNET to allow secure routing. Azure Databricks Workspace Deploy an Azure Databricks workspace if you don’t have one. Follow the official Azure Databricks documentation for detailed steps on creating workspaces and private endpoints Summary of Steps Deploy Azure Firewall and Networking Set up an Azure Firewall and create the necessary virtual networks (VNets) and subnets for your environment. Peer the Databricks VNet with your hub VNet to enable secure routing. Configure the Azure Load Balancer Create an internal Standard Load Balancer. Set up frontend and backend pools using NICs (not IP addresses). Add load balancing rules and configure a health probe (typically HTTP on port 8082). Create a Private Link Service Deploy the Private Link Service behind the load balancer. Associate it with the correct frontend and backend subnets. Set Up Route Tables Create route tables to direct backend VM traffic to the Azure Firewall. Ensure the route tables are associated with the correct subnets (e.g., backend subnet for the router VM). Deploy and Configure the Router VM Deploy a Linux VM to act as a router. Enable IP forwarding on the VM and in Azure settings. Configure IPTables for NAT and traffic forwarding. Install and configure NGINX to serve as a health probe for the load balancer. Configure Network Security Groups (NSGs) Set up NSGs to allow necessary traffic (SSH, load balancer, HTTP/HTTPS, health probe) to and from the router VM. Configure Azure Firewall Application Rules Define application rules to allow outbound access only to approved FQDNs (e.g., microsoft.com). Block all other outbound traffic by default. Configure Databricks Account Portal Enable outbound (serverless) Azure Private Link to customer-managed resources in the Databricks Account Portal. Create Network Connectivity Configurations (NCCs) and attach them to your workspaces. Add private endpoint rules for each Azure resource you want to allow access to. Approve Private Endpoints In the Azure Portal, approve the private endpoint connections created by Databricks for your resources. Troubleshooting Use tools like netstat, conntrack, and tcpdump on the router VM to diagnose routing issues. Double-check route table and NSG associations. Validate private endpoint rule configurations in both Databricks and the Azure Portal. References Serverless compute plane networking - Azure Databricks | Microsoft Learn Configure private connectivity to Azure resources - Azure Databricks | Microsoft Learn Key Takeaway This solution enforces a deny-by-default posture for outbound traffic from Azure Databricks Serverless, only allowing explicitly approved connections via Private Endpoints. All traffic is routed through your network for inspection and control, helping you meet strict compliance and security requirements. Ready to Get Started? Securing your Databricks Serverless environment doesn’t have to be daunting. With Azure Private Link, Azure Firewall, and a smart configuration, you get the best of both worlds: agility and airtight security. For more details, check out the official Azure Databricks documentation and start building your secure analytics platform today. Questions or want to share your experience? Drop a comment below or reach out to the Azure Databricks community.Approaches to Integrating Azure Databricks with Microsoft Fabric: The Better Together Story!
Azure Databricks and Microsoft Fabric can be combined to create a unified and scalable analytics ecosystem. This document outlines eight distinct integration approaches, each accompanied by step-by-step implementation guidance and key design considerations. These methods are not prescriptive—your cloud architecture team can choose the integration strategy that best aligns with your organization’s governance model, workload requirements and platform preferences. Whether you prioritize centralized orchestration, direct data access, or seamless reporting, the flexibility of these options allows you to tailor the solution to your specific needs.Announcing the new Databricks Job activity in ADF!
We’re excited to announce that Azure Data Factory now supports the orchestration of Databricks Jobs! Databrick Jobs allow you to schedule and orchestrate a task or multiple tasks in a workflow in your Databricks workspace. Since any operation in Databricks can be a task, this means you can now run anything in Databricks via ADF, such as serverless jobs, SQL tasks, Delta Live Tables, batch inferencing with model serving endpoints, or automatically publishing and refreshing semantic models in the Power BI service. And with this new update, you’ll be able to trigger these workflows from your Azure Data Factory pipelines. To make use of this new activity, you’ll find a new Databricks activity under the Databricks activity group called Job. Once you’ve added the Job activity (Preview) to your pipeline canvas, you can connect to your Databricks workspace and configure the settings to select your Databricks job, allowing you to run the Job from your pipeline. We also know that allowing parameterization in your pipelines is important as it allows you to create generic reusable pipeline models. ADF continues to provide support for these patterns and is excited to extend this capability to the new Databricks Job activity. Under the settings of your Job activity, you’ll also be able to configure and set parameters to send to your Databricks job, allowing maximum flexibility and power for your orchestration jobs. To learn more, read Azure Databricks activity - Microsoft Fabric | Microsoft Learn. Have any questions or feedback? Leave a comment below!General Availability: Automatic Identity Management (AIM) for Entra ID on Azure Databricks
In February, we announced that Automatic Identity Management in public preview and loved to hear your overwhelmingly positive feedback. Prior to public preview, you either had to set up an Entra Enterprise Application or involve an Azure Databricks account admin to import the appropriate groups. This required manual steps whether it was adding or removing users with organizational changes, maintaining scripts, or requiring additional Entra or SCIM configuration. Identity management was thus cumbersome and required management overhead. Today, we are excited to announce that Automatic Identity management (AIM) for Entra ID on Azure Databricks is generally available. This means no manual user setup is needed and you can instantly add users to your workspace(s). Users, groups, and service principals from Microsoft Entra ID are automatically available within Azure Databricks, including support for nested groups and dashboards. This native integration is one of the many reasons Databricks runs best on Azure. Here are some addition ways AIM could benefit you and your organization: Seamlessly share dashboards You can share AI/BI dashboards with any user, service principal, or group in Microsoft Entra ID immediately as these users are automatically added to the Azure Databricks account upon login. Members of Microsoft Entra ID who do not have access to the workspace are granted access to a view-only copy of a dashboard published with embedded credentials. This enables you to share dashboards with users outside your organization, too. To learn more, see share a dashboard. Updated defaults for new accounts All new Azure Databricks accounts have AIM enabled – no opt in or additional configuration required. For existing accounts, you can enable AIM with a single click in the Account Admin Console. Soon, we will also make this the default for existing accounts. Automation at scale enabled via APIs You can also register users, groups, or service principles in Microsoft Entra ID via APIs. Being able to do this programmatically enables the enterprise scale most of our customers need. You can also enable automation via scripts leveraging these APIs. Read the Databricks blog here and get started via documentation today!Part 2: Performance Configurations for Connecting PBI to a Private Link ADB Workspace
This blog was written in conjunction with Leo Furlong, Lead Solutions Architect at Databricks. In Part 1, we discussed networking options for connecting Power BI to an Azure Databricks workspace with a Public Endpoint protected with a workspace IP Access List. In Part 2, we continue our discussion and elaborate on private networking options for an Azure Databricks Private Link workspace. When using Azure Databricks Private Link with Allow Public Network Access setting set to Disabled, all connections to the workspace must go through Private Endpoints. For one of the private networking options, we’ll also discuss how to configure your On-Premise Data Gateway VM to get good performance. Connecting Power BI to a Private Link Azure Databricks Workspaces As covered in Part 1, Power BI offers two primary methods for secure connections to data sources with private networking: 1. On-premises data gateway: An application that gets installed on a Virtual Machine that has a direct networking connection to a data source. It allows Power BI to connect to data sources that don’t allow public connections. The general flow of this setup entails: a. Create or leverage a set of Private Endpoints to the Azure Databricks workspace - both sub-resources for databricks_ui_api and browser_authentication are required b. Create or leverage a Private DNS Zone for privatelink.azuredatabricks.net c. Deploy an Azure VM into a VNet/subnet d. The VM’s VNet/subnet should have access to the Private Endpoints (PEs) via either them being in the same VNet or being peered with another VNet where they reside e. Install and configure the on-premise data gateway software on the VM f. Create a connection in the Power BI Service via Settings -> Manage Connections and Gateways UIs g. Configure the Semantic Model to use the connection under the Semantic Model’s settings and gateway and cloud connections sub-section 2. Virtual Network Data Gateway: A fully managed data gateway that gets created and managed by the Power BI service. Connections work by allowing Power BI to delegate into a VNet for secure connectivity to the data source. The general flow of this setup entails: a. Create or leverage a set of Private Endpoints (PEs) to the Azure Databricks workspace - both sub-resources for databricks_ui_api and browser_authentication are required b. Create or leverage a Private DNS Zone for privatelink.azuredatabricks.net c. Create a subnet in a VNet that has access to the Private Endpoints (PEs) via either them being in the same VNet or being peered with another VNet where they reside. Delegate the Subnet to Microsoft.PowerPlatform/vnetaccesslinks d. Create a virtual network data gateway in the Power BI Service via Settings -> Manage Connections and Gateways UIs e. Configure the the Semantic Model to use the connection under the Semantic Model’s settings and gateway and cloud connections sub-section The documentation for both options is fairly extensive, and this blog post will not focus on breaking down the configurations further. Instead, this post is about configuring your private connections to get the best Import performance. On-Premise Data Gateway Performance Testing In order to provide configuration guidance, a series of Power BI Import tests were performed using various configurations and a testing dataset. Testing Data The testing dataset used was a TPC-DS scale factor 10 dataset (you can create your own using this Repo). A scale factor of 10 in TPC-DS generates about 10 gigabytes (GB) of data. The TPC-DS dataset was loaded into Unity Catalog and the primary and foreign keys were created between the tables. A model was then created in the Power BI Service using the Publish to Power BI capabilities in Unity Catalog; the primary and foreign keys were used to automatically create relationships between the tables in the Power BI semantic model. Here’s an overview of the tables used in this dataset: Fabric Capacity An F64 Fabric Capacity was used in the West US region. The F64 was the smallest size available (in terms of RAM) for refreshing the model without getting capacity errors - the compressed Semantic Model size is 5,244 MB. Azure Databricks SQL Warehouse An Azure Databricks workspace using Unity Catalog was deployed in the East US 2 and West US regions for the performance tests. A Medium Databricks SQL Warehouse was used. For Imports, generally speaking, the size of the SQL Warehouse isn’t very important. Using an aggressive Auto Stop configuration of 5 minutes is ideal to minimize compute charges (1 minute can be used if the SQL Warehouse is deployed via an API). Testing Architecture The following diagram summarizes a simplified Azure networking architecture for the performance tests. A Power BI Semantic Model is connected to a Power BI On-Premise Data Gateway Connection The On-Premise Data Gateway Connection connects to the Azure Databricks workspace using Private Endpoints. Azure Databricks provisions up a Serverless SQL Warehouse in ~5 seconds within the Serverless Data Plane within Azure. SQL queries are executed on the Serverless SQL Warehouse. Unity Catalog gives the Serverless SQL Warehouse a read-only, down-scoped, and pre-signed URL to ADLS. Data is fetched from ADLS and placed on the Azure Databricks workspace’s managed storage account via a capability called Cloud Fetch. Arrow Files are pulled from Cloud Fetch and delivered to the Power BI Service through the Data Gateway. Data in the Semanic Model is compressed and stored in Vertipaq In-Memory storage. Testing Results The following grid outlines the scenarios tested and the results for each test. We’ll review the different configurations tested below in specific sections. Scenario Gateway Scenario Avg Refresh Duration Minutes A East US 2, Public Endpoint 17:01 B West US, Public Endpoint 12:21 C West US, Public Endpoint via IP Access List 15:19 D West US, E VM Gateway Base 12:14 E West US, E VM StreamBeforeRequestCompletes 07:46 F West US, E VM StreamBeforeRequestCompletes + Logical Partitions 07:31 G West US, E VM Spooler (D) 12:57 H West US, E VM Spooler (E) 13:32 I West US, D VM Gateway Base 16:47 J West US, D VM StreamBeforeRequestCompletes 12:19 K West US, PBI Managed Vnet 27:04 Scenario VM Configuration D Standard E8bds v5 (8 vcpus, 64 GiB memory) [NVMe, Accelerated Networking], C Drive default (Premium SSD LRS 127 GiB) E Standard E8bds v5 (8 vcpus, 64 GiB memory) [NVMe, Accelerated Networking], C Drive default (Premium SSD LRS 127 GiB) F Standard E8bds v5 (8 vcpus, 64 GiB memory) [NVMe, Accelerated Networking], C Drive default (Premium SSD LRS 127 GiB) G Standard E8bds v5 (8 vcpus, 64 GiB memory) [NVMe, Accelerated Networking], D drive H Standard E8bds v5 (8 vcpus, 64 GiB memory) [NVMe, Accelerated Networking], E Drive (Premium SSD LRS 600 GiB) I Standard D8s v3 (8 vcpus, 32 GiB memory), C Drive default (Premium SSD LRS 127 GiB) J Standard D8s v3 (8 vcpus, 32 GiB memory), C Drive default (Premium SSD LRS 127 GiB) Performance Configurations 1. Regional Alignment Aligning your Power BI Premium/Fabric Capacity to the same region as your Azure Databricks deployment and your On-Premise Data Gateway VM helps reduce the overall network latency and data transfer duration. It should also eliminate cross-region networking charges. In scenario A, the Azure Databricks deployment was in East US 2 while the Fabric Capacity and On-Premise Data Gateway VM were in West US. The Import processing time when using the public endpoint between the regions was 17:01 minutes. In scenario B, while still using the public endpoint, there is complete regional alignment in the West US region and the Import times averaged 12:21 minutes which is a 27.4% decrease 2. Configure a Gateway Cluster A Power BI Data Gateway Cluster configuration is highly recommended for Prouduction deployments but this configuration was not performance tested during this experiment. Data Gateway clusters can help with data refresh redundancy and for overall volume / throughput of data transfer. This configuration is highly recommended for Production Power BI environments. 3. VM Family Selection The Power BI documentation recommends a VM with 8 cores, 8 GB of RAM, and an SSD for the VM used for the On-Premise Data Gateway. Through testing, it can be proven that using a VM with good performance characteristics can provide immense value in the Import times. In scenario D, data gateway tests were run using a Standard E8bds v5 with 8 cores and 64 GB RAM that also included NVMe, and Accelerated Networking, and a C drive using a Premium SSD. The import times for this scenario averaged 12:14 minutes which was slightly faster than the regionally aligned public endpoint test in scenario B. In scenario I, data gateway tests were run using a Standard D8s v3 with 8 cores and 32 GB RAM and a C drive using a Premium SSD. The import times for this scenario averaged 16:47 minutes which was noticeably slower than using the regionally aligned public endpoint in cenario B which was a 35.96% performance degradation. More tests could certainly be done to determine which VM characteristics help the most with Import performance, but it is clear certain features can be helpful like: Premium SSDs Accelerated Networking NVMe controller Memory optimized instances And while the better E8bds v5 Azure VM costs ~$820 per month in West US at list and the D8s v3 costs ~$610 per month at list (25% more expensive), this feels like a scenario where you pay the premium to get better performance and optimize through Azure VM reservations. 4. StreamBeforeRequestCompletes By default, the on-premise data gateway spools data to disk before sending it to Power BI. Enabling the StreamBeforeRequestCompletes setting to True can significantly improve gateway refresh performance as it allows data to be streamed directly to the Power BI Service without first being spooled to disk. In scenario E, when StreamBeforeRequestCompletes is set to True and restarted, you can see that the average Import times significantly improved to 07:46 minutes which is a 54% improvement compared to scenario A and a 36% improvement over the base VM configuration in scenario D. 5. Spooler Location As discussed above, when using the default setting for StreamBeforeRequestCompletes as False, Power BI spools the data to the data gateway spool directory before sending it to the Power BI Service. In scenarios D, G, and H, StreamBeforeRequestCompletes is False and the Spooler directory has been mapped to the C drive, D drive, and E drives respectively which all correspond to an SSD (of varying configuration) on the Azure VMs. In all scenarios, you can see the times are similar between 12:14, 12:57, and 13:32 minutes, respectively. In all three scenarios the tests were performed with SSDs on the E series VM configured with NVMe. Using this configuration mix, it doesn’t appear that the Spooler directory location provides significant performance improvements. Since the C drive configuration gave the best performance it seems prudent to keep the C drive default configuration. However, it is possible that that the Spooler directory setting might provide more value on a different VM configurations. 6. Logical Partitioning As outlined in the QuickStart samples guide, logical partitioning can often help with Power BI Import performance as multiple logical partitions in the Semantic Model can be processed at the same time. In scenario F, logical partitions were created for the inventory and store_sales table to have 5 partitions each. When combined with the StreamBeforeRequestCompletes setting, the benefit from adding Logical Partitions was negligible (15 second improvement) even though the parallelization settings were increased to 30 (Max Parallelism Per Refresh and Data Source Default Max Connections). While logical partitions are usually a very valuable strategy, combining them with StreamBeforeRequestCompletes, the E series VM configurations, and a Fabric F64 capacity yielded diminishing returns. It is probably worth more testing at some point in the future. Virtual Network Data Gateway Performance Testing The configuration and performance of a Virtual Network Data Gateway was briefly tested. A Power BI subnet was created in the same VNet as the Azure Databricks workspace and delegated to the Power BI Service. A virtual network data gateway was created in the UI with 2 gateways (12 queries can run in parallel) and assigned to the Semantic Model. In scenario K, an Import test was performed through the Virtual Network Data Gateway that took 27:04 minutes. More time was not spent trying to tune the Virtual Network Data Gateway as it was not the primary focus of this blog post. The Best Configuration The Best Configuration: Region Alignment + Good VM + StreamBeforeRequestsCompletes While the Import testing performed for this blog post isn’t definitive, it does provide good directional value in forming an opinion on how you can configure your Power BI On-Premise Data Gateway on an Azure Virtual Machine to get good performance. When looking at the tests performed for this blog, an Azure Virtual Machine, in the same region as the Azure Databricks Workspace and the Fabric Capacity, with Accelerated networking, an SSD, NVMe, and memory optimized compute provided performance that was faster than just using the public endpoint of the Azure Databricks Workspace alone. Using this configuration, we improved our Import performance from 17:01 to 07:46 minutes which is a 54% performance improvement.Closing the loop: Interactive write-back from Power BI to Azure Databricks
This is a collaborative post from Microsoft and Databricks. We thank Toussaint Webb, Product Manager at Databricks, for his contributions. We're excited to announce that the Azure Databricks connector for Power Platform is now Generally Available. With this integration, organizations can seamlessly build Power Apps, Power Automate flows, and Copilot Studio agents with secure, governed data and no data duplication. A key functionality unlocked by this connector is the ability to write data back from Power BI to Azure Databricks. Many organizations want to not only analyze data but also act on insights quickly and efficiently. Power BI users, in particular, have been seeking a straightforward way to “close the loop” by writing data back from Power BI into Azure Databricks. This capability is now here - real-time updates and streamlined operational workflows with the new Azure Databricks connector for Power Platform. With this connector, users can now read from and write to Azure Databricks data warehouses in real time, all from within familiar interfaces — no custom connectors, no data duplication, and no loss of governance. How It Works: Write-backs from Power BI through Power Apps Enabling writebacks from Power BI to Azure Databricks is seamless. Follow these steps: Open Power Apps and create a connection to Azure Databricks (documentation). In Power BI (desktop or service), add a Power Apps visual to your report (purple Power Apps icon). Add data to connect to your Power App via the visualization pane. Create a new Power App directly from the Power BI interface, or choose an existing app to embed. Start writing records to Azure Databricks! With this integration, users can make real-time updates directly within Power BI using the embedded Power App, instantly writing changes back to Azure Databricks. Think of all the workflows that this can unlock, such as warehouse managers monitoring performance and flagging issues on the spot, or store owners reviewing and adjusting inventory levels as needed. The seamless connection between Azure Databricks, Power Apps, and Power BI lets you close the loop on critical processes by uniting reporting and action in one place. Try It Out: Get started with Azure Databricks Power Platform Connector The Power Platform Connector is now Generally Available for all Azure Databricks customers. Explore more in the deep dive blog here and to get started, check out our technical documentation. Coming soon we will add the ability to execute existing Azure Databricks Jobs via Power Automate. If your organization is looking for an even more customizable end-to-end solution, check out Databricks Apps in Azure Databricks! No extra services or licenses required.Supercharge Data Intelligence: Build Teams App with Azure Databricks Genie & Azure AI Agent Service
Introduction Are you looking to unlock the full potential of your data investments in Azure Databricks while seamlessly harnessing the capabilities of Azure AI? At Microsoft BUILD 2025, we made the announcement of the Azure Databricks connector in Azure AI Foundry. This blog post is a follow-up to take advantage of this feature within Microsoft Teams. We'll guide you through leveraging the integration between Azure Databricks and Azure AI Foundry to build a Python-based Teams app that consumes Genie APIs using the secure On-Behalf-Of (OBO) authentication flow. Whether you are a data engineer, AI developer, or a business user seeking actionable insights, this guide will help you accelerate your journey in data intelligence with modern, secure, and streamlined tools. You can find the code samples here: AI-Foundry-Connections - Teams chat with Azure Databricks Genie | GitHub (git clone https://github.com/Azure-Samples/AI-Foundry-Connections.git ) Setting the Stage: Key Components Before we dive into the sample app, let’s quickly establish what each major component brings to the table. What is AI/BI Genie? AI/BI Genie is an intelligent agent feature in Azure Databricks, exposing natural language APIs to interact with your data. It empowers users to query, analyze, and visualize data using conversational AI, making advanced analytics accessible to everyone. Azure AI Foundry Azure AI Foundry is your control center for building, managing, and deploying AI solutions. It offers a unified platform to orchestrate AI agents, connect to various data sources like Azure Databricks, and manage models, workflows, and governance. Azure AI Agent Service Azure AI Agents are modular, reusable, and secure components that interact with data and AI services. They enable you to build multi-agent workflows, integrate with enterprise systems, and deliver contextual, actionable insights to your end-users. The Sample Solution: Python Teams App for Genie APIs To help you get hands-on, this blog features a sample Teams app that demonstrates how to: Connect Teams to Azure Databricks Genie via Azure AI Foundry using OBO (On-Behalf-Of) authentication. Query and visualize data using Genie APIs and LLMs in a secure, governed manner. Build and run the app locally using DevTunnel and easily extend it for production deployments. Purpose: The sample app is designed as a learning tool, enabling developers to explore the integration and build their own enterprise-grade solutions. It shows you how to wire up authentication, connect to Genie, and ask data-driven questions—all from within Teams. Architecture Overview Below is a logical architecture that illustrates the flow: Key Steps from the Sample (posted on Github) Prerequisites: a. Familiarity with Azure Databricks, DevTunnel, Azure Bot Service, and Teams App setup. b. An AI Foundry project with Databricks and LLM connection. Configuration: a. Set up the DevTunnel for local development. b. Register and configure your bot and app in Azure. c. Create a connection in Azure AI Foundry to your Databricks Genie space. d. Provision storage with public blob access for dynamic chart/image rendering. e. Populate the `.env` file with all necessary IDs, secrets, and endpoints. Teams App Manifest: a. Create and configure the Teams app manifest, specifying endpoints, domains, permissions, and SSO details. Run and Test: a. Host the app locally, activate the virtual environment, and run the Python application. b. Upload your Teams app manifest and test conversational queries directly in the Teams client. Sample Queries (Try these in the app!): "Show the top sales reps by pipeline in a pie chart." "What is the average size of active opportunities?" "How many opportunities were won or lost in the current fiscal year?" Get Started: Next Steps To replicate this solution: Review the sample in AI-Foundry-Connections - Teams chat with Azure Databricks Genie | GitHub (git clone https://github.com/Azure-Samples/AI-Foundry-Connections.git) —you'll need to review the README for the step-by-step setup and configuration details! Explore homework opportunities such as deploying the app on Azure App Service or Kubernetes, and experimenting with additional languages or the M365 Agents Toolkit. Final Thoughts Thank you for reading! By integrating Azure Databricks, Azure AI Foundry, and Genie APIs, you can deliver powerful, secure, and collaborative data intelligence experiences right within Microsoft Teams. Ready to go further? Check out the M365 Agents SDKand M365 Agents Toolkit for building advanced AI applications. Experiment, extend, and share your feedback with the community! Happy building! 🚀Enabling Open Data Sharing of Unity Catalog Assets with Microsoft Purview
As organizations scale their use of Databricks for data and AI workloads, enabling secure and discoverable access to Unity Catalog assets becomes increasingly important. In many cases, users such as analysts or business partners need to query specific datasets but do not have direct access to the Databricks workspace. Databricks Delta Sharing addresses this challenge by allowing secure access to data and AI assets through an open sharing protocol, making it possible to share governed datasets across organizational boundaries. Learn more about sharing data using the Delta Sharing open sharing protocol. While Delta Sharing provides the access layer, Microsoft Purview plays a critical role in enabling data discovery. Without access to Databricks Catalog Explorer, users may struggle to locate Unity Catalog-managed data. By integrating with Unity Catalog, Microsoft Purview makes metadata for these assets discoverable through its data map. This allows users across the organization to explore available datasets, including those in Unity Catalog, and request access in a secure and governed manner. Once data is discoverable, Microsoft Purview enables automated access provisioning through its self-service workflows. These workflows can be configured to trigger Databricks REST APIs, allowing Unity Catalog access requests to be processed in an automated fashion. This approach streamlines the approval process, reduces manual overhead, and ensures that access is granted in a secure and auditable way. Check out this detailed guide I wrote to implement this integration: A Guide to Enabling Open Data Sharing of Databricks Unity Catalog Assets with Microsoft Purview. The guide currently focuses on using Purview workflows, which are part of the classic data catalog experience. Guidance on the upcoming Power Automate connector—which will enable access requests directly from Purview—is expected to be available soon.
Workspace failure
Hi Community, I had my Databricks workspace up and running and it was managed through terraform, and encryption was enabled through cmk, there were some updation in the code, so I put terraform plan, one of the key changes(replace) it showed me was "azurerm_role_assignment.storage_identity_kv_access module.workspace.azurerm_role_assignment.storage_identity_kv_access" the terraform run was running for 30 min, and the workspace was in deployment for long time and then ultimately got failed. Again, as all the changes were not done, I reapplied, and I got this error "Performing CreateOrUpdate: unexpected status 400 (400 Bad Request ) With error: InvalidEncryptionConfiguration: Configure encryption for workspace at creation is not allowed, configure encryption once workspace is created and key vault access policies are added" Again, I applied and everything and terraform run succeeded but I can see in azure portal that workspace is in failed state, but if I go to Databricks account I can see Databricks as running and if I go to workspace, I am able to start clusters and execute some queries. I am not able to launch the workspace using azure portal, not sure there will be other issues due to this. Could anyone help me to resolve this issue. Let me know if you need anything further to investigate the issue.
