Decision Guide for Selecting an Analytical Data Store in Microsoft Fabric
Learn how to select an analytical data store in Microsoft Fabric based on your workload's data volumes, data type requirements, compute engine preferences, data ingestion patterns, data transformation needs, query patterns, and other factors.Announcing the availability of Azure Databricks connector in Azure AI Foundry
At Microsoft, Databricks Data Intelligence Platform is available as a fully managed, native, first party Data and AI solution called Azure Databricks. This makes Azure the optimal cloud for running Databricks workloads. Because of our unique partnership, we can bring you seamless integrations leveraging the power of the entire Microsoft ecosystem to do more with your data. Azure AI Foundry is an integrated platform for Developers and IT Administrators to design, customize, and manage AI applications and agents. Today we are excited to announce the public preview of the Azure Databricks connector in Azure AI Foundry. With this launch you can build enterprise-grade AI agents that reason over real-time Azure Databricks data while being governed by Unity Catalog. These agents will also be enriched by the responsible AI capabilities of Azure AI Foundry. Here are a few ways this can benefit you and your organization: Native Integration: Connect to Azure Databricks AI/BI Genie from Azure AI Foundry Contextual Answers: Genie agents provide answers grounded in your unique data Supports Various LLMs: Secure, authenticated data access Streamlined Process: Real-time data insights within GenAI apps Seamless Integration: Simplifies AI agent management with data governance Multi-Agent workflows: Leverages Azure AI agents and Genie Spaces for faster insights Enhanced Collaboration: Boosts productivity between business and technical users To further democratize the use of data to those in your organization who aren't directly interacting with Azure Databricks, you can also take it one step further with Microsoft Teams and AI/BI Genie. AI/BI Genie enables you to get deep insights from your data using your natural language without needing to access Azure Databricks. Here you see an example of what an agent built in AI Foundry using data from Azure Databricks available in Microsoft Teams looks like We'd love to hear your feedback as you use the Azure Databricks connector in AI Foundry. Try it out today – to help you get started, we’ve put together some samples here. Read more on the Databricks blog, too.Part 1: Power BI Service Connections to Azure Databricks with Private Networking
This blog was written in conjunction with Leo Furlong, Lead Solutions Architect at Databricks. Enhancing Security and Connectivity: Azure Databricks SQL, Unity Catalog, and Power BI Integration The combination of Azure Databricks SQL, Unity Catalog, and Power BI offers an unparalleled set of capabilities for modern data analytics. However, as organizations increasingly prioritize security, many Azure Databricks customers are deploying their Databricks workspace with private networking requirements which requires additional configuration for allowing connections from BI tools like Power BI. This blog post explores the options available for secure Azure Databricks deployments and how to maintain Power BI connectivity in these scenarios. Private Networking Options for Azure Databricks When deploying Azure Databricks with enhanced security, customers can choose from three main private networking configurations: Public Endpoint with an IP Access List for the Workspace: This option exposes a public endpoint for the Azure Databricks workspace but restricts access to specific IP ranges. Azure Databricks Private Link: Front-end private link provides fully private connectivity, routing all traffic through private endpoints. Hybrid Deployment: Combines front-end private link with a public endpoint protected by a Workspace IP Access List which is typically used for SaaS service connections. Connecting Power BI to a Private Azure Databricks Workspaces While private networking enhances security, it can require additional connection configurations from SaaS services like Power BI. Power BI offers two primary methods for secure connections to data sources with private networking: On-premises data gateway: an application that gets installed on a Virtual Machine that has a direct networking connection to the data source. It allows Power BI to connect to data sources that don’t allow public connections Virtual Network Data Gateway: a managed (virtual/serverless) data gateway that gets created and managed by the Power BI service. Connections work by allowing Power BI to delegate into a VNet for secure connectivity to the data source. While Power BI offers these two options, many customers prefer not to manage additional infrastructure or configurations required for these gateways. In such cases, Power BI can be allowed to access the private Azure Databricks workspace through the IP Access List. Implementing Power BI Connectivity via IP Access List To enable the Power BI Service connectivity to a private Azure Databricks workspace using an IP Access List: Obtain the Power BI Public IPs: Download the latest Azure IP Ranges and Service Tags file from the Microsoft Download Center. This file is updated weekly and contains IP ranges for various Azure services, including Power BI. Add Power BI IPs to Azure Databricks Workspace IP Access List: Extract the Power BI IP ranges from the downloaded file and add them to the Azure Databricks IP Access List using the API or SDK. Regular Updates: Since Power BI public IPs can change frequently, it's crucial to update the Workspace IP Access List regularly. This can be automated using a Databricks Job that periodically downloads the latest IP ranges and updates the Workspace IP Access List. The Job will need to be run by a Workspace Admin in order to set the configurations. You can run the Databricks Job as a Service Principal to make the updates. If you use the Databricks SDK from within a notebook in the Databricks Workspace, authentication is handled for you. The following sample code can be used to turn on your Workspace IP Access List which is more of a one-time operation. The Power BI IPs for IP Access List sample code can be used to refresh your Power BI IPs from a Databricks Workflow. Conclusion By leveraging IP Access Lists, organizations can maintain the security benefits of private Azure Databricks deployments while ensuring seamless connections from Power BI. This approach offers a balance between security and functionality with low maintenance overhead.Azure Stream Analytics releases slew of improvements at Ignite 2022: Output to Delta Lake and more!
Today we are excited to announce numerous new capabilities that unlock new stream processing patterns that work with your modern lakehouses. We are announcing native support of Delta Lake output, no code editor GA, improved development & troubleshooting experience and much more!
