One of the key features customers look for when adopting a Lakehouse strategy is the ability to efficiently and securely consume data directly from the data lake with BI tools. This typically reduces the additional latency, compute, and storage costs associated with the traditional flow of copying data already stored in a data lake to a data warehouse for BI consumption.
To help customers looking to consume data directly from data stored in their data lake using Power BI, Microsoft and Databricks have built a new Azure Databricks connector in Power BI Desktop.
This connector provides 3 key features or benefits over the previous Spark connector used to connect Power BI to Azure Databricks:
Azure Active Directory (AAD) Authentication - AAD is the basic authentication supporting most Azure services including Azure Databricks. Traditionally, to connect to Azure Databricks from tools like Power BI, a Personal Access Token (PAT) needed to be created for the user in Azure Databricks. The new Azure Databricks connector, however, allows users to log on with their AAD credential with support for familiar features such as Multi-Factor-Authentication (MFA).
ADLS Passthrough Capabilities - Customers want to secure the data where it lives. In the case of ADLS, customers can secure the files and folders with AAD user identities and groups. While Azure Databricks has long supported AAD credential passthrough in notebooks, it was not possible with BI tools that required PATs. Now that AAD authentication is supported with the new Azure Databricks Power BI connector, users can expect the same ADLS credential passthrough functionality experienced in their notebooks.
More Efficient Databricks ODBC Driver - The driver used in the existing Power BI Spark connector does not efficiently send queries to and return results from Azure Databricks. It often includes unnecessary metadata queries and unnecessary communication hops adding extra overhead to the round trip query. The new Azure Databricks connector in Power BI removes most of this unnecessary overhead resulting in round trip queries that more closely match the actual query time on the clusters.
Overall, the Azure Databricks connector in Power BI makes for a more secure, more interactive data visualization experience for data stored in your data lake. When you add the benefits of the Azure Databricks Delta Engine, including the Delta Cache and Photon, Azure Databricks provides one of the fastest experiences in Azure to visualize curated data lake data using Power BI.