Announcing the Azure Databricks connector in Power Platform
We are ecstatic to announce the public preview of the Azure Databricks Connector for Power Platform. This native connector is specifically for Power Apps, Power Automation, and Copilot Studio within Power Platform and enables seamless, single click connection. With this connector, your organization can build data-driven, intelligent conversational experiences that leverage the full power of your data within Azure Databricks without any additional custom configuration or scripting – it's all fully built in! The Azure Databricks connector in power platform enables you to: Maintain governance: All access controls for data you set up in Azure Databricks are maintained in Power Platform Prevent data copy: Read and write to your data without data duplication Secure your connection: Connect Azure Databricks to Power Platform using Microsoft Entra user-based OAuth or service principals Have real time updates: Read and write data and see updates in Azure Databricks in near real time Build agents with context: Build agents with Azure Databricks as grounding knowledge with all the context of your data Instead of spending time copying or moving data and building custom connections which require additional manual maintenance, you can now seamlessly connect and focus on what matters – getting rich insights from your data – without worrying about security or governance. Let’s see how this connector can be beneficial across Power Apps, Power Automate, and Copilot Studio: Azure Databricks Connector for Power Apps – You can seamlessly connect to Azure Databricks from Power Apps to enable read/write access to your data directly within canvas apps enabling your organization to build data-driven experiences in real time. For example, our retail customers are using this connector to visualize different placements of items within the store and how they impact revenue. Azure Databricks Connector for Power Automate – You can execute SQL commands against your data within Azure Databricks with the rich context of your business use case. For example, one of our global retail customers is using automated workflows to track safety incidents, which plays a crucial role in keeping employees safe. Azure Databricks as a Knowledge Source in Copilot Studio – You can add Azure Databricks as a primary knowledge source for your agents, enabling them to understand, reason over, and respond to user prompts based on data from Azure Databricks. To get started, all you need to do in Power Apps or Power Automate is add a new connection – that's how simple it is! Check out our demo here and get started using our documentation today! This connector is available in all public cloud regions. You can also learn more about customer use cases in this blog. You can also review the connector reference here
Announcing the availability of Azure Databricks connector in Azure AI Foundry
At Microsoft, Databricks Data Intelligence Platform is available as a fully managed, native, first party Data and AI solution called Azure Databricks. This makes Azure the optimal cloud for running Databricks workloads. Because of our unique partnership, we can bring you seamless integrations leveraging the power of the entire Microsoft ecosystem to do more with your data. Azure AI Foundry is an integrated platform for Developers and IT Administrators to design, customize, and manage AI applications and agents. Today we are excited to announce the public preview of the Azure Databricks connector in Azure AI Foundry. With this launch you can build enterprise-grade AI agents that reason over real-time Azure Databricks data while being governed by Unity Catalog. These agents will also be enriched by the responsible AI capabilities of Azure AI Foundry. Here are a few ways this can benefit you and your organization: Native Integration: Connect to Azure Databricks AI/BI Genie from Azure AI Foundry Contextual Answers: Genie agents provide answers grounded in your unique data Supports Various LLMs: Secure, authenticated data access Streamlined Process: Real-time data insights within GenAI apps Seamless Integration: Simplifies AI agent management with data governance Multi-Agent workflows: Leverages Azure AI agents and Genie Spaces for faster insights Enhanced Collaboration: Boosts productivity between business and technical users To further democratize the use of data to those in your organization who aren't directly interacting with Azure Databricks, you can also take it one step further with Microsoft Teams and AI/BI Genie. AI/BI Genie enables you to get deep insights from your data using your natural language without needing to access Azure Databricks. Here you see an example of what an agent built in AI Foundry using data from Azure Databricks available in Microsoft Teams looks like We'd love to hear your feedback as you use the Azure Databricks connector in AI Foundry. Try it out today – to help you get started, we’ve put together some samples here. Read more on the Databricks blog, too.Announcing general availability of Cross-Cloud Data Governance with Azure Databricks
We are excited to announce the general availability of accessing AWS S3 data in Azure Databricks Unity Catalog. This release simplifies cross-cloud data governance by allowing teams to configure and query AWS S3 data directly from Azure Databricks without migrating or duplicating datasets. Key benefits include unified governance, frictionless data access, and enhanced security and compliance.Announcing the availability of Azure Databricks connector in Azure AI Foundry
At Microsoft, Databricks Data Intelligence Platform is available as a fully managed, native, first party Data and AI solution called Azure Databricks. This makes Azure the optimal cloud for running Databricks workloads. Because of our unique partnership, we can bring you seamless integrations leveraging the power of the entire Microsoft ecosystem to do more with your data. Azure AI Foundry is an integrated platform for Developers and IT Administrators to design, customize, and manage AI applications and agents. Today we are excited to announce the public preview of the Azure Databricks connector in Azure AI Foundry. With this launch you can build enterprise-grade AI agents that reason over real-time Azure Databricks data while being governed by Unity Catalog. These agents will also be enriched by the responsible AI capabilities of Azure AI Foundry. Here are a few ways this seamless integration can benefit you and your organization: Native Integration: Connect to Azure Databricks AI/BI Genie from Azure AI Foundry Contextual Answers: Genie agents provide answers grounded in your unique data Supports Various LLMs: Secure, authenticated data access Streamlined Process: Real-time data insights within GenAI apps Seamless Integration: Simplifies AI agent management with data governance Multi-Agent workflows: Leverages Azure AI agents and Genie Spaces for faster insights Enhanced Collaboration: Boosts productivity between business and technical users To further democratize the use of data for those in your organization aren't directly interacting with Azure Databricks, you can also take it one step further with Microsoft Teams and AI/BI Genie. AI/BI Genie enables you to get deep insights from your data using your natural language without needing to access Azure Databricks. Here you see an example of what an agent built in AI Foundry using data from Azure Databricks available in Microsoft Teams looks like We'd love to hear your feedback as you use the Azure Databricks connector in AI Foundry. Try it out today – to help you get started, we’ve put together some samples here.Power BI & Azure Databricks: Smarter Refreshes, Less Hassle
We are excited to extend the deep integration between Azure Databricks and Microsoft Power BI with the Public Preview of the Power BI task type in Azure Databricks Workflows. This new capability allows users to update and refresh Power BI semantic models directly from their Azure Databricks workflows, ensuring real-time data updates for reports and dashboards. By leveraging orchestration and triggers within Azure Databricks Workflows, organizations can improve efficiency, reduce refresh costs, and enhance data accuracy for Power BI users. Power BI tasks seamlessly integrate with Unity Catalog in Azure Databricks, enabling automated updates to tables, views, materialized views, and streaming tables across multiple schemas and catalogs. With support for Import, DirectQuery, and Dual Storage modes, Power BI tasks provide flexibility in managing performance and security. This direct integration eliminates manual processes, ensuring Power BI models stay synchronized with underlying data without requiring context switching between platforms. Built into Azure Databricks Lakeflow, Power BI tasks benefit from enterprise-grade orchestration and monitoring, including task dependencies, scheduling, retries, and notifications. This streamlines workflows and improves governance by utilizing Microsoft Entra ID authentication and Unity Catalog suite of security and governance offerings. We invite you to explore the new Power BI tasks today and experience seamless data integration—get started by visiting the [ADB Power BI task documentation].
Fabric Data Agents: Unlocking the Power of Agents as a Steppingstone for a Modern Data Platform
What Are Fabric Data Agents? Fabric Data Agents are intelligent, AI-powered assistants embedded within Microsoft Fabric, a unified data platform that integrates data ingestion, processing, transformation, and analytics. These agents act as intermediaries between users and data, enabling seamless interaction through natural language queries in the form of Q&A applications. Whether it's retrieving insights, analyzing trends, or generating visualizations, Fabric Data Agents simplify complex data tasks, making advanced analytics accessible to everyone—from data scientists to business analysts to executive teams. How Do They Work? At the center of Fabric Data Agents is OneLake, a unified and governed data lake that joins data from various sources, including on-premises systems, cloud platforms, and third-party databases. OneLake ensures that all data is stored in a common, open format, simplifying data management and enabling agents to access a comprehensive view of the organization's data. Through Fabric’s Data Ingestion capabilities, such as Fabric Data Factory, OneLake Shortcuts, and Fabric Database Mirroring, Fabric Data Agents are designed to connect with over 200 data sources, ensuring seamless integration across an organization's data estate. This connectivity allows them to pull data from diverse systems and provide a unified analytics experience. Here's how Fabric Data Agents work: Natural Language Processing: Using advanced NLP techniques, Fabric Data Agents enable users to interact with data through conversational queries. For example, users can ask questions like, "What are the top-performing investment portfolios this quarter?" and receive precise answers, grounded on enterprise data. AI-powered Insights: The agents process queries, reason over data, and deliver actionable insights, using Azure OpenAI models, all while maintaining data security and compliance. Customization: Fabric data agents are highly customizable. Users can provide custom instructions and examples to tailor their behavior to specific scenarios. Fabric Data Agents allow users to provide example SQL queries, which can be used to influence the agent’s behavior. They also can integrate with Azure AI Agent Service or Microsoft Copilot Studio, where organizations can tailor agents to specific use cases, such as risk assessment or fraud detection. Security and Compliance: Fabric Data Agents are built with enterprise-grade security features, including inheriting Identity Passthrough/On-Behalf-Of (OBO) authentication. This ensures that business users only access data they are authorized to view, keeping strict compliance with regulations like GDPR and CCPA across geographies and user roles. Integration with Azure: Fabric Data Agents are deeply integrated with Azure services, such as Azure AI Agent Service and Azure OpenAI Service. Practically, organizations can publish Fabric Data Agents to custom Copilots using these services and use the APIs in various custom AI applications. This integration ensures scalability, high availability, and performance and exceptional customer experience. Why Should Financial Services Companies Use Fabric Data Agents? The financial services industry faces unique challenges, including stringent regulatory requirements, the need for real-time decision-making, and empowering users to interact with an AI application in a Q&A fashion over enterprise data. Fabric Data Agents address these challenges head-on through: Enhanced Efficiency: Automate repetitive tasks, freeing up valuable time for employees to focus on strategic initiatives. Improved Compliance: Use robust data governance features to ensure compliance with regulations like GDPR and CCPA. Data-Driven Decisions: Gain deeper insights into customer behavior, market trends, and operational performance. Scalability: Seamlessly scale analytics capabilities to meet the demands of a growing organization, without really investing in building custom AI applications which require deep expertise. Integration with Azure: Fabric Data Agents are natively designed to integrate across Microsoft’s ecosystem, providing a comprehensive end-to-end solution for a Modern Data Platform. How different are Fabric Data Agents from Copilot Studio Agents? Fabric Data Agents and Copilot Studio Agents serve distinct purposes within Microsoft's ecosystem: Fabric Data Agents are tailored for data science workflows. They integrate AI capabilities to interact with organizational data, providing analytics insights. They focus on data processing and analysis using the medallion architecture (bronze, silver, and gold layers) and support integration with the Lakehouse, Data Warehouse, KQL Databases and Semantic Models. Copilot Studio Agents, on the other hand, are customizable AI-powered assistants designed for specific tasks. Built within Copilot Studio, they can connect to various enterprise data sources like OneLake, AI Search, SharePoint, OneDrive, and Dynamics 365. These agents are versatile, enabling businesses to automate workflows, analyze data, and provide contextual responses by using APIs and built-in connectors. What are the technical requirements for using Fabric Data Agents? A paid F64 or higher Fabric capacity resource Fabric data agent tenant settingsis enabled. Copilot tenant switchis enabled. Cross-geo processing for AIis enabled. Cross-geo storing for AIis enabled. At least one of these: Fabric Data Warehouse, Fabric Lakehouse, one or more Power BI semantic models, or a KQL database with data. Power BI semantic models via XMLA endpoints tenant switchis enabled for Power BI semantic model data sources. Final Thoughts In a data-driven world, Fabric Data Agents are poised to redefine how financial services organizations operate and innovate. By simplifying complex data processes, enabling actionable insights, and fostering collaboration across teams, these intelligent agents empower organizations to unlock the true potential of their data. Paired with the robust capabilities of Microsoft Fabric and Azure, financial institutions can confidently navigate industry challenges, drive growth, and deliver superior customer experiences. Adopting Fabric Data Agents is not just an upgrade—it's a transformative step towards building a resilient and future-ready business. The time to embrace the data revolution is now. Learn how to create Fabric Data AgentsLlama 4 is now available in Azure Databricks
We are excited to announce the availability of Meta's Llama 4 in Azure Databricks. As you know, enterprises all over the world already use Llama models in Azure Databricks to power AI enterprise agents, workflows, and applications. Now with Llama 4 and Azure Databricks, you can get higher quality, faster inference, and lower cost than previous models. Llama 4 Maverick, the highest-quality and largest Llama model from today's announcement, is built for developers building the next generation of AI products that combine multilingual fluency, image understanding precision, and security. With Maverick on Azure Databricks, you can: Build domain specific AI agents with your data Run scalable inference with your data pipeline Fine-tune for accuracy and Govern AI usage with Mosaic AI Gateway Azure Databricks Intelligence Platform makes it easy for you to securely connect Llama 4 to your enterprise data using Unity Catalog governed tools to build agents with contextual awareness. Enterprise data needs enterprise scale, whether it is to summarize documents or analyze support tickets, but without the infrastructure overhead. With Azure Databricks workflows and Llama 4 at scale, you can use SQL/Python to run LLMs at scale without overhead. You can tune Llama 4 to your custom use case for accuracy and alignment such as assistant behavior or summarization. All this comes with built in security controls and compliant model usage via Azure Databricks Mosaic AI Gateway with PII detection, logging, and policy guardrails on Azure Databricks. Llama 4 is available now in Azure Databricks. More models will become available in phases. Llama 4 Scout is coming soon and you'll be able to pick the model that fits your workload best. Learn more about Llama 4 and supported models in Azure Databricks here and get started today.
Delivering Information with Azure Synapse and Data Vault 2.0
Data Vault has been designed to integrate data from multiple data sources, creatively destruct the data into its fundamental components, and store and organize it so that any target structure can be derived quickly. This article focused on generating information models, often dimensional models, using virtual entities. They are used in the data architecture to deliver information. After all, dimensional models are easier to consume by dashboarding solutions, and business users know how to use dimensions and facts to aggregate their measures. However, PIT and bridge tables are usually needed to maintain the desired performance level. They also simplify the implementation of dimension and fact entities and, for those reasons, are frequently found in Data Vault-based data platforms. This article completes the information delivery. The following articles will focus on the automation aspects of Data Vault modeling and implementation.
Creating a AI-Driven Chatbot to Inquire Insights into business data
Introduction In the fast-paced digital era, the ability to extract meaningful insights from vast datasets is paramount for businesses striving for a competitive edge. Microsoft Dynamics 365 Finance and Operations (D365 F&O) is a robust ERP platform, generating substantial business data. To unlock the full potential of this data, integrating it with advanced analytics and AI tools such as Azure OpenAI, Azure Synapse Workspace, or Fabric Workspace is essential. This blog will guide you through the process of creating a chatbot to inquire insights using Azure OpenAI with Azure Synapse Workspace or Fabric Workspace. Architecture Natural Language Processing (NLP): Enables customers to inquire about business data such as order statuses, item details, and personalized order information using natural language. Seamless Data Integration: Real-time data fetching from D365 F&O for accurate and up-to-date information. Contextual and Personalized Responses: AI provides detailed, context-rich responses to customer queries, improving engagement and satisfaction. Scalability and Efficiency: Handles multiple concurrent inquiries, reducing the burden on customer service teams and improving operational efficiency. Understanding the Components Microsoft Dynamics 365 Finance and Operations (D365 F&O) D365 F&O is a comprehensive ERP solution designed to help businesses streamline their operations, manage finances, and control supply chain activities. It generates and stores vast amounts of transactional data essential for deriving actionable insights. Dataverse Dataverse is a cloud-based data storage solution that allows you to securely store and manage data used by business applications. It provides a scalable and reliable platform for data integration and analytics, enabling businesses to derive actionable insights from their data. Azure Synapse Analytics Azure Synapse Analytics is an integrated analytics service that brings together big data and data warehousing. It allows users to query data on their terms, deploying either serverless or provisioned resources at scale. The service provides a unified experience to ingest, prepare, manage, and serve data for instant business intelligence and machine learning requirements. Fabric Workspace Fabric Workspace provides a collaborative platform for data scientists, analysts, and business users to work together on data projects. It facilitates the seamless integration of various data sources and advanced analytics tools to drive innovative solutions. Azure SQL Database Azure SQL Database is a cloud-based relational database service built on Microsoft SQL Server technologies. It offers a range of deployment options, including single databases, elastic pools, and managed instances, allowing you to choose the best fit for your application needs. Azure SQL Database provides high availability, scalability, and security features, making it an ideal choice for modern applications. Data from Dynamics 365 Finance and Operations (F&O) is copied to an Azure SQL Database using a flow that involves Azure Data Lake Storage (ADLS) and Azure Data Factory (ADF) Azure OpenAI Azure OpenAI enables developers to build and deploy intelligent applications using powerful AI models. By integrating OpenAI’s capabilities with Azure’s infrastructure, businesses can create sophisticated solutions that leverage natural language processing, machine learning, and advanced analytics. Step-by-Step Guide to Creating the Chatbot Step 1: Export Data from D365 F&O To begin, export the necessary data from your D365 F&O instance. This data will serve as the foundation for your analytics and AI operations. Ensure the exported data is in a format compatible with Azure Synapse or Fabric Workspace. Step 2: Ingest Data into Azure Synapse Workspace or Fabric Workspace Next, ingest the exported data into Azure Synapse Workspace or Fabric Workspace. Utilize the workspace’s capabilities to prepare, manage, and optimize the data for further analysis. This step involves setting up data pipelines, cleaning the data, and transforming it into a suitable format for processing. Step 3: Set Up Azure OpenAI With your data ready, set up Azure OpenAI in your environment. This involves provisioning the necessary resources, configuring the OpenAI service, and integrating it with your Azure infrastructure. Ensure you have the appropriate permissions and access controls in place. Step 4: Develop the Chatbot Develop the chatbot using Azure OpenAI’s capabilities. Design the chatbot to interact with users naturally, allowing them to inquire insights and receive valuable information based on the data from D365 F&O. Utilize natural language processing to enhance the chatbot’s ability to understand and respond to user queries effectively. Step 5: Integrate the Chatbot with Azure Synapse or Fabric Workspace Integrate the developed chatbot with Azure Synapse Workspace or Fabric Workspace. This integration will enable the chatbot to access and analyze the ingested data, providing users with real-time insights. Set up the necessary APIs and data connections to facilitate seamless communication between the chatbot and the workspace. Step 6: Test and Refine the Chatbot Thoroughly test the chatbot to ensure it functions as expected. Address any issues or bugs, and refine the chatbot’s responses and capabilities. This step is crucial to ensure the chatbot delivers accurate and valuable insights to users. Best Practices for Data Access Data Security Data security is paramount when exporting sensitive business information. Implement the following best practices: Ensure that all data transfers are encrypted using secure protocols. Use role-based access control to restrict access to the data exported. Regularly audit and monitor data export activities to detect any unauthorized access or anomalies. Data Transformation Transforming data before accessing it can enhance its usability for analysis: Use Synapse data flows to clean and normalize the data. Apply business logic to enrich the data with additional context. Aggregate and summarize data to improve query performance. Monitoring and Maintenance Regular monitoring and maintenance ensure the smooth operation of your data export solution: Set up alerts and notifications for any failures or performance issues in the data pipelines. Regularly review and optimize the data export and transformation processes. Keep your Azure Synapse environment up to date with the latest features and enhancements. Benefits of Integrating AI and Advanced Analytics Enhanced Decision-Making By leveraging AI and advanced analytics, businesses can make data-driven decisions. The chatbot provides timely insights, enabling stakeholders to act quickly and efficiently. Improved Customer Experience A chatbot enhances customer interactions by providing instant responses and personalized information. This leads to higher satisfaction and engagement levels. Operational Efficiency Integrating AI tools with business data streamlines operations, reduces manual efforts, and increases overall efficiency. Businesses can optimize processes and resource allocation effectively. Scalability It can handle multiple concurrent inquiries, scaling as the business grows without requiring proportional increases in customer service resources. Conclusion Creating a chatbot to inquire insights using Azure OpenAI with Azure Synapse Workspace or Fabric Workspace represents a significant advancement in how businesses can leverage their data. By following the steps outlined in this guide, organizations can develop sophisticated AI-driven solutions that enhance decision-making, improve customer experiences, and drive operational efficiency. Embrace the power of AI and advanced analytics to transform your business and unlock new opportunities for growth.Anthropic State-of-the-Art Models Available to Azure Databricks Customers
Our customers now have greater model choices with the arrival of Anthropic Claude 3.7 Sonnet in Azure Databricks. Databricks is announcing a partnership with Anthropic to integrate their state-of-the-art models into Databricks Data Intelligence Platform as a native offering, starting with Claude 3.7 Sonnet http://databricks.com/blog/anthropic-claude-37-sonnet-now-natively-available-databricks. With this announcement, Azure customers can use Claude Models directly in Azure Databricks. Foundation model REST API reference - Azure Databricks | Microsoft Learn With Anthropic models available in Azure Databricks, customers can use the Claude "think" tool with business data optimized promote to guide Claude efficiently perform complex tasks. With Claude models in Azure Databricks, enterprises can deliver domain-specific, high quality AI agents more efficiently. As an integrated component of the Azure Databricks Data Intelligence Platform, Anthropic Claude models benefit from comprehensive end-to-end governance and monitoring throughout the entire data and AI lifecycle with Unity Catalog. With Claude models, we remain committed to providing customers with model flexibility. Through the Azure Databricks Data Intelligence Platform, customers can securely connect to any model provider and select the most suitable model for their needs. They can further enhance these models with enterprise data to develop domain-specific, high-quality AI agents, supported by built-in custom evaluation governance across both data and models.
