Fabric Data Agents can choose Query Language based on Context
What if you could combine decades of historical analysis with live, real-time data in one seamless AI chat experience? Traditionally, analyzing large volumes of past data (Analytics / Business Intelligence) has been a separate architecture, or has at least required separate toolsets, from monitoring what’s happening right now (real-time, IoT, etc). With Fabric Data Agents, analytics for large volumes of historical data can be accessible with real-time data via a single AI query endpoint. Analytics can be used to gain understanding from historical data, and findings can be put into action for real-time scenarios, all through a single interface for the end users. Figure 1.0 – Fabric Data Agent can use different query languages for optimal performance with a lambda-style RTI architecture on Fabric Many of the demos I’ve seen for Fabric Data Agents will highlight the capability to connect to different types of queries and sources via a single endpoint such as Lakehouses (SQL), Warehouses (SQL), Semantic Models (DAX), Eventhouses (Kusto), and Ontologies. What I have not seen frequently discussed is adding different query engines on the same data for the purpose of optimizing query performance based upon the context of the query and the latency of the data, as per Figure 1.0 above. To be clear, I am not advocating duplication of data for the purpose of query performance. Rather, this architecture would enable Fabric Data Agents to choose the best query language for the context of the query. Hence the acronym I created for this blog article “NoDAX” which stands for “Not Only DAX.” NoSQL “Not Only SQL” is a real term referring to non-relational database systems designed for flexible schemas, horizontal scaling, and high-performance access to large, distributed datasets. NoDAX exists only within this blog post, and represents the availability of multiple query languages within a single Fabric Data Agent. The best query language can be used based on the context of the question and query. Not only DAX but also SQL, Kusto, and Ontologies can be queried to generate the most efficient contextual query. Figure 1.1 – Explaining the NoDAX acronym created for this article Why does this use case matter? The ability to query both the past and the present in one interface isn’t just a novel technical capability. Many analytic solutions can benefit from a pattern of [analysis + action]. Analytics by itself is great at understanding the past. We can look at years of historical data to figure out patterns, drivers of performance, and what went right or wrong. Historical context is incredibly valuable, but understanding the past doesn’t improve outcomes in the future. The real value comes when we connect findings from the past to decisions we make right now and in the future. If you discover historical data patterns that lead to an inventory shortage, a fraud event, or a patient risk score increasing, you can apply that insight to the latest operational data and act on ongoing workflows. Analytics and action have to reinforce each other. Analytics without action doesn’t create value. But action without understanding can easily make things worse. I once had a veteran analytics manager say to me “Without carefully considered and vetted KPIs you are flying blind. But be careful, because a KPI without proper context and understanding will quickly become a blunt object that an empty suit uses to whack somebody over the head.” Figure 1.2 – The purpose of this architecture is to learn from the past to improve the present and future Different Query Languages without data duplication A Fabric real-time architecture can be part of a design pattern that is similar to if not a version of a Lambda architecture. With a Lambda architecture, hot path data is available for real-time alerting and analytics while cold path data is stored for deep and complete historical analytics and data science. Figure 1.3 – NoDAX architecture can query a lambda-style architecture via multiple query endpoints Per the diagram above, real-time data is available in Fabric ASAP and cycled through an Eventhouse. Historical data can either be batched into a Fabric Lakehouse / Warehouse or copied over from the Eventhouse. A Fabric Data Agent can then generate Kusto queries against the Eventhouse, SQL queries against the Lakehouse / Warehouse, or DAX queries against the Warehouse / Lakehouse via the Direct Lake Semantic Model. DAX is often the best query language for data having deep history with complext analytic logic. SQL can be the best query language for retrieving historical row-level information from a robust relational database. Kusto can be used to query what’s happening right now via a real-time Eventhouse. Lambda architectures have been around for years, so why is this architecture a new option? Past lambda architectures would have hot path data available in a streaming toolset such as Azure Eventhub, and then store historical cold path data in a tool such as Azure Data Lake. Hot path alerting and reporting was usually disparate from historical cold path analytics. Per the diagram 3.4 below, with Microsoft Fabric, you can now: implement both the hot and cold path in a single Fabric environment (Eventstream, Eventhouse, Lakehouse / Warehouse). Ontologies are also an option. query the hot and cold path data via a single agentic endpoint using a Fabric Data agent Query either the hot or cold path using the optimal query language for the context of the question (DAX, SQL, Kusto, Ontologies) Figure 1.4 – Fabric not only unifies components of lambda-style architecture, but Data Agent also unites the query endpoints for AI unification Example use case for Healthcare Here’s an example of a Healthcare use case: A user might ask a question “Show me the percentage of patients who had their pain scores checked every hour for gall bladder removals on floor 5 over the last 3 years.” This will ideally filter three years worth of data for patients with specific procedure codes, filtered for specific rooms, and calculate the pain score check compliance for those visits. This query is ideal for the DAX language with a Semantic Model. The user might then want to see details for a specific time period, and ask “Show me the pain score results for patients who had their gall bladder removed on July 3 2024 on floor 5.” A SQL query might be the best option here against the Fabric Warehouse or Lakehouse, since SQL is better than DAX at retrieving row-level information. Then the user might want to know what is happening today. “Show me the pain score checks for inpatients right now who had their gall bladder removed on floor 5.” The Kusto language can retrieve the information that streams into a Fabric Eventhouse via an Eventstream. Based on the findings, the user may take an action. With the example above, a user was able to query deep history with analytic logic, retrieve historical row-level information from a robust relational database, and then view what’s happening right now for those patients. Action can then be taken in the here and now. Here are some additional use cases for Finance, Supply Chain, and Manufacturing in addition to Healthcare: Figure 1.5 – Industry use cases for Fabric Data Agent with multiple endpoints Video Summary Below is my video summary and demo of the Fabric Data Agent NoDAX architecture: Configure Fabric Data Agent for NoDAX query patterns When more than one source is added to a Fabric Data Agent, by default the source used for a specific query will be chosen based on interpreting available metadata. The Data Agents have a field called "Agent instructions" which can be used to provide detailed instructions about choosing the right source for the right question. Here’s a screenshot of the Agent instructions: Figure 1.6 – Agent instructions will guide the Fabric Data Agent to the best query endpoint I would recommend extensive unit testing and iterative improvements to the Agent instructions based upon your own data and use cases. Here’s a few examples that worked for my initial testing. I would recommend much more robust and carefully designed prompts for a production solution, but this is a baseline of an approach I found to work based on the demo in the video above: The KQL database named SeattleFireEventHouse is a live stream of 911 calls to the fire department in the city of Seattle. Whenever someone asks for “most recent” or “newest” or “latest” use SeattleFireEventHouse The lakehouse SeattleFireLakehouse should be queried with a SQL statement when someone asks for a list of incidents before the year 2026. Use SQL to retrieve row level requests for historical data. The semantic model SeattleFireSemantic Model should be queried when questions ask about historical analytic trends such as call volume averages, Year over year changes, and queries that aggregate data for analytic queries.
Identifying removed Fabric Assets
Is there any way for a data steward to identify a deleted physical asset ingested from Fabric into Purview. The original asset has been deleted in Fabric and another has been created with the exact same name, both assets now show in the catalogue and there doesn't seem to be a way to distinguish clearly which is the deleted and which is the current.
Struggling with running DQ Scans (Long queuing and Retry Count Error Issues)
Hi everyone, I have been exploring Microsoft Purview Data Quality quite extensively. At this point, I have configured more than 4,000 data quality rules across more than 10 Microsoft Fabric capacities, each with a minimum capacity of F16. Fabric is the source for all assets registered in Purview. I have identified several issues with the product, but the two that are currently impacting me the most are the following: DQ scans failing with a generic error“Max Retry Count Reached. Ending Workflow. Current Task HandleError”The challenge is that the error message does not identify which rule is causing the failure. As a result, I have to troubleshoot manually by disabling groups of rules, rerunning the scans, and repeating the process until I find the problematic rule. This trial-and-error approach is very time-consuming, especially at this scale. This seems to be caused by issues in some of the DQ rules, even though all rules are marked as “Good to go” in Purview. When running Data Quality scans, I often receive the following error: DQ scans remain queued for a long timeI am not sure why this happens or what resource, orchestration, or scheduling constraint is causing the delay. Whenever I run these DQ scans, they remain in a Queued state for at least 10 minutes, even when there is nothing running on the Fabric capacities. Has anyone experienced similar behavior with Purview Data Quality at this scale? Specifically, I would appreciate any guidance on: How to identify which DQ rule is causing a scan failure Why scans remain queued even when Fabric capacity appears to be idle Whether there are known limitations or best practices for running thousands of DQ rules in Purview Thank you.🎉APAC FY26 Fabric Partner Community Year‑End Celebration
As we prepare to wrap up FY26, we’re closing the year the same way we built it — together. This year‑end celebration will be held as part of the final Fabric Engineering Connection calls of FY26, giving us space to pause, look back on what we built together, and celebrate the partners who make this community what it is. 🌎Americas & EMEA Wednesday, June 24 | 8:00–9:00 AM PT 🌍APAC Thursday, June 25 | 1:00–2:00 AM UTC / Wednesday, June 24 | 5:00–6:00 PM PT) ✨ What to expect: A look back at the moments that defined FY26 along with partner updates to take you into FY27 Fun & games — including a Mad Libs–style community story built live by partners A community toast and a few surprises along the way 👉 Important: This call is open to members of the Fabric Partner Community on Microsoft Teams. If you’re not already a member, you can join here: https://aka.ms/JoinFabricPartnerCommunity This isn’t just a year‑end recap. It’s a thank‑you to the partners who showed up, shared openly, asked great questions, and helped each other grow real Microsoft Fabric practices. Mark your calendars. We can't wait to celebrate with you! 🥳 🥂🎉 Americas/EMEA FY26 Fabric Partner Community Year‑End Celebration
As we prepare to wrap up FY26, we’re closing the year the same way we built it — together. This year‑end celebration will be held as part of the final Fabric Engineering Connection calls of FY26, giving us space to pause, look back on what we built together, and celebrate the partners who make this community what it is. 🌎Americas & EMEA Wednesday, June 24 | 8:00–9:00 AM PT 🌍APAC Thursday, June 25 | 1:00–2:00 AM UTC / Wednesday, June 24 | 5:00–6:00 PM PT) ✨ What to expect: A look back at the moments that defined FY26 along with partner updates to take you into FY27 Fun & games — including a Mad Libs–style community story built live by partners A community toast and a few surprises along the way 👉 Important: This call is open to members of the Fabric Partner Community on Microsoft Teams. If you’re not already a member, you can join here: https://aka.ms/JoinFabricPartnerCommunity This isn’t just a year‑end recap. It’s a thank‑you to the partners who showed up, shared openly, asked great questions, and helped each other grow real Microsoft Fabric practices. Mark your calendars. We can't wait to celebrate with you! 🥳 🥂🎉 Save the Date: FY26 Fabric Partner Community Year‑End Celebration
As we prepare to wrap up FY26, we’re closing the year the same way we built it — together. This year‑end celebration will be held as part of the final Fabric Engineering Connection calls of FY26, giving us space to pause, look back on what we built together, and celebrate the partners who make this community what it is. 🌎Americas & EMEA Wednesday, June 24 | 8:00–9:00 AM PT 🌍APAC Thursday, June 25 | 1:00–2:00 AM UTC / Wednesday, June 24 | 5:00–6:00 PM PT) ✨ What to expect: A look back at the moments that defined FY26 along with partner updates to take you into FY27 Fun & games — including a Mad Libs–style community story built live by partners A community toast and a few surprises along the way 👉 Important: This call is open to members of the Fabric Partner Community on Microsoft Teams. If you’re not already a member, you can join here: https://aka.ms/JoinFabricPartnerCommunity This isn’t just a year‑end recap. It’s a thank‑you to the partners who showed up, shared openly, asked great questions, and helped each other grow real Microsoft Fabric practices. Mark your calendars. We can't wait to celebrate with you! 🥳 🥂On the Next Fabric Engineering Connection
Coming up on the next Fabric Engineering Connection calls, we’re focusing on one of the most important areas for partners right now: data protection, networking, and security in Microsoft Fabric. 🔐 What to expect 🎤 Recent Data Protection Value and Announcements (Americas & EMEA) presented by Yael Biss Covering the latest data protection capabilities in Fabric—designed to help partners meet security, compliance, and governance requirements while enabling customers to scale with confidence. 🎤 Updates + AMA with Networking and Data Security Team (Americas/EMEA & APAC) presented by Sarabjit D., Sumiran Tandon, Advaitha Karthikeyan, PMP®, and Bodhisatva Gautam This is a great opportunity to engage directly with the engineering team working on key scenarios including: ✅ Private Links ✅ Managed Private Endpoints (MPEs) ✅ Outbound Access Protection ✅ Customer Managed Keys (CMK) If you're advising customers on secure Fabric deployments, networking isolation, or enterprise‑grade governance, this session is definitely one to join. 🔒 Note: Fabric Engineering Connection calls are hosted exclusively in the Fabric Partner Community. Microsoft partners can join here the Community by submitting the form at https://aka.ms/JoinFabricPartnerCommunity. Looking forward to the discussion next week!
Driving AI‑Powered Healthcare: A Data & AI Webinar and Workshop Series
Across these sessions, you’ll learn how healthcare organizations are using Microsoft Fabric, advanced analytics, and AI to unify fragmented data, modernize analytics, and enable intelligent, scalable solutions, from enterprise reporting to AI‑powered use cases. Whether you’re just getting started or looking to accelerate adoption, these sessions offer practical guidance, real‑world examples, and hands‑on learning to help you build a strong data foundation for AI in healthcare. Date Topic Details Location Registration Link May 6 Webinar: Microsoft Fabric Foundations - A Simple Path to Modern Analytics and AI Discover how Microsoft Fabric consolidates fragmented analytics into a single integrated data platform, making it easier to deliver trusted insights and adopt AI without added complexity. Virtual Register May 13 Webinar: Reduce BI Sprawl, Cut Cost and Build an AI-Ready Analytics Foundation Learn how Power BI enables enterprise BI consolidation, consistent metrics, and secure, scalable analytics that support both operational reporting and emerging AI use cases. Virtual Register May 19-20 In Person Workshop: Driving AI‑Powered Healthcare: Advanced Analytics, AI, and Real‑World Impact Attend this two‑day, in‑person event to learn how healthcare organizations use Microsoft Fabric to unify data, accelerate AI adoption, and deliver measurable clinical and operational value. Day 1 focuses on strategy, architecture, and real‑world healthcare use cases, while Day 2 offers hands‑on workshops to apply those concepts through guided labs and agent‑powered solutions. Chicago Register May 27 Webinar: Unified Data Foundation for AI & Analytics - Leveraging OneLake and Microsoft Fabric This session shows how organizations can simplify fragmented data architectures by using Microsoft Fabric and OneLake as a single, governed foundation for analytics and AI. Virtual Register June 3-4 In Person Workshop: Driving AI‑Powered Healthcare: Advanced Analytics, AI, and Real‑World Impact Attend this two‑day, in‑person event to learn how healthcare organizations use Microsoft Fabric to unify data, accelerate AI adoption, and deliver measurable clinical and operational value. Day 1 focuses on strategy, architecture, and real‑world healthcare use cases, while Day 2 offers hands‑on workshops to apply those concepts through guided labs and agent‑powered solutions. New York Register June 10 Webinar: From Data to Decisions: How AI Data Agents in Microsoft Fabric Redefine Analytics Join us to learn how Fabric Data Agents enable users to interact with enterprise data through AI‑powered, governed agents that understand both data and business context. Virtual RegisterAPAC Fabric Engineering Connection Call
After a two‑week pause for FABCON & SQLCON - The Microsoft Fabric & SQL Community Conferences, we’re excited to welcome partners back for our first Fabric Engineering Connection call since the conference. Welcome back—and what a great way to restart the conversation! 🙌 This week’s sessions bring partners closer to the people building Microsoft Fabric, with timely insights and takeaways straight from FabCon. 🎙 What’s on the agenda: Fabric AI‑Powered Automation for Pro‑Developers (Americas & EMEA) presented by Evelina Alroy-Brin and Hasan Abo-Shally Recap of Data Warehouse announcements from FabCon presented by Rakesh Krishnan and Tino Tereshko 🇺🇦 🌍 Session times: Americas & EMEA: Wednesday, March 25 | 8–9 AM PT APAC: Thursday, March 26 | 1–2 AM UTC / Wednesday, March 25 | 5–6 PM PT These calls are a great opportunity to reconnect after FabCon, hear directly from engineering, and dig deeper into what’s new—and what’s next—for Microsoft Fabric. 👉 Participation is open to members of the Fabric Partner Community. Join here: https://aka.ms/JoinFabricPartnerCommunity
