Optimizing Oracle-to-Power BI Performance and the Path to Modern Analytics
Organizations frequently rely on Oracle databases for large-scale operational workloads while using Microsoft Power BI as their enterprise analytics and visualization platform. Although connecting Power BI directly to Oracle is fully supported, performance can become challenging as datasets grow into the millions of rows. A recent enterprise scenario involving millions of Oracle records illustrates an important principle: Oracle-to-Power BI performance should not be attributed to a single component without first isolating the entire retrieval path. This article provides a practical framework for optimizing that architecture today, while also examining how Microsoft Fabric Mirroring can provide a different data-access pattern for future analytics workloads. The Performance Challenge Consider an Oracle environment containing approximately 15 million records. Testing showed substantially different behavior depending on the client used to retrieve the same Oracle data: A direct Oracle query executed through TOAD returned millions of rows in about eight minutes. Tableau connected to the same Oracle table from the same laptop and refreshed millions of rows in 20 to 30 minutes. Power BI Desktop, however, continued processing for multiple hours before the operation was terminated by network or database controls. Importantly, the Power BI query did not contain additional Power Query transformations. The customer was directly executing SQL and loading the resulting data. This is an extremely useful diagnostic comparison because it demonstrates that Oracle is capable of returning the underlying dataset considerably faster than the Power BI test. It does not, however, prove that the Power BI Oracle connector itself is the single root cause. Performance Is a Pipeline, Not a Single Component Oracle-to-Power BI performance involves several layers: A bottleneck at any one of these layers can affect overall refresh or query performance. Therefore, the first architectural principle is: Measure every layer before attempting to tune the entire solution. A troubleshooting exercise should evaluate: Oracle server-side execution time SQL execution plan Oracle provider/client behavior FetchSize behavior Network transfer Power Query evaluation Query folding Power BI model processing Report design Gateway behavior when the Power BI Service is introduced This approach avoids prematurely concluding that Oracle, Power BI, the gateway, or the connector is inherently responsible for the slowdown. Start With the Oracle Provider Power BI currently provides two important approaches for Oracle connectivity. Built-in Oracle provider Modern versions of Power BI Desktop include Microsoft's bundled Oracle Managed ODP.NET provider. This reduces the dependency on separately installing Oracle client software. For environments using the built-in provider, testing should be performed with a current Power BI Desktop release because provider behavior and defaults can change between releases. Oracle Client for Microsoft Tools Oracle Client for Microsoft Tools, or OCMT, remains another supported option. OCMT configures Oracle Data Provider for .NET and can therefore be tested independently from the built-in provider. This gives architects a useful isolation mechanism: Test A: Built-in Oracle provider vs. Test B: OCMT provider. Use the same: workstation SQL statement Oracle database table credentials network path expected row count Changing several variables simultaneously makes the results considerably harder to interpret. Understand FetchSize One of the most important areas identified during troubleshooting is FetchSize. FetchSize controls how much data the Oracle provider retrieves during each fetch operation. For large datasets, a small FetchSize can require many retrieval operations, increasing overhead. Increasing FetchSize allows the provider to retrieve more data at a time and may improve throughput. However, FetchSize tuning should be tested independently from the choice of Oracle provider. Switching to OCMT, for example, does not by itself demonstrate whether FetchSize tuning improves performance. Benchmark the same SQL and dataset using consistent test conditions and compare elapsed time and throughput. FetchSize controls how much Oracle data the provider pulls back in each retrieval batch. Think of FetchSize like the size of a delivery truck. Imagine Power BI needs to retrieve a very large dataset from Oracle. Small FetchSize: Oracle → 📦 → Power BI Oracle → 📦 → Power BI Oracle → 📦 → Power BI Oracle → 📦 → Power BI ...many retrieval operations Larger FetchSize: Oracle → 📦📦📦📦📦 → Power BI Oracle → 📦📦📦📦📦 → Power BI ...fewer retrieval operations The idea is simple: FetchSize determines how much data the Oracle provider fetches at a time. Examine Oracle Data Types FetchSize is not the only provider consideration. Schema matters. A troubleshooting package should include the table DDL rather than only a list of column names. Pay particular attention to large-object types such as: BLOB CLOB Large-object columns may materially change retrieval behavior and can make simply increasing FetchSize insufficient. Therefore capture: complete table DDL column names Oracle data types row width presence of BLOB/CLOB columns expected total result size Do not optimize FetchSize without understanding the shape of the data being transferred. Filter Rows and Columns as Early as Possible Probably the most universally applicable Power Query optimization is also one of the simplest: Do not retrieve data you do not need. Instead of retrieving every row and column and filtering afterward, reduce the dataset at Oracle whenever possible. Good Oracle SQL should therefore: select only required columns apply restrictive predicates at the source aggregate at Oracle where appropriate avoid unnecessary data transfer This matters substantially when Power BI and Oracle are separated by a network boundary. Preserve Query Folding Power Query query folding allows Power BI transformations to be translated back into SQL executed by the relational source. For relational sources such as Oracle, maintaining folding can substantially reduce the amount of work performed by the Power Query mashup engine. The preferred pattern is: Where practical: filter early remove unnecessary columns early avoid transformations that unexpectedly break folding inspect where folding stops push relational operations back into Oracle For DirectQuery, folding is particularly important because the source remains part of the interactive report-query path. Be Careful With Native SQL and Incremental Refresh Native SQL can be extremely valuable for troubleshooting. Running the same targeted SELECT statement through Power BI that is executed directly against Oracle creates a useful A/B test. It helps isolate questions such as: Is Oracle itself taking a long time? Is Power BI generating a different query? Is additional processing happening after Oracle returns the data? Is the retrieval/provider layer adding significant latency? However, architecture matters. Power BI guidance warns that later Power Query steps cannot necessarily fold after a native SQL query, and native SQL has important implications for incremental refresh. Therefore, native SQL is useful both as an optimization technique in some architectures and as a diagnostic isolation technique, but it should not automatically be substituted for a well-folded Power Query design. Prefer Import for Traditional Oracle BI Workloads If the business does not require near-real-time data, Import should generally be evaluated before DirectQuery for large Oracle analytical workloads. With Import: The latency of Oracle, networking, and the provider is mostly experienced during refresh rather than repeatedly during user interaction. This usually provides a more predictable interactive experience. DirectQuery produces a different dependency chain: As a result, source latency can become report latency. Use DirectQuery when its architectural benefits are required, rather than simply because the source database supports it. Make Incremental Refresh Truly Incremental Import does not necessarily mean repeatedly loading the entire Oracle dataset. For large fact tables, incremental refresh should be evaluated so that only required partitions are refreshed. However, there is an important condition: RangeStart and RangeEnd filtering needs to be pushed back to Oracle. If partition filters do not fold, Power BI can end up retrieving considerably more source data than expected before applying the filtering. Therefore incremental refresh testing should verify that the Oracle query generated by Power BI contains the expected predicates. Treat the Gateway as Its Own Performance Layer Once the workload moves into the Power BI Service, however, the on-premises data gateway becomes another measurable component. For production deployments examine: gateway version Oracle provider/client version CPU memory network proximity to Oracle refresh concurrency DirectQuery concurrency competing workloads Where necessary, large refresh workloads and latency-sensitive DirectQuery workloads can be isolated onto different gateway resources. Do not diagnose gateway performance from a Power BI Desktop test where the gateway is not involved. Optimize DirectQuery If It Cannot Be Avoided Sometimes Import is not viable because users require data with very low latency. If Oracle must remain behind DirectQuery, reduce unnecessary work at every level. At Oracle index frequently filtered columns appropriately review execution plans select only necessary columns reduce returned rows avoid unnecessarily expensive calculations In Power Query maintain query folding push filtering to Oracle minimize transformations requiring local evaluation In the semantic model use efficient dimensional modeling avoid unnecessary high-cardinality columns consider aggregation strategies where appropriate In reports limit visual density use selective filters avoid generating unnecessary simultaneous source queries test with realistic user concurrency DirectQuery performance needs to be assessed as an end-to-end architecture, rather than as an individual SQL benchmark. Collect Diagnostics Before Declaring a Root Cause For difficult Oracle performance cases, establish a repeatable diagnostic package. Capture: Oracle database version execution plan SQL execution duration table DDL data types row count approximate returned data size Provider built-in provider versus OCMT provider/client version FetchSize configuration Power BI Power BI Desktop version generated SQL Query Diagnostics Mashup traces Applied Steps or M code total execution time Network Oracle-to-client path throughput latency connection interruptions or timeouts Comparison clients If Tableau or another Oracle client appears dramatically faster, reproduce the same SQL, same columns, same records, same database and preferably the same machine. Only then does the comparison become a useful engineering benchmark. Future Architecture: Oracle Mirroring in Microsoft Fabric Optimizing the direct Oracle connection solves one class of problem. Microsoft Fabric Mirroring provides a different architecture altogether. Instead of repeatedly placing Oracle in the analytical query path, Fabric can continuously replicate Oracle changes into OneLake. Conceptually: Fabric Mirroring uses Oracle LogMiner to capture changes and maintain an analytical copy in OneLake. This separates many downstream analytical reads from the operational Oracle database. The key architectural advantage isn't simply "a faster connector." The advantage is decoupling analytics from the operational database. Recommended Architecture Decision Tree Requirement: traditional BI with scheduled freshness Use: Oracle → Import → Power BI semantic model. Then optimize: FetchSize provider query folding source-side filtering incremental refresh Requirement: operational or near-real-time reporting where Oracle must remain the source Evaluate: Oracle → DirectQuery → Power BI. But aggressively optimize: Oracle SQL execution plans query folding visual density provider configuration concurrency gateway architecture Requirement: enterprise analytics with frequent Oracle changes and growing BI/AI requirements Evaluate: Oracle → Fabric Mirroring → OneLake → Fabric / Power BI. This reduces repeated analytical pressure against the operational Oracle environment and creates a reusable analytical data foundation for additional workloads. A Practical Troubleshooting Sequence A useful Oracle-to-Power BI performance investigation can be organized into five phases. Phase 1: Establish the Oracle baseline Execute exactly the same query directly against Oracle. Record: SQL → rows → bytes → execution time Phase 2: Isolate the provider Run the same dataset through: Built-in Oracle provider and OCMT. Record the results independently. Phase 3: Test FetchSize Test controlled FetchSize values while holding every other variable constant. Measure throughput rather than relying on subjective impressions. Phase 4: Optimize Power Query Validate: query folding source filtering column projection transformations incremental refresh behavior Phase 5: Decide whether the architecture itself should change If extremely large Oracle datasets must continually feed analytical workloads, consider whether repeatedly querying Oracle remains the appropriate long-term design. At that point, Fabric Mirroring becomes an architectural discussion rather than simply another troubleshooting step. Final Takeaway Oracle-to-Power BI performance problems should not immediately be categorized as an Oracle problem, a Power BI problem, or a gateway problem. They are usually best investigated as a data retrieval pipeline. For existing Power BI architectures, the strongest optimization sequence is: Efficient Oracle SQL For the longer term, Microsoft Fabric Mirroring changes the architecture: Oracle becomes the operational system of record while OneLake becomes the analytical data plane. That separation can be particularly valuable as organizations expand from traditional Power BI reporting into broader Fabric analytics, data engineering and AI workloads.
Amplify Healthcare Intelligence: Data, AI, and Agent-Powered Transformation
Join Microsoft for Amplify Healthcare Intelligence, a webinar and in-person workshop series designed for healthcare and life sciences organizations building the foundation for trusted AI. Every session starts from the same premise: you cannot deliver trusted AI without a trusted data foundation. Across the series you will see how leading organizations unify their data estate, ground AI agents in real business context, and turn that foundation into results they can measure. What You Will Learn How to build a unified, AI-ready data foundation across a fragmented healthcare data estate How to ground AI agents in trusted healthcare data and real business context How to modernize analytics while reducing complexity and cost How to accelerate innovation with Microsoft Fabric, Azure AI Foundry, Microsoft IQ, and Copilot technologies How to deliver measurable impact across clinical, operational, research, and business scenarios Whether you are defining your AI strategy, modernizing your analytics platform, or scaling AI across your organization, these sessions offer practical guidance, real-world customer examples, and hands-on learning to help you move from AI ambition to business impact. Webinars & In-Person Workshops 🎥 Webinars One-hour virtual sessions with actionable guidance, live demonstrations, customer stories, and best practices from Microsoft experts. Each webinar shows how leading healthcare organizations are turning data, AI, and enterprise intelligence into measurable business outcomes. All sessions are from 3-4 ET (12-1 PT) and are free to attend. Date Topic Register Oct 28 3-4 ET (12-1 PT) Reduce BI Sprawl, Cut Cost and Build an AI-Ready Analytics Foundation Register Nov 11 3-4 ET (12-1 PT) Connecting Microsoft Fabric with ExpressRoute: Enterprise Network Architecture Register Nov 18 3-4 ET (12-1 PT) From Question to Visuals: Data Insights with Fabric, Rayfin & GitHub Copilot Register Dec 2 3-4 ET (12-1 PT) The Intelligent Healthcare Enterprise: Microsoft IQ in Action Register Dec 9 3-4 ET (12-1 PT) From Trusted Data to Action: Microsoft Fabric + Copilot Cowork for Healthcare & Life Sciences Register Cannot join live? Register anyway. We will send you the recording and session materials after the event. Additional sessions will be added through the end of the year, so check back or register for one session to be notified as new dates are announced. 🏢 In-Person Workshops Our two-day workshops combine executive strategy, healthcare-specific use cases, architecture guidance, and hands-on labs designed to help teams identify and accelerate high-value AI opportunities. Attendance is free. Participants are responsible for their own travel and accommodation, and space at each location is limited. Workshops run 9am to 4pm local time on both days. Day 1: From Healthcare Data to Healthcare Intelligence Day 1 focuses on healthcare transformation strategy, customer examples, and the architectural patterns that make trusted AI possible at scale. The day closes with a networking reception and peer exchange. The Frontier Transformation imperative: from AI ambition to measurable impact Microsoft IQ: turning data into enterprise intelligence Building the unified data foundation Building trusted AI: security, governance, privacy, and compliance Healthcare transformation in action: clinical, operational, research, and finance scenarios Activating data with AI data agents and Copilot experiences Day 2: Hands-On Healthcare AI and Analytics Lab Day 2 is a guided, end-to-end lab. Participants build a working healthcare intelligence solution from raw data through to a grounded AI agent, using Microsoft Fabric, Azure AI Foundry, Copilot technologies, and modern data architectures. Build the foundation: data ingestion, lakehouse architecture, and data engineering Create actionable insights: semantic models and dashboards Prepare data for AI: AI-ready data assets and data governance Build and ground AI agents in trusted enterprise data From insight to intelligent action: planning your organization's next steps What to bring: a laptop with a current browser. Lab environments and credentials are provided on site, and no prior Fabric or Foundry experience is assumed. Date City Venue Register November 10-11, 2026 9am - 4pm Chicago Microsoft Chicago (AON Center) 200 East Randolph Drive, Suite 200 Chicago, IL 60601 Register December 8-9, 2026 9am - 4pm New York Microsoft Garage 300 Lafayette Street New York, NY 10012 Register Who Should Attend This series is built for the people who own the data estate and the people who depend on it. Sessions are technical enough for practitioners and strategic enough for the leaders who prioritize the work. Data & analytics leaders Data platform, data engineering, and business intelligence teams Data architects, engineers, and data scientists AI and innovation leaders Healthcare and life sciences executives Clinical, operational, research, and finance transformation leaders No prior Microsoft Fabric or AI experience is required for any session in this series. Questions Contact Camille Whicker if you have any questions about our upcoming events.Amplify Healthcare Intelligence In-Person Workshop Series
What You'll Learn How leading organizations are navigating the Frontier Transformation journey Strategies for turning healthcare data, workflows, and expertise into organizational intelligence Best practices for building a unified, AI-ready data foundation Approaches for implementing trusted AI with strong governance and security Real-world healthcare scenarios across clinical, operational, financial, and research domains How AI agents can transform the way people interact with data and make decisions Practical skills for deploying analytics, AI, and agent-powered solutions Day 1: From Healthcare Data to Healthcare Intelligence Explore how leading healthcare organizations are creating a trusted foundation for AI and transforming data into meaningful business outcomes. The Frontier Transformation Imperative The future of healthcare in the age of AI and agents Industry trends, opportunities, and challenges Moving from AI ambition to measurable impact Microsoft IQ: Turning Data into Enterprise Intelligence Connecting data, workflows, and organizational knowledge Building organizational intelligence for AI Establishing the foundation for trusted decision-making Building the Unified Data Foundation Modernizing the healthcare data estate Creating a single source of truth for analytics and AI Leveraging Microsoft Fabric and OneLake as a unified platform Building Trusted AI Security, governance, privacy, and compliance Responsible AI considerations for healthcare and life sciences Preparing data and processes for enterprise AI adoption Healthcare Transformation in Action Clinical transformation scenarios Operational transformation scenarios Research and innovation use cases Finance and performance management opportunities Activating Data with AI Agents AI Data Agents and Copilot experiences From data to decision-making Real-world examples of agent-powered healthcare innovation Networking Reception & Peer Exchange Day 2: Hands-On Healthcare AI & Analytics Lab Gain practical experience building an end-to-end healthcare intelligence solution using Microsoft technologies. Build the Foundation Data ingestion and modernization patterns Lakehouse architecture and data engineering Establishing a scalable healthcare data platform Create Actionable Insights Semantic models and business intelligence Power BI dashboards and reporting Real-time healthcare intelligence Prepare Data for AI Building AI-ready data assets Data governance and trust Optimizing data for intelligent applications Build and Ground AI Agents Creating AI Data Agents Leveraging business context and trusted enterprise data Connecting agents to analytics and operational workflows From Insight to Intelligent Action End-to-end healthcare AI scenario Data → Insights → Agents → Outcomes Planning your organization's next steps Upcoming Workshop Locations Date City Venue Register September 9-10, 2026 9am - 4pm Dallas Microsoft Las Colinas 1 7000 N State Hwy 161 Irving, TX 75039 Register September 29-30, 2026 9am - 4pm Minneapolis Microsoft Edina 3601 West 76th Street, Suite 600 Edina, MN 55435 Register October 13-14, 2026 9am - 4pm Boston Microsoft New England One Memorial Drive Cambridge, MA 02142 Register October 27-28, 2026 9am - 4pm Silicon Valley Microsoft Silicon Valley 1045 La Avenida Street Mountain View, CA 94043 Register November 10-11, 2026 9am - 4pm Chicago Microsoft Chicago (AON Center) 200 East Randolph Drive, Suite 200 Chicago, IL 60601 Register December 8-9, 2026 9am - 4pm New York Microsoft Garage 300 Lafayette Street New York, NY 10012 Register Who Should Attend? Healthcare and Life Sciences Executives CIOs, CTOs, and Chief Data Officers Clinical and Operational Leaders Analytics and Data Platform Teams AI and Innovation Leaders Finance Transformation Teams Architects, Engineers, and Data Professionals What You'll Walk Away With By the end of the workshop, you'll have a clear understanding of how to build a unified data and AI foundation, enable trusted AI and intelligent agents, and create a roadmap for accelerating healthcare transformation within your organization.Meet the IQ's: How Microsoft is Creating Context-Aware AI
Microsoft Architect's: Allison Rose allisonrose, Lavanya Sreedhar LavanyaSreedhar, Tom Dinh Tom-Dinh, Oviya Soundararajan oviyasound and Rafia Aqil Rafia_Aqil The AI era demands more than powerful language models. It demands context a deep understanding of what enterprise data means, how it connects, and how AI systems can reason and act on it intelligently. Microsoft has been building the foundational intelligence layer that makes this possible: a family of capabilities collectively known as the IQ Platform. The Microsoft IQ Platform is not a single product but a set of complementary intelligence layers: Work IQ, Fabric IQ, Foundry IQ and Web IQ each designed to inject rich contextual understanding into a different part of the enterprise technology stack. Together, they represent Microsoft’s strategic vision for how AI can move beyond isolated answers and become a true operating system for organizational intelligence. This article unpacks each IQ, explains the problems they solve, and explores how they work together to power the next generation of AI-driven enterprise workflows. How the IQs Work Together? Work IQ, Fabric IQ, Foundry IQ, and Web IQ are not competing products or overlapping investments. They are complementary intelligence layers designed to operate across different contexts within the enterprise, and they are most powerful when combined. Work IQ brings the intelligence of Microsoft 365 to every agent and Copilot experience- connecting people, conversations, documents, and organizational signals into a semantic layer that understands how work happens. Fabric IQ brings the intelligence of enterprise data and business context- teaching AI not just what the data says, but what it means in the language of your business: entities, relationships, rules, and governed actions. Foundry IQ brings the infrastructure intelligence that enables all of this to scale- eliminating the undifferentiated plumbing of agentic AI and letting teams focus on building the workflows that actually differentiate their business. Web IQ brings fresh, external web context, helping agents consider current and relevant public information alongside the knowledge and business context inside the organization. Microsoft Web IQ grounds most of the top AI platforms, including Copilot and ChatGPT. Together, the IQ platform represents Microsoft’s answer to one of the defining challenges of the AI era: not just making AI more capable, but making AI contextually aware-grounded in the real knowledge, relationships, and intent of your organization. Fabric IQ: Teaching AI the Language of Business Microsoft Fabric is an end-to-end, unified data analytics platform centered on OneLake- a centralized data lake that stores all analytical and operational business data in open Delta format. Because every Fabric compute experience (Data Engineering, Data Warehouse, Data Factory, Power BI, and Real-Time Intelligence) natively reads from OneLake, organizations gain a single source of truth without copying or duplicating data. OneLake also provides mirroring and shortcut capabilities so existing data can be accessed in place, wherever it lives. Most organizations have made significant progress consolidating their data. The harder challenge is giving AI- and the people who use it-the ability to reason about that data in business terms, not technical ones. Outside of data professionals, businesses do not talk about tables or schemas. They talk about entities that matter to them. Fabric organizes data. Fabric IQ teaches AI what that data means. Three Layers of Business Context Fabric IQ introduces three intelligence layers that together create a unified, contextually rich environment for enterprise AI: Unified Data Layer: Delivered through OneLake and the OneLake Catalog, this provides a single source of truth for all structured and unstructured data across the organization. Business Intelligence Layer: Delivered through Power BI Semantic Models, this layer provides curated measures, hierarchies, dimensions, and trusted KPIs- translating raw data into the analytical language of your business. Operational Intelligence Layer: This is where Fabric IQ’s most distinctive capability lives: Ontology. An Ontology is a model of your business- a graph of entities (such as Patient, Provider, Product, or Account), the relationships between them, the business rules that govern them, and the actions AI agents can take. It functions as the brain that enables AI to understand business context and act on it in a governed, explainable way. Together, these three layers create shared context across all business data stored in OneLake-enabling modern businesses, people, and AI to operate as one unified system. A Real-World Example: Healthcare Consider a care management executive asking: “Which diabetic patients discharged in the last 30 days are at high risk of readmission because they missed follow-up appointments, had medication adherence issues, and recently visited the Emergency Department?” Without Fabric IQ, answering this requires analysts to manually join EHR data, appointment systems, pharmacy records, and ED utilization data- writing SQL across multiple datasets and validating business logic with clinicians. It is slow, brittle, and error-prone. Semantic models can curate data for reporting and analysis, but they do not provide enterprise-scale context integration. With Fabric IQ, an Ontology can be created with entities like Patient, Encounter, Provider, Medication, Diagnosis, Appointment, and Care Plan- each bound to Lakehouse tables, Eventhouse tables, or Materialized Views. Relationships describe how patients connect to their diagnoses, medications, appointments, and treating providers. Business rules enforce data quality, identifying missed follow-ups, recent Emergency visits, and medication gaps. The result is a shift from siloed analytics to true system-level intelligence- an organization where data, AI, and people operate from a shared understanding of the business. Foundry IQ: From Infrastructure to Intelligence Building production-grade AI agents has traditionally meant writing a significant amount of undifferentiated plumbing, custom retrieval pipelines, memory systems, ranking logic, and orchestration code just to enable core RAG and agentic capabilities. While powerful, this approach often leads to complex, hard-to-maintain codebases that distract from the real goal: solving domain-specific problems. With Foundry IQ, Microsoft is fundamentally changing that model by turning these underlying capabilities into managed platform services, allowing teams to shift from building infrastructure to focusing on intelligent workflows. Foundry IQ acts as part of Microsoft's managed platform, enabling agents to use agentic reasoning to access, process, and act on knowledge from anywhere. It is Microsoft Foundry’s way of turning the undifferentiated plumbing behind a RAG agent, such as retrieval, ranking, citations, memory, and personalization, into managed, server-side services that you provision once and call through clean interfaces. Foundry IQ allows you to remove the infrastructure you never wanted to own in the first place. What This Means in Practice Instead of stitching together retrieval pipelines, embedding logic, ranking strategies, and memory mechanisms, Foundry IQ centralizes these capabilities into a single, opinionated platform layer that agents can directly consume. Developers no longer design and maintain each component individually. The knowledge base becomes the centerpiece of the workflow. Rather than coordinating multiple services and response handlers, applications make a single call to retrieve grounded context. Vector-semantic-hybrid querying, query planning, semantic ranking, and citation generation are all encapsulated within the provisioned knowledge base-with no retrieval or embedding logic to maintain in the client application. Memory follows the same pattern of abstraction. Instead of multiple classes and helper utilities to manage storage, user profiles, summarization, and context reconstruction, Foundry IQ replaces this entire layer with a single memory provider backed by a service-managed store with built-in capabilities for chat summarization and user-profile extraction. A Real-World Example: Clinical Workflows Consider building an AI-powered clinical workflow application. Previously, features like agent memory, knowledge base retrieval for grounding, and personalization all had to be written as custom logic and wired manually into the application. This resulted in thousands of lines of code, numerous helper functions, and brittle architecture that was difficult to evolve. With Foundry IQ, that same solution can be reimagined. A single provisioning script now stands up all required services and executes the data-plane steps to create a memory store, build the search index, and provision a Foundry IQ knowledge base for agentic retrieval. Because the top-level router agent carries its own memory, it can directly answer recalled context without relying on confidence thresholds, rule-based branching, or forced workflow paths. Conversation history is handled automatically at ingress- no custom thread management system required. What remains is only what was always worth building: domain-specific logic. Citation validation against grounded evidence. Hallucination checking using LLM-as-a-judge patterns. Agent revision loops. Everything else- retrieval, ranking, memory, user profiles, conversation management- is provisioned once and consumed as a platform capability. The result: a dramatically reduced surface area for bugs, significantly less code to maintain, and teams freed to focus entirely on the work that differentiates their product. Work IQ: Making Microsoft 365 Data Meaningful For years, Microsoft has given organizations API access to their Microsoft 365 data through the Microsoft Graph- emails, calendar events, OneDrive files, Teams conversations, and more. While valuable, this access essentially treated M365 as a structured database: query an endpoint, retrieve an artifact, parse the metadata. The problem was volume and context. With thousands of signals generated every day across the organization, customers needed a way to extract not just data but meaning. In the past year, Microsoft introduced a semantic index built on top of that raw M365 data- a layer that understands not just what exists in your ecosystem, but how everything relates to one another. This intelligence layer is Work IQ, and in an increasingly agent-driven world, it fundamentally changes what AI can do for your organization. In an AI-first world, the advantage is not simply in a model’s ability to reason- it’s in the richness of the context it can reason over. The Contrast in Action Consider asking an agent a simple question: “What’s the latest on Customer Contoso?” With the Microsoft Graph API alone, the agent must stitch together multiple endpoint queries- Teams chats, SharePoint documents, email threads and attempt to piece the results into a coherent answer. It lacks any connective tissue. It doesn’t know what’s relevant, what’s meaningful, or how these isolated data sources relate to each other. The burden of reasoning falls entirely on the agent. With Work IQ, that same prompt taps into a semantic layer that has already done the connecting. The agent knows Contoso-related details span a specific SharePoint folder, identifies the active Teams channel for progress tracking, and surfaces the key people involved. The response is grounded in a web of contextual relationships not just retrieved data. Three Core Components Work IQ is enabled by three powerful components: Data: Unifies signals from files, emails, meetings, chats, and other M365 business systems to capture how work actually gets done across your organization. Memory: Enables persistent context about how people and teams work: details inferred from past conversations, explicit memories stored with Copilot, and custom instructions you’ve configured. Each interaction allows Copilot to learn more about your priorities, preferences, and working style. Inference: Brings together skills, models, and tools to move work forward. It goes beyond understanding your work to deciding what should happen next. Data captures and indexes your M365 knowledge. Memory builds a personalized understanding of how you work. Inference translates this into action. Think of Work IQ as a specialized brain trained on who you are at work within the full context of what your organization knows. Web IQ: Bringing Real-World Intelligence into Agent Context An agent can understand your business and still miss information that matters outside it. For teams researching an account, investigating a supply-chain issue, or comparing products, internal knowledge is only part of the picture. The missing context may be on a company website, in a public announcement, or in newly published guidance. Web IQ is the part of Microsoft IQ that brings in outside information. It's a set of web search and grounding APIs that give agents current evidence from the web, beyond what the model learned in training and beyond a company's internal systems. Web IQ gives developers a way to ground AI applications and agents in fresh web content through Microsoft-hosted REST and MCP interfaces. An application can retrieve current, external information when it needs it, rather than relying only on the knowledge captured during model training. Web IQ supplies the grounding content; the application determines how to combine it with other context and use it in a response. This adds a distinct dimension to the IQ story: Information beyond the organization. The goal is not to replace trusted internal knowledge, but to help people examine it alongside current, relevant external evidence. Source links keep that evidence available for review, so users can check the information behind an answer. A Real-World Example: Sales Preparation Consider a seller preparing for a customer conversation. Work IQ might explain the relationship, outstanding commitments, and business priorities, and Web IQ could contribute public context, such as a recent company announcement. Bringing these inputs together in an agent could help the seller identify a more timely question to ask, while preserving the distinction between internal account information and external evidence. Get Started Whether you’re exploring how to ground your AI applications in richer organizational context, looking to reduce the infrastructure burden of building intelligent agents, seeking to make your enterprise data more actionable, or aiming to ground agents in current web information, the Microsoft IQ Platform offers a path forward. We encourage you to explore the Microsoft Fabric documentation, Azure AI Foundry resources, the Microsoft 365 developer platform, and the Web IQ documentation to learn more about how each IQ capability can fit into your architecture. Build Microsoft IQ powered agents, this cookbook walks through it step by step: files → Web IQ → Work IQ → Fabric IQ → MCP endpoint: https://lnkd.in/edrjG99F Select Microsoft IQ in your Copilot agent settings, follow step by step instructions here: Bring your enterprise data to every agent conversation We’d love to hear how you’re thinking about context-aware AI in your organization. Share your thoughts and questions in the comments below. Links: Microsoft IQ | Unified Enterprise Intelligence for AI Work IQ overview | Microsoft Learn What is Foundry IQ? - Microsoft Foundry | Microsoft Learn Fabric IQ documentation - Microsoft Fabric | Microsoft Learn Microsoft Web IQ documentation and Quick Start | Microsoft Web IQFabric Data Agents can choose Query Language based on Context
What if you could combine decades of historical analysis with live, real-time data in one seamless AI chat experience? Traditionally, analyzing large volumes of past data (Analytics / Business Intelligence) has been a separate architecture, or has at least required separate toolsets, from monitoring what’s happening right now (real-time, IoT, etc). With Fabric Data Agents, analytics for large volumes of historical data can be accessible with real-time data via a single AI query endpoint. Analytics can be used to gain understanding from historical data, and findings can be put into action for real-time scenarios, all through a single interface for the end users. Figure 1.0 – Fabric Data Agent can use different query languages for optimal performance with a lambda-style RTI architecture on Fabric Many of the demos I’ve seen for Fabric Data Agents will highlight the capability to connect to different types of queries and sources via a single endpoint such as Lakehouses (SQL), Warehouses (SQL), Semantic Models (DAX), Eventhouses (Kusto), and Ontologies. What I have not seen frequently discussed is adding different query engines on the same data for the purpose of optimizing query performance based upon the context of the query and the latency of the data, as per Figure 1.0 above. To be clear, I am not advocating duplication of data for the purpose of query performance. Rather, this architecture would enable Fabric Data Agents to choose the best query language for the context of the query. Hence the acronym I created for this blog article “NoDAX” which stands for “Not Only DAX.” NoSQL “Not Only SQL” is a real term referring to non-relational database systems designed for flexible schemas, horizontal scaling, and high-performance access to large, distributed datasets. NoDAX exists only within this blog post, and represents the availability of multiple query languages within a single Fabric Data Agent. The best query language can be used based on the context of the question and query. Not only DAX but also SQL, Kusto, and Ontologies can be queried to generate the most efficient contextual query. Figure 1.1 – Explaining the NoDAX acronym created for this article Why does this use case matter? The ability to query both the past and the present in one interface isn’t just a novel technical capability. Many analytic solutions can benefit from a pattern of [analysis + action]. Analytics by itself is great at understanding the past. We can look at years of historical data to figure out patterns, drivers of performance, and what went right or wrong. Historical context is incredibly valuable, but understanding the past doesn’t improve outcomes in the future. The real value comes when we connect findings from the past to decisions we make right now and in the future. If you discover historical data patterns that lead to an inventory shortage, a fraud event, or a patient risk score increasing, you can apply that insight to the latest operational data and act on ongoing workflows. Analytics and action have to reinforce each other. Analytics without action doesn’t create value. But action without understanding can easily make things worse. I once had a veteran analytics manager say to me “Without carefully considered and vetted KPIs you are flying blind. But be careful, because a KPI without proper context and understanding will quickly become a blunt object that an empty suit uses to whack somebody over the head.” Figure 1.2 – The purpose of this architecture is to learn from the past to improve the present and future Different Query Languages without data duplication A Fabric real-time architecture can be part of a design pattern that is similar to if not a version of a Lambda architecture. With a Lambda architecture, hot path data is available for real-time alerting and analytics while cold path data is stored for deep and complete historical analytics and data science. Figure 1.3 – NoDAX architecture can query a lambda-style architecture via multiple query endpoints Per the diagram above, real-time data is available in Fabric ASAP and cycled through an Eventhouse. Historical data can either be batched into a Fabric Lakehouse / Warehouse or copied over from the Eventhouse. A Fabric Data Agent can then generate Kusto queries against the Eventhouse, SQL queries against the Lakehouse / Warehouse, or DAX queries against the Warehouse / Lakehouse via the Direct Lake Semantic Model. DAX is often the best query language for data having deep history with complext analytic logic. SQL can be the best query language for retrieving historical row-level information from a robust relational database. Kusto can be used to query what’s happening right now via a real-time Eventhouse. Lambda architectures have been around for years, so why is this architecture a new option? Past lambda architectures would have hot path data available in a streaming toolset such as Azure Eventhub, and then store historical cold path data in a tool such as Azure Data Lake. Hot path alerting and reporting was usually disparate from historical cold path analytics. Per the diagram 3.4 below, with Microsoft Fabric, you can now: implement both the hot and cold path in a single Fabric environment (Eventstream, Eventhouse, Lakehouse / Warehouse). Ontologies are also an option. query the hot and cold path data via a single agentic endpoint using a Fabric Data agent Query either the hot or cold path using the optimal query language for the context of the question (DAX, SQL, Kusto, Ontologies) Figure 1.4 – Fabric not only unifies components of lambda-style architecture, but Data Agent also unites the query endpoints for AI unification Example use case for Healthcare Here’s an example of a Healthcare use case: A user might ask a question “Show me the percentage of patients who had their pain scores checked every hour for gall bladder removals on floor 5 over the last 3 years.” This will ideally filter three years worth of data for patients with specific procedure codes, filtered for specific rooms, and calculate the pain score check compliance for those visits. This query is ideal for the DAX language with a Semantic Model. The user might then want to see details for a specific time period, and ask “Show me the pain score results for patients who had their gall bladder removed on July 3 2024 on floor 5.” A SQL query might be the best option here against the Fabric Warehouse or Lakehouse, since SQL is better than DAX at retrieving row-level information. Then the user might want to know what is happening today. “Show me the pain score checks for inpatients right now who had their gall bladder removed on floor 5.” The Kusto language can retrieve the information that streams into a Fabric Eventhouse via an Eventstream. Based on the findings, the user may take an action. With the example above, a user was able to query deep history with analytic logic, retrieve historical row-level information from a robust relational database, and then view what’s happening right now for those patients. Action can then be taken in the here and now. Here are some additional use cases for Finance, Supply Chain, and Manufacturing in addition to Healthcare: Figure 1.5 – Industry use cases for Fabric Data Agent with multiple endpoints Video Summary Below is my video summary and demo of the Fabric Data Agent NoDAX architecture: Configure Fabric Data Agent for NoDAX query patterns When more than one source is added to a Fabric Data Agent, by default the source used for a specific query will be chosen based on interpreting available metadata. The Data Agents have a field called "Agent instructions" which can be used to provide detailed instructions about choosing the right source for the right question. Here’s a screenshot of the Agent instructions: Figure 1.6 – Agent instructions will guide the Fabric Data Agent to the best query endpoint I would recommend extensive unit testing and iterative improvements to the Agent instructions based upon your own data and use cases. Here’s a few examples that worked for my initial testing. I would recommend much more robust and carefully designed prompts for a production solution, but this is a baseline of an approach I found to work based on the demo in the video above: The KQL database named SeattleFireEventHouse is a live stream of 911 calls to the fire department in the city of Seattle. Whenever someone asks for “most recent” or “newest” or “latest” use SeattleFireEventHouse The lakehouse SeattleFireLakehouse should be queried with a SQL statement when someone asks for a list of incidents before the year 2026. Use SQL to retrieve row level requests for historical data. The semantic model SeattleFireSemantic Model should be queried when questions ask about historical analytic trends such as call volume averages, Year over year changes, and queries that aggregate data for analytic queries.Rayfin | Go from prompt to production backend
Describe the app you want using Rayfin’s open-source SDK with GitHub Copilot, and generate your full backend in code — schemas, relationships, and access policies included. Deploy to Microsoft Fabric with a single CLI command and immediately inherit enterprise data security, identity controls, and audit compliance already in place across your data estate. Connect your app’s live operational data to years of historical records in Fabric from the moment you deploy, no pipelines, no data movement. Query across both datasets using a Fabric data agent you spin up directly on your app’s data. Will Thompson, Microsoft Fabric Principal Product Manager, shares how to take an app from idea to governed production deployment in a single session. Describe the app you want. Rayfin and GitHub Copilot generate your full backend in code- schemas, relationships, and access policies. No manual wiring. See how it works. One CLI command. Fully managed SQL database, APIs, and static hosting in Microsoft Fabric. Rayfin handles provisioning automatically. Take a look. Spend less time configuring. Spend more time creating. Get managed backend services, authentication and authorization, data APIs, built-in analytics, and AI-ready architecture with Rayfin. Check it out. QUICK LINKS: 00:00 — Simplify backend complexity 01:20 — Home delivery service app 01:48 — Data analysis app 02:26 — See the build experience 03:08 — Copilot Generates Full Backend 03:47 — Authorization defined alongside schema 05:06 — One CLI Command Deploys to Fabric 05:21 — Create analytics app & add pages 06:31 — App Data Connects to Fabric Data Estate 06:55 — Conversational Data Agent on App Data 08:13 — Wrap up Link References Get started at https://aka.ms/rayfin Unfamiliar with Microsoft Mechanics? As Microsoft’s official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft. Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries Talk with other IT Pros, join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog Watch or listen from anywhere, subscribe to our podcast: https://microsoftmechanics.libsyn.com/podcast Keep getting this insider knowledge, join us on social: Follow us on Twitter: https://twitter.com/MSFTMechanics Share knowledge on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/ Enjoy us on Instagram: https://www.instagram.com/msftmechanics/ Loosen up with us on TikTok: https://www.tiktok.com/@msftmechanics Video Transcript: -What if building a real, production-ready enterprise app didn’t take months, but just a few hours? Today, AI can generate app logic and front ends almost instantly. And while everything works great with test data, the moment you try to connect that app to real enterprise data, everything slows down. Because now you have to wire up a database with access control, set up authentication, configure backend services, and make sure everything is secure and scalable. Your quick app turns into a complex backend project with multiple failure points, and that’s where Rayfin comes in. It’s designed to simplify backend definition in code and elevate the developer experience. Our open-source SDK and CLI grounds your agentic coding tools so that when you describe the app that you want to build, it can automatically define your app’s backend in code, including your data schemas, relationships, and access policies. -And then with a single CLI command, you deploy to our unified data and analytics platform, Microsoft Fabric. This spins up a fully managed backend with a SQL database with familiar billing and sharing models, as well as static hosting. APIs are generated automatically to ensure your app can communicate securely to data and services and Fabric’s identity and data security layer ensures that when your app accesses or queries data, it respects existing policies in place across your data estate. -To show how this works in action, I’ll start by showing two apps I built with Rayfin and then walk you through how I created them. My first app is a mobile home delivery service app for a home improvement retailer, with all orders delivered directly to customers’ doors. Drivers must be signed in to access the app, ensuring that every delivery is securely tied to an authenticated user. They can record deliveries, capture photos as proof, and even collect customer signatures to acknowledge receipt. And all this happens in real time. -Next, as a business, we want to learn from the data captured in our delivery app. So I’ve built a second app for data analysis. Because it runs in Fabric, the operational data is already connected to our broader enterprise data estate. It has access to years of customer data in Fabric, modeled and ready to use. And by connecting that data to the live data that our delivery app captures, retail managers now have immediate insight into how the new delivery service is impacting customer satisfaction. Now these apps were built in minutes. They’re not a proof of concept. They have real auth, real data, and real governance, and they work directly with enterprise data already in Fabric. -Let me walk you through the build experience. I’ll open a terminal and run the Rayfin create command, and this will walk me through my entire project setup. I can see that I have various templates that I can use to get started. So let’s go with the blank app template. It’s asking for a name and I’ll name it Delivery App. And you can see it’s creating the project. And under the covers, it’s installing all the necessary dependencies and adding the files I need, including the agent instructions for using the SDK and CLI. Now I can CD into my project and start the development server. Since deploying to Fabric, it prompts for a Fabric workspace name and I’m going to use Zava. And the deployment process is now in full swing with Fabric provisioning my foundational resources. -So now with our Rayfin foundation in place, let’s build the app. Using VS code with GitHub Copilot, I can now describe the app I want and Copilot knows how to build it, including the backend. Let’s try it out, I’ll type a prompt, build a Rayfin delivery app where drivers log deliveries with photos and item conditions and customers confirm receipt. I’m also going to start the Rayfin MCP server and this will help ground Copilot in all the latest best practices for Rayfin. As Copilot is getting to work, we can see it’s using the documentation for the MCP and CLI tool to understand the specifics of Rayfin. And Copilot will run independently using the skills defined by the Rayfin template. That’s going to take a few minutes, so we’re going to jump ahead to the results. -Here’s the app right here in code and there’s lots more to it, in fact, to understand the depth of what was created, let’s take a more detailed look. Inside the Rayfin folder, we can see our whole data schema, customers, deliveries, items, photos, drivers, and more. And if we take a look at how Copilot defined the driver entity, this is where Rayfin shines. The schema lives right in the code, which means instead of keeping the rules about your data in separate places like databases, SQL files, or configs, they’re built right into the code. So things are less likely to break and easier to keep consistent. Each field maps to a column in the database. ID name, email, and optional phone number with types and constraints built in. -And it captures relationships too. It calls out a driver has many deliveries. But here’s the key part, the access policy sits right next to the data. This rule says a user can only ever touch their own driver record. Authorization is defined alongside the schema not bolted on later. And if we open the delivery entity, you can see some of the scale Rayfin is capable of with richer relationships, defined with drivers, customers, and items, and a smarter policy where delivery can be read by an authenticated driver or its customer. And of course I can always customize any of this by hand if I need to. So that was the build experience. -Now let’s look at how we can deploy the app to Microsoft Fabric. I’ll ask Copilot to deploy our app and Copilot is using the Rayfin Up command to deploy everything we’ve defined straight to Fabric. And just like that, our app is deployed and available. Next, directly from Fabric, I’ll show you the steps to create our analytics app. I’ll start in the Fabric portal this time and I’ll call this app Delivery Analysis. This leverages a template that the Microsoft Power BI team built. It’s called a Data App template, and it gives our coding agent what it needs to connect to semantic models and create beautiful interactive visuals. -So I’ll select Data App, and now I can manually follow these steps or just copy the prompt and run it in VS Code to get started scaffolding my project. I’ll move over to VS Code and paste in the prompt and the agent knows exactly how to start up my new Rayfin app. Once it’s finished, you’ll see that the delivery analysis directory has been added. Now we can also add pages to our new app using Copilot. Here I want to add an insights page. So I’ll prompt, Add a customer satisfaction page, which shows historical CSAT scores from our linked semantic model and compare it against a given driver’s delivery ratings. The important detail here, I’m pointing it straight at our Fabric semantic model, so it’s using a preexisting and trusted business model. This process will run for several minutes, creating and executing a plan and querying our semantic model for what it needs. -To save time, I’ll jump back in near its completion. Now the app is ready and I can view it from the provided link. And we can see historical customer satisfaction data trending up since we launched the delivery service with details broken down by driver. Now, data isn’t moved between systems. Everything stays connected from the start. The backend is already connected to your enterprise data estate. And because all of your new data is now also landing in Fabric, there’s lots you can do on top of it. -Now that our app is live and generating data, the next step is making that data actionable through conversation. So instead of building another report or connecting the data to a Fabric notebook, I’m going to go ahead and create a data agent directly on top of this app’s data. I’ll name the agent, Zava Insights Agent, and next I’ll point it at the warehouse I configured earlier. This combines my operational database and historical data as the source. And now I can ask questions against my data just like I would to an analyst. -Let’s start with a simple question. I’ll ask, what is the average customer rating across all our deliveries? The agent understands the data model and returns the answer instantly as 4.56 out of five. But now I’ll go deeper. How does our customer satisfaction compare before and after we launch the new delivery service? It doesn’t just retrieve the data. It reasons across time and trends in the model. And we can see that it’s gone from 3.8 to 4.4 out of five since launching the new service. So instead of navigating dashboards, the business can now ask questions directly against the app and get answers grounded in the same governed data that powers the app. -And just like any Fabric artifact, I can share the app that we created earlier with other people in my organization. So let’s grant Lynne access. They were asking about the delivery analysis and now they can explore the data themselves without waiting on reports. Rayfin helps you go from idea to production faster. By using AI to build and deploying to Fabric, your app inherits existing data security controls, permissions and audit requirements, keeping everything compliant and under control. -To learn more, check out aka.ms/rayfin and keep watching Microsoft Mechanics for the latest tech updates and thanks for watching.Streaming and Batch Data Architectures with Microsoft Fabric to Azure Databricks
Author's: Aladdin Alchalabi AladdinAlchalabi, Oscar Alvarado oscaralvarado and Rafia Aqil Rafia_Aqil Note: This article describes a solution idea. Your cloud architect can use this guidance to help visualize the major components for a typical implementation. Use this article as a starting point to design a well-architected solution that aligns with your workload’s specific requirements. As organizations adopt Microsoft Fabric as their unified analytics platform, it has become a leading path for ingesting both streaming and batch data into Azure Databricks. This article covers integration approaches -via Microsoft Fabric- and details the five Fabric-specific paths that connect OneLake/ADLS and Databricks for end-to-end data processing. Medallion Architecture The following data flow corresponds to the architecture diagram: Data is ingested through Microsoft Fabric (via Mirroring, RTI, or Data Factory) lands data into OneLake/ADLS. With the medallion pattern, consisting of Bronze, Silver, and Gold storage layers, organizations have flexible access and extendable data processing: Bronze – Raw data entry point. Data arrives in its source format and is converted to the open, transactional Delta Lake format. Silver – Optimized for BI and data science. ETL and stream processing tasks filter, clean, transform, join, and aggregate Bronze data into curated datasets using SQL, Python, R, or Scala. Gold – Enriched data ready for analytics and reporting. Analysts use Power BI, PySpark, SQL, or Excel for insights and queries. Fabric Integration Paths Note: This architecture establishes a complete loop-back between Microsoft Fabric and Azure Databricks, enabling Gold layer tables to be seamlessly mirrored back to Microsoft Fabric for dashboarding through Azure Databricks Mirroring. The following five paths connect Microsoft Fabric to Azure Databricks: Fabric Mirroring to OneLake – A low-cost, low-latency turnkey solution that creates a replica of data from operational sources (SQL Server, Azure Cosmos DB, Oracle) in OneLake. Handles the initial load and ongoing CDC changes automatically, keeping data continuously up to date. Fabric RTI to OneLake – Fabric Real-Time Intelligence ingests streaming event data into OneLake with sub-second latency, enabling real-time analytics on live event streams. Fabric Data Factory to OneLake – Orchestrates ingestion from diverse sources not covered by Mirroring (such as Sybase or REST APIs) and lands data in OneLake, ensuring complete source coverage. OneLake to Azure Databricks – Unity Catalog connections to OneLake, secured via Managed Identities from Microsoft Entra ID, allow Databricks to query OneLake data items as a native catalog without data duplication. Fabric Data Factory to Azure Databricks (direct) – Orchestrates ingestion from diverse sources directly into Azure Data Lake Storage (ADLS), where Azure Databricks picks up the data for medallion architecture processing. Design Considerations Area Updated guidance Direct RTI-to-Databricks integration There is still no broad GA direct integration where Fabric RTI and Databricks operate as one native real-time runtime. Integration should be positioned through open protocols, Event Hubs/Kafka-style patterns, OneLake, Delta, and federation. OneLake federation in Azure Databricks OneLake federation in Azure Databricks is now the key integration story. It allows Databricks Unity Catalog to query Fabric Lakehouse and Warehouse data in OneLake without copying it. Access is read-only and depends on Fabric tenant settings, workspace permissions, and Databricks Unity Catalog setup. RTI data availability to Databricks Data ingested through Fabric RTI can be made available to Databricks by landing or exposing the data into OneLake-backed items, especially Lakehouse/Warehouse patterns. Eventhouse data can be made available in OneLake in Delta format through OneLake availability, but Databricks OneLake federation should be validated against the specific Fabric item type and access path. Existing Databricks customers Existing Databricks customers do not need to abandon Databricks. They can use Fabric RTI as the event ingestion, real-time detection, operational alerting, and business action layer, while continuing to use Databricks for engineering, ML, advanced analytics, and Unity Catalog-governed access. Activator and business action Fabric Activator is the cleanest business-user action layer. It can monitor streaming events and trigger Teams messages, email, Power Automate flows, Fabric pipelines, notebooks, Spark jobs, Dataflows, UDFs, and other downstream actions. This is a strong differentiator because it lets business users act on events without waiting for batch analytics. Operations Agents Operations Agents are in preview and should be positioned carefully. They monitor real-time data from Eventhouse or ontology sources, surface insights, recommend actions, and can connect to Activator/Power Automate action paths. They are not simply a pre-ingestion decision engine before data lands anywhere; they work from configured Fabric knowledge/data sources. Before landing in Lakehouse For decisioning before Lakehouse persistence, use Eventstream processing and Activator rules on streams. For AI-assisted operational recommendations, use Operations Agents once the relevant data is available in Eventhouse or ontology. Requirement-Specific Notes Data Ingestion Microsoft Fabric Mirroring currently supports SQL Server, Azure Cosmos DB, and Oracle as source systems. For sources not yet supported by Mirroring—such as Sybase or REST APIs—use Fabric Data Factory pipelines to ensure full coverage across all data systems. Once data is in the landing zone with the correct format, Mirroring’s CDC replication starts automatically and manages the complexity of merging changes (updates, inserts, and deletes) into Delta tables, keeping data in Fabric continuously up to date. Learn more about open mirroring Storage Format and Time Travel OneLake supports Delta tables, enabling schema evolution and time travel across all data stored in the lakehouse. Learn more about OneLake and Delta tables Security Encryption at rest: OneLake automatically encrypts all data at rest using Microsoft-managed keys, compliant with FIPS 140-2 standards. Learn more Encryption in transit: All data in transit is encrypted using TLS 1.2 or higher, securing data movement between Fabric, OneLake, and Azure Databricks. Learn more Data Governance OneLake can be registered and scanned by Microsoft Purview, enabling cataloging of stored metadata and data quality profiling. This protects sensitive information, including PHI and PII, across ingestion and analytics workflows. Learn more about Purview with Fabric Lakehouse Operations and Monitoring Use the Fabric monitor hub to track pipeline health, Spark application performance, and ingestion job status across all Fabric workloads. Learn more about the Fabric monitor hub Scenario Details This architecture applies to any organization that needs to unify streaming and batch data at scale. Common characteristics include: Multiple operational data sources (databases, SaaS applications, event streams) A requirement to process both real-time and historical data in the same platform Governance and compliance requirements for sensitive data (PHI, PII, financial records) Analytics consumers spanning BI (Power BI), data science (Databricks notebooks), and ML workloads Potential Use Cases Healthcare and life sciences – PHI/PII protection via Purview; real-time patient telemetry + batch EHR analytics Financial services – Real-time fraud detection streams + batch regulatory reporting Retail and e-commerce – Streaming clickstream analytics + batch inventory and supply chain processing Energy and utilities – IoT sensor telemetry streaming + batch consumption analytics Next Steps Get started with Microsoft Fabric Mirroring Build an ETL pipeline with Lakeflow Declarative Pipelines Configure Unity Catalog with OneLake shortcuts Monitor Fabric pipelines with the Fabric monitor hubOperations Context for AI | Ontology in Fabric IQ
Generate a full business ontology from an existing Power BI semantic model, map entities and relationships, and embed real-time operational signals alongside business rules that live inside the ontology with the data and its meaning. From there, trace cascading operational impacts across your business through the relationship graph, stand up Operations Agents in natural language with Teams-based actions, and connect the same ontology as a knowledge source in Copilot Studio or Azure AI Foundry. Chafia Aouissi, Fabric IQ Principal PM Manager, shares how to model your business operations, embed intelligence in your data, and deploy agents that act on it. Logic lives with your data. Embed business rules directly in the Fabric IQ ontology. Define thresholds, trigger notifications, and cascade decisions through connected entities. Watch the demo. Surface cascading impacts with a single query. Filter the Fabric IQ relationship graph by any entity and trace downstream operational impact across your entire business. Check out a built-in ontology graph in Fabric IQ. Build a Fabric IQ Operations agent in natural language. Define monitoring goals, connect your ontology as its knowledge base, configure Teams actions, & deploy. See the full build. QUICK LINKS: 00:00 — Unify models & data with Fabric IQ 01:12 — Generate an ontology 02:27 — Bring in Power BI reports 03:08 — View across multiple data sources 04:23 — Define rules 05:18 — Built-in ontology graph 06:03 — Fabric IQ agents 08:24 — Fabric IQ as Knowledge Source 08:54 — Wrap up Link References Get started at https://aka.ms/FabricIQ Unfamiliar with Microsoft Mechanics? As Microsoft’s official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft. Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries Talk with other IT Pros, join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog Watch or listen from anywhere, subscribe to our podcast: https://microsoftmechanics.libsyn.com/podcast Keep getting this insider knowledge, join us on social: Follow us on Twitter: https://twitter.com/MSFTMechanics Share knowledge on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/ Enjoy us on Instagram: https://www.instagram.com/msftmechanics/ Loosen up with us on TikTok: https://www.tiktok.com/@msftmechanics Video Transcript: -The agents you build and use need the right operational context of how your business runs to deliver the best outcomes. Today, that context is often fragmented across systems, defined differently by different teams, or buried in dashboards and logic, making outcomes inconsistent and agent behavior hard to predict. That’s where Microsoft Fabric IQ comes in. Fabric IQ introduces a semantic foundation that unifies models and data through an ontology. It defines the shared business entities and their relationships, and connects them to your data. It provides the operational context needed to understand how the business actually runs, without altering any underlying data. Analysts can not only work with the data they already trust, but also model how the business works. And agents can use that same shared context to reason and act more consistently. Today, I’ll show you both sides. -First, how a data analyst leverages Fabric IQ inside a Fabric workspace, using ontology to model the business concepts. Then, how Fabric IQ uses the same context to drive more reliable and predictable insights from agents. I’ll start from the point of view of an analyst looking to create a full fidelity view of how an airline operates, including processes such as ticketing, maintenance, and more. The first thing I need to do is to create a new ontology. I can either build one from scratch, or jumpstart by using an existing Power BI semantic model. As you see here, there’s a new option to generate an ontology from this semantic model. I just need to choose the workspace, give it a name. I’ll choose AirlineOperationsOntology. Then confirm by hitting Create. In just a few clicks, I’m able to see the different entities of our airline business. All data is now linked not only through keys, but also business relationships and semantics. We can see flights, airlines, routes, and more. If I click into airports, because Fabric IQ is semantically aware of the relationships between entities, it shows the routes connected to runways which are in turn connected to airports. -And for any entity, I can choose to add more live operational signals. In this case, I want to add details about the runway conditions using real-time data, including contamination, visual range for visibility, as well as the available cleared width and more. And you can also bring in your Power BI reports for a canonical view of how to monitor and manage these aircrafts. From the ontology, I’ll head over to the Report links tab that opens the OneLake catalog with all of my reports. I’ll search for air and there are three matching reports for gates, ground service, plus safety and runway. So I’ll add them and hit Connect to confirm. -So, in just a few clicks, we’ve expanded our ontology with the live operational view, using real-time signal, geospatial data, and more. Now, as an analyst, I can work immediately with it, and our agents can act on it as well. Let’s fast‑forward and see what I’ve unlocked. You can see that I now have a richer view over my data, which is connected to real‑world operations. We’re no longer optimizing one report or one dataset at a time. We’re looking at our operations across multiple data sources, in the language of our business, with meaning and relationships already understood. -Now let me show you how this makes it easier to turn insights into concrete decisions and actions. First, in the flight entity type overview, I can see how it relates to my other business processes. I see entities like bookings, gates, airlines, and more as a graph. I have links to all of my connected Power BI reports. There is a real-time weather data, including wind knots, as well as geo-spatial insights showing all of my flights. Using Fabric Maps, I have a fleet level view of all my active flights, and I can see live air traffic across the fleet. This, in fact, is a heat map view of three New York City area airports, and we can see that JFK in this case is impacted with lots of runway activity. -I can now understand the system as a whole, across bookings, flights, airports, and real‑time conditions. And I can drill in further to understand what is going on: I have opened the runways entity, and you’ll remember some of these categories from before. Since there’s snow in the area, I can immediately see the runway conditions that affect operations, things like surface friction and contamination levels, so I understand how safe it is for planes to take off and land. -Beyond connecting raw data, I can also define rules directly in the ontology, so this logic lives with the data and its business meaning, instead of being hard coded somewhere else. In this case, I’ll add a rule that says if runway contamination exceeds high threshold value of 25%, notify the passengers proactively of upcoming delays. We’ll also notify the ground crew, so they know that the runways need to be cleared. The rules are now embedded in the ontology, and the value comes from seeing how runway conditions impact the rest of the operations. That’s where the built‑in ontology graph helps. Let’s look at the relationship graph. I’ll expand the graph view. And add a filter for JFK airport Then run the query. No code needed here. And I get a filtered view for JFK. And immediately, I can see a poor condition that’s affecting Runway 25R. -From that insight, it’s easy to see the downstream impact. This runway issue is already affecting related gates and baggage operations that will need to be rescheduled. This is a unified view of our entire operations and how connected events will cascade across related business entities. This is how ontology helps you as an analyst. But remember, the same operational context is also available to AI agents, no matter how you build them. -Let me demonstrate this in the context of one our built-in Fabric IQ agents. From the New Item catalog, you can find the built-in agents by searching for agent. There is a Data agent designed to answer questions, and an Operations agent designed for real-time data and business action recommendations. The Operations agent is a perfect fit for our airline operations scenario, so I’ll choose that one. I want this agent to help with runway-related analysis and actions, so I’ll name it RunwayConditionsAgent, leave the location, and create it. -From there, I can add a bit more information to set up the agent, like adding the business goals for what it should accomplish I want this one to monitor surface conditions for runways and ensure things run smoothly based on logic like we used before. In fact, in the Agent instructions, using natural language, no code, I’ll describe that if surface contamination is reported above 10%, send ground crews to take care of it. Likewise, the clear width should be more than 25 meters, and the agent should send ground crew to visually assess whether planes can safely brake. -Now let’s add some knowledge. And for that, I’ll choose our AirlineOntology. And here’s where I can add actions. I’ll add one to assign ground crew for clearing, along with description for what needs to be done. Then I’ll give it the Runway ID as the one to clear and the Temperature to predict the type of clearing needed. And Create to add that one. Now I’ll add another for requesting visual assessment, and perform similar steps for the parameters. These will send status updates in Microsoft Teams. Now everything is defined and ready. I just need to save this new agent. That takes a moment. And once it’s finished, it creates a nice agent playbook with what it’s designed to do. -Now, with the agent running, the right people will get notified of what to do in Microsoft Teams Here, I’m looking at the Operations agent It’s alerting us that Runway 29L has only 22 meters of clear path. This is under our 25 meter threshold. It recommends to deploy the ground crew for a runway clearance operation. As the human in the loop, I can choose whether or not to proceed with the recommendation. I’ll do that. -Then it asks to confirm a few details. They look good, so I’ll confirm, and the ground crew is on its way. And here is the good news. If you’re building your own agent in Microsoft Copilot Studio or using Microsoft Foundry, Fabric IQ ontology will be an integrated knowledge source that you will be able to choose from. As you choose your knowledge types, you can select Fabric IQ. This will ground agents in the same semantic foundation that already runs your operations. And of course, the agents you build and connect to Fabric IQ will respect the permissions and security policies you already use in Fabric today. -As I have shown, Microsoft Fabric IQ gives agents shared understanding, with entities, relationships, rules, and actions so they can move from insight to decision more reliably. To learn more and get started, check out aka.ms/FabricIQ. Keep watching Microsoft Mechanics for the latest news and deep dives. And thank you for watching.General Availability Refresh: Mirroring Azure Database for PostgreSQL in Microsoft Fabric
Introduction We are excited to announce the General Availability Refresh of Mirroring Azure Database for PostgreSQL in Microsoft Fabric. This update introduces several new capabilities designed to reduce friction, improve transparency, and increase trust in PostgreSQL data for analytics and AI workloads. Database professionals, DBAs, and developers can now leverage a more robust, flexible, and transparent solution for integrating PostgreSQL data in Microsoft Fabric. Mirroring Azure Database for PostgreSQL enables seamless integration of transactional data into Microsoft Fabric, supporting advanced analytics and AI scenarios. Previously, users faced limitations in data type support, operational requirements, and troubleshooting transparency. The General Availability Refresh directly addresses these challenges, delivering a more reliable and user-friendly experience. Native Data Type Support One of the most significant enhancements is support for PostgreSQL native data types, including JSON and JSONB. Users can now mirror complex, semi-structured data without conversion, preserving the full fidelity of source data. Additionally, transparent replication to varchar(max) or varbinary is now supported, ensuring that even custom or less common types can be accurately mirrored in Fabric. Previously, data type limitations often required workarounds or manual transformations, introducing complexity and risk. With this update, data flows more naturally from PostgreSQL to Fabric, streamlining analytics and AI pipelines and reducing the time spent on data preparation. Mirroring now preserves PostgreSQL-native types—including JSON and JSONB—without requiring schema transformations or type coercion. This ensures: Full fidelity replication of semi-structured data Elimination of intermediate serialization (e.g., string encoding) Tables containing JSON/JSONB columns are mirrored into Fabric with their structure intact, enabling: Direct querying of nested JSON fields Consistent schema representation across source and analytical layers Once mirrored, JSON content can be queried natively within Fabric workloads, allowing: Hybrid analytical scenarios (relational + semi-structured) Use of familiar SQL patterns for JSON traversal and extraction Flexible Server High Availability Support for PG versions earlier than 17 The refresh expands support for Azure Database for PostgreSQL Flexible Server with high availability enabled on versions earlier than 17. This means organizations can leverage mirroring with HA configurations without needing to upgrade their database engine, improving operational flexibility and minimizing disruption. In the past, mirroring required specific PostgreSQL versions, limiting adoption for organizations with established HA deployments. Now, the broader compatibility allows teams to maintain their preferred configurations and benefit from seamless data integration in Fabric. Improved Transparency: Enhanced Error Messaging and Dedicated PostgreSQL UDFs Transparency is critical for troubleshooting and maintaining trust in data pipelines. The General Availability Refresh introduces enhanced error messaging, providing clear and actionable insights into replication issues. Dedicated PostgreSQL user-defined functions (UDFs) further support monitoring and diagnostics, enabling users to quickly identify and resolve problems. --Validates that all system and configuration requirements are met before starting CDC mirroring. SELECT * FROM azure_cdc.check_prerequisites(); -- Quickly verify which extension version is deployed — critical for troubleshooting SELECT azure_cdc.azure_cdc_version(); -- Returns a detailed list of errors and issues detected during CDC operations SELECT * FROM azure_cdc.get_health_status('', ''); -- Scans every eligible user table in the database and returns the mirroring readiness status for each SELECT * FROM azure_cdc.get_all_tables_mirror_status(); These features streamline troubleshooting, reduce downtime, and empower teams to maintain robust mirroring operations. The improvements are a substantial step forward from previous releases, which offered limited visibility into replication health and errors. Unblocking Mirroring for servers with Read Replica enables We also removed existing block for creating Mirrored databases on Flexible Servers with Read Replicas created. Now primary servers with one or more Read Replicas can serve as source for Mirroring their databases in Fabric with no limitations. Conclusion The new capabilities significantly reduce friction for analytics and AI workloads in Microsoft Fabric. By removing technical barriers and increasing operational simplicity, the General Availability Refresh empowers organizations to unlock the full potential of their PostgreSQL data for advanced analytics and AI applications. The General Availability Refresh of Mirroring Azure Database for PostgreSQL in Microsoft Fabric delivers critical enhancements: native data type support, expanded Flexible Server HA compatibility, replication identity improvements, and better transparency. Together, these features make mirroring more robust, flexible, and trustworthy for analytics and AI workloads. We encourage technical professionals, DBAs, and developers to adopt these new capabilities and explore how they can transform data integration in Microsoft Fabric. For more information, visit the official documentation and resources: Azure Database for PostgreSQL Documentation Microsoft Fabric DocumentationStep by Step Guide to Ontology and Plan for Financial Service
What We Will Build In this guide, we will construct a complete Fabric IQ solution that accomplishes the following: First, a Lakehouse that ingests publicly available data including bank financials, P2P lending statistics, borrower demographics, and licensing information. Second, a Semantic Model that defines the analytical layer with proper dimensions, measures, and relationships. Third, an Ontology that elevates these tables into business entities such as Bank, P2P Platform, Borrower, and Loan, connected by meaningful relationships and governed by regulatory rules. Fourth, a Planning sheet that enables supervisors to forecast enforcement workloads, allocate examination budgets, and model scenarios based on live data. Step 1: Preparing the Data Foundation in Fabric Lakehouse Every Fabric IQ solution begins with data. Before we can model business semantics or build planning sheets, we need a well structured Lakehouse that holds our source data in a governed and queryable format. Creating the Lakehouse Navigate to your Fabric workspace and create a new Lakehouse. In this example, we have named it P2PLendingLH, housed within the workspace P2P Lending CrossSector Demo. The Lakehouse serves as the Bronze and Silver layer of our medallion architecture, storing both raw ingested data and transformed analytical tables. Data Sources and Tables The Lakehouse is populated with data from publicly available publications. The table structure follows a dimensional modeling pattern with clear separation between dimension tables (prefixed with dim_) and relationship tables (prefixed with rel_). The following tables form the foundation of our model: Table Name Description dim_bank Bank profiles including KBMI tier, total assets, CAR, NPL, channeling exposure percentage dim_borrower Borrower demographics with credit score, employment type, province, and risk segment dim_p2p_platform Licensed P2P lending operators with TWP90 rate, outstanding balance, and total borrowers dim_loan Individual loan records with amount, tenure, interest rate, and repayment status dim_supervisor_team supervisory teams and their regional assignments dim_channeling_agreement Bank to P2P channeling contracts and exposure limits In addition to dimension tables, several relationship tables capture the connections between entities. These include rel_bank_channels_platform (which bank funds which P2P platform), rel_borrower_takes_loan (linking borrowers to their loans), rel_loan_funded_by_bank (tracing the funding chain), rel_platform_issues_loan (connecting platforms to the loans they originate), and rel_supervisor_oversees_platform and rel_supervisor_oversees_bank (mapping supervisory responsibility). Step 2: Creating the Semantic Model With data in the Lakehouse, the next step is to create a Semantic Model that defines the analytical interface. The Semantic Model is a Power BI construct that organizes your tables into a star schema with proper relationships, hierarchies, and measures. More importantly for our purpose, this Semantic Model will later serve as the blueprint from which we generate our Ontology. Generating the Model from Lakehouse From within the Lakehouse, click on "New semantic model" in the toolbar. A dialog appears allowing you to name your model and select which tables to include. In our case, we select all dimension and relationship tables to ensure the Ontology will have full visibility into the data landscape. Figure 1. Creating a new Direct Lake semantic model from the P2PLendingLH Lakehouse, selecting dimension and relationship tables for inclusion. Notice that the dialog shows the workspace name (P2P Lending CrossSector Demo) and provides a searchable list of all available tables. The Direct Lake mode is automatically selected, which means the Semantic Model will query data directly from the Lakehouse parquet files without importing a copy. This is important for our use case because it ensures that when regulator publishes updated monthly statistics and the Lakehouse is refreshed, the Semantic Model and subsequently the Ontology will reflect the latest data. Configuring Relationships and Properties After creation, the Semantic Model opens in the editing view where you can configure relationships, add calculated measures, and define display properties. The model view shows the entity cards with their fields and the lines connecting related tables. Figure 2. The Semantic Model editor showing entity cards for dim_bank and dim_borrower, with relationship lines and the full table listing in the Data panel. In the screenshot above, you can see two of the core dimension tables. The dim_bank table contains fields such as bank_id, bank_type, channeling_exposure_pct, channeling_total, name, regulator_team, and total_assets. The dim_borrower table holds borrower_id, credit_score, employment_type, name, province, and risk_segment. The Data panel on the right reveals the complete set of tables available in this model, including all the relationship tables that define the connections between entities. At this stage, you should verify that all necessary relationships are correctly established. For example, dim_bank should connect to rel_bank_channels_platform through bank_id, and dim_p2p_platform should connect to rel_platform_issues_loan through platform_id. These relationships are what enable the Ontology to reason across domains in the next step. You may also want to add calculated measures at this point, such as a weighted average TWP90 across all platforms funded by a specific bank, or a total channeling exposure as a percentage of the bank's total assets. These measures will be carried forward into the Ontology and can be used by AI agents for natural language querying. Step 3: Generating the Ontology This is the step where the magic of Fabric IQ truly comes alive. The Ontology transforms your Semantic Model from a reporting layer into an intelligence layer. While the Semantic Model answers the question "what does the data look like," the Ontology answers the question "what does the data mean." What the Ontology Does An Ontology in Fabric IQ is a machine understandable vocabulary of your business. It consists of entity types (the things in your environment, such as Bank, Borrower, or P2P Platform), properties (the facts about those entities, such as a bank's NPL ratio or a platform's TWP90 rate), and relationships (the ways entities connect, such as a Bank channels funding to a P2P Platform). Beyond static modeling, the Ontology also supports rules and constraints that can trigger automated actions when business conditions are met. Generating from the Semantic Model To create the Ontology, open your Semantic Model and look for the "Generate Ontology" button in the toolbar. Clicking it opens the generation dialog, which presents three key value propositions: Unify models into a semantic layer allows you to align concepts across domains and modeling paradigms, bringing banking data and P2P lending data into a shared vocabulary. Model expressively enables you to capture complex relationships, domain specific rules, and actions that drive business workflows, such as triggering an alert when a P2P platform's TWP90 crosses the 5 percent regulatory threshold. Reason over events and temporal patterns means that the Ontology can use sequences and trends to inform decisions and automation, such as detecting three consecutive months of TWP90 deterioration. Figure 3. The Ontology generation dialog, creating a new Ontology named NewP2P from the existing Semantic Model within the P2P Lending CrossSector Demo workspace. In the dialog, you specify the workspace (P2P Lending CrossSector Demo) and give your Ontology a name (in this example, NewP2P). After clicking Create, Fabric IQ analyzes the Semantic Model's structure, identifies entity types from dimension tables, infers relationships from the foreign key connections, and generates a navigable graph that represents your business domain. Enriching the Ontology with Rules Once the Ontology is generated, you can enrich it with business rules that reflect regulatory requirements. For the P2P lending use case, the following rules are particularly relevant: Rule Name Condition Action Elevated TWP90 P2P Platform TWP90 exceeds 5 percent Flag platform as high risk and alert PVML supervisor Contagion Risk Bank channeling exposure to flagged P2P platform exceeds 10 percent of portfolio Alert Banking supervisor and recommend joint examination Youth Overleveraged Borrowers aged 19 to 34 represent more than 60 percent of a platform's portfolio AND TWP90 is above average Trigger consumer protection review and education program allocation CAR Threshold Bank CAR drops below 10 percent while having active P2P channeling agreements Escalate to Kepala Eksekutif Pengawas Perbankan These rules integrate with Fabric Activator, enabling the Ontology to automatically initiate business processes through alerts and automated actions. This means that when new monthly P2P statistics are ingested and a platform's TWP90 crosses the threshold, the system does not wait for an analyst to discover it manually. The rule fires, the alert is sent, and the supervisory workflow begins. Querying with Natural Language One of the most powerful capabilities enabled by the Ontology is the ability to query across domains using natural language through a Data Agent. Because the Ontology defines the business vocabulary and binds it to real data, a supervisor can ask questions like: "Which banks have channeling agreements with P2P platforms whose TWP90 is currently above 5 percent, and what is their total exposure?" The Data Agent resolves this query by traversing the Ontology graph: from the Bank entity through the channels_funding_to relationship to P2P Platform, filtering by the TWP90 property, and aggregating the channeling_total measure. Step 4: Setting Up Planning Sheets While the Ontology tells you what is happening in your business right now, the Plan item in Fabric IQ helps you decide what should happen next. Planning in Fabric IQ brings budgeting, forecasting, and scenario modeling directly into the same environment where your data lives, eliminating the disconnect between analytical insights and forward looking decisions. Creating a Planning Sheet To create a Plan, navigate to your workspace and select New Item followed by Plan (preview). After naming the plan and connecting it to your Semantic Model, you can begin building Planning sheets that pull dimensions and measures directly from the same data that powers your Ontology. In the screenshot below, we see a Planning sheet named "Planning P2P" that presents a tabular view of all P2P lending platforms alongside their key risk metrics. Figure 4. The Planning sheet showing P2P lending platforms with their TWP90 rates, total outstanding balances (in trillions of Rupiah), total borrower counts (in thousands), and risk categories. The Planning sheet is structured with the platform name and risk_category as row dimensions, and three critical measures as values: Sum of twp90_rate, Sum of total_outstanding (displayed in trillions of Rupiah), and Sum of total_borrowers (displayed in thousands). The risk_category column provides an immediate visual classification of each platform's health status, with categories such as Elevated and Very High clearly indicating where supervisory attention should be directed. Looking at the data, several insights emerge immediately. DanaBijak and DanaCepat both carry a Very High risk category, with TWP90 rates of 18.77 and 17.79 respectively. CashWagon ID shows an Elevated risk designation despite a comparatively modest TWP90 of 8.26, likely due to its substantial outstanding balance of 144.97 thousand borrowers. The aggregate row at the top reveals the industry total: a combined TWP90 of 365.38 (this is a sum across all platforms), total outstanding of 29.86 trillion Rupiah, and 7,281.55 thousand borrowers across the monitored universe. Using Planning for Supervisory Resource Allocation The real power of the Planning sheet becomes apparent when supervisors begin using it for forward looking decisions. Consider the following scenarios that can be modeled directly within the Planning interface: Enforcement Forecasting: Based on the current data showing multiple platforms in the Very High risk category, supervisors can forecast the expected volume of warning letters and administrative sanctions for the coming quarter. If historical patterns show that each Very High platform typically receives two to three rounds of correspondence before resolution, the planning sheet can project staffing requirements for the enforcement team. Budget Allocation: The Planning sheet can incorporate budget dimensions alongside risk metrics. If the current quarterly examination budget allows for on site visits to 15 platforms, the risk category column helps prioritize which platforms should be visited first. The forecast capability can then project whether the budget is sufficient given the current risk trajectory, or whether a reallocation request should be submitted.
