Purview Data Quality with Microsoft Fabric
Hi, Our technical data quality checks are written in SQL inside the Fabric Warehouse. When a check fails, the ETL pipeline stops. Can business data quality rules defined in Purview Data Quality work the same way? Can they run inside the Warehouse and stop the ETL pipeline when they fail? Could someone also explain how to estimate the cost of running these rules in Purview Data Quality? How exactly is it billed: per compute unit, per compute time, or per number of rows processed? Thanks!
Struggling with running DQ Scans (Long queuing and Retry Count Error Issues)
Hi everyone, I have been exploring Microsoft Purview Data Quality quite extensively. At this point, I have configured more than 4,000 data quality rules across more than 10 Microsoft Fabric capacities, each with a minimum capacity of F16. Fabric is the source for all assets registered in Purview. I have identified several issues with the product, but the two that are currently impacting me the most are the following: DQ scans failing with a generic error“Max Retry Count Reached. Ending Workflow. Current Task HandleError”The challenge is that the error message does not identify which rule is causing the failure. As a result, I have to troubleshoot manually by disabling groups of rules, rerunning the scans, and repeating the process until I find the problematic rule. This trial-and-error approach is very time-consuming, especially at this scale. This seems to be caused by issues in some of the DQ rules, even though all rules are marked as “Good to go” in Purview. When running Data Quality scans, I often receive the following error: DQ scans remain queued for a long timeI am not sure why this happens or what resource, orchestration, or scheduling constraint is causing the delay. Whenever I run these DQ scans, they remain in a Queued state for at least 10 minutes, even when there is nothing running on the Fabric capacities. Has anyone experienced similar behavior with Purview Data Quality at this scale? Specifically, I would appreciate any guidance on: How to identify which DQ rule is causing a scan failure Why scans remain queued even when Fabric capacity appears to be idle Whether there are known limitations or best practices for running thousands of DQ rules in Purview Thank you.Should CRM Users Be Measured on Data Quality KPIs?
Most Organisations agree that high-quality data is essential for getting value from Dynamics 365. Accurate customer information supports better reporting, improved customer experiences, more reliable forecasting, and increasingly more effective AI-driven insights. Yet many Organisations continue to struggle with incomplete records, duplicate data, missing activities, and inconsistent data entry practices. This raises an interesting question: Should CRM users be measured on data quality KPIs? Consider a situation many Organisations have experienced. A sales team is expected to maintain customer records, update opportunities, and log key customer interactions in Dynamics 365. However, users are primarily measured on revenue, pipeline growth, and sales performance. As a result, CRM updates are often treated as a secondary task. During a quarterly sales review, leadership discovers that several opportunities forecasted as active were closed weeks earlier, while others had not been updated since the previous reporting cycle. Customer records are missing key information, activities have not been logged consistently, and reporting accuracy begins to suffer. The issue is often viewed as a reporting problem, but in reality, it starts with the quality and consistency of the data being maintained in Dynamics 365. To address these challenges, some Organisations introduce data quality metrics such as: Record completeness Duplicate record reduction Activity logging compliance Opportunity update accuracy Customer data validation rates Supporters argue that what gets measured gets managed, and that data quality should be considered part of everyone's responsibility. Others believe that introducing data quality KPIs may create an additional administrative burden, reduce user adoption, and shift focus away from core business objectives. There is also the question of whether users should carry the full responsibility. Modern Dynamics 365 environments include validation rules, duplicate detection, business process flows, Power Automate workflows, and governance frameworks that can help improve data quality. Some Organisations, therefore, argue that technology and governance should do more of the heavy lifting rather than relying solely on user behaviour. From your experience: Should CRM users be measured on data quality KPIs? Have data quality metrics improved CRM adoption or data accuracy in your Organisation? What KPIs have been most effective? Is data quality primarily a user responsibility, or should technology and governance frameworks carry most of the burden? Have you found a balance that improves data quality without creating additional friction for users? I'm interested in hearing how different Organisations balance user accountability, adoption, and data quality within Dynamics 365 environments.Who Should Be Accountable for Data Quality in Dynamics 365: IT or the Business?
Data quality remains one of the most common challenges in Dynamics 365 environments, regardless of industry or organisation size. When customer records are incomplete, duplicate data exists, or reporting becomes unreliable, the conversation often turns to ownership and accountability. Consider a simple example: A sales team creates customer records in Dynamics 365, while customer service updates contact details and finance systems synchronize billing information through integrations. Over time, duplicate records appear, customer information becomes inconsistent, and management reports start showing conflicting results. When this happens, who is accountable? Are the business users entering the data? Is the IT team managing the platform? The integration owners? Or should there be dedicated data stewards responsible for maintaining data quality standards? Some argue that data quality is primarily a business responsibility because users create and maintain most of the information stored in Dynamics 365. Others believe IT teams should take greater ownership through governance frameworks, validation rules, integrations, monitoring, and automated controls. In practice, many organisations struggle to find the right balance. When data issues arise, responsibility can become unclear, making it difficult to drive long-term improvements. From your experience: Who should ultimately be accountable for data quality in Dynamics 365? Should ownership sit with business teams, IT, dedicated data stewards, or a shared governance model? What approaches have worked well in your organisation? Have you seen a particular governance model deliver better results? I'm interested in hearing different perspectives and learning how others are addressing this challenge.
Data Quality Error (Internal Service Error)
I am facing an issue while running the DQ scan, when i tried doing manual scan and scheduled scans both time i faced Internal Service Error. ( DataQualityInternalError Internal service error occurred .Please retry or contact Microsoft support ) Data Profiling is running successfully but for none of the asset, DQ is working. After the lineage patch which MS had fixed, they had introduced Custom SQL option to create a rule, and after that only i am facing this issue. Is anyone else also facing the same? I tried with different data sources (ADLS, and Synapse) its same for both. If anyone has an idea, do share it here, it will be helpful.
Purview Data Quality Dashboard/ Report - Refresh
Hi All, Currently I am getting all blank in Purview Data quality dashboard, before two months dashboard shows all values across each data quality dimensions and showed graph for each quadrant in a dashboard. After two months when checked the dashboard everything is blank nothing is shown in the report. (Note : I have created two governance domain and each domain has five data products assigned with data assets, implemented data quality rules on top of each data assets that time scores were reflected in the Purview data quality dashboard), but suddenly now it all went blank scores showing as 'blank' Note : None of the data quality assessment were not deleted during that two months, data quality rules are still active and its still showing scores at data asset level. But its not showing in the dashboard currently. Can you please help me to sort out, is there any refresh policy associated for Purview Data quality dashboard.
Microsoft Purview Data Quality Report "Not-Available Data"
Hello! I'm working with data quality of Microsoft Purview. I have created a "Test Domain" and have data quality on its data assets. I have decided to delete the quality data as well as deleting the domain. It's been a week and the DQ Report is still like this. I think this comes from the deleted Test Domain. It affects the score in the report. Is there any way to remove the Not-Available Data? Thanks in advance!
Microsoft Purview - Establishing Data Quality Connection on Azure SQL Database
I have a service principal and already using it on the Data Map Solutions to scan and register Azure SQL Database as the source. It worked. Now, I am in the Unified Catalog and I am trying to establish a data quality connection on Microsoft purview for Azure SQL Database as its source type to run data quality scans. Why can't I use the Service Principal for the credential? Is there another way to establish the connection?
