Struggling with running DQ Scans (Long queuing and Retry Count Error Issues)
Hi everyone, I have been exploring Microsoft Purview Data Quality quite extensively. At this point, I have configured more than 4,000 data quality rules across more than 10 Microsoft Fabric capacities, each with a minimum capacity of F16. Fabric is the source for all assets registered in Purview. I have identified several issues with the product, but the two that are currently impacting me the most are the following: DQ scans failing with a generic error“Max Retry Count Reached. Ending Workflow. Current Task HandleError”The challenge is that the error message does not identify which rule is causing the failure. As a result, I have to troubleshoot manually by disabling groups of rules, rerunning the scans, and repeating the process until I find the problematic rule. This trial-and-error approach is very time-consuming, especially at this scale. This seems to be caused by issues in some of the DQ rules, even though all rules are marked as “Good to go” in Purview. When running Data Quality scans, I often receive the following error: DQ scans remain queued for a long timeI am not sure why this happens or what resource, orchestration, or scheduling constraint is causing the delay. Whenever I run these DQ scans, they remain in a Queued state for at least 10 minutes, even when there is nothing running on the Fabric capacities. Has anyone experienced similar behavior with Purview Data Quality at this scale? Specifically, I would appreciate any guidance on: How to identify which DQ rule is causing a scan failure Why scans remain queued even when Fabric capacity appears to be idle Whether there are known limitations or best practices for running thousands of DQ rules in Purview Thank you.
Data Quality Error (Internal Service Error)
I am facing an issue while running the DQ scan, when i tried doing manual scan and scheduled scans both time i faced Internal Service Error. ( DataQualityInternalError Internal service error occurred .Please retry or contact Microsoft support ) Data Profiling is running successfully but for none of the asset, DQ is working. After the lineage patch which MS had fixed, they had introduced Custom SQL option to create a rule, and after that only i am facing this issue. Is anyone else also facing the same? I tried with different data sources (ADLS, and Synapse) its same for both. If anyone has an idea, do share it here, it will be helpful.
Purview Data Quality Dashboard/ Report - Refresh
Hi All, Currently I am getting all blank in Purview Data quality dashboard, before two months dashboard shows all values across each data quality dimensions and showed graph for each quadrant in a dashboard. After two months when checked the dashboard everything is blank nothing is shown in the report. (Note : I have created two governance domain and each domain has five data products assigned with data assets, implemented data quality rules on top of each data assets that time scores were reflected in the Purview data quality dashboard), but suddenly now it all went blank scores showing as 'blank' Note : None of the data quality assessment were not deleted during that two months, data quality rules are still active and its still showing scores at data asset level. But its not showing in the dashboard currently. Can you please help me to sort out, is there any refresh policy associated for Purview Data quality dashboard.
Microsoft Purview Data Quality Report "Not-Available Data"
Hello! I'm working with data quality of Microsoft Purview. I have created a "Test Domain" and have data quality on its data assets. I have decided to delete the quality data as well as deleting the domain. It's been a week and the DQ Report is still like this. I think this comes from the deleted Test Domain. It affects the score in the report. Is there any way to remove the Not-Available Data? Thanks in advance!
Microsoft Purview - Establishing Data Quality Connection on Azure SQL Database
I have a service principal and already using it on the Data Map Solutions to scan and register Azure SQL Database as the source. It worked. Now, I am in the Unified Catalog and I am trying to establish a data quality connection on Microsoft purview for Azure SQL Database as its source type to run data quality scans. Why can't I use the Service Principal for the credential? Is there another way to establish the connection?
