Forum Discussion
Struggling with running DQ Scans (Long queuing and Retry Count Error Issues)
Hi everyone,
I have been exploring Microsoft Purview Data Quality quite extensively. At this point, I have configured more than 4,000 data quality rules across more than 10 Microsoft Fabric capacities, each with a minimum capacity of F16. Fabric is the source for all assets registered in Purview.
I have identified several issues with the product, but the two that are currently impacting me the most are the following:
- DQ scans failing with a generic error“Max Retry Count Reached. Ending Workflow. Current Task HandleError”The challenge is that the error message does not identify which rule is causing the failure. As a result, I have to troubleshoot manually by disabling groups of rules, rerunning the scans, and repeating the process until I find the problematic rule. This trial-and-error approach is very time-consuming, especially at this scale.
- This seems to be caused by issues in some of the DQ rules, even though all rules are marked as “Good to go” in Purview.
- When running Data Quality scans, I often receive the following error:
- DQ scans remain queued for a long timeI am not sure why this happens or what resource, orchestration, or scheduling constraint is causing the delay.
- Whenever I run these DQ scans, they remain in a Queued state for at least 10 minutes, even when there is nothing running on the Fabric capacities.
Has anyone experienced similar behavior with Purview Data Quality at this scale?
Specifically, I would appreciate any guidance on:
- How to identify which DQ rule is causing a scan failure
- Why scans remain queued even when Fabric capacity appears to be idle
- Whether there are known limitations or best practices for running thousands of DQ rules in Purview
Thank you.
2 Replies
lsims, I would extend Jamony's suggestion by measuring queue time separately from execution time. For each scan, record submission, start and completion times, asset, rule-set version and job ID. Fabric capacity appearing idle does not identify where the Purview job is waiting, and a retry-limit error alone does not prove a defective rule. Compare one previously successful rule on one unchanged asset with the larger rule set, keeping the connection and test window comparable.
If failure follows a particular rule group, narrow it in a test copy while preserving the original configuration. If the minimal scan also waits or fails, the evidence points toward a shared connection, service or orchestration issue worth escalating. Microsoft's troubleshooting page also distinguishes a skipped scan from a failed scan and lists source-format, connection and schema checks: https://learn.microsoft.com/en-us/purview/unified-catalog-data-quality-troubleshooting . I would not infer a supported concurrency limit or recommend a capacity upgrade from the symptoms alone.
The governance consequence is covered by the Measure and Prove stages in my published PALO Framework: https://paloframework.org/ . Report rule execution coverage, data quality results and evidence freshness separately. A failed assessment leaves quality unknown for the affected scope; it must not become a pass or a data defect. Keep the last successful result visibly dated, with an owner for the assessment gap. This makes the dashboard useful while the underlying scan issue is investigated.Hi lsims, with 4,000 rules across multiple Fabric capacities, I would treat this as both a scale issue and an observability issue. Start by splitting scans into smaller rule groups by domain or table family so you can isolate failures faster. Also open a Microsoft support case with scan IDs, capacity details, timestamps, and the Max Retry Count Reached examples, because the product should ideally expose which rule or asset caused the retry loop.
