Forum Discussion
Data Profiling on Multiple tables at once
I need a sanity check here. I have created a Data Product in my Unified Catalog and have added a handful of tables as Data Assets to that Data Product. I am not seeing anywhere in the UI where I can run a Profile data on all data assets in the data product at once. Is it really only possible in the UI to run a profile on a single data asset at a time and then review the results by clicking each column? I read that you could automate with the REST API but we were hoping that something that should be simple to do was available directly in the UI and didn't need coding to run.
NickDoughtyMicrosoft
The APIs for data profiling are not yet available but they are something we are planning to release after certification. The data profiling is not a part of the Atlas APIs that is why you will not see it there. This request sounds like bulk profiling but is it more to automatically profile all datasets where possible when added to a data product?
Hi ChrisLiley,
You're correct that in the UI, profiling data on multiple tables at once isn’t directly supported. You can only run profiling on one data asset at a time and view the results individually. While automation through the REST API is possible, there isn’t a built-in feature in the UI for profiling multiple assets simultaneously. I understand it would be more convenient to do it directly in the UI, but for now, automation via the API is the best alternative.
Hope that helps!
Thanks for confirming what I was seeing. When you say automation via the API, would you be able to point us in the right direction to some documentation? I have developer that has looked at the Rest API and the Atlas API and doesn't see anything regarding data profiling.
Hi ChrisLiley,
You're welcome! Regarding automation via the API, you're right that the data profiling functionality is not immediately apparent in the REST or Atlas API. However, depending on the platform you're using, there may be relevant endpoints for creating or scheduling profiling jobs, though they might not be explicitly labeled as "data profiling" in the API documentation.
I would recommend checking the documentation for any endpoints related to metadata operations or data asset management—those may provide ways to trigger or schedule profiling jobs. If the specific functionality isn’t documented, it might be worth reaching out to the support team for more guidance on how to script the profiling process.
Regards!
