Forum Discussion
Limiting scans being captured every time Data Factory pipeline is executed
Hi, How can you limit the scans that are done on the execution of a Data Factory pipeline? Currently it seems you cannot select how frequently, the scan executes every time a pipeline is execut...
DebbieH The feature is to get operational information at the time of factory runtime. Such as status, rows impacted and much more data quality aspects in the future. We cant get such operational info at the definition time. May I know the requirement to limit the lineage capture?
Hi ChandruS , if the cost to acquire the information is less than the value for decision making we may want to turn the feature off or limit how often it runs. For example the information captured on a weekly basis may be sufficient and limited benefit to getting the information at each execution.
- Thanks for checking. At this point in Azure preview, we don't have any implications on cost for tracking the lineage once or every time the pipeline runs. We will keep in mind though at any point if this becomes a cost factor.
