How to implement check points in my existing design to avoid duplicates in sinkin ADF

Copper Contributor


In adf one of the data flow is failed due to transist error in the middle of execution.the data flow is having insert ,update and delete operation based on scd2 scenario.To avoid the transist failures we have applied retry option.But while retrying second attempt there is some data loaded in the sink table already in the first attempt and causes duplicate records in the table during the second run for insert scenario.Is there any option to restart the job from the point of failure during the second retry.

I have not implemented cdc at source level that's the reason it seems I did not see any option at sink setting about check points to avoid duplicate Dara.


 Please suggest.

Thanks in advance.

0 Replies