When you move data from source to destination store, Azure Data Factory copy activity provides an option for you to do additional data consistency verification to ensure the data is not only successfully copied from source to destination store, but also verified to be consistent between source and destination store. Once inconsistent files have been found during the data movement, you can either abort the copy activity or continue to copy the rest by enabling fault tolerance setting to skip inconsistent files. You can get the copied and skipped file names by enabling session log setting in copy activity.
With data consistency verification enabled, when copying binary files, ADF copy activity will verify file size, lastModifiedDate, and MD5 checksum for each binary file copied from source to destination store to ensure the data consistency between source and destination store. When copying tabular data, copy activity will verify the row count after job completes. Be aware the copy performance will be affected by enabling this option.
In ADF copy activity, you can choose to enable "Data consistency verification" setting, and then decide if you want to abort the copy activity job or continue to copy the rest once inconsistent files found as below:
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.