We have enhanced the resume capability in ADF by which you can build robust pipelines for many scenarios. With this enhancement, if one of the activities fails, you can rerun the pipeline from that failed activity. When moving data via the copy activity, you can resume the copy from the last failure point at the file level instead of starting from the beginning, which greatly increase the resilience of your data movement solution especially on large size of files movement between file-based stores. This resume features in copy activity applies to the following file-based connectors: Amazon S3, Azure Blob, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, Azure File Storage, File System, FTP, Google Cloud Storage, HDFS, and SFTP.
Customers can benefit from this improvement when they are doing data lake migration, for example to migrate terabytes or petabytes of data from Amazon S3 to Azure Data Lake Storage Gen2. The benefits include increasing the success rate on data lake migration projects, gaining flexibility to control the process on data migration, and saving effort and time on data partitions or any solutions explicitly created for high resilience. Some early adopters trying out this feature under preview had successfully moved petabytes of data from AWS S3 to Azure with a simple pipeline containing a single copy activity. Once any unexpected failure happened during the data migration, they simply rerun this pipeline from last failure or specify retry count to let copy activity retry automatically, and the copy activity continued to copy rest of the files on AWS S3 without starting from any files which had been already copied to the destination last time.
When you copy data from Amazon S3, Azure Blob, Azure Data Lake Storage Gen2 and Google Cloud Storage, copy activity can resume from arbitrary number of copied files. While for the rest of file-based connectors as source, currently copy activity supports resume from a limited number of files, usually at the range of tens of thousands and varies depending on the length of the file paths; files beyond this number will be re-copied during reruns.
For more concrete information, please refer this.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.