Blog Post

Azure Data Factory Blog
1 MIN READ

Session log is now available in copy activity

Ye Xu's avatar
Ye Xu
Icon for Microsoft rankMicrosoft
Dec 02, 2020

 

Benefit of using session log in copy activity:

 

With session log enabled in copy activity, you can log your copied file names, which can help you to further ensure the data is not only successfully copied from source to destination store, but also consistent between source and destination store by verifying the copied file names with their metadata in the log files. When you enable fault tolerance setting in copy activity to skip faulty data, the skipped files and skipped rows can also be logged. For more information, you can get details from here.

 

How to get started:

 

You can create a pipeline with a copy activity, and then configure the copy activity to enable session log as below:

 

The sample log file is as below:

 

From the log file above, you can see sample1.csv has been skipped because it failed to be verified to be consistent between source and destination store. You can get more details about why sample1.csv becomes inconsistent is because it was being changed by other applications when ADF copy activity is copying at the same time. You can also see sample2.csv has been successfully copied from source to destination store.

Updated Dec 02, 2020
Version 5.0

1 Comment

  • brianbolger's avatar
    brianbolger
    Copper Contributor

    Hi Ye Xu ,

     

    We're using a copy activity with a REST sink, any plans to expand the logging to include that?
    Currently the logging only logs info on the source of the activity but not the sink.

     

    Thanks