Forum Discussion

Kris_KB's avatar
Kris_KB
Copper Contributor
Jul 29, 2024

Is there any way to increase source peak connections on "copy data" activity?

Max write connections are set to 32 but I can't find any option (if possible) to increase read connections.

 

It reads from physical SQL server and saves into Azure SQL as shown on screenshot.

 

 

  • zack-sponaugle's avatar
    zack-sponaugle
    Copper Contributor

    Hello Kris_KB 

     

    The following works for me when copying from Azure SQL DB. I haven't tried with SQL Server but I think it should work.

     

     

    You will need to decide on a partition column with an appropriate data type (integer, date, datetime). The lower and upper bound can be hard-coded (1 and 5000000 for example), or you can lookup the min/max of your partition column in a lookup step beforehand, and then set those dynamically using activity outputs (e.g.

    @{activity('Lookup Min and Max Partition Range').output.firstRow['Max_Partition_Key']}

    .

     

    For more information, refer to this document:

    https://learn.microsoft.com/en-us/azure/data-factory/connector-sql-server?tabs=data-factory#parallel-copy-from-sql-database

     

    As mentioned in the doc, higher parallelism doesn't always lead to better performance, so you may have to do some testing to find the best number.

     

    Hope this helps.

     

    Zack

Resources