Forum Discussion

Kris_KB's avatar
Kris_KB
Copper Contributor
Jul 29, 2024

Is there any way to increase source peak connections on "copy data" activity?

Max write connections are set to 32 but I can't find any option (if possible) to increase read connections.   It reads from physical SQL server and saves into Azure SQL as shown on screenshot.   ...
azure data factory
Copy Activity
zack-sponaugle's avatar
zack-sponaugle
Copper Contributor
Jul 30, 2024

Hello Kris_KB 

 

The following works for me when copying from Azure SQL DB. I haven't tried with SQL Server but I think it should work.

 

 

You will need to decide on a partition column with an appropriate data type (integer, date, datetime). The lower and upper bound can be hard-coded (1 and 5000000 for example), or you can lookup the min/max of your partition column in a lookup step beforehand, and then set those dynamically using activity outputs (e.g.

@{activity('Lookup Min and Max Partition Range').output.firstRow['Max_Partition_Key']}

.

 

For more information, refer to this document:

https://learn.microsoft.com/en-us/azure/data-factory/connector-sql-server?tabs=data-factory#parallel-copy-from-sql-database

 

As mentioned in the doc, higher parallelism doesn't always lead to better performance, so you may have to do some testing to find the best number.

 

Hope this helps.

 

Zack

Resources