Pipeline Execution Speed

Copper Contributor

Hi,

 

I have a Synapse pipeline that runs a mixture of notebooks and dataflows and want to know what I can do to make the routine run faster than it does currently. It's not doing anything too complex, collecting some data from an S3 bucket, writing the output into a lake database table and then performing some transformations before loading to an Azure SQL database. The part that is taking the longest in the pipeline are the notebooks but I don't really know what compute changes I can make and to what (spark pool, integration runtime etc) will make the difference.

 

Many thanks,

Dan

0 Replies