Pipeline Execution Speed

Copper Contributor



I have a Synapse pipeline that runs a mixture of notebooks and dataflows and want to know what I can do to make the routine run faster than it does currently. It's not doing anything too complex, collecting some data from an S3 bucket, writing the output into a lake database table and then performing some transformations before loading to an Azure SQL database. The part that is taking the longest in the pipeline are the notebooks but I don't really know what compute changes I can make and to what (spark pool, integration runtime etc) will make the difference.


Many thanks,


0 Replies