How to startup your data flows execution in less than 5 seconds! (Public Preview)
Published Apr 12 2021 12:27 AM 19K Views
Microsoft

ADF has added a new option in the Azure Integration Runtime for data flow TTL: Quick re-use. This feature is currently available as a public preview. By selecting the re-use option with a TTL setting, you can direct ADF to maintain the Spark cluster for that period of time after your last data flow executes in a pipeline. This will provide much faster sequential executions using that same Azure IR in your data flow activities. You may find that parallel data flow executions or data flows that take have long execution times, will be better served as single job clusters without a TTL.

 

resuse1.png

In the video below, I will walk you through how to use your TTL settings in the Azure IR so you can take your data flow sequential pipelines and move from 1-2 min warm pool startup times to activities that start in seconds:

 

 

 

11 Comments
Co-Authors
Version history
Last update:
‎Apr 12 2021 12:27 AM
Updated by: