How to startup your data flows execution in less than 5 seconds! (Public Preview)

Published Apr 12 2021 12:27 AM 10.8K Views

ADF has added a new option in the Azure Integration Runtime for data flow TTL: Quick re-use. This feature is currently available as a public preview. By selecting the re-use option with a TTL setting, you can direct ADF to maintain the Spark cluster for that period of time after your last data flow executes in a pipeline. This will provide much faster sequential executions using that same Azure IR in your data flow activities. You may find that parallel data flow executions or data flows that take have long execution times, will be better served as single job clusters without a TTL.



In the video below, I will walk you through how to use your TTL settings in the Azure IR so you can take your data flow sequential pipelines and move from 1-2 min warm pool startup times to activities that start in seconds:




Version history
Last update:
‎Apr 12 2021 12:27 AM
Updated by: