%3CLINGO-SUB%20id%3D%22lingo-sub-878380%22%20slang%3D%22en-US%22%3EADF%20adds%20TTL%20to%20Azure%20IR%20to%20reduce%20Data%20Flow%20activity%20times%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-878380%22%20slang%3D%22en-US%22%3E%3CP%3EADF%20has%20added%20a%20TTL%20(time-to-live)%20option%20to%20the%20Azure%20Integration%20Runtime%20for%20Data%20Flow%20properties%20to%20reduce%20data%20flow%20activity%20times.%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F133726iE14F64469E138721%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20alt%3D%22azureir2.png%22%20title%3D%22azureir2.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3EThis%20setting%20is%20only%20used%20during%20ADF%20pipeline%20executions%20of%20Data%20Flow%20activities.%20Debug%20executions%20from%20pipelines%20and%20data%20preview%20debugging%20will%20continue%20to%20use%20the%20debug%20settings%20which%20has%20a%20preset%20TTL%20of%2060%20minutes.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20you%20leave%20the%20TTL%20to%200%2C%20ADF%20will%20always%20spawn%20a%20new%20Spark%20cluster%20environment%20for%20every%20Data%20Flow%20activity%20that%20executes.%20This%20means%20that%20an%20Azure%20Databricks%20cluster%20is%20provisioned%20each%20time%20and%20takes%20about%205-7%20minutes%20to%20become%20available%20and%20execute%20your%20job.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHowever%2C%20if%20you%20set%20a%20TTL%2C%20ADF%20will%20maintain%20a%20pool%20of%20VMs%20which%20can%20be%20utilized%20to%20spin-up%20each%20subsequent%20data%20flow%20activity%20against%20that%20same%20Azure%20IR.%20This%20reduces%20the%20amount%20of%20time%20needed%20to%20start-up%20the%20environment%20before%20your%20job%20is%20executed.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EADF%20will%20maintain%20that%20pool%20for%20the%20TTL%20time%20after%20the%20last%20data%20flow%20pipeline%20activity%20executes.%20Note%20that%20this%20will%20extend%20your%20billing%20period%20for%20a%20data%20flow%20to%20the%20extended%20time%20of%20your%20TTL.%20However%2C%20your%20data%20flow%20job%20execution%20time%20will%20decrease%20because%20of%20the%20re-use%20of%20the%20VMs%20from%20the%20compute%20pool.%20The%20compute%20resources%20are%20not%20provisioned%20until%20your%20first%20data%20flow%20activity%20is%20executed%20using%20that%20Azure%20IR.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ERead%20more%20about%20the%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fconcepts-integration-runtime%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EAzure%20Integration%20Runtime%20here%3C%2FA%3E.%20And%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fconcepts-data-flow-performance%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehere%3C%2FA%3E%20is%20an%20ADF%20Data%20Flow%20performance%20guide%20to%20help%20you%20optimize%20your%20environment.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-878380%22%20slang%3D%22en-US%22%3E%3CP%3EADF%20has%20added%20a%20TTL%20(time-to-live)%20option%20to%20the%20Azure%20Integration%20Runtime%20for%20Data%20Flow%20properties%20to%20reduce%20data%20flow%20activity%20times.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-878380%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1079150%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20TTL%20to%20Azure%20IR%20to%20reduce%20Data%20Flow%20activity%20times%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1079150%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F175989%22%20target%3D%22_blank%22%3E%40Mark%20Kromer%3C%2FA%3E%26nbsp%3B%2C%20Where%20within%20my%20Azure%20Data%20factory%20(or%20data%20flows)%20do%20I%20set%20this%20Time%20to%20Live%20(TTL)%20property%3F%20I%20have%20looked%20quite%20a%20bit%20but%20can't%20find%20the%20screen%20you%20are%20showing%20above.%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1079361%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20TTL%20to%20Azure%20IR%20to%20reduce%20Data%20Flow%20activity%20times%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1079361%22%20slang%3D%22en-US%22%3E%3CP%3EFrom%20the%20ADF%20pipeline%20designer%20UI%2C%20go%20to%20Connections%20%26gt%3B%20Integration%20Runtimes%20%26gt%3B%20New.%20Select%20Azure%20IR%20and%20then%20open%20the%20Data%20Flow%20Run%20Time%20properties%20section.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1474612%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20TTL%20to%20Azure%20IR%20to%20reduce%20Data%20Flow%20activity%20times%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1474612%22%20slang%3D%22en-US%22%3E%3CP%3EHi%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F175989%22%20target%3D%22_blank%22%3E%40Mark%20Kromer%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20can%20find%20the%20option%20you%20indicate.%20On%20the%20Integration%20Runtime%20Setup%20windows%20there%20are%20only%20two%20options%3A%20Azure%20self-Hosted%20and%20Azure%20SSIS.%20However%2C%20I%20do%20not%20see%20the%20Data%20Flow%20Run%20Time%20properties%20in%20any%20of%20them.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1474658%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20TTL%20to%20Azure%20IR%20to%20reduce%20Data%20Flow%20activity%20times%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1474658%22%20slang%3D%22en-US%22%3E%3CP%3ETry%20creating%20a%20new%20Azure%20IR%20to%20see%20the%20options%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22ir.png%22%20style%3D%22width%3A%20615px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F199524i2D23EDFEBF43B690%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%22ir.png%22%20alt%3D%22ir.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1566824%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20TTL%20to%20Azure%20IR%20to%20reduce%20Data%20Flow%20activity%20times%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1566824%22%20slang%3D%22en-US%22%3E%3CP%3Eis%20there%20any%20option%20to%20update%20the%20TTL%20value%20for%20the%20an%20existing%20IR%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1566829%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20TTL%20to%20Azure%20IR%20to%20reduce%20Data%20Flow%20activity%20times%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1566829%22%20slang%3D%22en-US%22%3E%3CP%3Ealso%2C%20is%20there%20an%20option%20to%20pass%20the%20custom%20created%20IR's%20as%20parameters%20to%20pipelines%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1567521%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20TTL%20to%20Azure%20IR%20to%20reduce%20Data%20Flow%20activity%20times%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1567521%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F749935%22%20target%3D%22_blank%22%3E%40saikumare%3C%2FA%3E%26nbsp%3BCurrently%2C%20we%20do%20not%20support%20updating%20the%20TTL%20value%2C%20you%20must%20create%20a%20new%20Azure%20IR%20instead.%20The%20IRs%20are%20also%20not%20parameterizable%2C%20however%2C%20if%20you%20set%20the%20IR%20in%20the%20Data%20Flow%20activity%20as%20%22Auto-Resolve%22%2C%20you%20can%20parameterize%20the%20core%20count%20and%20compute%20type%20properties%20there%2C%20making%20them%20dynamic.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1567528%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20TTL%20to%20Azure%20IR%20to%20reduce%20Data%20Flow%20activity%20times%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1567528%22%20slang%3D%22en-US%22%3E%3CP%3Ebut%20this%20%22Auto-Resolve%22%20IR%20(with%20TTL%3D0)%20does%20not%20have%20the%20option%20of%20setting%2Fupdating%20the%20TTL%20which%20would%20consume%20more%20than%204%20mins%20of%20cluster%20startup%20for%20each%20job%20.%3C%2FP%3E%3CP%3EAny%20other%20option%20to%20handle%20both%20these%20core%20count%20and%20TTL%20dynamically%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1567551%22%20slang%3D%22en-US%22%3ERe%3A%20ADF%20adds%20TTL%20to%20Azure%20IR%20to%20reduce%20Data%20Flow%20activity%20times%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1567551%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F749935%22%20target%3D%22_blank%22%3E%40saikumare%3C%2FA%3E%26nbsp%3BTTL%20is%20a%20cluster-controlling%20mechanism%2C%20so%20it%20cannot%20be%20dynamic.%20But%20making%20the%20compute%20settings%26nbsp%3B%3CEM%3E%3CSTRONG%3Einside%3C%2FSTRONG%3E%3C%2FEM%3E%3CEM%3E%26nbsp%3B%3C%2FEM%3Ethe%20Azure%20IR%20configuration%20parameterized%2C%20instead%20of%20only%20inside%20the%20activity%2C%20is%20a%20common%20ask.%20You%20can%20add%20the%20feedback%20to%20Azure%20User%20Voice%20so%20that%20we%20can%20prioritize%20the%20feature%20request%20accordingly.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

ADF has added a TTL (time-to-live) option to the Azure Integration Runtime for Data Flow properties to reduce data flow activity times.

azureir2.png

This setting is only used during ADF pipeline executions of Data Flow activities. Debug executions from pipelines and data preview debugging will continue to use the debug settings which has a preset TTL of 60 minutes.

 

If you leave the TTL to 0, ADF will always spawn a new Spark cluster environment for every Data Flow activity that executes. This means that an Azure Databricks cluster is provisioned each time and takes about 5-7 minutes to become available and execute your job.

 

However, if you set a TTL, ADF will maintain a pool of VMs which can be utilized to spin-up each subsequent data flow activity against that same Azure IR. This reduces the amount of time needed to start-up the environment before your job is executed.

 

ADF will maintain that pool for the TTL time after the last data flow pipeline activity executes. Note that this will extend your billing period for a data flow to the extended time of your TTL. However, your data flow job execution time will decrease because of the re-use of the VMs from the compute pool. The compute resources are not provisioned until your first data flow activity is executed using that Azure IR.

 

Read more about the Azure Integration Runtime here. And here is an ADF Data Flow performance guide to help you optimize your environment.

9 Comments
Occasional Visitor

Hi @Mark Kromer , Where within my Azure Data factory (or data flows) do I set this Time to Live (TTL) property? I have looked quite a bit but can't find the screen you are showing above. 

Microsoft

From the ADF pipeline designer UI, go to Connections > Integration Runtimes > New. Select Azure IR and then open the Data Flow Run Time properties section.

Occasional Visitor

Hi @Mark Kromer 

I can find the option you indicate. On the Integration Runtime Setup windows there are only two options: Azure self-Hosted and Azure SSIS. However, I do not see the Data Flow Run Time properties in any of them.

Microsoft

Try creating a new Azure IR to see the options

 

ir.png

Occasional Visitor

is there any option to update the TTL value for the an existing IR

Occasional Visitor

also, is there an option to pass the custom created IR's as parameters to pipelines?

Microsoft

@saikumare Currently, we do not support updating the TTL value, you must create a new Azure IR instead. The IRs are also not parameterizable, however, if you set the IR in the Data Flow activity as "Auto-Resolve", you can parameterize the core count and compute type properties there, making them dynamic.

Occasional Visitor

but this "Auto-Resolve" IR (with TTL=0) does not have the option of setting/updating the TTL which would consume more than 4 mins of cluster startup for each job .

Any other option to handle both these core count and TTL dynamically?

Microsoft

@saikumare TTL is a cluster-controlling mechanism, so it cannot be dynamic. But making the compute settings inside the Azure IR configuration parameterized, instead of only inside the activity, is a common ask. You can add the feedback to Azure User Voice so that we can prioritize the feature request accordingly.