Forum Discussion

Lopamudra's avatar
Lopamudra
Copper Contributor
Sep 18, 2020

Orchestrating ADF pipelines from synapse

Currently we are working to come up with a scheduling architecture for azure data factory pipelines . 

As part of that one of the major requirements that we have is automated dependency detection based on scheduling metadata. 

One of the options that we also evaluated is ADF triggers , however that looks good for our requirements at a small scale . On a mass scale for example around 1800 jobs it looks very effort intensive to manage dependencies via triggers . 

Hence we are looking at some options whereby if we can control and automate dependency using  azure synapse tables and t-SQL procedures and trigger ADF pipelines using t-SQL. 

 

Looking for other scheduling tools that might meet this criteria . Open to guidance and suggestions. 

 

No RepliesBe the first to reply

Resources