Orchestrating ADF pipelines from synapse

%3CLINGO-SUB%20id%3D%22lingo-sub-1685446%22%20slang%3D%22en-US%22%3EOrchestrating%20ADF%20pipelines%20from%20synapse%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1685446%22%20slang%3D%22en-US%22%3E%3CP%3ECurrently%20we%20are%20working%20to%20come%20up%20with%20a%20scheduling%20architecture%20for%20azure%20data%20factory%20pipelines%20.%26nbsp%3B%3C%2FP%3E%3CP%3EAs%20part%20of%20that%20one%20of%20the%20major%20requirements%20that%20we%20have%20is%20automated%20dependency%20detection%20based%20on%20scheduling%20metadata.%26nbsp%3B%3C%2FP%3E%3CP%3EOne%20of%20the%20options%20that%20we%20also%20evaluated%20is%20ADF%20triggers%20%2C%20however%20that%20looks%20good%20for%20our%20requirements%20at%20a%20small%20scale%20.%20On%20a%20mass%20scale%20for%20example%20around%201800%20jobs%20it%20looks%20very%20effort%20intensive%20to%20manage%20dependencies%20via%20triggers%20.%26nbsp%3B%3C%2FP%3E%3CP%3EHence%20we%20are%20looking%20at%20some%20options%20whereby%20if%20we%20can%20control%20and%20automate%20dependency%20using%26nbsp%3B%20azure%20synapse%20tables%20and%20t-SQL%20procedures%20and%20trigger%20ADF%20pipelines%20using%20t-SQL.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ELooking%20for%20other%20scheduling%20tools%20that%20might%20meet%20this%20criteria%20.%20Open%20to%20guidance%20and%20suggestions.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Regular Visitor

Currently we are working to come up with a scheduling architecture for azure data factory pipelines . 

As part of that one of the major requirements that we have is automated dependency detection based on scheduling metadata. 

One of the options that we also evaluated is ADF triggers , however that looks good for our requirements at a small scale . On a mass scale for example around 1800 jobs it looks very effort intensive to manage dependencies via triggers . 

Hence we are looking at some options whereby if we can control and automate dependency using  azure synapse tables and t-SQL procedures and trigger ADF pipelines using t-SQL. 

 

Looking for other scheduling tools that might meet this criteria . Open to guidance and suggestions. 

 

0 Replies