Get metadata driven data ingestion pipelines on ADF within 10 minutes

By
Published Jul 08 2021 08:29 PM 3,818 Views
Microsoft

Now you can build large-scale data copy pipelines with metadata-driven approach on copy data tool within 10 minutes ! 

 

metadata driven blog.png

 

When you want to copy huge amounts of objects (for example, thousands of tables) or load data from large variety of sources to Azure, the appropriate approach is to input the name list of the objects with required copy behaviors in a control table, and then use parameterized pipelines to read the same from the control table and apply them to the jobs accordingly. By doing so, you can maintain (for example, add/remove) the objects list to be copied easily by just updating the object names in control table instead of redeploying the pipelines. What’s more, you will have single place to easily check which objects copied by which pipelines/triggers with defined copy behaviors.

 

Copy data tool in ADF eases the journey of building such metadata driven data copy pipelines. After you go through an intuitive flow from a wizard-based experience, the tool can generate parameterized pipelines and SQL scripts for you to create external control tables accordingly. After you run the generated scripts to create the control table in your SQL database, your pipelines will read the metadata from the control table and apply them on the copy jobs automatically.

 

You can get more details here.

3 Comments
Occasional Visitor

This is great news. I have been a fan of metadata-driven ingestion for many years already.

I had a deeper look at this today. please have a look at First look at metadata-driven copy task for Azure Data Factory | az Data Guy if you want to get a demo environment up and running quickly and if you want to read about my first impressions of this feature.

Frequent Visitor

Trying to use this.

How long should I be waiting after the review my selections until the scripts are generated this is my third try and each time it seems to get stuck after Review.
I have waited 30 minutes and process is still Saving... (with status on every option pending)  9 hours later still looking at the same screen.

ozhug_0-1626041042493.png

 

Occasional Visitor

@ozhug for me the last steps tales just a couple of seconds. I have not seen any issue there...

%3CLINGO-SUB%20id%3D%22lingo-sub-2528219%22%20slang%3D%22en-US%22%3EGet%20metadata%20driven%20data%20ingestion%20pipelines%20on%20ADF%20within%2010%20minutes%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2528219%22%20slang%3D%22en-US%22%3E%3CP%3ENow%20you%20can%20build%20large-scale%20data%20copy%20pipelines%20with%20metadata-driven%20approach%20on%20copy%20data%20tool%20within%2010%20minutes%20!%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22metadata%20driven%20blog.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F294379i38919B40EBF05C8B%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22metadata%20driven%20blog.png%22%20alt%3D%22metadata%20driven%20blog.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhen%20you%20want%20to%20copy%20huge%20amounts%20of%20objects%20(for%20example%2C%20thousands%20of%20tables)%20or%20load%20data%20from%20large%20variety%20of%20sources%20to%20Azure%2C%20the%20appropriate%20approach%20is%20to%20input%20the%20name%20list%20of%20the%20objects%20with%20required%20copy%20behaviors%20in%20a%20control%20table%2C%20and%20then%20use%20parameterized%20pipelines%20to%20read%20the%20same%20from%20the%20control%20table%20and%20apply%20them%20to%20the%20jobs%20accordingly.%20By%20doing%20so%2C%20you%20can%20maintain%20(for%20example%2C%20add%2Fremove)%20the%20objects%20list%20to%20be%20copied%20easily%20by%20just%20updating%20the%20object%20names%20in%20control%20table%20instead%20of%20redeploying%20the%20pipelines.%20What%E2%80%99s%20more%2C%20you%20will%20have%20single%20place%20to%20easily%20check%20which%20objects%20copied%20by%20which%20pipelines%2Ftriggers%20with%20defined%20copy%20behaviors.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ECopy%20data%20tool%20in%20ADF%20eases%20the%20journey%20of%20building%20such%20metadata%20driven%20data%20copy%20pipelines.%20After%20you%20go%20through%20an%20intuitive%20flow%20from%20a%20wizard-based%20experience%2C%20the%20tool%20can%20generate%20parameterized%20pipelines%20and%20SQL%20scripts%20for%20you%20to%20create%20external%20control%20tables%20accordingly.%20After%20you%20run%20the%20generated%20scripts%20to%20create%20the%20control%20table%20in%20your%20SQL%20database%2C%20your%20pipelines%20will%20read%20the%20metadata%20from%20the%20control%20table%20and%20apply%20them%20on%20the%20copy%20jobs%20automatically.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20can%20get%20more%20details%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fdata-factory%2Fcopy-data-tool-metadata-driven%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%22%3Ehere%3C%2FA%3E.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2528219%22%20slang%3D%22en-US%22%3E%3CP%3ENow%20you%20can%20build%20large-scale%20data%20copy%20pipelines%20with%20metadata-driven%20approach%20on%20copy%20data%20tool%20within%2010%20minutes%20!!%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2528219%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Data%20Integration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ECopy%20Activity%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2535357%22%20slang%3D%22en-US%22%3ERe%3A%20Get%20metadata%20driven%20data%20ingestion%20pipelines%20on%20ADF%20within%2010%20minutes%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2535357%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20is%20great%20news.%20I%20have%20been%20a%20fan%20of%20metadata-driven%20ingestion%20for%20many%20years%20already.%3C%2FP%3E%3CP%3EI%20had%20a%20deeper%20look%20at%20this%20today.%20please%20have%20a%20look%20at%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fazdataguy.com%2Fblog%2Fmetadatadriven-adf-available%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3EFirst%20look%20at%20metadata-driven%20copy%20task%20for%20Azure%20Data%20Factory%20%7C%20az%20Data%20Guy%3C%2FA%3E%26nbsp%3Bif%20you%20want%20to%20get%20a%20demo%20environment%20up%20and%20running%20quickly%20and%20if%20you%20want%20to%20read%20about%20my%20first%20impressions%20of%20this%20feature.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2536542%22%20slang%3D%22en-US%22%3ERe%3A%20Get%20metadata%20driven%20data%20ingestion%20pipelines%20on%20ADF%20within%2010%20minutes%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2536542%22%20slang%3D%22en-US%22%3E%3CP%3ETrying%20to%20use%20this.%3C%2FP%3E%3CP%3EHow%20long%20should%20I%20be%20waiting%20after%20the%20review%20my%20selections%20until%20the%20scripts%20are%20generated%20this%20is%20my%20third%20try%20and%20each%20time%20it%20seems%20to%20get%20stuck%20after%20Review.%3CBR%20%2F%3EI%20have%20waited%2030%20minutes%20and%20process%20is%20still%20Saving...%20(with%20status%20on%20every%20option%20pending)%26nbsp%3B%209%20hours%20later%20still%20looking%20at%20the%20same%20screen.%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22ozhug_0-1626041042493.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F294905iF53AEC3F80DBE2C2%2Fimage-size%2Fmedium%3Fv%3Dv2%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22ozhug_0-1626041042493.png%22%20alt%3D%22ozhug_0-1626041042493.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2536964%22%20slang%3D%22en-US%22%3ERe%3A%20Get%20metadata%20driven%20data%20ingestion%20pipelines%20on%20ADF%20within%2010%20minutes%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2536964%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F1100858%22%20target%3D%22_blank%22%3E%40ozhug%3C%2FA%3E%26nbsp%3Bfor%20me%20the%20last%20steps%20tales%20just%20a%20couple%20of%20seconds.%20I%20have%20not%20seen%20any%20issue%20there...%3C%2FP%3E%3C%2FLINGO-BODY%3E
Co-Authors
Version history
Last update:
‎Jul 08 2021 10:00 PM
Updated by: