Data Ingestion in Azure Data Factory

%3CLINGO-SUB%20id%3D%22lingo-sub-835877%22%20slang%3D%22en-US%22%3EData%20Ingestion%20in%20Azure%20Data%20Factory%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-835877%22%20slang%3D%22en-US%22%3E%3CP%3EI%20need%20guidance%20to%20ingest%20data%20from%20multiple%20tables%20in%20different%20RDBMS%20tables%2F.csv%20files%20(salesforce%2CSFTP%2C...)%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3ECreate%20a%20configuration%20table%20with%20column%20(schema-name%2C%20table-name%2C%20optional%20column%20list%2C%20filter%20condition)%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3E%C2%B7%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3EIn%20ADF%20pipeline%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3Eo%26nbsp%3B%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3ERead%20data%20from%20configuration-table%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3Eo%26nbsp%3B%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3Ecreate%20%5BCopy-each-table%5D%20activity%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3E%C2%A7%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3Ecreate%20%5Bcopy-table%5D%20activity%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3E%C2%B7%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3ECreate%20ADLS%20sink%20dataset%20to%20copy%20data%20into%20HD-Insight%20ADLS.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EBenefits%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3E%C2%B7%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3EBy%20this%2C%20the%20ADF%20pipeline%20will%20execute%20and%20ingest%20multiple-tables%20data%20in%20parallel%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3E%C2%B7%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%3EAdd%20a%20record%20in%20the%20configuration-table%20when%20a%20new%20table%20to%20be%20ingested.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3EKindly%20provide%20guidance...Thanks%20!!%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-835877%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Occasional Visitor

I need guidance to ingest data from multiple tables in different RDBMS tables/.csv files (salesforce,SFTP,...)

 

Create a configuration table with column (schema-name, table-name, optional column list, filter condition)

·         In ADF pipeline

o   Read data from configuration-table

o   create [Copy-each-table] activity

§  create [copy-table] activity

·         Create ADLS sink dataset to copy data into HD-Insight ADLS.

 

Benefits

·         By this, the ADF pipeline will execute and ingest multiple-tables data in parallel

·         Add a record in the configuration-table when a new table to be ingested.

 

Kindly provide guidance...Thanks !!

 

 

0 Replies