Data Ingestion in Azure Data Factory

Occasional Visitor

I need guidance to ingest data from multiple tables in different RDBMS tables/.csv files (salesforce,SFTP,...)

 

Create a configuration table with column (schema-name, table-name, optional column list, filter condition)

·         In ADF pipeline

o   Read data from configuration-table

o   create [Copy-each-table] activity

§  create [copy-table] activity

·         Create ADLS sink dataset to copy data into HD-Insight ADLS.

 

Benefits

·         By this, the ADF pipeline will execute and ingest multiple-tables data in parallel

·         Add a record in the configuration-table when a new table to be ingested.

 

Kindly provide guidance...Thanks !!

 

 

0 Replies