Forum Discussion

big_ozzie's avatar
big_ozzie
Copper Contributor
Dec 21, 2023

Create an ADF dataset that handles dynamic column headers

I want to create a source dataset in ADF that is linked to a blob storage container. The files landing in the container will be CSVs with two columns - the first column name will always be the same, the second will always be different. 

 

After this, I'd like to create a pipeline that skips the first 9 lines of the CSV, where the output results in a dataset with new column headers that were originally on line 10. I then want to remove the last 10 columns of this new dataset and copy the resultant data to a SQL database table.

 

I have bits and pieces of this figured out but cannot tie the whole thing together. Thanks! 

 

 

No RepliesBe the first to reply

Resources