Forum Discussion
big_ozzie
Dec 21, 2023Copper Contributor
Create an ADF dataset that handles dynamic column headers
I want to create a source dataset in ADF that is linked to a blob storage container. The files landing in the container will be CSVs with two columns - the first column name will always be the same, the second will always be different.
After this, I'd like to create a pipeline that skips the first 9 lines of the CSV, where the output results in a dataset with new column headers that were originally on line 10. I then want to remove the last 10 columns of this new dataset and copy the resultant data to a SQL database table.
I have bits and pieces of this figured out but cannot tie the whole thing together. Thanks!
No RepliesBe the first to reply