SOLVED

[Power Query] how to exclude x number of rows at the top when combining csv files

%3CLINGO-SUB%20id%3D%22lingo-sub-2121273%22%20slang%3D%22en-US%22%3E%5BPower%20Query%5D%20how%20to%20exclude%20x%20number%20of%20rows%20at%20the%20top%20when%20combining%20csv%20files%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2121273%22%20slang%3D%22en-US%22%3E%3CP%20class%3D%22_1qeIAgB0cPwnLhDF9XSiJM%22%3ENew%20power%20query%20user%20here%20(2019%20version)!%3C%2FP%3E%3CP%20class%3D%22_1qeIAgB0cPwnLhDF9XSiJM%22%3EI%20started%20by%20retrieving%20a%20single%20csv%20file%20from%20a%20folder%20and%20transforming%20the%20raw%20data.%20The%20data%20contained%208%20rows%20at%20the%20top%20that%20I%20didn't%20need%20so%20I%20removed%20them%20using%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-powerquery%22%3E%3CCODE%3E%3D%20Table.Skip(%23%22Changed%20Type%22%2C8)%3C%2FCODE%3E%3C%2FPRE%3E%3CP%20class%3D%22_1qeIAgB0cPwnLhDF9XSiJM%22%3EThe%20goal%20here%20is%20to%20be%20able%20to%20combine%20additional%20csv%20files%20in%20the%20same%20folder%20and%20click%20refresh%20so%20that%20the%20new%20data%20will%20follow%20my%20applied%20steps.%20The%20issue%20I'm%20having%20is%20that%20the%20column%20names%20(row%207)%20of%20the%20additional%20csv%20files%20are%20also%20being%20added%20even%20though%20they%20are%20within%20the%208%20rows%20of%20data.%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22Screenshot_9.png%22%20style%3D%22width%3A%20717px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F254112iCA651DA7AFB7D451%2Fimage-dimensions%2F717x174%3Fv%3D1.0%22%20width%3D%22717%22%20height%3D%22174%22%20role%3D%22button%22%20title%3D%22Screenshot_9.png%22%20alt%3D%22Screenshot_9.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%20class%3D%22_1qeIAgB0cPwnLhDF9XSiJM%22%3E%3CSPAN%3ERow%207%20is%20showing%20up%20when%20combining%20the%20data%20even%20though%20I%20applied%20the%20step%20of%20removing%208%20rows.%26nbsp%3BAll%20csv%20files%20have%20the%20same%20number%20of%20columns%20and%20similar%20enough%20column%20names.%20What%20am%20I%20missing%20here%3F%20I've%20tried%20filtering%20it%20out%20but%20it%20doesn't%20seem%20to%20work.%20Thank%20you%20in%20advance!%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2121273%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EBI%20%26amp%3B%20Data%20Analysis%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EExcel%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EFormulas%20and%20Functions%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMacros%20and%20VBA%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2122345%22%20slang%3D%22en-US%22%3ERe%3A%20%5BPower%20Query%5D%20how%20to%20exclude%20x%20number%20of%20rows%20at%20the%20top%20when%20combining%20csv%20files%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2122345%22%20slang%3D%22en-US%22%3E%3CP%3EThanks%20for%20the%20blog%20post!%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F9832%22%20target%3D%22_blank%22%3E%40Wyn%20Hopkins%3C%2FA%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F133371%22%20target%3D%22_blank%22%3E%40Mynda%20Treacy%3C%2FA%3E%26nbsp%3BThe%20part%20I%20found%20most%20helpful%20was%20Step%206%20-%20I%20realized%20I%20put%20my%20transforming%20queries%20under%20my%20final%20queries%20instead%20of%20my%20sample%20file%20query.%20All%20my%20files%20are%20now%20being%20consolidated%20properly!%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2121930%22%20slang%3D%22en-US%22%3ERe%3A%20%5BPower%20Query%5D%20how%20to%20exclude%20x%20number%20of%20rows%20at%20the%20top%20when%20combining%20csv%20files%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2121930%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F9832%22%20target%3D%22_blank%22%3E%40Wyn%20Hopkins%3C%2FA%3E%26nbsp%3BYou're%20welcome!%20Always%20good%20to%20promote%20good%20work%20from%20others%20%3CLI-EMOJI%20id%3D%22lia_slightly-smiling-face%22%20title%3D%22%3Aslightly_smiling_face%3A%22%3E%3C%2FLI-EMOJI%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EPerhaps%20quite%20obvious%2C%20but%20your%20video%20was%20an%20eye-opener.%20So%20simple%2C%20but%20yet%20so%20clear.%20And%20it%20made%20me%20forget%20about%20the%20%22From%20folder%22%20option%20that%20%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F521%22%20target%3D%22_blank%22%3E%40Sergei%20Baklan%3C%2FA%3E%26nbsp%3Bmentioned.%20Am%20a%20relative%20new%20PQ%20user%20as%20I%20have%20been%20on%20a%20Mac%20since%202011.%20Still%20am%2C%20but%20running%20W10%20virtually%20on%20Parallels%20for%20the%20past%209%20months%20or%20so.%20Still%20learning%2C%20every%20day!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2121586%22%20slang%3D%22en-US%22%3ERe%3A%20%5BPower%20Query%5D%20how%20to%20exclude%20x%20number%20of%20rows%20at%20the%20top%20when%20combining%20csv%20files%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2121586%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F963058%22%20target%3D%22_blank%22%3E%40Claudia000%3C%2FA%3E%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3ENice%20blog%20here%20about%20this%26nbsp%3B%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fwww.myonlinetraininghub.com%2Fpower-query-get-files-from-a-folder%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fwww.myonlinetraininghub.com%2Fpower-query-get-files-from-a-folder%20%3C%2FA%3E%3C%2FP%3E%0A%3CP%3Efrom%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F133371%22%20target%3D%22_blank%22%3E%40Mynda%20Treacy%3C%2FA%3E%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThanks%20for%20the%20promotion%20of%20my%20video%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F403176%22%20target%3D%22_blank%22%3E%40Riny_van_Eekelen%3C%2FA%3E%26nbsp%3B%20%3Agrinning_face%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2121470%22%20slang%3D%22en-US%22%3ERe%3A%20%5BPower%20Query%5D%20how%20to%20exclude%20x%20number%20of%20rows%20at%20the%20top%20when%20combining%20csv%20files%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2121470%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F963058%22%20target%3D%22_blank%22%3E%40Claudia000%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWhy%20don't%20you%20use%20From%20Folder%20connector%20and%20adjust%20only%20one%20%3CEM%3ETransform%20from%20Sample%20File%3C%2FEM%3E%20query%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2121464%22%20slang%3D%22en-US%22%3ERe%3A%20%5BPower%20Query%5D%20how%20to%20exclude%20x%20number%20of%20rows%20at%20the%20top%20when%20combining%20csv%20files%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2121464%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F963058%22%20target%3D%22_blank%22%3E%40Claudia000%3C%2FA%3E%26nbsp%3BYou%20start%20by%20doing%20all%20the%20transformations%20needed%20on%20one%20file.%20Then%20you%20create%20a%20parameter%20for%20the%20file%20name%20used%20in%20the%20Source%20step.%20Replace%20the%20hardcoded%20file%20name%20with%20the%20parameter.%26nbsp%3BNow%2C%20you%20can%20create%20a%20function%20of%20the%20first%20query.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ECreate%20a%20table%20with%20all%20the%20file%20names%20you%20need%20to%20transform%2C%20query%20it%20and%20add%20a%20column%20through%20%22Invoke%20Custom%20function%22.%20Select%20the%20function%20you%20just%20created%2C%20select%20the%20column%20that%20contains%20the%20file%20names%20and%20press%20OK.%20Expand%20the%20new%20column.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EActually%2C%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F9832%22%20target%3D%22_blank%22%3E%40Wyn%20Hopkins%3C%2FA%3E%26nbsp%3Brecently%20posted%20a%20link%20to%20an%20easy%20to%20follow%20video%20that%20demonstrates%20this%20principle.%20His%20video%20deals%20with%20named%20ranges%20within%20one%20sheet%2C%20but%20you%20apply%20the%20same%20technique%20to%20files%20in%20a%20folder.%3C%2FP%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcPN24NK3_68%22%20target%3D%22_self%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcPN24NK3_68%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2124179%22%20slang%3D%22en-US%22%3ERe%3A%20%5BPower%20Query%5D%20how%20to%20exclude%20x%20number%20of%20rows%20at%20the%20top%20when%20combining%20csv%20files%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2124179%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F963058%22%20target%3D%22_blank%22%3E%40Claudia000%3C%2FA%3E%26nbsp%3Bglad%20it%20was%20helpful.%20And%20thanks%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F9832%22%20target%3D%22_blank%22%3E%40Wyn%20Hopkins%3C%2FA%3E%26nbsp%3Bfor%20sharing%20my%20video!%3C%2FP%3E%3C%2FLINGO-BODY%3E
New Contributor

New power query user here (2019 version)!

I started by retrieving a single csv file from a folder and transforming the raw data. The data contained 8 rows at the top that I didn't need so I removed them using 

= Table.Skip(#"Changed Type",8)

The goal here is to be able to combine additional csv files in the same folder and click refresh so that the new data will follow my applied steps. The issue I'm having is that the column names (row 7) of the additional csv files are also being added even though they are within the 8 rows of data.Screenshot_9.png

Row 7 is showing up when combining the data even though I applied the step of removing 8 rows. All csv files have the same number of columns and similar enough column names. What am I missing here? I've tried filtering it out but it doesn't seem to work. Thank you in advance!

6 Replies

@Claudia000 You start by doing all the transformations needed on one file. Then you create a parameter for the file name used in the Source step. Replace the hardcoded file name with the parameter. Now, you can create a function of the first query.

 

Create a table with all the file names you need to transform, query it and add a column through "Invoke Custom function". Select the function you just created, select the column that contains the file names and press OK. Expand the new column.

 

Actually, @Wyn Hopkins recently posted a link to an easy to follow video that demonstrates this principle. His video deals with named ranges within one sheet, but you apply the same technique to files in a folder.

https://www.youtube.com/watch?v=cPN24NK3_68 

 

 

 

 

best response confirmed by Claudia000 (New Contributor)
Solution

@Claudia000 

Why don't you use From Folder connector and adjust only one Transform from Sample File query?

@Wyn Hopkins You're welcome! Always good to promote good work from others

 

Perhaps quite obvious, but your video was an eye-opener. So simple, but yet so clear. And it made me forget about the "From folder" option that @Sergei Baklan mentioned. Am a relative new PQ user as I have been on a Mac since 2011. Still am, but running W10 virtually on Parallels for the past 9 months or so. Still learning, every day!

Thanks for the blog post! @Wyn Hopkins@Mynda Treacy The part I found most helpful was Step 6 - I realized I put my transforming queries under my final queries instead of my sample file query. All my files are now being consolidated properly! 

@Claudia000 glad it was helpful. And thanks @Wyn Hopkins for sharing my video!