Jul 19 2019 01:36 PM
Hi all,
I get quarterly excel files that I have been manually combining and I am wondering if there is a more efficient way that I can be doing it with less chance of user error?
Each quarter I get the following excel workbooks:
Currently, each quarter I take the new data and append it to the bottom of a master excel workbook with a worksheet for each of the above books. This can be quite time consuming as occasionally columns are added into the quarterly reports so they don't line up with the columns of the master, and I have to adjust things.
Is there a way in excel that I can append the rows from each new workbook to its corresponding sheet in the master where the data will be placed in the column with the same header, and in situations where there is no existing matching header, insert a new column with that header to the master file? This would leave a bunch of blanks in the previous rows, but that wouldn't really be an issue for me.
Any guidance or suggestions would be greatly appreciated!
Cheers,
Brian
Jul 20 2019 01:45 AM
Hi Brian
To my humble point of view, you have 2 options (depending upon the structure of your files)
Option 1:
Creating a Code in VBA
I am attaching an excel file with about 90 sheets and by triggering the code using the shortcut CTRL+ SHIFT + P , all the sheets are consolidated and appended, while the original sheets are deleted. (of course modifications can be done as needed, I am just showing you an example.
Option 2
Using Power Query to append any number of files together if they were put in the same folder. All what you need to do with every new file is to refresh the Query.
Here is a link to a similar situation explained by Excel MVP Mike Girvin
https://www.youtube.com/watch?v=_csX8sCzJd0
Hope that helps
Nabil Mourad
Jul 26 2019 02:52 PM
Thanks for your answer! Macros seem to be blocked by my IT department, so I will see if I can make the power query method work - the tutorial you linked looks very helpful.
I have never used power query before - Will I be able to use it even if the columns are not consistent? As in, if there is a new column in the latest workbook, will it just create a new column in the file and have null values for the previous rows?
Cheers,
Brian
Jul 26 2019 03:10 PM
Hi Brian
Although Power Query is doing magic, however, at the end it just repeats the same steps in the same sequence, assuming that you run the function to similar sources.
Also note that if you are recording among the steps, a step that keeps 3 columns and Deletes other columns >> so it will delete Other columns whether they are 5 or 6 or 7...
Once you get the concept, you apply it according to your work situation
Good Luck
Nabil Mourad
Jul 27 2019 03:33 AM
You may create template file with dummy data with all possible columns in your files and, if columns have names (texts in first row before data), with exactly the same names as in your files. Combining the files take that template as the sample.
If in one of the files compare to the template extra columns appear, or column(s) have another order, or column(s) have another names - such data will be missed or you'll have an error, depends on situation.
if in the file there are no some columns as in template, data will be filled by null.
Jul 29 2019 07:17 AM
@Sergei Baklan and @nabilmourad,
Thanks for your replies!
So if make a template file with all the columns and headers that will be there in the end, it will add null data to columns where there is no data and add the data it has to the appropriate column?
For instance if I make a table template that looks like this:
Col A | Col B | Col C | Col D | Col E | Col F | Col G |
And my first worksheet looked like this:
Col A | Col B | Col D | Col E | Col F |
abc | abc | abc | abc | abc |
And a second looked like this:
Col A | Col B | Col C | Col E | Col G |
abc | abc | abc | abc | abc |
I would get an output like this?
Col A | Col B | Col C | Col D | Col E | Col F | Col G |
abc | abc | abc | abc | abc | ||
abc | abc | abc | abc | abc |
And if so, would it matter if the columns weren't necessarily in the same order as long as they had the same headers?
Cheers,
Brian
Jul 29 2019 02:20 PM - edited Jul 29 2019 02:22 PM
Hi Brian,
Yes, practically. Initial result will be as
After filtering the template file and removing first column
You may add as many files as needed into the folder
Jul 29 2019 02:25 PM
And attached are template and two dummy files. Put them into the folder c:\test (only these files, nothing more), file from previous post in any other place and be sure all together works. After that you may try to play with your real files in similar way.
Jul 30 2019 07:36 AM
Thanks for posting the files. Unfortunately I don't have access to my C:/ drive... Will it work if I put it in my z:/ drive? Or would the code need to be changed?
Cheers,
Brian
Jul 30 2019 10:07 AM
Brian, I added the parameter to the attached file.
When you start with Power Query most probably you'll see the warning what your version could be incompatible with queries - ignore it.
Open right pane with queries (Data-> double clink on Queries and Connections). It shall be like
Double click on pFilePath and change it on yours. Now double click on Test and Refresh it. Before that ignore all errors.
Group of queries under Transform... is generated automatically, you query is only Test.