Combining multiple quarterly workbooks into one master workbook.

[Brianbuchanan]

Hi all,

I get quarterly excel files that I have been manually combining and I am wondering if there is a more efficient way that I can be doing it with less chance of user error?

Each quarter I get the following excel workbooks:

1. An inpatient listing for my hospitals
2. A listing of inpatients who had a complication
3. A listing of inpatients who returned to the Emergency Department after discharge
4. A listing of inpatients who were readmitted within 30 days of discharge.

Currently, each quarter I take the new data and append it to the bottom of a master excel workbook with a worksheet for each of the above books. This can be quite time consuming as occasionally columns are added into the quarterly reports so they don't line up with the columns of the master, and I have to adjust things.

Is there a way in excel that I can append the rows from each new workbook to its corresponding sheet in the master where the data will be placed in the column with the same header, and in situations where there is no existing matching header, insert a new column with that header to the master file? This would leave a bunch of blanks in the previous rows, but that wouldn't really be an issue for me.

Any guidance or suggestions would be greatly appreciated!

Cheers,

Brian

[#2]

@Brianbuchanan 

Hi Brian

To my humble point of view, you have 2 options (depending upon the structure of your files)

Option 1:

Creating a Code in VBA

I am attaching an excel file with about 90 sheets and by triggering the code using the shortcut CTRL+ SHIFT + P , all the sheets are consolidated and appended, while the original sheets are deleted. (of course modifications can be done as needed, I am just showing you an example.

Option 2

Using Power Query to append any number of files together if they were put in the same folder. All what you need to do with every new file is to refresh the Query.

Here is a link to a similar situation explained by Excel MVP Mike Girvin

https://www.youtube.com/watch?v=_csX8sCzJd0

Hope that helps

Nabil Mourad

[#3]

@nabilmourad,

Thanks for your answer! Macros seem to be blocked by my IT department, so I will see if I can make the power query method work - the tutorial you linked looks very helpful.

I have never used power query before - Will I be able to use it even if the columns are not consistent? As in, if there is a new column in the latest workbook, will it just create a new column in the file and have null values for the previous rows?

Cheers,

Brian

[#4]

@Brianbuchanan 

Hi Brian

Although Power Query is doing magic, however, at the end it just repeats the same steps in the same sequence, assuming that you run the function to similar sources.

Also note that if you are recording among the steps, a step that keeps 3 columns and Deletes other columns >> so it will delete Other columns whether they are 5 or 6 or 7...

Once you get the concept, you apply it according to your work situation

Good Luck

Nabil Mourad

[#5]

@Brianbuchanan 

You may create template file with dummy data with all possible columns in your files and, if columns have names (texts in first row before data), with exactly the same names as in your files. Combining the files take that template as the sample.

If in one of the files compare to the template extra columns appear, or column(s) have another order, or column(s) have another names - such data will be missed or you'll have an error, depends on situation.

if in the file there are no some columns as in template, data will be filled by null.

[#6]

@Sergei Baklan and @nabilmourad,

Thanks for your replies!

So if make a template file with all the columns and headers that will be there in the end, it will add null data to columns where there is no data and add the data it has to the appropriate column?

For instance if I make a table template that looks like this:

Col A	Col B	Col C	Col D	Col E	Col F	Col G

And my first worksheet looked like this:

Col A	Col B	Col D	Col E	Col F
abc	abc	abc	abc	abc

And a second looked like this:

Col A	Col B	Col C	Col E	Col G
abc	abc	abc	abc	abc

I would get an output like this?

Col A	Col B	Col C	Col D	Col E	Col F	Col G
abc	abc		abc	abc	abc
abc	abc	abc		abc		abc

And if so, would it matter if the columns weren't necessarily in the same order as long as they had the same headers?

Cheers,

Brian

[#7]

@Brianbuchanan 

Hi Brian,

Yes, practically. Initial result will be as

After filtering the template file and removing first column

You may add as many files as needed into the folder

[#8]

@Brianbuchanan 

And attached are template and two dummy files. Put them into the folder c:\test (only these files, nothing more), file from previous post in any other place and be sure all together works. After that you may try to play with your real files in similar way.

[#9]

@Sergei Baklan,

Thanks for posting the files. Unfortunately I don't have access to my C:/ drive... Will it work if I put it in my z:/ drive? Or would the code need to be changed?

Cheers,

Brian

[#10]

@Brianbuchanan 

Brian, I added the parameter to the attached file.

When you start with Power Query most probably you'll see the warning what your version could be incompatible with queries - ignore it.

Open right pane with queries (Data-> double clink on Queries and Connections). It shall be like

Double click on pFilePath and change it on yours. Now double click on Test and Refresh it. Before that ignore all errors.

Group of queries under Transform... is generated automatically, you query is only Test.