incremental refresh in query editor

%3CLINGO-SUB%20id%3D%22lingo-sub-433382%22%20slang%3D%22en-US%22%3Eincremental%20refresh%20in%20query%20editor%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-433382%22%20slang%3D%22en-US%22%3E%3CP%3EI%20am%20using%20query%20editor%20in%20excel%20to%20pull%20raw%20data%20from%20multiple%20files%20within%20a%20folder%20that%20contains%20day%20to%20day%20data%20in%20it.%20I%20am%20experiencing%20a%26nbsp%3Blengthy%20waiting%20period%20when%20refreshing%20the%20query%2C%26nbsp%3Band%20each%20day%20the%20waiting%20time%20increases%20as%26nbsp%3Bit%20searching%20through%20more%26nbsp%3Band%20more%20files.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIs%20there%20a%20setting%20that%20can%20be%20applied%2C%20or%20an%20approach%20to%20tell%20the%20query%20editor%20to%20only%20look%20for%20new%20files%2C%20as%20the%20previously%20loaded%20files%20will%20not%20change%20as%20they%20will%20be%20historical%20data.%20This%20way%20the%20refresh%20times%20will%20be%20fast%20as%20it%20will%20only%20be%20looking%20for%20a%20days%20or%202%20days%20worth%20of%20data%2C%20rather%20than%203%20months%20worth%2C%20or%20by%20the%20end%20of%20year%2C%2012%20months%20worth..%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-433382%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EBI%20%26amp%3B%20Data%20Analysis%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EExcel%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EPower%20BI%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1068342%22%20slang%3D%22de-DE%22%3ESubject%3A%20incremental%20refresh%20in%20query%20editor%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1068342%22%20slang%3D%22de-DE%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F318937%22%20target%3D%22_blank%22%3E%40cchiappazzo%3C%2FA%3E%20could%20you%20solve%20this%3F%20I%20have%20the%20same%20problem.%20Regards%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1068358%22%20slang%3D%22en-US%22%3EBetreff%3A%20incremental%20refresh%20in%20query%20editor%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1068358%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F490379%22%20target%3D%22_blank%22%3E%40hwoe1%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20guess%20incremental%20refresh%20works%20only%20in%20Power%20BI%20Premium%2C%20not%20in%20Excel.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1591839%22%20slang%3D%22en-US%22%3ERe%3A%20incremental%20refresh%20in%20query%20editor%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1591839%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F318937%22%20target%3D%22_blank%22%3E%40cchiappazzo%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20was%20just%20researching%20this%20question.%20As%20you%20apparently%20know%2C%20there%20is%20a%20dedicated%20feature%20in%20PowerBI%20for%20incremental%20refresh.%3C%2FP%3E%3CP%3E.%3C%2FP%3E%3CP%3EThere%20is%20not%20a%20similar%20feature%20in%20Excel%20PowerQuery.%26nbsp%3B%20But%20I%20did%20find%20a%20couple%20of%20articles%20that%20suggest%20manual%20work%20arounds%3C%2FP%3E%3CP%3E%3CSTRONG%3EIncremental%20data%20loads%20in%20Microsoft%20Power%20Query%20for%20Excel%20(DIY%20incremental)%202015%2005%2015%3C%2FSTRONG%3E%3CBR%20%2F%3E%3CA%20href%3D%22https%3A%2F%2Fwww.oraylis.de%2Fblog%2Fincremental-data-loads-in-microsoft-power-query-for-excel%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fwww.oraylis.de%2Fblog%2Fincremental-data-loads-in-microsoft-power-query-for-excel%3C%2FA%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%3CP%3E%3CSTRONG%3EHow%20to%20create%20a%20Load%20History%20or%20Load%20Log%20in%20Power%20Query%20or%20Power%20BI%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%202016%2011%2019%3CBR%20%2F%3E%3C%2FSTRONG%3E%3CA%20href%3D%22https%3A%2F%2Fwww.thebiccountant.com%2F2016%2F02%2F09%2Fhow-to-create-a-load-history-or-load-log-in-power-query-or-power-bi%2F%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fwww.thebiccountant.com%2F2016%2F02%2F09%2Fhow-to-create-a-load-history-or-load-log-in-power-query-or-power-bi%2F%3C%2FA%3E%3CBR%20%2F%3E%3CBR%20%2F%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1599724%22%20slang%3D%22en-US%22%3ERe%3A%20incremental%20refresh%20in%20query%20editor%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1599724%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F1532%22%20target%3D%22_blank%22%3E%40ron%20S.%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIMHO%2C%20that's%20more%20about%20incremental%20data%20appending%2C%20not%20about%20incremental%20refresh%3C%2FP%3E%3C%2FLINGO-BODY%3E
Occasional Visitor

I am using query editor in excel to pull raw data from multiple files within a folder that contains day to day data in it. I am experiencing a lengthy waiting period when refreshing the query, and each day the waiting time increases as it searching through more and more files.

 

Is there a setting that can be applied, or an approach to tell the query editor to only look for new files, as the previously loaded files will not change as they will be historical data. This way the refresh times will be fast as it will only be looking for a days or 2 days worth of data, rather than 3 months worth, or by the end of year, 12 months worth..

5 Replies

@cchiappazzo could you solve this? I have the same problem. Regards

@hwoe1 

I guess incremental refresh works only in Power BI Premium, not in Excel.

@cchiappazzo 

I was just researching this question. As you apparently know, there is a dedicated feature in PowerBI for incremental refresh.

.

There is not a similar feature in Excel PowerQuery.  But I did find a couple of articles that suggest manual work arounds

Incremental data loads in Microsoft Power Query for Excel (DIY incremental) 2015 05 15
https://www.oraylis.de/blog/incremental-data-loads-in-microsoft-power-query-for-excel

How to create a Load History or Load Log in Power Query or Power BI      2016 11 19
https://www.thebiccountant.com/2016/02/09/how-to-create-a-load-history-or-load-log-in-power-query-or...

@ron S. 

IMHO, that's more about incremental data appending, not about incremental refresh

In the interest of keeping the solution with the question in case those websites are no longer accessible, the basic premise of incremental updates in the Excel for Windows version of Power Query is as follows:

 

  1. Create your initial PQ with the transforms that you want and output to an Excel Table. 
  2. Create a second PQ referencing the Excel Table you just created, and set it as “Connection Only”. This preserves the existing data.
  3. Return to edit the PQ you created in Step 1 by adding a step to append the output of the second PQ. This runs the first query to get more recent data, then appends the existing data that you preserved in Step 2 onto that result for the finished output.
  4. Add steps to the first query after that to de-dupe and sort the results, if necessary. This will obviously require some kind of unique identifier on each record (a unique date/time, an ‘id’ field, whatever provides some unique value). You can always remove that unique id column after de-duping as a last step if you don’t want to see it in the final output.

If you are using Excel for Mac, while it now supports refreshing PQs and creating PQs using VBA, you’ll find it’s a whole lot easier to build a PQ with the Windows version of Excel. Good reason for even the most die-hard Apple fan-boy to keep a Windows laptop or PC around, even if it’s a cheap $200  netbook or a small VM, as long as it can run Office.