Forum Discussion
cchiappazzo
Apr 11, 2019Copper Contributor
incremental refresh in query editor
I am using query editor in excel to pull raw data from multiple files within a folder that contains day to day data in it. I am experiencing a lengthy waiting period when refreshing the query, and ea...
Steven Buehler
Jun 25, 2021Copper Contributor
In the interest of keeping the solution with the question in case those websites are no longer accessible, the basic premise of incremental updates in the Excel for Windows version of Power Query is as follows:
- Create your initial PQ with the transforms that you want and output to an Excel Table.
- Create a second PQ referencing the Excel Table you just created, and set it as “Connection Only”. This preserves the existing data.
- Return to edit the PQ you created in Step 1 by adding a step to append the output of the second PQ. This runs the first query to get more recent data, then appends the existing data that you preserved in Step 2 onto that result for the finished output.
- Add steps to the first query after that to de-dupe and sort the results, if necessary. This will obviously require some kind of unique identifier on each record (a unique date/time, an ‘id’ field, whatever provides some unique value). You can always remove that unique id column after de-duping as a last step if you don’t want to see it in the final output.
If you are using Excel for Mac, while it now supports refreshing PQs and creating PQs using VBA, you’ll find it’s a whole lot easier to build a PQ with the Windows version of Excel. Good reason for even the most die-hard Apple fan-boy to keep a Windows laptop or PC around, even if it’s a cheap $200 netbook or a small VM, as long as it can run Office.