Forum Discussion
JCouture17
Jun 12, 2023Copper Contributor
Very slow Power Query refresh
Hey everyone, first-time poster here! I am having some issues with a Query refresh that we have to do on a weekly basis. It basically goes into a Sharepoint folder, opens and merges all of the files...
Lorenzo
Jun 13, 2023Silver Contributor
Context differs so don't draw conclusions from what's below. It's just to give you a sense of some refresh times
PC Specs: Core i7 / SSD / 8Gb / Win11 / MSFT 365-64Bits
Created following 2 Tables in an Excel workbook, each 1 Million rows. Required columns only. All [Serial #] match:
Query1 - Merge tables + Expand
let
Source = Table.NestedJoin(
SharePoint, {"Serial #"},
CAPScrapWeekly, {"Serial #"},
"CAPScrapWeekly", JoinKind.LeftOuter
),
Expanded = Table.ExpandTableColumn(Source, "CAPScrapWeekly",
{"CQ", "Prod Date"}, {"CQ", "CAPScrapWeekly.Prod Date"}
)
in
Expanded
Query2 - Same as above + your extra steps
let
Source = Table.NestedJoin(
SharePoint, {"Serial #"},
CAPScrapWeekly, {"Serial #"},
"CAPScrapWeekly", JoinKind.LeftOuter
),
Expanded = Table.ExpandTableColumn(Source, "CAPScrapWeekly",
{"CQ", "Prod Date"}, {"CQ", "CAPScrapWeekly.Prod Date"}
),
AddedProdDate = Table.AddColumn(Expanded, "Prod Date", each
if DateTime.Time([Datetime]) < Time.From("07:00")
then DateTime.Date(Date.AddDays([Datetime],-1))
else DateTime.Date([Datetime]),
type date
),
AddedWeekNum = Table.AddColumn(AddedProdDate, "WeekNum", each
Text.PadStart(Text.From(Date.WeekOfYear([Prod Date])),2,"0"),
type text
),
ReplacedNullCQ = Table.ReplaceValue(AddedWeekNum,null,"-",Replacer.ReplaceValue,{"CQ"}),
ReplacedProdDate = Table.ReplaceValue(ReplacedNullCQ,
each [CAPScrapWeekly.Prod Date],
each [CQ],
(a,b,c) as date => if c <> "-" then b else a,
{"Prod Date"}
),
RemovedCapsProdDate = Table.RemoveColumns(ReplacedProdDate,{"CAPScrapWeekly.Prod Date"}),
AddedMonth = Table.AddColumn(RemovedCapsProdDate, "Month", each Date.MonthName([Prod Date]), type text),
AddedDateShift = Table.AddColumn(AddedMonth, "Date-Shift", each
if DateTime.Time([Datetime])>=Time.From("7:00") and DateTime.Time([Datetime])<Time.From("19:00")
then Date.ToText([Prod Date],"yyyy-MM-dd") & "-" & "D"
else Date.ToText([Prod Date],"yyyy-MM-dd") & "-" & "N",
type text
)
in
AddedDateShift
Hope this helps a bit