Forum Discussion
JCouture17
Jun 12, 2023Copper Contributor
Very slow Power Query refresh
Hey everyone, first-time poster here! I am having some issues with a Query refresh that we have to do on a weekly basis. It basically goes into a Sharepoint folder, opens and merges all of the files...
Lorenzo
Jun 12, 2023Silver Contributor
Hi JCouture17
Logically, if time to refresh increases every week this means you have at least one table that grows every week... Generic recommendation: get rid of the rows & columns you don't need as early as possible (cf. Curt H. keep things in "table space")
If you Buffer your tables presumably you already tested perf. with and w/o buffering. Otherwise be aware that buffering large tables can be counterproductive (cf. Curt H. in this post)
- Optimising The Performance Of Power Query Merges In Power BI, Part 1: Removing Columns
- Optimising The Performance Of Power Query Merges In Power BI, Part 2: Does When You Remove Columns Matter?
- Optimising The Performance Of Power Query Merges In Power BI, Part 3: Table.Join And SortMerge
- Optimising The Performance Of Power Query Merges In Power BI, Part 4: Table.Join And Other Join Algorithms
- Speed/Performance aspects
As a side note, very tiny possible optimization. You can replace:
#"Replaced Value1" = Table.ReplaceValue(#"Replaced Value",
each [Prod Date],
each if [CQ] <> "-" then [CAPScrapWeekly.Prod Date] else [Prod Date],
Replacer.ReplaceValue,{"Prod Date"}
),
#"Changed Type4" = Table.TransformColumnTypes(#"Replaced Value1",{{"Prod Date", type date}}),
#"Removed Columns1" = Table.RemoveColumns(#"Changed Type4",{"CAPScrapWeekly.Prod Date"}),
with
#"Replaced Value1" = Table.ReplaceValue(#"Replaced Value",
each [CAPScrapWeekly.Prod Date],
each [CQ],
(a,b,c) as date => if c <> "-" then b else a,
{"Prod Date"}
),
#"Removed Columns1" = Table.RemoveColumns(#"Replaced Value1",{"CAPScrapWeekly.Prod Date"}),