Excel performance improvements now take seconds running Lookup-type functions
Published Sep 25 2018 06:00 AM 223K Views
Microsoft

This is our second wave of Excel Office 365 performance improvements after our first Sept ’17 one, to reinforce our commitment to fixing top impacting freezing, slow and not responding performance issues derived from Excel user feedback and usage learnings. This time around we have even more improvements than before starting with...

 

Lookup functions.png

Lookup functions VLOOKUP, HLOOKUP, and MATCH are one of the most used functions in Excel. If you use them with exact match to find items in a table or range in Excel, and see it noticeably take time in seconds or minutes or more, chances are you'll now see very noticeable improvement in the speed at which you see results.

 

 

We made them faster by more efficiently finding the item you are looking up. We create an index on-demand when you first search of a range of cells and then reuse it in subsequent lookups from the same range, until data changes in the lookup range. For details refer to our updated Excel Performance and limit improvements whitepaper.

 

Also, we have improved the speed of many use cases involving operations like copy/paste, undo, conditional formatting, cell editing, cell selection, scrolling, filtering, file open, and programmability, such as:

 

  • Copying the entire column in a sheet and selecting the clipboard is faster now because we optimized our large memory allocations with a more efficient data structure.
  • Undoing pasted cells with conditional formatting was slow because a paste operation was inefficient in generating an undo record for every cell with conditional formatting and with its own changed priority. It now combines them more efficiently into one undo record with changed priorities for all in it.
  • Undoing inserting a new row with a copied range with merged cells is now faster because we optimized handing of merged cells during an undo operation.
  • Deleting one or more rows with merged cells is now more performant because we skip redundant calls to render the visible grid in the midst of the deletion operation.
  • Viewing filter drop down for a column with lots of cells with conditional formatting is now faster because we optimized the filter drop down algorithm to be more efficient computing highlight, unique, and duplicate values.
  • Scrolling in the visible sheet, after an operation wasn’t optimal, because we re-rendered (or at least evaluated) all rows top to bottom (including filtered rows) in the visible sheet. This can be expensive depending on the number of rows and our ability to calculate for animation related rendering. Our optimization introduced in our first wave of performance improvements
  • Scrolling, editing of cells, cell selection, filtering operations in the grid are much faster when lots of rows are filtered / hidden because our rendering is more optimized to intelligently defer expensive rendering calls until we have calculated last row in the visible range.
  • Slow transitioning to the next cell after editing a cell adjacent to a large table or range of cells with data, because after editing a cell adjacent to a large table or range of cells with data, we would generate a preview of all cells in that column to Flash Fill, which is on by default. Depending on the size of the adjacent table or range of cells and whether the cells contain additional metadata like conditional formatting, this can be a time consuming and expensive operation. We now, limit scope of Flash Fill preview to the visible area improving Excel responsiveness. 
  • When opening any workbook we searched all existing ribbon content to ensure every component of the ribbon renders correctly. We have now made open faster by more intelligently searching in common cases like when the ribbon item being searched for doesn’t exist. Also, during open we now skip updating MRU links synchronously and instead update them asynchronously after open. Finally, we scope the rendering of the grid to the visible grid area when opening workbooks with lots of wrapped text in locales like Japanese.
  • Opening a local workbook when using a 3rd party anti-virus software is faster because we are more intelligent in avoiding searching for the 3rd party anti-virus vendor registered for scanning when a scan is determined to be redundant.
  • We open simple CSV text files much faster by employing a more efficient memory allocation.
  • Flashing scrollbars and slower running VBA code (than Excel 2010) even when ScreenUpdating property is set to false. We have addressed this by ensuring scrollbars will no longer update when ScreenUpdating property is set to false (just like in Excel 2010). This in turn diverts CPU cycles away from redundant rendering of the scrollbars and to running user VBA code, improving code execution speed.
  • Programmatically selecting a cell, a range of cells, or a sheet using VBA is faster because we avoid redundant calls for updating a document’s upload status when running VBA macro code performing operations like selecting a cell, a range of cells, or a sheet.

Office 365 can help you be even more productive.1 Give it a try today!


Now we know we are not done with this second wave and there is more, that said we would really like to hear from you to ensure our fixes align closely with the freezing, or slow, or not responding issues you experience. Hence please stay connected to Excel and its community, read Excel blog posts, send us a frown, and send us ideas and suggestions for the next version of Excel through UserVoice. You can also follow Excel on Facebook and Twitter.

 

Prash Shirolkar

Program Manager, Excel

 

1 Available in monthly channel version 1809 or later

35 Comments
Copper Contributor

Good news. One thing I'd still like to see in regards to lookups is to give us the option to do an exact match binary search, as per my comment at

http://dailydoseofexcel.com/archives/2015/04/23/how-much-faster-is-the-double-vlookup-trick/#comment...

 

I'd also be interested to hear about whether your performance enhancements extend to optimizing the deletion of rows within filtered tables. This is currently so slow on large Tables (i.e. minutes) that it is much faster to copy the rows you want to keep from a filtered Table, rather than trying to delete the rows you don't want. See https://www.excelcampus.com/tips/sort-data-before-deleting-rows/ for the kind of scenario I'm talking about here.

Copper Contributor

I'm using version 1808 on O365 and it seems to be working for me already

I compared 4 VLOOKUPs on 1,000,000 rows against a table with 18,000 rows

It took 12 CPU seconds using VLOOKUP and the same time using one column searching with MATCH and 4 columns using INDEX.

With the previously implementation the version using INDEX should be 4 times faster but now it takes the same time.

On Excel online the difference should be more dramatic because a large number of VLOOKUPs is a best case for multi threaded calculation which is not available in Excel online.

Copper Contributor

Excel hang upon opening after update to latest updates 10/10/2018. Version 1809 (Build 10827.20150)

Great work @Prash Shirolkar and team.  

Copper Contributor

This is great news! I'd like to know if there are any plans to include the "roles" functionality to Power Pivot, as it is in Power BI. If not, I've opened a suggestion on Excel's user voice.

Bronze Contributor

@Miguel Ulloa - how would roles work in Power Pivot? In Power BI, the service provides the authentication security to know what data to provide. There is no authentication needed for Excel. None possible in fact. You can get the currently logged in user with VBA code, but it is easy to bypass that by simply not running the VBA.

 

You can get roles in Excel by using Power BI as the source then connecting Excel to the Power BI data source. It is the "Analyze in Excel" feature. It adds a Power BI connector ribbon to Excel, and the addin has a login feature that the Power BI service authenticates.

Copper Contributor

@Ed Hansberry Thanks for your reply Ed. Both Power BI and Power Pivot include the username() function. In Power BI, I can use it in a DAX function that limits what the user sees, regardless of what's present in the file. How it works in in Power BI is exactly how I'd like it to work. If it needs the user to be authenticated, then even better, I'd say. This is on a O365 environment, so every user would be on the company network and properly authenticated. I realize the roles allow stuff like writing back to the model, but at the moment, it's only limiting what the user sees what I think it's missing.

 

In my research on this, I haven't seen your suggestion, I'll try to give it a shot and see what comes out of it.

Copper Contributor

@Ed Hansberry Is this enabled  by the "Power BI publisher" thing I just found?

Bronze Contributor

Partially @Miguel Ulloa - In your Power BI portal, click the download arrow in the upper right, then "Analyze in Excel Updates." It will guide you to a download that will install a set of files on your PC that will allow Excel to connect directly to your Power BI reports. It honors row level security in the service. A full overview is here.

Copper Contributor

Fast Match / Lookups are available only for the Monthly Channel (Targeted) - Build 1810 (11001.20074)

They are not available for Insider Build 1811 (11022.200000)

Intentional or oversight ?

 

Regards

Sam

Microsoft

Thanx for the comments folks! Keep them coming.

Looks like there may be 3 outstanding issues left. Responding in order of when the comments were posted...

  1. Thanx Jeffrey! Btw, unless I am missing something, our accelerated EXACT VLOOKUP search with Speedy Lookup applies for both sorted or unsorted data... and so a user shouldn't have to resort to specifying a new -1 option that you propose. Also, I was unable to reproduce the 8-10 second delay in latest Monthly channel; when deleting the rows with Color = Red in the unsorted sheet in the workbook attached to the article. Can you give it a try too with the latest monthly channel and LMK if it still repros?

  2. Thanx for confirming the speed-ups Dany! And thanx for the multi-threaded recalc in Online reminder. It is something on our TODO list.

  3. Thanx for reaching out Sameer! You are correct Insider Fast audience doesn't experience lookup function speed up. Sorry for the inconvenience. Please refer to details in article here


HTH
Prash

Copper Contributor

I have 4 type packing in my products 1 main packing and 3 sub packing

main pack is =Bag

2nd pack is =Pkt

3rd pack is =Strip

4th pack is =Pcs

i want formula for get result in all qty

in cell G4  QTY 1

in cell H4 QTY 2

in cell I4 QTY 3

in cell J4 QTY 4

in cell K4 QTY 5

in cell L4 =G4+H4+I4+J4+K4

in cell M4 is my main packing qty in pcs (pcs in main bag)

in cell N4 is my Sub packing (pcs in Pkt)

in cell O4 is my sub/sub packing (Strip in pkt)

in cell P4 is my sub/sub/sub Packing (pcs in Strip)

i know formula for 3 cell but i want formula for 4 cell

i used below formula now in 3 cell

D=TRIM(IF(L4>=N4,INT(L4/N4)&" Pkt ",""))

E=IF(MOD(L4,N4)>=P4,INT(MOD(L4,N4)/P4)& " Strip ","")

F==IF(MOD(L4,P4),INT(MOD(L4,P4))&" Pcs","")

i want forumal for cell C=

 

i.e

G4=270

H4=90

I4=540

J4=105

K4=30

L4=G4+H4+I4+J4+K4=1035

M4=600PCS

N4=40PCS

O4=4 STRIP

P4=10PCS

 

RESULT IS

C4=1 BAG

D4=10 PKT

E4=3 STRIP

F4=5PCS

 

I HAVE FORMULA FOR D4,E4,F4 BUT I DONT HAVE FORMULA TO GET RESULT IN C4

 

PLEASE GIVE ME FORMULA FOR CELL C4

 

 

Copper Contributor

Nice work @Prash Shirolkar. Thanks for sharing valuable information.

Copper Contributor

Is this available only on windows or on macOS as well?

Microsoft

Updates...

Hey Sameer - you should see the VLOOKUP speedup in Insider Fast too... please let me know if you do not. 

Thanx SynchronizerExcelCompareTool! 

Also Oscar - our plan is to enable for MacOS too. We are monitoring inner audience rings for MacOS currently... will keep you posted when we release to Production. 

 

thanx,

Prash

Copper Contributor

@Prash Shirolkar 

I am on Office Insider Ver 1904 Build 11504.20000

I don't see the speed improvements

 

Sam

Copper Contributor

Hi @Prash Shirolkar ,

Thanks for the information, was good to meet you last week. What I really like about this blog is the detail you have gone into explaining what the problem was and what you have done to make things faster.

Do you have any further targets for speed improvements?

Regards

Copper Contributor

Is there any plan to look into the efficiency of the cube formula functions like cubevalue(), cubemember(), cubeset()? These functions still write out mdx formulas (poorly) against tabular cubes and then split their query into sequential mdx queries that return a limited number of cells. Most analysis services servers can handle a larger load of queries, could likely send them in parallel, and preferably send dax statements to tabular cubes.

 

Would be great to hear what the plan is for these functions!

Silver Contributor

What about the old LOOKUP function?

Does it take advantage from this update!

Copper Contributor

The old LOOKUP function used a Binary search algorithm and was already blazing fast like the 1 or -1 options of MATCH or the 1 option of VLOOKUP

Copper Contributor

Hi, are these performance updates only available in Office 365, or can they be obtained via a volume license?

Copper Contributor

Hi, just as Excel 1902 is being rolled out in my organisation, I discovered a very nasty calculation bug which is having a major business impact.

 

This occurs when opening a file which was made with any previous Excel release (I tested it back to Excel 2010) up to Excel Version 1808 on Windows 10 is now opened using Excel 1902. Some cells using a LOOKUP formula are returning a #NA result straight after opening the file.

I also found a way to reproduce this in a new file / with new cells.

 

How to reproduce on a new file:

  1. Put value "ABC" in A1 and in A2
  2. Put the following formula in A3: "=LOOKUP(2,1/(A1:A2<>""),A1:A2)" --> the purpose of this formula is to repeat the last non blank item from the referred set of cells
  3. Drag this formula down from A3 down to A10
    1. You will see that all cells from A3 to A10 repeat the value "ABC"
  4. Clear the content of one cell in this list (A7 by instance)
    1. You will see that A8, A9, and A10 still repeat the value "ABC" (which is the purpose of the formula)
  5. Edit the formula of the cell below the emptied one (A8 in this case), just click in the formula bar, and press "Enter" without changing anything
  6. Force a workbook recalculation by pressing CTRL+ALT+SHIFT+F9
  7. Cell A8, A9 and A10 now will all carry a #N/A

 

This procedure allows to reproduce from scratch in a new workbook using Excel 1902.

 

However the criticality of this calculation bug is that when opening a workbook which was saved using 1808 (and make use of the formula described in step #2), while it worked fine with 1808, once opened using 1902 it throws errors.

 

Steel Contributor

@Sebastien_Spas shouldn't it throw an error? If A7 is empty, then your array above A6:A7 evaluates to 1/{TRUE;FALSE} and FALSE = 0, so you get a #DIV/0! error when evaluating it, which causes the entire formula to ultimately error out.

 

Seems to be the bug is it takes a CTRL-ALT-SHIFT-F9 to get the error to show up.

 

I am wondering if this is one of those obscure issues that the new dynamic array calcs is bubbling to the top. But not sure why you are seeing it in 1902 unless some of the Dynamic Array logic is there without the full functionality.

Copper Contributor

I don’t know about “should”, but this function worked without throwing errors in the past 10 years versions of Excel.

 

The issue is when opening field saves with previous versions of Excel where this did not throw an error, it now does after opening the file. On complex spreadsheets (this is my use case), it has a cascading effect, and due to several IFERROR functions in between returns wrong totals (when compared to all prior Excel versions).

Steel Contributor

You might post a link to your comment above in the dynamic array thread here. I suspect the two are related. There might be something pop up about implicit intersection, a topic I try to avoid discussing at all costs. :lol:

Copper Contributor

In a calculation program made by Leica Geosystems, I receive a CSV with the followind data: The space between number 11 835,7777 it is not possible to change to my excel numbers (2nd table below- numbers without any space in between)  and therefore calculations are not made

Anyone cam help me ??

A14311 835,77775 237,3752
A15311 838,84685 239,0157
A15111 841,43385 231,9890
A14211 842,79525 224,2548

 

V111806,62135213,5126-12,0446
V211808,11495210,8544-13,1251
V311805,60225215,5197-12,9434
Copper Contributor

Hello,

open the csv file, copy the Space between the numbers and replace them by nothing. Then it will be numbers and you can make calculations

 

 

 

Copper Contributor

Have you added Stockhistory?

Copper Contributor

在单元格格式对话框中:数字—特殊—大写,中能否在增加一个货币大写的功能呢?

Copper Contributor

Thanks for this, but come on guys.  Where are the new text formulas.  Why are they not rolling out.  You are very eager  for your MVP's to make Youtube videos on all of these new features, just for us, the paying customer to wait months (if not more than a year) for them to roll out?  And don't tell me to join the programme for these pre-releases.  Excel with Windows 11 is not even working properly, can you imagine using these pre-release versions.  Somewhere Microsoft is going off the rails.  Seems you are more interested in getting more customers than actually providing the actual services.  I think I am your biggest fan since day one (Excel) but I am seriously considering moving to Google or Apple.  Make a rule regarding new functions rolling out.  Make a rule that MVP's are not allowed to share them unless they will be rolled out within the next month.  You keep hanging carrots in front of the donkey and not delivering.  And please fix the issues with Windows 11.  Excel just stops responding unless you move the scroll bar and then it wakes up again.  I am seriously loosing interest in my Office 365 subscription  (and we have more than 200 in our company).

@Chris Botha , as a comment

1) How new functionality is rolling out is described here Overview of update channels for Microsoft 365 Apps - Deploy Office | Microsoft Docs and related links for insiders

2) MVP:s have no priority in receiving new functionality, they have it as any other Beta insiders. 

Copper Contributor

I know they using the Insider, my point is they go out and advertise these new functions all over the place and gets everyone excited and then the waiting game starts.....  wait......  wait.........  wait.....

My point is, do not allow them to start sharing these unless you guys are sure it will be rolling out to everyone sooner than later (and usually it is MUCH later)

For example Leila Gharani showed us the new text functions already three months ago on her channel and still nothing in the roll outs.

Why not? That all is published here Office Insider Blog and that not only MVP:s who share information about new Beta functionality appeared with their own samples. You may check LinkedIn, whatever.

Beta actually is for testing purposes. Based on people feedback and stats collected Microsoft could change announced functionality on that stage. That happened with recent text functions, with smooth scrolling, some other functionality. Not sure if everyone is ready to use in production functionality which could be changed in few months.

And the question who is "everyone". If wait for semi-annual it could take more than an year.

Copper Contributor

@Sergei Baklan 

The problem is more with the messaging.

For e.g. I am on the current channel (preview) previously (insider slow)

As per https://learn.microsoft.com/en-us/officeupdates/current-channel-preview searchable validation drop downs rolled in to this channel on 26th Jan 2023 via Version 2301 (Build 16026.20094). 

I am currently on the most updated build of 14th Feb 2023 - Version 2302 (Build 16130.20156) and I still do not have this feature rolled out.

This is what is frustrating and there is no point is hiding behind "Don’t have it yet? It’s probably us, not you"

Cheers

Sam

 

@sameer bhide , in general build number doesn't matter. It could include code for this or that functionality, but it is activated by channels and by waves within channel. That not at once for all users on the channel, even if they have proper build. Moreover, on insider channels functionality could be activated, in a while deactivated due to some issues and later on returned back with improved version. Exact activation schedule depends on many things, I don't think Microsoft will ever start sharing it. However, information like "we activated this functionality for 100% of channel users" could be useful. 

Version history
Last update:
‎Sep 25 2018 05:22 PM
Updated by: