Home
Microsoft

Excel performance improvements now take seconds running Lookup-type functions

This is our second wave of Excel Office 365 performance improvements after our first Sept ’17 one, to reinforce our commitment to fixing top impacting freezing, slow and not responding performance issues derived from Excel user feedback and usage learnings. This time around we have even more improvements than before starting with...

 

Lookup functions.png

Lookup functions VLOOKUP, HLOOKUP, and MATCH are one of the most used functions in Excel. If you use them with exact match to find items in a table or range in Excel, and see it noticeably take time in seconds or minutes or more, chances are you'll now see very noticeable improvement in the speed at which you see results.

 

 

We made them faster by more efficiently finding the item you are looking up. We create an index on-demand when you first search of a range of cells and then reuse it in subsequent lookups from the same range, until data changes in the lookup range. For details refer to our updated Excel Performance and limit improvements whitepaper.

 

Also, we have improved the speed of many use cases involving operations like copy/paste, undo, conditional formatting, cell editing, cell selection, scrolling, filtering, file open, and programmability, such as:

 

  • Copying the entire column in a sheet and selecting the clipboard is faster now because we optimized our large memory allocations with a more efficient data structure.
  • Undoing pasted cells with conditional formatting was slow because a paste operation was inefficient in generating an undo record for every cell with conditional formatting and with its own changed priority. It now combines them more efficiently into one undo record with changed priorities for all in it.
  • Undoing inserting a new row with a copied range with merged cells is now faster because we optimized handing of merged cells during an undo operation.
  • Deleting one or more rows with merged cells is now more performant because we skip redundant calls to render the visible grid in the midst of the deletion operation.
  • Viewing filter drop down for a column with lots of cells with conditional formatting is now faster because we optimized the filter drop down algorithm to be more efficient computing highlight, unique, and duplicate values.
  • Scrolling in the visible sheet, after an operation wasn’t optimal, because we re-rendered (or at least evaluated) all rows top to bottom (including filtered rows) in the visible sheet. This can be expensive depending on the number of rows and our ability to calculate for animation related rendering. Our optimization introduced in our first wave of performance improvements
  • Scrolling, editing of cells, cell selection, filtering operations in the grid are much faster when lots of rows are filtered / hidden because our rendering is more optimized to intelligently defer expensive rendering calls until we have calculated last row in the visible range.
  • Slow transitioning to the next cell after editing a cell adjacent to a large table or range of cells with data, because after editing a cell adjacent to a large table or range of cells with data, we would generate a preview of all cells in that column to Flash Fill, which is on by default. Depending on the size of the adjacent table or range of cells and whether the cells contain additional metadata like conditional formatting, this can be a time consuming and expensive operation. We now, limit scope of Flash Fill preview to the visible area improving Excel responsiveness. 
  • When opening any workbook we searched all existing ribbon content to ensure every component of the ribbon renders correctly. We have now made open faster by more intelligently searching in common cases like when the ribbon item being searched for doesn’t exist. Also, during open we now skip updating MRU links synchronously and instead update them asynchronously after open. Finally, we scope the rendering of the grid to the visible grid area when opening workbooks with lots of wrapped text in locales like Japanese.
  • Opening a local workbook when using a 3rd party anti-virus software is faster because we are more intelligent in avoiding searching for the 3rd party anti-virus vendor registered for scanning when a scan is determined to be redundant.
  • We open simple CSV text files much faster by employing a more efficient memory allocation.
  • Flashing scrollbars and slower running VBA code (than Excel 2010) even when ScreenUpdating property is set to false. We have addressed this by ensuring scrollbars will no longer update when ScreenUpdating property is set to false (just like in Excel 2010). This in turn diverts CPU cycles away from redundant rendering of the scrollbars and to running user VBA code, improving code execution speed.
  • Programmatically selecting a cell, a range of cells, or a sheet using VBA is faster because we avoid redundant calls for updating a document’s upload status when running VBA macro code performing operations like selecting a cell, a range of cells, or a sheet.

Office 365 can help you be even more productive.1 Give it a try today!


Now we know we are not done with this second wave and there is more, that said we would really like to hear from you to ensure our fixes align closely with the freezing, or slow, or not responding issues you experience. Hence please stay connected to Excel and its community, read Excel blog posts, send us a frown, and send us ideas and suggestions for the next version of Excel through UserVoice. You can also follow Excel on Facebook and Twitter.

 

Prash Shirolkar

Program Manager, Excel

 

1 Available in monthly channel version 1809 or later

11 Comments
Occasional Visitor

Good news. One thing I'd still like to see in regards to lookups is to give us the option to do an exact match binary search, as per my comment at

http://dailydoseofexcel.com/archives/2015/04/23/how-much-faster-is-the-double-vlookup-trick/#comment...

 

I'd also be interested to hear about whether your performance enhancements extend to optimizing the deletion of rows within filtered tables. This is currently so slow on large Tables (i.e. minutes) that it is much faster to copy the rows you want to keep from a filtered Table, rather than trying to delete the rows you don't want. See https://www.excelcampus.com/tips/sort-data-before-deleting-rows/ for the kind of scenario I'm talking about here.

Occasional Visitor

I'm using version 1808 on O365 and it seems to be working for me already

I compared 4 VLOOKUPs on 1,000,000 rows against a table with 18,000 rows

It took 12 CPU seconds using VLOOKUP and the same time using one column searching with MATCH and 4 columns using INDEX.

With the previously implementation the version using INDEX should be 4 times faster but now it takes the same time.

On Excel online the difference should be more dramatic because a large number of VLOOKUPs is a best case for multi threaded calculation which is not available in Excel online.

Frequent Visitor

Excel hang upon opening after update to latest updates 10/10/2018. Version 1809 (Build 10827.20150)

Great work @Prash Shirolkar and team.  

Occasional Visitor

This is great news! I'd like to know if there are any plans to include the "roles" functionality to Power Pivot, as it is in Power BI. If not, I've opened a suggestion on Excel's user voice.

Regular Contributor

@Miguel Ulloa - how would roles work in Power Pivot? In Power BI, the service provides the authentication security to know what data to provide. There is no authentication needed for Excel. None possible in fact. You can get the currently logged in user with VBA code, but it is easy to bypass that by simply not running the VBA.

 

You can get roles in Excel by using Power BI as the source then connecting Excel to the Power BI data source. It is the "Analyze in Excel" feature. It adds a Power BI connector ribbon to Excel, and the addin has a login feature that the Power BI service authenticates.

Occasional Visitor

@Ed Hansberry Thanks for your reply Ed. Both Power BI and Power Pivot include the username() function. In Power BI, I can use it in a DAX function that limits what the user sees, regardless of what's present in the file. How it works in in Power BI is exactly how I'd like it to work. If it needs the user to be authenticated, then even better, I'd say. This is on a O365 environment, so every user would be on the company network and properly authenticated. I realize the roles allow stuff like writing back to the model, but at the moment, it's only limiting what the user sees what I think it's missing.

 

In my research on this, I haven't seen your suggestion, I'll try to give it a shot and see what comes out of it.

Occasional Visitor

@Ed Hansberry Is this enabled  by the "Power BI publisher" thing I just found?

Regular Contributor

Partially @Miguel Ulloa - In your Power BI portal, click the download arrow in the upper right, then "Analyze in Excel Updates." It will guide you to a download that will install a set of files on your PC that will allow Excel to connect directly to your Power BI reports. It honors row level security in the service. A full overview is here.

Regular Visitor

Fast Match / Lookups are available only for the Monthly Channel (Targeted) - Build 1810 (11001.20074)

They are not available for Insider Build 1811 (11022.200000)

Intentional or oversight ?

 

Regards

Sam

Microsoft

Thanx for the comments folks! Keep them coming.

Looks like there may be 3 outstanding issues left. Responding in order of when the comments were posted...

  1. Thanx Jeffrey! Btw, unless I am missing something, our accelerated EXACT VLOOKUP search with Speedy Lookup applies for both sorted or unsorted data... and so a user shouldn't have to resort to specifying a new -1 option that you propose. Also, I was unable to reproduce the 8-10 second delay in latest Monthly channel; when deleting the rows with Color = Red in the unsorted sheet in the workbook attached to the article. Can you give it a try too with the latest monthly channel and LMK if it still repros?

  2. Thanx for confirming the speed-ups Dany! And thanx for the multi-threaded recalc in Online reminder. It is something on our TODO list.

  3. Thanx for reaching out Sameer! You are correct Insider Fast audience doesn't experience lookup function speed up. Sorry for the inconvenience. Please refer to details in article here


HTH
Prash