Scenario:
When searching for an item in a large list/library using the "Find a file/item" the search results do not appear to return all files/items that are expected. This will generally be seen in lists and libraries that have exceeded the list view threshold.
How does the "Find a file/Item" work?
- The user enters a string to search for in the "Find an item".
- The list control gets that query string, and sends it to the SearchAdapter.
- The SearchAdapter now has the query string and the list that requested the search. It goes into that list (the ENTIRETY of the list) and finds the list item IDs that correspond to that query string, and returns the first 500 (60 if the list exceeds the list view threshold) that match the query string (regardless of view/folder etc).
- The list takes those 500 (60 if the list exceeds the list view threshold) results and adds them to a CAML query which contains the original view information.
- The list submits the CAML query and can result in only a subset of the list item IDs that are in that view and displays those to the user.
NOTE: The 500 (60 if the list exceeds the list view threshold) results that are returned are not affected by current view. Instead they are the first 500 (60 if the list exceeds the list view threshold) that the query finds within the search index of the list/library.
In short, the find a file/item will search through all folders and items and return the first 500 items (60 if the list exceeds the list view threshold) within the search index of the library that meet the search criteria, then apply the view filter which can result in only a subset of items/files being returned.
Real world Scenario:
Tailspin Toys has a large document library (5,000+ items) that keeps track of local vendors and what items need to be ordered. These items are also organized by year that they are ordered within folders. They require that they have certain metadata on each of these documents that track the city in which these items are located. See below for scenario visual.
A user is working within the 2013 folder and reviewing order forms. The 2013 folder has around 2,000 items. The user wants to search these documents by the city metadata column, so they type in "New York" into the find a file search box. The user notices a couple things:
- The user knows that there are at least 1,000 items within this folder that have the "New York" metadata tag yet it is only returning 500 that have this tag.
- They also notice that some of these items are from other folders.
Why did this happen?
When the search query is run for these items, SharePoint will run the query on the entire document library and will not distinguish documents that are only in the 2013 folder that the user is working in. Since the CAML query view has a hard limit of 500 items (60 if the list exceeds the list view threshold) that it will return that meet the "New York" criteria, it only returned the first 500 (60 if the list exceeds the list view threshold) that it found within the document library.
How to determine if you are running into this limitation or if content isn't available in the index?
Search in the "Find an item: search using the list item id.
If you can identify one of the items in the library that is not showing when running a find an item search, you can do a quick test by searching for the list ID of the item. Here are some instructions for doing this.
- You will need to first modify the view of the list/library to show the ID. Simply click on library settings > Modify view > Check the ID field.
- You will then see the ID of the document as you can see below:
- Next, you will want to run a search for that ID managed property by typing in the Find an Item Search using Listitemid:(x)
If the item shows, that means that when you are doing the initial query, you are hitting the 500 item limitation. If it still does not show, that means that the item may not be available in the index. More information on re-indexing a list/library can be found here.
Search for one of the files that you are expecting from the search for the site collection.
You can also search for the document within the search center. This can be search box can be located on the top right side of the document library shown below:
Note: If you are doing a generic search from the Site search center there are instances that the results you are expecting to see are not returned which could be expected behavior depending on configuration or dynamic duplicate trimming.
What we recommend if you are hitting this limit?
Divide the content into multiple libraries
- By dividing the content and reducing the item count within a single library, you reduce the chance of reaching the 500 item (60 if the list exceeds the list view threshold) find a file search limit.
- Pros: Usability and management of the list is easier long term
- Cons: Initial effort needed to divide the content into separate list and libraries.
- More information on managing large lists and libraries in SharePoint can be found here
Utilize Enterprise search.
- Enterprise search utilizes a type of query that does not have the same limitation as the Find a File search query. This would prevent you from having to separate your content and is the Best Practice recommendation for searching within SharePoint Online if you are hitting these limits.
- Pros: Will return more than 500 items/files.
- Cons: Search results are not limited to the list/library so additional search knowledge is needed to isolate the results to the areas needed.
- We are in the process of creating this in a separate blog post. We will post it here when completed.
Create search web parts configured to search the document library in question.
- If you require a custom search option that allows you to query the document library in question while also bypassing the 500-item limit (60 items if the list is over the list view threshold), creating a custom site and putting a search web part on it that is sourced to the library in question may be a viable option.
- More information on search and search web parts can be found here.
- We are also in the process of creating a separate blog post on how to do this. We will update this article to include a link when completed.
Note: This same process applies to modern list and library search however there are additional filters that are applied to the query sent to the SearchAdapter that will change the result set returned verses classic search.
Contributors: Tony Radermacher[MSFT] and Tania Menice[MSFT]
-TNT