We’re making changes to search in SharePoint Online

[Bill Baer]

Last month we announced improvements coming to search in SharePoint Online (https://techcommunity.microsoft.com/t5/microsoft-search-blog/we-re-improving-search-usage-reports-in...).  As we continue to innovate across Microsoft Search to bring you a better, more precise search experience – we’re making some changes and improvements to search in SharePoint Online.

No matter where people work or what kind of device they use, they need the ability to quickly and easily find the information that will help them be more productive. As part of our continued effort to bring Microsoft Search to all your favorite productivity apps and services we’re making changes and improvements to several classic search experiences in SharePoint.

NOTE These changes are not applicable to Government and/or Sovereign clouds.  

Changes and improvements to the query experience in SharePoint Online and OneDrive

Deprecation of relevance tuning via ranking models

We’re making changes to the handling of ranking models for search in Microsoft 365 (see https://docs.microsoft.com/sharepoint/dev/general-development/customizing-ranking-models-to-improve-...).

Ranking models are a SharePoint Server feature that have been supported in Microsoft 365 (SharePoint Online) to a limited extent for backwards compatibility.

With several search relevance improvements this feature in Microsoft 365 has become less applicable and may in certain cases hurt relevance rather than improving it. Certain low-level elements of the SharePoint Search ranking model have become obsolete due to other changes to the Microsoft 365 search core. This change is also a part of a larger shift towards machine learning based relevance tuning of search.

The change may affect search results if you

• have configured a Classic search web part to use a non-default ranking model.
• use the CSOM/REST search API and specify a ranking model in the query. See also https://docs.microsoft.com/dotnet/api/microsoft.sharepoint.client.search.query.query.rankingmodelid?....
• use the Ranking Model Tuning app. See also https://appsource.microsoft.com/product/office/wa104192565?tab=overview.
  • This app has been possible to use also with Microsoft 365 search, but it is generally hard to maintain hand-tuned rank models in this way.
• use the open-source query tool. See also https://github.com/pnp/PnP-Tools/releases. This tool also has an option to return a low-level rank detail info for diagnostics purposes.

Earlier this year, a change was applied to search queries which override the selection of a non-default rank model in queries. The change results in queries falling back to using the default rank model to take advantage of recent relevance improvements in SharePoint Online.

Upcoming change: We will stop supporting the ranking model tuning app. This app will no longer work in Microsoft 365 from February 2021.

We do not expect this change to impact your search application unless you have applied very deliberate changes in a ranking model via low-level tuning of a model. In this case you may need to consider if this tuning is no longer relevant, or may alternatively be achieved by using the XRANK feature.

Searching private content in Microsoft 365 OneDrive for Business

In Microsoft 365 enterprise search experiences, the search results include:

• All SharePoint site collection content you have read access (or stronger) to, including site collections associated with groups and teams.
• Your own OneDrive for Business content, including content that you have not shared with anyone (private).
• OneDrive for Business content that has been shared with you.

This applies to all search experiences scoped to SharePoint/OneDrive, including native OneDrive for Business web and mobile search experiences.

Your privacy is important to us, and while we work to earn your trust every day, we discover areas of improvement where we can align our service with the users’ expectations of privacy. We are therefore rolling out a change in Microsoft 365 Search to ensure search results are consistent with what users perceive as private vs. shared content. We will do so by adapting the definition of private which is already in use by OneDrive, resulting in a consistent search experience with the users’ privacy at the center.

How will I see this change?

This change will only be visible in the search results for users who have been granted site collection admin privileges (directly or via a security group) to a OneDrive for Business site collection that they are not the owner of, through governance policies or other changes in ownership.

Prior to this change, OneDrive content that was not explicitly shared with you, but where access was granted through governance policies or changes in ownership, could be returned in search results. With this change, OneDrive content will not be returned in search results unless it has been shared with you through any Microsoft 365 sharing mechanism.

Full details of the scenarios associated with this change

An important aspect of this change is to ensure a consistent definition of ‘Private OneDrive content’ that is in line with the user’s expectation of privacy. In the OneDrive for Business Web experience today, the ‘Sharing’ column reflects the definition of private OneDrive content which we will adapt in search.

‘Sharing’ in this context means explicit sharing of items or folders within the OneDrive, or other low-level means to change the access rights for sites, folders, or items within the OneDrive. We know that customers may be concerned about perceived over-sharing via search, and thus we want to ensure that the content returned in search results are consistent with what users perceive as private vs. shared.

A scenario that is not defined as ‘sharing’ in this context is when someone else than the OneDrive owner is defined as Site Collection Administrator for the OneDrive site collection in addition to the OneDrive owner. This could be another user within the Microsoft 365 subscription or a security group. In SharePoint, a site collection admin will always have full access to the content of the associated site collection. With this change, if a user has read access to content in another user’s OneDrive via being Site Collection Administrator only, then the content is still not surfaced in search experiences.

Prior to this change, search results did reflect any read access to content, i.e., you would be able to see other user’s private OneDrive content even if the user has not shared the content with you if you had been assigned the site collection admin role on the site collection.

These are the most relevant scenarios where this change may have an impact on search results:

1. Some customers may have a governance policy where they add a security group (e.g., some special security group for privileged tenant admin roles) as an additional site collection admin on all site collections, including OneDrive for Business site collections).

o   The target scenario is normally not a search scenario, but other reasons why it must be possible for some privileged tenant admin to access specific content regardless of access rights.

o   This is not a recommended pattern, but we have understood that certain customers have deployed this pattern using SharePoint scripting.

o   If a privileged admin that is member of such a security group expects to be able to search in all content, this has previously worked in SharePoint Online search, but will no longer return private content based on the new definition.

2. A user has left the company, and the OneDrive for Business contains content that may be relevant for the organization. In this case a manager or some other appointed peer may be added as site collection admin to the ODB to have access to the content of the OneDrive.
3. A user changes employee identity (e.g., from vendor to full-time employee) where the user’s email alias changes, and the user may get a new OneDrive for Business site associated with the new identity. In this case the user (with the new identity) may want to have full access to the old OneDrive site, potentially by being assigned as additional site collection administrator.

Mitigation

In all the scenarios listed above there may be legitimate reasons for a supplementary site collection admin to be able to search for private content within the OneDrive.

• For scenario 1, we recommend using eDiscovery if there is a specific need for search across all content including private content. This will also ensure that proper governance policies are followed and thus privacy concerns are being respected.
• For scenario 2 and 3, if a user has legitimate reasons to search for private content in such an ODB, the user (which have full access via being Site Collection Admin) may share the relevant content of the OneDrive with himself/herself, using normal sharing mechanisms on files or folders.

Updates to FQL (FAST Query Language)

As we continue to modernize Microsoft Search, we’ll retire some elements of FQL described below in February 2021.

NOTE Deprecated FQL features will only imply a subtle difference in relevance ranking of results from full-text query expressions, the queries will continue to function as before. The exception to this is the COUNT() operator (which has no known usage in SharePoint Online today), which will fail the query.

FAST Query Language (FQL) is a powerful query language that enables developers to perform exact searches and to narrow the scope of search to values that belong to a specific managed property or a full-text index. The FQL query language is only intended for programmatic query integration.

As part of this deprecation several operators related to FQL will be removed.  Beginning on February 1, 2021 the following FQL operators will be removed:

COUNT operator 

The COUNT operator In FQL specifies the of number query term occurrences an item must include for the item to be returned as a result. 

FILTER operator 

The FILTER operator in FQL is used to query metadata or other structured data.  Once this operator is retired, the FILTER operator will be ignored.  This change will not impact the user experience; however, ranking of results may change.

Dynamic rank ‘weight’ parameter to the ‘string’ operator 

Enables custom ranking where the expressions enclosed in the affected string() operator will get a different rank. This change will not impact the user experience; however, ranking of results may change. 

Per string configuration of linguistics on/off 

Enables linguistics control where stemming is not applied to the expressions enclosed in the affected string() operator.  This change will not impact the user experience; however, ranking of results may change.

Per string configuration of wildcard on/off 

Enables wildcard expansion control where the expected behavior is that when set to off then any wildcard character in the string must be treated as a character.  This change will not impact the user experience; however, ranking of results may change.

FQL dynamic rank difference between OR and ANY 

The ANY operator, is like the OR operator except that the dynamic rank (the relevance score in the result set.md) is affected by neither the number of operands that match nor the distance between the terms in the item. The OR operator in FQL is returns only items that match at least one of the OR operands. Items that match will get a higher dynamic rank if more of the OR operands match.  Once this operator is retired, the ANY operator will be implemented similarly to the OR operator.

We recommend, where applicable, using the default SharePoint query language, KQL where your business requirements can be similarly met.

KQL is the default query language for building search queries. Using KQL, you specify the search terms or property restrictions that are passed to the SharePoint search service.

Learn more about KQL query syntax at https://docs.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-synta....

Search Query Rules

Search Query Rules in SharePoint help support the users’ search intent, by creating pre-defined rules which apply to the user queries.  In a query rule, you specify conditions and correlated actions. When a query meets the set conditions, search performs the actions to improve the relevance of the search results. 

For example, you might specify a condition that checks whether the query matches a term in a SharePoint term set, or another condition that checks whether the query is frequently performed on a particular search vertical in your search system, such as Videos. 

Beginning on February 1^st, 2021 we’ll be removing the following conditions in SharePoint

• Context condition: Category 
• Context condition: User segment 
• Query condition: Taxonomy 

  • Found in query condition option "Query contains action term" -> "Action term is an entry in this dictionary".

    Found in query condition option "Advanced Query Text Match" -> "Query contains an entry in this dictionary". 

  • Found in query condition option “Advanced Query Text Match.” 
• Query condition: Common query 
• Query condition: Commonly clicked property 
• Query condition: Regular expression 

In addition to these changes, we’ll additionally deprecate the following out-of-the-box query rules that take a dependency on the conditions to be deprecated:  

• Location in People Search (depends on Taxonomy query condition) 
• Location in SharePoint Search (depends on Taxonomy query condition) 
  • Phone Number in People Search (depends on Regular expression query condition)  
  • Phone Number in SharePoint Search (depends on Regular expression query condition)  
  • Tags in People Search (depends on Regular expression query condition)  
  • Tags in Conversation Search (depends on Regular expression query condition)  
  • People Expertise Search 

NOTE These deprecations only apply to classic search experiences, they do not affect the modern search experiences.  

For scenarios in which you would like to promote a result above existing ranked results, Microsoft Search provides a set of Answers, both editorial and AI mined, that can be used in place of classic search functionality such as Best Bets and Promoted Results.

An Answer is a highly relevant and high confidence result that satisfies a user intent expressed as a query/question in search, presenting the most relevant information needed to get a job done and help users to faster task completion.

An Answer is a way to address user intent. When searching, the user typically types in characters and keywords to express an intent. Recognizing the keywords that are triggers for specific intents is important, but it is even more important that the content that is shown in search satisfies the user intent.  

Answers are useful when you want to promote a search result to appear above ranked results. For example, for the query “sick leave”, you could specify a particular result, such as a link to a site that has a statement of company policy regarding time off work.  You can think of Answers as being navigational aids to assist employees in getting directions to the information that matters most to help them keep productive and informed.

In Microsoft Search, an Answer can come from a variety of sources.  Learn more about Microsoft Search Answers at https://blog.wbaer.net/2020/10/06/making-the-most-of-answers-in-microsoft-search/.

Changes and improvements to relevance in SharePoint Online

Changes to Authoritative Pages

Currently, as a global or SharePoint admin in Microsoft 365, you can influence the pages or documents that should appear at the top of your list of search results by identifying high-quality pages, also known as authoritative pages. Authoritative pages link to the most relevant information. A typical example of an authoritative page could be the home page of your company portal.  Beginning February 1, 2021, we’ll remove the ability to configure authoritative pages in SharePoint Online.

Like query rules, Answers in Microsoft Search can be implemented to influence specific sites, documents, and more to promote a result above ranked results.  Refer to the information above to learn more about Answers in Microsoft Search.

Improving Personal Favorites

Search is something we use every day, a lot, and it’s hard to keep track of what you last searched for now you’ll no longer need to search for what you’ve searched for adding a new option to view and manage your personal query history.

In SharePoint Online, personal favorites were used to display previous queries when a threshold was met, for example, if frequently searching for “Contoso Marketing Presentation”, this query would become a candidate to be displayed in search.  Beginning February 1, 2021, we’ll remove personal favorites and recommend personal query history in Microsoft Search.

With personal query history, you'll see your recent queries as you begin typing in the search box to help you get back to insights and information you recently used or accessed so you'll no longer have to bookmark your queries or memorize the right query to get you back to where you were. Your personal query history can be managed through your Office 365 My Account settings and new My Account privacy controls allow you to delete your query history or download your query history for future reference.

To learn more about Microsoft Search in SharePoint visit https://aka.ms/MicrosoftSearch/Ignite2020/Sessions/5002.  To watch related sessions from Microsoft Ignite visit https://techcommunity.microsoft.com/t5/microsoft-search-blog/microsoft-search-at-ignite-2020/ba-p/16....