We’re making changes to search in SharePoint Online
Published Dec 09 2020 05:47 AM 42.1K Views
Microsoft

Last month we announced improvements coming to search in SharePoint Online (https://techcommunity.microsoft.com/t5/microsoft-search-blog/we-re-improving-search-usage-reports-in...).  As we continue to innovate across Microsoft Search to bring you a better, more precise search experience – we’re making some changes and improvements to search in SharePoint Online.

 

No matter where people work or what kind of device they use, they need the ability to quickly and easily find the information that will help them be more productive. As part of our continued effort to bring Microsoft Search to all your favorite productivity apps and services we’re making changes and improvements to several classic search experiences in SharePoint.

 

NOTE These changes are not applicable to Government and/or Sovereign clouds.  

 

Changes and improvements to the query experience in SharePoint Online and OneDrive

Deprecation of relevance tuning via ranking models

We’re making changes to the handling of ranking models for search in Microsoft 365 (see https://docs.microsoft.com/sharepoint/dev/general-development/customizing-ranking-models-to-improve-...).

 

Ranking models are a SharePoint Server feature that have been supported in Microsoft 365 (SharePoint Online) to a limited extent for backwards compatibility.

 

With several search relevance improvements this feature in Microsoft 365 has become less applicable and may in certain cases hurt relevance rather than improving it. Certain low-level elements of the SharePoint Search ranking model have become obsolete due to other changes to the Microsoft 365 search core. This change is also a part of a larger shift towards machine learning based relevance tuning of search.

 

The change may affect search results if you

 

Earlier this year, a change was applied to search queries which override the selection of a non-default rank model in queries. The change results in queries falling back to using the default rank model to take advantage of recent relevance improvements in SharePoint Online.

 

Upcoming change: We will stop supporting the ranking model tuning app. This app will no longer work in Microsoft 365 from February 2021.

 

We do not expect this change to impact your search application unless you have applied very deliberate changes in a ranking model via low-level tuning of a model. In this case you may need to consider if this tuning is no longer relevant, or may alternatively be achieved by using the XRANK feature.

Searching private content in Microsoft 365 OneDrive for Business

In Microsoft 365 enterprise search experiences, the search results include:

 

  • All SharePoint site collection content you have read access (or stronger) to, including site collections associated with groups and teams.
  • Your own OneDrive for Business content, including content that you have not shared with anyone (private).
  • OneDrive for Business content that has been shared with you.

This applies to all search experiences scoped to SharePoint/OneDrive, including native OneDrive for Business web and mobile search experiences.

 

Your privacy is important to us, and while we work to earn your trust every day, we discover areas of improvement where we can align our service with the users’ expectations of privacy. We are therefore rolling out a change in Microsoft 365 Search to ensure search results are consistent with what users perceive as private vs. shared content. We will do so by adapting the definition of private which is already in use by OneDrive, resulting in a consistent search experience with the users’ privacy at the center.

 

How will I see this change?

This change will only be visible in the search results for users who have been granted site collection admin privileges (directly or via a security group) to a OneDrive for Business site collection that they are not the owner of, through governance policies or other changes in ownership.

 

Prior to this change, OneDrive content that was not explicitly shared with you, but where access was granted through governance policies or changes in ownership, could be returned in search results. With this change, OneDrive content will not be returned in search results unless it has been shared with you through any Microsoft 365 sharing mechanism.

 

Full details of the scenarios associated with this change

An important aspect of this change is to ensure a consistent definition of ‘Private OneDrive content’ that is in line with the user’s expectation of privacy. In the OneDrive for Business Web experience today, the ‘Sharing’ column reflects the definition of private OneDrive content which we will adapt in search.

 

‘Sharing’ in this context means explicit sharing of items or folders within the OneDrive, or other low-level means to change the access rights for sites, folders, or items within the OneDrive. We know that customers may be concerned about perceived over-sharing via search, and thus we want to ensure that the content returned in search results are consistent with what users perceive as private vs. shared.

 

A scenario that is not defined as ‘sharing’ in this context is when someone else than the OneDrive owner is defined as Site Collection Administrator for the OneDrive site collection in addition to the OneDrive owner. This could be another user within the Microsoft 365 subscription or a security group. In SharePoint, a site collection admin will always have full access to the content of the associated site collection. With this change, if a user has read access to content in another user’s OneDrive via being Site Collection Administrator only, then the content is still not surfaced in search experiences.

 

Prior to this change, search results did reflect any read access to content, i.e., you would be able to see other user’s private OneDrive content even if the user has not shared the content with you if you had been assigned the site collection admin role on the site collection.

 

These are the most relevant scenarios where this change may have an impact on search results:

  1. Some customers may have a governance policy where they add a security group (e.g., some special security group for privileged tenant admin roles) as an additional site collection admin on all site collections, including OneDrive for Business site collections).

o   The target scenario is normally not a search scenario, but other reasons why it must be possible for some privileged tenant admin to access specific content regardless of access rights.

o   This is not a recommended pattern, but we have understood that certain customers have deployed this pattern using SharePoint scripting.

o   If a privileged admin that is member of such a security group expects to be able to search in all content, this has previously worked in SharePoint Online search, but will no longer return private content based on the new definition.

  1. A user has left the company, and the OneDrive for Business contains content that may be relevant for the organization. In this case a manager or some other appointed peer may be added as site collection admin to the ODB to have access to the content of the OneDrive.
  2. A user changes employee identity (e.g., from vendor to full-time employee) where the user’s email alias changes, and the user may get a new OneDrive for Business site associated with the new identity. In this case the user (with the new identity) may want to have full access to the old OneDrive site, potentially by being assigned as additional site collection administrator.

 

Mitigation

In all the scenarios listed above there may be legitimate reasons for a supplementary site collection admin to be able to search for private content within the OneDrive.

  • For scenario 1, we recommend using eDiscovery if there is a specific need for search across all content including private content. This will also ensure that proper governance policies are followed and thus privacy concerns are being respected.
  • For scenario 2 and 3, if a user has legitimate reasons to search for private content in such an ODB, the user (which have full access via being Site Collection Admin) may share the relevant content of the OneDrive with himself/herself, using normal sharing mechanisms on files or folders.

Updates to FQL (FAST Query Language)

As we continue to modernize Microsoft Search, we’ll retire some elements of FQL described below in February 2021.

 

NOTE Deprecated FQL features will only imply a subtle difference in relevance ranking of results from full-text query expressions, the queries will continue to function as before. The exception to this is the COUNT() operator (which has no known usage in SharePoint Online today), which will fail the query.

 

FAST Query Language (FQL) is a powerful query language that enables developers to perform exact searches and to narrow the scope of search to values that belong to a specific managed property or a full-text index. The FQL query language is only intended for programmatic query integration.

 

As part of this deprecation several operators related to FQL will be removed.  Beginning on February 1, 2021 the following FQL operators will be removed:

 

COUNT operator 

The COUNT operator In FQL specifies the of number query term occurrences an item must include for the item to be returned as a result. 

 

FILTER operator 

The FILTER operator in FQL is used to query metadata or other structured data.  Once this operator is retired, the FILTER operator will be ignored.  This change will not impact the user experience; however, ranking of results may change.

 

Dynamic rank ‘weight’ parameter to the ‘string’ operator 

Enables custom ranking where the expressions enclosed in the affected string() operator will get a different rank. This change will not impact the user experience; however, ranking of results may change. 

Per string configuration of linguistics on/off 

Enables linguistics control where stemming is not applied to the expressions enclosed in the affected string() operator.  This change will not impact the user experience; however, ranking of results may change.

 

Per string configuration of wildcard on/off 

Enables wildcard expansion control where the expected behavior is that when set to off then any wildcard character in the string must be treated as a character.  This change will not impact the user experience; however, ranking of results may change.

 

FQL dynamic rank difference between OR and ANY 

The ANY operator, is like the OR operator except that the dynamic rank (the relevance score in the result set.md) is affected by neither the number of operands that match nor the distance between the terms in the item. The OR operator in FQL is returns only items that match at least one of the OR operands. Items that match will get a higher dynamic rank if more of the OR operands match.  Once this operator is retired, the ANY operator will be implemented similarly to the OR operator.

 

We recommend, where applicable, using the default SharePoint query language, KQL where your business requirements can be similarly met.

 

KQL is the default query language for building search queries. Using KQL, you specify the search terms or property restrictions that are passed to the SharePoint search service.

 

Learn more about KQL query syntax at https://docs.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-synta....

 

Search Query Rules

Search Query Rules in SharePoint help support the users’ search intent, by creating pre-defined rules which apply to the user queries.  In a query rule, you specify conditions and correlated actions. When a query meets the set conditions, search performs the actions to improve the relevance of the search results. 

 

For example, you might specify a condition that checks whether the query matches a term in a SharePoint term set, or another condition that checks whether the query is frequently performed on a particular search vertical in your search system, such as Videos. 

 

Beginning on February 1st, 2021 we’ll be removing the following conditions in SharePoint

 

  • Context condition: Category 
  • Context condition: User segment 
  • Query condition: Taxonomy 
    • Found in query condition option "Query contains action term" -> "Action term is an entry in this dictionary".

      Found in query condition option "Advanced Query Text Match" -> "Query contains an entry in this dictionary". 

    • Found in query condition option “Advanced Query Text Match.” 
  • Query condition: Common query 
  • Query condition: Commonly clicked property 
  • Query condition: Regular expression 

In addition to these changes, we’ll additionally deprecate the following out-of-the-box query rules that take a dependency on the conditions to be deprecated:  

 

  • Location in People Search (depends on Taxonomy query condition) 
  • Location in SharePoint Search (depends on Taxonomy query condition) 
    • Phone Number in People Search (depends on Regular expression query condition)  
    • Phone Number in SharePoint Search (depends on Regular expression query condition)  
    • Tags in People Search (depends on Regular expression query condition)  
    • Tags in Conversation Search (depends on Regular expression query condition)  
    • People Expertise Search 

NOTE These deprecations only apply to classic search experiences, they do not affect the modern search experiences.  

 

For scenarios in which you would like to promote a result above existing ranked results, Microsoft Search provides a set of Answers, both editorial and AI mined, that can be used in place of classic search functionality such as Best Bets and Promoted Results.

 

An Answer is a highly relevant and high confidence result that satisfies a user intent expressed as a query/question in search, presenting the most relevant information needed to get a job done and help users to faster task completion.

 

An Answer is a way to address user intent. When searching, the user typically types in characters and keywords to express an intent. Recognizing the keywords that are triggers for specific intents is important, but it is even more important that the content that is shown in search satisfies the user intent.  

Answers are useful when you want to promote a search result to appear above ranked results. For example, for the query “sick leave”, you could specify a particular result, such as a link to a site that has a statement of company policy regarding time off work.  You can think of Answers as being navigational aids to assist employees in getting directions to the information that matters most to help them keep productive and informed.

In Microsoft Search, an Answer can come from a variety of sources.  Learn more about Microsoft Search Answers at https://blog.wbaer.net/2020/10/06/making-the-most-of-answers-in-microsoft-search/.

 

Changes and improvements to relevance in SharePoint Online

 

Changes to Authoritative Pages

Currently, as a global or SharePoint admin in Microsoft 365, you can influence the pages or documents that should appear at the top of your list of search results by identifying high-quality pages, also known as authoritative pages. Authoritative pages link to the most relevant information. A typical example of an authoritative page could be the home page of your company portal.  Beginning February 1, 2021, we’ll remove the ability to configure authoritative pages in SharePoint Online.

 

Like query rules, Answers in Microsoft Search can be implemented to influence specific sites, documents, and more to promote a result above ranked results.  Refer to the information above to learn more about Answers in Microsoft Search.

 

Improving Personal Favorites

Search is something we use every day, a lot, and it’s hard to keep track of what you last searched for now you’ll no longer need to search for what you’ve searched for adding a new option to view and manage your personal query history.

 

In SharePoint Online, personal favorites were used to display previous queries when a threshold was met, for example, if frequently searching for “Contoso Marketing Presentation”, this query would become a candidate to be displayed in search.  Beginning February 1, 2021, we’ll remove personal favorites and recommend personal query history in Microsoft Search.

 

With personal query history, you'll see your recent queries as you begin typing in the search box to help you get back to insights and information you recently used or accessed so you'll no longer have to bookmark your queries or memorize the right query to get you back to where you were. Your personal query history can be managed through your Office 365 My Account settings and new My Account privacy controls allow you to delete your query history or download your query history for future reference.

 

To learn more about Microsoft Search in SharePoint visit https://aka.ms/MicrosoftSearch/Ignite2020/Sessions/5002.  To watch related sessions from Microsoft Ignite visit https://techcommunity.microsoft.com/t5/microsoft-search-blog/microsoft-search-at-ignite-2020/ba-p/16....

16 Comments
Iron Contributor

Hello! Question about the following:

  • Query condition: Taxonomy 

Does the following note apply to the Taxonomy condition?

NOTE These deprecations only apply to classic search experiences, they do not affect the modern search experiences.  

 

I'm probably misunderstanding this, but does this have an impact on the ability to issue queries against results filtered by terms? If so, does that limitation apply only to classic search, or both classic and modern?

 

Thank you!

Ryan

 

Brass Contributor

When will the search function in Teams include searching SPO pages that are available in Teams?

Microsoft

@Ryan Helmer, thank you for your question related to "Query condition: Taxonomy"!

 

You will still be able to issue queries against results filtered by terms. The removal of the taxonomy query condition is limited to query rules in classic search where an administrator has configured the query rule to trigger on a condition referencing a term from term store. 

 

Thanks, 

Karoline

Brass Contributor

When exactly will these changes be available in "First Release" mode within our tenants? We really need some time for adoption to check everything is still working as expected before this goes productive!

 

Thanks

Bjoern

Microsoft

@Björn Nettingsmeier, you can expect to see the initial changes in First Release from Feb 1st. We plan on rolling these changes out very slowly to allow for customers to adopt to the changes, but it's difficult to be more precise on the timing as the change will not occur for all customers worldwide at the same time. 

Brass Contributor

Hi,

 

We have some clients who are still firmly on classic site collections that can't yet be migrated to modern. How definite is the M365 roadmap January 2021 ETA for modern search being made available in classic? Can we expect to see that happen before these classic search deprecations take effect on all tenants?

 

Thanks!

Brass Contributor

The link to Bill's blog article needs to be updated but this is the new location if you want to learn more about Microsoft Search Answers.

Brass Contributor

@karolikl , do you have a quick tipp on how we can ensure that these changes are allready rolled out to our tenants? Is there a search statement or some other functionality that we could check to see if the update is allready active?

Brass Contributor

@karolikl , perhaps it is not related to this change but since yesterday (1st Feb) we see some unexpected behaviours on our tenants of SharePoint Search and date formats. Dates are not delivered anymore in UTC format and are somehow localized to the users language. Is this something that could belong to this change? 

 

It actually appears on productive tenants, so not within a first release.

Microsoft

@Björn Nettingsmeier The changes will roll out to first release tenants before production, so you should be able to see them there first. Both the authoritative pages change and the query rule change will be seen in the Admin UX (by the controls being removed). So if the controls are still present, the change has not rolled to your tenant yet. 

 

The date behavior you are describing should not be related to any of these changes. I would suggest you raise this through the regular escalation channels to ensure it's investigated. 

Copper Contributor

We are seeing the exact same thing as @Björn Nettingsmeier our search based applications are having issues since the dateformat has been changed.

Microsoft

@Björn Nettingsmeier & @thomas-christiansen: Although unrelated to the changes covered in this blog post, I can confirm that there is an incident related to the date format and the team is working on getting this fixed. 

Copper Contributor

@karolikl Any update on the fix? We have experienced the date format from a search managed property changing several times over the last week. It's current format is returned in ISO 8601.

Iron Contributor

Thank you for the detailed article.  Could we get some clarifications on 'query condition: common query' that is impacted by the change.  Is this one listed as is or is this "query more common in source"?  What about "query condition: regular expressions"?  This one doesn't appear to be listed either.

Microsoft

@Chris Ficek, you are correct that "common query" is displayed as "query condition more common in source". The "regular expression" query condition does not appear in the drop down list, but it is used by several of the out-of-the-box query rules (Phone Number and Tags) and is the cause behind us also deprecating those. 

Iron Contributor

@karolikl,  Thank you for your prompted reply.  It helps. 

Co-Authors
Version history
Last update:
‎Jan 12 2022 08:35 AM
Updated by: