Cowritten By: suhelp and arishojaswi
Update November 19, 2024: Bing web search query citations for users has entered general availability, Audit logging and eDiscovery for Bing web search queries are now rolling out as generally available. The new policy option: Allow web search in Copilot is rolling out in November.
To receive the highest quality responses from Copilot, users have the option to allow Copilot to reference web content. Querying the web improves the quality of Copilot responses by grounding them in the latest information from the web. Throughout the process the prompt and response remain within the service boundary; the queries Copilot generates have the user and tenant identifiers removed and are sent securely to the Bing search service.
Screenshot: Web content plugin end user toggle for Microsoft 365 Copilot
As more customers experience the benefits of web search, we’ve heard a strong interest in being able to see the queries Copilot generates to fetch that information from the Bing search service. Today, we’re excited to announce new features that will allow both users and admins greater visibility into Copilot generated web queries.
Bing web search query citations for users
Web search query citations for users (generally available in November for Microsoft 365 Copilot Business Chat) will include the exact web search queries derived from the user's prompt in the linked citation section of the Copilot response, helping users understand what search queries, along with the sites searched (available today), were used to enhance the response.
This provides users with valuable feedback as to exactly how their prompts are used to generate web queries that are sent over to Bing search. This transparency helps users with the information they need to improve prompts and use Copilot more effectively.
To learn more about data, privacy, and security for web queries in Microsoft 365 Copilot, please refer to the Microsoft Learn article Data, privacy, and security for web queries in Microsoft 365 Copilot.
Web search queries will be displayed in the linked citation section of the Copilot response
Audit logging and eDiscovery for Bing web search queries
Web search query logging (generally available in November for Microsoft 365 Copilot Business Chat) will enable admins to perform search, audit, and eDiscovery on the exact web search queries Copilot derived from the user's prompt. Admins can already perform these actions for prompts and responses and will be able to use their familiar tools to extend those actions to search queries.*
Providing transparency into web queries supports our mission of Trustworthy AI, as customers can confirm that the information searched by Copilot is relevant and appropriate.
Admins will be able to use the audit log to find events involving web queries. In Purview Auditing, run a search query to filter and export all relevant CopilotInteraction audit logs. This can be done by selecting “Interacted with Copilot” in the Operation dropdown. Once this query finishes, the results can be exported into a CSV file. Within the CSV file, for each entry, there is a property AISystemPlugin.Id which can be used to identify whether a specific Copilot interaction had a query sent to Bing. If it did, then AISystemPlugin.Id will contain the value ‘BingWebSearch’.
In order to identify all Copilot interactions where a web search query was generated, admins will first filter and export all relevant CopilotInteraction audit logs. This can be done by filtering operation = CopilotInteraction. Once the search results are exported, admind can then filter by AISystemPlugin.Id offline.
Admins will be able to use the audit log to find events involving web queries
For eDiscovery, the Bing search query is stored in the user’s Exchange mailbox as part of the Copilot response in the form of a metadata. Admins and eDiscovery Manager can search for the prompts and responses using Copilot Activity filter in eDiscovery condition builder and view the detailed conversations including the Bing search queries. The actual queries that were sent to Bing search can be viewed by downloading the query results and viewing this offline in a text editor. If a Copilot interaction resulted in a query sent to Bing, then the term “WebSearchQuery” will exist in the downloaded results. Alongside it will be the actual query text that was sent to Bing. Additionally, the CopilotInteraction audit record contains a messageId which points to this response message, which can be retrieved using eDiscovery. Learn more about using eDiscovery in the Microsoft Learn article Search for and delete Microsoft Copilot for Microsoft 365 data.
Expanded controls for managing web searches in Copilot
To offer more ways for IT admins to manage web search in Microsoft 365 Copilot and Microsoft Copilot, we also recently introduced a new policy option: Allow web search in Copilot. This new control provides the ability to manage web search separately from other optional connected experiences for Microsoft 365. It will also support turning off web search in Microsoft 365 Copilot Business Chat (work) while keeping web search on in Business Chat (web).
Both the Allow web search in Copilot policy feature and the optional connected experiences policy can be managed at the user and group levels from the Microsoft 365 Apps admin center. This feature is rolling out in November.
Our continuing commitment to enhancing web search control
As the examples above illustrate, we're continuing to invest in giving customers increased visibility and control over web search for Copilot. To learn more, please see the Microsoft Learn article How Microsoft handles generated queries.
*Note: The specific controls will vary depending on the underlying subscription plan.
Microsoft
