Video Transcript:
-A lot of the information you work with today might sit outside of Microsoft 365 locations. So how can you safely bring that information into your Microsoft 365 Copilot experiences to ground generative AI results? There are two primary options. First, you can use Microsoft Graph connectors to integrate the data sources you want with Semantic Index for Copilot. This option pre-indexes defined read-only information used for retrieval with your prompts to drive the most relevant AI generated responses.
-The second option uses managed and trusted plugins which can be a few different types, including OpenAI plugins, Microsoft Teams message extensions, and Power Platform connectors. And with plugins instead of pre-indexing information, they optimize for real-time API based data retrieval and they can optionally even write back data. Both options follow Microsoft’s comprehensive security compliance and privacy approach while respecting the boundaries that you define within your organization’s data services. And both types can even work together when responding to a single prompt.
-In fact, let me show you an example using a connector and a plugin simultaneously. Let’s say I’m a help desk technician in Microsoft Teams and I want to respond to all users facing an issue with DNS. I’ll start by submitting a prompt to help me find who’s impacted and to write an email for me that I can send them with instructions that reference our internal knowledge base article. Copilot has taken my prompt and is generating a response, in this case, by pulling in information from both a plugin configured for Jira which we’re using for issue tracking, as well as a Graph connector for an intranet site which we’re using as a knowledge base for our help desk. And it responds with a matching DNS related ticket along with the steps for flushing DNS and even an email it authored that I can use to respond to our user with the matching open ticket. You’ll also see at the bottom of the response that both the plugin’s Jira ticket and the connector’s intranet KB article are referenced.
-Additionally, if I scroll up and click on the Jira ticket details, you’ll see that my plugin also presents an adaptive card to show more information and it even allows me to take action. I’ll hit “Edit issue” and right from within Teams, I can change the record in Jira, for example, if I sent the email to the user already and resolved the issue, I can update the ticket status from “To Do,” to “Done.” And by hitting “Save,” it’ll record that change to our Jira system. So we just saw how you can extend Microsoft 365 Copilot to external data sources coming into the Microsoft Graph using connectors and two-way integration using plugins. Next, let me show you what was going on behind the scenes and how each one works.
-The first option where we referenced our intranet article, leveraged Semantic Index for Copilot which works with Microsoft 365 Copilot by default over internal Microsoft 365 data. When you submit a prompt, Copilot brings in the Microsoft Graph with a Semantic Index to correlate activity signals, retrieving indexed information uniquely relevant to you. It presents the combination of your prompt with what was retrieved to the large language model to generate an informed response. Importantly, Semantic Index works on two levels. By default, it indexes information within SharePoint inside your organization’s tenant. And second, each user also has their own separate index for email and their individual important documents like files shared with them, where they’ve commented or were at mentioned, and more.
-This two-pronged approach is what increases relevancy, allowing Microsoft 365 Copilot to orchestrate AI generated results that are unique to you as a user, again, based on your approved access to information. And a great way to demonstrate Semantic Index for real is by showing two sets of search results in SharePoint, one without leveraging the Semantic Index and one with it. Here we’re searching on Project Melbourne Shiproom. Notice we have a typo. On the left, without the Semantic Index, you can see results are retrieved based on keywords and ranking as well as last modified date for relevance and a file summary for context but it’s not the specific information we’re looking for. Conversely, on the right with the Semantic Index, instead of a list of links, notice the informative answer right at the top. You can see that the index has mapped semantic signals such as people relationships and recent activities surrounding the content the user has access to, to help determine the intent of your search and formulate an answer. So you can see how Semantic Index improves the search experience across Microsoft 365 and that includes Microsoft 365 Copilot.
-So how do you take advantage of Semantic Index for your external data, like we saw in our example? First, if you already use Graph connectors in Microsoft 365, they will work without any changes. If not, before you connect to external data, from the Microsoft 365 Admin Center, you do need to enable the service or start a trial and from there you can add a connection. There are options for enterprise websites and a few other common data sources listed here as well. For example, if I open ServiceNow, you’ll see that there are three options for ticketing, knowledge base, and catalog, but in my case, I’ll set up an enterprise website and this will show me a few simple options to define the site, who can search the data source, and whether connector results are shown in search.
-Then for even more options, I can go into advanced settings for more granular configuration with options to define meta tags, specific exclusions, semantic property labels, and customizations to the schema. And once you have a connection up and running, you’ll be able to see integrated results using SharePoint search with the connected site. In this case, I’ll just perform a simple keyword search for troubleshooting instructions and results are found in SharePoint locations as well as from the connected site. And here is the DNS article that was used as part of the retrieval process to respond to my prompt from before.
-Beyond Microsoft Graph connectors with Semantic Index, let’s look at our second option for data retrieval with Copilot, using plugins. Again, starting with how they work. So with a plugin configured, when we submitted the prompt, Microsoft 365 Copilot leveraged that plugin to retrieve information directly from its source, again, in real time without pre-indexing the data. It was also able to perform additional actions on behalf of the user and with respect to their permissions, like changing the status of the ticket. This time, I’ll demonstrate the plugin by itself without connection to our knowledge base. In Microsoft Teams, I’ll write a new prompt to look up VPN related tickets in our Jira system and submit it. Then Copilot responds using that information and finds two tickets in the system related to VPN issues. And as I mentioned, plugins work without pre-indexing and there are three types. These can be OpenAI plugins, Microsoft Teams message extensions, and Power Platform connectors.
-Now I’ll explain how plugins work with a short tour of their code. All three of these types of plugins are similar with similar packaging and components. This is an example of a Teams app manifest file that uses NuGet Package Manager to find app packages and their details. You can see that it refers to a YAML file. And in that file, is where the API is defined for connecting to the external data or service as well as any actions it can take on behalf of the user. Also note the natural language descriptions of the API get operation and its responses. These are used by the LLM to assess what the plugin can do to help respond to a prompt. And because these are portable components and can be used for OpenAI, Teams, and Power Platform, as a developer, you’ll be able to maintain a single shared code base. Each will leverage the same underlying open API spec.
-Then to make plugins available in your Microsoft 365 environment, you’ll do this from the Microsoft 365 Admin Center in the Integrated Apps page. Here you can choose from existing plugins from the store, just like you would get other apps today. You can also upload your own plugin packages directly in the Teams Admin Center. Either way you add the plugin, you’re in control of who can access the plugin and its underlying data.
-So those are the two main options available to you for safely bringing in data external to Microsoft 365 into your Microsoft 365 Copilot experiences to ground generative AI results. To learn more, check out aka.ms/CopilotExtensibility and keep following Microsoft Mechanics for the latest in tech updates, thanks for watching.
