Building your own copilot – yes, but how? (Part 1 of 2)

In a technology landscape deeply changed by generative AI, businesses look for more effective ways to deliver customer service solutions to their customers. On the other hand, customers have higher expectations of a faster and more precise self-service support offering.
 

Today, there’s a wide range of built-in services and features designed to meet these needs and enable organizations and developers to build their own copilots, able to answer questions based on their own knowledge bases and data sources.
 

But how to choose the most suitable one for each scenario? This blog post wants to provide an overview of some of the main choices you have in the Microsoft technology ecosystem. Part 1 will look into low code tools and out-of-the-box features, while part 2 will focus on code-heavy and extensible options.

Empowering Generative AI models with RAG

Generative AI models – commonly known as Large Language Models or LLMs – are very powerful at generating different types of content, like grammatically correct text, art creatives and code, with just a short textual input – called prompt.

However, when it comes to using them in a real-world production scenario, they have some limitations, mainly due to the fact that they can answer questions related only to the data they were trained on. This means that they do not know facts that happened after their date of training, and they do not have access to data protected by firewalls.
 

Retrieval Augmented Generation (RAG) is a pattern designed to overcome the limitations mentioned above, by providing the LLM with the relevant and freshest data to answer a user question, injecting the information through the prompt.
 

A system that implements the RAG pattern includes in its architecture a data store, hosting the validated data sources (usually private data as enterprise data) on which the model should base its answer on. Each time a user question comes to the system:

1. The question is converted into a query to search into the data store for relevant information (an index is commonly used to optimize the search process).
2. The matching results are combined with the user prompt and the system message – including instructions for the model on how to use the data context provided – and injected into the model prompt.
3. The output of the model, written in natural language, is returned to the final user.

In the upcoming sections, we’ll explore how to implement this pattern with different tools and services in the Microsoft ecosystem.

Azure AI Studio

Azure AI studio is a brand-new hub – currently in preview – enabling developers and AI engineers with a centralized place to explore all the Azure AI services. The studio allows users to test, deploy, and evaluate pre-trained LLMs available in a catalog and build innovative solutions with them, making it simpler to collaborate on one AI project among teammates.

Once you provisioned an Azure AI resource and created an Azure AI project, a great first step to build your custom copilot in Azure AI Studio is the Playground.
 

There, you can set up your assistant by customizing through the GUI:

• The OpenAI model deployment used as the bot engine.
• The system message, which is a component of the model prompt designed to define an assistant’s personality, with desired style and tone, response format, and any guardrails.
• The length of the chat history to include.
• Examples of interactions between the user and the agent.

It’s also possible to implement the RAG pattern, by using the ‘Add your data’ feature, which uses the data sources you select to ground the generated model’s results with your data. Allowed data sources are local files, Azure Blob storage, or a vector index – created through the Azure AI search service.

Once you are done with the configuration and you have tested it through the chat session, you can export it as a JSON file for further usage.

You can also evaluate your assistant using the ‘Manual Evaluation’ feature, by providing a set of input prompts and expected responses.

Once you are satisfied with the result you can deploy your copilot to a web application, using the ‘Deploy’ button.

Microsoft Copilot Studio

Microsoft Copilot Studio enables the creation of copilots through a guided, no-code graphical experience, embedding all the capabilities that used to be part of Power Virtual Agents (PVA).

Once you sign in with your Microsoft 365 work or school account, and you select ‘Create a copilot’, you’ll launch a wizard to set up the main configurations of your assistant, such as the name, the supported language, and the URL to a website or SharePoint your copilot should use to ground its responses (e.g. your organization website).

Once your copilot is created, you can customize it further in the ‘Generative AI’ tab, by adding other website URLs and directly uploading documents as data sources.
 

Also, from the ‘Topics’ page, you can create a new topic - a dialog tree that describes how your copilot responds to a user's trigger.
 

In addition to manually defining the dialog trees – as the standard experience with PVA – you can also create a topic from a natural language description, leveraging generative AI.
 

Within a copilot's topic, you can add a ‘Create generative answers’ node, allowing you to specify more sources that the node searches based on your inputs. Information sources defined in the ‘Generative answers’ node override sources you specified at the copilot level (like the website URL you set up at the beginning), which functions as a fallback.
 

These data source options include external sources (such as Bing Search) and internal ones, like Azure OpenAI on your data, documents uploaded to Dataverse, SharePoint, OneDrive for Business, or custom data from any sources (such as a Power Automate Flow).
 

Once you are satisfied with your copilot configuration, you can publish your copilot to a website or Microsoft Teams and monitor its performance through the ‘Analytics’ feature.
 

Summary

In this article, we have been looking at how to build a custom copilot using Azure AI Studio and Microsoft Copilot Studio. These two tools both offer graphic developer environments to build a copilot on your data using generative AI.
 

Azure AI studio might be a natural choice for customers who have some familiarity with the Azure OpenAI service or the OpenAI playground and wish to have higher control over the LLM used by the copilot, including the ability to evaluate and compare different model versions as well as to design the model prompt.
 

On the other hand, Microsoft Copilot Studio might be a smoother transition for customers familiar with the Power Virtual Agent suite and need the flexibility to pre-define some closed dialogue journeys for frequently asked questions and then use generative answers for fallback.
 

In the upcoming part of this blog post, we’ll be going through 2 other options for building a copilot with the RAG pattern in the Microsoft ecosystem: using Prompt Flow and custom solutions.