Building Copilot Agents with User Experience in Mind
Understand where and how the user will be accessing the Copilot agent Create multiple-modal experiences for your Copilot using voice To enhance the accessibility and user-friendliness of your copilot agent, consider developing a voice agent for voice interactions. This hands-free experience is not only inclusive but also particularly beneficial in customer scenarios where associates are using their hands and need to interact with the agent quickly and effectively without typing. Before deploying the agent to your user base, ensure thorough testing in the work environment to validate a good end-user experience, especially considering the loudness of the environment. For step-by-step instructions on testing the voice agent experience, you can view this blog Determine the proper channel deployment Selecting the appropriate channel for deployment is crucial. Ensure you reach users where they perform their work, whether it's through M365 Copilot, M365 Copilot chat, embedding into a website, or other channels. It's essential to deploy the agent to the channels that best suit the use case. Creating multilingual agents Creating multilingual agents is essential for providing a personalized and inclusive experience. By enabling your agent to interact in multiple languages, you can cater to a diverse user base and ensure effective communication without language barriers. This feature enhances accessibility, user satisfaction, and expands the agent's reach in global contexts. For more specific information on multilingual agents see below. Take Time During the Configuration to Set Up the Agent for Success Creating a name for the agent In a world where multi-agents are on the rise, it's essential to be as descriptive as possible in the name and description for the intended use. This helps users understand the agent's purpose and functionality. Planning out your multi-agent scenarios early and defining clear names is especially important to avoid confusion. Add starter prompts Starter prompts guide users on how to interact with the agent effectively. These prompts should be clear and concise. When selecting starter prompts, consider defining the most common use cases for the agent. For example, if you are creating an IT Support agent, you might include prompts like "create a support ticket" or "check request status" to cover the most asked questions. Provide description and general instructions A detailed description of the agent's capabilities and intended use helps users understand what they can expect from the interaction. This description also assists the language model in identifying and utilizing your agent for specific tasks or situations. Ensure that the description is concise yet informative. For the instructions (declarative and generative mode), you should be as detailed as possible. Include the purpose, guidelines on what the agent should and should not do, tone, and the skills the agent will possess. Below are resources that provide detailed instructions on how to construct the verbiage effectively. Add key knowledge resources Adding key knowledge sources to the agent is crucial for its effectiveness and reliability. By integrating relevant documentation and connecting to essential business systems, the agent can provide accurate and timely information, ensuring users receive the support they need. These connections enable the agent to perform specific tasks efficiently, leveraging up-to-date data and resources. This not only enhances the user’s experience but also ensures the agent remains a valuable tool in various scenarios, from customer support to internal business operations. Offer alternative ways to answer questions Connecting to a live agent Provide users with the option to connect to a live agent for more complex queries or issues that the agent cannot resolve. Customizing the escalation topic is another way to track potential opportunities for agent improvement. Creating a step in the topic to record the escalation to ensure it is captured for agent performance is key to ensuring agent success and continued feedback on areas of improvement. Create adaptive cards Using adaptive cards in a Copilot agent enhances user interaction by providing a visually appealing and interactive interface. They facilitate structured data collection, ensuring consistency and ease of processing. Adaptive cards offer customization and flexibility, allowing them to be tailored to various scenarios and use cases. They improve communication by presenting information clearly and concisely, reducing ambiguity. Additionally, adaptive cards integrate seamlessly across different platforms and devices, ensuring consistent user experience. Overall, they significantly enhance the functionality and user experience of Copilot agents. Provide documentation references or links In the agent's response, include links to additional resources. Offer links to documentation or external sites where users can find more detailed information. Automate, Automate, Automate Enhancing efficiency with automation Using automation in Copilot agents (agent flows and agent actions) can significantly improve efficiency and reduce manual workload by identifying key processes that can be automated. Incorporating messages that inform users when the agent is processing their request helps manage user expectations and enhances the overall experience. By automating repetitive tasks and providing real-time updates, Copilot agents can streamline workflows, allowing users to focus on more complex and value-added activities. This approach not only boosts productivity but also ensures smoother and more transparent interaction with the agent. Monitor Agents for Insights Actively review the analytics for the agent Regularly reviewing analytics is essential for assessing the agent's performance and identifying areas for improvement, ensuring a high-quality user experience. By analyzing metrics such as engagement outcomes, knowledge source usage, action success rates, and user feedback, you can gain valuable insights into how effectively your agent is meeting user needs. Resources like Microsoft Copilot Studio Analytic, Topic Analytic Autonomous agent health, and Application Insights telemetry provide detailed guidance on measuring and enhancing agent performance. For more information see the discussion on monitoring performance and effectiveness of M365 Copilot agents. Collect Feedback Integrate feedback mechanisms Use adaptive cards to collect user feedback after every interaction with your agent. This allows you to assess and enhance the quality of agent responses while ensuring user satisfaction. Implementing this feedback mechanism helps in obtaining valuable insights into how well your agent is performing and identifying areas for improvement. Conclusion To create an effective Copilot agent, focus on providing easy accessibility by developing voice interactions and multilingual capabilities. Ensure you set up the agent properly in a world of multi-agents to decrease complexity and avoid confusion. Be creative in how you answer prompts, using adaptive cards and alternative ways to respond. Automate crucial business processes associated with the agent's purpose to enhance efficiency and reduce manual workload. Finally, actively collect feedback and apply it to continuously improve the user experience, ensuring the agent remains a valuable tool. A special thanks to IvicaIvancic ((3) Create your first custom AI enabled Copilot Studio Voice Agent – step-by-step guide | LinkedIn) and TrevorNorcross (Monitoring performance and effectiveness of Microsoft 365 Copilot agents | Microsoft Community Hub) for their incredible support in allowing me to link to their insightful blogs.Clarity around the different types of connectors available for Microsoft 365 Copilot
Working with some of the largest and most technical of Microsoft's partners in the Global System Integrators (GSI) team, we are still often met with confusion around the different types of connectors available in M365 Copilot. While agent extensibility could easily be a full week or more of discussion topics and sessions, I wanted to spend a moment to focus specifically on some of the various types of connectors available to agent makers working with M365 Copilot. GRAPH CONNECTORS Graph Connectors play a crucial role in enhancing the knowledge scope of Microsoft 365 Copilot, by enabling the integration of external data sources into the Microsoft Graph. This integration allows Copilot to reason over a broader range of enterprise content, thereby improving the relevance and accuracy of responses to user queries. By ingesting unstructured business data through Graph Connectors, organizations can ensure that their critical content is indexed and accessible within Microsoft 365 Copilot. This process involves semantic indexing, which optimizes data retrieval and enhances the matching of search queries to content, providing more relevant results than simple keyword matches. The relevance of Graph Connectors to Microsoft 365 Copilot is particularly significant in the context of enterprise search and data utilization. Copilot leverages these connectors to access and summarize information from various external sources, such as third-party applications, databases, and cloud services. This capability allows users to find, summarize, and learn from their business data through natural language prompts in Copilot. For example, when a user asks Copilot to summarize recent communications or project updates, Graph Connectors enable Copilot to utilize relevant data from integrated sources, ensuring comprehensive and accurate responses. As an agent maker using M365 Copilot, one important consideration of Graph Connectors is that they offer high-performance operations, due to direct access to information via the Large Language Model (LLM). However, keep in mind that data accessed is not pulled from the external data sources in real-time, as the Graph Connectors sync this business data on a set schedule. Additionally, your operations will be limited to retrieval-based ("read only") tasks when interacting with externally-synchronized Graph data. It should also be noted that these types of connectors will work best for working with unstructured data. Moreover, Graph Connectors are not limited to Microsoft 365 Copilot; they also power other intelligent experiences within the Microsoft ecosystem, such as Microsoft Search and Context IQ. This extensibility allows users to hover over in-text citations to preview external items referenced in responses and dive deeper into the referenced content by selecting links at the bottom of Copilot responses. The ability to configure custom connectors and utilize pre-built ones further enhances the flexibility and utility of Graph Connectors, making them a valuable tool for organizations looking to leverage their external data within Microsoft 365. By integrating these connectors, organizations can ensure that their data is not only accessible but also actionable, driving better decision-making and productivity. POWER PLATFORM (FLOW) CONNECTORS Power Platform Connectors are integral to extending the capabilities of Microsoft 365 Copilot by enabling seamless integration with various external applications and services. These connectors act as proxies or "wrappers" around APIs, allowing Copilot to interact with other apps and services within the Microsoft ecosystem and beyond. By leveraging Power Platform connectors, users can connect their accounts and utilize prebuilt actions and triggers to build sophisticated workflows and applications. This integration enhances Copilot's ability to retrieve and process data from diverse sources, thereby providing more comprehensive and actionable insights to users. In the context of Microsoft 365 Copilot, Power Platform connectors enable the creation of custom agents that can perform specific actions based on enterprise data. For instance, connectors can be used to retrieve sales opportunities, manage orders, or even check the weather at a customer's location for site visits. This functionality is particularly valuable for businesses looking to automate routine tasks and streamline operations. By integrating these connectors, Copilot can ground its responses in real-time data from various enterprise systems, ensuring that users receive accurate and relevant information. While the performance of a Power Platform Connector will inherently not be as speedy as data retrieval via Graph Connectors, agent makers should consider advantages to using Power Platform Connectors in the right scenarios. The API wrapper can perform real-time operations with the external business data, not requiring data synchronization to the Graph to access. Additionally, these types of connectors offer the ability to add and update records, in addition to retrieval. I have also seen use cases where Power Platform Connectors are preferred when dealing with structured data, even in retrieval-only agents and tasks. Beyond that, Power Platform Connectors are essential for extending Copilot's capabilities through Copilot Studio. This platform allows makers to add custom knowledge and skills to agents using connectors, thereby expanding the range of actions that Copilot can perform. For example, connectors can bring in data from Microsoft Graph, Dynamics 365, and other non-Microsoft enterprise sources, enabling Copilot to provide more nuanced and detailed responses. The ability to use these connectors in a no-code environment makes it accessible for users with varying technical expertise, empowering them to create intelligent, automated experiences that address unique business challenges. CONCLUSION By understanding the different types of connectors, the advantages and disadvantages of each, and the scenarios where each work best, agent makers can use one of the most powerful tools for working with external business data. For additional technical detail, please see the links below: Copilot connectors and actions overview (preview) - Microsoft Copilot Studio | Microsoft Learn Use Power Platform connectors (preview) - Microsoft Copilot Studio | Microsoft Learn Extend Copilot for Microsoft 365 with connectors | Microsoft Learn Build Microsoft Graph Connectors for Microsoft 365 Copilot | Microsoft LearnMonitoring performance and effectiveness of Microsoft 365 Copilot agents
When asked by a systems integrator partner about the best ways to "monitor" M365 Copilot agents (and this comes up frequently), I always follow up with questions to clarify what, exactly, they are trying to monitor. Are we trying to monitor performance, looking at speed and potential bottlenecks? Are we trying to find out who is using the agent, how often it's being abandoned, and how effective the functionality appears to be for end users? Microsoft offers multiple means of "monitoring" M365 Copilot agents, based on the intent of the question. By understanding what tools are available to the administrator, implementer, agent maker, or product owner, it is possible to gain deeper insight into what does (and doesn't) work well in the custom agent solutions. MONITORING PERFORMANCE Monitoring the performance of custom M365 Copilot agents is essential to ensure they operate efficiently and end users aren't waiting unnecessarily long a specific outcome. One of the primary aspects to monitor is the speed of the agents. This involves tracking how quickly the agents respond to user queries and perform their designated tasks. Tools like the Copilot Control System provide comprehensive analytics and reporting capabilities that allow administrators to view core usage telemetry and track the speed of agent responses. By analyzing these metrics, you can identify any delays or slowdowns in agent performance and take corrective actions to optimize their speed. Identifying and addressing bottlenecks is another critical aspect of performance monitoring. Bottlenecks can occur at various points in the agent's workflow, such as during data retrieval, processing, or response generation. Using tools like the PVA test framework for load testing and performance monitoring can help you simulate different scenarios and identify potential bottlenecks. Additionally, connecting via DirectLine API allows for programmatic testing of Copilot Studio agents, helping you pinpoint specific areas where performance may be lagging. By addressing these bottlenecks, you can ensure that your agents operate smoothly and efficiently. MONITORING USAGE AND EFFECTIVENESS Monitoring the usage, analytics, and effectiveness of custom M365 Copilot agents is crucial for ensuring they deliver optimal performance and meet organizational needs. One of the primary tools for tracking usage is the Microsoft 365 admin center, which provides detailed reports on agent activity. These reports become available within 72 hours of the end of the day and can be filtered by different periods, such as the last 7 days, 30 days, 90 days, or 180 days. The agent usage report captures metrics on active agents, user interactions, and adoption rates, helping administrators understand how frequently agents are being used and identify any patterns or trends in their usage. Analytics play a significant role in measuring the effectiveness of custom M365 Copilot agents. The Copilot Control System offers comprehensive analytics and reporting capabilities that allow administrators to view core usage telemetry and track license utilization. By analyzing these metrics, organizations can gain insights into how well their agents are working for end users, and identify areas for improvement. For example, the Analytics tab in Copilot Studio provides detailed reports on agent effectiveness, including metrics such as escalation rate, abandon rate, and resolution rate. These reports help administrators understand the impact of agents on user productivity and business outcomes, enabling them to make data-driven decisions to enhance agent performance. Effectiveness can also be monitored through user feedback and interaction logs. The upgraded analytics page in Copilot Studio allows administrators to review user feedback, knowledge source use, and the outcomes of conversations between agents and users. This helps identify any recurring issues or areas where agents may not be performing as expected. Additionally, reviewing historical activity and logs of AI inference or decision-making can help address inconsistencies and optimize agent performance. By proactively monitoring these aspects, organizations can ensure that their custom M365 Copilot agents deliver accurate and timely responses to users. CONCLUSION By understanding various tools available for monitoring custom M365 Copilot agents, stakeholders can ensure their solutions are performing well and effectively solving problems for the business users in the organization. I would not be surprised to see additional focus on these areas in future evolution of the Copilot and Copilot Studio products and administration tools, as this is one of the most asked-about topics in my discussions with global systems integrator partners in our ongoing partnership discussions.Copilot Chat vsus. Microsoft 365 Copilot What's the difference?
While their names sound similar at first glance - Microsoft 365 Copilot and Microsoft 365 Copilot Chat - they differ in several aspects. And more importantly, one can't be without another. What is Microsoft 365 Copilot Chat? First since first. Microsoft 365 Copilot Chat is also called Copilot Chat. Copilot Chat generates answers based on web content, while Microsoft 365 Copilot is also grounded on users' data, like emails, meetings, files, and more. Since January 15, Microsoft 365 Copilot Chat is available for everyone. Everyone in organizations. Also customers with a Microsoft 365 Business Basic subscriptions can enjoy using Copilot Chat securily. The screenshot below shows how Copilot Chat looks like and highlights its main capabilities. Note that EDP - Enterprise Data Protection is available. What is Microsoft 365 Copilot? Microsoft 365 Copilot is an add-on available for specific Microsoft 365 Subscriptions: Microsoft 365 E3, E5, A3, A5, and Business Standard & Premium. It includes Copilot Chat in addition to other Copilot features: Microsoft 365 Copilot also includes a chat grounded on users' meetings, emails, chats, and documents. It integrates into Microsoft 365 apps, like Outlook, Teams, Word, Excel, and more. It brings the capability to create agents and additional Copilot management features such as SharePoint Advanced Management and Copilot Dashboard. The screenshot below shows how the Copilot chat experience for those users who got the Microsoft 365 Copilot license. Note that EDP - Enterprise Data Protection is available here too. Copilot Chat can be pinned in MS Teams and MS Outlook as App. How can I access Microsoft 365 Copilot Chat? Copilot Chat is nowadays accessible via m365copilot.com using your Entra account. In contrast to Microsoft 365 Copilot licensed users, Microsoft 365 Copilot Chat users cannot see, by default, Copilot Chat pinned on the Microsoft 365 homepage. Microsoft 365 Copilot Administrators will have to pin the chat in the admin center so it is easy for Copilot Chat users to access it. What's the difference? There are some aspects, such as licensing requirements, subscription fees, data sources, or access to organization content, that determine the differences between Copilot Chat and Microsoft 365 Copilot. I have listed it in the screenshot below. 👇 Image showing a 3-column table: Aspect, Microsoft 365 Copilot Chat, and Microsoft 365 Copilot.When not to purchase Copilot Studio
Copilot agents are everywhere and for everyone. They are experts on a topic, able to talk to other agents, start an automation, and complete tasks for you. Agents can boost team productivity and process efficiency. Productivity dreams may come true with agents. That's how the need to use and create agents increases every day. ✅ Stay calm and read further before making any purchase decision on the M365 admin center. It may save you over 224,64 EUR monthly with a 1-year upfront commitment. 👇 Who can create agents, and how? M365 Copilot licensed users can create agents using Copilot Studio. Although the creation is not available for M365 Copilot non-licensed users, they can add and interact with agents. Nowadays, the easiest way to create Copilot agents is as follows: Use the built-in version of Copilot Studio. Do not start from scratch. Instead, use an agent template like the Prompt Coach to build your own. Why do agents bridge the gap between M365 Copilot licensed and non-licensed users? Agents are an excellent option for providing M365-Copilot non-licensed users access to Copilot capabilities grounded on organizational data. Even if users are not licensed with M365 Copilot, they can benefit from Copilot grounded on organizational data via agents. Can I purchase Copilot Studio? Copilot Studio is a no-code Power Platform tool that allows the creation of Copilot agents. The M365 Copilot add-on license already includes licensing for Copilot Studio. You can find it as Copilot Studio in M365 Copilot. There are 3 different ways to license Copilot Studio: 🤖 Copilot Studio pay-as-you-go meter: 0.01 USD per message (no upfront commitment) 🤖 Copilot Studio message packs: 200 USD per tenant, per month 🤖 Copilot Studio in M365 Copilot: 30 USD per user, per month Nevertheless, there's a product called Microsoft Copilot Studion on the purchase services on M365 Admin Center. I am not quite sure, what this product is for. If you know it, please let me know in the comments. 🛑 The challenge is: Microsoft Copilot Studio on the M365 admin center purchase services can easily be mistaken for Copilot Studio in M365 Copilot. Be cautious!
Can we enable Copilot in Azure Devops for AI assisted user story and test case generation?
I am currently using Copilot as an enterprise user for github copilot in VSCode and want to have copilot enable in Azure Devops so that it can assist in user story and test case generation. But i am not sure if it's even possible. Please let me know how can i achieve it?M365 Copilot requirements explained!
Why do you need a OneDrive account? What for a MS Entra ID account to use M365 Copilot? If you're wondering about these questions and more, read on. I will briefly explain the technical requirements for M365 Copilot. What is M365 Copilot? Skip this if you're already familiar with it. M365 Copilot is a conversational AI tool that can help you with your tasks at work. M365 Copilot includes access to: Copilot Chat Seamless integration in M365 Apps like Outlook, Word, Excel, PowerPoint, Teams, more... Creation and usage of Copilot Agents A key feature of M365 Copilot is Enterprise Data Protection or EDP. EDP does not limit M365 Copilot users. Instead it is a relief to use the advantages of AI at work knowing, that your organizational data is protected and secured by the terms of the Data Protection Addendum and the Product Terms. EDP ensures, for example, that your data isn't used to train foundation models or that your data remains within the boundaries of the EU, supporting regulatory and compliance requirements such as GDPR. M365 Copilot Prerequisites I enjoy having conversations about various topics with different people. Here and there, I get asked about M365 Copilot. And here and there, I confirm how organizations want to bake the cake without first gathering all the ingredients. A: Let's deploy Copilot. B: Sure! Have you already adopted OneDrive? A: No, we are still using our file server. Why? An optimal usage of M365 is a prerequisite for enabling M365 Copilot. Happy M365 users = happy M365 Copilot users. It is harmonized with the technical prerequisites. Even if technically, your users have a OneDrive account available. It counts as checked if they are using OneDrive regularly for saving and sharing their work files. An eligible base license Microsoft 365 Copilot is an add-on license, therefore to assign the add-on, there needs to be a base license first. Eligible base licenses include: - M365 Apps for enterprise or M365 Apps for business - M365 Business Basic, Business Standard or Business Premium - M365 E3, E5, F1, or F3 - M365 A1, A3, or A5 for faculty - Office 365 A1, A3, or A5 for faculty or higher education students aged 18+ And more. MS Entra ID account Formerly Azure Active Directory, MS Entra ID is the cloud-based identity and access management service. Users must have MS Entra ID account. Why? As the motor for identity and access control and management in M365, Microsoft Entra ID plays a key role in Enterprise Data Protection. In other words, without MS Entra ID account, there's no chance within M365 architecture to ensure Enterprise Data Protection. OneDrive account Have you tried to see what happens if you use M365 Copilot without a OneDrive account? I haven't myself, if you've tried, share your experience with me, send me a message. 👇 Some features in M365 Copilot require users to have a OneDrive account, like uploading a file from your device to enrich a prompt. Uploading a file to a Copilot Chat prompt will send a copy to that location OneDrive/my files/Microsoft Copilot Chat Files on your OneDrive for Business account. Microsoft 365 Apps Cloud-based Microsoft 365 Apps are required. Make sure there's no device-based licensing for M365 Apps for enterprise, since Copilot isn't available in such a case. M365 Copilot also integrates in the web versions of Word, Excel, and PowerPoint. To ensure a good user experience, make sure thrid-party cookies are enabled. Update Channel To ensure your users receive the latest M365 Copilot features, make sure you set up as M365 Apps update channel either the Current or Monthly Enterprise Channel. Outlook Have you tried to use M365 Copilot in Outlook Classic? M365 Copilot works in both, the new and classic versions. In the Outlook classic version, it loses the interactivity features, which I personally love. Teams If your users are licensed with M365 Copilot, then let them try the M365 Copilot integration in MS Teams. Copilot in Teams can help people summarize Teams messages as well as take notes during meetings. Make sure to configure transcription and captions for Teams meeting and Teams meeting recording.
Get files from SharePoint, send the result to Copilot, show the documents as list inside Copilot
I am trying to create a copilot inside copilot studio, which do the following:- 1) Call a Power Automate Flow to return the documents from a SharePoint online document library. 2) I will select the File name and File URL only 3) Send the documents to Copilot. 4) then i need to build a list inside a Copilot Question, to select a Document, as i want to send the document to another power automate flow. Here what i have :- 1) Inside Copilot i am calling the flow:- 2) Here is the flow. Where i got the files, Select the Name & Url, initialize the variable with the output of the Select, and return the variable to Copilot, after converting the variable into json() :- here is sample of the response from the flow:- { "r": " [ {\"Name\":\"New Microsoft Word Document (4)\",\"Url\":\"https://**.sharepoint.com/sites/Copilot/New/New%20Microsoft%20Word%20Document%20(4).docx?d=w36a23c408c844a4ebf4d444f8777a975\"}, {\"Name\":\"VScompleted\",\"Url\":\"https://**.sharepoint.com/sites/Copilot/New/VScompleted.png\"}, {\"Name\":\"Salary\",\"Url\":\"https://***.sharepoint.com/sites/Copilot/New/Salary.docx?d=wcf24819ffdcf4616873cd50c24f1b1d0\"} ] " } but how i can convert this JSON to a table inside Copilot and show this table as a question to the user, to select a specific document? Thanks
