Monitoring performance and effectiveness of Microsoft 365 Copilot agents

When asked by a systems integrator partner about the best ways to "monitor" M365 Copilot agents (and this comes up frequently), I always follow up with questions to clarify what, exactly, they are trying to monitor. Are we trying to monitor performance, looking at speed and potential bottlenecks? Are we trying to find out who is using the agent, how often it's being abandoned, and how effective the functionality appears to be for end users?

 

Microsoft offers multiple means of "monitoring" M365 Copilot agents, based on the intent of the question. By understanding what tools are available to the administrator, implementer, agent maker, or product owner, it is possible to gain deeper insight into what does (and doesn't) work well in the custom agent solutions.

 

MONITORING PERFORMANCE

Monitoring the performance of custom M365 Copilot agents is essential to ensure they operate efficiently and end users aren't waiting unnecessarily long a specific outcome. One of the primary aspects to monitor is the speed of the agents. This involves tracking how quickly the agents respond to user queries and perform their designated tasks. Tools like the Copilot Control System provide comprehensive analytics and reporting capabilities that allow administrators to view core usage telemetry and track the speed of agent responses. By analyzing these metrics, you can identify any delays or slowdowns in agent performance and take corrective actions to optimize their speed.

 

Identifying and addressing bottlenecks is another critical aspect of performance monitoring. Bottlenecks can occur at various points in the agent's workflow, such as during data retrieval, processing, or response generation. Using tools like the PVA test framework for load testing and performance monitoring can help you simulate different scenarios and identify potential bottlenecks. Additionally, connecting via DirectLine API allows for programmatic testing of Copilot Studio agents, helping you pinpoint specific areas where performance may be lagging. By addressing these bottlenecks, you can ensure that your agents operate smoothly and efficiently.

 

MONITORING USAGE AND EFFECTIVENESS

Monitoring the usage, analytics, and effectiveness of custom M365 Copilot agents is crucial for ensuring they deliver optimal performance and meet organizational needs. One of the primary tools for tracking usage is the Microsoft 365 admin center, which provides detailed reports on agent activity. These reports become available within 72 hours of the end of the day and can be filtered by different periods, such as the last 7 days, 30 days, 90 days, or 180 days. The agent usage report captures metrics on active agents, user interactions, and adoption rates, helping administrators understand how frequently agents are being used and identify any patterns or trends in their usage.

 

Analytics play a significant role in measuring the effectiveness of custom M365 Copilot agents. The Copilot Control System offers comprehensive analytics and reporting capabilities that allow administrators to view core usage telemetry and track license utilization. By analyzing these metrics, organizations can gain insights into how well their agents are working for end users, and identify areas for improvement. For example, the Analytics tab in Copilot Studio provides detailed reports on agent effectiveness, including metrics such as escalation rate, abandon rate, and resolution rate. These reports help administrators understand the impact of agents on user productivity and business outcomes, enabling them to make data-driven decisions to enhance agent performance.

 

Effectiveness can also be monitored through user feedback and interaction logs. The upgraded analytics page in Copilot Studio allows administrators to review user feedback, knowledge source use, and the outcomes of conversations between agents and users. This helps identify any recurring issues or areas where agents may not be performing as expected. Additionally, reviewing historical activity and logs of AI inference or decision-making can help address inconsistencies and optimize agent performance. By proactively monitoring these aspects, organizations can ensure that their custom M365 Copilot agents deliver accurate and timely responses to users.

 

CONCLUSION

By understanding various tools available for monitoring custom M365 Copilot agents, stakeholders can ensure their solutions are performing well and effectively solving problems for the business users in the organization. I would not be surprised to see additional focus on these areas in future evolution of the Copilot and Copilot Studio products and administration tools, as this is one of the most asked-about topics in my discussions with global systems integrator partners in our ongoing partnership discussions.