Abstract:
Copilot Studio's Standard harness is built for conversation: several isolated model calls per turn, a reply within seconds, and control handed back to the user as soon as a step needs input. Many business tasks are not simple Q&A. They run for many turns, touch thousands of records, and need code, files and retries along the way. We implemented the Microsoft Copilot harness: a single model-centered agentic loop running inside a secure container, with command line and file tools and a way to materialize large data sets on disk so the model can reason over them iteratively.
This paper describes the two architectures and measures the difference. On the same evaluation tasks and the same models, the Microsoft Copilot harness makes 2 times more model calls per task, yet because of efficient caching (on average +40 pp), its cost-weighted token consumption falls 15 to 21% while its eval pass rate rises 10.8 points on GPT-5.5 Chat and 22.1 points on Opus 4.8 and per-call latency falls 10% to 36%. In production, Microsoft Copilot harness agents run longer tasks with more tool calls, use the container's built-in tools in more than half of turns, and keep working after a failed tool call 84% to 91% of the time where the Standard harness does so under 7% of the time. On a business process benchmark the Microsoft Copilot harness scored 65% against 36% for a leading competitor, and hill climbing tool behavior in production, one tool at a time, moved the Excel Online connector's success rate from 85.6% to 96.3% with one telemetry-driven tuning change and no maker action.
Read the full paper by clicking the PDF attachment below.
Microsoft