From Good to Great: We Put Agent Optimizer to the Test in Microsoft Foundry

Authors: Abi Komma, Charles Kim, Luffy Chen, Morteza Ziyadi

Why hand-tuning agents does not scale 

An agent’s behavior comes mostly from its natural-language configuration: its instructions, skills, and tool descriptions. Most teams tune that configuration by hand. You write the instruction, run some cases, read the failures, reword, and try again. The Foundry documentation (link) calls this the manual trial-and-error loop. It is slow, subjective, and hard to scale across many agents. 

The loop is also brittle. Unlike a code bug, a quality failure has no stack trace: the cause can sit in the instruction, a tool description, or the model. Fix one case by editing the instruction, and another that used to pass often breaks. Your evaluators tell you what is wrong, but not the one instruction that satisfies every case at once. Retraining is not the answer either, because reinforcement-learning style tuning is costly and needs many trial runs. 

How agent optimizer works 

Every run leaves a useful signal: a natural-language trace of the agent’s reasoning and tool calls. Reflection based optimization learns from those traces. Rather than rely on a score alone, it reads a failing run in plain language, works out why the agent scored poorly, and proposes a targeted change to the agent’s configuration, such as its instructions, skills, or tool descriptions, not just a single prompt. Methods like GEPA (link) can match or beat reinforcement-learning tuning with far fewer trials. Agent Optimizer in Foundry Agent Service builds this reflection based optimization into it. 

You give agent optimizer the agent, the evaluators you already trust, and a dataset of representative cases, and it searches for a better configuration. Each round, it proposes a candidate change and tries it on a small batch of cases first. Only promising candidates go on to a full evaluation, which keeps the cost down. It keeps a Pareto frontier of the strongest candidates, the ones that win on different cases rather than a single overall best, and draws the next candidate from that pool. You review the leaders and their scores, then approve one or send it back for another pass. The version you approve becomes the agent’s next version, with the same model and tools. You decide what ships. 

How we tested it, and what we found 

To assess the improvements, we ran agent optimizer across six prompt agents (standard, declarative agents) that span different domains and evaluation types. We chose datasets with ground-truth based evaluators so the quality signal is objective and easy to check. We also included one agent, tax-assistant, with a rubric based judge to score overall response quality. Every run used the same setup: gpt-4o-mini as the agent model, a stronger model as the reflector that proposed mutations, and optimization of the system instructions. Each agent was evaluated the same way before and after, so the delta reflects the instruction change alone. 

Agent optimizer improved the score on every agent. The largest gains came on the agents that had the most room to improve, while an agent already close to the ceiling, such as sentiment, had less to gain. The changes it makes are readable: instead of rewording the prompt at random, it adds concrete rules that target the failure patterns in your data, and you can see exactly what changed before you approve it. The approach is not limited to text-only agents either. One agent in the set, retail-tools, calls tools to look up orders, accounts, and product data before it decides, and agent optimizer improved it while it kept calling those tools during evaluation. 

These are early, single-seed runs on small datasets, so the exact magnitudes are directional, but the gains were consistent across every agent and evaluation type. Agent optimizer helps most when an agent already runs on real evaluators and a dataset that reflects production cases, when fixes tend to trade one failure for another, and when you maintain several agents or re-tune often. It is less useful when the root cause is infrastructure, such as a tool that returns bad data, rather than the agent’s configuration, such as its instructions, tool descriptions, skills, or model. 

What’s next 

This first round benchmarks prompt agents tuned on their system instruction. In the next post, we will share experiments and results on hosted agents, with larger evaluation sets and more optimization targets, including skills, tool descriptions, and model selection. 

Get started 

For a full walkthrough of optimizing agents, see the Agent optimizer overview (link), which covers both prompt and hosted agents. To optimize a prompt agent, follow the Quickstart (link). To try it on a hosted agent, install the Azure Developer CLI (link) with the Foundry extension, then scaffold a sample agent, deploy it, and optimize: 

azd ext install microsoft.foundry               # one-time: add the Foundry extension 
azd ai agent init -m <optimization-sample>      # scaffold the sample agent 
azd provision && azd deploy                     # create resources and deploy 
azd ai agent eval generate                      # generate a dataset and evaluators 
azd ai agent optimize --max-candidates 2        # run the optimizer 
azd ai agent optimize apply --candidate <id>    # apply the winning candidate 
azd deploy                                      # redeploy the improved agent 

To optimize your own agent, see Make your agent optimizer-ready (link). For more agent examples, browse the foundry-samples repository (link).

Agent optimizer and the azd evaluation experience are in preview, so capabilities can change. The plans described here are work in progress, not a commitment to deliver any feature or timeline. The benchmark numbers come from internal runs on the datasets above, so your own results will vary with your agent, your data, and your evaluators. Microsoft Foundry and Azure are trademarks of the Microsoft group of companies. 

Learn more 

• Read agent optimizer announcement blog: link 

• See agent optimizer in action - view Build 2026 Observability session: aka.ms/build26-BRK252