Authors: Priyanka Paul, Abi Komma, Charles Kim, Luffy Chen, Morteza Ziyadi
Agent optimizer in Foundry Agent Service automatically improves prompt agents and hosted agents by evaluating their behavior and generating candidate configurations. Depending on the agent type, it can improve instructions, skills, tool descriptions, and model selection. It evaluates each candidate against the same dataset and criteria, ranks the results, and lets the developer choose the best candidate to deploy.
Each optimization run uses an agent model and a reflection model. The agent model completes the tasks in the dataset, while the reflection model reviews the evaluation results and proposes changes to the agent configuration. When the evaluator uses an LLM judge, a separate judge model scores the responses and explains failures.
In our previous post, we used GPT-5 as the reflection model and found that it needs enough reasoning ability to turn failures into useful configuration changes. This post asks the next question: which reflection models work well for different agent workloads?
We tested five reflection models on four agent workloads. Across the four workloads we tested, no single reflection model consistently came out on top. GPT-5 placed first or second on three of the four workloads, and GPT-5.6 Sol improved the starting agent in all 12 of its runs. Comparing quality with runtime, Sol was on the efficient frontier for every workload and GPT-5.6 Luna for three of four, with Luna the fastest overall. Luna is a practical starting point, but we recommend comparing it with at least one other model on your own workload.
How we tested
We compared GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna, GPT-5, and GPT-5 mini as reflection models, with GPT-5 as the reference. Each run optimized the agent's instructions and generated up to five candidates. We ran each reflection model three times on each workload, for 60 runs in total (4 workloads x 5 models x 3 runs).
Within a workload, only the reflection model changed. The dataset, evaluator, instruction target, and candidate limit stayed the same. Most runs used gpt-4o-mini as the agent model.
|
Workload |
What it tests |
Train / validation |
Evaluator |
|
Facility |
Category, sentiment, and urgency prediction |
15 / 30 |
Deterministic JSON field match |
|
Taxman |
Extraction, calculations, and tax-policy reasoning |
16 / 10 |
GPT-5 LLM judge |
|
Retail Policy |
Applying customer-service rules to structured decisions |
50 / 30 |
Deterministic JSON field match |
|
GSM8K |
Word-problem reasoning and final-answer extraction |
52 / 30 |
Deterministic exact match |
For each reflection model and workload, we report three numbers, each averaged over the three runs:
- Baseline score: the agent's validation score before optimization.
- Final score: the validation score of the best candidate after optimization.
- Score lift: final score minus baseline score.
For example, GPT-5 on Taxman averaged a 70.00% baseline and a 90.00% final score, a lift of 20.00 percentage points. Final score is the better measure for choosing which agent to deploy. Lift shows how much optimization helped, but it also depends on where a run started, since a lower baseline leaves more room to improve.
Findings
1. The best model depended on the workload
No reflection model led across all four workloads.
|
Workload |
Highest final score |
Largest lift |
|
Facility |
GPT-5.6 Sol (79.96%) |
GPT-5.6 Sol (9.11 points) |
|
Taxman |
GPT-5 (90.00%) |
GPT-5 and GPT-5.6 Luna (20.00 points) |
|
Retail Policy |
GPT-5 mini (70.07%) and GPT-5 (70.06%) |
GPT-5 (6.85 points) |
|
GSM8K |
GPT-5.6 Terra (93.33%) |
GPT-5.6 Terra (7.78 points) |
Baseline and final validation score by workload for five reflection models
These results reflect three runs per model on each workload. They can help identify models to test first for similar tasks, and we recommend confirming the choice on your own validation set.
2. Sol improved the baseline in all 12 runs
Sol raised the baseline in all 12 of its runs. Luna improved it in 10 of 12 runs and GPT-5 in 9 of 12, with the remaining runs ending with no lift. This measures how often a reflection model finds a useful change, which is different from how high the final score goes.
3. Sol and Luna were most often on the runtime and quality frontier
Runtime and quality frontier for Facility and TaxmanThe frontier changes by workload, but two models appear on it most often. Sol is on the frontier for all four workloads, and Luna for three of four (Facility, Taxman, and Retail Policy). Terra, GPT-5, and GPT-5 mini each appear on it for two. Luna had the smallest runtime spread across its runs. It was the fastest model on Facility and Retail Policy and closest to the fastest on Taxman.
This comparison uses runtime, not cost. Runtime includes agent and judge calls, and reflection models have different per-token prices, so a cost comparison may rank the models differently.
Final recommendations
- Consider GPT-5.6 Luna as a starting point. It was the fastest model overall, had the smallest runtime spread across its runs, and its final score stayed within about 3 points of the leader on every workload.
- Try GPT-5.6 Sol when maximizing score is the priority. Sol is on the runtime and quality frontier, led Facility, and improved the baseline in all 12 of its runs. GPT-5, the reference model from our previous post, remains useful for development testing.
- Choose using your own validation results. Prioritize final score, use score lift to see how much optimization helped, and use runtime as a tiebreaker when quality is close.
What's next?
We will keep evaluating new reflection models as they become available, update this guidance as the model catalog changes. These recommendations are starting points. The best reflection model still depends on your agent, dataset, evaluator, and optimization target.
Get started
To compare reflection models on your own agent, begin with the agent optimizer overview. For a prompt agent, follow the prompt-agent optimization quick start. For a hosted agent, first make the agent optimizer-ready. Use a representative validation set and an evaluator that measures the behavior you want to improve. Review the configuration changes before you promote or apply a candidate.
Microsoft