Skill Optimization in Microsoft Foundry
Authors: Abi Komma, Charles Kim, Vivek Bhadauria, Morteza Ziyadi, Kavitha Budumuri, Vinutha Karanth, Lori Li, Shivank Goel, Yingying Zhao, Yi Fang
Each time an agent runs, it leaves a trace: the request, its reasoning, its tool calls, and its answer. When an agent keeps making the same mistake, the traces usually show it. Turning those traces into a fix is still slow, manual work.
Here is a real example. A retail support agent got this message: “I want to cancel order #CN-6633 placed 4 hours ago. Total $33.00. Status still says ‘processing’.”
The agent said the cancellation was free and promised a full $33.00 refund. The policy in its own system prompt says a $5 fee applies after two hours, so the refund should have been $28.00. Nothing was logged as an error, and the response time was normal. The only record of the mistake was a trace, and other traces from the same week had the same mistake.
To fix this by hand, someone has to find the bad runs, see the pattern, and write an instruction that fixes it without breaking the runs that already work. Most teams only do this after an incident. In our earlier posts (part 1, part 2), agent optimizer improved an agent’s instructions once it was given a dataset and an evaluator. Building a labeled dataset and a reliable evaluator is often the hardest part for teams, so we said our next step was to lower that barrier: start from the production traces teams already have and turn them into skills that improve the agent.
On internal benchmarks modeled on common enterprise agent tasks, the skills Skill Optimization learned improved agents by 9.2 points on average and by up to 19.9 points on questions the agents had never seen. Even without an evaluator, the rubric that Skill Optimization generates pointed in the right direction in 31 of 35 runs.
What Skill Optimization does
Skill Optimization reads what an agent did in production, along with any feedback on those runs, and looks for three things: mistakes that keep coming back, answers that only just passed, and habits worth keeping. It turns those lessons into a new or updated Agent Skill, a plain file of instructions that lives alongside the agent’s system prompt. Skills load progressively, so the agent sees only a short description until a task calls for the full instructions. Because the skill sits on top of the prompt the team owns rather than rewriting it, every change can be read and reviewed, nothing is fine-tuned, and the agent keeps its model and tools.
Figure 1. Skill Optimization turns traces into new or updated skills, tests and checks them, and saves each as a new version. Attaching them to the agent is always the developer’s decision.
Each run looks at a limited sample of recent traces and at any skills the agent already uses, so it can build on them rather than duplicate them. From that, it plans which skills to add or update, often several in one pass, and drafts them with a reason for every change. If evaluation is set up, the candidate is then scored against the current agent on traces it didn’t learn from, to show whether it actually helps. Every candidate also goes through quality checks and a static security scan. An accepted candidate is saved as a new skill version that can’t be edited later, so there is always a clear record of what changed and when.
The final step stays with the agent’s developer. Skill Optimization never attaches a skill to a production agent or deploys it automatically. The developer decides whether to use it and can roll back to the previous version if a new one doesn’t work out.
Why it matters
The main benefit is a lower cost to get started. Teams don’t need a curated dataset or a hand-built evaluator before the first run, because Skill Optimization learns from traces already in Application Insights and the feedback already being collected. The second benefit is trust. A run doesn’t just produce a file. It explains what kept going wrong, why each skill was created or updated, and which traces each lesson came from, so a reviewer can check the reasoning instead of guessing. The third benefit is safety. Candidates are tested before anyone adopts them, checked for quality and security, and published as fixed versions, so nothing changes in production without a person approving it. The fourth benefit is continuous improvement. Because each run learns from recent production traces, teams can run it again as traffic changes, so the agent’s skills keep pace with what users actually ask instead of drifting away from it.
How it works, in brief
The pipeline has three stages: learn from sampled traces, decide what to change, and write the skill. Figure 2 follows the cancelled order through all three.
Figure 2. A trace that needs improvement becomes a lesson, joins the run’s plan, and ends up as the “Cancel requests” section of a skill.
Every lesson needs evidence. Each trace is labeled acceptable or needs improvement, and an analyst model writes a lesson from it that must quote the trace. Quotes we can’t find in the trace are dropped, and so are lessons left with no quotes. A wrong lesson does more harm than a missing one.
Learning from contrasts. In production, the same question rarely comes twice, so there is seldom an acceptable answer and a weaker one to the same task to compare. Instead, we analyze each trace next to traces from the same batch that had the opposite outcome. The analyst is asked to look past the topic and focus on how each answer was put together. This works best when the content of an answer is right but its form is wrong, for example a missing field, an amount written as text, or steps done in the wrong order.
The smallest change that fixes the pattern. A planner looks at all the lessons and the agent’s existing skills together. It decides which skills to update and which to create, and can change several at once. It prefers updating to creating, keeps the set small, and changes nothing when the lessons are weak or already covered. Running it again on the same traces doesn’t rewrite the skills.
General, not specific. Similar lessons are merged into one rule. When two lessons conflict, the one with more support wins. Details that only apply to one task are removed, so the skill doesn’t memorize single cases. In our example, the agent already had the fee rule in its prompt. The skill turned it into a step to follow at the moment of answering, in the exact fields the evaluator checks.
The signal: a custom evaluator, or a steerable rubric
What the skill learns depends on the signal it gets. The best signal is the team’s own definition of success, written as a custom evaluator. Its verdicts label the traces, and its notes explain what needs improvement.
If the production agent doesn’t have an evaluator yet, Skill Optimization generates a rubric: a short list of pass/fail checks written from the agent’s instructions, its tools, and a sample of its traces. An LLM judge then uses the rubric to score answers. Developers can add a short steering note, such as “focus on refund calculations” or “don’t break what already works.” The note guides both the rubric and what the skill learns.
Benchmarks
To see how well this works, we ran Skill Optimization on internal benchmarks, adapted from public datasets, that each come with a code-based checker: a script that knows the correct answers and gives the true score. IFBench tests instruction following, Retail-Policy tests store policy decisions, and Airline-Tools tests booking decisions made with tools. We used three agent models: gpt-4o-mini (with gpt-5 as the optimizer model), and gpt-5.4-nano and gpt-5.6-luna (with gpt-5.6-sol as the optimizer model).
Every score comes from held-out test questions the optimizer never saw, and each one is the average of 3 evaluations. Gains are measured against the agent without any skill, averaged over all of its no-skill runs. We treat changes within ±3 points as noise. We compared the two setups a customer would use: learning from a custom evaluator (here, the benchmark’s checker), and learning from a generated rubric guided by a steering note.
Figure 3. Gain over the agent without a skill, for each agent model and benchmark. Anything to the right of the gray band is a real gain.
With the better of the two setups, every agent and benchmark improved, 6 of 7 by more than the noise margin, and the average gain was 9.2 points. The biggest jump took gpt-4o-mini on IFBench from 37% to 57%. On Airline-Tools, the same agent went from 81% to 93%. Even gpt-5.6-luna, which already scores 75–88%, gained up to 6.6 points.
| Agent model | Benchmark | Without skill | With skill (best) | Gain with custom evaluator | Gain with rubric + steering note |
|---|---|---|---|---|---|
| gpt-4o-mini | IFBench | 36.7% | 56.6% | +15.8 | +19.9 |
| gpt-4o-mini | Retail-Policy | 69.2% | 77.6% | +4.9 | +8.4 |
| gpt-4o-mini | Airline-Tools | 81.2% | 93.4% | +12.2 | +6.0 |
| gpt-5.4-nano | IFBench | 76.2% | 84.0% | +5.1 | +7.8 |
| gpt-5.4-nano | Retail-Policy | 78.1% | 85.0% | +6.9 | +4.8 |
| gpt-5.6-luna | IFBench | 75.0% | 81.6% | +2.4 | +6.6 |
| gpt-5.6-luna | Retail-Policy | 88.1% | 90.6% | +2.4 | −2.8 |
Change in points (0–100 scale) on held-out test questions. The rubric column shows the best of the steering notes we tried.
What we learned
- Skills fix procedure, not reasoning. The largest gains came where the agent knew what to do but didn’t do it every time: following output rules, applying a store policy, or calling the right booking tools. The more room an agent had to improve, the bigger the gain. When a task depends on deep reasoning, a skill helps less. On a subset of LiveMath, a hard math benchmark, gpt-5.6-luna started at 37% and gained at most 3.9 points. For tasks like this, a more capable agent model is the better fix.
- A skill can patch instructions, but not a broken agent. A skill can fill gaps in an agent’s instructions, such as a missing step or an unclear policy, as long as the evaluator or rubric catches the mistake. It can’t repair a broken tool, supply data the agent has no access to, or make a model capable enough for the task. Fix those first, then use Skill Optimization to make the agent more consistent.
- A custom evaluator works best, and a generated rubric is a good stand-in. Learning from a custom evaluator (here, the benchmark’s checker) held or improved the true score. Without one, the generated rubric still pointed the right way: in 31 of 35 runs, it moved in the same direction as the true score, or the true change was within noise (Figure 4). It is less precise about how much. In one run, the rubric rose 4.0 points while the true score rose 19.4; in another, it rose 16.7 while the true score rose only 2.4. That’s expected, since a rubric written from traces can’t know every rule in a hidden checker.
Two things make the rubric more reliable. A short steering note generally beat the rubric alone. Building the rubric from a larger set of traces than the skill learns from, which can be just the most recent ones, can also help it capture more of what matters.
Our advice: teams with an evaluator they trust should use it. Otherwise, review the generated rubric to make sure it measures what the developer cares about, use it to compare candidates, and confirm the winner on real traffic.
Figure 4. Rubric change against true change for every rubric-based run on the same agents and benchmarks as the table. Each agent and benchmark pair was run with several steering notes, so there are more dots than table rows; the table shows the best one. Dots in the green areas agree. The four circled runs disagree. Because the rubric is scored within each run, both changes here are measured from that run’s own starting score, so they can differ slightly from the gains in the table, which are measured from the average no-skill score.
- Strong agents need a light touch, and that’s why the developer stays in control. gpt-5.6-luna already scores 88% on Retail-Policy, so it has few mistakes left. Without a steering note, the rubric rewarded caution, and the skill made the agent refuse or escalate cases it used to get right, costing 18.1 points. A steering note cut that loss to 2.8 points, and learning from the custom evaluator gained 2.4, so the problem came from what the rubric rewarded, not from skills themselves. This is exactly why Skill Optimization tests every candidate and never attaches it automatically: a skill is kept only if it helps.
Get started
Skill Optimization is in public preview in Microsoft Foundry. To get started, see the Skill Optimization documentation. To improve the agent’s instructions and tool descriptions as well, see the Agent optimizer overview. The two work well together.
We’d love to hear what works and what doesn’t on real agents. Feedback during the preview will shape the feature.
Skill Optimization is in preview, so capabilities can change. The plans described here are work in progress, not a commitment to deliver any feature or timeline. Benchmark numbers come from internal runs on benchmarks adapted from public datasets, and results will vary by agent, data, and evaluator. Microsoft Foundry and Azure are trademarks of the Microsoft group of companies.
Skill Optimization builds on a growing body of research on agents that learn from their own experience: distilling many trajectories into one skill (Trace2Skill), turning feedback into written guidance that improves over time (ACE), and keeping a change only when it holds up on unseen data (EvoSkill). It also extends Microsoft’s own work on optimizing prompts and skills from feedback (PromptWizard, Trace, SkillOpt).