Blog Post

Microsoft Foundry Blog
14 MIN READ

Your Agent Found the Right Schema. Then Ignored It.

shihyaolin's avatar
shihyaolin
Icon for Microsoft rankMicrosoft
Sep 15, 2026

We changed one assumption in a standard evaluation and lost 40 points. Here is the failure mode, and the fine-tuning recipe that recovered 27 of them on Microsoft Foundry.

Authors: Shih-Yao Lin, Rayan Khoury, Pratyush Mishra

The assumption nobody tests

Most agent evaluations quietly assume something that never happens in production: that the model is handed exactly the right information, and nothing else.

We changed that one assumption — and watched a model drop from 58% to 18%.

Retrieval was not the problem. The correct information was sitting in the prompt. The model just didn't use it.

TL;DR — Standard agent evaluations hand the model exactly the right schema. Real retrieval does not. Changing only that cost 40 points of accuracy, and neither a larger model nor better prompting recovered it. A $150 fine-tuning job won two-thirds of it back and beat prompted gpt-5 by 28 points at a twenty-fifth of the per-turn cost. Steps 2–4 of the runbook below reproduce the gap on your own registry and need no fine-tuning.

We measured this on schemas, but if you build agents that retrieve tools, functions, or MCP servers into a prompt, the shape of the problem is the same. More on that at the end.

Here is the setup. In schema-guided dialogue agents, the model is given a schema — the list of fields it is allowed to fill — and must return a filled state. Benchmarks put the one correct schema in the prompt and report strong numbers: 86.4% and 74.7% on SGD [2, 3, 4]. But a deployed assistant has no idea which schema is correct. It retrieves candidates from a registry, so the prompt ends up holding the right schema plus whatever else the retriever dragged in.

So we changed exactly one thing: how the schema reaches the prompt. Same dataset, same metric, same model. We call the realistic condition RetDist (retrieval + distractors) — BM25 top-1 plus two random distractor schemas from the same registry, shuffled. Note what this does not guarantee: BM25 ranks the correct schema first on only 65.4% of turns, so on about a third of them the right answer is not in the prompt at all. That matters later.

The score we measure is joint goal accuracy (JGA): the fraction of turns where every field is exactly right. It is unforgiving on purpose — one wrong field fails the turn, which is also how a user experiences it.

So if a turn needs three fields and the model gets two of them right, that turn scores zero. This matters for reading the numbers below: 58 does not mean "58% of fields correct" — it means 58% of turns were completely correct. Slot-level accuracy on the same runs is far higher, and far less honest about what a user would experience.

58.0% with the gold schema. 18.3% under RetDist. Same model — gpt-4.1-mini, same weights, same dialogues, same metric. Only the prompt assembly changed.

Figure 1. The only thing that changes is how the schema reaches the prompt.Standard evaluation (A) hands gpt-4.1-mini the one correct schema, so the task reduces to filling in fields. A deployed assistant (B) retrieves candidates from the 21-service test registry, so the correct schema arrives alongside confusable ones. Same dialogue, same weights, same metric — 58.0 JGA becomes 18.3.

It is not a retrieval problem

The obvious explanation is that retrieval simply missed. It didn't.

We split every turn by whether the correct schema was actually in the prompt. On the turns where it was present — retrieval did its job — gpt-4.1-mini still only reached 24.0%.

 

Figure 2. Retrieval is not the bottleneck.Restricted to the 27,085 turns where the correct schema was verifiably present in the prompt, the base model still reaches only 24.0 JGA. Fine-tuning on the same distribution lifts the same stratum to 62.7.

So the model isn't failing to find the schema. It's failing to follow it. It falls back on field names memorised during pre-training instead of reading the schema in front of it. We call this Schema-Following Collapse.

This is why better retrieval would not have saved us. Prior robustness work on this benchmark varied the schema's wording [5]; the failure here survives even perfect retrieval.

 

Bigger models don't fix it

Figure 3. Scale and prompt engineering do not close the gap.Every prompted configuration — Llama-3.3-70B, gpt-5, and gpt-5 again with few-shot examples and strict decoding — lands inside the same 15–19 JGA band, alongside the un-tuned gpt-4.1-mini. Only fine-tuning leaves it. All bars are SGD-200 (n=1,810) under RetDist; the dashed line is the gold-schema ceiling.

 

Prompt engineering moved gpt-5 by 2.8 points. Everything stayed in a 15–19% band. This is not a capability problem you can scale or prompt your way out of.

 

Two caveats on that chart, because it names names.

This is a prompting stress test, not a capability ranking. All three models see one fixed deployment prompt. Give gpt-5 a different prompt, a schema-selection tool, or permission to ask a clarifying question and the number changes. The finding is that no prompted configuration escaped the band — not that any particular model is weak. gpt-5 scoring 16.6 here says something about the task, not about gpt-5.

And no, we did not fine-tune them. At the time of these runs gpt-5 exposed no supervised fine-tuning path on Foundry — it is reasoning-mode and fixed at temperature 1.0 — and fine-tuning a 70B model costs orders of magnitude more than the $150 that makes this recipe worth writing about. So this compares a fine-tuned small model against prompted large ones. That is the choice a team on a budget actually faces, but it is not a claim about what those models could do if they were trained.

What did work: train on the noise

RAFT (Retrieval-Augmented Fine-Tuning) [1], originally proposed for document question answering, trains a model with the evidence it should use alongside realistic distractors it should ignore. Unlike RAG, which adds retrieval at inference time, RAFT changes the training distribution, so the model learns how to behave after retrieval has already happened — including how to behave when retrieval brought back the wrong thing.

We adapt that idea from retrieved documents to retrieved schemas. We call the recipe SchemaRAFT: build the fine-tuning examples to look like the deployment prompt, including both the correct schema and confusable alternatives. This is a task-specific application of RAFT, not a new training algorithm.

Figure 4. The recipe.Every training prompt is assembled the way the serving prompt is assembled — the correct schema plus two confusable distractors. One in five examples contains no correct schema at all and is labelled with an empty state, which is what teaches the model to abstain rather than guess.
  • 19,300 chat-format examples (~2,100 prompt tokens, ~40 completion tokens each)
  • each example: 1 gold schema + 2 random distractors, exactly as at serving time
  • 20% of examples omit the gold schema entirely and are labelled {} — this teaches the model to withhold output rather than invent slots
  • 3 epochs, ~124M training tokens, one fine-tuning job on Microsoft Foundry

Here is one training example, trimmed. The system message carries the schemas, the user message carries the dialogue, and the assistant message is the only part the model is scored on.

system

You are a dialogue state tracker for task-oriented conversations.
...
Rules:
- Output ONLY a valid JSON object on a single line.
- Use only slot names that appear in the schemas below.
- If no relevant slots have been mentioned, output: {}

Service schemas (one is relevant; others may be distractors):
Service: Hotels_4
  Accommodation searching and booking portal
  Slots:
    - location [free-text]: City or town where the hotel is located
    - number_of_rooms [categorical] (values: 1, 2, 3): Rooms to reserve
--------------------------------------------------
Service: Restaurants_2
  Restaurant search and reservation service
  Slots:
    - city [free-text]: City where the restaurant is located
    - cuisine [free-text]: Type of food served
    - party_size [categorical] (values: 1, 2, 3, 4): Number of seats
    ...
--------------------------------------------------
Service: Buses_3
  ...

user

Conversation history:
User: I need a table for two in San Francisco.
System: What kind of food?
User: Italian, somewhere moderate. Around 7 pm tomorrow.

Extract the current belief state for service: Restaurants_2

assistant — this is the label, and the only tokens in the loss

{"city": "san francisco", "cuisine": "italian", "party_size": "2",
 "price_range": "moderate", "time": "19:00"}

Two things are worth noticing.

The model is told which service it needs. The name sits at the end of the user message and matches one of the three schema headers verbatim. Choosing the right schema is string matching. The only real task is to then stay inside it — and that is what collapses. A base model reaches for location from Hotels_4 because "San Francisco" and hotels co-occur constantly in pre-training, even though Restaurants_2 is named right there and has a city field. The gap is not a search failure. It is a grounding failure, and we made the search step as easy as it could possibly be.

Nothing in the loss punishes using a distractor's field. There is no auxiliary objective, no reranker, no constrained decoding. It is ordinary next-token prediction on the assistant message. The model sees ~19,300 examples in which three schemas are present and the answer only ever draws on one of them, and that is the entire training signal.

The inference prompt is the same object; only the assembly changes. At serving time BM25 chooses the schemas instead of the data builder, so the correct one is present 65.4% of the time rather than 80%, and the assistant message is generated instead of given.

Result on gpt-4.1-mini — the same model from Figure 1, now fine-tuned: 18.3 → 45.0 JGA, recovering about two-thirds of the gap to the gold-schema ceiling. On gold-in-prompt turns it goes 24.0 → 62.7.

We re-ran the identical recipe on gpt-4o-mini with no code changes and got +28.0 points (14.5 → 42.5). It's a recipe, not a lucky run.

The economics

Measured on the same deployment, per turn:

 LatencyRelative costJGA
gpt-5, prompted4,401 ms24.4×16.6
gpt-4.1-mini + SchemaRAFT1,095 ms1.0×45.0

Four times faster, roughly 25× cheaper per turn, and 2.7× the accuracy. The one-time training job costs about $150 and pays for itself after roughly 18,000 served turns.

 

Where else this shows up

We measured this on dialogue state tracking, but nothing in the failure is specific to dialogue. The ingredients are generic: several semantically similar capability descriptions retrieved into one prompt, and a model asked to commit to one of them.

It is worth being precise about how close the analogy is. An SGD service schema is a name, a description, and a list of typed parameters with their own descriptions. So is an OpenAI tool definition. So is an MCP tool listing. These are the same object wearing different labels, and the model is doing the same thing with all of them: reading several near-identical interface descriptions and picking one.

That pattern is everywhere in agent systems today — function calling with a large tool catalogue, MCP server selection, tool and plugin routing, multi-agent handoff, enterprise API registries.

What we did not test is whether the collapse reproduces when the decoding target is a tool call rather than a belief state. So treat the extension as a hypothesis, not a result. The cheap move is to measure it on your own registry — which is exactly what the runbook below does, and it needs no fine-tuning.

When this won't apply to you

This gap needs candidates that are genuinely confusable. SGD defines 45 services across 20 domains with heavily overlapping field names — the slot city alone appears in 9 of them, and 15 services carry some variant of it (from_city, to_city, and so on). If the schemas in your registry are clearly distinguishable, a model that retrieves the wrong one tends to notice, and you won't see a collapse this large.

Measure before you fine-tune.

What we haven't answered yet

We are reporting a finding, not a finished agenda. The open questions we think matter most, roughly in order of how cheaply you could settle them:

  • Does the collapse survive a better retriever? We used BM25 top-1 on character 3–5-grams — deliberately weak, because that is what a registry lookup usually is. A dense retriever with a cross-encoder reranker would reduce how often the wrong schema is in the prompt, but Figure 2 says it cannot fix the turns where the right schema was already there.
  • Does it reproduce when the output is a tool call? Everything here decodes a belief state. A function-call target has its own constrained decoding and its own schema-validation layer, and either could mask or amplify the effect. Untested.
  • Can schema selection replace fine-tuning? Making the model pick one schema first and fill it second is a much cheaper intervention than training. It also moves the failure rather than removing it — the picking step is the same discrimination problem. Worth measuring before anyone spends $150.
  • What is the right abstention ratio? 20% distractor-only examples is a round number that worked, not a tuned one. Abstention also turned out to be a separate skill from schema-following: fine-tuning lifted the gold-present turns enormously and barely moved behaviour on turns where the correct schema was genuinely absent.
  • Does one fine-tune transfer to a different registry? The recipe transfers across base models — we re-ran it unchanged on a second one for +28.0 points. Whether a model trained on one organisation's schemas follows a different organisation's schemas is a separate question, and the answer decides whether this is a one-time cost or a per-registry cost.

If you run the runbook on your own registry, we would like to know what number you get — especially if you get no gap at all. Negative results on the "when this won't apply" boundary are the most useful thing anyone could send us. Leave a comment on this post with how many schemas are in your registry, how confusable they are, and the two numbers from Steps 2 and 3. That is enough for us to tell whether it is the same failure mode.

Try it yourself (~20 minutes)

The step numbers match the sample's README, so you can move between the two. Steps 2–4 reproduce the gap and need no fine-tuning; they cost only inference on about 400 turns. Step 5 is the fix, written out so you can read the recipe without running it.

Prerequisites: Python 3.10+, a Microsoft Foundry deployment of a small chat model (we used gpt-4.1-mini), and the Foundry User role on that resource. The sample is keyless — it authenticates with DefaultAzureCredential, so there is no API key to paste anywhere. The sample README has the full access walkthrough.

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/python/finetuning/schemaraft-dst
python -m venv .venv; .\.venv\Scripts\Activate.ps1
pip install -r requirements.txt

Copy-Item .env.example .env   # set AOAI_ENDPOINT and DEPLOYMENT
az login

Step 0 — confirm the checkout works. Offline and free: no Azure calls, no tokens. If this fails, nothing after it will work.

python selftest.py            # 42 offline checks

Step 1 — get the data. The test split only; training data is not needed until Step 5.

python download_sgd.py --splits test

Step 2 — the number papers report.

python eval_jga.py --mode oracle `
    --deployment <your-deployment> --n 200 `
    --output-dir results/oracle

Step 3 — the number your users get. One flag changes.

python eval_jga.py --mode retrieved_distractor `
    --deployment <your-deployment> --n 200 `
    --n-schemas 1 --n-distractors 2 `
    --output-dir results/retdist

Compare results/*/summary.json. Expect roughly 58 vs 18.

Step 4 — confirm it isn't retrieval. Score only the turns where the correct schema really made it into the prompt:

python goldpresent_strata.py --results results/retdist

They stay far below the oracle number.

Step 5 — the fix, for reference. Don't run this one unless you mean it: it is a real fine-tuning job (~$150, 6–10h in queue). It is here so the recipe is something you can read, not just a link.

python download_sgd.py --splits train
python build_sft_data.py --split train --n-distractors 2 --gold-absent-frac 0.2
python submit_finetune.py --train-file data/schemaraft_train.jsonl --epochs 3

The job writes finetune_job.json, so --resume reattaches instead of paying for a second run. Expect it to sit in the queue for a few hours before training starts; you can watch both phases in the Foundry portal's fine-tuning section. Once the status reads succeeded, the new model appears alongside the base models when you deploy. Point --deployment at it and re-run Step 3 — same command, same data, same metric. That is the whole experiment.

What to do with this

If you are shipping an agent that retrieves anything into its prompt — schemas, tools, MCP servers, API definitions — the cheap version of this post is one afternoon:

  1. Re-run your own eval with distractors in the prompt. Not a better retriever, not a bigger model — just stop handing the model the right answer. Steps 2–4 above do exactly this and cost only inference.
  2. If the number holds, you are done. Your registry is not confusable enough for this failure mode and you should not spend money on it.
  3. If it collapses, you now know that prompting won't save you — that is what Figure 3 bought you — and the fix is a supervised fine-tuning job on the same deployment you are already using.

The part we would push back on is skipping step 1. The gap in this post was invisible to a standard evaluation, survived perfect retrieval, and survived gpt-5. It would have shipped.

 

Appendix — exact setup

Collected here so the narrative above stays readable and the numbers above stay auditable.

Data. Schema-Guided Dialogue (SGD) [2]: 45 services across 20 domains. The test-time retrieval registry is the 21 services in the test split; 15 of them never appear in training, which is the point of the benchmark. Full test split: 39,324 turns.

Retrieval (RetDist). BM25 over character 3–5-grams of the schema text, top-1, plus two distractor schemas sampled uniformly from the same registry. The model never sees a retrieval score — BM25's job is selection, not ranking signal. On the test split the gold schema is top-1 for 65.4% of turns; random distractor sampling surfaces it on some of the rest, leaving 31.1% of turns where the correct schema is genuinely absent from the prompt.

Evaluation populations. Three, and they must not be mixed:

PopulationnUsed for
SGD-2001,810 turns (200 dialogues)Figure 3, the economics table, all frontier comparisons — frontier models are too expensive to run on the full split
Gold-in-prompt stratum27,085 turnsFigure 2
Full test split39,324 turnsFine-tuning effects only

The gold-in-prompt stratum is every turn where the correct schema reached the prompt — whether BM25 ranked it top-1 (65.4% of turns) or a random distractor draw happened to surface it. The remaining 12,239 turns (31.1%) are the ones where it is genuinely absent.

Models. Base and fine-tuned: gpt-4.1-mini. Prompted baselines: gpt-5, Llama-3.3-70B. All three are served from the same Foundry deployment, which is what makes the latency and relative-cost columns comparable. The prompted baselines were not fine-tuned: gpt-5 exposed no SFT path on Foundry at the time of these runs, and fine-tuning a 70B model is far outside the cost envelope this post is about.

Fine-tuning. ~19,300 chat-format examples, ~2,100 prompt tokens and ~40 completion tokens each; every example carries one gold schema plus two random distractors; 20% are distractor-only and labelled {}. Supervised fine-tuning, 3 epochs, ~124M training tokens, one job on Microsoft Foundry.

Cost basis. Latency and relative cost are measured per turn on that same deployment, not quoted from a price sheet. The ~$150 figure is the one-time training job; the ~18,000-turn break-even is that cost divided by the measured per-turn saving against gpt-5.

Reproducibility. Steps 2–4 of the runbook reproduce the gap and cost only inference. Step 5 reproduces the fix and is a real fine-tuning job.

References

  1. Zhang, T., Patil, S. G., Jain, N., Shen, S., Zaharia, M., Stoica, I., and Gonzalez, J. E. RAFT: Adapting Language Model to Domain Specific RAG. COLM, 2024.
  2. Rastogi, A., Zang, X., Sunkara, S., Gupta, R., and Khaitan, P. Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset. AAAI, 2020. (the SGD benchmark used throughout)
  3. Zhao, J., Gupta, R., Cao, Y., Yu, D., Wang, M., Lee, H., Rastogi, A., Shafran, I., and Wu, Y. Description-Driven Task-Oriented Dialog Modeling. ACL, 2022. (D3ST — 86.4% JGA under an oracle schema)
  4. Bang, N., Lee, J., and Koo, M.-W. Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System. Findings of ACL, 2023. (TOATOD — 74.7% JGA under an oracle schema)
  5. Lee, H., Gupta, R., Rastogi, A., Cao, Y., Zhang, B., and Wu, Y. SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems. AAAI, 2022. (prior robustness probe — stylistic schema variants, still oracle delivery)
  6. Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR, 2024.
Updated Oct 21, 2026
Version 2.0
No CommentsBe the first to comment