Making evaluations cost-efficient

Every evaluation run is, under the hood, a stack of LLM-as-judge calls. Behind Microsoft Foundry's evaluation service, every graded row becomes judge traffic: hundreds of thousands of judge calls a day, billions of prompt tokens in any given two-week window, and roughly a dozen judge calls beneath every batch run. At that scale, the judge's token bill is a systems surface you architect, not a line item.  

This post is where we document our cost-efficiency initiatives as they ship. The first three attack that bill from three different angles. Restructuring judge prompts unlocked prefix caching, and on our agent-evaluation judge traffic the cached share of prompt tokens climbed from under 1% to roughly 70% on a typical day, and has held there for three weeks. TOON (Token-Oriented Object Notation), a compact tabular serialization we adopted for rubric-evaluator verdicts, cuts judge output tokens by roughly 30% relative to JSON on the model families where we enable it. And composite evaluators score five or six evaluation rubrics in a single judge call instead of one call per evaluator, cutting input tokens by 61 to 71% and wall-clock time by 36 to 46% on the workloads we tested. Caching applies across the board; TOON applies specifically to rubric evaluators, whose verdicts are flat, uniform tables, which is exactly the shape TOON is built for (it is not a fit for deeply nested output); composite evaluators are new evaluators you can choose. Every product running evaluations on the platform inherits the first two without changing a line of code, and adopts the third by picking it.  

The key to why the first two levers work is that a judge call spends from three pools priced very differently. Cached input is deeply discounted: 50 to 90% off list depending on the model. Uncached input costs list price. Output is the expensive pool: output tokens list at a multiple of the input rate, typically 4 to 6 times depending on the model, and that asymmetry is industry-wide. At current list prices it holds across flagship, mid, and small tiers alike at OpenAI, Anthropic, and Google, and prefix caching itself is now offered by all three. So optimizing a judge is not "use fewer tokens". It is moving tokens into cheaper pools. The third lever is different in kind: it does not change the price of a token, it stops paying for the same tokens five or six times over.  

 

 

 

Input and output list price per million tokens, three tiers per provider. The output segment dominates every bar, which is why a 30% cut to judge output tokens matters far more than the same cut to input. Public list prices as of writing; they change. 

Initiative 1: Prompt caching for every judge  

Most of a judge prompt never changes  

A production judge call is lopsided. It carries a few thousand tokens of instructions, grading criteria, and few-shot examples that are byte-identical across every row of a dataset, plus a few hundred tokens of per-row data: the {{query}}, {{response}}, and {{tool_calls}} being judged. It returns a structured verdict billed entirely at output prices.  

Two substrate facts turn that shape into a plan. Prefix caches are prefix-match: the server reuses computation only for a leading span that is byte-identical to a prior request, usually above a minimum length and in fixed-size blocks (on Azure OpenAI, a 1,024-token minimum and 128-token blocks). Call this the static-prefix contract: everything above the first dynamic byte is cacheable; everything below it is not, no matter how static it is. And output tokens are generated serially and billed at that 4 to 6 times multiple: every token the judge writes costs you in dollars and in wall-clock.  

The two optimizations are corollaries. Hot prefix: arrange the prompt so that at least the minimum cacheable span of static content, 1,024 tokens on Azure OpenAI, forms the prefix. Tabular verdicts: the output is a uniform table, so stop serializing it as JSON.  

Put the static tokens where the cache can see them  

Our evaluator templates were written the way anyone writes them: instructions, then the data being judged, then the grading criteria and few-shot examples that explain scoring. Pedagogically natural, cache-hostile, and true across the whole evaluator library, not just one kind of judge. The first byte of {{query}} differs on every row, so prefix matching dies a few hundred tokens in and the static tail never caches, on any call, ever. The fix is a content-neutral reorder:  

 

Before the change, the per-row data sat second in every template, so the cache stopped matching a few hundred tokens in and the grading criteria and examples below it never cached. After the change, every static block is stacked first and the per-row data moves to the bottom, so a long byte-identical prefix is served from cache on every row and only a short dynamic suffix is processed fresh.  

 

 

The reorder changes byte order, not semantics. But that is a hypothesis about a position-sensitive model, so we measured score match instead of assuming it: templates were restructured with the originals frozen as controls, and the same reorder shipped to the public catalog evaluators. The pre-merge A/B:  

 

Controlled-environment A/B with varying query, response, and tool-call payloads; input tokens only. Identical scores on every row, and 5 to 16% lower latency, because cached prefill is work the server never redoes. One caveat to carry forward: the larger the dynamic payload, the smaller the cached share of input tokens, so these gains vary with workload shape. 

 

 

 

Prompt caching on agent-evaluation judge traffic, daily, over a recent month: the share of prompt tokens served from cache (blue) and the share of judge calls with at least one cache hit (orange). Nothing cached before the restructured prompts rolled out; roughly 70% of tokens and about four in five calls since. The dips fall on days dominated by unusually large dynamic payloads, not on caching regressions. 

Prompt caching is active and earning in production judge traffic: on agent-evaluation calls, roughly 70% of prompt tokens are served from cache on a typical day, up from nothing before the restructure rolled out, and about four in five judge calls now hit the cache. Both figures have held for three weeks. The day-to-day token share moves with workload mix: days dominated by large dynamic payloads dilute the cached share even when the prefix caches perfectly, which is why the call-level hit rate stays flat on the days the token share dips. Treat any single day as a sample, not a ceiling.  

The per-call anatomy shows the mechanism working exactly as designed. On a call that hits, the cache serves the large majority of the prompt, essentially the entire static prefix, and the median cached span lands on an exact multiple of the provider's 128-token cache block, day after day. Calls that miss are mostly structural: caching requires the prompt to clear the provider's minimum length, and the shortest judge calls sit near that floor.  

What is it worth? That depends on the model's cached-input discount, and the rates are public. At a 90% discount, a cached prompt token costs one-tenth of an uncached one, so a judge call whose prompt is 70% cached pays under 40% of its uncached input price. Current-generation models from all three major providers discount cached input by 90%; older model generations discount less, so the same hit rate is worth more on newer judge models, and caching gains grow as judges move up-tier.  

 

 

 

Cached-input discount off list price, by model. Deep discounts are the norm across providers, with older model generations discounting less. Providers differ on the write side (Anthropic prices cache writes at a premium, Google charges for cache storage), so realized savings also depend on how often a prefix is reused. Public list prices as of writing; they change. 

Initiative 2: TOON for rubric-evaluator verdicts  

The verdict is a table, so send a table  

The output side has no cache to exploit; the only lever is emitting fewer tokens. The mechanism is what JSON does to a uniform table under a modern tokenizer. Our rubric judge returns the same three fields, applicable, score, and reason, for every rubric dimension. Serialized as JSON, every key, quote, and brace is re-emitted per dimension, and every repetition is a completion token billed at the output multiple. Call it the pay-per-key serialization tax: JSON makes the model restate the schema once per row, at output prices. A tabular encoding states the schema once, in a header.  

TOON is that encoding. The same verdict both ways, tokenized with o200k_base:  

JSON verdict, 364 tokens pretty-printed, 310 minified:  

{  

  "dimension_scores": {  

    "factual_accuracy": {  

      "applicable": true,  

      "score": 5,  

      "reason": "The definition is correct and free of hallucinated details."  

    },  

    "response_completeness": {  

      "applicable": true,  

      "score": 3,  

      "reason": "Covers the core question but omits practical usage guidance the user could act on."  

    }  

    // ... six more dimensions, identical shape ...  

  },  

  "overall_reasoning": "..."  

}  

TOON verdict, 217 tokens, the full verdict:  

scores[8]{id | applicable | score | reason}:  

  factual_accuracy | true | 5 | The definition is correct and free of hallucinated details.  

  response_completeness | true | 3 | Covers the core question but omits practical usage guidance the user could act on.  

  clarity | true | 5 | Prose is direct and well-organized; the reader can follow without effort.  

  conciseness | true | 4 | No filler, though a bit denser than strictly necessary.  

  tone_appropriateness | true | 5 | Neutral, professional tone throughout.  

  actionability | true | 2 | No concrete next steps or usage patterns are provided.  

  code_correctness | false | null | Not applicable - no code was requested or produced.  

  example_illustrativeness | false | null | Not applicable - no examples were requested.  

overall_reasoning: response_completeness (3) and actionability (2) drove the verdict downward because no practical usage guidance was provided; factual accuracy and clarity are strong.  

Identical content: 40% fewer tokens than the pretty JSON a model naturally emits, and still 30% fewer than minified. Minification only removes whitespace; the tax is the keys.  

Vetting is what makes a format change shippable, so we budgeted more engineering for it than for the format. Not every model family holds a bespoke output format reliably, and falling back to a JSON-formatted retry would mean paying for the judge call twice, which defeats the purpose. So before any production traffic switched, we validated TOON end to end in our test environment, running thousands of evaluator runs across different rubric sizes and input sizes, with every input and rubric combination executed at least three times to iron out statistical noise and hallucinations. Model families that held the format 100% of the time were enabled; families that did not stay on JSON. In production, the decoder still tries TOON first and falls back to parsing JSON if a model ever ignores the instruction; the parse fallback costs nothing extra, the result is reshaped into exactly the structure the JSON path produces, and both paths pass identical strict validation. The worst case is "row scored via the JSON parse path, counter ticks", never "wrong score recorded silently". That fallback has fired zero times across every production TOON call to date, which doubles as the compliance result: the vetted models hold the format.  

Measured in production: roughly 30% fewer judge output tokens  

Savings numbers are easy to get wrong, so we measured this one conservatively. The evaluators running on TOON are not the same set as the evaluators running on JSON, and different evaluators write verdicts of different lengths, so simply comparing all TOON traffic against all JSON traffic would measure which evaluators happened to be in each group, not the format. We therefore compared the two formats three ways over more than 10,000 production runs, each way holding more constant than the last:  

 

All three land at roughly 30% fewer output tokens per judge call, with prompt tokens unchanged. As a final check, we compared JSON against itself: the same evaluators, still on JSON, before and after the rollout window. That moved by well under a tenth of a percent, which tells us nothing else in the environment shifted during the window, so the reduction belongs to TOON.  

 

 

 

Judge output tokens per call in production rubric-evaluator traffic, indexed to the JSON arm within each model family. Three of four families show a 29 to 46% reduction, including a large reasoning model. The small reasoning families (mini and nano) show no gain and do not hold the format reliably, so they stay on JSON. That is exactly why TOON enablement is a per-model-family decision made by vetting, not by model class. 

Output is roughly 15% of a judge call's total tokens, so a 30% output reduction is about 4 to 5% of the whole token count. But output carries the 4 to 6 times price multiple and is the serial part of the call you wait on, so the output reduction is what shows up in cost and in perceived speed. Caching's latency win is directly measured (5 to 16% in the A/B above); TOON's should track the output reduction for the same serial-decode reason, and wall-clock measurement is in progress.  

Initiative 3: Composite evaluators, one judge call for many evaluators 

Why: the same trajectory, sent once per evaluator  

A single agent trajectory is rarely judged on one dimension. A quality pass alone may check fluency, coherence, intent resolution, task adherence, groundedness, and task completion; a tool-use pass adds tool selection, call accuracy, input accuracy, call success, and output utilization. In a conventional pipeline each of those is its own evaluator with its own judge call, and every call receives the same messages, tool calls, tool results, and tool definitions. The shared context is paid for once per evaluator.  

Caching cannot help here. The trajectory is the dynamic suffix, the part of the prompt that differs on every row, so it sits below the static-prefix contract by definition. The cost grows with the number of rubrics multiplied by the size of the shared context, and long agent trajectories make that multiplication expensive.  

What: send the shared context once, score every rubric  

A composite evaluator inverts the pipeline: send the trajectory once and ask a single judge call to score several rubrics together. We built two. An Output Quality composite scores the six quality rubrics above in one call, and a Tool Use composite scores the five tool-use rubrics in one call. Each returns one score per rubric, the same shape the individual evaluators produce, so downstream thresholds, dashboards, and pass/fail logic do not change.  

 

 

 

Conventional evaluation sends the full context to each evaluator in a separate judge call. A composite evaluator sends it once and returns every rubric's score from a single call. 

How we validated it  

We compared parallel individual evaluators against one composite call per row on matched samples of single-turn agent conversations (the judge scores the final response with the full conversation history as context), one sample per workload, and repeated the runs to confirm the results were stable rather than lucky. Trajectory lengths in the samples clustered around a few thousand tokens with a long tail, and the longer the trajectory, the larger the absolute saving, exactly as the mechanism predicts. On quality, frontier judge models held stable scores across several high-value rubrics. A few rubrics proved workload-sensitive, and for those we recommend keeping a targeted individual evaluator alongside the composite.  

Impact: five or six calls become one  

 

Composite evaluation against one call per evaluator, per evaluated row; ranges span the quality and tool-use workloads. 

 

 

Per evaluated row, composite evaluation needed 5 to 6 times fewer model calls, cut input tokens by 61 to 71%, cut output tokens by 47 to 64%, and cut measured wall-clock time by 36 to 46%. Input falls because the shared context is sent once. Output falls largely because one call carries one round of reasoning and formatting overhead for six rubrics instead of six rounds, while still emitting every per-rubric verdict. Wall-clock falls because five or six serial decodes collapse into one. The results support composite evaluation as the cost-efficient default, with targeted individual evaluation reserved for the workload-sensitive rubrics.  

The three initiatives compose. Caching makes the static part of every judge call cheap; composite evaluation stops re-sending the dynamic part; and where a verdict is a flat table, TOON shrinks what the judge writes back.  

Applying this to your own judges  

• Measure your prompt length first. Below the provider's caching minimum, restructuring is a no-op; just above it, every block of static prefix you consolidate is money.  

• Put every static byte before the first dynamic byte. The cache stops matching at the first difference. Grading criteria and examples placed below the data being judged never cache, no matter how static they are.  

• TOON pays for flat, uniform, multi-row output, where one header amortizes across rows. Deeply nested output and single-scalar judges have little key repetition to eliminate.  

• Vet per model family, and keep a costless fallback. Test the format offline at scale, enable only where compliance is total, keep a parse-side fallback with identical validation, and never architect a fallback that requires a second paid call.  

• Batch rubrics that share context. If several evaluators read the same trajectory, one composite call that scores them together removes the duplicated context, and the saving grows with trajectory length. Keep individual evaluators for the rubrics your workload is sensitive to.  

• Validate savings with controlled comparisons, including a null check. Compare like populations, and measure your unchanged configuration against itself over the same period. If the unchanged setup moves, you are looking at drift, not savings.  

What's next  

This post will grow as further initiatives ship; each will get its own numbered section above. For the three so far, TOON already runs in every region and the composite evaluators are available to use; the frontier now is model families, coverage, and measurement. That means extending the TOON-vetted set as new judge models arrive, putting a wall-clock number on TOON's latency win, pushing the remaining short-prompt evaluators past the caching minimum, where controlled tests project cache-hit gains of 60% or more each, and extending composite evaluation to more rubric groupings as the quality checks allow. Caching is already spreading: beyond the agent-evaluation judges, the rest of the judge traffic on the same path went from nothing cached to roughly 40 to 50% of prompt tokens over the same month.  

From the outside, one of these changes looks like reordering text, another looks like swapping braces for pipes, and the third looks like asking one question instead of six. Shipping them safely is what took the work: weeks of controlled testing, score-fidelity checks on every changed template, thousands of vetting runs per model family, and production measurement designed so we would catch any regression before customers could. That investment lives at the platform layer, so every evaluation run on the platform inherits it: roughly 70% of agent-evaluation prompt tokens now arrive from the discounted pool, rubric-evaluator judges emit roughly 30% fewer output tokens, teams that adopt the composite evaluators send each trajectory once instead of five or six times, and nobody had to change a line of code to get the first two. The token bill arrives either way. The only question is whether you architected it.  

Get Started Today

• Start building in Microsoft Foundry: ai.azure.com
• Check out Build session on Foundry observability covering rubric evaluators aka.ms/build26-BRK252
• Read the docs: Rubric evaluators - Microsoft Foundry | Microsoft Learn
• Join the community: aka.ms/ai/discord

Dive deeper into the techniques covered:

Composite evaluators documentation - Learn how to implement batched rubric evaluation

TOON format specification - Explore the tabular serialization format