Blog Post

Microsoft Foundry Blog
6 MIN READ

Auto-Generated Rubric Evaluators: Building Context-Aware Evaluators for AI Agents

Shuo's avatar
Shuo
Icon for Microsoft rankMicrosoft
Jun 18, 2026

Pre-built evaluators may not always be specific enough to real-world agents. The auto-generated rubric evaluator in Microsoft Foundry is designed to help address this by generating task-specific rubric evaluators from the context you already have. We validated it across four aspects, and found that with the recommended configuration, the auto-generated rubric evaluator demonstrated validity and reliability under the tested conditions

Authors: Shuo Qiu, Sydney Lister, Ilya Matiach, Ali Mahmoudzadeh, Salma Elshafey, José Santos, Vivek Bhadauria, Morteza Ziyadi, April Kwong

Why Your Agent Needs a Task-Specific Evaluator

Picture a customer-service agent for a telecom company. A customer messages in asking to switch plans and get a refund for last month's overcharge. The agent needs to verify the customer's identity and confirm the new plan before ending the conversation. Miss the verification step and you have a security incident. Those success criteria are specific to this one scenario.

The auto-generated rubric evaluator is designed to help address this: use the context you already have to generate a task-specific rubric evaluator that returns a weighted score with per-dimension explanations, then can be reused across iterations.

How We Validated Evaluator Quality

We validate auto-generated rubric evaluators across four aspects:

  • Verdict Validity — whether judgments on real cases reflect what a competent reviewer would conclude.
  • Rubric Validity — whether generated rubrics capture the task requirements and failure modes.
  • Manual Quality Inspection — whether judgments on real cases look right to a human reviewer.
  • Reliability and Separability — whether judgments are stable across repeated runs and distinguish stronger from weaker candidate agents.

Validation Results

1. Agreement with Trusted Reference Signals

We first validate the auto-generated rubric evaluator end-to-end: we use the rubric generator to produce the rubric's dimensions, then the rubric evaluator scores each case against them. We use GPT-5.4 for both rubric generator and rubric evaluator.

The first question is whether those end-to-end scores move with signals teams already trust. For example, does the rubric evaluator give lower scores to failed cases, and higher scores to successful ones? We start by choosing benchmarks the community already uses as reference points:

Dataset

What It Tests

JSON Editing

Deterministic structured-editing tasks where outputs can be checked exactly.

TauBench Telecom

Customer-service agent tasks requiring policy following, tool use, and task completion.

The Agent Company

Long-horizon workplace-agent tasks with multi-step tool use. We InspectAI’s 10-case subset.

BFCL Multi-Turn Tool Calling

Multi-turn function-calling behavior across realistic tool-use scenarios.

LiveClawBench

Open-ended web-agent tasks that require browsing, interaction, and judgment.

Retail-Agent Customer Service

Real production-style retail support conversations.

We then ask the generation pipeline to generate rubric evaluators for each scenario, and measure the correlation between the evaluator's scores and the trusted reference signals. For the three datasets with per-case reference signals, we can directly check whether the evaluator gives higher scores to successful cases than failed ones.

We then create traces from different candidate agents. In these experiments, each candidate agent uses the same task setup and prompt but a different underlying model, which gives us a controlled range of stronger and weaker agent behaviors.

Because the evaluator returns a continuous score, we use receiver operating characteristic area under the curve (ROC AUC) when the trusted case-level signal can be read as success versus failure. It measures how often, when comparing a successful case with a failed case, the evaluator assigns the successful case the higher score.

 

 

In these experiments, generated rubric evaluators align well with trusted signals at the case level, with ROC AUC of 0.794 on TauBench Telecom, 0.869 on The Agent Company, and 0.972 on JSON Editing.

An important goal of evaluation is to score candidate agents that perform better on the reference signal also higher by the evaluator. This is more directly relevant when choosing among candidate agents, and it is a more forgiving test of alignment because aggregated scores are less sensitive to noise in individual judgments.

We measure this with aggregate candidate-agent Spearman ρ, which checks whether the evaluator ranks candidate agents the same way as the oracle — a ρ of 1.0 means the evaluator's ranking is perfectly aligned with the oracle's, while 0 means no relationship. For BFCL and LiveClawBench, the oracle ranking comes from their official leaderboard scores.

 

 

At the aggregate candidate-agent level, Spearman ρ ranges from 0.69 on The Agent Company to 0.98 on JSON Editing across all five benchmarks. Aggregation reduces per-case noise, so the candidate-agent ranking is the more relevant view when the goal is agent selection.

2. Rubric Quality on GDPVal

GDPVal is a benchmark that measures how well AI models perform real-world, economically valuable work in sectors such as government, manufacturing, and technical services. This benchmark includes a rubric for each task, authored by a domain expert, which is useful for rubric-validity measurement.

We ask the rubric generator to produce a rubric for each test case, then use a separate matching judge to match the generated dimensions to the expert dimensions. This gives us two metrics for rubric quality:

  • Recall. For each annotated dimension, did at least one generated dimension express a similar requirement?
  • Precision. For each generated dimension, did at least one annotated dimension express a similar requirement?

Under this setup, the generated rubric achieved 72.1% recall and 86.4% precision against the expert dimensions on GDPVal tasks.

 

 

3. Manual Quality on Retail-Agent Conversations

For a real-world retail-agent customer-service dataset, we generated a rubric with six dimensions, then graded 12 conversations over those dimensions, and manually inspected every case-by-dimension judgment.

 

 

In this small sample (12 conversations), the reviewer disagreed with only one of the 72 case-by-dimension judgments. Most neutral cases involved applicability questions that the evaluator flagged inconsistently.

 

 

Reliability and Separability

Another key question is how reliable the evaluator's scores are. We look at two things: reliability (does the same case get the same score next time?) and separability (can the evaluator confidently rank two candidate agents against each other?).

Reliability

If you re-grade the same case tomorrow, do you get the same score? We measure this two ways: single-measure intraclass correlation, ICC(3,1) measures how much of the score variance comes from real case differences rather than repeat noise, and Kendall's W measures rank reliability across repeats — 1.0 means the evaluator ranks cases in the same order every time.

 

 

On JSON Editing, single-measure intraclass correlation, ICC(3,1), is 0.852 and Kendall's W is 0.767, which means re-running the evaluator on the same case gives similar numbers under repeated runs in this experimental setup. TauBench Telecom shows similarly strong reliability, with ICC(3,1) of 0.85 and Kendall's W of 0.89 under the same recommended configuration.

Separability

Separability measures whether the score is decisive: when you put two candidate agents side by side, can the evaluator confidently say which one is better?

We report mean pairwise bootstrap confidence, which measures ranking stability. For each pair of candidate agents, we resample cases and recompute each agent's mean evaluator score. The pair confidence is the fraction of bootstrap samples supporting the more common ordering: a value near 0.5 means the ordering is unstable, while a value near 1.0 means the evaluator consistently separates that pair. We average this across all candidate-agent pairs.

The candidate-agent intervals are tight on JSON Editing and TauBench Telecom. Mean pairwise bootstrap confidence is 0.96 on JSON Editing dataset and 0.95 on TauBench Telecom dataset.


Get Started

The auto-generated rubric evaluator's results may vary depending on task design, input quality, and evaluation setup. Start with a clear, well-defined description for your evaluation in the prompt field, include as much high-quality context as possible, such as the agent definition and examples, and review the generated rubric carefully before using it. Run it against a small set of known-good and known-bad cases to understand how the score reflects different failure modes.

Try the workflow in the Foundry portal and follow the rubric evaluator tutorial. For a demo that covers Rubric in the broader observability workflow, watch the Build breakout session From observability to ROI for AI agents on any framework. For the full set of Build observability announcements, read Build 2026: From observability to ROI for AI agents on any framework.

Updated Jun 16, 2026
Version 1.0
No CommentsBe the first to comment