Human-in-the-loop for evaluation quality is now in public preview in Microsoft Foundry, announced at Microsoft Ignite on November 17, 2026.
Your agent handled 10,000 conversations last week. Your team reviewed 50 and uncovered critical issues. But what happens to those findings? Too often, they end up in a spreadsheet, disconnected from your evaluation pipeline, never becoming reusable tests or measurable quality signals.
Production traces reveal how your agent behaves with real users. Human reviewers bring the expertise to identify what worked, what failed, and what the agent should have done differently. The challenge is to turn these human feedbacks into evaluation signals that continuously measure and improve agent quality.
With Human-in-the-loop for evaluation quality in Microsoft Foundry, teams can turn real-world agent interactions and human feedback into trusted evaluation datasets, actionable quality signals, and repeatable tests - connecting human judgment directly to the evaluation and optimization loop.
Why agent evaluation needs human feedback
Automated evaluators make it possible to assess AI agents at scale. But scale is only valuable when you can trust the results. When an LLM judge says your agent passed, is it evaluating what your business actually cares about?
Unlike traditional software, good agent behavior is often defined by business requirements, domain expertise, and customer expectations. This knowledge typically lives in review comments, spreadsheets, and people's heads rather than in evaluation workflows.
AI Agent teams face three challenges:
- Valuable test cases are buried in production. Real-world failures, edge cases, and unexpected behaviors are hidden among thousands of traces.
- Human feedback is difficult to reuse. Reviewer comments explain what went wrong, but cannot directly evaluate a different response from a future agent version.
- Not every dataset is a trusted benchmark. A large collection of examples doesn't necessarily represent the quality bar a team wants to maintain.
What's new
Human-in-the-loop addresses these challenges by connecting production behavior, human feedback, evaluation and agent improvement in one workflow. It supports three goals:
|
Goal |
How Human-in-the-loop helps |
|
Build trusted evaluation datasets |
Convert production traces and conversations into representative test cases with human-reviewed expected outputs and annotations. |
|
Make evaluation trustworthy |
Generate test-case-specific scoring criteria from human feedback and evaluate agent responses against them. |
|
Close the improvement loop |
Reuse trusted datasets and task-level rubric evaluators to identify regressions and measure agent improvements. |
From production behavior to evaluation datasets
Start with traces or conversations from an agent that’s already running. Use trace filters to identify representative scenarios, including successful interactions, recurring failures, and edge cases. For multi-turn tasks, review the full conversation to capture enough context for evaluation.
Reviewers can capture four types of feedback directly on traces:
- Ratings: Overall assessment, such as thumbs up or down.
- Tags: Categories for scenarios and failure patterns.
- Comments: Why a response is right or wrong.
- Expected outputs (ground truth): What the agent should have said or done.
Select annotated traces or conversations and add them to a new or existing evaluation dataset. When you enable test-case rubric generation, choose a model deployment. During dataset generation, Foundry preserves annotations and expected outputs, summarize conversations, and generate scoring criteria for an individual test case. Review generated rows and criteria before evaluating so they reflect the right context, policies, and expected behavior.
Build a benchmark your team trusts
Not all evaluation datasets serve the same purpose.
A golden dataset is a small, intentionally curated collection of high-value examples that your team trusts as a benchmark for agent quality. These examples are typically grounded in production behavior and reviewed by domain experts. Unlike broad datasets used for experimentation or testing, golden datasets establish a consistent quality baseline.
In Foundry, teams can mark a dataset as golden. Teams should continue reviewing and maintaining these benchmarks as production behavior and business requirements evolve.
Keep the improvement loop grounded
This is where human feedback becomes a repeatable evaluation signal.
Consider a reviewer comment: "Explained how to request a refund but omitted the 30-day deadline." This feedback describes a specific response. But when the agent changes, it may generate a completely different response. How can the original feedback evaluate the new response?
Foundry addresses this with task-level rubric evaluators - automatically generating test-case scoring criteria that can be used to evaluate future responses to the same scenario.
When adding annotated traces or conversations to a dataset with test-case rubrics generation enabled, teams select a model deployment. Foundry uses conversation history and human annotations to generate scoring criteria for each test case.
For example, the reviewer comment can become a reusable criterion: "The response must state that refund requests must be submitted within 30 days." Expected outputs are preserved exactly as written and evaluated separately rather than being paraphrased into rubric criteria.
For multi-turn evaluation, Foundry generates a conversation summary describing the scenario represented by each test case. Rubrics are generated during dataset creation, not dynamically during evaluation, so teams can inspect the generated criteria before using them.
Teams should still use the shared quality rubric evaluators to consistently apply configured criteria across multiple test cases. The task-level rubric evaluator is a complementary approach for allowing teams to evaluate both scenario-specific requirements and broader quality expectations.
To validate whether human feedback improves the effectiveness of rubric-based evaluation, we tested this approach on HelpSteer3, a public dataset of human preference judgments. Rubrics informed by human annotations improved alignment with human preferences from 65% to 81% - a 16-percentage-point gain, approaching the 89% agreement achieved by a second human annotator.
By incorporating human feedback into test-case rubric generation, the task-level evaluator became more effective at distinguishing responses that better align with human preferences. These results demonstrate how human expertise can improve the fidelity of automated evaluation to human judgment while generating reusable, task-specific criteria for consistently assessing agent quality.
Get started
Human-in-the-loop for evaluation connects production traces or conversations, human feedback, curated datasets, and evaluation. It focuses on building agent improvement loop.
To get started, you’ll need:
- A Microsoft Foundry project with an agent that emits traces.
- Tracing configured with a connected Application Insights resource.
- Permissions to access traces and create evaluation datasets.
- A supported region for trace-based data generation.
- A model deployment (required for test-case rubric generation)
Production traces and annotations may contain sensitive information. Review privacy, retention, and dataset-access requirements before creating or sharing evaluation datasets.
Try it Now: Follow the step-by-step guide to create your first golden dataset and task-level rubrics from production traces for evaluation [Human in the loop (preview) - Microsoft Foundry | Microsoft Learn]
Watch the Ignite Session: [From traces to better agents: The Foundry improvement loop]