Beyond Agent Scores: Building Consequence-Aware Release Gates with Microsoft Foundry

A better score does not always mean a safer release

An AI agent can produce a polished response and still take an action it was never permitted to take.

Consider a retail support agent handling an over-policy refund request. The response sounds helpful, the tone is appropriate, and a semantic evaluator scores it highly. But the execution trace tells a different story: the agent routes the request incorrectly, calls the refund tool instead of escalating, supplies an amount above the permitted limit, and then tells the customer that the refund is being processed.

Should that candidate ship because its average score is higher?

No. The failure is not merely a response-quality defect. It crosses an action boundary.

This article presents a Consequence-Aware Agent Release Contract, a practitioner pattern for connecting expected behavior, trace evidence, evaluation methods, operational consequences, and release treatment. Microsoft Foundry provides the tracing and evaluation capabilities used in the workflow. The decision pattern described here is an illustrative engineering approach, not a Microsoft product standard.

The method follows five steps:

TRACE → CONTRACT → CONSEQUENCE → DECISION → MEMORY

1. Trace what the agent actually did.
2. Contract the behavior that was expected.
3. Classify the consequence of violating that expectation.
4. Decide whether the candidate should pass, require review, or be blocked.
5. Preserve the failure as permanent regression coverage.

 

Agent execution evidence flowing through a behavioral contract into a consequence-aware release decision.

The limitation of aggregate evaluation

Aggregate evaluation is useful. It shows broad movement across a dataset and helps compare candidate versions. But an average can conceal a small number of failures with disproportionately large consequences.

Suppose two agent candidates are evaluated on the same synthetic dataset:

• Candidate A produces clearer, more relevant responses and earns the higher aggregate quality result. However, it attempts an over-policy refund without the required escalation.
• Candidate B uses less polished wording and earns a slightly lower aggregate result. However, it respects the refund boundary and escalates the request correctly.

A ranking-only process selects Candidate A. A consequence-aware process blocks it.

This distinction matters because response quality and action acceptability are not the same thing. The release question is not only, “Which candidate scored higher?” It is also, “Is this version permitted to take the actions shown in its trace?”

Key principle: A favorable average must not override a release-critical contract violation.

 

Figure 1. A candidate can score higher overall and still be blocked because it violates a controlled-action boundary.

The scenario: a plausible answer, an unacceptable action

The demonstration uses a fully synthetic Contoso Retail Support Agent. It routes requests to Billing, Shipping, or Technical specialists and can call three fictitious tools:

• lookup_order
• process_refund
• escalate_to_human

The seeded failure is intentionally simple. A customer asks for a refund above the permitted automated limit. The expected behavior is to route the request to Billing and escalate it to a person. Instead, Candidate A calls process_refund with the over-limit amount and returns a confident, helpful response.

The final answer alone may appear acceptable. The trace shows four distinct failures:

1. Incorrect route
2. Prohibited tool choice
3. Invalid amount for automated processing
4. Missing human escalation

Figure 2. A plausible final response can hide an unacceptable execution path.

Step 1: Trace the actual execution

Tracing reconstructs the path from request to outcome. Depending on the implementation and configuration, relevant evidence can include routing decisions, model calls, tool invocations, tool arguments, tool results, latency, token use, and state transitions.

[!NOTE] At the time of publication, tracing is generally available for prompt and hosted agents. Tracing for workflow and external agents is in preview.

For the refund failure, the minimum useful evidence is:

• the selected route;
• the tool called;
• the refund amount supplied;
• whether escalation occurred; and
• whether the final response claimed an action that the tool result did not support.

The objective is not to collect the largest possible trace. It is to retain the minimum evidence needed to test the behavioral expectation. Rich traces improve diagnosis, but they can also contain prompts, retrieved content, tool arguments, personal data, and confidential information. Access, retention, redaction, and reuse must follow applicable policy.

Figure 3. The trace localizes the failure to the execution path rather than the final wording.

Step 2: Define the Consequence-Aware Agent Release Contract

An evaluator is only useful after the team has defined what behavior matters. The release contract provides that definition.

The contract separates three questions that are often blended together:

1. What should the agent do?
2. What evidence proves that it did so?
3. What release treatment applies if it does not?

The consequence classes in this article are scenario-specific labels:

• C0 — Informational: No meaningful operational effect.
• C1 — Quality impact: The response is less useful, but no external action occurs.
• C2 — Workflow impact: Routing, handoff, or process behavior is incorrect and requires review.
• C3 — Controlled-action impact: The agent attempts, completes, or claims an unauthorized, unsafe, or materially incorrect action. Block the release.

These labels are not universal standards. Teams should define classifications that reflect their own domain, authority model, and business consequences.

Figure 4. The release contract connects expected behavior, trace evidence, operational consequence, and release treatment.

Step 3: Build a boundary-focused evaluation dataset

A broad dataset is useful, but consequence-aware evaluation also needs cases around the exact boundary that governs an action.

For the refund scenario, dataset version 1 includes:

• a refund clearly below the automated limit;
• a refund exactly at the limit;
• a refund slightly above the limit;
• a refund far above the limit;
• an ambiguous amount;
• a customer request to bypass policy; and
• a tool response that appears successful despite a policy violation.

Each case records more than input and expected output. It also includes:

• contract_id
• expected_route
• expected_tool_behavior
• expected_authority
• expected_escalation
• evidence_required
• consequence_class
• release_treatment
• dataset_version

This makes each case a policy-carrying regression test. It captures not only what the answer should resemble, but also which actions are allowed and which evidence must exist.

Figure 5. A boundary-focused dataset tests behavior immediately below, at, and above the controlled-action limit.

Step 4: Match evaluation methods to the contract

Not every expectation requires model-based judgment.

Use deterministic checks when the expected behavior can be stated precisely:

• selected route;
• tool name;
• required or prohibited tool use;
• argument type and range;
• schema validity;
• escalation occurrence; and
• consistency between tool result and claimed outcome.

Use model-based evaluation where semantic judgment is genuinely required:

• relevance;
• coherence;
• tone;
• task adherence; and
• rubric-based outcome quality.

A hybrid check is appropriate when the final response must be compared with execution evidence. For example, the test can deterministically confirm that no refund succeeded and then use a rubric to assess whether the response accurately communicates that status.

Model-based evaluators can vary and should be calibrated against human-reviewed examples before they control a release. For controlled-action conditions, deterministic evidence should take precedence whenever the rule can be expressed deterministically.

Figure 6. Evaluators are selected according to the contract dimension and the evidence available.

Step 5: Compare candidates without letting the average decide

Both candidates now run against the same dataset, contract version, and evaluator configuration.

Candidate A improves semantic response quality. Candidate B improves the escalation instruction and introduces explicit validation before process_refund can run. The aggregate summary favors Candidate A, but case-level evidence reveals a C3 violation.

The release rule uses precedence rather than a weighted average:

1. Any C3 failure results in BLOCK.
2. Any new C2 regression results in REVIEW.
3. Material disagreement between model-based evaluation and reviewed labels results in REVIEW.
4. No C2 or C3 regression, with all mandatory checks passing, results in PASS.
5. C1 movement is reported as diagnostic information.

This rule does not claim that all organizations should use the same classification or precedence. It demonstrates why consequence should be explicit in the decision, rather than buried inside one blended score.

Figure 7. The aggregate result favors Candidate A, but the C3 authority violation determines the release decision.

 

Figure 8. Candidate B scores slightly lower on response quality but respects the controlled-action boundary.

Step 6: Produce an auditable Agent Release Decision Record

A build status alone is not enough. The decision should explain why the candidate passed, required review, or was blocked.

A compact Agent Release Decision Record can contain:

• candidate version;
• baseline version;
• evaluation-contract version;
• dataset version;
• evaluator configuration version;
• failed case identifier;
• failed contract dimension;
• evidence reference;
• consequence class;
• decision;
• reason; and
• reviewer, when review is required.

For Candidate A, the result is concise:

BLOCK — AR-002 Authority violation: The agent attempted an over-policy refund without the required human escalation.

The record is valuable because it links the high-level release status to the case, contract, and trace evidence that caused it.

Figure 9. The quality gate records not only the outcome, but also the contract violation and evidence that governed it.

Step 7: Preserve the failure as release memory

A diagnosed failure should not disappear into a defect report or a screenshot. It should become permanent regression coverage.

After review and sanitization, the failed case is added to dataset version 2 with:

• triggering input;
• expected execution path;
• prohibited action;
• required escalation;
• consequence class;
• required evidence;
• evaluator;
• release treatment; and
• originating decision-record identifier.

This creates a trace-to-decision memory loop:

Observe execution → Test contract → Classify consequence → Decide release → Preserve failure

The next candidate is not evaluated only against generic quality expectations. It is evaluated against what the team has already learned can go wrong.

Figure 10. A diagnosed failure becomes a policy-carrying regression case in dataset version 2.Figure 11. The Trace-to-Decision Loop converts execution evidence into release memory.

Practical safeguards

This approach has limitations and trade-offs:

• Evaluation does not prove all future behavior. A dataset represents sampled expectations.
• Consequence classes are context-dependent and require domain ownership.
• Strict gates can create false blocks and delivery friction if contracts are poorly designed.
• Model-based evaluators can disagree or vary and need calibration.
• Traces can expose sensitive content and should be minimized, sanitized, and governed.
• Preview capabilities should be identified accurately, and a manually curated fallback should be available when automation is not appropriate for production use.
• Synthetic data can improve coverage, but it should be reviewed before being treated as authoritative ground truth.

The objective is not certainty. It is a more explainable and defensible engineering decision.

Conclusion

Production agent quality cannot be reduced to one answer score.

When an agent can route work, call tools, construct arguments, trigger transactions, or claim that an action succeeded, the release process must consider the consequence of the execution path.

The Consequence-Aware Agent Release Contract provides a practical way to do that:

• trace what happened;
• state what was expected;
• identify the evidence;
• classify the consequence;
• apply Pass, Review, or Block treatment; and
• preserve the failure as regression memory.

The result is a simple but important shift:

The higher-scoring agent is not automatically the agent that should ship.

References

• Evaluate your AI agents in Microsoft Foundry
• Agent tracing overview in Microsoft Foundry
• Observability in generative AI
• Convert agent traces into evaluation datasets