Global enterprise multi-agent systems in production: Release and operations (Part 2)

With hosted agents in Microsoft Foundry, the Azure Developer CLI packages and deploys each agent in a multi-agent system. Release work goes further: proving that users reach the exact release you tested, in every region, and that it keeps working. An agent that starts successfully can still be the wrong version, run with the wrong settings, or leave an uncertain business outcome.

Part 1 covered how to get started with hosted agents and the design and security practices that made enterprise multi-agent systems dependable: ownership, identity, state, safe actions, networking, knowledge, and controls. Its lesson, that a convincing response is where the engineering starts, holds for releases too. Taking those systems into production across many geographies showed that release evidence matters as much as deployment automation.

This post gives developers and architects seven practices for testing a specific release, protecting diagnostic data, measuring useful outcomes, and rehearsing recovery for artificial intelligence (AI) applications. It closes with the improvement loop that keeps quality rising after release. The goal is to release the system you tested and keep verifying what it delivers.

From hosted runtime to multi-region release

Hosted agents already provide much of what release work depends on. Each deployed agent gets a dedicated identity and endpoint and immutable versions, and agents built on the hosting protocol libraries emit OpenTelemetry traces to Application Insights by default. Immutable versions let you test and promote the exact candidate, and built-in traces supply release evidence. The platform manages session compute and the container lifecycle; your team still owns orchestration, application permissions, dependencies, and business outcomes.

Like Part 1, these lessons draw on multiple production implementations. Part 1's generalized example runs through this post: an enterprise assistant whose orchestrator agent delegates domain questions to specialist agents that use governed data and internal knowledge. The design also handles long-running jobs, and isolated deployments span multiple regions across several geographies.

What changes across geographies

• Each region is its own release. Verify the exact serving version, model and tool availability, effective configuration, access, and diagnostic-data boundaries in each destination.
• Capacity and cost add up. Every added deployment multiplies capacity planning, regional dependencies, consistent controls, and operational ownership, even when per-deployment concurrency stays modest. Compare representative regional workloads, and account for aggregate capacity and cost.
• Failover counts only after it's exercised. Define ownership when a region or dependency is unavailable. Fleet-wide acceptance needs evidence from each region; placement and one healthy region prove neither.

Feature availability: At the time of writing, several features in this post are in preview, including Insights in Foundry, several agent evaluators (among them the Output Quality and Tool Use Quality composite evaluators), routing sensitive trace content to a dedicated table, protected tables in Azure Monitor Logs, and Microsoft Defender protection for Foundry agents. Sending agent activity data from Foundry to Agent 365 requires enrollment in the Frontier preview program, and hosted-agent optimization requires an allow-listed subscription. Preview features have no service-level agreement and aren't recommended for production workloads; generally available (GA) features are supported for production use. Check each feature's current status in its documentation before you rely on it.

What you will learn

8. Separate deployment from promotion

Validate a candidate before any user can prompt it, and promote only the exact version you tested. In the production rollouts behind this series, this requirement proved simple to state and easy to miss; traffic splitting could wait. For Microsoft Foundry hosted agents, the order matters: pin the serving version before creating the candidate, because default routing follows the latest version. Capture the candidate's concrete version once, and don't resolve a moving latest again after testing.

One team designed a blue-green release around that order. Their application programming interface (API) reads the active agent version from a value in Azure App Configuration. The deployment pipeline provisions and validates the new version and only then updates that value. If provisioning fails, the value doesn't change and traffic stays on the old version.

Figure 1. A release process based on versioned evidence. A generalized flow, not a specific customer's deployment.

Release in this order:

1. Record the serving version and preserve rollback artifacts.
2. Create the candidate without overwriting the serving selector.
3. Verify its image, configuration, dependencies, and effective identity.
4. Wait for the candidate version's own status to reach active, then test that exact candidate through a correctly bound session, inspecting the terminal result rather than stopping at the initial status.
5. Complete functional, authorization, control, privacy, history, recovery, performance, and cost checks.
6. After approval, recheck the candidate and serving configuration, promote under coordinated ownership, and verify a fresh unpinned session. Hold promotion if a required check fails or approval is missing.

Plan exposure and rollback. The endpoint selector serves one version at 100%, so progressive exposure needs an explicitly supported gateway, application, or separate-endpoint design. Existing sessions stay bound to their version, so plan session draining and schema compatibility.

Pin immutable artifacts and available explicit versions: source revision, application and base-image digests, dependency locks, framework, protocol, prompts, and state schema. Record effective tool, model, policy, and retrieval configuration, and revalidate mutable dependencies when they change. A Microsoft Foundry Toolbox consumer endpoint, for example, follows its mutable default version, which can change without an agent redeployment.

Retain verification evidence for image signatures and a signed software bill of materials (SBOM), and keep scanning and patching. Treat model migration as a behavior change, track model retirement, and preserve and retest a supported rollback target.

Watch for:

• The selector doesn't provide native weighted traffic splitting, and changing it doesn't migrate existing sessions.
• Rollback doesn't repair state a candidate already wrote.
• Publishing signatures and an SBOM doesn't establish deployment enforcement or vulnerability-free code.
• Pinning can't extend a retired model's availability.
• Shadow tests that duplicate consequential actions.

Check before release: Keep one immutable evidence package, demonstrate promotion and recovery, and test both old and new sessions. Gate deployments on health signals as well as deployment status.

Rejected alternative: Promoting based on an active label, a mutable image tag, or a merged change. Merged, released, installed, effective, and successfully retested are different milestones.

Further reading: Manage hosted agents · Use Toolbox with hosted agents · Attach a signed SBOM · Model migration · Model retirements · Safe deployment practices

9. Assign configuration ownership

Give every setting an owner and verify it end to end, so portal users, pipelines, and security automation don't silently overwrite each other. Infrastructure as code improves repeatability, but it doesn't automatically cover every setting or coordinate every writer.

In one rollout, a logging setting reached a stable service API version, but the Terraform provider didn't support it yet, so the team's infrastructure code couldn't manage it. That rollout showed why a successful API call isn't enough: verify the setting's whole chain.

For each required field, verify the chain: backend behavior, published API and schema, released software development kit (SDK) or provider, desired-state owner, governance enforcement, and effective behavior. Give version selection, identity, environment variables, guardrail associations, telemetry, and policy defaults explicit owners.

The hosted Terraform guide separates infrastructure from agent deployment. A post-create patch can bridge a tooling gap, but it adds sequencing, drift, and possible exposure before the setting takes effect, so read back effective state after changes and propagation. Test create and update separately, including new and resumed sessions.

Watch for:

• A backend accepting a field doesn't prove that the provider preserves it on update.
• A provider resource doesn't establish that an Azure Policy alias exists, and an AI guardrail policy and Azure Policy are different controls.
• A provider's latest output is not an atomic receipt for the candidate you created, and a Terraform state lock doesn't coordinate independent portal or SDK writers.
• A checked control in an editor is not proof of a saved, assigned, active policy.

Check before release: Record field-level owners and support evidence. Repeat a no-change deployment, introduce authorized test drift, and confirm that required settings survive. Keep exception and rollback procedures auditable.

Rejected alternative: Ignoring drift on every field. That can hide a missing owner or a control that keeps getting turned off; each exception needs an accountable owner and verification.

Further reading: Deploy hosted agents with Terraform

10. Evaluate agents, tools, and outcomes

Evaluate the agent, its tools, and the business outcome separately, because a convincing answer can hide an unauthorized lookup, incorrect tool arguments, or a failed business action. The agent-evaluator guidance separates system outcomes from process behavior. Test at three levels:

• Agent: interpretation, evidence use, abstention, scope, and output structure.
• Tool: authorization, arguments, timeouts, duplicate-action handling, and authoritative results.
• System: multi-turn completion, coordination, recovery, safety, latency, and total cost.

Build a versioned dataset covering representative tasks, languages, user entitlements, incomplete information, ambiguity, legitimate sensitive-data use, adversarial content, dependency failures, and rare, high-impact cases. Break results down by cohort, such as user groups or task types; one average score can hide a serious failure affecting a smaller group.

Preserve the structured trajectory each evaluator requires, and gate mandatory criteria separately. Run in-memory evaluation in the delivery pipeline as a first gate. Combine deterministic assertions with human review and calibrated model judges, and investigate disagreements and false passes. In the example, verify the job service's receipt and its completed result; the agent saying "job completed" is not enough.

Run regression checks when prompts, models, tools, policies, retrieval, frameworks, adapters, or state schemas change. Start model comparisons with an unchanged workload before adapting it, so differences stay interpretable. Repeat runs when identical inputs can produce different outcomes, and feed reviewed failures and feedback into the suite. Cloud evaluation provides workflows for datasets, agents, and traces, and the improvement loop at the end of this post turns this practice into a repeatable cycle with Insights, rubric evaluators, Agent Optimizer, and cloud red teaming.

Watch for:

• Tool and evaluator support varies, and a skipped or not_applicable check is not a pass.
• A local logic test doesn't establish hosted identity, network, history, or restart behavior.
• Output Quality and Tool Use Quality are composite agent evaluators, distinct from the deprecated, unrunnable Quality Grader. They require every applicable component to pass, while rubric aggregation weights applicable criteria; neither establishes that a skipped required check passed.
• Trace-to-dataset convenience is not permission to copy unrestricted production content into another store.

Check before release: Agree thresholds before running tests. Attach versioned results, cohort breakdowns, reviewed disagreements, and explicit skipped checks to the release manifest.

Rejected alternative: Accepting a release on one favorable demonstration or a high model-judge score. A broader suite costs time and compute, so scale its depth by risk, but don't call a smoke test complete acceptance.

Further reading: Agent evaluators · Rubric evaluators · Cloud evaluation

11. Protect diagnostic data from its first write

Protect diagnostic data from its first write, because tracing creates more copies of prompts, retrieved content, tool arguments, and outputs. Following diagnostic data end to end changes the question from "Which exporter is configured?" to "Where else is this content copied?"

Inventory every place content lands, then assign purpose, ownership, access, retention, and deletion rules to each:

• Telemetry: application and framework instrumentation, hosting exporters, collectors, Azure Monitor Application Insights, and workspaces.
• Governance and security copies: Agent 365 and Defender prompt-evidence collection.
• Operational copies: tool logs, exports, support packages, and historical copies.

Review telemetry destinations with their owners. The hosted telemetry guide documents independent Application Insights and OpenTelemetry Protocol (OTLP) destinations; configuring both sends telemetry to both. Project monitoring injects the reserved Application Insights connection string, and agent-version environment settings are immutable, so review project monitoring and version configuration as separate owners when changing destinations. Prefer safe correlation and timing metadata, review content-bearing custom attributes and error paths, and keep bearer tokens and secrets out of prompts, traces, checkpoints, and support packages.

Route sensitive content, then protect it. The sensitive-trace guide describes dedicated routing for input and output messages, system instructions, tool definitions, tool arguments and results, and evaluation explanations; in Log Analytics, the table is AppGenAIContent. Protected-table configuration separately governs access, including privileged and time-bound access. Review custom and inherited grants, role conditions, and resource and workspace access modes.

Test before real sensitive content arrives. Use unique test markers, establish that each was ingested, then test ordinary, inherited, and privileged access through resource and workspace contexts, including exports, historical copies, suppression, and the first write. Minimize or suppress content at a supported upstream boundary. If the required protection can't be demonstrated, suppress content or choose a supported alternative.

Check Agent 365 collection separately: the Microsoft Foundry setting is account-scoped and conditional on licensing and tenant consent. Hosted-session changes apply to newly started sessions, so test existing sessions, explicit exporters, and Microsoft 365-side copies.

Review the privacy terms of every tool and destination. The cited model-data privacy commitments cover designated Foundry Models sold by Azure, not every external tool, judge, or exporter. Model statelessness and the commitment not to use your data to train foundation models without your permission or instruction do not eliminate stateful service storage or abuse monitoring, and preview models and features may follow different privacy practices. Review deployment type, processing geography, storage location, and each tool or destination's terms.

Figure 2. Protecting diagnostic data across independent destinations. A generalized review flow, not a prescribed routing topology or a specific customer's architecture.

Watch for:

• A collector isn't necessarily a replacement for another destination, and span sampling doesn't stop log export.
• A content-capture flag isn't universal. In Python, @trace_function records supported arguments and results independently of built-in content recording, and those code.function.* attributes aren't among the seven gen_ai.* attributes routed to the dedicated content table.
• Routing is not protection. Suppression, redaction, routing, read restrictions, retention, and erasure solve different problems.
• One protected table doesn't govern an independent exporter or every custom attribute, and routing changes don't remove historical values from older tables.
• Protected tables have cross-workspace-query and alert-access restrictions, and source access conditions don't follow exported copies.
• The Microsoft Foundry and Application Insights data-model guides give different migration dates at the time of writing, so verify actual destinations rather than trusting either date.
• Zero query rows alone can mean missing data, not denied access.
• The AppGenAIContent table reference lists no workspace data collection rule (DCR) transformation or Ingestion API support, so don't count on downstream scrubbing.
• The reviewed documentation doesn't establish universal protection before the first sensitive write, and a later permission change doesn't erase an earlier exposure.
• Explicit SDK configuration can override resource-level Agent 365 disablement.
• "In Azure" is not a compliance conclusion.

Check before release: Verify first-write behavior and effective access for every intended destination, using non-production test data. Keep safe diagnostics while accounting for downstream and historical copies.

Rejected alternative: Disabling one telemetry product and declaring privacy solved. Suppressing every diagnostic is also costly: define the minimum safe diagnostic information your operators need.

Further reading: Hosted agent telemetry · Client-side tracing · Sensitive trace content · Application Insights data model · Configure protected tables · Protected tables access · Granular role-based access control and exports · AppGenAIContent table · Agent 365 data collection · Model data privacy · Defender prompt evidence

12. Measure time to useful progress

Measure the time until users see useful progress; an open connection doesn't count. On the way to production, a connection kept alive by a heartbeat still left users waiting for anything useful. Treat latency as several clocks: user submission, gateway arrival, provisioning and readiness, response headers, first body event, first model token, first useful progress, and the verified terminal outcome.

In the example, a useful status might explain that a job is running. But a grounded explanation and a completed job are separate outcomes. Only the job service's authoritative result shows that a job completed; a submission receipt doesn't.

Correlate the same request across client, proxy, platform, agents, models, and tools. Record timestamp sources and units, and allow for clock skew, buffering, reconnects, and missing spans. The hosted debugging guide defines response-begin at header commitment, not first token or useful output, and its timings are not billing or service-level agreement (SLA) measurements.

Compare cold, warm, resumed, long-history, tool-heavy, and concurrent journeys, holding protocol, network, region, image, model and reasoning settings, controls, and workload constant. Report sample counts, failures, percentiles, distributions, and uncertainty. Optimize one factor at a time: dependencies, imports, initialization, correctly scoped connections, tool batching, model choice, context, or idle behavior.

Watch for:

• A token can begin generation without resolving the request.
• A timing gap near a guardrail is a hypothesis until matched evidence attributes it.
• Completion trailers are best-effort, and a missing component is not necessarily a zero-duration component. Asynchronous calls expose only platform overhead on this timing path.
• An echo endpoint isolates overhead; it doesn't establish a business service-level objective (SLO).
• The median is not the mean, and a range is not variance.
• Moving initialization after readiness may only shift the delay to the first request.

Check before release: Define the clocks, capture matched-request traces, and compare representative workloads without sacrificing quality, controls, or cost.

Rejected alternative: Calling every delay a cold start, blaming an unmeasured component, or using an unmatched competitor benchmark as a target.

Further reading: Debug hosted agents

13. Measure cost per useful outcome

Measure cost per useful outcome, because the cheapest token isn't necessarily the cheapest completed task. Account for applicable model, hosted compute, idle lifecycle, tool, retrieval, storage, network, security, monitoring, and evaluation charges, and check which meters apply.

The hosted concepts guide describes session-based scaling: there is no native customer-sized warm pool to configure. Longer idle windows can reduce wait time while increasing cost and consuming capacity. Shared compute may reduce duplication while adding isolation and debugging obligations, and a smaller model or lower reasoning effort may cut one charge while increasing failed tasks, retries, or manual rework. Record both sides of every change.

Use Microsoft Foundry cost-management guidance to distinguish estimates, applicable rates, and actual financial records, and enforce budgets when admitting and executing work, alongside financial alerts.

For a defined reporting window and accounting basis, put the cost of all attempts (successful, failed, canceled, and retried) plus allocated shared and idle costs in the numerator. Divide by unique, authoritatively verified useful outcomes, not requests or attempts. Define success in the task contract, where a required safe refusal can be a useful outcome, and deduplicate retries of the same outcome.

Track task-level attribution and fleet-wide totals so shared infrastructure and running-idle costs don't disappear between teams, and keep estimates, actual charges, and any allocated manual-rework cost distinguishable. Budget the improvement loop too: judge calls, baseline and candidate evaluations, red-team runs, real test-tool execution during optimization, and Insights analysis all consume model, compute, and telemetry resources.

Watch for:

• A completed request doesn't prove compute stopped immediately.
• Overlapping products may not count usage identically.
• A displayed amount is not a unit rate, and trials, licenses, allowances, enrollment, and paid usage have different scopes.
• Budget alerts don't stop consumption.
• With zero verified outcomes, report spend and zero outcomes; the ratio is undefined, not zero.
• Longer optimized instructions may improve a score while increasing token use, so compare the accepted business outcome and total cost, not the optimizer's composite score alone.

Check before release: Record meter units, workload assumptions, quality targets, retry and idle behavior, and cost ownership. Reconcile estimates with actual usage before scaling.

Rejected alternative: A token-cost or latency improvement that weakens required quality or protection. The regression is real even if one dashboard looks better.

Further reading: Hosted agents · Manage Foundry costs · Create budgets

14. Rehearse recovery and operational handover

Rehearse recovery until an authorized user can complete the same task again; infrastructure that's back online doesn't prove it. Define recovery point and recovery time objectives from business impact. Then inventory everything that task depends on: business records, conversations, checkpoints, action receipts, user mappings, files, sources, indexes, deletion records, images, schemas, configuration, identities, permissions, networking, models, monitoring, and operator access. Decide whether recovery must be zonal or regional; for deployments spread across regions, zonal resilience alone may not meet the requirement.

Distinguish restoring data from reconnecting it through a supported service configuration. Microsoft Foundry Agent Service recovery guidance describes mode-specific limitations, and capability hosts can't be updated in place, only deleted and recreated. The documented incident path treats supported dependency restores and project reconstruction as different operations, and reconstruction need not recover prior history. If the supported path can't meet the required objective, change the design or the explicitly agreed requirement.

Follow Azure Well-Architected recovery guidance: rehearse dependency order, failover, failback, integrity, permissions, deletion behavior, and the business flow, and reconcile uncertain external actions. Start in authorized proof-of-concept (PoC) or minimum viable product (MVP) environments with non-production data and a limited impact area, and give any production drill separate safeguards and approval.

Handover also needs named owners, service-level indicators, objectives, error-budget and escalation policies, dashboards, rollback authority, runbooks, access procedures, and approved residual risks. For incidents, retain safe request and trace correlation, the concrete version and configuration, reproduction conditions, expected and actual behavior, and the decision needed.

Keep disclosure and legal review separate

Handover should also say how AI-generated content is disclosed and who reviews it. Content provenance for supported Foundry models uses signed Content Credentials, based on the Coalition for Content Provenance and Authenticity (C2PA) standard, and invisible watermarks. Coverage varies by model family, modality, and deployment type. Generation-side disclosure and detection-side support are different: the provenance detection API covers supported image, audio, and video signals, not every AI-generated text or media output. Editing, cropping, or format conversion can remove provenance information. Neither capability proves ownership, truth, or legal compliance, and missing signals don't prove human origin.

Watch for:

• A restored index that brings back revoked documents is not a correct recovery.
• A database's native restore doesn't prove that every Microsoft Foundry-managed association can be reattached, and standard-mode procedures don't automatically cover hosted sandbox files or other hosted state.
• Existing project settings and generated connections in use can't be repointed as a restore shortcut.
• Promising transparent recovery from restored bytes alone.
• A conversation identifier may not exist when failure occurs before a conversation is created.
• A support case or merged fix is not evidence of installation and successful retest.

Check before release: Demonstrate a supported recovery path, correct restored access and state, usable runbooks, and an accepted operating handover. Record support decisions and remaining risks.

Rejected alternative: "Backups exist, so recovery is done," or "the project launched, so every acceptance criterion passed." Both leave the operating requirements unproven.

Further reading: Agent Service disaster recovery · Capability hosts · Disaster recovery guidance · Reliability testing · Content provenance · Provenance detection

Keep improving after release

Quality work continues after release. A loop kept these systems improving: find recurring behavior in production traces, make domain quality explicit, compare candidate configurations, challenge the top-scoring one, and promote it only with evidence.

Four Microsoft Foundry capabilities support that loop. Insights surfaces recurring production behavior worth testing, rubric evaluators make domain quality explicit, Agent Optimizer compares candidate configurations, and cloud red teaming challenges the selected candidate. Authorization, mandatory safety checks, and human approval remain gates that no tool replaces.

Figure 3. An evidence-driven improvement loop: let tools suggest, test, and measure; keep every change under human approval.

Across geographies, separate an agent behavior change from a regional dependency failure, throttling, or missing telemetry before you act. A composite score can improve overall while one regional cohort regresses, so break results down by region. Keep trace analysis within approved processing geography. Hosted optimization isn't available in every region; the Agent Optimizer overview currently excludes Norway East.

Find recurring behavior with Insights

Start with what production traces show. Insights in Foundry analyzes traces in the Application Insights resource connected to a Foundry project to find recurring behavior that predefined evaluations may miss. It links affected versions and representative traces to a likely cause and a proposed action. Treat it as decision support that complements monitoring and evaluation; its Learn guide says not to use it as your only production monitoring, security, safety, or compliance control.

Review problematic and healthy traces before you accept a finding, and route each problem to the responsible owner. Turn confirmed behavior into a targeted fix or a new evaluation case. Give the analysis identity authorized model and telemetry access, including protected-content permissions where required, and keep least privilege, data minimization, retention, and model-analysis costs in the review.

Watch for:

• Missing spans, identity, version, or content limit what Insights can conclude.
• A likely cause is a hypothesis, not a confirmed root cause, and no findings is not a clean bill of health.
• Broadening trace capture merely to improve a finding.
• Insights can propose prompt or code changes for supported configurations, but it doesn't open pull requests, create evaluators, or deploy fixes on its own. The handoff is a developer decision, not automatic remediation.

Make domain quality explicit with rubric evaluators

Write down what "good" means for your domain. Rubric evaluators turn domain expectations into weighted, versioned criteria with stable identifiers, precise descriptions, and a pass threshold. For the example, useful dimensions include evidence fidelity, appropriate abstention when data is missing, scope and disclosure policy, and truthful reporting of tool and job outcomes.

Review an automatically generated rubric against the actual implementation. As the rubric guidance recommends, calibrate against human judgment before scaling, using human-adjudicated examples that include failures and short responses. Version criteria, weights, threshold, dataset, and judge deployment, and inspect per-dimension reasons as well as the overall pass label.

Watch for:

• A rubric score is a quality signal, not a compliance certification.
• Generation can use a hosted agent's description, which is not its full code or tool contract. Traces can supplement an agent, system prompt, or reference files, but they can't be the sole base input.
• The judge scores applicable dimensions from 1 to 5, and the weighted average is normalized to 0–1. Non-applicable dimensions contribute nothing, and the default threshold of 0.5 shouldn't silently become your release standard.
• always_applicable changes scoring behavior; it is not deterministic policy enforcement. Keep authorization, forbidden-action, and other mandatory gates separate.

Optimize a reviewed baseline, not the release gate

Use Agent Optimizer to compare candidate configurations against a reviewed baseline. It evaluates the baseline, generates candidate configurations, and compares them against selected datasets and evaluators. For hosted agents, expose the configuration that the running agent loads: instructions and, where used, skills, function-calling tool descriptions, and model selection. Optimization improves that configuration, not the application's core orchestration or authorization logic.

Hosted optimization requires the Responses protocol and, per the current hosted quickstart, subscription allow-list access. Verify enrollment, supported models, and regional eligibility before you plan on it.

An optimizer evaluation invokes the hosted agent's tools for real, so use test endpoints or mocks with non-production data, bounded spend, and no consequential business effects. Separate optimization examples from held-out release cases to check for overfitting, and keep the baseline if candidates regress. Test the selected configuration against hard security gates, rare cases, latency, and useful-outcome cost. Then apply the reviewed configuration locally, as in the hosted quickstart, redeploy, and verify it through lesson 8's exact-candidate process. Prompt agents take a different path: promotion to a new agent version.

Watch for:

• Prompt-agent tool-description optimization doesn't evaluate client-side tool execution, so its results don't transfer to hosted agents.
• A local configuration loader or generated wiring code is not proof that the deployed agent used the intended candidate.
• The starred candidate has the highest composite score: evaluator scores are rescaled to 0–1, reversed when lower is better, then averaged across evaluators and tasks. That doesn't guarantee that every required criterion or regional cohort improved. Inspect configuration changes, evaluation errors, skipped checks, per-evaluator results, and token trade-offs.
• The documentation's improvement bands are guidance, not statistical significance or automatic deployment approval.

Challenge the candidate with cloud red teaming

Attack the exact candidate before users do. The cloud AI Red Teaming Agent supports Foundry project deployments, model deployments from Azure OpenAI in Microsoft Foundry Models, and project agents, including prompt and container agents. Choose the actual target and version, and verify current feature, region, identity, and SDK requirements before adoption rather than assuming another red-teaming path's contract applies.

For prohibited actions, review and confirm the generated taxonomy against approved policy before running attacks. Choose risk categories, attack strategies, and multi-turn depth to match the workload's risks; the guide's agentic examples evaluate prohibited actions, task adherence, and sensitive-data leakage. Use an authorized, isolated environment with test identities, non-production data, and safe tools, so attacks can't cause consequential changes. Retain the taxonomy, target version, strategies, run status, attack scope, row-level outputs, and unresolved findings, and add confirmed failures to regression coverage.

Watch for:

• Testing a model alone doesn't validate the orchestration, authorization, retrieval, and tool effects around it.
• A completed run or a low aggregate attack success rate is not proof that untested risks, paths, languages, or regions are safe. Review individual failures, too.

Approve, promote, and observe again

Hand the loop over as an operating process with a named owner for each step: confirm the finding, add representative tests, compare candidates, pass mandatory gates, approve the exact release, and verify fresh outcomes. Review, test, approve, and deploy through the normal release process, and record a resolution only after fresh evidence supports it. Its results feed release approval, supported controls, regional acceptance, and recovery drills, and each of those still applies.

Budget the loop like any other workload, and count its costs per useful outcome as lesson 13 describes.

Check before release: For each loop iteration, record the confirmed finding, the test cases added, the candidates compared, the mandatory gates passed, and who approved the exact release.

Rejected alternative: Letting a tool promote changes on its own. Insights, Agent Optimizer, and red teaming suggest, test, and measure; a person approves each release.

Further reading: Insights in Foundry · Use Insights · Rubric evaluators · Agent Optimizer results · Optimize a hosted agent · Cloud red teaming

Build a small release evidence pack

Keep the evidence versioned and close to the release. These gates connect the series:

Give each gate an owner, result, and disposition. An exception is not a pass: record scope, accountable approval, compensating controls, and expiry. It cannot override product-use restrictions or turn an unsupported feature into a supported one.

For a small starting pack, use Part 1's five release checks, adapted to your workload. Keep the expected outcome separate from the observed result, and link each result to its exact candidate and safe evidence. A blank, skipped, or Not run row is not a green gate.

Your next step: test one complete user journey

Choose one representative user journey and approved non-production test data, and use Part 1's boundary checks to identify its highest-risk failures. Then pin a candidate and assemble evidence for authorization, repeated actions, privacy, useful completion, and recovery before promotion. Repeat these checks as prompts, models, tools, policies, frameworks, networking, and telemetry change. Start with the reliability-testing guidance and hosted release-management guide, then adapt the gates to your workload's risk and support requirements.

Make responsibilities explicit, then keep verifying the outcomes users depend on. Add agents and features only after that discipline holds.