rss.livelink.threads-in-node

Cross-Region Model Connectivity Options in Microsoft Foundry: Supported Patterns and Tradeoffs

rayankhoury — Fri, 19 Jun 2026 15:00:00 GMT

Model availability in Microsoft Foundry is region-dependent. The region approved for your project may not be the one where the model or Foundry Agent Service support you need is available. That creates a common cross-region design choice: connect directly to another Foundry resource, or add a gateway layer for more control and governance.

This article complements the official Bring Your Own Model guidance by focusing specifically on cross-region scenarios with Foundry resources, including practical architecture tradeoffs and enterprise deployment patterns.

Executive summary

Foundry supports more than one way to access models across regions. The simplest option is when you add another Foundry resource as a connection and access models created in the foundry resource. An Azure API Management (APIM) connection adds a customer-controlled gateway for routing, policy, observability, and network isolation. This article explains the tradeoffs and then walks through an APIM-based implementation in a VNet-secured topology and cross-region patterns for Azure AI Foundry, including how to connect Foundry agents (agents built with the Foundry Agent Service) to models in other regions.

Architecture options at a glance

Before going deeper on the APIM implementation, it helps to visualize the main options. The diagrams below show different options. The VNet-secured variant is covered in more details later in this article.

Direct Connection (No gateway)

Use when the goal is to reach models in another Foundry resource with the fewest moving parts.

Azure API Management AI Gateway

Use when the platform team wants a governed model surface and a customer-owned control plane in front of backend model traffic.

VNet-secured APIM variant

In this variant, APIM ingress is itself fronted by a private endpoint in the project VNet's `pe-subnet`, resolved via the `privatelink.azure-api.net` private DNS zone. Combined with the backend Foundry account's private endpoint, the entire request path — caller → APIM → backend — stays on the Azure backbone with no public hostname in the loop.

Choosing the right cross-region model pattern

The table below compares different options and their tradeoffs

Pattern	Best fit	Control plane and auth model	Tradeoffs
Directly connecting to another Foundry resource	Best when a team needs the simplest supported path to models in another Foundry resource and does not need an additional gateway tier.	The model connection points directly to the backend Foundry project endpoint. Authentication can use API key or managed identity, and model exposure is controlled by the connection configuration.	Lowest architectural overhead, but less room for a platform-owned policy, routing, throttling, or observability layer on every model call.
Azure API Management model connection	Best when the platform team wants a stable gateway surface in front of model traffic and expects routing, throttling, centralized policy, or richer telemetry requirements.	The model connection targets an Azure API Management endpoint that fronts the backend Foundry resource. Azure API Management can normalize auth, apply policies, route across backends, and enforce governance before forwarding the call.	Adds another service hop and requires Azure API Management design, subnet planning, and operations ownership.
Azure API Management in front of the agent surface	Governance and telemetry on agent ingress — throttling, rate-limit-by-key, request/response logging, schema validation, regional egress policy.	Inbound traffic into the agent endpoint, but Foundry continues to validate the caller's token directly so MCP On-Behalf-Of, customer-data tool authorization, and per-user audit trails still work. The underlying model calls stay inside Foundry-managed surfaces.	APIM cannot replace identity on the agent surface today — Foundry needs to see the caller (at least for MCP OBO). Treat APIM here as a governance and observability layer, not an auth layer.

In practice, teams have two supported ways to reach models across regions. Direct connections are the simplest starting point. Azure API Management is useful when the platform team wants a governed model surface with centralized routing, policy, or observability. The right choice depends on ownership, control needs, and operational overhead.

What this architecture proves: agents, models, and tracing still work

This architecture answers a key question: does the experience still hold together when models are reached cross-region through a gateway? In this setup, the answer is yes. Both Foundry agents (state-bearing agent runs in the project Playground) and prompt agents (Responses-API invocations through the same BYO connection) execute end-to-end against models reached through APIM.

Operational workflows also remain intact. APIM tracing shows the rewritten backend URL and managed-identity token path for each model call. Per-request values such as apim-request-id and apim-trace-id continue to flow through the standard APIM telemetry surfaces that platform teams already use. The routing policy is reusable across model deployments because the /inference/deployments/{deploymentName}/chat/completions path is parameterized. APIM does not need a separate API or policy for every backend model that follows the same inference route. To expose a new model end to end, the model still needs to be deployed on the backend Foundry account and registered in the project-side AI Gateway or connected-model configuration so agents can select it.

This URL-parameterized pattern works as written for the chat-completions surface, where the deployment name lives on the URL. The Responses API carries the model name in the JSON payload ("model": "<deployment>"), so a single APIM route cannot fan out by URL parameter alone — APIM would need a body-inspection policy that reads model from the request JSON and rewrites the backend per call. The cleaner answer for that surface is Foundry's dynamic model connection, which resolves model → backend on the project side at invocation time.

Dynamic connection

Dynamic model connection (Responses API). For payload-based routing — Responses API, any future surface that carries the model name in the body — Foundry's dynamic connection resolves model → backend on the project side at invocation time. The advantage is that APIM never has to inspect the request body and the AI Gateway can stay payload-agnostic. The trade-off is that the routing decision moves back into the project, so the single observable APIM URL no longer covers every model call. Teams that want centralized APIM control should stay with the parameterized chat-completions template; teams adopting the Responses API at scale should pair APIM (for backend access + Managed Identity) with the dynamic connection (for per-call dispatch).

BYOM feature compatibility at the time of publication

This table summarizes how current Foundry features interact with BYOM via APIM at the time this post is published. The experience continues to evolve, and teams across Foundry and related platform areas are actively improving coverage and integration over time.

Foundry feature	Status with BYOM via APIM	Notes
Chat completions / Responses API	✅ Works	Both surfaces invoke the BYO connection like any HTTP backend; APIM forwards to the backend Foundry account and the response is returned unchanged.
Foundry agents (Assistants / Threads / Runs)	✅ Works	Agent state lives on the project side; only the model call traverses APIM. The agent runtime, tool calls, and conversation memory remain managed by Foundry.
Prompt agents (Responses API)	✅ Works	Prompt-agent invocations go through the BYO connection on every call. No project-side state is required beyond the prompt definition.
1P on-behalf-of tools (SharePoint Grounding, Fabric Data Agent, etc.)	❌ Not supported	Foundry returns bad_request because forwarding a 1P-token tenant context to a non-Microsoft-managed model endpoint would leak it outside the Microsoft trust boundary. Use a directly-connected Foundry deployment for those tools, or split into two agents (one for 1P-tool calls, one for BYOM calls).
Evaluations (offline, on a dataset)	⚠️ Partial	Eval runs that call the model via the BYO connection execute fine. Built-in eval quality metrics that rely on Foundry-side model introspection (token-level scoring, log-probs, internal model metadata) may degrade because Foundry only sees the request/response surface that APIM exposes.
Continuous evaluations (production)	⚠️ Partial	Same caveat as offline evals. Sampling/scoring works on the request-response pair; deeper telemetry that needs first-party model metadata is reduced.
Red teaming / adversarial testing	✅ Works	Red-team probes are payload-level — they invoke the BYO connection like any other caller and score the response. No first-party hook into the model deployment is required.
Foundry IQ (model evaluation suggestions)	⚠️ Reduced fidelity	IQ recommendations rely on Foundry's view of model usage patterns. BYO traffic shows up as opaque connection-level usage, so recommendations are coarser than for directly-connected deployments.
Content Safety in Foundry Control Plane (input/output filters)	⚠️ Configure on APIM	Foundry-side content filters do not apply to BYO connections automatically. Either attach the Content Safety policy at APIM (recommended — every BYO caller is scanned in one place) or invoke Content Safety as an explicit pre/post step in the agent.
Prompt Shields (jailbreak / indirect injection)	⚠️ Configure on APIM	Same as Content Safety in Foundry Control Plane — attach the Prompt Shields policy at APIM so every request is scanned before the model sees it, regardless of which agent or caller originated it.
Fine-tuning	❌ Not applicable to the connection	Fine-tuning targets a specific deployment on a Foundry account, not a connection. Fine-tune on the backend Foundry account directly; once the new deployment exists, it becomes routable through the same parameterised /inference/deployments/{deploymentName}/chat/completions template with no APIM change.
Batch API	⚠️ Direct only	Batch jobs use long-running deployment-scoped APIs that don't fit the synchronous APIM passthrough model. Submit batch jobs directly against the backend Foundry account; APIM is bypassed for those calls.

Sample implementation topology for APIM in a vnet and core components

This implementation uses two Foundry resources with distinct roles. The project-side resource hosts the agent-facing project in the approved region and remains VNet-integrated. The backend resource hosts model deployments in the capacity region and is exposed only through a private endpoint. APIM sits between them as the customer-owned gateway, with private DNS resolving backend FQDNs to private IPs.

Template 16 provides the baseline private-network and APIM setup for Foundry. This extension builds on that template by adding one more subnet for the backend Foundry model resource, so APIM can reach the backend privately in a cross-region setup.

The request flow is straightforward: the agent runs in the project-side Foundry resource, the project calls APIM through the AI Gateway connection, and APIM forwards the request privately to the backend Foundry account that hosts the model deployment. The table below maps each component in that path to its role in the overall design.

Component	Resource	Notes
Project resource	<foundry-account> / <project-name> (project region, e.g., canadaeast)	Hosts the agent-facing project in the approved region. The project remains VNet-integrated and uses a system managed identity.
APIM AI Gateway	<apim-name> (StandardV2)	Serves as the customer-owned gateway between the project and the backend model resource. Its system-assigned managed identity is granted Cognitive Services User on the backend account so it can forward model calls without API keys.
Backend account	<backend-account> (backend region, e.g., japaneast)	Hosts the model deployments in the capacity region. After setup, public network access is disabled so model traffic reaches the account only through the private path.
Private endpoint	<backend-account>-pe in the backend-pe subnet	Provides a private, cross-region path from the project VNet to the backend Foundry account. Private DNS resolves the backend hostname to the private endpoint so APIM can reach it without using a public route.
AI Gateway connection	<connection-name> on the project	The project uses this connection to send model calls to APIM by using managed identity authentication and a defined list of available models.

Together, these components make the pattern practical for production use. Standard service names resolve privately, public network access is disabled on the backend Foundry account, model calls authenticate through APIM by using managed identity, and every model is reachable through a single observable gateway URL.

What is actually inside the VNet today

Today, the project VNet (<vnet-name>) contains four subnets, each with a specific role in the design:

agent-subnet — delegated to Microsoft.App/environments for the Foundry agent runtime.

pe-subnet — private endpoints for the project Foundry account, Storage, Cosmos DB, AI Search.

apim-outbound — APIM SV2 outbound VNet integration, defaultOutboundAccess: false.

backend-pe — the cross-region private endpoint to the backend Foundry account, defaultOutboundAccess: false.

The Foundry project, the APIM outbound integration, and the cross-region private endpoint to the backend account all sit inside the same VNet. The backend Foundry account itself remains a regional managed resource in the backend region, such as japaneast, but with public network access disabled, the cross-region private endpoint becomes the only route to it. The apim-outbound and backend-pe subnets also use defaultOutboundAccess: false, which means neither subnet has implicit public internet egress.

Hop	Auth artefact	Identity used	Configured by
Agent → Project	None — in-process	n/a	Foundry SDK
Project → APIM	AAD bearer token	Foundry Project managed identity	validate-azure-ad-token policy needs the project MI's app ID (projectMiClientId template param)
APIM → Backend	Fresh AAD bearer token	APIM managed identity	authentication-managed-identity policy + Cognitive Services User role (Azure RBAC) on the backend account for Azure AI Services acces
Network path	Private throughout	n/a	Backend publicNetworkAccess: Disabled; resolution via privatelink DNS zones; data path via the cross-region PE on the Microsoft backbone

No API keys are stored anywhere. No APIM subscription key is required (subscriptionRequired: false on the inference API). Token rotation is handled automatically by both managed identities.

For the full hop-by-hop trace and the source-of-truth auth-chain breakdown, see the extension README ("How the pieces are wired together" section).

Consuming the connection from an agent

Once the connection exists, every Foundry surface that accepts a deployment name — Foundry agents, prompt agents, raw chat-completions and Responses calls from the SDK, the portal Playground — accepts the alias <connection-name>/<deployment-name> in the same slot:

from azure.ai.projects import AIProjectClient 
from azure.identity import DefaultAzureCredential 
 
project = AIProjectClient( 
    endpoint="https://<foundry-account>.services.ai.azure.com/api/projects/<project-name>", 
    credential=DefaultAzureCredential(), 
) 
 
agent = project.agents.create_agent( 
    model="ai-gateway/gpt-5",        # <connection-name>/<deployment-name> 
    name="cross-region-agent", 
    instructions="You are a helpful assistant.", 
)

What's Next

If you want to keep going, these are the next resources and actions that matter most.

Bring your own model with the AI Gateway — Canonical BYOM pattern for Foundry Agent Service.

Configure AI Gateway in your Foundry resources — Enable Azure API Management as the gateway in front of Foundry from the portal.

AI gateway capabilities in Azure API Management — The APIM side of the story: policies, throttling, observability, and token limits for AI traffic.

Import a Microsoft Foundry API — Bring a Foundry endpoint into APIM as a managed API.
Microsoft Foundry documentation — Explore the full platform.

Deployment walkthrough

The full implementation lives in the companion repo: foundry-samples/infrastructure/infrastructure-setup-bicep/16-private-network-standard-agent-apim-setup/extensions/byom-cross-region at main · microsoft-foundry/foundry-samples

It contains the phased Bicep modules, a PowerShell orchestrator, a preflight checker, and a sample configuration. The condensed sequence below is enough to stand up the pattern and verify the gateway path works end to end.

Prerequisites

Azure CLI 2.67+ and PowerShell 7+

An Azure subscription with Contributor on the target resource group

Az PowerShell module (Install-Module Az -Scope CurrentUser)

Owner or Contributor + User Access Administrator on the target resource group (the template creates role assignments).

Foundry Account Owner to create the Foundry account.

Quota for gpt-4o/gpt-5/gpt-5.1 in the backend region, and gpt-4o in the project region.

Quota for APIM StandardV2 in the project region.

The Foundry project's managed-identity client ID (projectMiClientId). Look this up after creating the project; APIM needs it to validate inbound tokens.

Deploy

git clone https://github.com/azure-ai-foundry/foundry-samples.git 
cd foundry-samples/samples/microsoft/infrastructure-setup/16-private-network-standard-agent-apim-setup/extensions/byom-cross-region 
 
az group create --name <rg> --location <project-region> 
 
az deployment group create \ 
  --resource-group <rg> \ 
  --template-file main.bicep \ 
  --parameters @samples/parameters-cross-region.json \ 
  --parameters projectMiClientId=<paste-client-id>

After deployment, the gateway URL is in outputs.apimGatewayUrl. The connected models appear in the Foundry portal under Connected resources as <connection-name>/<deployment-name> (e.g. ai-gateway/gpt-5).

Smoke test

The /inference API has subscriptionRequired: false because authentication is managed-identity end to end. To verify the gateway is reachable from your own session:

$apim       = '<your-apim-name>' 
$deployment = 'gpt-5' 
 
$token = az account get-access-token --resource 'https://cognitiveservices.azure.com' --query accessToken -o tsv 
$body  = @{ messages = @(@{ role = 'user'; content = 'Say hi in five words.' }); max_tokens = 30 } | ConvertTo-Json -Compress 
 
Invoke-RestMethod -Method POST ` 
  -Uri "https://$apim.azure-api.net/inference/deployments/$deployment/chat/completions?api-version=2024-10-21" ` 
  -Headers @{ Authorization = "Bearer $token"; 'Content-Type' = 'application/json' } ` 
  -Body $body

A 200 with x-aigw-backend and x-aigw-region response headers proves the policy chain (MI validation → backend rewrite → cross-region PE) is healthy. For end-to-end validation via a Foundry hosted agent — agent creation, model selection, run loop — see the Use the connected model from a Foundry agent section of the extension README.

Appendix — what the APIM trace proves

A live POST against /inference/deployments/gpt-4o/chat/completions confirms the policy chain: APIM acquires a managed-identity token for the Cognitive Services audience, deletes the caller api-key, rewrites the backend to the japaneast Foundry account, forwards the request privately, and returns self-describing gateway headers.

Inbound 

  authentication-managed-identity -> token acquired for https://cognitiveservices.azure.com 

  set-header Authorization        -> Bearer token from APIM SMI, not the caller 

  set-header api-key (delete)     -> removed 

  set-backend-service             -> https://ctcfvce-jpe.openai.azure.com/openai 

  set-header x-aigw-route         -> customer-apim-fvce to japaneast 

 

Backend 

  request-forwarder               -> POST /openai/deployments/gpt-4o/chat/completions 

  forward-request                 -> 200 OK, x-ms-region: Japan East 

 

Outbound 

  set-header x-aigw-backend       -> apim-fvce-aigw / ctcfvce-jpe 

  set-header x-aigw-region        -> canadaeast to japaneast 

  transfer-response               -> response sent to caller in full

Auto-Generated Rubric Evaluators: Building Context-Aware Evaluators for AI Agents

Shuo — Thu, 18 Jun 2026 19:00:00 GMT

Authors: Shuo Qiu, Sydney Lister, Ilya Matiach, Ali Mahmoudzadeh, Salma Elshafey, José Santos, Vivek Bhadauria, Morteza Ziyadi, April Kwong

Why Your Agent Needs a Task-Specific Evaluator

Picture a customer-service agent for a telecom company. A customer messages in asking to switch plans and get a refund for last month's overcharge. The agent needs to verify the customer's identity and confirm the new plan before ending the conversation. Miss the verification step and you have a security incident. Those success criteria are specific to this one scenario.

The auto-generated rubric evaluator is designed to help address this: use the context you already have to generate a task-specific rubric evaluator that returns a weighted score with per-dimension explanations, then can be reused across iterations.

How We Validated Evaluator Quality

We validate auto-generated rubric evaluators across four aspects:

Verdict Validity — whether judgments on real cases reflect what a competent reviewer would conclude.
Rubric Validity — whether generated rubrics capture the task requirements and failure modes.
Manual Quality Inspection — whether judgments on real cases look right to a human reviewer.
Reliability and Separability — whether judgments are stable across repeated runs and distinguish stronger from weaker candidate agents.

Validation Results

1. Agreement with Trusted Reference Signals

We first validate the auto-generated rubric evaluator end-to-end: we use the rubric generator to produce the rubric's dimensions, then the rubric evaluator scores each case against them. We use GPT-5.4 for both rubric generator and rubric evaluator.

The first question is whether those end-to-end scores move with signals teams already trust. For example, does the rubric evaluator give lower scores to failed cases, and higher scores to successful ones? We start by choosing benchmarks the community already uses as reference points:

Dataset	What It Tests
JSON Editing	Deterministic structured-editing tasks where outputs can be checked exactly.
TauBench Telecom	Customer-service agent tasks requiring policy following, tool use, and task completion.
The Agent Company	Long-horizon workplace-agent tasks with multi-step tool use. We InspectAI’s 10-case subset.
BFCL Multi-Turn Tool Calling	Multi-turn function-calling behavior across realistic tool-use scenarios.
LiveClawBench	Open-ended web-agent tasks that require browsing, interaction, and judgment.
Retail-Agent Customer Service	Real production-style retail support conversations.

We then ask the generation pipeline to generate rubric evaluators for each scenario, and measure the correlation between the evaluator's scores and the trusted reference signals. For the three datasets with per-case reference signals, we can directly check whether the evaluator gives higher scores to successful cases than failed ones.

We then create traces from different candidate agents. In these experiments, each candidate agent uses the same task setup and prompt but a different underlying model, which gives us a controlled range of stronger and weaker agent behaviors.

Because the evaluator returns a continuous score, we use receiver operating characteristic area under the curve (ROC AUC) when the trusted case-level signal can be read as success versus failure. It measures how often, when comparing a successful case with a failed case, the evaluator assigns the successful case the higher score.

In these experiments, generated rubric evaluators align well with trusted signals at the case level, with ROC AUC of 0.794 on TauBench Telecom, 0.869 on The Agent Company, and 0.972 on JSON Editing.

An important goal of evaluation is to score candidate agents that perform better on the reference signal also higher by the evaluator. This is more directly relevant when choosing among candidate agents, and it is a more forgiving test of alignment because aggregated scores are less sensitive to noise in individual judgments.

We measure this with aggregate candidate-agent Spearman ρ, which checks whether the evaluator ranks candidate agents the same way as the oracle — a ρ of 1.0 means the evaluator's ranking is perfectly aligned with the oracle's, while 0 means no relationship. For BFCL and LiveClawBench, the oracle ranking comes from their official leaderboard scores.

At the aggregate candidate-agent level, Spearman ρ ranges from 0.69 on The Agent Company to 0.98 on JSON Editing across all five benchmarks. Aggregation reduces per-case noise, so the candidate-agent ranking is the more relevant view when the goal is agent selection.

2. Rubric Quality on GDPVal

GDPVal is a benchmark that measures how well AI models perform real-world, economically valuable work in sectors such as government, manufacturing, and technical services. This benchmark includes a rubric for each task, authored by a domain expert, which is useful for rubric-validity measurement.

We ask the rubric generator to produce a rubric for each test case, then use a separate matching judge to match the generated dimensions to the expert dimensions. This gives us two metrics for rubric quality:

Recall. For each annotated dimension, did at least one generated dimension express a similar requirement?
Precision. For each generated dimension, did at least one annotated dimension express a similar requirement?

Under this setup, the generated rubric achieved 72.1% recall and 86.4% precision against the expert dimensions on GDPVal tasks.

3. Manual Quality on Retail-Agent Conversations

For a real-world retail-agent customer-service dataset, we generated a rubric with six dimensions, then graded 12 conversations over those dimensions, and manually inspected every case-by-dimension judgment.

In this small sample (12 conversations), the reviewer disagreed with only one of the 72 case-by-dimension judgments. Most neutral cases involved applicability questions that the evaluator flagged inconsistently.

Reliability and Separability

Another key question is how reliable the evaluator's scores are. We look at two things: reliability (does the same case get the same score next time?) and separability (can the evaluator confidently rank two candidate agents against each other?).

Reliability

If you re-grade the same case tomorrow, do you get the same score? We measure this two ways: single-measure intraclass correlation, ICC(3,1) measures how much of the score variance comes from real case differences rather than repeat noise, and Kendall's W measures rank reliability across repeats — 1.0 means the evaluator ranks cases in the same order every time.

On JSON Editing, single-measure intraclass correlation, ICC(3,1), is 0.852 and Kendall's W is 0.767, which means re-running the evaluator on the same case gives similar numbers under repeated runs in this experimental setup. TauBench Telecom shows similarly strong reliability, with ICC(3,1) of 0.85 and Kendall's W of 0.89 under the same recommended configuration.

Separability

Separability measures whether the score is decisive: when you put two candidate agents side by side, can the evaluator confidently say which one is better?

We report mean pairwise bootstrap confidence, which measures ranking stability. For each pair of candidate agents, we resample cases and recompute each agent's mean evaluator score. The pair confidence is the fraction of bootstrap samples supporting the more common ordering: a value near 0.5 means the ordering is unstable, while a value near 1.0 means the evaluator consistently separates that pair. We average this across all candidate-agent pairs.

The candidate-agent intervals are tight on JSON Editing and TauBench Telecom. Mean pairwise bootstrap confidence is 0.96 on JSON Editing dataset and 0.95 on TauBench Telecom dataset.

Get Started

The auto-generated rubric evaluator's results may vary depending on task design, input quality, and evaluation setup. Start with a clear, well-defined description for your evaluation in the prompt field, include as much high-quality context as possible, such as the agent definition and examples, and review the generated rubric carefully before using it. Run it against a small set of known-good and known-bad cases to understand how the score reflects different failure modes.

Try the workflow in the Foundry portal and follow the rubric evaluator tutorial. For a demo that covers Rubric in the broader observability workflow, watch the Build breakout session From observability to ROI for AI agents on any framework. For the full set of Build observability announcements, read Build 2026: From observability to ROI for AI agents on any framework.

How to Score a User Simulator: Introducing USR-8

jcasantos — Fri, 19 Jun 2026 13:00:00 GMT

Authors: José Santos, Shuo Qiu, Morteza Ziyadi

User simulators have become a standard part of the agent-building toolkit: cheaper than real-user pilots, faster than scripted dialogues, and the only way to push an agent through hundreds of plausible conversations between every code change. The hard question is the one that comes next: how do you know the simulator itself is any good?

It's easy to underestimate how much can go wrong. A simulator that ends every turn with "Thanks so much for your help!" will quietly inflate your agent's scores. A simulator that coaches your agent ("could you check the fare rules first?") will hide failures behind helpful nudges. A simulator that scores well on a holistic "user-coherence" metric can still produce conversations that cluster around the same two openings and the same four invented names. None of these failure modes are visible from the agent's score sheet; all of them distort it. If the test harness isn't measured with the same rigor as the agent, you're back to vibes — just with bigger numbers attached.

This post is a research study on how we developed and measured the Microsoft Foundry’s user simulator quality. The post offers two contributions: 1) an Eight-Metric User Simulation Rubric (USR-8), a minimal, orthogonal, and sufficient framework for scoring a user simulator that separates behavior from style and surfaces failure modes a single composite score erases, and 2) a set of empirical findings from running that rubric across 1,200 conversations, three domains, and four simulator configurations.

The short version: across 1,200 conversations spanning three domains and four simulator configurations, the Foundry user simulator scored at or near ceiling on every per-conversation metric except realism, and a small prompt revision substantially improved realism in our evaluation. More interestingly, the same prompt loaded into a third-party harness produced very similar results on the metrics we measured, thus, in our evaluation we conclude that most of what makes a "good" simulator is driven by the prompt policy, not the orchestration code. That last finding likely generalizes past Foundry.

Two philosophies of user simulation

Before you can evaluate a user simulator, you have to decide what you want it to be. There are two coherent answers, and they pull in opposite directions.

Philosophy A — Realistic foil. The simulator stays in character and never coaches the agent. If the agent skips a step, the simulator notices the way a real customer would — by getting confused, getting frustrated, or moving on — not by saying "you forgot to check the fare rules." Conversation failures attributable to the agent stay visible and clean. This is what you want when you're measuring the agent's intrinsic quality.

Philosophy B — Helpful tester. The simulator is part of an end-to-end task-success benchmark. It may nudge the agent back on track when it goes off-script, but only as a real cooperative user would: by answering clarifying questions and supplying missing details, not by coaching the agent on how to do its job. That help can mask failures, so this setup measures whether the task gets completed by the agent and simulator together. Treat the result as an optimistic system-level success rate: it credits the simulator’s help and can overstate the agent’s standalone ability if that help slips into coaching.

Neither philosophy is wrong. They serve different product use cases, and the choice has direct consequences for what your scores mean.

You're trying to…	Use Philosophy A	Use Philosophy B
Compare two agent prompts	✅	⚠️ (simulator help can flatten differences)
Catch regressions during development	✅	⚠️
Measure end-to-end task success	⚠️ (no help may lead to lower pass rates)	✅
Red-team for adversarial inputs	needs persona seeds	✅
Generate realistic-looking sample conversations	✅	❌

Microsoft Foundry's simulator is built around Philosophy A: stay in character, don't coach. Several widely used third-party user-simulator frameworks default to Philosophy B — a fixed opening prompt and explicit instructions for the simulated user to correct the agent when it misses a step. We'll come back to that contrast when we look at the results.

The first thing to do, before you score anything, is pick the philosophy that matches your use case. Otherwise, you'll end up measuring a simulator on a job it wasn't designed to do.

Eight metrics for scoring a user simulator

Once you've picked a philosophy, you need a rubric. Generic agent metrics — task adherence, intent resolution, tool-call accuracy — don't apply, because the simulator isn't trying to complete the task, it's trying to be a user.

We built USR-8: eight LLM-judge metrics specifically for evaluating user-simulator output. We designed these metrics to be minimal, orthogonal, and sufficient — each captures a distinct failure mode that cannot be reliably inferred from the others. Seven are per-conversation; one is cohort-level. All score on a 1–5 integer scale with a short rationale, and all were judged by a current-generation GPT model running at low reasoning effort.

Per-conversation metrics (each judge sees the full transcript and the original scenario):

Clarity — Does the user state their request clearly enough that a competent agent could act on it without guessing? Penalizes ambiguous, fragmented, or under-specified asks.
Relevance — Does the user stay on the topic the scenario specifies? Penalizes drift, off-topic detours, and wholesale topic swaps (e.g., scenario asks for a California vendor agreement; user opens with an English NDA). Does not penalize the user when the agent goes off-topic.
Steering — Does the user keep the conversation moving toward their goal without coaching the agent — i.e., without telling the agent how to do its job? A high score requires productive steering that stops short of becoming the agent's manager. This is the Philosophy A guard rail; expect a Philosophy B simulator to score lower here by design.
Responsiveness — Does each user turn acknowledge and respond to what the agent just said, rather than ignoring it or repeating prior asks? Picks up on options the agent actually offered.
Consistency (coherence) — Does the user maintain a single coherent identity, set of facts, and conversational thread across the whole conversation? Penalizes self-contradiction — e.g., changing a booking code mid-conversation.
Realism — Does the user sound like a human writing in the moment, hesitations, false starts, contractions, emotional register that fits the scenario, plausible imperfection — rather than a polished ghostwriter playing a role? This metric scores how human the prose reads, not whether the behavior is correct.
Persona fidelity — When the scenario specifies a persona ("frustrated returning customer", "junior on-call engineer", "in-house counsel"), does the user embody it faithfully? Defaults to 5 when the scenario specifies no particular traits.

Cohort metric (judged on an entire run of conversations):

Diversity — Across, say, 100 conversations of the same simulator on the same scenario set, how varied are the conversations? Considers distinct names, distinct opening framings, distinct emotional registers, and unique scoring terms. 1 = visibly clustered (same names, same opening template); 5 = high variety across all axes.

A few notes on the design:

Separate behavior from style. Clarity, relevance, steering, responsiveness, consistency, and persona fidelity describe what the simulator does. Realism describes how it sounds. They tend to move independently and conflating them hides important signal.
Include a no-coaching guard. If you're under Philosophy A, you need at least one metric that explicitly penalizes the simulator for coaching the agent. Otherwise the judge's prior about "helpful" user turns will let coaching slip through.
Score the cohort, not just the conversation. Per-conversation metrics can be at ceiling while the cohort is visibly clustered (same names, same opening, same closer). Diversity catches that.
Use scenarios, not scripts. A "scenario" in our setup is a short prose task description (typically 80–200 words) given to the simulator as "the thing the user is trying to accomplish today." The simulator invents concrete details — names, booking codes, dates, error counts — that the scenario doesn't specify. This intentionally probes the simulator's ability to stay in character while improvising plausibly.

The minimum subset, in our experience, is relevance + steering + consistency + realism: it covers on-topic behavior, no-coaching, within-conversation coherence, and prose quality. The other four sharpen the picture, especially when comparing two simulators that look close on the minimum set.

How we put it to the test

Three agents, in three domains. We created three Microsoft Foundry agents, each in simulation mode (tool responses generated by a current-generation GPT model against a tool catalog rather than a live backend), chosen to span both consumer-vs-professional tone and short-turn-vs-long-turn conversational shape:

airline-customer-service-sim — refunds, rebookings, fare-rule lookups, baggage allowance, seat changes, 12 tools.
sre-incident-triage-sim — on-call engineer that confirms alerts, looks up service ownership, opens incidents, paginates runbooks, coordinates rollback, 9 tools.
legal-contract-review-sim — vendor agreements, redline liability and indemnity clauses, deviations from a contract template, negotiation drafts, 8 tools.

Ten scenarios per domain, ten conversations per scenario. Per simulator configuration: 10 scenarios × 10 conversations × 3 domains = 300 conversations. With four simulator configurations, that's 1,200 conversations scored. The 10 × 10 design gives 95% CI half-widths of 0.05–0.15 on per-conversation metrics — tight enough to detect effects above ~0.1.

Four simulator configurations. This is where the comparative claims come from:

Foundry user simulator, baseline prompt - an earlier prompt, before the realism revision, included here as an ablation baseline.
Foundry user simulator, production prompt - the current production prompt, whose realism-focused revision is the change we isolate below.
A publicly available third-party user-simulator framework with its default prompt.
The same third-party framework with our production prompt ported in through its prompt-customization mechanism. This was the isolation experiment — same underlying model, same third-party harness, same scenarios; only the prompt swapped. If results changed dramatically, the prompt was the lever; if they held steady, the harness was.

Every conversation in all four configurations was scored by the eight judges. Total judgments: roughly 10,000.

What we found

Three findings are worth reporting in detail.

1. The Foundry simulator is at or near ceiling on every metric except realism

For the baseline prompt (before the realism revision), at n=100 conversations per domain:

Metric	Airline	SRE	Legal
Clarity	5.00	4.89	5.00
Relevance	5.00	5.00	5.00
Steering	5.00	4.86	5.00
Responsiveness	5.00	5.00	5.00
Consistency	5.00	5.00	5.00
Persona fidelity	4.48	4.89	4.86
Realism	3.97	4.13	3.34

Diversity, the eighth USR-8 metric, is cohort-level rather than per-domain, so it does not appear in this per-conversation table; on our current scenario set it does not yet discriminate between simulators. Five of the seven per-conversation metrics are consistently at 5.00 in our evaluation. Persona fidelity is strong without being at ceiling. Realism is the lowest-scoring metric in our evaluation. The judge's notes pinpoint what's missing in plain language: clean polished phrasing, no hesitation, no false starts, no contractions, fictional names that read like marketing copy ("Priya Singh", "Olivia Hartwell"), and a polished closing thanking the agent the way a real customer rarely does.

In other words, the simulator sounds less like a customer and more like a professional ghostwriter playing a customer.

2. A single prompt revision substantially improved realism

The realism revision (now the production default) focused entirely on register, hesitation, and dropping the closing-thanks tic, moved realism by +0.61 on airline, +0.62 on SRE, and +1.06 on legal, with no regression on any other metric. The deltas are 4–10× the 95% CI half-width on each domain, well outside noise.

This says something specific about what "improving a user simulator" actually looks like in practice: at the production-prompt stage, the high-leverage edits are about prose register, not about behavior. The behavior-side metrics were already pinned at ceiling; the work that moved the needle was style work.

3. Most of the gap lived in the prompt, not the harness

A surprising finding came from the isolation experiment. The third-party simulator with its default prompt scored realism in the range 1.5–2.0 across the three domains — 2.5–3.2 points below the Foundry simulator. The judge's notes named two recurring failures: meta-reasoning leakage in the user turn ("Step 1: Analyze what the Agent just said. Step 2: …") and explicit coaching of the agent ("It looks like you skipped a step. Could you please check the fare rules first?"). Both behaviors are required by the third-party default prompt and forbidden by Foundry's.

When we ported the Foundry production prompt into the third-party framework's prompt-customization hook — same model, same harness, same scenarios, only the prompt swapped — realism on the third-party harness jumped to 4.4–4.6, essentially matching the Foundry production reference (4.4–4.7). Across all per-conversation metrics, gap reduction was measured in the 96–99% range in our evaluation. Both failure modes we had flagged in the default run — the reasoning leakage and the coaching — were gone once the Foundry prompt was in place.

This means simulator behavior is encoded primarily in the prompt policy, not in the orchestration code. The two harnesses are similar enough that, given the same prompt, they produce very similar conversations on every measured per-conversation metric. Where they diverge is in the default policy each one ships with — and that policy is editable.

The practical consequence, with the caveat that we compared only two harnesses: within that pair, the choice of framework mattered far less than the prompt running on top of it. We would not over-generalize from two systems, but the direction is clear — most of the behavior we measured lived in the prompt policy, and that is the asset worth investing in.

A note on diversity

Across all four configurations, cohort diversity scored a flat 2/5. This is a feature of the scenario set, not the simulator: when the simulator is given the same 10 prose scenarios 10 times, it produces conversations that cluster on names, openings, and emotional registers. Discriminating between simulators on diversity will require either much richer scenarios or a persona-seed mechanism that supplies named persona archetypes, e.g. cooperative happy-path; frustrated; misinformed; adversarial; non-native speaker; low-effort. That's on our roadmap; it's not a near-term fix you can drop into an existing simulator.

Methodological recommendations

A checklist for any team doing this kind of evaluation — on Foundry's simulator or a different one — based on the gaps we ran into and the choices that paid off:

Pick the philosophy first. Decide whether you're measuring a realistic foil or a helpful tester, write that decision down, and design your metrics around it. Mixing them inside one rubric produces incoherent scores.
Separate behavior from style. Use at least one metric per category. Concretely: in our baseline-versus-production comparison, the entire improvement landed on realism while every behavior metric stayed flat — proof that a single composite score would have averaged away the only signal that moved.
Add an explicit no-coaching metric (under Philosophy A). LLM judges tend to reward helpfulness; without an explicit penalty, coaching slips through and gets scored as conscientious behavior.
Run at n ≥ 100 per condition. Below that, CIs are too wide to call effects under ~0.5 reliably. We caught one apparent catastrophic failure at n=10 that turned out to be a sampling artifact at n=100.
Score the cohort, not just the conversation. A cohort-level diversity judge is what told us our diversity weakness was scenario-set-bound, not simulator-bound.
Use scenarios, not scripts. Forcing the simulator to improvise specifics is what surfaces persona fidelity and realism. Scripted dialogues let a weak simulator pass by having nothing to invent.
Build a human-rated calibration set. A few dozen conversations, hand-rated by at least two annotators, is a reasonable starting point for bounding judge bias via Spearman correlation, though the right size depends also on inter-human agreement and needs to be validated. We have not done human calibration for this study and this is an important methodological gap to close in future work.
Compare against at least one external baseline. A score on a custom rubric only means something relative to another simulator scored on the same rubric. "4.7 out of 5" reads as excellent in isolation, but if a baseline simulator also scores 4.7, you've learned nothing about your simulator — only that the rubric is generous. External baselines defend you against your own optimism.
When you compare, isolate the variable. A simulator's output depends on multiple components — at minimum the prompt and the surrounding harness, often also the judge configuration and the model. A head-to-head between two simulators changes all of them at once, so the result is uninterpretable: you can't tell which component drove the gap. Run at least one swap experiment that holds everything constant except the variable you care about — port your prompt into the other harness, or theirs into yours, and re-score. Without a controlled swap, framework-vs-framework numbers are a confounded measurement, not a result.

Where this kind of measurement has limits

User simulators are not the right instrument for every evaluation. Three honest caveats:

A deterministic oracle beats a simulated conversation, when one exists. If you can assert programmatically that a refund record was created, that check is tighter and cheaper than judging a whole conversation, so reserve user simulation for what no oracle can see: tone, persistence, and recovery from confusion.
You cannot validate invented facts against a real system. Because the user simulator will make up specifics such as booking codes, account numbers, and dates, nothing it asserts can be cross-checked against ground-truth records, so simulation cannot test correctness that depends on real identifiers. This is structural, not a defect of the simulator.
Adversarial behaviors are a different design. A simulator built to stay in character is, by construction, not built to actively try to break the agent. Red-teaming belongs in a separate evaluation track with its own metrics.

Within those limits, the methodology does what we need: it quantifies realism, coherence, on-topic behavior, and persona fidelity at a scale hand-rating cannot reach.

Final thoughts

Three takeaways are worth carrying out of this work, whether or not you ever look at the Foundry user simulator specifically.

Evaluation tools can be wrong too. Evaluation is recursive: any tool you use to score your agent is itself a system that can be wrong. A simulator that sounds polished can flatter a mediocre agent; a simulator that coaches when it shouldn't can hide real regressions.
Behavior and style are separate axes. Style failures (the polished-ghostwriter problem) and behavior failures (the coaching problem) move independently and call for different fixes. A single composite score erases that signal.
The prompt did most of the work. Swapping the prompt between two harnesses reduced 96–99% of the measured gap on every per-conversation metric in our evaluation. A fair comparison between simulator frameworks has to control for prompt — otherwise you're measuring prompts, not frameworks. We saw this across the two harnesses we compared; a harness with more simulation-side logic could move outcomes more, so treat it as a strong signal, not a universal law.

User simulators are how teams ship conversational agents without waiting for real users. Measuring them with the same rigor as the agents they're meant to test is what keeps the end-to-end evaluation honest.

Get started

Start building in Microsoft Foundry: ai.azure.com
Get the BRK252 session: aka.ms/build26-BRK252
Build 2026 Observability team blog post: From observability to ROI for AI agents on any framework
Read the docs: Microsoft Foundry user simulation documentation
Join the community: aka.ms/ai/discord

Build an Automated SLA Risk Agent with Routines in Microsoft Foundry

LauraVerghote — Wed, 17 Jun 2026 12:00:00 GMT

What if your support team could start every morning with an AI-generated summary of tickets most likely to breach SLA? This tutorial shows you how to build exactly that: a grounded agent in Microsoft Foundry that automatically analyzes your ticket data and surfaces risks. Using Routines in Microsoft Foundry, we run the agent on a recurring schedule so risks are identified early and addressed before they escalate.

You'll learn how to:

Connect Azure Blob Storage data to Azure AI Search for RAG
Create a Foundry IQ knowledge base from your search index
Build a grounded agent with SLA triage instructions
Schedule daily automated runs using Routines

Time to complete: ~45 minutes | Prerequisites: Azure subscription, Microsoft Foundry access

Prepare the dataset in Blob Storage

Before the agent can summarize SLA risk, it needs a trusted place to retrieve ticket context from. In this example, we used a sample dataset from Hugging Face. A subset of this data is uploaded to a private Azure Blob Storage container and later indexed by Azure AI Search.

Instructions:

Open Azure Portal and go to Storage Accounts.
Create or open the storage account that you want to use.
In the storage account left navigation, select Data storage → Containers.
Create a new container for your data. This is where the sample dataset will be stored. Open the newly created container.
Permission note: if the portal cannot list or upload blobs with the signed-in Entra account, go to the storage account Access Control (IAM) and assign Storage Blob Data Contributor to the signed-in user. The later Search indexing step also requires the Search service managed identity to have Storage Blob Data Reader on the storage account.
In the container toolbar, select Upload.
In the Upload blob pane, select Browse for files.
Select the desired files. Select Upload.

Create the Azure AI Search vector index

Now that the sample data is available in Blob Storage, the next step is to make it searchable for grounding. Azure AI Search reads the content from the blob container, chunks and vectorizes the text, and creates the index that Foundry IQ will use later as the knowledge source for the agent.

Instructions:

Go to Azure AI Search. Create an Azure AI Search service or select the one you want to use. Click import data.
On Choose a data source, select Azure Blob Storage.
On What scenario are you targeting? select RAG.
Configure the blob source. Fill in your subscription, storage account and blob container. Leave parsing mode on default and authentication using managed identity on enabled. Select Next.
Permission note: if the import wizard cannot read from the container, make sure the Search service managed identity has Storage Blob Data Reader on the storage account. If needed, also assign Reader so the service can resolve the storage resource during import.
On Vectorize your text, in the Kind dropdown, select Microsoft Foundry (this replaces the default Azure OpenAI option with Foundry's integrated embedding capabilities). Select the preferred Foundry project and embedding model. Put authentication type on System assigned identity. Check the acknowledgement box for additional costs, then select Next.
On Vectorize your images, leave image vectorization and image text extraction disabled. (The sample data is text-first, so text vectorization is enough for this scenario.). Click Next.
On Advanced settings, Leave everything on default and select Next.
On Review and create, leave default and select Create.

When the import completes, Azure AI Search creates the index, indexer, and skillset.

Connect Azure AI Search to Foundry IQ and create the Foundry IQ knowledge base

Let's now connect the Azure AI search to Foundry IQ so we can do agentic RAG across data sources and

Go to Microsoft Foundry at https://ai.azure.com. And select the preferred project.
In the new Microsoft Foundry experience, click Build on the top right.
In the left navigation under Build, click Knowledge.
In the Foundry IQ resource field, click the dropdown/open button.
Select the AI search we created.
Leave Auth Type as API Key.
Click Connect. Foundry IQ is now connected to the Azure AI Search resource and ready to create a knowledge base from one of its indexes.
Click Create a knowledge base.
In Choose a knowledge type, select: Azure AI Search Index.
Click Connect.
In the knowledge-base wizard, enter the knowledge source details. Open the Search index dropdown and select our created search index.
Click Create / Next to create the knowledge source.
After the knowledge source was created, select the chat/completions model for the knowledge base. Open the Model dropdown and select the desired model. Click Save / Save knowledge base.

Foundry IQ now has a knowledge base that wraps the Azure AI Search index and can be connected to a Foundry agent.

Create the grounded SLA risk agent

Create a prompt agent named and instruct it to use the connected Foundry IQ knowledge base for SLA risk analysis.

In the left navigation, click Agents. On the Agents page, click New agent and then Build an agent.
In the agent name field, enter a name for your agent.
In the agent setup, select a model/deployment.
In the Instructions field, entered instructions for example:
You are an SLA risk triage agent for IT support operations.
Use the connected Foundry IQ knowledge base / Azure AI Search knowledge source to answer all SLA ticket questions.
Do not answer SLA ticket questions from memory. Use the connected knowledge source whenever ticket details, breach risk, categories, queues, priorities, deadlines, or summaries are requested.
Identify the tickets at highest risk of SLA breach. Prioritize tickets with:- Critical or High priority- Open or In progress status- Low hours_until_breach values- Clear business impact in the ticket summary
For each high-risk ticket, include:- Ticket ID- Priority- Status- Category- Support queue- Hours until breach, if available- Why it is risky- Recommended next action.
If the answer is not available in the connected knowledge base, say: “I don’t know based on the available SLA ticket knowledge base.”
Cite sources where available.
Click into the Knowledge section of the agent. Click Add. Choose the option to connect Foundry IQ knowledge. In the Foundry IQ connection dialog, select the knowledge base we created. Click Connect.
Click Save. The agent now exists in Foundry, has SLA triage instructions, and is grounded on the Foundry IQ knowledge base.
Test the agent by asking: Test prompt: Using the connected SLA ticket knowledge base, summarize the tickets at highest risk of SLA breach and recommend next actions.

Create the daily Routine

In the left navigation, click Agent again and open the agent we created.
On the top right, click publish and then create routine.
In the routine setup form, entered a routine name and choose a suitable routine schedule and run time.
In the routine prompt/instructions field, enter the user prompt you want the agent to be triggered with. E.g. Run the daily SLA risk summary from the connected SLA ticket knowledge base. Click Create & start.
Wait for the routine creation to complete. Open the routine detail/run area.
Click ‘test run’ to trigger a manual test run from the portal.
Confirm the routine dispatched successfully. Wait for the run history/status to update to ‘Completed’.
Open the completed run entry / trace result by clicking on the response name.
We can observe that the output came from the grounded agent connected to the SLA ticket knowledge base.

So the routine is active, and will run daily, scheduled at 9:00 AM, and has been manually test-run once successfully.

Conclusion

You've built a production-ready SLA risk agent that runs automatically every morning, grounded on your actual ticket data. The same pattern works for any recurring analysis task e.g. inventory alerts, compliance checks, or customer health scores.

Next Steps

Ready to go deeper?

📚 Foundry IQ documentation – Learn about advanced knowledge base configurations
⏰ Routines in Foundry Agent Service documentation – Learn details about Routines
🔧 Azure AI Search tutorials – Optimize your search index for better grounding
💬Microsoft Foundry community – Share your implementation and get help

Want to extend this solution?

Add Microsoft Teams notifications when high-risk tickets are detected
Connect multiple data sources to your Foundry IQ knowledge base
Build a dashboard to track SLA trends over time

Have questions or built something cool with Microsoft Foundry Routines? Drop a comment below!

A Guided Tour of the New Microsoft Foundry Labs

vaidyas — Mon, 15 Jun 2026 23:41:16 GMT

There's a moment every developer chases — the first time you take something genuinely at the edge of what's possible and watch it run on your screen, against your problem. Microsoft Foundry Labs exists to give you that moment on repeat.

This is the place where Microsoft's most ambitious AI research stops being a headline and becomes something you can click, run, fork, and ship. Frontier models for computer use, biomolecular structure prediction, real-time 3D generation, multilingual speech — they're not locked behind a paper or a waitlist. They're live, they're interactive, and a lot of them are one button away from your next project. Let's walk through what's waiting for you.

The mission: closing the gap between research and reality

Foundry Labs starts from a conviction stated plainly on its About page: AI advances fast, and access to it should too. Historically, the distance between a breakthrough research paper and real-world impact has been far too wide — months or years of reimplementation between the idea and anyone actually using it.

Labs is built to close that gap. It takes experiments straight out of Microsoft's labs and puts them directly in the hands of builders and researchers, framed around three moves: discover what's at the frontier, experiment with it hands-on, and connect with the people pushing it forward.

And the work isn't scattered randomly — it concentrates in six domains where AI is having outsized real-world impact:

Pick the domain that maps to your work, and Labs becomes a map of the frontier in your field.

Featured and fresh: the homepage

The homepage opens with "Breakthrough AI you can try today" — a rotating showcase of the experiments worth knowing about right now, each with a Try it now button straight into Microsoft Foundry and a Learn more link for the deep dive. A quick look at the marquee:

MAI-Image-2.5 — Microsoft AI's image-generation model with image-to-image editing and "control with preservation," a recent No. 2 debut for editing capabilities among image-model families on Arena.ai.
MAI-Thinking-1 — Microsoft AI's first large language model, tuned for strong reasoning and math at a fraction of frontier-scale cost.
MagenticLite — an open-source agentic app for small models and the successor to Magentic-UI, pairing the MagenticBrain orchestrator with Fara 1.5 to run autonomously and transparently on your own hardware.
Fara 1.5 — Microsoft's agentic small language models for computer use, perceiving the screen from screenshots and predicting clicks directly.
EO/OS Object Detection — a first-party Earth Observation model that detects and localizes objects in satellite and aerial imagery at petabyte scale, anchoring the new GeoAI category alongside the Planetary Computer team.

Scroll on and the "Just added to Labs" strip keeps a finger on the pulse, surfacing the newest arrivals — recently VibeVoice ASR, MAI-Image-2-Efficient, Magentic Marketplace, and BugPilot. It's the fastest way to see what shipped this week without going hunting.

Going deeper: the Innovations catalog

The homepage is the highlight reel; the Innovations tab is the whole library — 50+ experiments and growing, built from frontier research and available to you today. The images across the experience are generated using Microsoft’s in-house MAI image models.

This is where the experience really opens up, because you're not stuck scrolling. You can filter and sort the entire catalog to cut straight to what matters to you — narrow by the six domains, by what's newest, or by the kind of artifact you're after. Hunting for a vision model in Creative & Generative Media? A production-ready framework in Code & Software Engineering? A few clicks and the list is exactly your shortlist. Each result opens onto its own page, where the experiment goes from a name to something you can understand and try.

A closer look: TRELLIS

To see what one of those pages actually delivers, open Trellis — and it's a great one to start with, because it doesn't just describe the model, it lets you use it on the spot.

An interactive playground, right on the page. TRELLIS turns a single image (or a text prompt) into a fully textured 3D asset, and the page gives you a live playground to do exactly that. Drop in an image, tune the parameters — latent and sparse-structure CFG scale, sampling steps, seed — hit Generate 3D Model, and preview the result. When you like it, export a GLB and pull it straight into your pipeline. No setup, no notebook, no GPU of your own.

The substance behind the demo. Below the playground, the page lays out what makes TRELLIS notable. It's built on a novel Structured LATent (SLat) representation with rectified-flow transformers scaled to 2 billion parameters, trained on 500,000 diverse 3D objects. Its trick is decoupling the latent from the decoder, so a single model can output meshes, radiance fields, and 3D Gaussians — pick the format your pipeline needs without retraining. It can go from one image to a textured mesh in under 10 seconds on a single A100, supports local editing that changes specific 3D regions while preserving the rest, and was adopted by NVIDIA AI Blueprints in September 2025. (The usage counter on the page — north of 2.4 million users — tells you it's resonating.)

Everything you need to go further. Clear tags label it (Creative & Generative Media · Model · Vision · Experimental), the technology stack is spelled out (PyTorch, diffusion models, NeRF, 3D Gaussians, CUDA), and a "Ready to Explore?" section links you to the GitHub repo, the research paper, and the project blog. That's the full arc on one page: try it, understand it, then take the code.

Once you've experimented in the playground and you're ready to go from prototype to production, head to Microsoft Foundry to deploy the model — and turn what you just tried into part of your own application.

Every experiment page follows this same shape — so once you've explored TRELLIS, you know how to read any of them.

Proof it works: Customer Stories

Trying a model is exciting; betting a product on it takes evidence. That's what the Stories page delivers — accounts from teams who took Labs experiments into production and came back with hard numbers.

Space Intelligence — A UK-based organization mapping the world’s forests. By combining Microsoft Foundry with the Microsoft Planetary Computer, they increased data production 100× and reduced forest mapping timelines from six months to six weeks (a 75% reduction), covering 3 billion hectares across 50+ countries in a single year.
Sight Machine — Achieved a 10% lift in manufacturing productivity using Foundry-powered capabilities.
Commerzbank — Scaled to 30,000 customer conversations per month on Foundry Agent Service.
MediaTek — Delivered 50% faster on-device AI performance using Phi models.

If you're building a case for adoption, these are concrete signals of impact. And if you've shipped something with Labs, you can submit your own story to be featured.

Better together: Community

Great tools build communities, and the Community page is where 25,000+ developers and researchers gather under a simple banner: build together, think further. It brings three things into one place:

An events calendar — where the Foundry team and community will be, from Microsoft Build and Ignite to the KubeCon circuit, GitHub Universe, and NeurIPS, each with a registration link so you can plan to meet up in person.
The latest from the labs — a feed of blogs and publications straight from the teams behind the experiments, so you follow the research as it unfolds.
The conversation — direct lines into Discord, Reddit, the Microsoft Research Blog, and Tech Community, where you can ask questions, swap ideas, and watch your feedback help steer what comes next.

Foundry Labs is the shortest path from Microsoft's most ambitious research to something running in front of you. The frontier is open, the experiments are live, and the community is already building. Go explore it — and tell us what you create.

Intelligent sampling in Microsoft Foundry: the science behind selecting better production traces

imatiach — Wed, 17 Jun 2026 18:00:00 GMT

Authors: Ilya Matiach, Morteza Ziyadi, José Santos, Ali Mahmoudzadeh, Shuo Qiu, Salma Elshafey, April Kwong, Vivek Bhadauria

TL;DR

Microsoft Foundry's intelligent sampling feature (used when creating an evaluation or fine-tuning dataset from production agent traces) uses a MinHash farthest-first diversity sampler. On WildChat (the primary validation dataset, sampling 100 items from a 5,000-trace pool), diversity sampling produces +29.1% higher lexical diversity and +44.8% larger vocabularies than a uniform-random baseline; across five additional datasets (Dolly, No Robots, OASST2, ShareGPT-GPT4, UltraChat), vocabulary gains range from +5.7% to +86.3%. An LLM judge prefers diversity-sampled data 78% of the time for evaluation and 71% for training (268 paired judgments). By design, the technique prioritizes coverage of agent behavior over mirroring production frequencies — which is exactly what most evaluation and fine-tuning workflows benefit from. This post shares the science behind the approach, how we validated it, and where it shines.

Figure 1. The intelligent-sampling flow at a glance: a pool of agent traces, the MinHash farthest-first selection step, and the resulting curated subset that maximizes coverage of agent behavior.

What we mean by “better.” Throughout this post, we evaluate intelligent sampling against three operational definitions of “better”:

Lexical variety — unigram diversity (unique tokens / total tokens) and vocabulary size of the selected subset; capturing whether the subset covers more of the input space.
LLM-judge preference — pairwise GPT-4.1 judgments comparing the diversity-sampled subset to a uniform-random subset, under two framings (“which is the better evaluation dataset?” and “which is the better training dataset?”).
Human-rated quality — for three datasets with genuine human quality annotations (HelpSteer2, OASST2, OpenAI’s Summarize from Feedback), the mean human-annotator quality score of the items each method selects; confirming diversity sampling doesn’t systematically pick worse items.

Why trace selection matters

The simplest conceptual approach to picking traces is uniform random selection. It's statistically unbiased and preserves the true input distribution of your production traffic — a clear baseline that's the right choice when mirroring production frequencies matters. But uniform sampling has a well-known weakness: when your real traffic is dominated by a small number of common patterns, a uniform sample is dominated by those same patterns. Rare prompts, unusual tool-call sequences, and edge cases are systematically under-represented — exactly the cases evaluation is supposed to stress-test, and exactly the cases fine-tuning needs to learn from.

Diversity sampling explicitly addresses this gap. The goal is to select a subset that covers as much of the input space as possible — including the less-frequent regions a uniform sample would systematically miss. By design, it prioritizes coverage over mirroring production frequencies, which makes it particularly well-suited for evaluation and fine-tuning workflows where breadth of behavior matters.

The technique: MinHash + farthest-first traversal

Intelligent sampling runs server-side in Foundry with no LLM calls and no external embedding-model dependencies — selection is pure hashing-based, so it adds zero per-token cost and completes in seconds-to-minutes. It combines two classic, well-understood components:

MinHash signatures. Each trace's user text is tokenized into shingles and hashed with 128 permutations. The resulting fixed-size signature lets us estimate the Jaccard similarity between any two traces in constant time, without storing or comparing the original text. This is the same trick that powers near-duplicate detection in search engines.

Farthest-first traversal. Starting from a seed trace, repeatedly select the trace whose minimum similarity to any already-selected trace is smallest — in other words, the trace that is most different from everything chosen so far. This greedy algorithm is a standard approximation for the maximum-diversity subset selection problem.

Diversity sampling — the MinHash farthest-first algorithm — is the core selection mechanism used when you create a dataset from traces. It runs after a small set of supporting steps: exact deduplication to remove redundant traces, hard filters to drop malformed or trivial ones, and aggregation across an agent's runs. Our validation work focuses on whether the diversity-sampling step itself drives the quality gains we measure.

Selection cost is independent of trace content (no LLM or embedding-model calls), scales linearly in pool size, and is dominated by hashing — negligible compared with running the evaluation or fine-tuning job that consumes the result.

How we validated the method

A diversity sampler can be evaluated along several axes. We split our work into four complementary studies, designed to assess the technique's value from independent angles.

Intrinsic diversity metrics. Unigram diversity (unique tokens divided by total tokens, a length-normalized measure of lexical variety) and vocabulary size (count of unique tokens across the selected subset). Both are aggregated across 5 random seeds and compared via paired t-tests.

LLM-as-judge preference. A blinded GPT-4.1 judge scored pairwise comparisons (diversity-sampled vs. uniform-random subsets, each containing 10 trace examples) across 268 judgments spanning three dataset sizes (1k / 5k / 10k) and five seeds. The judge is asked which subset would produce a better evaluation dataset, and separately which would produce a better training dataset, with reasoning. A second judge (GPT-5.2) was run on 50 shared comparisons to test directional robustness across judge models.

Downstream supervised fine-tuning. We fine-tuned gpt-4.1 using the standard OpenAI fine-tuning API (supervised fine-tuning on <prompt, response> pairs) on two 80-example WildChat subsets — one diversity-sampled, one randomly selected — held out 20 additional examples for validation, and ran 3 epochs each. We measured training convergence (train loss, validation loss, token accuracy) and held-out pairwise generation quality on 48 unseen prompts judged by a blind GPT-4.1 evaluator.

Golden-dataset quality. We re-ran the sampler against three datasets with genuine human quality annotations — HelpSteer2 (Scale AI professionals), OASST2 (13.5K crowd-sourced volunteers), and OpenAI's Summarize from Feedback (crowd workers) — to ask whether selecting for diversity systematically picks lower-quality items according to human raters.

All experiments use five deterministic seeds for the random baseline, paired t-tests for statistical significance, and the same sampler configuration (num_perm=128, alpha=0.35, target=100). The primary dataset is WildChat — 1k, 5k, and 10k subsets of real user / chatbot conversations — supplemented by five cross-dataset studies on Dolly, No Robots, OASST2, ShareGPT-GPT4, and UltraChat. Golden-dataset quality validation uses HelpSteer2, OASST2, and OpenAI's Summarize from Feedback.

What the data says

Before the aggregate numbers, here is what diversity sampling actually does on real production traces. We embedded all 5,000 WildChat user prompts with a sentence-transformer model and projected them into 2D — every grey dot is one trace, and the colored markers are the 100 traces each method selected:

Figure 2. Semantic view — UMAP of sentence-transformer embeddings. Each grey dot is one trace; colored markers are the 100 traces each method picked. Both methods look broadly spread in semantic space, yet they share only 1 of 100 selected traces — the methods reach genuinely different content, even where the spatial coverage looks similar.

Second, the MinHash–Jaccard space the algorithm actually optimizes on. Instead of a 2D projection (which loses the fine-scale distance information the algorithm cares about), we plot the distance distributions directly:

Figure 3. Pairwise (left) and nearest-neighbour (right) Jaccard-distance distributions between the 100 selected items, on Dolly and WildChat. Random sampling produces a long left tail in nearest-neighbour distance — some random picks have near-duplicate siblings in the selected set. Diversity sampling explicitly maximises every selection’s min-distance, lifting the entire distribution to the right. On Dolly the gap is dramatic (mean +0.08, with random’s left tail reaching down to 0.3). On WildChat the gap is smaller because the pool already has high baseline diversity — there are simply fewer near-duplicates to avoid.

The two methods select almost entirely different traces (99 unique each, 1 in common) — and the visual difference foreshadows what the aggregate metrics will show. The rest of this section quantifies the gap on four independent axes.

Diversity gains are large and consistent

On the primary WildChat dataset (5k pool, 100 selected, 5 seeds), diversity sampling produces measurably and significantly more diverse subsets than a uniform-random baseline:

Metric	Diversity sampling	Random	Δ%	p-value
Unigram diversity	0.307	0.238	+29.1%	0.010
Vocabulary size	7,019	4,849	+44.8%	0.003

Vocabulary gains generalize to five additional datasets (Dolly +5.7%, No Robots +44.5%, OASST2 +55.7%, ShareGPT-GPT4 +86.3%, UltraChat +49.3%), with the strongest effects on human-authored data.

Figure 4. Vocabulary increase relative to a uniform-random baseline, across six datasets. Vocabulary gains are positive on every dataset but vary widely with how lexically rich the underlying pool is — ShareGPT-GPT4 (long, varied conversational turns) has the most room for the sampler to spread out, while Dolly (short, formulaically-written instructions) leaves the sampler with the least lexical variety to surface.

LLM judges prefer diversity-sampled data

We ran 268 pairwise judgments using GPT-4.1 across two framings (which sample produces a better evaluation dataset, and which produces a better training dataset). Aggregated win rates:

Framing	Diversity sampling wins	Random wins
Evaluation dataset	78.0% (209/268)	22.0% (59/268)
Training dataset	71.3% (191/268)	28.7% (77/268)

Diversity-sampled subsets win across all three dataset sizes (1k, 5k, 10k). A secondary GPT-5.2 judge run on 50 shared comparisons reproduced the same directional result, with 76% raw agreement on the evaluation framing and 68% on training — supporting the conclusion that the gap is not specific to a single judge model.

Figure 5. Per-pool-size pairwise win rates for the diversity-sampled subset, across both evaluation and training framings. The 50% dashed line is the coin-flip baseline; every bar is well above it.

Vocabulary gain and judge preference measure different things — lexical variety in the selection vs. a holistic quality signal. They mostly agree, but not always:

Figure 6. Vocabulary gain vs. LLM-judge preference, one point per dataset. Four datasets (No Robots, OASST2, WildChat, Dolly) sit in the win-win region — diversity sampling delivers measurable lexical gains and judges prefer the curated subset. ShareGPT-GPT4 produces the largest vocabulary lift but a more modest judge preference. UltraChat is the clear outlier: a +49% vocabulary gain that doesn’t translate to any judge preference — on this fully-synthetic dataset the lexical novelty is real but judges don’t see the resulting subset as meaningfully better. Cross-dataset judge percentages come from a small (~4 judgments each) experiment, so individual-dataset values are noisy; WildChat’s 78% is the high-confidence anchor from 268 judgments.

The UltraChat result is informative on its own: on fully-synthetic datasets — where both sides of the conversation are model-generated — diversity sampling still surfaces lexical variety, but judges don’t see the resulting subset as meaningfully better, likely because the underlying conversations are already homogeneously phrased and there is less long-tail variation left to surface.

Fine-tuning: faster convergence, similar final quality

We ran a supervised fine-tuning experiment on gpt-4.1 using the standard OpenAI fine-tuning API: two 80-example WildChat subsets (diversity-sampled vs. randomly selected), 20 held-out validation examples, 3 epochs each.

Metric	Diversity sampling	Random	Δ
Final train loss	0.547	0.908	−40%
Train token accuracy	85.3%	76.7%	+8.6pp
Validation loss	0.869	0.873	comparable
Holdout pairwise (N=48)	37.5% wins	33.3% wins	not significant

Training dynamics differ sharply: the diversity-sampled model converges by epoch 2 and reaches 40% lower training loss. On held-out generation quality, both models perform comparably (37.5% vs. 33.3% pairwise wins, not statistically significant at N=48) — confirming that the diversity advantage does not hurt downstream model quality.

Quality holds up across human-annotated datasets

A natural concern with diversity sampling is that it might over-select hard, weird, or low-quality items. We tested this on three datasets with genuine human quality annotations:

Dataset	Annotators	Quality dimension	Diversity	Random	Δ%
HelpSteer2	Scale AI pros	Helpfulness (0–4)	2.66	2.94	−9.7%
OASST2	13.5K volunteers	Quality (0–1)	0.72	0.67	+8.1%
Summarize FB	Crowd workers	Overall (1–7)	5.16	4.77	+8.3%

Figure 7. Human quality scores for diversity-sampled vs. uniform-random subsets, on each dataset’s own annotation scale. Diversity sampling produces statistically-significant quality gains on OASST2 and Summarize from Feedback, and a statistically-significant quality drop on HelpSteer2 — see the qualitative examples in the next section for why.

On two of three datasets, diversity sampling selects items rated as higher-quality by human annotators (OASST2 +8.1%, Summarize from Feedback +8.3%). On HelpSteer2 the trend reverses: diversity sampling picks more creative-writing, roleplay, and niche-topic prompts which human annotators systematically rate as harder, and therefore lower in helpfulness. Both methods select a similar number of score-zero items; the gap is in the mid-to-high range of the scale.

To see where the HelpSteer2 quality gap actually lives, here is the full distribution of helpfulness scores for items selected by each method:

Figure 8. Distribution of HelpSteer2 helpfulness scores for the 500 items each method selected across 5 seeds. Diversity sampling slightly over-picks low- and mid-scoring items (scores 0–2) and ties random at score 3, while random over-picks the highest-scoring score-4 prompts (45% vs. 35%) — the common, well-trodden requests where models already produce reliable answers. The 9.7% mean-helpfulness gap is driven almost entirely by that single score-4 bin.

Note: OpenAI's Summarize from Feedback is a summarization task (Reddit / CNN articles), not general instruction-following. We include it for annotation methodology breadth.

What the sampler actually picks

To make the diversity-vs-coverage trade-off concrete, here are real examples of items selected by each method on HelpSteer2 (seed=42). Of 100 items selected by each method, only 3 overlapped — the two methods choose almost entirely different subsets.

Diversity-sampled examples (selected by diversity sampling, skipped by random):

#	User prompt (truncated)	Helpfulness	Correctness	Coherence
1	I have a vacation rental website and I am looking for alliterative and descriptive headlines that are 4–5 words in length. Examples: "Get Away to Galveston", "Sleep Soundly in Seattle". Each headline should have at least 50% alliteration…	2	2	4
2	You are a branding consultant with a creative mind. Give me 30 naming ideas for a baby’s website for parents in a table format.	1	1	2
3	my table has dates and USD values. give me historical exchange rates for each date for USD to Euro. also the converted USD to EU value: 01/02/2021 84.62 / 01/03/2021 79.48 / 01/04/2021 79.69 / 01/05/2021 38.06 / 01/06/2021 58.46…	0	0	2

Randomly-selected examples (selected by random, skipped by diversity sampling):

#	User prompt (truncated)	Helpfulness	Correctness	Coherence
1	How to cook t-bone in the oven	4	4	4
2	Create a 4-day dumbbell and EZ-bar workout program to build over 10lbs of muscle in 3 months	3	3	4
3	How did life originate?	3	3	4

The diversity-sampled prompts are unusual, creative, and underspecified — writing tasks with unusual constraints, open-ended branding requests, and ill-formed data tasks the model can’t reliably answer — exactly the kinds of inputs an evaluation suite or fine-tuning corpus should stress-test. The randomly-selected prompts are common, well-trodden requests where models already produce reliable, high-scoring responses. The quality-score gap (3–10%) is a direct consequence of this content shift, not a sign that diversity sampling is picking lower-quality work.

Considerations

Diversity sampling is a powerful default for evaluation and fine-tuning workloads, but a few honest considerations are worth flagging for practitioners:

Coverage, not representativeness. By design, diversity sampling emphasizes breadth of behavior over mirroring production frequencies. For use cases that need a faithful population estimate (latency, error rates), a uniform sample is the better fit.
Synthetic data sees smaller gains. On fully-synthetic datasets (e.g. UltraChat), the underlying conversations are already pre-diversified by the generation process, so the technique has less signal to work with.
Fine-tuned model quality is comparable, not dramatically better. Diversity sampling accelerates training convergence and produces richer datasets, but at the scales we tested, held-out generation quality is on par with a uniform-random baseline.
The diversity-vs-quality trade-off is dataset-specific. Two of three human-annotated datasets show higher quality scores with diversity sampling; one (HelpSteer2) shows lower scores, driven by selection of harder, more creative prompts.

Where diversity sampling shines

Across our evaluation, the technique consistently helps for workloads where broad input coverage drives downstream quality, and adds less value for workloads that depend on faithful production frequencies:

Workload	Diversity sampling fit
Fine-tuning data selection	Strong — faster convergence, broader coverage
Rubric & evaluator generation	Strong — surfaces clearer evaluation themes
Long-tail evaluation suites	Strong — maximizes input-space coverage
Quality-critical training	Strong — pair diversity coverage with manual review of selected items
Production-distribution benchmarking	Weaker — uniform sample mirrors true frequencies
Pre-curated or synthetic data	Weaker — data is already diverse

Closing thoughts

Intelligent sampling — MinHash diversity sampling — gives Foundry developers a fast, zero-extra-cost way to curate higher-coverage datasets from their production traces. In our evaluation, the technique consistently delivered measurable diversity gains, was strongly preferred by LLM judges for both evaluation and training framings, and accelerated fine-tuning convergence — while running in under a minute on typical trace pools.

It’s worth keeping the scope clear: the technique is designed for breadth of behavior, not faithful production-frequency representation. For most evaluation and fine-tuning workloads, broader coverage is exactly what you want — and that’s where this feature shines.

Try it out. If you’re building agents on Microsoft Foundry, enable Application Insights on your project, head to the Traces tab, and try creating your first dataset from traces. See the official documentation for step-by-step instructions: Convert agent traces into evaluation datasets (preview). Two Python samples — one for evaluation, one for fine-tuning — are available in the azure-ai-projects SDK: azure-sdk-for-python / datasets samples. We’d love to hear how it works for your workloads.

Author contributions
Shuo Qiu conceptualized and implemented the core method and developed the initial LLM judge-based validation method. Ilya Matiach designed and ran the validation experiments and authored the blog post. April Kwong and Morteza Ziyadi reviewed the core methodology. José Santos, Salma Elshafey, Ali Mahmoudzadeh, and Vivek Bhadauria contributed to methodology and validation review.

Get Started
Start building in Microsoft Foundry: ai.azure.com
Check out Observability BRK252 session: aka.ms/build26-BRK252
Read the docs: Foundry observability documentation
Join the community: aka.ms/ai/discord
Build blog: From observability to ROI for AI agents on any framework

Benchmarks in Microsoft Foundry (preview): Standardized model and agent quality checks

imatiach — Wed, 17 Jun 2026 16:00:00 GMT

Authors: Ilya Matiach, Morteza Ziyadi, Han Che, José Santos, Ali Mahmoudzadeh, Shuo Qiu, Salma Elshafey, April Kwong, Vivek Bhadauria

Introduction

Benchmarks in Microsoft Foundry (preview) make that kind of measurement a first-class part of the development workflow. You can run well-known open-source benchmarks against any model deployment or agent in your project, compare runs side by side in the evaluation group view, and drive the whole flow from the portal or the REST API.

Figure 1. Benchmarks appear in the Microsoft Foundry Evaluations list alongside your evaluations.

How is this different from the model leaderboard?

Microsoft Foundry already includes a model leaderboard that surfaces precomputed benchmark scores across the model catalog. The leaderboard answers, “How do these models perform in general?”

Benchmarks answer a different question: “How does my deployment or my agent, with my judge model and my configuration, perform right now?” That distinction matters when you’re comparing two deployments of the same model, validating a fine-tune, checking whether an agent regressed after a tool change, or re-running a benchmark after a model version upgrade.

What a benchmark contains

A benchmark is a predefined evaluation package. Instead of asking you to upload evaluation data and define scoring criteria, the benchmark provides them for you:

Benchmark dataset — a curated, fixed-size dataset of prompts and expected answers.
Task category — the capability being measured (reasoning, math, science, truthfulness, …).
Evaluation logic — a built-in evaluator such as builtin.regex_match, or a benchmark-specific scorer.
Optional judge model — for benchmarks that use model-based judging (for example, FrontierScience), you also pick a deployment to act as the grader. Most regex- or scorer-based benchmarks (such as GPQA Diamond, BBH, MuSR, BBEH, and TruthfulQA) do not need a Judge model.

The result of each run is typically a score: a percentage plus a pass count, such as 82% with 645 / 790 examples passing the benchmark metric.

Target vs. judge model

The target is what you are evaluating — a model deployment or an agent. The judge model is a separate deployment that the evaluation service uses to score outputs when a benchmark requires model-based judging. The judge model’s score does not represent its own quality; it is part of the scoring pipeline. Keep the judge model consistent across runs you want to compare, the same way you would keep a measurement instrument fixed in a scientific experiment.

Benchmarks available in preview

The initial set of benchmarks covers reasoning, math, science, and truthfulness, with example counts ranging from small smoke-test benchmarks (AIME 2025, 30 examples) to larger suites (BBEH, 4,520 examples).

Benchmark	Task	Examples	Evaluation logic
AIME 2025 Benchmark	reasoning, quality, math	30	AIME 2025
BBEH Benchmark	reasoning, quality	4,520	builtin.bbeh
BIG-Bench Hard Benchmark	reasoning, quality	934	builtin.regex_match
ChemBench Benchmark	reasoning, quality, sciences	2,785	ChemBench
FrontierScience Benchmark	reasoning, quality	160	FrontierScience
GPQA Diamond Benchmark	reasoning, quality	198	builtin.regex_match
MuSR Benchmark	reasoning, quality	756	builtin.regex_match
TruthfulQA Benchmark	truthfulness, quality, reasoning	790	TruthfulQA

Common benchmark scenarios

Comparing model deployments

Suppose you’re deciding between two model deployments — maybe two versions of the same model family, or a frontier model versus a smaller, cheaper one. With benchmarks you create a single evaluation group (for example, GPQA Diamond), then add one run per target. The group view shows each run’s score and token usage side by side, so you can weigh quality against cost in the same place.

This same pattern catches regressions after a model version upgrade or a fine-tune: re-run the established benchmark, look at the group view, and the delta tells you whether things moved in the right direction.

Benchmarking an agent

Agents introduce variability that pure model evaluations miss: prompts, tools, orchestration logic, and connected data all influence outputs. Pointing benchmarks at an agent gives you a model-agnostic, reproducible signal that complements the agent-specific evaluators you may already be using (intent resolution, task adherence, tool-call accuracy).

A practical pattern: pick a reasoning-heavy benchmark like GPQA Diamond or MuSR, target your agent (and, if relevant, one or two of its versions), and run it whenever you change the underlying model, the system prompt, or the tool set. The benchmark score becomes a stable yardstick across changes that would otherwise be hard to compare. From an agent’s detail page in Microsoft Foundry, the Evaluation tab opens the same wizard scoped to that agent.

Running a benchmark in the portal

From your Foundry project’s Build experience, open Evaluations and choose Create. The wizard walks through three short steps.

Target — pick Model or Agent, then select one or more deployments or agent versions.
Data — choose Benchmarks as the dataset source, pick the benchmarks you want, and select a Judge model if the benchmark requires one.
Review — confirm targets, datasets, and evaluators, then submit.

Because the benchmark owns its dataset, prompts, and evaluator, the wizard skips the dataset upload, prompt setup, and most evaluator configuration that custom evaluations require.

Figure 2. Start by selecting one or more model deployments (or an agent) as the evaluation target.

Figure 3. In the Data step, choose Benchmarks and pick one or more from the catalog. When you select a benchmark that uses model-based judging (here, FrontierScience), a Judge model dropdown appears for picking the deployment that will grade outputs.

Figure 4. The Review step summarizes targets, benchmark datasets, and evaluators before submission.

Figure 5. A completed evaluation group shows each run’s status, token usage, and benchmark score.

Reading benchmark results

Benchmark results show up at two levels. The evaluation group page lists every run in the group with its target, dataset, status, token usage, and benchmark score — convenient for side-by-side comparisons. Open an individual run for run status, Raw JSON, Download results, Download user logs, and the overall metric breakdown including token usage and the benchmark score.

A score can be expressed as a percentage (for example, 82%) and as a pass/total count (for example, 645 / 790). When detailed per-example metrics aren’t available, the overall metric result is still surfaced, and Download results gives you the row-level output for deeper analysis.

Token usage is reported as a first-class metric. Benchmarks can run hundreds or thousands of examples, and model-based judging multiplies the token cost, so it pays to keep an eye on this column when scaling up.

Driving benchmarks from the REST API (preview)

Everything in the portal is also available through the Foundry evaluations REST API. You create an evaluation group that pins the benchmark (and judge model, when one is required), then add one run per target you want to evaluate. The example below shows the shape of a group-create request; the full reference — authentication, the judge-model variant, adding runs, error responses, limitations, and troubleshooting — lives on Microsoft Learn.

POST {project-endpoint}/openai/evals?api-version=2025-11-15-preview
Authorization: Bearer {token}
Content-Type: application/json

{
  "name": "truthfulqa-benchmark-eval-group",
  "display_name": "TruthfulQA Benchmark",
  "data_source_config": {
    "type": "azure_ai_source",
    "scenario": "benchmark_preview",
    "benchmark_name": "builtin.truthful_qa",
    "benchmark_version": "3"
  }
}

Each evaluation group needs at least one run — the run is what actually evaluates a target model or agent against the group’s benchmark. Add one run per target you want to compare.

POST {project-endpoint}/openai/evals/{evaluation-id}/runs?api-version=2025-11-15-preview
Authorization: Bearer {token}
Content-Type: application/json

{
  "name": "truthfulqa-run-target-a",
  "display_name": "TruthfulQA - {target-deployment-a}",
  "data_source": {
    "type": "azure_ai_benchmark_preview",
    "target": {
      "type": "azure_ai_model",
      "model": "{connection-name}/{target-deployment-a}"
    }
  }
}

See the Microsoft Learn how-to for the full reference: https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/benchmark-evaluations

Best practices

Start with one or two benchmarks to validate your setup before scaling to larger suites.
Pin a stable judge model and a stable benchmark_version when comparing runs over time.
Watch token usage. Large benchmarks and model-based judges add up; preflight a single run before launching a batch.
Download results for failure analysis when an aggregate score doesn’t tell the whole story.
Treat preview features as preview. Pin api-version, validate benchmark identifiers, and review release notes before depending on the API in production.

Final thoughts

Reliable AI applications depend on reliable measurement. As models, prompts, fine-tuning datasets, and agents continue to evolve, the teams that move fastest will be the ones that can quickly and credibly answer: “did this change make things better?”

Benchmarks in Microsoft Foundry bring industry-standard measurement into the same place you build, deploy, and observe your AI systems — through a streamlined portal flow and a programmable REST API. To try it, open your Foundry project, go to Build > Evaluations, choose Create, and pick a small benchmark like AIME 2025 (30 examples) or GPQA Diamond (198) against a model deployment or agent. Once you have a baseline, add larger suites and additional targets, and let standardized benchmarks become part of how your team ships AI.
Get Started
Start building in Microsoft Foundry: ai.azure.com
Check out Observability BRK252 session: aka.ms/build26-BRK252
Read the docs: Foundry observability documentation
Join the community: aka.ms/ai/discord
Build blog: From observability to ROI for AI agents on any framework

How to Measure Token Impact of MCP Tool Invocation in Microsoft Foundry

vrajakishore — Fri, 12 Jun 2026 13:00:00 GMT

Your MCP-enabled agent just ran, and the token counts don't add up. The API says 773 tokens. The portal trace shows 581/141. The trajectory view displays something else entirely. Before you file a bug report, here's what's actually happening—and how to build an evidence pattern that makes enterprise token accounting defensible.

What you'll get: A reproducible A/B comparison method, visual evidence templates, and a reconciliation pattern that reduces FinOps review cycles.

Quick Reference

MCP (Model Context Protocol): A standard for connecting AI models to external tools and data sources.

Token: The basic unit of text processing and billing for AI models—roughly 4 characters or three-quarters of a word.

Trajectory view: Microsoft Foundry's visualization of the step-by-step execution path of an agent.

Token chips: The inline token count indicators shown in Microsoft Foundry's trace UI.

devtunnel: A Microsoft tool for exposing local services to the internet for testing.

The Challenge

A common assumption is that Execute Tool spans should directly expose billed token deltas for the tool call itself. In practice, this is not how the telemetry is usually interpreted in Microsoft Foundry traces.

What typically goes wrong:

Teams treat tool spans as billing boundaries.
Teams compare numbers from different runs as if they were the same transaction.
Teams mix thread token chips, trace table token columns, and API usage fields without reconciliation.

Example scenario:

An MCP-enabled run shows tool activity and large total token count in API usage.
A portal traces screenshot shows different Tokens In/Out values.
Reviewers conclude the system is inconsistent, when the data is actually from different response IDs and run contexts.

Deep Dive

This validation used a prompt agent in Microsoft Foundry with an inline MCP tool that calls a weather MCP server.

Core components:

Microsoft Foundry prompt agent: weather-mcp-token-test-agent
MCP server: remote endpoint over devtunnel
Evidence sources:
- API invocation usage object (input, output, total tokens)
- Microsoft Foundry Traces table (Tokens In, Tokens Out)
- Trajectory view (Execute Tool and Chat spans)

Observed architecture behavior:

MCP invocation is visible through mcp_list_tools and execute_tool spans.
Token accounting appears in the model's response metadata—specifically the usage object that reports input tokens (what you sent) and output tokens (what the model generated).
Tool outputs can increase subsequent model context, which increases turn-level token usage.

Visual evidence from the run:

Image 1: Agent MCP tool configuration

Image 2: Traces table with token columns

Image 3: Traces trajectory dialog

Solution Approach

Establish two comparison paths explicitly:
- Path A: strict API A/B comparison on the same prompt.
- Path B: portal trace evidence for invocation and trace-level token columns.
Run MCP-enabled and baseline (no-tool) agents with the same prompt.
Capture API usage values for both runs:
- MCP-enabled: input 581, output 192, total 773
- Baseline: input 57, output 57, total 114
- Delta: +659 total tokens
Capture portal screenshots for trace-table and trajectory views, then record their row-level token values as separate run evidence:
- Example portal rows observed: 581/141, 581/141, 868/97
Add a reconciliation statement in the report:
- API A/B totals and portal trace rows can represent different response IDs.
- They are complementary evidence and should not be forced into one exact row match.

Token variability note:

Absolute token counts can vary across runs due to differences in system instructions, tool payload length, model behavior, and response formatting.
Preserve the same prompt and baseline conditions when producing A/B comparisons.

Key decisions:

Use API usage as primary delta proof.
Use traces and trajectory as operational proof of tool invocation and run behavior.
Keep evidence linked by response IDs when possible.

Key Learnings / Best Practices

Always separate token evidence sources:
- API usage fields for strict per-response accounting
- Portal trace table for run observability
- Trajectory spans for invocation semantics
Never compare token rows across different response IDs without labeling them as separate runs.
Treat Execute Tool spans as invocation evidence, not standalone billing truth.
Capture both baseline and MCP-enabled runs using the exact same prompt for defensible deltas.
Preserve screenshots and IDs together in the same evidence package.
Add one reconciliation paragraph whenever mixed evidence sources are presented.
For enterprise reporting, prefer clear A/B tables over narrative-only claims.

Conclusion

MCP tool invocation in Microsoft Foundry can materially increase turn-level token usage, but the increase must be measured with disciplined evidence handling. In this validation, API A/B comparison showed a +659 total-token increase for the same prompt when MCP tools were enabled.

Going forward, enterprise teams should standardize an evidence pattern that combines API usage for accounting with trace/trajectory views for operational transparency.

Recommended next steps:

Engineers: Create a prompt agent in Microsoft Foundry and run the A/B method on one of your production prompts. Record response IDs with token evidence.
Platform/FinOps owners: Standardize one token evidence reporting template that separates API usage evidence from portal trace evidence, then connect the results to your existing Azure cost analysis workflow.
Launch post authors: Reference this article's evidence pattern: include one trace screenshot, one A/B token table, and one reconciliation note to reduce review cycles.
Questions? Join the discussion in the comments or connect with the Microsoft Foundry community on Microsoft Q&A.

References

Now in Foundry: Command A+ (W4A4), Chandra OCR 2, and GLM-OCR

Osi — Wed, 10 Jun 2026 22:59:40 GMT

We are seeing two distinct trends this week. The first is around how low-bit quantization has developed to the point where large reasoning models can fit on a single accelerator with less quality loss. Second, a new wave of OCR-specialized vision-language models are redefining the accuracy-throughput frontier for document understanding.

This week we are highlighting three Hugging Face models in Microsoft Foundry: Cohere Labs' Command A+ (W4A4), a 218B-parameter Sparse Mixture-of-Experts (MoE) reasoning model optimized for agentic, multilingual, and reasoning-heavy tasks; Datalab's Chandra OCR 2, a 5.3B vision-language model that converts images and PDFs to markdown, HTML, and JSON while preserving layout, with state-of-the-art results on the olmOCR benchmark and 90+ language coverage; and Z.ai's GLM-OCR, a 0.9B compact OCR model—roughly 6× smaller than Chandra OCR 2—built on the GLM-V encoder–decoder architecture that ranks first on OmniDocBench V1.5 while serving at high concurrency.

Models of the week

Cohere Labs: Command A+ (W4A4)

Model Specs

Parameters / size: 218B total, 25B active per token

Context length: 128K input, 64K output

Primary task: Text generation with vision input, reasoning, and tool use

Why it's interesting

Efficient, low compute deployment: Command A+ is designed to run on relatively minimal hardware for its size while maintaining high performance. It achieves this through advanced quantization and optimization techniques that reduce compute, latency, and cost. However, reasoning models are especially sensitive to quantization, as errors can accumulate over long decoding sequences. To mitigate this, the quantized student model is post-trained against the full-precision teacher’s output distribution, using fake quantization in the forward pass and straight-through estimators during backpropagation. CohereLabs recommends the W4A4 quantization for its strong balance of speed and latency.

Multilingual, multimodal, and reasoning focused performance gains: Command A+ extends to 48 different languages (previously 23) and is built for complex reasoning and multimodal tasks with measureable improvements across document understanding, math reasoning, and enterprise QA workflows.

Try it

Test this prompt in the CohereLabs Hugging Face Space before deploying the model in Foundry:

Sample prompt: You are Command, a legal AI for multinational contract review with access to CONTRACT_VAULT_QUERY and POLICY_TEMPLATE_RETRIEVAL tools. Analyze the input clause by first detecting language and classifying obligation type, then use CONTRACT_VAULT to find comparable {jurisdiction} clauses and retrieve the relevant policy template. Output structured JSON with obligation classification, comparative findings, risk assessment, and English recommendations with exact document citations. Include confidence scores, similarity metrics, and a reasoning trace showing each analysis step. Handle Polish/Japanese legal terminology accurately, preserve legal precision, and ensure all citations reference actual source documents. Use chain-of-thought reasoning, stay within 128K tokens, and never hallucinate references—state limitations explicitly when tools fail.

Datalab: Chandra OCR 2

Model Specs

Parameters / size: 5.3B

Output formats: Markdown, HTML, and JSON

Primary task: Document OCR (image-text-to-text)

Why it's interesting

State-of-the-art on the olmOCR benchmark: Chandra OCR 2 recieved 85.9% bench score on the olmOCR Benchmark and a 77.8% multilingual bench score (12% improvement over Chandra 1).

Support for 90 world languages: Indic script, European languages, and languages that read right to left say substantial improvemtns based on Datalab’s internal benchmarking. View the full list of languages and the benchmark results here: Chandra 2 Language List

Better complex layout understanding: Handles multi-level tables, nested structures, forms, math, and mixed handwriting with structured outputs (HTML/JSON/Markdown + bounding boxes), removing the need for post-OCR layout reconstruction. Take a look here:

Try it

Build an automated compliance intake pipeline using Chandra OCR 2 for structured extraction across complex, handwritten and form-based documents.

In this scenario, you’re supporting a state election commission processing large volumes of candidate filings submitted as scanned forms or mobile-captured images. These documents often include mixed handwriting quality, checkbox selections, signatures, and structured fields that must be validated for compliance.

Chandra OCR 2 can extract both printed and handwritten fields, identify form structure, and capture key elements such as candidate information, filing details, checkbox states, and signed declarations in a consistent JSON format. This structured output can then be passed into a compliance workflow to validate completeness, detect inconsistencies, and flag filings that require manual review.

This approach helps streamline high-volume intake while improving accuracy and reducing manual processing across complex document types.

Sample prompt: Extract all fields from this filing and return a structured JSON output including form type, candidate name, office sought, district, committee name, treasurer, filing date, checkbox states, and a transcription of the signed declaration. Include bounding boxes for each extracted field.

Z.ai: GLM-OCR

Model Specs

Parameters / size: 0.9B

Languages: Chinese, English, French, Spanish, Russian, German, Japanese, Korean

Primary task: Document OCR (image-text-to-text)

Why it's interesting

High accuracy at a compact scale: GLM-OCR achieves a score of 94.62 on OmniDocBench V1.5, showing strong performance on tasks such as formula recognition, table extraction, and document parsing—even at sub-1B scale

Designed for structured document understanding: The model performs well across complex document layouts, enabling extraction of tables, forms, and mixed text-image content

Optimized training for consistency across tasks: Uses Multi-Token Prediction (MTP) and full-task reinforcement learning to improve stability and accuracy across diverse document types

Efficient for real-world deployment: Its smaller footprint makes it well suited for scalable OCR pipelines where cost, latency, and throughput matter

Try it

Build a high-throughput document ingestion pipeline using GLM-OCR for structured extraction across diverse document types.

Imagine you are operating a customer onboarding platform that processes identity documents, invoices, and proof-of-income statements across multiple languages. GLM-OCR can be used to extract key fields—such as names, ID numbers, dates, and addresses—and output them in a consistent structured format for downstream systems.

The model’s compact footprint makes it well suited for scaling high-volume OCR workflows, enabling you to process large batches of documents efficiently while maintaining accuracy across layouts like tables, forms, and mixed text-image content.

Sample prompt: Extract the following fields from this document and return a structured JSON output: full name, ID number, date of birth, address, document type, and expiration date. Ensure all fields match the document exactly, including formatting.

Getting started

Whether you are coming straight from the Hugging Face hub or are already in Microsoft Foundry, deploying new open models is getting simpler. You can deploy models on Foundry by browsing the Hugging Face collection in the model catalog or you can choose "Deploy on Microsoft Foundry" on the Hugging Face website, which brings you straight into Foundry with secure, scalable inference already configured. Read the documentation to learn more:

Transitioning from Azure Language Features to Foundry Models

renaliu — Thu, 11 Jun 2026 18:06:43 GMT

Azure AI Language in Foundry Tools will begin deprecating eight features:

Key Phrase Extraction, Entity Linking, Sentiment Analysis & Opinion Mining, Summarization (document & conversation), Conversational Language Understanding (CLU), Custom Question Answering (CQA), Orchestration Workflow, and Custom Text Classification,

to be ultimately retired by March 2029 (Entity Linking even sooner, by September 20, 2028).

Microsoft’s guidance is to implement these functionalities using Microsoft Foundry-based models. This series of recommendation guides is intended to help you get started as you explore the breadth of Foundry-based options at your fingertips. Each standalone guide includes:

A brief overview of the Azure AI Language in Foundry Tools feature.
A recommended Foundry model and deployment for a similar solution.
A tutorial to implement the new solution via Foundry’s REST API.
A point-in-time comparison table listing alternate Foundry models you might consider for each feature.

Endpoint conventions used in this guide

The REST examples below use the unified Microsoft Foundry inference endpoint, which is the same Azure OpenAI path used for every model (OpenAI, Mistral, Cohere, Phi, and more): https://<your-resource>.services.ai.azure.com/openai/deployments/<your-deployment>/chat/completions?api-version=2024-10-21. Authenticate with header api-key: <YOUR_API_KEY>. <your-deployment> is the deployment name you assigned when adding the model to your Foundry resource.

For keyless (Microsoft Entra ID) authentication, swap the URL to https://<your-resource>.openai.azure.com/openai/v1/chat/completions and use header Authorization: Bearer <ENTRA_TOKEN> (issue the token with scope https://ai.azure.com/.default).

The older Azure AI Inference beta route (<endpoint>.<region>.models.ai.azure.com/chat/completions) is being retired on August 26, 2026 — use the unified path above for any new code.

New: Microsoft’s first-party MAI models in Foundry. Microsoft’s in-house MAI models are now available in Microsoft Foundry. For the text and language features covered in this guide, the relevant new option is MAI-Thinking-1 — Microsoft’s first reasoning model (sparse Mixture-of-Experts, ~35B active / ~1T total parameters, 256K-token context), introduced at Build 2026 and currently in private preview. It is Chat Completions API-compatible and supports function calling, so it slots into the same call patterns shown below, and it now appears as an alternate option in the comparison tables for the reasoning- and context-heavy features. The MAI media models released in April 2026 — MAI-Transcribe-1 (speech-to-text), MAI-Voice-1 (text-to-speech), and MAI-Image-2 (image generation) — are not covered here, since they address audio and vision rather than the text features being retired.

1. Key Phrase Extraction → Foundry with mistral-small-2503 (24B)

Key Phrase Extraction identifies main ideas or topics in unstructured text. For example, given the sentence “The food was delicious and the staff were wonderful.”, it returns the phrases “food” and “staff” as the key topics. In Foundry, you can replicate this by prompting a knowledge-dense model to extract significant terms from text and return them in JSON format. We recommend trying out mistral-small-2503, a 24-billion-parameter open model that rivals larger proprietary models on comprehension tasks.

Step 1: Deploy the Mistral model in Foundry. In your Foundry project, deploy the Mistral Small 3.1 model (mistral-small-2503). This model is licensed for enterprise use and can process extremely long documents (128,000 tokens) with state-of-the-art accuracy. Once deployed, note its deployment name and region from the Azure portal.

Step 2: Write a prompt for key phrase extraction. Instruct the model to extract key phrases and output them as JSON by writing a prompt for key phrase extraction. For example, your prompt can say: “Extract the key phrases from the given text and output them in JSON as a list.”

Step 3: Call the Foundry REST API with your text. Use your Foundry endpoint and API key to send a POST request to the model’s chat completion API (similar to how you called Azure’s API). Include a system message with extraction instructions, and a user message containing your text. For example:

1     POST https://<your-resource>.services.ai.azure.com/openai/deployments/<your-deployment>/chat/completions?api-version=2024-10-21
2     Headers: { "api-key": "<YOUR_API_KEY>", "Content-Type": "application/json" } 
3     Body: {
4       "messages": [
5         {
6           "role": "system",
7           "content": "Extract the key phrases (main topics) from the given text. Respond in JSON as {\"keyPhrases\": [ ... ]}."
8         },
9         {
10           "role": "user",
11           "content": "Text: \"The food was delicious and there were wonderful staff.\"\nTask: What are the key phrases?"
12         }
13       ]
14     }

Step 4: Receive the JSON output. The Mistral model will parse the text and return the key concepts in JSON form. For this example, it responds with:

1     {
2       "keyPhrases": [
3         "food",
4         "staff"
5       ]
6     }

You can write your prompt to return a desired structured output to ensure that it returns a list of extracted key phrases. Your application can then consume the "keyPhrases" array.

Other recommended models for Key Phrase Extraction:

Model	Rationale (When to Consider)
phi-3.5-mini-instruct	3.8B-parameter LLM from Microsoft’s Phi family, offering faster inference. Use if you need low latency or on-premise deployment – it’s smaller but trained on high-quality data and outperforms other models of similar size. It’s effective for straightforward key extraction, though not as nuanced as larger models.
claude-sonnet-4-6	Anthropic’s advanced model with multilingual skills and extremely long context (up to 200K tokens). Choose this if summarizing or extracting topics from very large or complex documents across many languages is required, and top-tier quality is worth the higher cost.

2. Entity Linking → Foundry with gpt-4o + Retrieval

Azure AI Entity Linking identifies named entities in text (people, places, organizations, etc.) and links each to a reference ID or entry. While Azure AI’s stand-alone Entity Linking will be retired, Microsoft’s officially recommended replacement is Azure AI Language in Foundry Tools’ Named Entity Recognition (NER), which continues to be supported and identifies the same entity types. If you need URL-linked entities, multilingual disambiguation, or richer reasoning beyond NER’s tag set, Foundry offers an alternative using a combination of knowledge retrieval and a generative model. One option is to try out gpt-4o, which excels at reasoning and handling complex instructions. By pairing it with your enterprise or public knowledge sources, you can link entities accurately.

Step 1: Set up an entity knowledge base. Prepare the reference data for your entities. For public knowledge, you might use Wikipedia; for internal terms, index the definitions using Azure AI Search. This knowledge base will allow the LLM to retrieve factual descriptions for any given entity.

Step 2: Deploy gpt-4o in Foundry. In your Foundry resource, deploy gpt-4o from the Foundry model catalog (or choose an equivalent high-accuracy model). This model is designed for broad reasoning and “agentic” tasks – meaning it can use tools and contextual data to solve problems. With strong general knowledge and reasoning, it’s ideal for disambiguating entities.

Step 3: Implement retrieval + linking prompt. When processing text, first retrieve relevant information about each potential entity. For instance, if the text says “Jane moved to Paris to work at Microsoft.”, search your knowledge base or call an API to get info on “Paris” and “Microsoft”. Then, craft a prompt that provides this context to the model and asks it to output a JSON mapping of entities to their IDs/URLs. For example:

1     {
2       "messages": [
3         {"role": "system", "content":
4             "You are an entity linking assistant. Identify entities in text and link each to its official reference ID and URL. Use provided context as needed and respond with JSON."},
5         {"role": "user", "content":
6             "Text: 'Jane moved to Paris to work at Microsoft.'\n\n" +
7             "Context:\n" +
8             "Paris – City in France (ID: Paris, URL: https://en.wikipedia.org/wiki/Paris).\n" +
9             "Microsoft – Technology company (ID: Microsoft, URL: https://en.wikipedia.org/wiki/Microsoft).\n\n" +
10             "Task: List each entity with its name, id, and url in JSON format as {\"entities\": [...] }."
11         }
12       ]
13     }

Step 4: Get the JSON linking output. The GPT-4o model, grounded by the context you provided, will output a JSON block such as:

1     {
2       "entities": [
3         {
4           "name": "Paris",
5           "id": "Paris",
6           "url": "https://en.wikipedia.org/wiki/Paris"
7         },
8         {
9           "name": "Microsoft",
10           "id": "Microsoft",
11           "url": "https://en.wikipedia.org/wiki/Microsoft"
12         }
13       ]
14     }

Other recommended models for Entity Linking:

Model	Rationale (When to Consider)
claude-sonnet-4-6	Anthropic’s advanced model, which has broad world knowledge and a large context window (up to 200K tokens). Useful if you want a non-OpenAI option with strong multilingual understanding and the ability to process very long texts for entity linking. It’s comparable in accuracy to GPT-4o, with a different safety profile.
phi-3.5-mini	A 3.8B-parameter open model fine-tuned by Microsoft. Consider fine-tuning this model on your specific entity domain if you need a self-hosted solution. Paired with a high-quality search index, it can perform domain-specific entity linking at a fraction of the cost – ideal for smaller knowledge bases and offline use (with some tuning).
MAI-Thinking-1	Microsoft’s first in-house reasoning model (sparse Mixture-of-Experts, ~35B active parameters, 256K-token context), in private preview on Microsoft Foundry. Its step-by-step reasoning suits disambiguating entities over retrieved context, and it supports function calling for tool-based knowledge lookups. Trained only on commercially licensed data with no third-party distillation, which helps with provenance in regulated scenarios. Requires preview access approval.

3. Sentiment Analysis & Opinion Mining → Foundry with phi-3.5-mini (3.8B)

Azure AI Sentiment Analysis evaluates whether text expresses a positive, neutral, or negative tone, while Opinion Mining provides granular sentiment on specific aspects or targets in the text. For example, in a hotel review “The room was great, but the staff was unfriendly,” the overall sentiment might be mixed, with aspect-level results: room: positive; staff: negative. We suggest you try a model, like Phi-3.5-mini model, which supports 128K context and was fine-tuned by Microsoft for quality instruction following. This model handles sentiment classification with high accuracy and speed, rivaling much larger models.

Step 1: Deploy the Phi model. In the Foundry portal model catalog, search for Phi-3.5-mini-instruct (128k) and deploy it. Once deployed, record the deployment name and have your API key ready for the following steps.

Step 2: Prepare your sentiment analysis prompt. Write a prompt that instructs the model to produce a sentiment analysis JSON output. Use a prompt like: "Analyze the following text. Provide 'sentiment' (positive, neutral, negative) and, if applicable, list 'opinions' of specific aspects and their sentiment. Respond in JSON." This explicitly asks for an overall sentiment and aspects that inform the predicted sentiment.

Step 3: Send the text to the model’s REST endpoint. Using your Foundry endpoint and key, call the phi-3.5-mini chat API with the text. For an example input:

1     {
2       "messages": [
3         {"role": "system", "content":
4             "Analyze the sentiment. Output JSON with overall 'sentiment' and 'opinions' for aspects mentioned."},
5         {"role": "user", "content":
6             "Review: 'The room was great, but the staff was unfriendly and rude.'"}
7       ]
8     }

Step 4: Review the output JSON. Once you adjust your prompt to your use case, the model can provide an analysis similar to Azure’s service output:

1     {
2       "sentiment": "mixed",
3       "confidenceScores": {
4         "positive": 0.5,
5         "neutral": 0.0,
6         "negative": 0.5
7       },
8       "opinions": [
9         {
10           "aspect": "room",
11           "sentiment": "positive"
12         },
13         {
14           "aspect": "staff",
15           "sentiment": "negative"
16         }
17       ]
18     }

The Phi model’s output is factual and concise, capturing the correct tone for each element. You can adjust the prompt for more or fewer details (e.g., add confidenceScores or not) as needed.

Other recommended models for Sentiment:

Model	Rationale (When to Consider)
mistral-small-2503	A larger (24B) model that may catch subtle nuances in sentiment. Its extra capacity can help in very complex or lengthy documents with mixed sentiment signals. Use if you need maximum accuracy for multi-paragraph analysis, or multi-language sentiment detection.
claude-sonnet-4-6	Anthropic’s advanced model with a deep understanding of context and tone. It’s especially capable for long-form text (e.g., multi-page reviews or chat transcripts) due to its large context window. Consider this if you require highly detailed sentiment reasoning and summarization in one go (with higher compute cost).

4. Text & Conversation Summarization → Foundry with claude-sonnet-4-6

Azure AI Summarization automatically produces a concise summary of input text. Azure AI’s summarization includes extractive (selecting key sentences) and abstractive (generating new summary text) modes for documents and conversations. To continue summarizing content, we recommend Claude Sonnet 4.6, known for its strength in summarization and very long context support. Claude can ingest large documents or transcripts and generate coherent, human-like summaries.

Step 1: Deploy the Claude model. In Foundry, deploy Claude Sonnet 4.6 as your model. With Claude deployed, note the deployment’s name for use in the API call.

Step 2: Create a summarization prompt. Decide on the summary style (short paragraph, bullet points, etc.). For example: “Summarize the following document in 2-3 sentences.” For conversation transcripts, you might ask: “Summarize the conversation below, highlighting key issues and resolutions.” Also specify that the response should be in JSON (e.g., {"summary": "..."}) for a more structured output style.

Step 3: Call Claude with your content. Use the Foundry API to send the text and prompt. For example, to summarize a document:

1     {
2       "messages": [
3         {"role": "system", "content": "You summarize texts. Output JSON with a 'summary' field."},
4         {"role": "user", "content": "Document: At any given moment, turnaround coordinators at Lufthansa CityLine monitor video feeds of airplanes at gates, ensuring each step of unloading and servicing planes happens on time. Even minutes of delay can add up, costing airlines millions of dollars a year. ...\n\nTask: Summarize the main point of this document in 2-3 sentences."}
5       ]
6     }

Step 4: Inspect the summary output. Claude will reply with a short summary, for example:

1     {
2       "summary": "Lufthansa CityLine uses real-time video monitoring to streamline aircraft turnarounds. By tracking tasks like unloading, refueling, and cleaning, they minimize delays and save significant costs."
3     }
4

Claude’s summarization quality is excellent for both extractive and abstractive summaries. Additionally, because Claude Sonnet 4.6 can incorporate enormous context, you can feed very long content directly without splitting it.

Conversation summarization: per-aspect starter prompts

The example above covers plain document summarization. If you are migrating the Conversation Summarization API, note that it let you request one or more summaryAspects: issue, resolution, recap, chapterTitle, and narrative — each with a distinct purpose, so a single generic summary prompt does not reproduce them. In Foundry, reproduce each aspect with its own targeted system prompt on claude-sonnet-4-6, sending the speaker-labeled transcript (for example, Agent: / Customer:) as the user message. The output field names below mirror the Azure API (summaries: [{aspect, text}]) so your integration code barely changes. Request several aspects in parallel calls, or combine them into a single call.

Issue — the customer’s problem (call-center focused)

{"role":"system","content":"You replicate the Azure AI Language conversation summarization 'issue' aspect. Read the agent/customer transcript and summarize ONLY the customer's primary problem — what they contacted support about — from the customer's perspective. Do NOT include any resolution or troubleshooting steps. Respond ONLY with JSON: {\"summaries\":[{\"aspect\":\"issue\",\"text\":\"<concise issue>\"}]}"}

Example output:

{"summaries":[{"aspect":"issue","text":"Customer could not set up the Wi-Fi connection for their Smart Brew 300 espresso machine."}]}

Resolution — what fixed it, or didn’t (call-center focused)

{"role":"system","content":"You replicate the Azure AI Language conversation summarization 'resolution' aspect. Summarize ONLY the actions the agent took to resolve the customer's issue and the final outcome, including any steps that failed or remain open. Do NOT restate the issue itself. Respond ONLY with JSON: {\"summaries\":[{\"aspect\":\"resolution\",\"text\":\"<concise resolution>\"}]}"}

Example output:

{"summaries":[{"aspect":"resolution","text":"The agent had the customer reset the Wi-Fi button and attempt a factory reset; neither restored the connection, so the issue was escalated for further troubleshooting."}]}

Recap — a brief one-paragraph summary of the whole conversation

{"role":"system","content":"You replicate the Azure AI Language conversation summarization 'recap' aspect. Condense the entire conversation into a single brief paragraph (2–4 sentences) capturing the participants, the main topic, and the outcome. Respond ONLY with JSON: {\"summaries\":[{\"aspect\":\"recap\",\"text\":\"<brief recap>\"}]}"}

Chapter title — topic-based segmentation with titles

{"role":"system","content":"You replicate the Azure AI Language conversation summarization 'chapterTitle' aspect. Segment the conversation into sequential chapters by topic, in chronological order, and give each a short title (max ~10 words). For each chapter, also return the verbatim opening line so offsets can be recomputed client-side. Respond ONLY with JSON: {\"summaries\":[{\"aspect\":\"chapterTitle\",\"text\":\"<chapter title>\",\"startsAt\":\"<first utterance of the chapter, verbatim>\"}]}"}

Azure returned character offsets per chapter. Because an LLM cannot reliably count characters, this prompt returns the verbatim opening line of each chapter; map it back to an offset with a string search on the client side.

Narrative — detailed call or meeting notes

{"role":"system","content":"You replicate the Azure AI Language conversation summarization 'narrative' aspect. Produce detailed call notes / meeting notes for the conversation — more thorough than a recap — in coherent prose or structured notes, preserving the order events occurred and any decisions or follow-ups. Respond ONLY with JSON: {\"summaries\":[{\"aspect\":\"narrative\",\"text\":\"<detailed notes>\"}]}"}

Migration note: where you previously passed summaryAspects: ["issue","resolution"] in a single call, now select the matching prompt(s) above. For multiple aspects, make parallel calls (cleanest — keeps each output focused) or combine the instructions and ask for one summaries array containing every requested aspect.

Other recommended models for Summarization:

Model	Rationale (When to Consider)
gpt-4o	OpenAI’s frontier model, superb at concise writing. It excels in accuracy and clarity when summarizing text, often capturing nuances. Use if you prefer OpenAI’s style or need highly polished results; it’s a bit slower but sets the state-of-art for quality.
mistral-small-2503	A strong open model (24B) that often matches larger LLMs on NLP tasks. It’s a cost-effective summarizer for moderate-length documents, or if you plan to fine-tune a summarization model. It runs on self-hosted infrastructure under an Apache license, giving flexibility and no dependency on external APIs.
MAI-Thinking-1	Microsoft’s first-party reasoning model, in private preview on Microsoft Foundry. Its 256K-token context window can ingest very long documents or transcripts in a single pass (Microsoft cites roughly a 600-page document), making it a first-party option for abstractive summarization where multi-step reasoning over the whole text matters. Chat Completions API-compatible, so it uses the same call pattern shown above. Requires preview access approval.

5. Conversational Language Understanding (CLU) → Foundry (claude-sonnet-4-6, or fine-tuned model)

Azure AI’s Conversational Language Understanding (CLU) allows chatbots to interpret user utterances by predicting a top intent and extracting key entities. For example, given “Book a flight to Paris tomorrow”, a custom CLU model might return Intent: BookFlight and Entities like Destination = Paris, Date = tomorrow. Foundry offers two solutions:

Approach A – Prompt-based CLU with an LLM: Use a single advanced model (like Claude Sonnet 4.6) to classify intents zero-shot and format a JSON response with the predicted intent and entities. This avoids any training and leverages the model’s high generalization ability.
Approach B – Fine-tuned custom model: Fine-tune an open model (e.g., Ministral 3B) on your existing labeled utterances to get a dedicated classifier for your scenario. This requires training (similar to CLU today) but yields maximum domain accuracy and offline deployment.

Approach A: Use claude-sonnet-4-6 as an Intent & Entity Model

Step 1: Gather CLU schema and examples. Assemble your existing list of intents (and what each means), plus any entity types. Include a None intent for irrelevant utterances as CLU does. For example, suppose your bot’s intents are: GetWeather (ask about weather), BookFlight (book a flight), CancelBooking (cancel a booking), and None (no match).

Step 2: Build a detailed prompt. Incorporate the above schema into a prompt for the model. You might create a system message like: “

"You are a conversational NLU classifier that mimics Azure Conversational Language Understanding (CLU). Your job is to read the user's text and return a CLU-like prediction JSON for intents and entities.\n\nTASK\n1) Identify the single best matching intent from the ALLOWED INTENTS list below.\n2) Extract zero or more entities from the text using the ALLOWED ENTITY TYPES list below.\n3) Return ONLY valid JSON that follows the OUTPUT SCHEMA exactly (no markdown, no explanation, no extra keys).\n\nALLOWED INTENTS (choose exactly one)\n- GetWeather: user is asking about weather conditions or forecast.\n- BookFlight: user is requesting to book, reserve, or search for a flight.\n- CancelBooking: user is requesting to cancel an existing reservation/booking (flight/hotel/etc.).\n- None: none of the intents above apply.\n\nALLOWED ENTITY TYPES (extract only these)\n- Location: city, region, airport, or place name that is relevant to the request.\n- Date: a date or date-like phrase (e.g., \"tomorrow\", \"next Friday\", \"2026-03-24\").\n\nSCORING RULES\n- intentConfidence must be a number from 0.00 to 1.00.\n- If the intent is None, set intentConfidence between 0.00 and 0.49 and return an empty entities array.\n- If an intent matches well, set intentConfidence between 0.70 and 1.00.\n- If uncertain between two intents, pick the best one and set intentConfidence between 0.50 and 0.69.\n\nENTITY RULES\n- Each entity object MUST include: category, text, offset, length, confidenceScore.\n- offset is the 0-based character index of the first character of the entity span in the ORIGINAL user text.\n- length is the number of characters in the span.\n- confidenceScore must be a number from 0.00 to 1.00.\n- Do not invent entities that are not explicitly present in the text.\n\nOUTPUT SCHEMA (return exactly this structure)\n{\n \"query\": \"<original user query>\",\n \"prediction\": {\n \"topIntent\": \"<best matching intent name>\",\n \"intents\": {\n \"GetWeather\": {\"confidenceScore\": <0.00-1.00>},\n \"BookFlight\": {\"confidenceScore\": <0.00-1.00>},\n \"CancelBooking\": {\"confidenceScore\": <0.00-1.00>},\n \"None\": {\"confidenceScore\": <0.00-1.00>}\n },\n \"entities\": [\n {\n \"category\": \"<entity type, e.g. Location or Date>\",\n \"text\": \"<entity span text>\",\n \"offset\": <integer>,\n \"length\": <integer>,\n \"confidenceScore\": <0.00-1.00>\n }\n ]\n }\n}\n\nCONSTRAINTS\n- Output MUST be valid JSON and MUST match the schema.\n- Populate confidence scores for ALL intents; the topIntent should have the highest score.\n- If no entities are found, return an empty array for prediction.entities.\n- Do not output any additional text before or after the JSON."},”

This explicit enumeration guides Claude to stick to the known intents only.

Step 3: Call the model with user utterances. Send a chat completion request for each user query. For example, for “Is it going to rain in Seattle tomorrow?”:

1     {
2       "messages": [

3         {"role": "system", "content":
4           "Intents:\n- GetWeather: user asks about weather.\n- BookFlight: user wants to book a flight.\n- CancelBooking: user wants to cancel a booking.\n- None: input doesn't match above.\nEntities to extract: Location, Date.\nOutput JSON with {\"intent\":..., \"entities\":...} for each user query."
5         },
6         {"role": "user", "content": "Is it going to rain in Seattle tomorrow?"}
7       ]
8     }

Step 4: Examine the JSON response. Claude uses your schema to infer the best intent and fills in entities:

1     {
2       "intent": "GetWeather",
3       "entities": [
4         {
5           "category": "Location",
6           "text": "Seattle"
7         },
8         {
9           "category": "Date",
10           "text": "tomorrow"
11         }
12       ]
13     }

Note that the example above is the simplified shape your application typically consumes — the prompt in Step 2 actually requires the model to return the full CLU-compatible schema (query, prediction.topIntent, prediction.intents.<each>.confidenceScore, and entities with offset, length, and confidenceScore). Extract just the fields you need on the client side, or relax the Step 2 schema to match the simplified output. With careful prompting, Claude Sonnet 4.6’s classifier can approach the accuracy of your old CLU model – without any fine-tuning – and is easy to maintain because you can tweak the schema in the prompt itself.

Approach B: Fine-tune a Custom Intent Classification Model

If you have many intents and domain-specific language, training a custom model gives the highest accuracy. Fine-tuning an open model on your utterances yields a tailor-made classifier you control and can be deployed off-cloud if needed.

Step 1: Export CLU training data. Collect the utterances, intent labels, and annotated entities from your CLU project. Ensure each example is labeled correctly. This will be your training dataset.

Step 2: Fine-tune an open model in Foundry. Use Foundry’s fine-tuning UI or API to train one of the supported base models — Ministral-3B, Qwen-32B, or Llama-3.3-70B-Instruct for open-source; gpt-4.1-mini or gpt-4o-mini for OpenAI-managed — with your data. During training, the model learns to classify your intents and tag entities as in CLU. For more detailed information on fine-tuning in Foundry, check out Customize a model with fine-tuning - Azure AI Foundry | Microsoft Learn. After training, you can deploy the fine-tuned model as a new endpoint.

Step 3: Integrate the fine-tuned model. Once deployed, use the Foundry REST API to send user utterances and get predictions. The response format can be customized, but you can tailor it to predict the predicted intent and possibly entity spans. For instance, if you send “I want to cancel my Contoso booking”, the model might output Intent: CancelBooking; Entity: { type: Subscription, value: “Contoso” } (assuming it learned those classes).

Step 4: Evaluate and iterate. Validate the new model against test data or real queries. You should find it highly accurate on your domain-specific utterances, often matching or exceeding CLU’s performance. Fine-tuning yields a model that doesn’t stray beyond your provided intents, ensuring no unexpected outputs. Additionally, you can re-train periodically with new data to continuously improve it.

Why fine-tune? If your chatbot has a large set of intents or very specialized language, a fine-tuned model will be more consistent and precise than prompting a general LLM. It also runs without sending data to third-party APIs (since the model is your custom deployment). This path is ideal for enterprises needing predictable, controllable NLU – you invest once in training and get a dedicated model that aligns exactly with your taxonomy and can be packaged for offline or on-prem use.

Other recommended models for CLU:

Model	Rationale (When to Consider)
gpt-4o	A cutting-edge OpenAI model suitable for Approach A if you prefer OpenAI’s LLM. It has superior reasoning skills and might handle extremely complex utterances or mixed-language queries more gracefully. Use it for prompt-based CLU if maximum accuracy is needed out-of-the-box (with corresponding cost).
Ministral-3B (fine-tune)	One of the open-source base models that Foundry supports for fine-tuning, with 3B parameters trained by Mistral AI. Despite the smaller size, Ministral-3B is competitive on instruction-following benchmarks and produces a tailor-made NLU classifier you can host privately. Use it when you want an Apache 2.0–licensed Mistral lineage that is actually supported in the Foundry fine-tuning UI.
MAI-Thinking-1	Microsoft’s first-party reasoning model (private preview on Foundry). Useful for Approach A zero-shot intent and entity classification, where careful reasoning over ambiguous utterances improves top-intent selection; multi-layered instruction following helps keep outputs on your fixed schema, and function calling lets you route intents to tools. A first-party alternative to gpt-4o or Claude. Requires preview access approval.

6. Custom Question Answering (CQA) → Foundry with gpt-4o or Cohere Rerank + Retrieval

Azure AI’s Custom Question Answering (CQA) allows you to upload your documents or FAQs and answer natural language queries from that content. You can use a Retrieval-Augmented Generation (RAG) pattern: search for relevant info in your data using Azure Search (L1), then use another model to rank and generate an answer based only on your knowledge base (L2).

Approach A: Use Cohere Rerank v4.0 Pro as a Deterministic L2 Ranker

Step 1: Export your existing CQA knowledge base. Export your current Custom Question Answering knowledge base. You should retain all existing question answer pairs. No retraining or rewriting of answers is required.

Step 2: Create an Azure AI Search index. Create a new Azure AI Search index that will store your existing QA pairs. Each QA pair should be uploaded as a document containing the stored question text and stored answer text. This index will replace the internal retrieval layer previously used by the CQA runtime.

Step 3: Deploy the Cohere Rerank v4.0 Pro model in Foundry. In Microsoft Foundry, deploy the Cohere Rerank v4.0 Pro model to your project. This model will replace the confidence scoring functionality that was previously handled by the CQA service and will be used to determine which stored answer best matches the incoming user query.

Step 4: Update your application’s query endpoint. Locate the existing service call in your application that sends user questions to the CQA endpoint. Replace this call so that user queries are now sent to your Azure AI Search index instead. A minimal index schema for QA retrieval is: id (key), question (searchable text), answer (retrievable text), and questionVector (vector field populated by an embedding model such as text-embedding-3-small). Configure Azure AI Search to run a hybrid query — combining BM25 keyword match on the question field with vector similarity on questionVector — and return the top 10 candidate QA pairs for each query. The reranker in Step 5 will reduce those candidates to the single best answer.

Step 5: Pass search results to the Cohere Reranker. Modify your application workflow so that the QA pairs returned from Azure AI Search are passed to the deployed Cohere Rerank model along with the user’s original query. This replaces the scoring logic that your application previously received from the CQA runtime.

Step 6: Select the highest ranked QA pair. Update your application logic to 1) sort the reranked results returned by Cohere and 2) select the highest ranked result. The answer associated with that QA pair can then be returned directly to the user using the same response logic that currently exists in your application.

Step 7: Return the selected answer to the user. Update your application response logic so that the answer returned to the user is taken from the top ranked QA pair produced by the reranker. This preserves the same deterministic answer selection behavior that existed in the original CQA workflow while allowing your application to operate using Foundry hosted models.

Step 8: Optional GPT-Based Fallback. If needed, you can introduce an optional GPT-based validation or fallback step to evaluate the top reranked results or generate a synthesized response when no candidate answer meets your required relevance threshold. This allows you to gradually introduce retrieval augmented generation without changing the primary deterministic selection path used by your application today.

Approach B: Use GPT-4o as a Generative Answerer (RAG)

Step 1: Index your content for retrieval. Take the knowledge bases (documents, manuals, Q&A pairs, etc.) you were using in CQA and load it into an index, such as Azure Search. Enable keyword vector search so that queries can find relevant passages, not just keyword matches.

Step 2: Deploy the GPT-4o model. Use Foundry to deploy gpt-4o as your answer generator. Also consider deploying a smaller embedding model for vector search if needed (e.g., OpenAI Ada or Mistral Embeddings).

Step 3: Formulate a retrieval-augmented prompt. When a user asks a question, your application will first retrieve the top matching content from the index (e.g., a paragraph from an FAQ or a snippet from a manual). Then, you send the question along with that content to GPT-4o, instructing it to use only the provided content to answer. For example: “Content: [insert sample retrieved passage] \n\n Question: [insert sample user question] \n\n If the answer is in the content, answer in one sentence; otherwise say “Answer not found”.”

Step 4: Call GPT-4o via REST API. The user message will include both the Context (retrieved content) and the Question. GPT-4o will then generate an answer grounded in that context. For instance, if a user asks “How do I reset my device?”, you pass in the relevant instruction from your manual as context. The model’s answer would be factual and derived from those instructions.

Step 5: Validate the answer output. The model should return an accurate answer, e.g.: “Press and hold the reset button for 10 seconds until the device restarts.” In scenarios where the context doesn’t cover the question, the model can be configured to say “Answer not found.” This ensures no hallucinated answers. You can also instruct the model to output references or citations to the source content if needed.

By combining search and an LLM, you can effectively build a question answering model on Foundry: the search step finds the answer, and the model produces a user-friendly statement from it.

Other recommended models for CQA:

Model	Rationale (When to Consider)
claude-sonnet-4-6	Another excellent choice for RAG solutions. Claude’s tremendous context window (hundreds of thousands of tokens) means you can feed large documents directly, without splitting, which is useful if you prefer a single-model approach. Its output is known for being detailed and polite, which some Q&A use-cases prefer.
Ministral-3B (fine-tune)	A Mistral-family open model supported by Foundry fine-tuning. With 3B parameters it’s lightweight enough to host on your own infrastructure, and after fine-tuning on your QnA dataset it can be specialized to extract answers from text with minimal deviation. Use it when you want a lower-cost, privately-hosted alternative to GPT-4o for narrow Q&A domains.
MAI-Thinking-1	Microsoft’s first-party reasoning model (private preview on Foundry). A strong RAG answerer when responses require reasoning across multiple retrieved passages; its 256K-token context lets you pass many candidate QA pairs or long source documents at once, and instruction following helps enforce “answer only from the provided context” grounding. Requires preview access approval.

7. Orchestration Workflow → Foundry Agent (using gpt-4o)

Azure AI’s orchestration workflow directs incoming requests to the correct Language sub-service (CLU, CQA, etc.) by first determining the user’s goal. Instead of maintaining a separate orchestrator, you can now rely on a Foundry agent – essentially a single LLM that can interpret user input and decide how to fulfill it using the available tools. GPT-4o, for instance, can analyze an utterance and either answer it or format an action, as needed.

Step 1: Define the agent’s tasks. Identify what distinct actions or knowledge the orchestrator used to handle. For example, perhaps the bot either answered a question (using QnA) or executed a command (like scheduling). We will rely on the LLM to detect which scenario applies and respond appropriately.

Step 2: Construct an agent prompt. Write a system message that explains the assistant’s capabilities. For example: “You are an AI assistant. If the user asks a knowledge question, answer it using the provided content. If the user makes a request to perform an action, output a JSON specifying the action.” This effectively gives GPT-4o a simple set of tools (answering vs. acting).

Step 3: Provide context or tools per query. In each conversation turn, include any info needed. For instance, if the user query is factual (“What’s our security policy?”), attach a relevant snippet from your data (like in the QnA solution). If the query is an action (“Schedule a meeting with IT”), you might not have extra context, but expect an action-oriented output.

Step 4: Evaluate the outputs. GPT-4o will analyze the input and produce either a direct answer or an action, following the instructions. For the fact question, it might output a paragraph answer; for the command, it could output a JSON like {"action": "ScheduleMeeting", "topic": "IT Support", "time": "tomorrow 10am"}. Because GPT-4o is an “agentic” model, it can reason about the user’s intent and choose the correct format on its own. This means the separate orchestration layer is no longer needed – the model’s reasoning replaces it.

Using an AI agent for orchestration leads to a simpler flow: you only call the LLM (and perhaps a search for facts) rather than maintaining multiple endpoints and routing logic. Foundry’s advanced models like GPT-4o have demonstrated they can handle multi-step tasks and tool usage in a single conversation, making them a powerful replacement for manual orchestration.

Other recommended models for Orchestration:

Model	Rationale (When to Consider)
claude-sonnet-4-6	A strong alternative for building an AI agent. Claude is known for coherent multi-turn reasoning and decision-making. It can be used if you prefer a non-OpenAI model; it integrates well with external tools and its large context helps when orchestrating complex, multi-step processes.
MAI-Thinking-1	Microsoft’s first-party reasoning model (private preview on Foundry). Built for agentic, multi-step reasoning and supports function calling, so a single MAI-Thinking-1 agent can interpret intent and route to the right tool or answer — a first-party alternative to gpt-4o for the orchestration role. Requires preview access approval.

8. Custom Text Classification → Foundry (fine-tuned Ministral-3B)

Custom Text Classification allowed you to train a model to classify text into your custom categories (e.g., labeling customer reviews as Complaint, Suggestion, or Praise). In Foundry, you will achieve the same by fine-tuning a foundation model with your data. The ideal base model is Ministral-3B, since it is one of the open-source base models Foundry currently supports for fine-tuning, and it is an Apache 2.0 licensed model with top-tier accuracy in language tasks. Fine-tuning this model with your training data will produce a robust classifier that matches or exceeds your previous model’s performance, with the added benefit that you can fully control and deploy it as needed.

Step 1: Prepare training data. Export your labeled text samples and their categories from the Language Studio (or however you stored them). Ensure you have a representative dataset for each label. Clean the data if needed (e.g., remove duplicates, ensure consistent labeling).

Step 2: Fine-tune Ministral-3B in Foundry. In the Foundry UI or using the Python SDK, start a fine-tuning job on Ministral-3B with your dataset. Foundry will handle the training process; you simply provide the data and some configuration (like number of epochs). Once training is completed, you’ll have a new model version specialized to predict your categories.

Step 3: Deploy and test the model. Deploy the fine-tuned model to get an endpoint. Then send test texts to its REST API. For example, if you send the text “The product arrived broken and I’m very unhappy”, the model might return:

1     {
2       "category": "Complaint",
3       "confidenceScore": 0.98
4     }

This indicates the text was classified as a “Complaint” with high confidence. The output format can mimic the original service’s JSON structure (as shown above), so integration changes are minimal.

Step 4: Integrate and iterate. Update your application to call the new Foundry model endpoint for classification requests. Because the model was trained on your exact labels and examples, it will produce one of your predefined classes every time (no extraneous outputs), and often with greater accuracy than before. Over time, you can retrain with more data or adjust the model if needed, giving you full control over the classifier’s evolution.

Other recommended models for Custom Classification:

Model	Rationale (When to Consider)
gpt-oss-20b (fine-tune)	An OpenAI-published open-weight model (20B) supported by Foundry fine-tuning. Train it on your labels for a classifier that runs on your own infrastructure or as a Foundry deployment. Use this when you want OpenAI-style quality at a fraction of GPT-4o’s inference cost and need an Apache 2.0–licensed open-weight option.
Llama-3.3-70B-Instruct (fine-tune)	The current open-weight Meta Llama model supported by Foundry fine-tuning. At 70B parameters it’s the largest of the open-source fine-tune options and gives you the highest ceiling for accuracy on complex or multilingual classification taxonomies. Choose this when you need maximum quality from an open model and have the budget to host a 70B deployment.

By following these guides, you’ll smoothly transition each Azure Language feature to a Microsoft Foundry solution. No direct replacements exist, but by leveraging foundation models + prompt engineering, you can achieve equal or better functionality. This approach consolidates tasks onto a unified, powerful platform with models that are continuously improving. The result is continuity for your applications and an upgrade in capabilities – from static APIs to the flexible world of Foundry’s LLMs. Each guide above equips you to implement the change with minimal friction, ensuring your users enjoy the benefits of modern AI with the structurally similar JSON outputs they expect.

Date extraction regression: 2025-05-01-preview vs 2025-11-01 (GA) in Azure Content Understanding

SGRoborana — Wed, 10 Jun 2026 09:28:58 GMT

Issue: When using the documentFieldExtraction scenario in Azure Content Understanding, the GA version (2025-11-01) produces significantly worse results compared to the preview version (2025-05-01-preview) for date field extraction on scanned (Dutch) medical documents.

Observed behavior:

With 2025-05-01-preview: all date fields are extracted correctly, including dates that are split across three separate handwritten fields (day, month, year).
With 2025-11-01 (GA): multiple date fields are either not found, returned as null, or extracted with day and year swapped (e.g. 2027-12-01 instead of 2001-12-27).

Document characteristics:

Scanned PDF (not native digital)
Dutch language
4-16 pages
Dates are handwritten and split across three separate labeled fields: dag (day), maand (month), jaar (year)
Year is written as 2 digits (e.g. "26" for 2026, "01" for 2001)

Schema used: documentFieldExtraction with type: date fields and explicit descriptions instructing the model to read day → month → year in order and expand 2-digit years to 4 digits.

Expected behavior: The GA version should perform at least on par with the preview version when using the exact same prompts. Is this a known regression? Any recommended workarounds while waiting for a fix?

Unable to Connect Localhost MCP Server from Azure AI Foundry Hosted Agent (o4-mini)

aayush7g — Wed, 10 Jun 2026 06:04:47 GMT

I'm using the Azure AI Foundry Toolkit in VS Code and have configured an MCP server running on my local machine (localhost).

When I run my Azure AI Foundry-hosted agent (o4-mini), it fails to connect to the MCP server. Based on the error logs, it appears that the hosted agent cannot reach the localhost endpoint.

My understanding is that the MCP server is running correctly locally, but the hosted agent seems unable to access services running on my machine.

Has anyone successfully connected a locally hosted MCP server to an Azure AI Foundry-hosted agent while using the Foundry Toolkit in VS Code?

Online Meeting Effectiveness Index (OMEI)

juhisingh — Tue, 09 Jun 2026 17:26:07 GMT

Samidha Shikhare*, Tim Malueg*, Shreya Goel*, and Charlie Goldenberg*
Department of Business Analytics, Santa Clara University - Leavey School of Business,
Santa Clara, California 95053, United States
*These authors contributed equally.
{sshikhare, tmalueg, sgoel, cgoldenberg}@scu.edu

This research was conducted as part of a Microsoft-sponsored Practicum Project at Santa Clara University's Leavey School of Business, in direct partnership with Microsoft's Commercial Engineering and AI Business division. The project was led by Juhi Singh, Bonnie Ao, and Suneel Innamuri, with a focus on delivering enterprise-grade, computationally informed insights to accelerate commercial AI adoption and drive scalable business transformation across Microsoft's engineering and go-to-market strategy.

INTRODUCTION
What if your meetings could tell you how to make them better? We’ve all been there: another hour-long meeting that could have been an email. Research shows 17% of meetings are perceived as ineffective — and that’s a conservative estimate. But what if AI could analyze your meetings and give you specific, actionable feedback on how to improve them? That’s exactly what we built. The Online Meeting Effectiveness Index (OMEI) uses Azure AI to score your virtual meetings across five dimensions — Participation, Engagement, Structure, Sentiment, and Tech Quality — and tells you precisely what to fix. Here’s how it works and what we learned from analyzing 68 real meetings.

1. Introduction
This paper presents the methodologies and evaluation parameters we developed to define a meeting effectiveness score using large language models. Meetings are central to organizational productivity yet remain poorly measured and frequently ineffective — Cutler et al. found that 17% of meetings were perceived as ineffective[3], a significant cost in wasted employee time, while Scott et al. identified that support for meeting intentionality is surprisingly scant in existing workflow technologies[6]. We designed this project to address that gap by developing an automated, LLM-driven framework capable of evaluating meeting quality across various real-world formats. Data was drawn from 68 publicly available recorded meetings on YouTube, categorized by size into small (3 to 5 participants), medium (6 to 10), and large (11 to 20), excluding one-on-one meetings where effectiveness can be gauged directly by whether the purpose was resolved. The resulting framework consists of five umbrella metrics — Participation, Engagement, Structure, Sentiment, and Tech Quality — each containing three sub-metrics, producing fifteen measurable indicators of meeting quality and providing a structured lens through which meeting recordings can be assessed against research-validated indicators of effectiveness, inclusiveness, and participation equity.

TL;DR: We built an AI system that grades your meetings on 5 dimensions. After testing 68 real meetings, we found most score well on tech quality (92/100) but struggle with engagement (77/100). The fix? Clearer agendas, intentional interaction points, and explicit action items at the end.

Key Findings at a Glance: 68 meetings analyzed across corporate, government, and academic settings | Average effectiveness score: 86/100 | Biggest opportunity: Engagement (77.3) | Strongest area: Tech Quality (92.3) | Speaking for just 10% of a meeting increases perceived inclusiveness 4x (Hosseinkashi et al., 2024)

2. How We Measure Meeting Effectiveness

Dimension 1: Emotional Tone & Psychological Safety

Workplace collaboration is shaped by the emotional tone set during meetings. Cutler et al. found that psychological safety — people's readiness to share their thoughts — is the strongest predictor of meeting success (OR=3.3), outweighing agenda, video, and meeting size, with inclusive meetings twice as likely to be viewed positively (OR=2.1) [3]. This dynamic intensifies in remote settings, where Cao et al. found that the absence of nonverbal signals increases multitasking and distraction, making verbal tone the primary determinant of emotional climate[1]. Hosseinkashi et al. found that communication failures reduce meeting success likelihood by 90% (OR=0.1) [4], Scott et al. identified social conflict as a major productivity obstacle[6], and Chen et al. demonstrated that AI-assisted reflection can drive measurable behavior change[2]. Based on this evidence, OMEI scores sentiment tone on a 0-100 scale: 80-100 reflects a psychologically safe environment; 50-79 moderate tone with friction; 20-49 neutral tone; and 0-19 negative or dismissive tone.

Dimension 2: Active Engagement & Agenda Adherence
Active participation is the strongest predictor of meeting inclusiveness and effectiveness in virtual settings. Hosseinkashi et al. after analyzing 15,000 real meetings across five multinationals, found that speaking for at least 10% of the meeting increased the likelihood of an 2 inclusive meeting fourfold (OR=4.0), with small groups of eight or fewer producing seven times higher odds of vocal engagement (OR=7.1), and effectiveness dropping by one percentage point for every two additional participants[4]. Scott et al. found that mixed-role groups and unclear objectives consistently produced off-topic discussion even when agendas were present[6], while Chen et al. observed that 7 of 12 real workplace meetings drifted from their objectives mid-discussion despite having reference materials available[2]. Based on this evidence, OMEI scores engagement across responsiveness and agenda adherence on a 0-100 scale: 80-100 reflects broad vocal participation consistent with Hosseinkashi et al.'s OR=4.0 threshold; 50-79 moderate but uneven participation; 20-49 limited participation with frequent off-topic discussion; and 0-19 one-sided communication with no agenda adherence.

Dimension 3: Who Spoke, How Much, and How Equitably
Participation is the strongest predictor of meeting effectiveness and inclusiveness. Hosseinkashi et al. found that vocal involvement correlates with a fourfold increase in perceived inclusiveness, yet group size consistently works against this — attendees in groups under eight are seven times more likely to contribute, and effectiveness drops by roughly one percentage point for every two people added[4]. Cutler et al. found that 17% of meetings are perceived as ineffective, and that 20% of respondents felt excluded even when 83% had spoken at least twice, suggesting that how voice and time are distributed matters more than whether someone spoke at all[3]. Chen et al. adds that silent attendees are a proxy for lost psychological safety, as junior staff often stay quiet due to social hierarchies[2]. To capture this, our participation metric is built on three pillars: unique speakers, which tracks whether contribution is truly distributed; speaking time balance, operationalized through the Gini coefficient (a measure of how evenly speaking time is distributed — 0 means perfectly equal, 1 means one person dominated) and dominant speaker share; and turn-taking, which distinguishes real dialogue from disjointed monologues and reflects Cao et al.’s finding that multitasking spikes in roughly 30% of remote meetings when discussion is dominated by a few voices[1]. Together these sub-metrics create a composite score that addresses the lack of intentionality in current workflow tools[6].

Dimension 4: Meeting Structure & Outcomes
Structure is a critical determinant of meeting effectiveness. Scott et al. found that unclear goals are among the most significant obstacles to productivity, noting that current workflow tools provide surprisingly little support for meeting intentionality[6]. Their research draws a direct line between structure and outcomes — if goals set the destination, agendas outline the route, functioning as implementation intentions that consistently increase the likelihood of goal achievement. Yet Chen et al. found that even when agendas existed, fewer than half of observed meetings articulated specific goals, none allocated time per agenda item, and time management behaviors typically only emerged when participants realized they were running out of time — 3 resulting in overly long discussions in 6 of 12 meetings and under-addressed topics in 4 of 12[2]. Our structure metric evaluates three sub-dimensions that together capture whether a meeting was intentionally organized, purposefully conducted, and productively concluded. Agenda adherence assesses not merely whether an agenda was present, but whether the meeting followed it, stayed within scope, and gave appropriate time to stated items. Topic threads measure whether discussion followed a coherent arc, with topics opened and closed rather than abandoned — Chen et al. found demonstrable off-target discussion in 3 of 12 meetings and core issues left unaddressed in 4 of 12[2]. Action items assess whether the meeting translated discussion into concrete commitments with named owners, reflecting Cutler et al.'s finding that post-meeting summaries correlate directly with meeting quality and effectiveness[3].

Dimension 5: Technical Quality as a Prerequisite
Technical quality is a foundational prerequisite for participation in remote meetings, not a peripheral IT concern. Hosseinkashi et al. found after analyzing 15,000 meetings that reliability failures reduced participation odds by 87% (OR=0.13) and poor audio and video quality reduced meeting success likelihood by 90% (OR=0.1), drawing a useful distinction between reliability — staying connected — and quality, which determines whether a meeting is inclusive and effective[4]. Cao et al. found that camera-off behavior strongly predicts multitasking and disengagement[1], Scott et al. identified technology fragmentation as a key reason meetings lose intentionality[6], and Chen et al. found that technical failures directly disrupt goal tracking[2]. Cutler et al. established that audio and video reliability each independently increase the likelihood of an inclusive meeting (OR=1.7) — comparable in effect to using a formal agenda — with headset use alone increasing participation odds by 20% (OR=1.2) [3]. Based on this, OMEI scores technical quality on a 0-100 scale: 80-100 reflects no detected issues enabling full participation; 50-79 minor degradation without reliability failures; 20-49 significant quality barriers limiting engagement; and 0-19 reliability failures preventing meaningful participation.

3. The Technical Pipeline: From Video to Score
The project’s architecture follows a four-layer process designed to transform raw video recordings into structured meeting effectiveness insights. The pipeline consists of a data ingestion layer, perception layer, intelligence layer, and output layer. Together, these layers convert videos into transcripts, metadata, metric level LLM evaluations, and a final meeting effectiveness score.

3.1 High Level Architecture
The data ingestion layer collects meeting recordings from publicly available video sources. A YouTube URL is passed into a video download pipeline using yt-dlp and FFmpeg. The processed video file is then uploaded to Azure Blob Storage, where it becomes available for downstream analysis. The perception layer extracts meeting information from the stored video. Azure Video Indexer is used to generate diarization, transcription, sentiment signals, and meeting metadata. These outputs provide the foundation for evaluating who spoke, what was discussed, and what interaction patterns appeared during the meeting. The intelligence layer combines preprocessed transcript data with the Azure Video Indexer JSON output. These structured inputs are sent to Microsoft Foundry for LLM based analysis. For each OMEI metric, the system uses a customized system prompt and requests structured output, allowing the model to evaluate each of the 5 metrics in a replicable and consistent format. The output layer presents the final results through a lightweight web demo formatting the returned structured output. The interface displays the core OMEI metrics designed to help users understand not only the numerical score rating the system has provided, but also which dimensions contributed to the score as well as positive comments and potential improvements.

3.2 LLM Methodology
The evaluation pipeline orchestrated with Microsoft Foundry combines a guided system prompt with an appended diarization user prompt for LLM assessment. System prompts were iteratively refined using findings from research, with each prompt aligned to its corresponding metric. GPT-5.4-mini was selected as the final model for its ability to process full transcripts within its context window while providing reliable scoring and output. Since none of the models tested were capable of complex time-based calculations from diarization data, selected quantitative fields — including total speaking time per participant, balance ratios, and silences — were preprocessed before LLM evaluation to improve consistency and reduce model error.

4. What We Learned from 68 Meetings

4.1 Quality Assurance Approach
To ensure the integrity and objectivity of our model's outputs, we adopted a structured quality assurance approach grounded in independent human review. Seven student analysts were recruited for their unbiased perspective, having had no prior exposure to the model's outputs or scoring criteria. Analysts reviewed a standardized dataset of 14 unique corporate and institutional meeting recordings spanning government, academic, and professional contexts, evaluating only the first 30 minutes of each recording to ensure consistency. Evaluations were submitted through a structured Google Form aligned to the five core metrics, normalized to a composite scale of 100 points with Participation, Engagement, Structure, and Sentiment each contributing 22.5 points and Tech Quality contributing 10 points. The resulting dataset comprised 20 reviews yielding a global composite average of 66.84 out of 100, with Engagement the strongest metric at 18.36 points and Tech Quality the lowest at 5.5. Video-level scores ranged from 79.26 to 44.25 — a 35-point spread confirming the rubric differentiates meaningfully between strong and weak meeting performance, as illustrated in Figure 2. This human-generated dataset serves as the ground truth against which our model's outputs are measured, validated, and iteratively refined.

To evaluate the impact of prompt calibration, we compared original LLM scores, revised LLM scores, and human QA scores across all 16 videos, shown in Figure 3. Prior to calibration, the original LLM averaged 87.1 — approximately 20 points above the human QA average of 66.84 — confirming that generous default logic was systematically inflating scores regardless of actual meeting quality. Prompt revisions were applied across four metrics, replacing default scoring logic with explicit requirements for affirmative evidence and hard scoring caps tied to measurable signals such as dominant speaker percentage, silence ratio, and agenda adherence. Following revision, the revised LLM average dropped to 72.5, closing the gap with human reviewers from roughly 20 points to approximately 6, with the revised scores showing greater variance across videos — confirming that the model was now differentiating between stronger and weaker meetings in a way that more closely mirrors human judgment.

4.2 Overall OMEI Results
Across 68 tested meetings, the average OMEI score was 86.0 out of 100 based on 340 metric scores. Overall performance was strong, with the highest averages in Tech Quality at 92.3, Sentiment at 90.3, and Structure at 86.6, as shown in Figure 4. Engagement was the lowest at 77.3, indicating the greatest area for improvement. Scores ranged from 18 to 100, with the widest variation appearing in Sentiment, suggesting that while most meetings were technically stable, the LLM was more sensitive to differences in tone than in other dimensions. By meeting type, Review Sessions scored highest at approximately 90.4, reflecting consistently strong performance across all five dimensions, while Presentations scored lowest at approximately 85.6, with Engagement at 72 and Structure at 83 likely reflecting the more one-directional nature of that format. Meeting size was indirectly captured through the Participation metric using speaker occurrences identified by Azure Video Indexer as the primary proxy for contribution patterns.

4.3 LLM Model Performance
The OMEI pipeline tested GPT-5.4-mini, GPT-5.4-nano, DeepSeek-R1, GPT-5.4, and Phi-4 at the individual metric level, with all models able to process the full Azure Video Indexer JSON within their context windows. GPT-5.4-mini was selected as the final model for its best balance of scoring quality, consistency, and actionable recommendations — larger models tended to over-interpret the task while smaller models produced weaker rationales. Models performed well on qualitative assessments including tone, structure, and engagement, and generally avoided hallucinating evidence. The primary limitation was numerical reliability, as models were weaker at tracking counts, proportions, and speaker activity without preprocessing — particularly for the Participation metric, where speaker occurrences were preprocessed before scoring to improve consistency. Feedback across all models tended toward generic recommendations, likely reflecting the constraints of the rule-based evaluation framework.

4.4 Recommendations for Meeting Organizers
Meeting organizers can use OMEI results as a practical guide for improving meeting design and facilitation. The strongest opportunities come from clearer agendas, defined objectives, stronger facilitation, and more explicit closing practices — meetings should begin with a stated purpose and end with summarized decisions, action items, and named owners. In presentation-heavy or larger meetings, organizers should intentionally build in interaction points such as questions, discussion pauses, or participant check-ins. OMEI scores should be interpreted as directional indicators rather than definitive judgments, helping teams identify recurring patterns and improve meeting habits over time.
4.5 Recommendations for Software Integration
The current OMEI workflow is a proof of concept pipeline for native Microsoft Azure integration, deliberately designed to be lightweight and straightforward to extend. Although Azure Video Indexer was used in this prototype, Azure Speech may be better suited for production use since scoring relies primarily on transcripts, speaker activity, and timing data rather than visual analysis. Preprocessing numerical signals such as speaker counts, timing, and participation patterns before LLM scoring remains necessary to improve consistency and reduce model error, and future integration should preserve the framework's simplicity while improving transcript processing, metadata extraction, and preprocessing reliability.
4.6 Future Work
Future work should expand the dataset across more organizations, sectors, and meeting types, with richer metadata including participant counts, roles, meeting purpose, and duration. Human evaluator ratings should be used to validate OMEI scores against expert judgment, while further development should focus on improving preprocessing, reducing generic feedback, and deepening analysis of turn-taking, interruptions, silence, and action item quality. Longitudinal dashboards would allow teams to track meeting quality trends over time and evaluate whether changes in meeting practices lead to measurable improvement.
5. Get Started with OMEI
Want to evaluate your own meetings? Here’s how to get started:

Build your own pipeline: Azure Video Indexer documentation | Microsoft Foundry quickstart | Sample prompts from this research

Connect with the team: Questions or feedback? Reach out via the Microsoft Tech Community discussion thread linked above.

Learn more: Meeting effectiveness research from Microsoft | Azure Speech Services for production implementations

6. Conclusion
This project demonstrates that large language models can transform unstructured virtual meeting data into meaningful evaluations of meeting effectiveness. Through the Online Meeting Effectiveness Index, we developed a structured framework that scores meetings across five core dimensions — Participation, Engagement, Structure, Sentiment, and Tech Quality — by combining Azure Video Indexer outputs, deterministic preprocessing, and LLM-based qualitative reasoning to generate interpretable scores and actionable recommendations. Our findings show that while most virtual meetings perform well in technical quality and sentiment, engagement and structure remain consistent areas for improvement, particularly in presentation-heavy or larger formats. OMEI serves as a proof of concept for integrating reflective meeting intelligence into platforms such as Microsoft Teams, providing organizers with targeted feedback on agendas, participation balance, discussion flow, and follow-up practices.
Future work should expand the dataset, validate scores against expert human judgment, and improve analysis of turn-taking, interruptions, action items, and long-term meeting trends. With continued refinement, OMEI has the potential to help organizations move beyond simple transcription toward measurable, AI-supported improvements in collaboration, productivity, and decision-making.

References
[1] Cao, H., Lee, C.-J., Iqbal, S., Czerwinski, M., Wong, P. N. Y., Rintel, S., Hecht, B., Teevan, J., & Yang, L. (2021). Large scale analysis of multitasking behavior during remote meetings. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Article 448. https://doi.org/10.1145/3411764.3445243
[2] Chen, X., Tankelevitch, L., Vanukuru, R., Scott, A. E., Panda, P., & Rintel, S. (2025). Are we on track? AI-assisted active and passive goal reflection during meetings. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Article 705. https://doi.org/10.1145/3706598.3714052
[3] Cutler, R., Hosseinkashi, Y., Pool, J., Filipi, S., Aichner, R., Tu, Y., & Gehrke, J. (2021). Meeting effectiveness and inclusiveness in remote collaboration. Proceedings of the ACM on Human-Computer Interaction, 5, (CSCW1), Article 173. https://doi.org/10.1145/3449247
[4] Hosseinkashi, Y., Tankelevitch, L., Pool, J., Cutler, R., & Madan, C. (2024). Meeting effectiveness and inclusiveness: Large-scale measurement, identification of key features, and prediction in real-world remote meetings. Proceedings of the ACM on Human-Computer Interaction, 8, (CSCW1), Article 93. https://doi.org/10.1145/3637370
[5] Park, G. W., Panda, P., Tankelevitch, L., & Rintel, S. (2024). The CoExplorer technology probe: A generative AI-powered adaptive interface to support intentionality in planning and 12 running video meetings. Proceedings of the 2024 ACM Designing Interactive Systems Conference, (pp. 1638–1657). https://doi.org/10.1145/3643834.3661507
[6] Scott, A. E., Tankelevitch, L., & Rintel, S. (2024). Mental models of meeting goals: Supporting intentionality in meeting technologies. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, Article 1025. https://doi.org/10.1145/3613904.3642670

Driving Stop-and-Go Business Processes to Closure with Foundry Hosted Agents

srikantan — Mon, 08 Jun 2026 16:30:00 GMT

Why This Matters

Your AI agent just forgot everything. Again.

If you've built agents that need to track work across days or weeks—not just single conversations—you've hit this wall: the agent loop is stateless. Each turn reasons over what it's handed, then forgets. But real business workflows don't fit in one conversation.

This post shows how to build agents that remember, using Foundry's per-project sandbox VMs to maintain state across weeks of discontinuous work. No external database required.

What you'll learn: - How to persist agent state in Foundry microVMs - Session handling patterns for long-running workflows - When to choose hosted vs. local agent architecture

Quick glossary for this post:

MAF (Microsoft Agent Framework): The framework for building AI agents that can use tools and orchestrate sub-agents
Foundry: Microsoft's platform for hosting and running AI agents in production
MicroVM: A lightweight, isolated virtual machine that Foundry provisions per project—think of it as your agent's persistent workspace
MCP (Model Context Protocol): A standard for connecting AI models to external tools and data sources
WorkIQ: Our custom component that exposes M365 capabilities as MCP endpoints

The business workflow and the core challenge

The reference workflow here is Statement of Work (SOW) response orchestration. A SOW Owner kicks off an RFP response and the agent runs the whole engagement across Microsoft 365: it finds the RFP, drafts a project charter, extracts owners and tasks from the kickoff meeting, allocates the work, fans out kickoff briefs (after the owner approves), then polls email and chat to track each task to closure and drafts nudges for stragglers. The catch is that this isn't a single conversation — it plays out over days or weeks, with people contributing at their own pace and the agent waking up intermittently to advance the work. That is the core challenge: an agent loop is stateless — each turn reasons over what it's handed and then forgets — yet the process demands continuity across long, discontinuous gaps. Something has to remember every outstanding task, every submission, and every next step, and reconstruct that picture flawlessly each time the loop wakes.

The solution approach

Rather than push that burden onto the client, we let the Foundry hosted agent carry its own state in a per-project sandbox VM (microVM). The high-level idea:

State lives in the sandbox, not the client. Each project gets its own Foundry microVM with a durable $HOME. The agent writes everything it needs there — the charter, a structured project log of tasks and status, channel cursors, an activity trail.
Every wake-up reconstructs the world. The loop doesn't rely on a dangling in-memory pointer. On each turn it reads its own state files back into context, rebuilds the full status of the engagement, advances exactly the phase that's due, and writes the updated state back. Whether the human returns in five minutes or five days, the agent recreates an authoritative picture from disk.
The client stays thin. Because the substance is server-side, the desktop app only needs to remember which sandbox belongs to which project — a session id — and nothing sensitive ever lands on the endpoint.

This turns a stateless loop into a genuinely stateful, long-running workflow without an external database and without parking business data on the local machine.

Architecture and design

The system is two components.

1) The Foundry Hosted Agent is the brain: built using the Microsoft Agent Framework (MAF), reasons with OpenAI models on Foundry, reaches Microsoft 365 through a single Foundry Toolbox fronting the WorkIQ MCP endpoints, and owns the agent loop, skills, memory, session, state, and orchestration — calling M365 as the signed-in user via Foundry OAuth Identity Passthrough, so it never holds a user token.

The workflow is built as an orchestrator skill (sow-response) with specialist sub-skills (RFP search, charter draft, kickoff extract, task allocate, reply poll); the orchestrator inspects state, decides the current phase, and delegates to the right sub-agent.

Every skill is a warm MAF Agent built once at boot, all sharing one FoundryChatClient — so sub-agent invocations reason through the same model client. The skill-based design is what makes it extensible: a skill is just a folder with a SKILL.md; drop a new one in and the loader builds it an agent and starts routing to it by its description — a brand-new business process, no framework changes.

The agent runs locally during development and deployed as a Foundry hosted agent in production, identical code either way.

2) The second component, the Desktop App, is a deliberately thin shim: it signs the user in, attaches their bearer to the call, remembers a session id per project, polls in the background so the workflow advances unattended, and renders the result — holding session pointers and a non-sensitive UI cache, nothing more. The two talk over the OpenAI Responses API. MAF supplies the loop itself — the per-turn reasoning, tool dispatch over MCP, and sub-agent orchestration that drives every phase to completion.

Session and state handling in code

On a project's first turn, the Desktop client asks Foundry to mint a session. The returned agent session id is the handle to the distinct microVM provisioned for that project, and the client persists it immediately:

def _ensure_session_sdk(self, project_dict, isolation_key) -> str: # project_dict: this project's persisted record (label, ids, etc.). # isolation_key: a STABLE client-owned key — we pass the project_id — that # tells Foundry "all turns with this key belong to the same session/microVM". # If we already minted a session for this project on a previous run, reuse it. # session_id is the handle to the project's durable sandbox VM. sid = project_dict.get("session_id") if sid: return sid # sandbox already exists — reattach later # First turn for this project: ask Foundry to provision a new session. # create_session spins up a dedicated microVM whose $HOME will hold ALL # of this project's state. The platform MINTS the id; we don't choose it. session = self._get_project_client().beta.agents.create_session( agent_name=agent_name, # which deployed agent to bind to isolation_key=isolation_key, # = project_id, pins one session per project ) sid = session.agent_session_id # the server-minted sandbox handle # Persist the id BEFORE the first request goes out, so a crash mid-turn # never strands the project without a pointer to its sandbox. project_dict["session_id"] = sid _save_projects(self._projects_data) # write the local pointer file return sid

From then on, every request embeds that session id (reattaching the sandbox) over the Responses API surface — responses.create(stream=True, ...) — plus a previous_response_id to chain the transcript:

# oc: the OpenAI-compatible Responses client for the hosted agent. It's obtained # from the Foundry project client (AIProjectClient.get_openai_client(...)) and # is what actually puts the request on the wire to the agent's /responses endpoint. oc = self._get_openai_client() # Build the payload for one turn against the OpenAI Responses API surface. create_kwargs = { "input": prompt, # the user/auto message for this turn "stream": True, # stream SSE events back as they happen # agent_session_id is the load-bearing field: it tells Foundry to REATTACH # this project's existing microVM (and its $HOME state) for this turn, # instead of starting fresh. This is what makes the loop stateful. "extra_body": {"agent_session_id": _session_id}, } # previous_response_id chains the in-memory transcript from the prior turn so the # model has recent conversational context. It's best-effort (the transcript can # roll/expire); the durable state always comes from $HOME, not from this id. if previous_response_id: create_kwargs["previous_response_id"] = previous_response_id # Fire the turn. The hosted agent mounts the right sandbox, runs the agent loop, # and streams events (text deltas, tool calls, completion) back to the client. stream = oc.responses.create(**create_kwargs)

The agent_session_id pins the durable sandbox (survives restarts and idle); the previous_response_id chains the in-memory transcript (best-effort). Inside the sandbox, the agent rehydrates and updates its own state through system state tools exposed alongside the Toolbox — chiefly a JSON-patch operation so each skill updates only the keys it owns:

@tool # exposed to the agent as a callable tool, alongside the WorkIQ MCP tools def state_patch_json(path: str, patch: dict[str, Any]) -> str: """Merge top-level keys into a JSON file without overwriting unrelated fields. Used for every project_log update. path: a path RELATIVE to the sandbox $HOME (e.g. "project_log.json"). state.patch_json validates it stays inside $HOME — no escaping out. patch: the keys to merge in. Only these keys are touched; everything else in the file is preserved. This lets one skill update, say, a task's status without clobbering fields another skill owns. """ return state.patch_json(path, patch) # read-modify-write, atomic

The whole "reconstruct the world on every wake-up" guarantee rests on these reads and patches against $HOME.

Telemetry, the easy way

Because the agent runs on Foundry with MAF, connecting the Foundry project to Application Insights gives you the entire agent loop as OpenTelemetry traces, turnkey — model calls, every MCP tools/call, every sub-agent invocation, every state patch. The only custom wiring is a span processor that stamps project.id on each span at boot; the platform owns the exporter. You see the full multi-week loop end to end with no bespoke logging pipeline to build.

The alternative: run the loop on the client

The other option is to run the agent loop and its state on the local machine. The trade-offs:

Data residency. Hosted: sensitive substance stays in an Azure-governed sandbox; the endpoint holds only session ids — ideal where compliance limits business data on local devices. Local: charter, artifacts, and task ledger all live on the laptop, expanding the DLP and vulnerability footprint.
Scale & sharing. Hosted: per-project sandboxes are provisioned by the platform and any signed-in teammate can attach. Local: bounded by one user's machine; multi-user collaboration is awkward.
Simplicity for a single user. Local: no session plumbing, no cold-start latency, fully offline-capable — genuinely simpler for one power-user on a permissive setup. Hosted: a little more wiring (session minting, resume handling) in exchange for the governance and scale.

When to choose what: go hosted for teams, multi-project portfolios, or any regulated environment; keep it local for a single power-user on a data-permissive machine who values zero roundtrips.

See it in action

Get Started

Ready to build your own stateful agent? Here are three paths forward:

🚀 Explore the code: The complete Charter Agent source is on GitHub: Charter-Agent-Repo — clone it, run it locally, and adapt the patterns for your workflow.

📚 Learn the fundamentals:

Microsoft Agent Framework documentation — understand the agent loop and skill architecture
Foundry hosted agents quickstart — deploy your first agent in minutes
Working with Sessions in Foundry Hosted Agents
Foundry Toolbox overview — connect your agent to M365 and other services

What workflow would you build with a stateful agent? Drop a comment below.

A Multi-Region Microsoft Foundry Pattern for Enterprise Private Networking

pratyushmishra — Fri, 05 Jun 2026 17:30:00 GMT

Enterprise AI projects do not move to production on model quality alone. They move when architecture, networking, observability, and evaluation all work together in a way that fits existing enterprise constraints. That is why a multi-region Microsoft Foundry pattern matters: it lets teams deploy Foundry where capacity and platform readiness are available, while continuing to use existing private resources in another region without breaking security or operational expectations.

The architecture pattern: Foundry in one region, enterprise resources in another

In this pattern, a Foundry account and project are deployed in the region that best fits platform availability, quota, or rollout needs, while enterprise-owned services such as Azure AI Search, Azure Cosmos DB, Key Vault, storage, and Application Insights remain in their approved home region. The two sides are connected through private networking, private DNS, managed identity, and controlled outbound access. The result is an architecture that preserves existing landing-zone investments while still enabling modern agent scenarios in Foundry.

Why this pattern is useful in real enterprise environments

This approach is especially useful when organizations already have production resources, compliance approvals, and operational processes anchored in a specific region. Instead of rebuilding every dependency for a new AI project, teams can place Foundry where they have capacity and feature readiness, then securely connect back to the resources they already trust. This is what makes the pattern practical: it reduces migration friction while keeping the deployment model realistic for enterprise platform teams.

What this architecture proves: tools, evaluations, and tracing still work

The most important question for customers is not whether the network diagram looks clean on paper. It is whether the key platform experiences still function when Foundry is deployed in one region and the data plane dependencies remain somewhere else. This architecture demonstrates that agent tools can still execute against private resources, that prompts and hosted agents can still access enterprise systems through approved paths, and that the overall solution remains usable under real enterprise constraints.

Just as importantly, the architecture shows that critical operational workflows continue to work as expected. Batch and agent evaluations can run with Foundry orchestrating execution while reaching private dependencies across regions. Telemetry can still flow into Application Insights for logs and diagnostics. Conversation history, tool activity, and operational traces remain visible through the supported observability surfaces. For enterprise teams, that is the real validation: not just that the deployment succeeds, but that the development, testing, and troubleshooting lifecycle still holds together.

The screenshot below shows how a prompt agent can use Knowledge IQ (powered by Azure AI search)

And tool call details can be seen in foundry

And the agent invocation details in the configured application insights

The foundry agent could also be quickly evaluated using the evaluation feature of the Foundry platform

Core architectural components behind the pattern

At the center is a Foundry project deployed in the region selected for platform readiness. Around it are the services most enterprise agent solutions already depend on: search for grounding and retrieval, Cosmos DB for conversation or application state, Key Vault for secrets, storage for artifacts and datasets, and Application Insights for diagnostics. These services do not have to be recreated in the Foundry region. Instead, the architecture connects to them through private endpoints, private DNS, and identity-based authorization.

That combination is what makes the pattern credible for production use. Standard service names resolve privately, public ingress can remain disabled on sensitive resources, and agent runtimes communicate only through approved paths. The architecture is designed so the question is not whether enterprise controls must be relaxed for AI adoption, but how Foundry can fit cleanly into the controls that already exist.

This pattern is already in use by enterprise teams running production Foundry workloads. If your organization has been waiting for a way to adopt Foundry without rebuilding your existing infrastructure, the templates and guidance below will get you started.

Next Steps

Want to learn more first?
Private networking skill documentation - Deep dive into the networking concepts.
Microsoft Foundry documentation - Explore the full platform capabilities.

Ready to deploy?
Deploy the multi-region pattern with Template 19 - Bicep templates to help you get started quickly.

Azure AI Foundry Agent Unable to Use Credentials Stored in Key Vault Through Playwright MCP Tool

vaibhav01 — Fri, 05 Jun 2026 07:22:22 GMT

Hello everyone,

I am trying to understand how Azure AI Foundry agents interact with Azure Key Vault when using custom MCP tools, and I would appreciate any guidance from the community.

My Setup

- Created an Azure AI Foundry agent.
- Created an Azure Key Vault and configured all permissions according to Microsoft's official documentation.
- Stored the required website credentials (username and password) in the Key Vault.
- Deployed the official Playwright MCP Docker image.
- Exposed the MCP server using ngrok and verified that the endpoint is accessible.
- Connected the MCP endpoint as a Custom MCP Tool in Azure AI Foundry.
- Performed all configuration through the Azure portal, Foundry UI, and Playground only (no SDK or custom application code involved).

The Issue

The agent can access and use the Playwright MCP tool. However, when I ask it to log in to a website using credentials that are already stored in Key Vault, it does not populate the username and password fields.

My expectation was that the agent would be able to retrieve the secrets from Key Vault and provide them to the Playwright tool during execution.

Questions

Is there currently a supported mechanism for Azure AI Foundry agents to automatically retrieve Key Vault secrets and pass them to a Custom MCP tool?
Does the Playwright MCP Docker image have any built-in integration with Azure Key Vault?
When using only the Foundry UI (without SDK code), can a Foundry agent securely inject Key Vault secrets into MCP tool calls?
Are additional configurations required beyond Key Vault permissions and agent connections?
Has anyone successfully implemented a similar setup where a Foundry agent uses credentials stored in Key Vault to perform browser automation through Playwright MCP?

Any clarification on the expected architecture and whether this scenario is currently supported in Azure AI Foundry would be greatly appreciated.

Thank you.

Connecting Claude Clients with Azure API Management and Claude Models in Microsoft Foundry

MuraliKumanduri — Wed, 10 Jun 2026 20:02:12 GMT

Summary

You want to give your engineering org Claude Code without handing out Anthropic API keys, without per-developer billing sprawl, and without losing visibility into who is spending what. This post shows a battle-tested pattern:

Claude models run in Microsoft Foundry, billed through your Azure subscription — no Anthropic contract or keys required.
Azure API Management (APIM) sits in front as an LLM gateway: it authenticates each developer with Entra ID, enforces per-user rate limits and token quotas, and emits per-user usage metrics for chargeback.
Foundry lives in its own Azure subscription, and APIM authenticates to it with a Foundry API key — so there's no cross-subscription RBAC to untangle.
Developers hold only short-lived Entra tokens. The Foundry key never leaves APIM.

Everything below is grounded in the Claude Code LLM gateway requirements and Azure API Management's GenAI gateway policies. All command-line steps are shown in PowerShell for Windows developers.

The problem

Claude Code is a terminal- and IDE-native coding agent that talks to Claude over the Anthropic Messages API. Pointing it directly at Anthropic (or even directly at Foundry) creates three headaches for any organization beyond a handful of users:

Key sprawl and billing. Direct API keys mean either a shared key (no per-user attribution, a rotation nightmare) or many keys (procurement and offboarding overhead).
No throttle. Claude Code is token-heavy — it reads files, plans, and edits in long loops. One runaway session or an over-enthusiastic team can produce a surprising bill with nothing standing between the developer and the model.
No visibility. Finance wants to know cost per team. Security wants to know who is calling what. A raw key gives you neither.

The fix is a gateway that every request flows through — one that knows who the developer is (Entra ID), enforces how much they can use (APIM GenAI policies), and records what they used (Azure Monitor). Claude Code supports exactly this through its gateway configuration.

Architecture

Claude Code on a developer laptop authenticates to Azure API Management with an Entra ID bearer token; APIM validates the token, applies per-user token and request limits, swaps in the Foundry API key, and forwards the Anthropic Messages request to Claude in Microsoft Foundry in a separate subscription; per-user token usage is emitted to Application Insights.

The request path:

Developer laptop (Claude Code CLI / VS Code) | Authorization: Bearer <Entra access token for the APIM app> v Azure API Management (the LLM gateway) [Subscription A] | 1. validate-jwt confirm Entra identity, audience, app role | 2. extract oid per-user counter key | 3. llm-token-limit per-user tokens/min + monthly token quota | 4. rate-limit-by-key per-user requests/min | 5. strip Authorization; set api-key from secret named value | 6. llm-emit-token-metric per-user usage to App Insights v (forwards Anthropic Messages format; anthropic-* headers preserved) Microsoft Foundry https://{resource}.services.ai.azure.com/anthropic/v1/messages v [Subscription B] Claude deployments (Sonnet 4.6 / Haiku 4.5 / Opus 4.6)

The key idea: developer-facing auth and backend auth are independent. Developers always authenticate as themselves with Entra ID at the gateway. How the gateway authenticates to Foundry is a separate decision — and you have two good options.

Choosing how the gateway authenticates to Foundry

Both options below are independent of the developer-facing Entra ID auth, and both work whether Foundry is in the same subscription as APIM or a different one. The only hard constraint for managed identity is that both resources live in the same Entra tenant.

	Option A — Foundry API key	Option B — Managed identity
How APIM authenticates	api-key header from a secret named value	Entra token from APIM's managed identity, in the Authorization header
Setup	Read the key once, store it in APIM	Enable APIM's identity, assign Cognitive Services User on Foundry
Same subscription	Works	Works
Cross-subscription	Works — no RBAC crosses the boundary	Works — role assignment spans subscriptions in the same tenant
Cross-tenant	Works	Not supported — use a key
Shared secret to rotate	Yes	None
Best for	Fastest start; cross-tenant; key-only environments	Production; eliminates the shared secret

This guide builds the key-based path end to end, then shows the managed-identity swap inline at each step (Parts 3 and 4). Pick one — you don't need both.

What this design achieves

Goal	How it's met
Developers use Claude Code with no Anthropic billing or keys	Claude runs in Microsoft Foundry, billed through your Azure subscription
Foundry can live in a different subscription	APIM reaches Foundry by URL + API key only — no cross-subscription RBAC
Every developer authenticates as themselves	Entra ID tokens validated at the APIM gateway
Per-developer rate limits and quotas	rate-limit-by-key + llm-token-limit keyed on the Entra oid claim
Per-developer usage and cost tracking	llm-emit-token-metric → Application Insights / Log Analytics
No Foundry keys on developer laptops	The Foundry key lives only inside APIM; developers hold short-lived Entra tokens

Prerequisites

Two Azure subscriptions, both pay-as-you-go. Subscription A holds APIM; Subscription B holds Foundry. (Foundry Claude does not run on free, trial, sponsored, or CSP subscriptions.)
A Microsoft Foundry resource (Subscription B) in a region where Claude is available — currently East US 2 or Sweden Central — with Claude deployments created and at least one API key under Keys and Endpoint.
An API Management instance (Subscription A). Developer SKU is fine for a pilot; Standard v2 or Premium for production and VNet integration.
Permission to read the Foundry key in Subscription B, contributor on the APIM instance, and the ability to register Entra apps.
Developers on Windows with PowerShell (5.1 built-in, or 7), the Azure CLI (winget install Microsoft.AzureCLI), and the Claude Code CLI installed.
Pilot costs: APIM Developer SKU ~$50/month + Claude usage based on token consumption.

Option A (key): no cross-subscription role assignment — the only cross-subscription action is reading the Foundry key once (Part 3), which you can also do from the Foundry portal. Option B (managed identity): one cross-subscription role assignment (Cognitive Services User), supported as long as APIM and Foundry share an Entra tenant.

Part 1 — Deploy Claude in Foundry (Subscription B)

In the Foundry portal, open Model catalog, search Claude, and deploy the models Claude Code uses. Name each deployment to match its model ID so the gateway can pass the model field through unchanged:
Role Deployment name (recommended)
Primary (general coding) claude-sonnet-4-6
Fast (file reads, small edits, background tasks) claude-haiku-4-5
Extended thinking (optional) claude-opus-4-6
Pin versions — select a specific version, not auto-update to latest. Without pinning, a new model release can break every developer at once.
On the resource's Keys and Endpoint blade, copy the endpoint and one of the two API keys. The Anthropic endpoint base is:

Role	Deployment name (recommended)
Primary (general coding)	claude-sonnet-4-6
Fast (file reads, small edits, background tasks)	claude-haiku-4-5
Extended thinking (optional)	claude-opus-4-6

https://{resource}.services.ai.azure.com/anthropic

Critical: Foundry's Claude endpoint is the Anthropic surface (/anthropic/v1/messages), not the OpenAI surface (/openai/deployments/.../chat/completions?api-version=...). When you build the APIM API, do not apply the OpenAI policy template, do not add an api-version query parameter, and do not rewrite to an /openai/... path. Any of these produces the "not supported" or "resource not found" errors people commonly hit.

✅ Checkpoint: You now have Claude deployed in Foundry. Verify your deployment before continuing to Part 2.

Part 2 — Entra ID app registration (developer-facing)

This registration lives in Subscription A's tenant. It defines the audience developers' tokens are issued for, and what APIM validates. It's unaffected by where Foundry lives.

App registrations → New registration → name it e.g. Claude Code Gateway.
Expose an API → set the Application ID URI, e.g. api://claude-code-gateway. Add a scope access_as_user (admin + user consent).
(Optional, for tiering) App roles → add roles such as Claude.Standard and Claude.Premium. Assign developers or groups under Enterprise applications → this app → Users and groups.
Note the Application (client) ID, the Application ID URI, and your Tenant ID.

Developers request tokens for this app's audience; APIM validates aud = api://claude-code-gateway.

Part 3 — Provision the APIM API and Foundry backend (Subscription A)

3.1 Option A — Store the Foundry API key in APIM

First read the key from Foundry in Subscription B (use --subscription so you don't have to switch your active context):

# Read a Foundry key from Subscription B $FOUNDRY_KEY = az cognitiveservices account keys list ` --name <foundry-resource> ` --resource-group <foundry-rg> ` --subscription <SUBSCRIPTION_B_ID> ` --query key1 -o tsv

Then store it as a secret named value in APIM (Subscription A). The policy references it as {{foundry-api-key}}:

# Create a secret named value in APIM holding the Foundry key az apim nv create -g <apim-rg> --service-name <apim-name> ` --named-value-id foundry-api-key ` --display-name foundry-api-key ` --value "$FOUNDRY_KEY" ` --secret true

Hardening: instead of the raw key in APIM, put it in Key Vault and create a Key Vault-backed named value, so rotation lives in one place. APIM needs a managed identity with Get/List secret access on that vault — but the vault is in Subscription A alongside APIM, so this is still not a cross-subscription role assignment.

3.2 Option B — Give APIM a managed identity instead

If you'd rather not manage a shared key, skip 3.1 and give APIM an identity that Foundry trusts. This works in the same subscription and across subscriptions alike, as long as both resources are in the same Entra tenant.

# Enable a system-assigned managed identity on APIM (Subscription A) az apim update -g <apim-rg> --name <apim-name> ` --set identity.type=SystemAssigned # Get the identity's principal (object) ID $APIM_MI = az apim show -g <apim-rg> --name <apim-name> ` --query identity.principalId -o tsv # Get the Foundry resource ID (Subscription B) $FOUNDRY_ID = az cognitiveservices account show ` --name <foundry-resource> --resource-group <foundry-rg> ` --subscription <SUBSCRIPTION_B_ID> ` --query id -o tsv # Grant Cognitive Services User on the Foundry resource (works cross-subscription) az role assignment create ` --assignee-object-id $APIM_MI ` --assignee-principal-type ServicePrincipal ` --role "Cognitive Services User" ` --scope $FOUNDRY_ID

The Cognitive Services User role (a97b65f3-24c7-4388-baec-2e87135dc908) grants data-plane access to call the model without key-management rights. Role assignments can take a few minutes to propagate. A user-assigned identity works too — assign it to APIM and reference its client ID in the policy (Part 4, Option B). On this path there is no foundry-api-key named value to create or rotate.

3.3 Create the backend and API

# Named backend pointing at the Foundry Anthropic endpoint (Subscription B URL) az apim backend create -g <apim-rg> --service-name <apim-name> ` --backend-id foundry-claude ` --url "https://<foundry-resource>.services.ai.azure.com/anthropic" ` --protocol http # API with NO path suffix so callers hit /v1/messages at the gateway root az apim api create -g <apim-rg> --service-name <apim-name> ` --api-id claude-anthropic --display-name "Claude (Foundry)" ` --path="" --protocols https ` --service-url "https://<foundry-resource>.services.ai.azure.com/anthropic"

PowerShell + empty strings: write --path="" (joined with =), not --path "" as two tokens. PowerShell strips a bare "" before the az wrapper sees it, so the CLI reports argument --path: expected one argument. The = form keeps it a single token (--path=) that az reads as an empty string. The same trick applies to any empty-string value you pass to az from PowerShell.

Add the operations Claude Code calls (a wildcard covers them all):

POST /v1/messages
POST /v1/messages/count_tokens
GET /v1/models (only if you enable gateway model discovery — see Part 5.3)

az apim can't apply XML policies. Apply the Part 4 policy via the portal (APIs → Claude (Foundry) → Inbound processing → policy editor) or via Bicep/ARM.

Part 4 — The APIM policy (auth + rate limiting + metering)

Apply this at the API level. Replace the tenant ID and audience. The policy below is the key-based (Option A) version — its step 6 removes the developer's Authorization header and sets the api-key header from the secret named value. For managed identity (Option B), swap step 6 as shown immediately after the policy; every other step is identical.

<policies> <inbound> <base />  <set-header name="Authorization" exists-action="skip"> <value>@("Bearer " + context.Request.Headers.GetValueOrDefault("x-api-key",""))</value> </set-header>  <validate-jwt header-name="Authorization" failed-validation-httpcode="401" failed-validation-error-message="Unauthorized: invalid or missing Entra token."> <openid-config url="https://login.microsoftonline.com/{{tenant-id}}/v2.0/.well-known/openid-configuration" /> <audiences> <audience>{{gateway-audience}}</audience> </audiences> <issuers> <issuer>https://login.microsoftonline.com/{{tenant-id}}/v2.0</issuer>  <issuer>https://sts.windows.net/{{tenant-id}}/</issuer> </issuers> <required-claims> <claim name="roles" match="any"> <value>Claude.Standard</value> <value>Claude.Premium</value> </claim> </required-claims> </validate-jwt>  <set-variable name="callerId" value="@{ var jwt = context.Request.Headers .GetValueOrDefault("Authorization","").Split(' ').Last().AsJwt(); return jwt.Claims.GetValueOrDefault("oid", "unknown"); }" />  <set-variable name="tier" value="@{ var jwt = context.Request.Headers .GetValueOrDefault("Authorization","").Split(' ').Last().AsJwt(); return jwt.Claims.GetValueOrDefault("roles","").Contains("Claude.Premium") ? "premium" : "standard"; }" /> <set-variable name="modelName" value="@{ var body = context.Request.Body.As<JObject>(preserveContent: true); return body?["model"]?.ToString() ?? "unknown"; }" />  <choose> <when condition="@(((string)context.Variables["tier"]) == "premium")"> <llm-token-limit counter-key="@((string)context.Variables["callerId"])" tokens-per-minute="200000" estimate-prompt-tokens="true" remaining-tokens-header-name="x-tokens-remaining" token-quota="20000000" token-quota-period="Monthly" /> <rate-limit-by-key calls="300" renewal-period="60" counter-key="@((string)context.Variables["callerId"])" retry-after-header-name="retry-after" remaining-calls-header-name="x-ratelimit-remaining" /> </when> <otherwise> <llm-token-limit counter-key="@((string)context.Variables["callerId"])" tokens-per-minute="50000" estimate-prompt-tokens="true" remaining-tokens-header-name="x-tokens-remaining" token-quota="5000000" token-quota-period="Monthly" /> <rate-limit-by-key calls="100" renewal-period="60" counter-key="@((string)context.Variables["callerId"])" retry-after-header-name="retry-after" remaining-calls-header-name="x-ratelimit-remaining" /> </otherwise> </choose>  <llm-emit-token-metric namespace="claudecode"> <dimension name="UserId" value="@((string)context.Variables["callerId"])" /> <dimension name="Tier" value="@((string)context.Variables["tier"])" /> <dimension name="Model" value="@((string)context.Variables["modelName"])" /> </llm-emit-token-metric>   <set-header name="Authorization" exists-action="delete" /> <set-header name="x-api-key" exists-action="override"> <value>{{foundry-api-key}}</value> </set-header> <set-backend-service backend-id="foundry-claude" /> </inbound> <backend> <base /> </backend> <outbound> <base /> </outbound> <on-error> <base /> </on-error> </policies>

Option B — authenticate to Foundry with managed identity

If you chose the managed-identity path (3.2), replace step 6 above with the block below. Instead of injecting an api-key, APIM acquires an Entra token for its own identity and forwards it as the Authorization bearer token. Token validation, rate limits, and metering are unchanged.

<authentication-managed-identity resource="https://cognitiveservices.azure.com" output-token-variable-name="msi-token" /> <set-header name="Authorization" exists-action="override"> <value>@("Bearer " + (string)context.Variables["msi-token"])</value> </set-header> <set-backend-service backend-id="foundry-claude" />

The token audience for Azure AI Services / Foundry is https://cognitiveservices.azure.com. For a user-assigned identity, add client-id="<uami-client-id>" to the authentication-managed-identity element. There's no api-key named value and no secret to rotate on this path — which is exactly why it's the preferred production posture.

Policy notes

Stripping the developer's Authorization header before forwarding (step 6) matters: that Entra token is for APIM only. Foundry must receive only the api-key header.
{{tenant-id}}, {{gateway-audience}}, and {{foundry-api-key}} are APIM named values. Mark foundry-api-key as secret; the first two can be plain named values.
llm-token-limit and llm-emit-token-metric are APIM's GenAI gateway policies — they understand the Anthropic/OpenAI message formats and parse token usage, so you meter tokens, not just requests. That's the right cost lever for token-heavy Claude Code.
These counters are per-region per-gateway. With multi-region APIM, limits are enforced per region.

Part 5 — Configure Claude Code, Claude Desktop and Cowork on developer machines

Developers point Claude Code, Claude Desktop and Cowork at APIM (Anthropic Messages gateway mode) and authenticate with their own Entra token. The backend-auth swap is invisible to clients.

5.1 Entra token helper (per-developer, auto-refreshing)

Create %USERPROFILE%\.claude\get-claude-gateway-token.ps1:

# Returns a short-lived Entra access token for the APIM gateway audience. # Supports CLAUDE_HELPER_CONTEXT for Claude Desktop Cowork/Code silent refresh. $context = $env:CLAUDE_HELPER_CONTEXT # For non-interactive contexts, try silent token acquisition only. # If it fails, exit non-zero so Cowork prompts the user instead of blocking. if ($context -and $context -ne 'interactive' -and $context -ne 'setup-test') { try { $token = az account get-access-token ` --resource "api://claude-code-gateway" ` --query accessToken -o tsv 2>$null if ($LASTEXITCODE -ne 0 -or -not $token) { Write-Error "Silent token refresh failed (context: $context)" exit 1 } Write-Output $token exit 0 } catch { Write-Error "Silent token refresh failed: $_" exit 1 } } # Interactive or setup-test context: allow az CLI to prompt if needed. $token = az account get-access-token ` --resource "api://claude-code-gateway" ` --query accessToken -o tsv if ($LASTEXITCODE -ne 0 -or -not $token) { Write-Error "Token acquisition failed. Run 'az login' first." exit 1 } Write-Output $token exit 0

PowerShell scripts need no chmod. If execution policy blocks the helper, allow local scripts for your user once:

Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned

5.2 Claude Code settings (%USERPROFILE%\.claude\settings.json)

In enabling configuration of below environment variables in settings.json under .claude folder allows its usage for all Claude Code sessions (VS Code, terminal CLI, JetBrains, etc.)

{ "env": { "ANTHROPIC_BASE_URL": "https://<apim-name>azure-api.net", "ANTHROPIC_MODEL": "claude-opus-4-8", "ANTHROPIC_DEFAULT_OPUS_MODEL": "claude-opus-4-8", "CLAUDE_CODE_API_KEY_HELPER_TTL_MS": "600000" }, "apiKeyHelper": "powershell -NoProfile -ExecutionPolicy Bypass -File C:\\Users\\<you>\\.claude\\get-claude-gateway-token.ps1" }

In JSON, backslashes must be doubled — hence C:\\Users\\.... Use pwsh in place of powershell if you run PowerShell 7.

apiKeyHelper output is sent as the Authorization (and X-Api-Key) header, validated by APIM's validate-jwt. The developer never holds the Foundry key.
CLAUDE_CODE_API_KEY_HELPER_TTL_MS=3600000 refreshes the token hourly (Entra access tokens last ~60–90 minutes).
Pinning the three ANTHROPIC_DEFAULT_*_MODEL IDs ensures Claude Code sends model names that match your Foundry deployment names, so the gateway passes model through untouched.
Other Anthropic models like Sonnet and Haiku can be provided. Default model to be used is provided with ANTHROPIC_MODEL.

Then developers run claude from their project folder.

5.3 Optional — model discovery

To list gateway models in the /model picker, expose GET /v1/models on the API and set CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 (Claude Code v2.1.129+). Only IDs starting with claude or anthropic appear.

5.4 VS Code extension

Settings.json in .claude folder will control both VS Code Extension and Claude Code CLI.

🚀 Ready to test? Jump to Part 7 to validate your setup, or run claude from your project folder to try it live.

5.5 Claude Desktop and Cowork

Install the Claude Desktop and Cowork application on your desktop. Enable the Developer mode in Help -> Troubleshooting -> Enable Developer Mode to reveal the Developer menu. Then go to Developer -> Configure third-party inference to open the configuration.

In the connection section, Inference Provider will be Gateway and will point to APIM endpoint. You can provide Model list that has been exposed. You will need to use credential helper script by point to PowerShell script that step 5.1 above that will enable authentication using Entra ID.

Administrators can follow Export and deploy via MDM to distribute this configuration once they have tried it on their machine.

Part 6 — Rate-limiting and usage-tracking design

Per-developer keying. Everything is keyed on the Entra oid claim — stable and unique per user, unlike email or upn which can change. For service accounts or CI, key on appid instead.

Two enforcement layers:

llm-token-limit — tokens/min plus a monthly token quota. The real cost control.
rate-limit-by-key — requests/min. Guards against runaway loops.

Tiering is driven by Entra app roles (Claude.Standard / Claude.Premium) read from the JWT — no separate APIM subscription management needed.

Usage tracking flows from llm-emit-token-metric into Application Insights with UserId, Tier, and Model dimensions. Example Log Analytics query for per-user monthly token spend:

customMetrics | where name == "Total Tokens" and customDimensions.namespace == "claudecode" | extend UserId = tostring(customDimensions.UserId), Model = tostring(customDimensions.Model) | summarize Tokens = sum(valueSum) by UserId, Model, bin(timestamp, 1d) | order by Tokens desc

Foundry doesn't return Anthropic's standard rate-limit headers, so manage and observe limits through APIM (the headers above) and Azure Monitor rather than relying on upstream headers.

Part 7 — Test and validate

# 1. Get a token as a developer $TOKEN = az account get-access-token --resource "api://claude-code-gateway" --query accessToken -o tsv # 2. Call the gateway directly in Anthropic Messages format $body = @{ model = "claude-sonnet-4-6" max_tokens = 64 messages = @(@{ role = "user"; content = "Say hello in one word." }) } | ConvertTo-Json Invoke-RestMethod -Method Post ` -Uri "https://<apim-name>.azure-api.net/v1/messages" ` -Headers @{ "Authorization" = "Bearer $TOKEN" "anthropic-version" = "2023-06-01" "content-type" = "application/json" } ` -Body $body

Invoke-RestMethod returns the parsed body but hides response headers. To see x-tokens-remaining / x-ratelimit-remaining, use Invoke-WebRequest with -ResponseHeadersVariable resp (then read $resp), or call curl.exe -i (the real curl, not PowerShell's curl alias).

Validation checklist

No token / expired token → 401 from validate-jwt (confirm before trusting rate limits).
Valid token → 200 with a Claude completion; response carries x-tokens-remaining / x-ratelimit-remaining.
401 from Foundry on a valid developer token → the api-key named value is wrong or not injected (see Troubleshooting).
Exceed the limit → 429 with retry-after.
App Insights → customMetrics shows token counts dimensioned by UserId.
Then run claude end to end from a project folder.

Part 8 — Operations and hardening

Key rotation (Option A). Foundry gives you two keys. Rotate by updating the foundry-api-key named value to key2, then regenerating key1 — zero downtime. A Key Vault-backed named value makes this a one-place change.
Prefer managed identity in production (Option B). If you started on the key path, switch to managed identity (Parts 3.2 and 4, Option B) to remove the shared secret entirely. Because the Cognitive Services User role assignment works across subscriptions in the same tenant, the cross-subscription topology doesn't block this upgrade — and developers see no change, since their side of the contract is always authenticate to the gateway as yourself.
Private networking. Put APIM in internal VNet mode and reach Foundry over a Private Endpoint; disable Foundry public network access so the gateway is the only path in. Cross-subscription private endpoints are supported.
Resilience. Deploy Claude in two regions and use APIM's load-balanced backend pool with retry on 429 and 5xx.
Cost guardrails. Pair per-user llm-token-limit quotas with an Azure Budget and alert on the Foundry resource in Subscription B.

Troubleshooting

Symptom	Cause / fix
404 resource not found from Foundry	Backend URL or path wrong, or an OpenAI-style rewrite applied. Backend must end in /anthropic; callers hit /v1/messages. Remove any /openai/... rewrite and api-version query param.
401 from Foundry (developer token is valid) — Option A	The api-key header is missing/wrong, or the foundry-api-key named value wasn't saved as expected. Confirm the named value, and that the policy deletes the developer Authorization header and sets api-key.
401 / 403 from Foundry — Option B (managed identity)	The role assignment is missing or hasn't propagated yet, or the token audience is wrong. Confirm APIM's identity has Cognitive Services User on the Foundry resource, wait a few minutes, and ensure the policy requests resource="https://cognitiveservices.azure.com". For a user-assigned identity, confirm the client-id is set.
Managed identity works same-sub but not cross-sub	The two resources are in different Entra tenants. Cross-tenant managed identity isn't supported — use the API key (Option A) instead.
401 at the gateway even with a token	aud or issuer mismatch. Confirm the token's aud = api://claude-code-gateway and you used the v2.0 OIDC config and issuer.
403 from Foundry	The key belongs to a different Foundry resource, or the resource disabled key auth. Re-copy a key from Keys and Endpoint, or re-enable local/key auth.
Reduced Claude Code functionality	Gateway stripped anthropic-beta / anthropic-version. Ensure both headers pass through.
Model not available	Claude Code's model ID doesn't match the Foundry deployment name. Align names, or rewrite the body model field in policy.
ChainedTokenCredential authentication failed (client side)	Developer not logged in. Run az login so the helper has a usable Azure credential.

Wrapping up

With about an afternoon of setup you get a gateway that every Claude Code request flows through: Entra ID proves who the developer is, APIM GenAI policies cap how much each person can spend, and Application Insights tells you exactly where the tokens went. For the APIM → Foundry hop you pick what fits: a Foundry API key held only inside APIM (fastest start, works cross-tenant) or a managed identity with no shared secret at all (the production posture). Either way Claude can live in its own subscription, and developers hold nothing more sensitive than a short-lived Entra token.

When you're ready to tighten the screws, the upgrade path is clean: if you started on the key, move it into Key Vault, then graduate to managed identity to eliminate the secret entirely, and put the whole path on a private network.

'None of those steps disrupt developers, because their side of the contract — authenticate to the gateway as yourself — never changes.

Start your pilot today: Deploy a Developer-tier APIM instance, connect it to Foundry, and have your first developer running Claude Code through the gateway by end of day. The Prerequisites section has everything you need to begin.'

All command-line steps target Windows with PowerShell 5.1 or 7. Model IDs and Foundry regions reflect availability at time of writing; check the Foundry model catalog for current options.

Share Your Experience

Share your setup and implementation approach based on this pattern in the comments.

Next Steps

Get started now: - Deploy Claude models in Microsoft Foundry — browse the model catalog and create your first deployment - Create an API Management instance — spin up a Developer SKU for your pilot

Go deeper: - Claude Code LLM gateway requirements — full specification for gateway compatibility - APIM GenAI gateway policies reference — all available token and rate limiting options

Get help: - Questions? Post in the Azure AI Community with tag #ClaudeCode - Found an issue with this guide? Open a GitHub issue

Introducing the New Browser Automation Tool with Toolboxes in Foundry

NandiniMuralidharan — Wed, 10 Jun 2026 20:05:18 GMT

Your AI agent just analyzed 500 invoices—but it can't click the 'Submit' button to process them. AI agents can reason through complex problems, but they hit a wall when workflows require interacting with web interfaces The Browser Automation Tool bridges that gap, giving your agents the ability to interact with any website the same way a human would. Here's what's new in our latest release with Toolboxes support in hosted agents.

Browser Automation Tool is available as an MCP (Model Context Protocol) tool powered by Playwright workspaces as its headless browser infrastructure layer. This gives teams a faster path from idea to automation, with better visibility into what is happening in real time and more control when workflows hit complex edge cases. If you’re working on web automation, this release is for you—it lets you use any CDP (Chrome Dev Tools Protocol) supported browser framework or reasoning SDK to automate your workflows with ease.

Why this release matters

Today’s AI agents are powerful at reasoning but fall short when workflows move beyond APIs.

Enterprise reality is:

Critical systems live behind web interfaces, not APIs
Workflows require navigation, decision-making, and multi-step interaction
Automation breaks at the last mile of execution
Private and internal websites are difficult to validate safely at scale.

The revised Browser Automation Tool with Toolboxes in Foundry is designed to close those gaps with a practical, operator-friendly workflow.

What is new

MCP-native Browser Automation Tool in Toolboxes: Browser Automation Tool (Public Preview) is now available as an MCP tool in Toolboxes, working seamlessly with hosted agents in Foundry Agent Service. This makes it easier to adopt browser automation in your existing agentic workflows. Teams get standardized browser tasks with built-in Microsoft Entra ID authentication (formerly Azure AD).
Playwright workspaces as the infrastructure layer: The underlying execution model is powered by Playwright workspaces (generally available) which brings robust browser automation foundations and reliable runtime behavior for modern web applications.
Live View for issue detection: Live View (Public preview) helps customers detect issues as they happen. You can observe browser actions and quickly identify - selector drift (when UI elements change and automation can no longer find them), timing issues, navigation failures, unexpected UI states. This improves debugging velocity.
Take Control for edge cases: When automation reaches a non-deterministic path (unpredictable user flows like CAPTCHAs or dynamic content), customers can use Take Control (Public preview) to intervene and steer execution through edge-case conditions. This blends automation scale with human judgment when it is needed most.
Support for private website browsing [Private Preview]: Customers can browse and automate private websites in hosted environments, enabling more realistic enterprise workflows such as internal portals, authenticated research paths, and secured form flows. Playwright Workspaces Private Website Feature - Private Preview Enrollment
Built-in Observability & Optimizations: Logs and evaluations are integrated within observability in Foundry Control Plane and full traceability is available for debugging and reliability
Choose Orchestration layer: Customers can choose from wide variety of open-source reasoning layer for their use case while writing the hosted agents.

Get Started Here are the samples: Sample 1 (BYO) and Sample 2 (Microsoft Agent Framework)

Sample Scenarios could be Form filling (automate repetitive, high-volume form interactions while maintaining traceability and visibility during execution) or Web research (Scale browser-based research flows across multiple sources, while retaining the option to step in manually when pages or data structures vary)

How teams can use Browser Automation Tool in practice :

Define the desired browser workflow in your Hosted Agent setup.
Run Browser Automation Tool execution with Live View enabled.
Detect and resolve issues quickly as they appear.
Use Take Control on complex or unexpected branches.
Continue automation and capture outcomes for iteration.

What this unlocks

With hosted agents with Browser Automation Tool , you can now build:

✅ End-to-end agents that complete real workflows with reliable automation
✅ Enterprise-grade systems with identity, compliance, and observability
✅ Human-in-the-loop experiences when precision matters

This unlocks a new class of applications across sectors (finance & operations, back office automation, testing, web-based data workflows) where teams want to automate with confidence and not just speed.

Learn more

This update is part of a broader investment in practical, agent-driven browser automation. We are focused on making automation more adaptable to real-world website behavior. If your workflows depend on forms, web navigation, and research at scale, now is a great time to evaluate the revised Browser Automation Tool experience in hosted agents in Foundry Agent Service.

Azure Speech at Build 2026: Powering Voice Agents with Real-Time and Life-like Experiences

DONGLI — Fri, 05 Jun 2026 03:14:41 GMT

Voice is rapidly becoming the default interface for AI. At Build 2026, Azure Speech is making it dramatically easier to ship production-grade voice experiences that feel real, responsive, and global. From agentic, low-latency Voice Live experiences built natively into Foundry Agent Service, to a new generation of LLM powered Speech to text and Text to speech voices - every layer of the Speech stack is getting faster, more expressive, and more customizable. With a new unified speech experience in the latest Foundry update that brings playgrounds and self-service fine-tuning to every Speech capability, developers now have a clear path from prototype to production for real-time, multilingual, and truly agentic voice applications - faster and more scalable than ever before.

Build Real-Time Voice Agent with Voice Live and Foundry Agent Service

As developers move beyond traditional chatbots, they are building a new class of real-time agents that can listen, reason, take actions, and respond naturally in live conversations. From customer support agents and virtual assistants to healthcare intake, retail concierge, field operations, in-car assistants and multilingual employee support, voice agents are becoming a key interface for how people interact with AI.

At Build 2026, we’re announcing major updates that support building enterprise-ready, voice agents at scale.

Voice Live for Foundry Prompt Agents is now generally available. This is a strong fit for developers that want enterprise-ready capabilities with minimal operational overhead. Voice Live brings the essential pieces of voice interaction into a single API, from Speech to Text and Text to Speech to turn detection, interruption handling, avatars, and other conversational capabilities. Customers can combine real-time speech-to-speech interaction with managed agent orchestration, knowledge, memory, enterprise governance, observability, and scalable deployment - all within a single developer workflow (learn more).

Hosted agents with Voice Live is available in Public Preview. Some customers need full control over their agent’s runtime, orchestration framework, and execution model. For those scenarios, Microsoft Foundry supports hosted agents with Voice Live (public preview), so developers can build with the frameworks they prefer and deploy on managed infrastructure. Whether using Microsoft Agent Framework, LangChain, or a custom orchestration stack, they can host those agents on Foundry Agent Service and connect them directly to Voice Live. Both Response API and Invocations Protocol are supported (learn more).

Hosted agents also adds support for real-time voice interfaces such as WebSocket and WebRTC, which allows developers to deploy real-time voice workloads as managed containers while continuing to use frameworks such as Microsoft Voice Live, Pipecat, and LiveKit etc. These interfaces are bidirectional and full duplex, which makes them well suited to both cascaded pipelines and native speech-to-speech models (learn more).

In addition, we are advancing the Voice Live API with the following enhancements that developers can integrate into their agents:

New all-in-one speech-to-speech models to help developers build highly responsive voice experiences. These include GPT-Realtime 1.5 and the new Azure-Realtime model (public preview), which delivers more natural voice output across multiple languages and accents, including en-US, zh-CN, es-ES, fr-FR, de-DE, hi-IN and more. This is a strong option for customers prioritizing speed, simplicity, and natural conversational quality in multilingual voice experience (learn more).

Integration with MAI Transcribe-1 (public preview) for more accurate multilingual speech input, Neural HD V3 voices for more conversational and realistic voice experience, and four new full-body standard avatars (public preview) to make the voice agent more engaging. More details about the models can be found in next sections.

Full integration with speech customization/fine-tuning capabilities in Foundry, including custom speech for better recognition accuracy, custom voice for branded voice experience and custom avatar for one-of-a-kind visual representations of the agent. More details about the features can be found in next sections.

WebRTC (Web Real-Time Communication) connection as public preview, enabling low‑latency, real‑time voice interactions directly from web and mobile clients (learn more).

The solution template call center voice agent accelerator now expands the telephony capabilities by integrating more third-party providers such as Twilio and Infobip, giving customers greater flexibility to connect with their preferred telephony infrastructures.

A new Voice Live Evaluation Harness gives developers a one-command-deployable pipeline to score their voice agents on 13 Foundry evaluators - intent resolution, task adherence, task completion, response completeness, and more - using pre-recorded multi-turn audio in Push-toTalk (PTT), Voice-Activity-Detection (VAD), or Foundry Agent mode (learn more).

Next Generation Speech Models in Azure Speech

We're advancing Azure Speech to text with a new generation of LLM-powered recognition models that raise accuracy, expand language coverage, and give developers more control across both batch and real-time scenarios.

LLM Speech API is now generally available in Azure speech for LLM-powered transcription and translation of audio files (learn more): 25 languages / 90+ locales with locale hint, renewed speech-LLM model with better context and entity recognition and reduced hallucination, up to 5-hour long-form audio, prompt-tuning with 20,000-character input and 2,000 phrase-list entries, and broader regional availability. This model achieves industry-leading accuracy, ranking No.1 across all models on the Open ASR Leaderboard. We also upgraded the MAI-Transcribe model from 1.0 to 1.5 with the phrase list support and verbatim mode.

We're upgrading the TTS and TTS avatars in Azure Speech. With flexible instruction controls brought into the HD voices, upgraded recipes in personal voice and new TTS avatar capabilities, customers can build voice agents that feel real, human-like and personalized.

Neural HD V3 (En-US Ava-Preview/Andrew-Preview/Serena-Preview) is now in public preview, delivering best-in-class quality with prompt-level instruction control. We also upgraded the MAI-Voice from 1.0 to 2.0 in public preview with 10+ languages support. Personal Voice is upgraded the OmniHD and MAI-Voice-2, optimized for conversational AI, creative applications, and long-form narration with emotion and style control.

 Avatar updates include Photo Avatar and Custom Photo Avatar are in generally available (demo). Also, four new full-body standard avatars are now in public preview in the Foundry Voice Live and Text-to-Speech Avatar playground. Kobie Burrell, Director of Development of Optimal Blue, sharing: "The photo avatar and speech service made it incredibly easy for our team bring our Virtual Economist to life. The photo avatars in particular helped us create something that feels human and intuitive - giving our users the experience of engaging with an economist, not just an interface to a set of powerful models."

Azure Speech & Customization experience in Microsoft Foundry

Speech Playgrounds are now available for every Speech capability in Microsoft Foundry

Every Azure Speech capability now has a hands-on playground in one place, so developers can try different models, compare them and prototype in the different speech capabilities such as speech-to-text, text-to-speech, avatars, Voice Live, and speech translation - no code required - and go from experimentation to production without ever leaving Foundry. try it here

Azure Speech is fine-tunable through the new Foundry experience

For the first time in the Microsoft Foundry, custom speech, voice and avatar allow the developers to tailor models to their own domain vocabulary, brand identity, and visual presence so their agents sound, understand, and look distinctly their own.

Custom Speech: adapt speech-to-text to domain vocabulary and acoustic conditions.

Custom Voice: train brand voices with Professional Voice, or zero-shot cloning with Personal voice, including Omni and MAI-Voice-1 and 2 models.

Custom Avatar: create high quality avatars using video, or a quick avatar with a single image. See the self-serving photo avatar creation in the foundry experience as follows:

Get started today  

The easiest way to explore is through the Microsoft Foundry portal and the Foundry Tools catalog. From there you can follow the documentation and Microsoft Learn courses, and start building with Azure Speech referring to Azure Speech Documentation

Foundry Labs @ Build 2026

Saumil-Shrivastava — Wed, 03 Jun 2026 15:30:00 GMT

We launched Foundry Labs last year as the home for cutting-edge AI experiments from across Microsoft. It is meant to be a place where developers could try, fork, and build with our earliest prototypes. The response from the community has been tremendous, and we're doubling down. Today at Build 2026, we're rolling out a refreshed experience for Foundry Labs. Here's what's new.

A refreshed home for Microsoft’s AI experiments

We’ve rebuilt labs.ai.azure.com as a central hub for every AI experiment coming out of Microsoft. The catalog is now searchable and easier to scan whether you’re hunting for a specific model or just curious about what’s shipping next.

One theme we heard loud and clear from developers: more connection to the community. So, we added two new sections that put that front and center:

Stories — a dedicated space for the stories from the teams who are building real systems with experiments from Foundry Labs. The first stories are live, and we're always looking for more — if you're building with Foundry Labs, we'd love to hear from you.
Communities — a page for the places where our team and our developer community gather. The current calendar includes Microsoft events as well as conferences we'll be attending and sponsoring. If you're going to be at any of them, come find us!

Six areas where AI is having significant real-world impact

Over the past year, we’ve watched the same pattern repeat: AI breaking out of demos and into the hands of builders, in six specific domains. We’ve reorganized Foundry Labs around those six areas, so you can navigate by the problem you’re solving for:

Biomedical Sciences — accelerating pathology, drug discovery, and clinical research with multimodal AI (ex. GigaTIME, RosettaFold3)
Chemistry & Materials — generating and screening new compounds before they ever exist in a lab (ex. MatterGen, Skala, RetroChimera)
Code & Software Engineering — debugging, refactoring, and reasoning about codebases at agentic scale (ex. BugPilot, Debug-gym)
Creative & Generative Media — text-to-image, voice, and 3D generation with enterprise-grade control (ex. MAI-Image-2, MAI-Voice-1, Trellis)
Geospatial & Earth Science — turning petabytes of satellite and overhead imagery into usable signal (ex. EO/OS Object Detection). This was developed in collaboration with the Microsoft Planetary Computer team.
Robotics & Physical AI — translating language into action for embodied systems (ex. Rho-Alpha)

We’re also rolling out these categories into Microsoft Foundry Model Catalog, making it easier for you to find and choose models specific to your use case.

New innovations from the Lab: MAI models in Foundry across text, image, voice, and speech

We're continuing the momentum of frontier-quality, cost-efficient models from Microsoft AI (MAI) with the availability of new releases across 4 modalities:

Text/Reasoning: MAI-Thinking-1 is our first large language model, designed to deliver strong reasoning, math, and general intelligence at a fraction of the cost of frontier-scale models.
Image: MAI-Image-2.5 is an updated image generation model that adds image-to-image editing and a suite of "control with preservation" capabilities, once again debuting at No. 3 on Arena.ai for image generation model families. We also have MAI-Image-2.5 Flash for a faster and more efficient option.
Voice: MAI-Voice-2 is an updated multilingual text-to-speech model that brings voice cloning and voice prompting to more than 10 languages. We also have MAI-Voice-2 Flash for a faster and more efficient option coming soon.
Speech: MAI-Transcribe-1.5 is an updated speech-to-text model that adds content biasing and improved accuracy.

These are the same models already powering experiences across Copilot, Bing, OneDrive, PowerPoint, and Azure Speech, and now they're available in Foundry for developers to build with.

Learn more about the new MAI models in Foundry here.

Catch us at Build 2026

If you are at Build 2026, two sessions are especially relevant if you want to dig deeper into AI experiments from Microsoft in Foundry Labs:

BRK230 — Build smarter AI systems in Foundry as models and costs evolve. How to quickly choose, integrate, and validate AI models inside Microsoft Foundry: navigating thousands of model options, benchmarking performance, and streamlining the workflow with deep IDE support.
LTG419 — Turn ideas into AI applications with Microsoft Foundry Labs. A lightning talk covering the new Foundry Labs website with interactive demos of the latest releases from Microsoft Research. On-site only in San Francisco.

Come find us in the expo hall – we’d love to connect!

What’s Next

Foundry Labs is where Microsoft’s most ambitious AI research becomes accessible to builders. Whether you’re forecasting hurricanes, building voice agents, designing molecules, or shipping multimodal pipelines — the tools are here, and the next wave is already in the lab.

Stay tuned — there’s more coming soon:

Explore more AI innovations on https://labs.ai.azure.com/
Join the Microsoft Foundry Discord community to shape the future of AI together