rss.livelink.threads-in-node

From Good to Great: We Put Agent Optimizer to the Test in Microsoft Foundry

abikomma — Fri, 07 Aug 2026 14:00:00 GMT

Authors: Abi Komma, Charles Kim, Luffy Chen, Morteza Ziyadi

Why hand-tuning agents does not scale

An agent’s behavior comes mostly from its natural-language configuration: its instructions, skills, and tool descriptions. Most teams tune that configuration by hand. You write the instruction, run some cases, read the failures, reword, and try again. The Foundry documentation (link) calls this the manual trial-and-error loop. It is slow, subjective, and hard to scale across many agents.

The loop is also brittle. Unlike a code bug, a quality failure has no stack trace: the cause can sit in the instruction, a tool description, or the model. Fix one case by editing the instruction, and another that used to pass often breaks. Your evaluators tell you what is wrong, but not the one instruction that satisfies every case at once. Retraining is not the answer either, because reinforcement-learning style tuning is costly and needs many trial runs.

How agent optimizer works

Every run leaves a useful signal: a natural-language trace of the agent’s reasoning and tool calls. Reflection based optimization learns from those traces. Rather than rely on a score alone, it reads a failing run in plain language, works out why the agent scored poorly, and proposes a targeted change to the agent’s configuration, such as its instructions, skills, or tool descriptions, not just a single prompt. Methods like GEPA (link) can match or beat reinforcement-learning tuning with far fewer trials. Agent Optimizer in Foundry Agent Service builds this reflection based optimization into it.

You give agent optimizer the agent, the evaluators you already trust, and a dataset of representative cases, and it searches for a better configuration. Each round, it proposes a candidate change and tries it on a small batch of cases first. Only promising candidates go on to a full evaluation, which keeps the cost down. It keeps a Pareto frontier of the strongest candidates, the ones that win on different cases rather than a single overall best, and draws the next candidate from that pool. You review the leaders and their scores, then approve one or send it back for another pass. The version you approve becomes the agent’s next version, with the same model and tools. You decide what ships.

How we tested it, and what we found

To assess the improvements, we ran agent optimizer across six prompt agents (standard, declarative agents) that span different domains and evaluation types. We chose datasets with ground-truth based evaluators so the quality signal is objective and easy to check. We also included one agent, tax-assistant, with a rubric based judge to score overall response quality. Every run used the same setup: gpt-4o-mini as the agent model, a stronger model as the reflector that proposed mutations, and optimization of the system instructions. Each agent was evaluated the same way before and after, so the delta reflects the instruction change alone.

Agent	Task	Train / Val	Evaluator	Before Optimization	After Optimization	Delta
sentiment	Classifies customer reviews as positive, negative, or neutral, including sarcasm handling.	25 / 15	exact-match	93.3%	100.0%	+6.7%
gsm8k-cot	Solves grade-school math word problems and extracts the final numeric answer.	52 / 30	exact-match	83.3%	93.3%	+10.0%
facility	Triages facility tickets into structured JSON fields for category, sentiment, and urgency.	15 / 30	json-field-match	71.6%	78.9%	+7.3%
retail-policy	Applies retail policy rules to return single-turn customer service requests.	50 / 30	json-field-match	63.3%	74.6%	+11.3%
retail-tools	Uses tool calls to resolve retail customer service scenarios with account and order data.	50 / 25	json-field-match	61.3%	72.9%	+11.6%
taxman-2023	Extracts W-2 data and computes tax liability and credit eligibility for tax scenarios.	16 / 10	llm-as-judge rubric	75.0%	92.5%	+17.5%

Agent optimizer improved the score on every agent. The largest gains came on the agents that had the most room to improve, while an agent already close to the ceiling, such as sentiment, had less to gain. The changes it makes are readable: instead of rewording the prompt at random, it adds concrete rules that target the failure patterns in your data, and you can see exactly what changed before you approve it. The approach is not limited to text-only agents either. One agent in the set, retail-tools, calls tools to look up orders, accounts, and product data before it decides, and agent optimizer improved it while it kept calling those tools during evaluation.

These are early, single-seed runs on small datasets, so the exact magnitudes are directional, but the gains were consistent across every agent and evaluation type. Agent optimizer helps most when an agent already runs on real evaluators and a dataset that reflects production cases, when fixes tend to trade one failure for another, and when you maintain several agents or re-tune often. It is less useful when the root cause is infrastructure, such as a tool that returns bad data, rather than the agent’s configuration, such as its instructions, tool descriptions, skills, or model.

What’s next

This first round benchmarks prompt agents tuned on their system instruction. In the next post, we will share experiments and results on hosted agents, with larger evaluation sets and more optimization targets, including skills, tool descriptions, and model selection.

Get started

For a full walkthrough of optimizing agents, see the Agent optimizer overview (link), which covers both prompt and hosted agents. To optimize a prompt agent, follow the Quickstart (link). To try it on a hosted agent, install the Azure Developer CLI (link) with the Foundry extension, then scaffold a sample agent, deploy it, and optimize:

azd ext install microsoft.foundry # one-time: add the Foundry extension azd ai agent init -m <optimization-sample> # scaffold the sample agent azd provision && azd deploy # create resources and deploy azd ai agent eval generate # generate a dataset and evaluators azd ai agent optimize --max-candidates 2 # run the optimizer azd ai agent optimize apply --candidate <id> # apply the winning candidate azd deploy # redeploy the improved agent

To optimize your own agent, see Make your agent optimizer-ready (link). For more agent examples, browse the foundry-samples repository (link).

Agent optimizer and the azd evaluation experience are in preview, so capabilities can change. The plans described here are work in progress, not a commitment to deliver any feature or timeline. The benchmark numbers come from internal runs on the datasets above, so your own results will vary with your agent, your data, and your evaluators. Microsoft Foundry and Azure are trademarks of the Microsoft group of companies.

Learn more

Read agent optimizer announcement blog: link

See agent optimizer in action - view Build 2026 Observability session: aka.ms/build26-BRK252

The Hidden Reason Your Foundry Agent Can't Reach Any of Your Private (Bring-Your-Own) Resources

vishal_kalal — Fri, 07 Aug 2026 14:00:00 GMT

The setup

You've built the deployment the enterprise architecture review asked for. Nothing touches the public internet.

Your Microsoft Foundry resource is private — public network access disabled, a private endpoint fronting it inside your virtual network. Every resource the agent depends on is locked down the same way:

Azure AI Search — private endpoint, public access off.
Azure Storage — private endpoint, public access off.
Azure Cosmos DB — private endpoint, public access off.
An MCP server (say on Azure Functions, Container Apps or APIM) — private, reachable only from inside the VNet.

You've delegated a subnet for network injection, stood up Private DNS zones for the Foundry account and every data resource, kept the Foundry resource and the VNet in the same region, and configured network injection at account-creation time like the docs require. On paper, this is a textbook private standard agent setup.

Then you run your first agent.

The problem: network errors everywhere

The agent can't reach anything. And the errors look exactly like what you'd expect from a private-networking problem — connection failures and name-resolution errors reaching Search, Storage, and Cosmos; MCP tool calls timing out. Every message points at the network.

(Some of you will see the quieter version of this bug instead: no errors at all, but your Cosmos container, your blob container, and your search index stay stubbornly empty. Same root cause — hold that thought.)

So, you do the natural thing. You open Network Watcher and start debugging the network.

That's the trap. In the vast majority of these cases, the network is fine. The failure is happening one layer up, in a resource most people barely touch — and it presents as a network problem because of what it does when it's misconfigured.

The layer you're probably staring at

The network layer is the obvious suspect, and to be fair, private standard agent setup has a lot of moving parts that can genuinely break:

Private endpoints and Private DNS zones for the Foundry account and for each data resource, so name resolution stays inside the VNet.
A dedicated subnet delegated to Microsoft.App/environments for network injection.
Foundry resource and VNet in the same region.
Network injection configured at account creation — it can't be added to an existing account afterward.

Here's the useful distinction, though, and it's the pivot of this whole post: not every "unreachable resource" belongs to the same layer.

The layer that's actually failing: the capability host

A capability host is a sub-resource you configure at both the Foundry account scope and the Foundry project scope. Its entire job is to tell Agent Service where to store and process agent data — conversation history, file uploads, and vector stores. Capability hosts let you bring your own Azure resources instead of using the default Microsoft-managed platform resources. This gives you:

Data sovereignty - Keep all agent data within your Azure subscription.
Security control - Use your own storage accounts, databases, and search services.
Compliance - Meet specific regulatory or organizational requirements.

If you don't create capability hosts, Agent Service doesn't error out. It quietly uses Microsoft-managed resources for all the communications. That's a fine default for development. The moment you bring your own resources, you're in "standard agent setup," and the contract changes: you now have to declare your resources explicitly, in two places, through connections that resolve correctly — or things break.

And here's why it masquerades as a network problem. The capability host resolves each of your resources through a connection, and that connection has to carry enough information to bind to the right resource at runtime. Four properties matter:

Property	What it must be
authType	The auth type, e.g. AAD
category	The resource type: AzureStorageAccount, AzureCosmosDb, or CognitiveSearch
target	The service endpoint URL — not the resource ID
metadata.ResourceId	The full Azure resource ID of the resource

If target is wrong, or metadata.ResourceId is missing or malformed, the agent tries to reach the wrong endpoint — or can't resolve the resource at all. The symptom you see is a connection or resolution failure. The symptom looks like DNS or a private endpoint. The cause is a bad connection object. This is a resolution problem wearing a **connectivity** costume.

Hidden reason #1: There is no inheritance from account to project

This is the big one, and it's the reason most private setups fail. It's natural to assume that if you configure things at the account level — the account capability host, account-level connections — everything cascades down to the projects underneath. It doesn't.

The project-level capability host is the only thing Agent Service reads to decide which storage, conversation, and vector store resources a project uses. There is no automatic inheritance of BYO resource configuration from the account capability host to the project. Even if your account capability host references every connection perfectly, Agent Service will not use those resources for a project unless that project has its *own* capability host that explicitly references them.

Account-level connections are inherited by new projects — but a connection and a capability host are different things. Inheriting the connection only means the project can see it. You still have to create a project capability host that points at it.

So the classic failure looks like this: you create the account capability host and your connections, maybe reference them at the account level, and stop — because it feels complete. You never created a project capability host, or created one that doesn't reference your connections. Now Agent Service has no project-level instruction to use your resources, so it either falls back to managed defaults (the silent, empty-container variant) or fails to resolve your resource entirely (the loud, network-error variant). Either way, from your private resource's point of view, the agent never correctly reached it.

The fix is the project capability host that ties it all together:

PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.CognitiveServices/accounts/{accountName}/projects/{projectName}/capabilityHosts/{name}?api-version=2025-06-01 { "properties": { "capabilityHostKind": "Agents", "threadStorageConnections": ["my-cosmos-db-connection"], "vectorStoreConnections": ["my-ai-search-connection"], "storageConnections": ["my-storage-account-connection"], "aiServicesConnections": ["my-azure-openai-connection"] } }

aiServicesConnections is optional — include it only if the project should use models from your own Azure OpenAI resource. And ordering matters: you can't create a project capability host until an account-level one already exists. Account host first, then project host.

How to diagnose this in five minutes

Before you touch Network Watcher again, run these checks. Capability hosts are managed through the REST API today (no SDK surface yet), so this is all ARM. If required, you can use Azure Cloud Shell:

Get Access Token:

$TOKEN=$(az account get-access-token --resource https://management.azure.com --query accessToken -o tsv)

Confirm the account capability host exists:

curl -X GET "https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.CognitiveServices/accounts/{accountName}/capabilityHosts?api-version=2025-06-01" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" | jq .

If account capability host doesn't exit, create one:

curl -X PUT \ "https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.CognitiveServices/accounts/{accountName}/capabilityHosts/{capabilityHostName}?api-version=2025-06-01" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "properties": { "capabilityHostKind": "Agents", "customerSubnet": "{subnetId}", "threadStorageConnections": [ "my-cosmos-db-connection" ], "vectorStoreConnections": [ "my-ai-search-connection" ], "storageConnections": [ "my-storage-account-connection" ], "aiServicesConnections": [ "my-azure-openai-connection" ] } }'

Confirm the project capability host exists — and references the connections you expect. This is the one people are missing.

curl -X GET "https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.CognitiveServices/accounts/{accountName}/projects/{projectName}/capabilityHosts?api-version=2025-06-01" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" | jq .

Look for threadStorageConnections (Cosmos), vectorStoreConnections (AI Search), and storageConnections (Storage). Empty or missing? That's your bug - the agent has no instruction to use your resources.

If project capability host doesn't exit, create one:

curl -X PUT \ "https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.CognitiveServices/accounts/{accountName}/projects/{projectName}/capabilityHosts/{capabilityHostName}?api-version=2025-06-01" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "properties": { "capabilityHostKind": "Agents", "threadStorageConnections": [ "my-cosmos-db-connection" ], "vectorStoreConnections": [ "my-ai-search-connection" ], "storageConnections": [ "my-storage-account-connection" ], "aiServicesConnections": [ "my-azure-openai-connection" ] } }'

Inspect each referenced connection. Verify authTyp, category, target, and especially metadata.ResourceId. Watch the classic swap: target is the endpoint URL, metadata.ResourceId is the ARM resource ID. Reversing them is an easy hand-authoring mistake, and it produces failures that read as network errors. A correct Storage connection looks like this:

{ "properties": { "authType": "AAD", "category": "AzureStorageAccount", "target": "https://{storageAccountName}.blob.core.windows.net/", "metadata": { "ResourceId": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Storage/storageAccounts/{storageAccountName}" } } }

Run a test conversation and check where the data actually landed. Conversations should appear in your Cosmos DB, uploaded files in your Storage account, and vector data in your AI Search index and communication to private MCP server should work.

The gotchas that keep you stuck

Even once you know the fix, a few constraints conspire to make this harder than it should be:

Capability hosts can't be updated. Wrong connection name? You can't PATCH it — you delete and recreate. Build your automation around delete-then-create, not an idempotent upsert.
One capability host per scope. A second one with a different name at the same scope returns 409 Conflict. Same name + same config is idempotent; same name + different config returns a 400.
Deleting a capability host affects every dependent agent. Delete the project and account hosts and your agents lose access to the files, conversations, and vector stores they were using. Treat deletion as a breaking change.
Permissions. You need Contributor on the Foundry account to create capability hosts, and User Access Administrator or Owner to grant the agent access to your Azure resources.

The mental model to take away

Split the question in two.

Connectivity asks: can the agent reach this resource? That's the network layer — private endpoints, DNS, subnet delegation, the data proxy. It owns your MCP server and any tool egress.
Resolution asks: is the project even told to use this resource, and is the connection pointing at the right place? That's the capability host layer. It owns your Search, Storage, and Cosmos DB.

They fail in completely different ways, and - this is the whole point - a resolution failure can look like a connectivity failure. When all of your data resources go dark at once, that's your signal it's resolution, not connectivity. No amount of DNS troubleshooting fixes a project that was never wired to your resources in the first place.

Match the symptom to the layer, and you'll stop losing days to the wrong one.

A 60-second checklist

[] Account-level capability host exists.

[] Project-level capability host exists (the one everyone misses).

[] The project capability host references your Cosmos, AI Search, and Storage connections by name.

[] Each referenced connection has a valid metadata.ResourceId.

[] target is the endpoint URL and metadata.ResourceId is the ARM resource ID — not swapped.

[] authType and category are correct for each resource type.

If all of these pass and it still doesn't work — *then* go back to the network.

Further reading

Closing

Private-by-default is the right posture for enterprise agents — but it moves the failure modes around. The instinct to blame the network is strong precisely because the network is where you spent all your effort. Next time your agent can't reach its private resources, resist that instinct for five minutes and check the project capability host first. More often than not, that's where the day was hiding.

Cannot verify email

jonfrontfoot — Fri, 07 Aug 2026 09:12:49 GMT

I have put in a request for quota to deploy an OpenAI model. I was then sent a link to verify my email address - I've signed out everywhere, tried incognito windows in multiple browsers, copied the link out of the email and pasted it into a browser etc. but when I click the link to verify I always get:

The request is blocked.

Any ideas?!

Microsoft Foundry's New Model Wave

Christian_Coello — Thu, 06 Aug 2026 16:00:00 GMT

The era of the single best model is over

Microsoft Foundry is no longer just a shelf of foundation models. It is becoming an enterprise model portfolio: frontier models for complex reasoning, cost-optimized models for scale, specialized models for documents and media, and first-party Microsoft AI models for text, image, voice, and speech. That shift changes the operating question for customers and partners.

The question is not simply, “Which model is best?” It is, “Which model is best for this workload, risk profile, latency target, context size, modality, and budget?” In practice, the winning architecture is increasingly a model-routing architecture, where different workloads are routed to different models based on business value and operational constraints.

GPT-5.6: the premium Microsoft Foundry reasoning lane

GPT-5.6 (available in Microsoft Foundry), including gpt-5.6-sol, gpt-5.6-terra, and gpt-5.6-luna, is the model family many enterprises will evaluate first for their most demanding reasoning and agentic workloads. Microsoft positions the family for advanced reasoning, coding, long-context understanding, research, cybersecurity analysis, and sophisticated enterprise workflows. The value is not only raw capability, but the broader Foundry operating model: Azure billing, Microsoft support, enterprise controls, familiar APIs, and integration with the Foundry development experience.

GPT-5.6 is a strong starting point for high-value applications such as codebase analysis, contract and compliance review, complex agent planning, and long-document synthesis. The tradeoff is that premium frontier models typically require more thoughtful quota, latency, and cost planning than smaller or more specialized options.

Kimi: what it is and why it matters

Kimi is a family of models from Moonshot AI that is now available in Microsoft Foundry for coding, reasoning, and agentic software engineering scenarios. The most important model for this blog is Kimi-K2.7-Code, which Microsoft describes as a coding-focused agentic model built on Kimi-K2.6. In plain language, Kimi is not just trying to answer one prompt well. It is designed for work where an AI system needs to reason, use tools, follow instructions across a long context, and keep making progress across multiple engineering steps.

That makes Kimi relevant for long-horizon coding: refactoring across a codebase, implementing a feature across multiple files, debugging a complex issue, generating tests, or helping an agent plan and execute a workflow. Microsoft’s Kimi K2.7 Code materials call out improvements in end-to-end task completion, multi-step execution, and long-context coding. Moonshot also reports that K2.7 Code reduces thinking-token usage by about 30 percent compared with K2.6, which is why Kimi is often discussed as a price-performance option rather than only a quality benchmark.

For partners and customers, the practical positioning is simple: evaluate Kimi when the workload is coding-heavy, agentic, high-volume, or cost-sensitive. It may not replace GPT-5.6 or Claude for every high-stakes enterprise workflow, but it is exactly the kind of model that can lower cost or increase throughput when the task is narrower, measurable, and repeatable. Kimi is especially worth testing for developer copilots, DevOps automation, internal engineering agents, software lifecycle workflows, and partner demos where strong coding performance and cost discipline both matter.

Claude and DeepSeek: strong alternatives for agents and scale

Claude remains a trusted enterprise comparator for agentic workflows, coding, financial analysis, security operations, and long-running knowledge work. In Foundry, Claude models are available through Foundry and are often the model to benchmark next to GPT-5.6 when quality, safety posture, and enterprise readiness matter more than lowest unit cost.

DeepSeek is best framed as a reasoning-at-scale option. DeepSeek-V4-Pro and related models are designed for math, scientific reasoning, coding analysis, multilingual reasoning, and large-context processing. For teams running heavy evaluation pipelines or high-throughput reasoning workloads, DeepSeek should be part of the bake-off, especially when operational scale is as important as answer quality.

Microsoft MAI: the first-party multimodal stack

The other important addition is Microsoft AI’s MAI model family. MAI expands Foundry beyond general-purpose chat into a first-party multimodal stack across reasoning, image generation and editing, voice generation, and transcription. Microsoft has announced MAI-Thinking-1 for text and reasoning, MAI-Image-2.5 and MAI-Image-2.5 Flash for image generation and image-to-image editing, MAI-Voice-2 for multilingual text-to-speech, and MAI-Transcribe-1.5 for speech-to-text.

For enterprise builders, MAI matters because many AI applications are not just chatbots. A customer-facing experience may need reasoning, branded visuals, localized voice, and high-accuracy transcription in one workflow. MAI should sit in the routing matrix as the first place to look for Microsoft-first multimodal workloads, especially when the app needs tight alignment with Microsoft product experiences, Foundry deployment, and enterprise-grade governance.

Specialized models still matter

The broader Foundry catalog also includes specialized options such as Mistral for OCR and document intelligence, Llama for open-model and portability-sensitive scenarios, Grok for additional reasoning and coding evaluations, and FLUX for image generation. These models matter because not every workload should be forced through a frontier reasoning model. A cheaper specialized model can often produce a better business outcome when the task is narrow and measurable.

Recommended routing matrix

Workload	First model to test	Why
Most demanding enterprise reasoning and agents	GPT-5.6	Premium Foundry reasoning lane, long context, enterprise controls
Enterprise coding and trusted agents	Claude Sonnet 5	Strong quality and enterprise fit for coding and agent workflows
Cost-sensitive coding agents	Kimi-K2.7-Code	Long-horizon coding, multi-step execution, and price-performance
High-throughput reasoning	DeepSeek V4-class	Reasoning-heavy workloads and large-scale evaluation pipelines
Microsoft-first multimodal apps	MAI models	Reasoning, image, voice, and transcription across a first-party Microsoft stack
OCR and document extraction	Mistral OCR / Document AI	Purpose-built document intelligence capabilities
Open-model or portability-sensitive workloads	Llama / Kimi	Flexible evaluation path and portability options

Bottom line

For most enterprises, the right answer is not model standardization. It is model routing. Benchmark real prompts, measure quality, latency, cost, safety behavior, tool-call accuracy, context handling, and operational fit, then route by task.

GPT-5.6 may become the premium default for difficult reasoning; Claude may serve as a trusted enterprise comparator; Kimi may become the coding-agent value play; DeepSeek may power high-volume reasoning; MAI may become the Microsoft-first multimodal layer; and specialized models will continue to handle OCR, images, extraction, and targeted workflows. Foundry’s model wave is not just about more models. It is about giving builders enough model diversity to optimize for quality, cost, throughput, modality, and governance at the same time.

Get Started

Ready to build your routing architecture?

Start with the Foundry Model Catalog to compare models side-by-side, or try our Deploy Microsoft Foundry Models in the Foundry portal - Microsoft Foundry | Microsoft Learn quickstart guide to deploy your first model in minutes.

Additional Resources

From Distillation to Reinforcement Fine-Tuning: Hill-Climbing in Microsoft Foundry

srikantan — Thu, 06 Aug 2026 14:00:00 GMT

TL; DR: Hill climbing means improving the deployable model package one measured step at a time across quality, latency, and cost. SFT made this workload cheaper, but did not transfer the teacher's planning quality. Adding explicit reasoning produced the major quality gain. RFT in Microsoft Foundry then retained near-teacher quality, achieved 100% feasibility on 150 held-out scenarios, and reduced token cost from $0.09975 to $0.03101 per scenario. Jump to the measured hill climb.

What you will learn

How to frame model customization as a measurable hill climb rather than a one-shot training decision.
Where SFT and grader-based RFT differ for multi-constraint planning.
How a deterministic Python grader turns business outcomes into an RFT reward.
How Microsoft Foundry manages datasets, training, monitoring, checkpoints, deployment, and evaluation.
What the training metrics and deployed-model comparison actually establish.

Quick glossary

Microsoft Foundry: Microsoft's platform for building, customizing, deploying, and evaluating AI applications and models.
SFT (supervised fine-tuning): Model customization using labeled input-and-output demonstrations.
RFT (reinforcement fine-tuning): Model customization in which sampled outputs receive rewards from a grader and the training process increases the likelihood of higher-reward behavior.
Grader: A scoring function used during RFT. This experiment uses deterministic Python code rather than another model as the judge.
Feasibility: Whether a complete plan satisfies schema, coverage, inventory, capacity, substitution, quantity, warehouse, shipping-mode, and expedite-budget constraints.
Hill climb: A sequence of measured model, prompt, reasoning, and fine-tuning changes that seeks a better operational package at each step.

Why Hill Climb the Model Package?

Model selection is not a single-axis leaderboard exercise. A package that produces the highest-quality answer may be too slow or expensive for the workload. A smaller model may be economical but fail the business constraints. The useful unit of comparison is therefore the complete package:

model + prompt + reasoning configuration + fine-tuning method

For this experiment, I evaluated every package on three dimensions:

Dimension	What was measured
Quality	Feasibility plus priority-weighted service, retained margin, and shipping-cost efficiency
Latency	End-to-end time to produce one coordinated plan
Cost	Input, output, and reasoning-token charges; fine-tuned hosting was tracked separately

The hill climb used quality as the primary objective. Only after the task was solved well did it make sense to reduce cost and latency without giving that quality back.

Why Microsoft Foundry Changes the Fine-Tuning Workflow

Microsoft Foundry has long supported supervised fine-tuning for supported models. It now also supports reinforcement fine-tuning for supported reasoning models, including grader-based training, structured response formats, RFT-specific hyperparameters, reward monitoring, automatic validation evaluations, checkpoints, and deployment.

This matters because RFT is more than uploading a different JSONL file. A practical workflow must:

Prepare and validate training and validation datasets.
Attach a grader that can score generated responses reliably.
Sample candidate behavior during managed training.
Run reward-based optimizer updates.
Monitor training reward, validation reward, reasoning tokens, grader latency, and step duration.
Produce deployable checkpoints.
Deploy the customized model and evaluate it on a separate held-out test set.

Foundry turns that lifecycle into a turnkey, managed workflow through its portal and APIs. In this project, GitHub Copilot agent mode in Visual Studio Code drove the workflow code-first, while Foundry supplied the managed training and model-hosting infrastructure. The datasets, grader, job payloads, deployment scripts, and evaluation artifacts remain reviewable in source control.

Model support is method-specific. At the time of this experiment, in Microsoft Foundry, gpt-4.1-mini supported SFT and o4-mini supported RFT; the same base model was not available for both methods. The results should therefore be read as a comparison of deployable model-and-method packages, not as a controlled claim that RFT universally outperforms SFT when the base model is held constant. Always check the current fine-tuning model support table before starting a new experiment.

The Supply-Chain Recovery Problem

The experiment runs on a benchmark of procedurally generated warehouse-disruption scenarios — hundreds used for training and validation, plus a separate held-out set of 150 for evaluation. Each scenario represents a disruption in which one warehouse is unavailable. The model receives:

12 to 16 affected orders, each with a SKU, quantity, priority, deadline, margin, and optional approved substitute.
Three warehouses with availability, SKU inventory, shipment capacity, transit time, and standard and expedite costs.
One shared expedite budget.

For every order, the model must choose a warehouse, the requested SKU or an approved substitute, a quantity, and standard or expedited shipping, or defer the order. The answer must be one complete JSON plan.

This is difficult because decisions are coupled. Assigning three units of SKU A to one order changes the inventory available to all other orders. Expediting a flexible order may consume the budget needed for an urgent, high-priority order. A plan can use only valid choices and still be suboptimal; worse, one shared-constraint violation makes the entire plan infeasible.

Figure 1. The comparison application exposes orders, priorities, deadlines, substitutes, margins, and the shared expedite budget. Readers can change the scenario and run the deployed packages without changing code.

SFT and RFT Receive Different Learning Signals

The SFT dataset contained a scenario and one selected teacher plan as the labeled assistant response. Only teacher plans that were feasible and scored at least 0.75 were eligible. This produced 148 training and 38 validation demonstrations.

SFT is a strong choice when the desired answer can be demonstrated directly. It teaches the model to reproduce the labeled target token by token. An SFT model can still reason about constraints at inference time, but its training objective does not ask it to generate several valid alternatives, evaluate their business outcomes, or reward a different plan that is better than the single label. Shared constraints are learned only to the extent that the demonstrations and prompt represent them successfully.

RFT rows contained the prompt and scenario but no target plan. The model sampled plans during training, and the grader evaluated each sampled outcome. Every captured scenario was retained, producing 300 training and 60 validation rows across equal loose, mixed, and tight budget-pressure families.

This distinction is central:

SFT	RFT
Learns from one labeled response per example	Learns from rewards on sampled responses
Optimizes agreement with demonstrations	Optimizes the grader-defined outcome
A different valid plan can still disagree with the label	Different plans can earn similar rewards if their outcomes are similar
Requires high-quality target answers	Requires a trustworthy, discriminating grader

Turning Business Outcomes into a Reward

The grader has two jobs. First, it acts as a hard feasibility gate. Malformed plans, missing or duplicate orders, unavailable warehouses, incorrect quantities, prohibited substitutes, exhausted inventory or capacity, invalid shipping modes, and excess expedite spending receive zero reward.

Second, it scores every feasible plan using normalized service, margin, and cost components:

Q=0.55S+0.25M+0.20Ce

Here, S is priority-weighted on-time service, M is retained margin, and Ce is fulfillment-adjusted shipping-cost efficiency. Service receives the highest weight, but the model cannot ignore margin or spend.

The following shortened excerpt shows the important shape of the Python scorer used in the experiment:

from collections import defaultdict def score_plan(plan, scenario): # Schema, coverage, type, warehouse, quantity, substitution, and # shipping-mode checks run before this shortened scoring excerpt. orders = {order["order_id"]: order for order in scenario["orders"]} warehouses = { warehouse["warehouse_id"]: warehouse for warehouse in scenario["warehouses"] } inventory_used = defaultdict(int) capacity_used = defaultdict(int) expedite_spend = 0.0 shipping_cost = 0.0 service_earned = 0.0 margin_earned = 0.0 shipped_orders = 0 for decision in plan["decisions"]: order = orders[decision["order_id"]] warehouse = warehouses[decision["warehouse_id"]] quantity = decision["quantity"] inventory_used[(warehouse["warehouse_id"], decision["sku"])] += quantity capacity_used[warehouse["warehouse_id"]] += quantity unit_cost = warehouse[f"{decision['shipping_mode']}_cost"] shipping_cost += unit_cost * quantity if decision["shipping_mode"] == "expedite": expedite_spend += unit_cost * quantity delivered_on_time = ( warehouse[f"{decision['shipping_mode']}_hours"] <= order["deadline_hours"] ) if delivered_on_time: service_earned += order["priority"] retention = 0.90 if decision["sku"] != order["sku"] else 1.0 if not delivered_on_time: retention *= 0.50 margin_earned += order["margin"] * retention shipped_orders += 1 if any( used > warehouses[warehouse_id]["inventory"].get(sku, 0) for (warehouse_id, sku), used in inventory_used.items() ): return {"score": 0.0, "feasible": False, "reason": "inventory exceeded"} if any( used > warehouses[warehouse_id]["shipment_capacity"] for warehouse_id, used in capacity_used.items() ): return {"score": 0.0, "feasible": False, "reason": "capacity exceeded"} if expedite_spend > scenario["expedite_budget"]: return {"score": 0.0, "feasible": False, "reason": "budget exceeded"} service = service_earned / sum(order["priority"] for order in orders.values()) margin = margin_earned / sum(order["margin"] for order in orders.values()) fulfilled_fraction = shipped_orders / len(orders) reference_cost = max(1.0, sum(o["quantity"] for o in orders.values()) * 15.0) cost_efficiency = fulfilled_fraction * max(0.0, 1.0 - shipping_cost / reference_cost) return { "score": 0.55 * service + 0.25 * margin + 0.20 * cost_efficiency, "feasible": True, }

Microsoft Foundry packages this logic as a Python grader. During RFT, the model output is available to the grader as the sample, while the structured scenario remains private grader context. This avoids leaking a target answer to the model and keeps the reward deterministic.

SFT Run: Excellent Imitation Metrics, Weak Business Generalization

The SFT run used gpt-4.1-mini-2025-04-14 with the following configuration:

Setting	Value
Training rows	148
Validation rows	38
Epochs	3
Seed	42
Training type	Global Standard
Foundry-selected batch size	1
Foundry-selected learning-rate multiplier	2
Training steps	444

The job completed in approximately 1 hour and 6 minutes wall-clock and billed 699,000 training tokens. Final training loss was 0.01, final validation loss was 0.01, and both training and validation mean token accuracy reached 0.99.

Figure 2. The completed SFT job in Microsoft Foundry. Low loss and high token accuracy show that the model learned the selected teacher demonstrations closely.

Those metrics looked excellent, but they measured imitation of the labeled responses. They did not establish that a newly generated plan would satisfy every shared constraint or maximize the business reward. That distinction became visible only after deployment: across 150 held-out scenarios, SFT achieved 0.313 mean quality and only 38.7% feasibility.

This was a valuable failed hill-climb step. SFT reduced P50 latency from 4.54 to 4.18 seconds and token cost from $0.00858 to $0.00119 per scenario, but the quality regression meant the package had not improved overall.

RFT Run: Reward Increased as the Policy Improved

The RFT run used o4-mini-2025-04-16 and the deterministic grader:

Setting	Value
Training rows	300
Validation rows	60
Epochs	2
Reasoning effort	Medium
Compute multiplier	2
Learning-rate multiplier	1.0
Evaluation interval	Every 5 training steps
Validation samples per evaluation	10
Seed	42
Training type	Global Standard
Foundry-selected batch size	9
Approximate training steps	67

The run completed in approximately 25.6 hours wall-clock, including about one hour for service orchestration, data import, and GPU queueing. Foundry billed 24.750 training hours. Final mean training reward reached 0.80, and final full-validation mean reward reached 0.82.

Figure 3. Foundry's RFT monitor shows training reward rising from roughly 0.69 toward 0.80. It also exposes validation reward, reasoning-token usage, grader execution latency, and per-step duration. Periodic validation produces the regular duration spikes.

The curve is noisy because each step uses a different training batch, so the trend matters more than any one point. The validation reward did not diverge from training reward, but training metrics alone were not the decision gate. A model can exploit a grader, overfit its scenarios, or find a mediocre shortcut that still earns reward. The final decision therefore used a separate 150-scenario test set that did not overlap training or validation data.

The Measured Hill Climb

The experiment progressed through four packages:

GPT-5.2 teacher without explicit reasoning: Fast, but frequently violated coupled constraints.
GPT-4.1-mini with SFT: Cheaper and slightly faster, but did not preserve planning quality.
GPT-5.2 teacher with a strengthened prompt and medium reasoning: Solved the task well, with a large latency and token-cost increase.
o4-mini with grader-based RFT and medium reasoning: Preserved near-teacher quality while reducing latency and token cost.

Figure 4. The hill climb uses a documented composite index: 70% quality, 15% inverse token cost, and 15% inverse P50 latency. The measured components remain visible so the weighting is auditable.

Package	Quality	Feasible	P50 latency	Token cost/scenario
GPT-5.2 teacher, no explicit reasoning	0.335	42.7%	4.54 s	$0.00858
GPT-4.1-mini SFT	0.313	38.7%	4.18 s	$0.00119
GPT-5.2 teacher, medium reasoning	0.851	99.3%	66.68 s	$0.09975
o4-mini RFT, medium reasoning	0.833	100%	58.59 s	$0.03101

The reasoning teacher achieved the highest absolute quality. RFT trailed it by 0.018 quality points, but reduced P50 latency by about 12% and token cost by about 69%. That made RFT the strongest operational balance in this experiment, not the universal winner on every dimension.

Fine-tuned model hosting is separate from the token charges in the table and accrues while a customized deployment is active. Workload volume and concurrency therefore matter when calculating production cost per scenario.

What the Deployed Models Did in the Comparison App

After training, I deployed the SFT and RFT models and exposed them in the Warehouse Planning Gym comparison application. The app sends the same editable scenario to both packages, displays token usage, latency, estimated token cost, and allocation decisions, and then runs the production-style grader to explain the operational result.

Note: The input provided below to the model is different from the one used in the finetuning and evaluation run

Figure 5. On this live scenario, SFT produced a plausible-looking plan but spent $42 on expedite shipping against a $35 shared budget. One coupled-constraint violation made the complete plan infeasible, so the grader returned 0.000. RFT respected the budget and every other hard constraint, earning 0.900.

The SFT failure is exactly the kind that can hide in a row-by-row review. Each allocation decision looks reasonable in isolation. The error appears only after expedite spending is accumulated across the complete plan. RFT took longer for this request because it used 4,160 reasoning tokens, but it produced an executable outcome.

Figure 6. On the same scenario, the reasoning teacher and RFT produced the same feasible allocation and both scored 0.900. RFT's estimated token charge was $0.0219, compared with $0.0306 for the teacher in this run.

This single scenario makes the constraint behavior understandable, but it is not the performance claim. In this sample, the teacher happened to be faster than RFT (18.2 versus 40.9 seconds). The 150-scenario paired evaluation is the stronger evidence: there, RFT retained near-teacher quality, was faster at P50, and had substantially lower average token cost.

What This Experiment Establishes, and What It Does Not

Five conclusions stand out.

Hill climbing kept the optimization honest. The cheap SFT package looked attractive until held-out quality was included. The strongest package emerged only after quality, latency, and cost were measured together.
Training metrics must match the business decision. SFT's 0.99 token accuracy proved excellent imitation, not constraint satisfaction. RFT's reward metric was better aligned with the business objective, but still required held-out evaluation.
A grader converts policy into a learnable signal. Hard rules received zero reward; feasible alternatives received continuous credit for service, margin, and cost. That let training reward outcomes instead of one canonical answer.
RFT learns a policy, not an embedded optimizer. At inference time the model does not normally run the grader or prove global optimality. For guaranteed feasibility, keep validation, repair, best-of-N selection, or a mathematical optimizer in the runtime architecture.
Foundry made the lifecycle operational. Dataset management, grader attachment, managed training, reward monitoring, validation evaluations, checkpoints, deployment, and model invocation were part of one platform workflow rather than separate training infrastructure.

There are also important limitations. SFT and RFT used different base models because Foundry's supported methods were model-specific. The teacher packages used different prompting and reasoning settings. Raw o4-mini could not be deployed for a same-base pre/post comparison because that version was deprecated. This experiment therefore demonstrates a practical package hill climb for one supply-chain workload; it does not prove that RFT always beats SFT or mathematical optimization.

For a production fulfillment system, an optimization solver remains the right choice when a formally optimal allocation is required. The model can still add value by interpreting disruptions, mapping unstructured inputs into constraints, proposing strategies, or producing candidate plans that deterministic components validate.

Get Started

Ready to run the hill climb or adapt it to another constrained planning problem?

Explore the complete implementation: Clone the Supply-Chain Hill-Climb Showcase. The repository includes dataset generation, SFT and RFT submissions, the Python grader, deployment scripts, held-out evaluation, charts, and the model comparison application.

Follow the experiment: Use the step-by-step runbook, review the fine-tuning stage, and inspect the held-out evaluation methodology.

Learn the Foundry workflow:

Join the conversation: Which business process in your organization has several valid answers, objective outcome measures, and enough complexity that prompt engineering alone has plateaued? Share the scenario in the comments, or open an issue in the repository so the grader and evaluation design can be discussed with the implementation in view.

Exploring Multi-Agent Workflows with Microsoft Agent Framework

SonakshiA — Wed, 05 Aug 2026 20:00:00 GMT

As organizations race to automate decision-making, content generation, analysis, and execution at scale, the future is shifting from isolated AI agents to multi-agent workflows where specialized agents collaborate, delegate, challenge, and refine each other's work. By allowing multiple AI agents to work together, organizations can tackle complex business processes more efficiently, improve accuracy, and scale operations in ways that would be difficult for a single agent to achieve alone.

In this blog, we will explore 5 types of multi-agent workflows offered by the Microsoft Agent Framework to design, manage, and scale complex multi-agent workflows.

Prerequisites for the Tutorial:

Azure Subscription
Microsoft Foundry resource deployed in a resource group. Deploy any model (such as gpt-4.1-mini) in the Foundry project.

1. Concurrent Orchestration

In this orchestration, the same input is sent to multiple agents simultaneously and consolidated. Each agent handles tasks independently and the results are combined. All agents work at the same time.

Used when:

You need different approaches/perspectives for a problem
Group decision making-based scenarios
Voting-based scenarios

Use Case: Ticket Assessment on Various Criteria

Consider a customer-support use case in which a new ticket must be assessed quickly and routed correctly. With concurrent orchestration, the same incoming ticket is sent to three specialized agents at the same time.

import os import asyncio from typing import cast from agent_framework import Message from agent_framework.foundry import FoundryChatClient from azure.identity import DefaultAzureCredential from agent_framework.orchestrations import ConcurrentBuilder from dotenv import load_dotenv load_dotenv() async def main(): credential = DefaultAzureCredential() chat_client = FoundryChatClient( credential=credential, project_endpoint=os.getenv("AZURE_AI_PROJECT_ENDPOINT"), model=os.getenv("AZURE_AI_MODEL_DEPLOYMENT_NAME") ) sentiment_agent = chat_client.as_agent( name="Sentiment Agent", instructions="You are a helpful assistant that analyzes the sentiment of a support ticket." ) category_agent = chat_client.as_agent( name="Category Agent", instructions="You are a helpful assistant that categorizes a support ticket into categories such as Billing, Technical, Refund, or Account." ) priority_agent = chat_client.as_agent( name="Priority Agent", instructions="You are a helpful assistant that determines the priority of a support ticket as High, Medium, or Low." ) workflow = ConcurrentBuilder( participants = [sentiment_agent, category_agent, priority_agent] ).build() result = await workflow.run("I was charged twice and I'm furious — refund me now!") outputs = result.get_outputs() i = 1 for response in outputs: for msg in cast(list[Message], response.messages): name = msg.author_name or ("assistant" if msg.role == "assistant" else "user") print(f"{'-' * 60}\n{i:02d} [{name}]\n{msg.text}") i += 1 if __name__ == "__main__": asyncio.run(main())

2. Sequential Orchestration

Used when the output of one agent is consumed by subsequent agents one after another. This pattern is ideal for workflows where each step depends on the previous one.

Used When:

There is a multi-step process, where each step relies on the output of the previous one.
Situations that benefit from iterative refinement, such as drafting, reviewing, and improving content.
Each stage produces an output and the next output builds upon that output.

Use Case: Automated Support Ticket Triage

Consider a customer-support operation that receives large volumes of unstructured tickets and must route each one accurately. In this sequential workflow, a Summarizer Agent first condenses the raw ticket into one or two sentences that capture the customer’s core intent. Its output is then passed to a Classifier Agent, which assigns exactly one category—Billing, Technical, Refund, or Urgent.

import os import asyncio from typing import cast from agent_framework import Message from agent_framework.foundry import FoundryChatClient from azure.identity import DefaultAzureCredential from agent_framework.orchestrations import SequentialBuilder from dotenv import load_dotenv load_dotenv() async def main(): credential = DefaultAzureCredential() chat_client = FoundryChatClient( credential=credential, project_endpoint=os.getenv("AZURE_AI_PROJECT_ENDPOINT"), model=os.getenv("AZURE_AI_MODEL_DEPLOYMENT_NAME") ) summarizer_agent = chat_client.as_agent( name="Summarizer Agent", description="Summarizes a support ticket into 1-2 sentences of core intent.", instructions="You are a helpful assistant that summarizes support tickets into concise summaries." ) classifier_agent = chat_client.as_agent( name="Classifier Agent", description="Classifies a ticket summary into: Billing, Technical, Refund, or Urgent.", instructions="You are a helpful assistant that classifies a support ticket summary strictly into one of the following categories: Billing, Technical, Refund, or Urgent." ) workflow = SequentialBuilder( participants=[summarizer_agent, classifier_agent], output_from = "all" ).build() ticket = "I was charged twice for my subscription this month and need a refund ASAP." result = await workflow.run(ticket) outputs = result.get_outputs() i = 1 for response in outputs: for msg in cast(list[Message], response.messages): name = msg.author_name or ("assistant" if msg.role == "assistant" else "user") print(f"{'-' * 60}\n{i:02d} [{name}]\n{msg.text}") i += 1 if __name__ == "__main__": asyncio.run(main())

3. Group Chat Orchestration

Manages a collaborative conversation between multiple agents, optionally involving a human in the process (Human-in-the-Loop). There is a central chat manager that decides which agent responds next and when to request for human input.

Used when:

Scenarios that require debates or group brainstorming.

Use Case: Cross-Functional Feature Proposal Review

A product team uses specialist agents to review a feature proposal in one shared discussion. Product Agent assesses value, Engineering Agent feasibility, Design Agent usability, and Security Agent compliance. A Manager Agent guides the debate and concludes with a recommendation to proceed, revise, or reject.

import os import asyncio from typing import cast from agent_framework import AgentResponseUpdate, Message from agent_framework.foundry import FoundryChatClient from azure.identity import DefaultAzureCredential from agent_framework.orchestrations import GroupChatBuilder from dotenv import load_dotenv load_dotenv() async def main(): credential = DefaultAzureCredential() chat_client = FoundryChatClient( credential=credential, project_endpoint=os.getenv("AZURE_AI_PROJECT_ENDPOINT"), model=os.getenv("AZURE_AI_MODEL_DEPLOYMENT_NAME") ) product_agent = chat_client.as_agent( name="Product Agent", instructions="You are a helpful assistant that argues for user value and business priority.", default_options={"store": False} # turn off server-side storage, so the client keeps history in the local session and re-sends the full conversation every turn. ) engineering_agent = chat_client.as_agent( name="Engineering Agent", instructions="You are a helpful assistant that raises engineering feasibility, effort, and technical risks.", default_options={"store": False} ) design_agent = chat_client.as_agent( name="Design Agent", instructions="You are a helpful assistant that focuses on UX and usability concerns.", default_options={"store": False} ) security_agent = chat_client.as_agent( name="Security Agent", instructions="You are a helpful assistant that flags compliance and data-protection issues.", default_options={"store": False} ) manager_agent = chat_client.as_agent( name="Manager Agent", instructions=( "You moderate a design-review discussion. Each turn, choose the SINGLE next " "participant to speak from: Product Agent, Engineering Agent, Design Agent, " "Security Agent. Never select the same participant twice in a row. Once every " "perspective has been heard and a clear decision is reached, terminate the " "conversation with a short recommendation." ), default_options={"store": False} ) workflow = GroupChatBuilder( participants=[product_agent, engineering_agent, design_agent, security_agent], orchestrator_agent=manager_agent, max_rounds=5, intermediate_output_from="all", ).build() stream = await workflow.run("Proposal: add biometric login to the mobile app. Should we build it next quarter?", stream=True) last_executor = None async for event in stream: if event.type in ("intermediate", "output") and isinstance(event.data, AgentResponseUpdate): executor = event.data.author_name if executor != last_executor: print(f"\n{'-' * 60}\n{executor}:\n") last_executor = executor print(event.data.text, end="", flush=True) result = await stream.get_final_response() print(f"\n{'=' * 60}\nFinished ({len(result.get_outputs())} output messages).") if __name__ == "__main__": asyncio.run(main())

4. Handoff Orchestration

This orchestration lets agents assign a task to other agents based on their expertise.

Used when:

Multiple agents are involved, but the order of execution is unknown/non-deterministic.

Use Case: Dynamic Customer Support Routing

A Triage Agent greets the customer, identifies the issue, and hands the conversation to the right specialist: a Refund Agent for refunds and returns, or an Order Status Agent for shipping updates. If the customer’s need changes, the specialist hands the conversation back to triage for seamless rerouting.

import os import asyncio from typing import cast from agent_framework import AgentResponseUpdate, Message from agent_framework.foundry import FoundryChatClient from azure.identity import DefaultAzureCredential from agent_framework.orchestrations import HandoffBuilder from dotenv import load_dotenv load_dotenv() async def main(): credential = DefaultAzureCredential() chat_client = FoundryChatClient( credential=credential, project_endpoint=os.getenv("AZURE_AI_PROJECT_ENDPOINT"), model=os.getenv("AZURE_AI_MODEL_DEPLOYMENT_NAME"), ) triage_agent = chat_client.as_agent( name="triage_agent", instructions=( "Greet the customer and briefly acknowledge their issue in one sentence, " "then hand off to the right specialist." ), default_options={"store": False}, require_per_service_call_history_persistence=True ) refund_agent = chat_client.as_agent( name="refund_agent", instructions="You process refunds and returns. Ask for the order number and resolve the request.", default_options={"store": False}, require_per_service_call_history_persistence=True ) order_status_agent = chat_client.as_agent( name="order_status_agent", description="Answers questions about order status and shipping.", instructions="You answer order status and shipping questions.", default_options={"store": False}, require_per_service_call_history_persistence=True ) workflow = ( HandoffBuilder(participants=[triage_agent, refund_agent, order_status_agent]) .with_start_agent(triage_agent) .add_handoff(triage_agent, [refund_agent, order_status_agent]) # triage can route to either .add_handoff(refund_agent, [triage_agent]) # specialists can hand back .add_handoff(order_status_agent, [triage_agent]) .build() ) # Interactive loop: run once, then answer each request_info until it ends. user_message: str | None = "Hi, I was charged twice for order #12345 and want a refund." responses: dict | None = None last_executor = None while True: if responses is not None: stream = workflow.run(responses=responses, stream=True) else: stream = workflow.run(user_message, stream=True) pending_request_id = None async for event in stream: if event.type in ("intermediate", "output") and isinstance(event.data, AgentResponseUpdate): executor = event.data.author_name if executor != last_executor: print(f"\n{'-' * 60}\n{executor}:\n") last_executor = executor print(event.data.text, end="", flush=True) elif event.type == "request_info": pending_request_id = event.request_id # workflow is waiting for the user if pending_request_id is None: break # no input requested -> conversation finished user_text = input("\n\nYou: ") if not user_text.strip(): break # empty answer terminates the handoff workflow responses = {pending_request_id: [Message(role="user", contents=[user_text])]} if __name__ == "__main__": asyncio.run(main())

5. Magentic Orchestration

This orchestration allows for dynamic collaboration of multiple agents. Used when the exact workflow is not known upfront, and for complex open-ended problems.

There is a task ledger which keeps track of the tasks that need to be done. The progress ledger keeps track of the tasks completed and what all was learnt.

The manager agent does the following:

Maintains the overall goal
Creates and updates the task ledger
Chooses which agent should work next
Tracks progress in the progress ledger
Replans when stuck
Synthesizes the final answer

Use Case: Autonomous Blog Drafting and Review

Given a topic, a Magentic manager plans and drives an iterative draft-and-review cycle: it directs a writer agent to produce the post, routes the draft to an editor agent for clarity and length feedback, loops back for revisions when needed, and finalizes the post once it meets the quality bar — all without a human specifying the step order.

import os import asyncio from typing import cast from agent_framework import AgentResponseUpdate from agent_framework.foundry import FoundryChatClient from azure.identity import DefaultAzureCredential from agent_framework.orchestrations import MagenticBuilder from dotenv import load_dotenv load_dotenv() async def main(): credential = DefaultAzureCredential() chat_client = FoundryChatClient( credential=credential, project_endpoint=os.getenv("AZURE_AI_PROJECT_ENDPOINT"), model=os.getenv("AZURE_AI_MODEL_DEPLOYMENT_NAME") ) writer_agent = chat_client.as_agent( name="Writer Agent", instructions="You write clear, engaging blog posts on the requested topic.", default_options={"store": False}, ) editor_agent = chat_client.as_agent( name="Editor Agent", instructions="You review drafts for clarity and length, and suggest concise improvements.", default_options={"store": False}, ) manager_agent = chat_client.as_agent( name="Manager Agent", instructions="You coordinate the writer and editor to produce a polished final blog post.", default_options={"store": False}, ) workflow = MagenticBuilder( participants=[writer_agent, editor_agent], manager_agent=manager_agent, max_stall_count=2, max_round_count=10, intermediate_output_from="all").build() stream = workflow.run( "Write a 300-word blog post explaining why sleep matters for productivity.", stream=True, ) last_executor = None async for event in stream: if event.type in ("intermediate", "output") and isinstance(event.data, AgentResponseUpdate): executor = event.data.author_name if executor != last_executor: print(f"\n{'-' * 60}\n{executor}:\n") last_executor = executor print(event.data.text, end="", flush=True) result = await stream.get_final_response() print(f"\n{'=' * 60}\nFinished ({len(result.get_outputs())} output messages).") if __name__ == "__main__": asyncio.run(main())

Difference Between Group Chat, Handoff, and Magentic Orchestration

	Group Chat	Handoff	Magentic
Who's in charge?	A manager agent picks who speaks next	No manager agent - agents route to each other based on their expertise	The manager agent plans to attain the final goal

Next Steps:

Try it yourself - Clone the GitHub Repo to get started!
Build upon the Magentic Orchestration use case - "Write a well-researched 800-word article on whether biometric login improves security. Verify claims, add real statistics, and include a counter-argument section." Add participants agents such as researcher_agent, fact_checker_agent, writer_agent, and editor_agent

Related Resources:

Reference Used in the Blog (Including images): Introduction - Training | Microsoft Learn
More on Magentic Orchestration: Use Magentic Orchestration - Training | Microsoft Learn

Introducing Real-Time Multichannel Speech-to-Text

JanGoergen — Wed, 05 Aug 2026 18:00:00 GMT

Organizations working with stereo audio often face a difficult choice.

In many contact center, conversational AI, and communication scenarios, participants are already recorded on separate channels. Keeping those channels separate provides valuable conversational context, especially when speakers overlap or interrupt each other.

Yet supporting channel-separated transcription has traditionally required additional complexity. Teams often split audio channels and build parallel transcription pipelines, increasing operational overhead and, in many cases, processing costs. Others merge channels into a single audio stream, simplifying processing but potentially sacrificing transcription quality and speaker attribution.

Today, we're launching Real-Time Multichannel Speech-to-Text in public preview in Azure Speech, part of Microsoft Foundry.

This new capability enables Azure Speech to process stereo audio while preserving channel separation throughout the real-time transcription workflow, making it easier to build contact center intelligence, agent-assist solutions, conversational AI experiences, and analytics applications.

Why channel separation matters

When separate speakers are already recorded on separate channels, that information is valuable.

Consider a customer service call where the customer is recorded on one channel and the agent on another. During natural conversation, participants may speak simultaneously, interrupt one another, or react while the other person is still talking.

A single mixed audio stream can make these situations more challenging to process accurately. By preserving channel boundaries, speech recognition can maintain clearer speaker separation and better preserve conversational structure.

This is why many organizations choose to retain stereo recordings in the first place.

How Real-Time Multichannel Speech-to-Text works

Until now, organizations generally relied on one of two approaches.

The first approach is to split channels and process each independently. This preserves channel separation and recognition quality but requires managing multiple transcription workflows and combining results later.

The second approach is to merge channels into a single stream and transcribe the mixed audio. This simplifies implementation but can reduce the benefits of having separate audio channels, particularly when overlapping speech occurs.

Real-Time Multichannel Speech-to-Text is designed to eliminate this tradeoff.

The concept is straightforward. Instead of requiring separate recognition pipelines for each channel, Azure Speech can process stereo audio directly and return transcription results with channel attribution preserved. Because the channels remain separate throughout recognition, applications can retain valuable conversational structure without building additional processing infrastructure.

The result is a simpler architecture that remains aligned with how the audio was originally captured.

Built into the Speech SDK experience

If you're already using Azure Speech, getting started is intentionally simple.

Real-Time Multichannel Speech-to-Text integrates directly into the existing Speech SDK experience starting with SDK version 1.51.0.

Enabling stereo processing requires only a simple configuration change, allowing you to continue using the same recognition workflows, event handling patterns, and application architecture you're already familiar with.

Here is a complete, copy-paste Python example:

import azure.cognitiveservices.speech as speechsdk import time speech_config = speechsdk.SpeechConfig(subscription="YourSpeechKey", endpoint="YourSpeechEndpoint") speech_config.speech_recognition_language = "en-US" done = False # Enable Multichannel processing speech_config.set_property(speechsdk.PropertyId.Speech_EnableMultiChannelProcessing, "true") audio_config = speechsdk.AudioConfig(filename="stereo.wav") speech_recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config, audio_config=audio_config) # Print Multichannel results def recognized_cb(evt): if evt.result.reason == speechsdk.ResultReason.RecognizedSpeech: print("RECOGNIZED (channel {}): {}".format(evt.result.channel, evt.result.text)) def stop_cb(evt): global done done = True speech_recognizer.recognized.connect(recognized_cb) speech_recognizer.session_stopped.connect(stop_cb) speech_recognizer.canceled.connect(stop_cb) speech_recognizer.start_continuous_recognition() while not done: time.sleep(0.5) speech_recognizer.stop_continuous_recognition()

💡There is no separate service to deploy and no need to create parallel recognition sessions for each channel. Just as importantly, your application continues to receive real-time transcription results while preserving channel information.

Designed for real-world conversational scenarios

Real-Time Multichannel Speech-to-Text is particularly valuable in scenarios where separate audio channels already exist.

Contact centers

Customer and agent audio are commonly captured on separate channels. Preserving those channels can help maintain conversational context while supporting agent assistance, quality monitoring, compliance workflows, and post-call analytics.

Conversational AI

Voice agents and copilots benefit from understanding who said what. Channel-aware transcription provides cleaner conversational structure for downstream AI workflows such as summarization and insight generation.

Communications and collaboration

Applications that capture participants on separate channels can generate richer transcripts while preserving the structure of the original conversation.

Part of the Azure Speech transcription platform

Real-Time Multichannel Speech-to-Text builds on the broader Azure Speech transcription platform, part of Microsoft Foundry.

Organizations can continue using Azure Speech across real-time, fast transcription, and batch transcription workflows while taking advantage of stereo audio processing where appropriate. Individual transcription modes may differ in how they interact with capabilities such as speaker diarization, but they remain part of the same Azure Speech developer experience.

Get started today

Real-Time Multichannel Speech-to-Text is now available in Azure Speech.

If you've been choosing between the complexity of parallel transcription pipelines and the compromises of mixed-audio transcription, this feature offers a new option: preserve stereo channel separation within a single Azure Speech transcription workflow.

We'd love to hear your feedback. Try it in your applications and let us know how it improves your transcription quality. You can reach us through:

Read the Documentation — How to transcribe multichannel audio in real time - Speech service - Foundry Tools | Microsoft Learn
Azure Support — File a support request through the Azure Portal
Stack Overflow — Tag your questions with azure-speech-service
GitHub — Open issues on the Azure AI Speech SDK samples repository

When Does RL Actually Help Fine-Tuning? A Difficulty-Controlled Study on Structured Generation

shihyaolin — Wed, 05 Aug 2026 17:11:08 GMT

The uncomfortable question

Reinforcement learning is often the finishing move of the modern fine-tuning stack: run SFT first, then add RL (GRPO, PPO, DPO) to squeeze out the last few points. In practice the return is wildly inconsistent — sometimes a real jump, sometimes nothing after a burned GPU budget. The folk rule "RL helps when the task is hard" is directionally right but too vague to budget against: it doesn't say how much, which fields, or how to check in advance. This post turns that intuition into a decision rule you can apply before committing to an RL run.

TL;DR. In a controlled text-to-JSON experiment, GRPO added only +0.0111 absolute after SFT on a straightforward formulation where target attributes were explicitly provided. On a harder formulation requiring attributes to be inferred and normalized from prose, the weighted score increased from 0.6983 to 0.7498 — +0.0515 absolute, or approximately +7.4% relative. The largest gains occurred in Semantic-IoU (SIoU)-scored attribute fields, while cosine-scored free-text fields improved less. The practical takeaway is that low post-SFT performance alone is not enough to justify RL: first confirm that SFT has plateaued, that the missing information is recoverable, and that the reward can provide meaningful sequence-level credit. Fully reproducible on public ABO data.

The setup: one task, two difficulties

The task is a large class of enterprise fine-tuning jobs: turn semi-structured product data into a fixed 8-field JSON object. Four fields are free-text editorial descriptions scored by cosine similarity; four are sparse attribute lists (category, color, material, style — surfaced in the schema as inference-type tags like vibe, quick_semantic, trend_signals, and overall_vibe) scored by Semantic-IoU (SIoU) (fuzzy set overlap). The reported metric is the weighted aggregate of these eight per-field scores. GRPO uses the same underlying cosine and SIoU scoring functions as evaluation; during training both arms apply the same reward shaping (threshold annealing and per-field weighting), while final test reporting uses a fixed SIoU threshold (τ = 0.7) and equal field weights.

We use Amazon Berkeley Objects (ABO) (public, CC BY 4.0) as a stand-in for a proprietary product catalog, and build two arms of the same task:

Easy — the target attributes (productType / color / material) are handed to the model in the prompt, so the gold is copyable. SFT can reformat its way to a high score; RL should have little to add.
Hard — those same label lines are hidden in prose (the info still lives in the product name and bullets), so the gold must be inferred and normalized, not copied.

Both arms use the same Qwen3-32B base and the identical SFT-then-GRPO training recipe — same LoRA config, learning rates, sampling settings, reward functions, reward curriculum (threshold annealing), and per-field weighting. The only thing that changes is whether the target attributes are exposed in the prompt or must be inferred from prose. Train and test are disjoint product sets, so the reported gains are measured on held-out items the model did not see in training.

What the two arms look like (same product)

A representative pair makes the difficulty knob concrete — the rows are the same item, differing only in whether the structured attributes are exposed:

Easy — attributes exposed (gold is copyable):

Every gold token already appears in the prompt, so SFT can reformat its way to a high score — RL has little to add.

Hard — the same three label lines are hidden (gold must be inferred):

The attributes still live in the product name ("yellow gold", "earrings"), so the model must locate and normalize them from prose instead of copying a labeled field — and the free-text gold is rephrased so the bullet-copy shortcut is closed too. That is where SFT leaves more headroom and GRPO produces a larger improvement.

The headline result

Task	SFT	SFT → GRPO	Δ absolute	Notes
Easy (attributes exposed)	0.8885	0.8996	+0.0111	small gain after SFT
Hard (attributes hidden)	0.6983	0.7498	+0.0515 (≈ +7.4% rel.)	larger gain; concentrated in SIoU fields

Two things stand out. First, the observed SFT ceiling depends strongly on task formulation: exposing the attributes lets SFT learn a largely copy-and-format transformation, reaching 0.8885; hiding them drops it to 0.6983 because the model can no longer copy. Second, the GRPO gain is much larger on the hard formulation (+0.0515) than the easy one (+0.0111).

The pattern in one line: headroom is necessary but not sufficient — GRPO helped most where SFT had plateaued on recoverable, inference-heavy fields and the task-level reward provided direct semantic credit. In this experiment the size of the GRPO gain was strongly associated with the headroom remaining after SFT, particularly for inference-oriented fields.

Figure 1. Weighted held-out scores before and after GRPO. The easy formulation improves by +0.0111 absolute, while the hard formulation improves by +0.0515 absolute, equivalent to approximately +7.4% relative.

Where the gain lands: it's the field type, not just the score

If the pattern holds, the hard-arm gain should concentrate in the SIoU list fields — the ones that require inference — and touch the already-strong cosine fields less. That is what we observe.

Field	Type	Easy SFT	Easy GRPO	Δ easy	Hard SFT	Hard GRPO	Δ hard
quick_summary	cosine	1.000	1.000	+0.000	0.832	0.856	+0.024
rich_semantic	cosine	0.995	0.995	−0.000	0.788	0.804	+0.016
summary	cosine	0.959	0.962	+0.003	0.890	0.903	+0.013
trend	cosine	0.884	0.894	+0.010	0.819	0.858	+0.039
quick_semantic	SIoU	0.881	0.892	+0.012	0.588	0.694	+0.105
trend_signals	SIoU	0.706	0.724	+0.018	0.471	0.532	+0.061
overall_vibe	SIoU	0.861	0.886	+0.025	0.665	0.717	+0.052
vibe	SIoU	0.823	0.846	+0.023	0.533	0.635	+0.101
weighted_overall		0.8885	0.8996	+0.0111	0.6983	0.7498	+0.0515

Notice the contrast. On the hard arm, the four cosine free-text fields also have headroom (rich_semantic 0.788, trend 0.819), yet GRPO produced smaller and less consistent gains there (e.g., trend +0.039, quick_summary +0.024). The four SIoU inference fields, sitting at 0.47–0.67 after SFT, are where the largest gains land: quick_semantic +0.105, vibe +0.101, trend_signals +0.061, overall_vibe +0.052.

Figure 2. Per-field GRPO-minus-SFT differences. The largest hard-arm improvements occur in SIoU-scored attribute fields, especially quick_semantic and vibe.

Why? Standard SFT optimizes token-level likelihood against one reference output. Although it can generalize beyond exact memorization, the objective does not directly assign sequence-level partial credit based on the downstream SIoU or cosine metric. GRPO instead evaluates sampled outputs with the task-level grader and reinforces higher-scoring generations. This gives the optimizer a direct signal for recovering semantically correct attributes even when the generated sequence differs from the reference wording. In this experiment, GRPO's gains concentrated where the missing information was recoverable from the input but poorly served by token-level imitation — the hard-arm SIoU fields.

A pre-flight check: should you run RL at all?

You don't need to run RL to estimate whether it will help. Because GRPO's gains here concentrated where SFT left recoverable headroom, a short SFT-baseline inspection is informative. But mind the trap: a low score is necessary, not sufficient. The check is three cheap steps:

Always run SFT first. RL isn't a replacement for SFT — it comes after it. So the question is never "SFT vs RL," but "after SFT, is it worth spending more on RL?" SFT is relatively cheap, so do it first.
Score the SFT checkpoint per-field on a held-out test set (not just the aggregate). This is nearly zero-cost — a single inference pass plus grading.
Apply the checklist below to each low-scoring field.

When to consider RL. Consider RL when SFT has genuinely plateaued, important fields remain below target, the missing information is recoverable from the input, sampled outputs can meaningfully vary, and a reliable task-level reward can distinguish partial improvements. In this study that reward is SIoU on attribute lists — a concrete choice for this task, not a universal requirement.

Post-SFT condition	Diagnosis	Recommended action
Aggregate and field scores are already near target	Limited headroom	Keep SFT; skip RL
Important fields remain low and the information is recoverable	Potential reward-credit gap	Consider RL
Scores are low because labels are noisy or unrecoverable	Data or reward problem	Fix data/reward first
SFT validation is still improving	SFT not converged	Continue tuning SFT
Reward does not reliably distinguish better outputs	Weak RL signal	Improve the grader before RL

The point is that a low score alone is not enough. GRPO helped here specifically where the missing information was recoverable and the task-level reward could credit partial progress. A low free-text (cosine) score, a noisy label, and an unconverged SFT are different problems with different fixes, and GRPO should not be assumed to solve them automatically.

Figure 3. Exploratory relationship between post-SFT field score and subsequent GRPO gain. Each point is one field on one arm: x = its post-SFT score (inspectable on your own checkpoint), y = the GRPO gain it received. Color = arm (orange = hard, blue = easy); shape = field type (circle = inference / SIoU, square = free-text / cosine). In this experiment the largest gains occur among low-scoring SIoU attribute fields, while free-text fields generally show smaller gains over the observed score range. (Headroom size alone still correlates with gain, pooled R²≈0.74 — the field-type axis is what a headroom-only reading misses.)

A prerequisite: make sure you’ve actually maxed out SFT

The whole diagnostic assumes the post-SFT score reflects a genuine ceiling, not an under-trained run. Hyperparameters absolutely matter here: if SFT is left under-tuned, its "headroom" is illusory — RL will appear to help when a better-tuned SFT would have closed the same gap for far less compute. So before you read any field as "stranded," confirm SFT has converged:

Train to a plateau, then measure. Push epochs/steps until the held-out score stops improving. A still-rising validation curve means you're measuring an unfinished SFT, not headroom.
Sweep the few hyperparameters that move the needle — learning rate, epochs, and (for LoRA) rank — and keep the best-validation checkpoint. You don't need an exhaustive grid; you need evidence the val metric is no longer sensitive to these.
Sanity-check the copyable fields. Fields whose answer is directly copyable from the input should saturate near 1.0. If even those are low, SFT is under-trained — fix that first; recovering copyable content is SFT's job, not RL's.

In this study both arms share one SFT recipe (Qwen3-32B, LoRA rank 32, lr 5e-4) that we confirmed can saturate: on the easy arm it reaches 0.8885 with many test items at exact 1.0 — direct evidence the recipe isn't leaving easy points unclaimed. Because the identical recipe (SFT, GRPO, and reward shaping) sits under both arms, the difference in GRPO gain is attributable to task difficulty rather than to a change in the training setup

What we can conclude — and what remains open

Shows: in this experiment, GRPO's post-SFT gain was strongly associated with the headroom remaining after SFT, particularly for inference-oriented fields — small on the easy formulation (+0.0111) and larger on the hard one (+0.0515), concentrated in the SIoU attribute fields.

Does not establish: that this holds unchanged for hard-verifiable rewards (math, code) or other task families. The gold is a rule-derived silver label, not a human-preference target. The easy +0.0111 is a small gain, and the reported numbers use a 48-example held-out subset with a single seed, so we do not treat it as a significance test; the hard +0.0515 is larger and lands on the predicted field-level structure, but additional seeds and a larger test set would be needed to quantify uncertainty.

Confounded factors. In this study, inference requirement, output format, and grader type are partly coupled: the four hidden attribute fields are list-valued and scored with SIoU. The experiment identifies a useful empirical pattern but does not fully disentangle these factors. The hard formulation also removes explicit attribute lines and reduces direct lexical overlap with the target output, so it changes more than a single microscopic difficulty knob. The GRPO recipe (its reward curriculum and per-field weights) was tuned on the harder task and applied unchanged to both arms; holding it fixed keeps the arm-to-arm comparison clean, but the recipe is not independently optimized for the easy arm.

What’s next

Extend to hard-verifiable rewards (math-correctness, code-execution) to test whether the observed headroom–gain relationship holds under other reward types — the most important open question.
Explore a predictor for whether the post-SFT per-field distribution can forecast the GRPO gain.
Sweep more difficulty points (a graded reveal-fraction) to trace the full curve.

Reproduce it on public data (CC BY 4.0)

Everything runs on the public Amazon Berkeley Objects dataset (Collins et al., CVPR 2022; abo-listings text metadata only) — no proprietary data, checkpoints, or code required. The study is specified end-to-end below, and because evaluation uses the same cosine and SIoU graders (at a fixed threshold τ = 0.7 and equal field weights), the headline numbers can be verified from a set of model predictions with no fine-tuning access at all; only retraining the models needs the preview.

Get the data. Download the ABO listings metadata (public, no credentials):

aws s3 cp --no-sign-request --recursive s3://amazon-berkeley-objects/listings/ abo-listings/listings/

Build the two arms. Filter to fashion listings and map each product's real attributes (brand, product_type, color, material, style, pattern, bullet points, keywords) into the 8-field editorial JSON — four free-text fields + four attribute-list fields. Build both arms aligned by product_id, differing by the difficulty knob: easy exposes the productType / color / material lines in the prompt (gold is copyable); hard hides those label lines (the words still live in the product name and bullets) and rephrases the free-text gold so the copy shortcut is closed. Split disjointly by product_id (4,800 / 600 / 600 train / val / test). Because of inference and grading cost, the numbers in this post are computed on a fixed 48-example subset of the 600-example test split (seed 42), used for every checkpoint (both arms, SFT and GRPO).
Verify the numbers (no special access). The evaluation grader is fully specified in the Appendix: cosine similarity on the four free-text fields + SIoU (embedding overlap at τ = 0.7, text-embedding-3-small) on the four list fields, averaged with equal weights into weighted_overall. Implement it and score predictions on that fixed 48-example evaluation subset drawn from the held-out test split; you should land within small-sample range of Easy ≈ 0.89, Hard SFT ≈ 0.70 → GRPO ≈ 0.75.
Rerun the training (needs preview access). The fine-tuning itself uses the Interactive Post-Training API (currently in private preview). The base model, SFT configuration, GRPO configuration, and reward shaping are identical across arms; only the data (attributes exposed vs. inferred) changes:

- SFT: Qwen3-32B LoRA r32, lr 5e-4, batch 32, 1 epoch, 4,800 train examples.
- GRPO (from the SFT checkpoint): lr 2e-5, temperature 1.0, group_size 8, groups_per_batch 8, importance-sampling loss, 64 steps, seed 42. Both arms anneal the SIoU threshold 0.5→0.7 and up-weight the inference-type list fields (overall_vibe/trend_signals ×3, vibe ×2, quick_semantic ×1.5).

Score both checkpoints with the same evaluation grader (τ = 0.7, equal weights) and compare the SFT vs GRPO deltas. This is a custom reward on public data, not a preset recipe.

Bottom line

Run and tune SFT first. Then inspect held-out performance by field rather than relying only on an aggregate score. When SFT has saturated on a copy-and-format task, RL may offer little additional value. When important, recoverable attributes remain difficult and a reliable task-level reward can distinguish partial semantic improvements, RL becomes a more promising investment. These results establish that pattern for one structured-generation task; testing whether it generalizes to math, code, preference optimization, and other reward types remains future work.

Appendix — Scoring metrics

This is the evaluation grader used to report all test metrics in this post. Each of the eight fields is scored in [0, 1] by one of two functions, then averaged into weighted_overall. Training additionally applies reward shaping (threshold annealing and per-field weights, the same on both arms); the numbers reported here use the fixed evaluation configuration below (τ = 0.7, equal weights). All embeddings use text-embedding-3-small; emb(·) is the embedding of a normalized (lowercased, whitespace-collapsed) string.

Cosine fields (quick_summary, rich_semantic, summary, trend) — free-text descriptions. Gold and prediction are each reduced to one string, embedded, and compared:

Semantic-IoU fields (quick_semantic, trend_signals, overall_vibe, vibe) — sparse attribute lists. Let G, P be the gold and predicted lists. Embed every item, greedily match pairs whose cosine ≥ τ (neither already matched), and let m be the number of matched pairs (the semantic intersection |G ∩ P|):

This is the Jaccard index |G ∩ P| / |G ∪ P|, but intersection is defined by embedding similarity ≥ τ (τ = 0.7) instead of exact string match — so "sneakers" matches "shoes" without being identical.

Aggregation. With per-field scores sₖ and equal weights (wₖ = 1 over all 8 fields at scoring time):

The denominator is fixed at the full field count (8), so a missing or malformed field contributes 0 to the numerator but still counts in the denominator — the model can't inflate its score by emitting only the easy fields, and output that fails to parse as JSON scores 0.

In one line: cosine fields ask "is the sentence semantically close?"; SIoU fields ask "how much do the two attribute sets overlap, allowing synonyms?" The hard task's headroom lives almost entirely in the second question — which is exactly where GRPO delivers its gain.

References

This work uses the Amazon Berkeley Objects (ABO) dataset (Collins et al., ABO: Dataset and Benchmarks for Real-World 3D Object Understanding, CVPR 2022, arXiv:2110.06199). Data © Amazon.com, used under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). We use only the abo-listings text-metadata subset (not the 3D assets), which we reshaped and adapted into the text-to-JSON task described above.

Disclaimer

© 2026 Microsoft Corporation. All rights reserved. This document is provided "as-is." Information and views expressed in this document, including URL and other internet website references, may change without notice. You bear the risk of using it.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

Some examples are for illustration only and are fictitious. No real association is intended or inferred.

Microsoft's six Responsible AI pillars: how do you operationalize them in practice?

rafaellimaesilva — Tue, 04 Aug 2026 13:00:17 GMT

Deploying AI without a trust criterion isn't innovation. It's risk taken on without awareness.

Microsoft organizes this problem into six pillars: reliability and safety, privacy and security, inclusiveness, transparency, accountability, and fairness. It's not a marketing list. It's a governance structure applicable to any AI adoption decision.

The pillar that generates the most debate in practice, at least in the conversations I've had, is transparency. Systems that can't explain why they reached a given output create a real accountability problem: who signs off on the decision, the model or the manager?

Accountability doesn't transfer to the tool. It stays with whoever decided to use it.

Fairness also tends to be underestimated. Models trained on historical data carry the biases in that data. If the historical record was biased, the model replicates it at scale. Speed amplifies the problem, it doesn't fix it.

Privacy and security are about where the data goes, who accesses it, what gets retained. Questions that need answers before deployment, not after an incident.

The framework doesn't guarantee an outcome. It creates the right questions before any adoption decision.

I'd like to hear how this plays out in practice for you all: which of these six pillars tends to be the hardest to operationalize on your team? And how are you documenting that decision process today?

Securing On-Device AI: Evaluating Foundry Local Models with AI Red Teaming Agent

mhadiputro — Thu, 30 Jul 2026 14:00:00 GMT

On-Device AI Inference Changes the Architecture, Not the Risk

As organizations transform into Frontier Firms and integrate AI into more applications, cloud-hosted inference is not always the optimal architectural choice. Some applications operate with limited connectivity, require low-latency responses, or involve data that organizations prefer to process locally. As AI usage scales, token economics also becomes an architectural consideration as organizations look for greater predictability and control over cloud inference cost. By running AI inference directly on the device, applications can eliminate cloud round trip, support offline operations, avoid cloud token costs, and provide greater control over data during processing.

However, moving AI inference onto the device changes the application architecture, not the model’s exposure to behavioral risks. On-device language models can still encounter adversarial inputs intended to bypass safeguards, manipulate instructions, or elicit unsafe and unexpected responses. This makes systematic security and safety evaluation, including AI red teaming, an important part of adopting on-device AI.

AI red teaming evaluates an AI system from an adversarial perspective to uncover safety and security weaknesses before they are encountered in production. While functional testing asks whether a model works as intended under expected conditions, red teaming examines how it behaves when adversarial user intentionally attempts to make it fail or misbehave.

This article shows how organizations can use Microsoft technology to run AI on end-user devices while systematically evaluate model behavior against safety and security risks. To make this practical, I provide a sample solution that development teams can use to automate repeatable evaluations across models.

Running and Evaluating On-Device AI Security Risks

Microsoft addresses these on-device AI challenges with complementary solutions: Foundry Local runs end-to-end AI inference on the device, while AI Red Teaming Agent helps developers evaluate model behavior before release.

Foundry Local (generally available since April 2026), is a cross-platform runtime for integrating local language models directly into applications. It runs on Windows, macOS, and Linux, with SDKs for Python, JavaScript, C#, and Rust. The runtime manages the model lifecycle within the application: downloading, caching, loading, inference, and unloading.

Foundry Local is designed for single-user inference embedded within an application, rather than multi-user model serving through server-based inference stack. It provides a curated catalog of hardware-optimized models, while allowing developers to extend beyond the catalog using Bring Your Own Model (BYOM) in the ONNX format. To learn more, refer to the Foundry Local site.

AI Red Teaming Agent complements the solution by enabling developers to evaluate the safety and security risks in on-device language models before release. It combines Microsoft’s open-source Python Risk Identification Tool (PyRIT), with Foundry Risk and Safety Evaluations to generate adversarial probes, assess attack-response pairs, calculate metrics such as Attack Success Rate, and produce detailed reports.

The AI Red Teaming Agent supports adversarial testing of cloud models hosted in Microsoft Foundry, while its local scan workflow (in Public Preview) extends this capability to on-device models. Using the azure-ai-evaluation[redteam] package, developers can provide a model configuration or application callback as the scan target. The sample solution that follows uses this callback pattern to evaluate a model running through Foundry Local.

Try It Yourself: Red-Teaming a Foundry Local model

To make this article practical, I provided a sample solution in the foundry-local-eval GitHub repository. The sample solution lets you select and download a model from the Foundry Local curated catalog, runs it entirely on your machine, and evaluate its responses to adversarial prompts using the Azure AI Evaluation red teaming SDK. The Azure Developer CLI (azd) provisions the required Azure resources, helping you move from a fresh clone to a scored risk report seamlessly.

The sample uses a hybrid architecture, in which the target model runs locally through Foundry Local, while the Azure AI Evaluation red teaming SDK uses Microsoft Foundry project to generate adversarial prompts and asses the model’s responses. Model inference remains on the device, while attack generation and risk scoring use Azure services.

The scan probes the model across four risk categories: violence, hate and unfairness, sexual content, and self-harm. It applies a range of attack strategies and reports key measurements: Attack Success Rate (ASR) that you can track across models and tune your safeguards against. For more details, such as prerequisites, configuration, and step-by-step run instructions, refer to the repository.

Final Thoughts: Building On-Device AI with Trust from the Start

As organizations progress toward becoming Frontier Firms, bringing AI into more applications must be accompanied by deliberate governance and evaluation. On-device AI is no exception. In this article, I explored how Microsoft technologies can help organizations deploy AI models to end-user devices and conduct systematic, automated safety and security evaluations through AI red teaming. The sample solution provides a practical reference implementation that development teams can use as a starting point.

By incorporating repeatable red teaming into the development lifecycle, teams can compare results across candidate models, strengthen safeguards before deployment, and treat trust as a design requirement. As applications with on-device AI continue to evolve, this evaluation discipline can help organizations innovate with greater confidence while keeping safety and security central to their Frontier Firm transformation journey.

Introducing GPT-transcribe and GPT-live-transcribe in Microsoft Foundry

Dave_Jacobs — Wed, 29 Jul 2026 23:07:34 GMT

A transcription model hears “account number 8-4-7-2” but returns “account number eighty-four seventy-two.” A single error can break a downstream automation workflow. Developers building voice applications need transcription models that can handle real-world audio conditions, natural speech patterns, and business-critical details, including codes, dates, addresses, account numbers, mixed-language conversations, specialized terminology, and quiet or low-volume speech.

GPT-transcribe and GPT-live-transcribe do just that and are available in Microsoft Foundry today. Two updates to the audio model family designed to improve automatic speech recognition across asynchronous transcription and live streaming scenarios.

Built for More Accurate Transcription in Real-World Audio

GPT-transcribe is the highest accuracy ASR model from Open AI, designed for asynchronous speech-to-text transcription of completed audio files and batch workloads. It accepts audio input and returns text output, making it a strong fit for workflows that process recorded, uploaded, or submitted audio, including meeting recordings, voicemails, and media files.

GPT-live-transcribe is designed for low-latency streaming transcription through the Realtime API. It supports real-time audio input and text output, helping developers build live experiences where speech needs to be transcribed continuously as audio arrives. This model also introduces “tunable latency” where developers can adjust the latency/accuracy trade-off for streaming. It is a strong fit for live captions, voice assistants, contact center workflows, accessibility experiences, field service applications, real-time intake, and monitoring systems.

Together, these models give developers transcription options in Microsoft Foundry for stored audio and live voice interactions. Their text output can support downstream workflows such as search, summarization, routing, analytics, automation, and quality review.

What’s New in Both Models

The features of the new transcription models focus on improving transcription quality in real-world audio environments where speech can be brief, noisy, accented, quiet, domain-specific, or mixed across languages. Key capabilities include:

Background noise: Helps isolate speech in noisy environments so transcription quality can remain more reliable when audio conditions are not controlled.
Short utterances: Improves recognition of brief commands, confirmations, interruptions, and clipped speech that can be difficult to capture accurately.
Alphanumeric perception: Strengthens transcription of IDs, codes, phone numbers, dates, addresses, account numbers, and mixed letter-number sequences.
Domain terminology understanding: Improves recognition of specialized vocabulary used in product, workflow, industry, and business-process contexts.
Codemix: Improves understanding when speakers switch between languages within a conversation or utterance.
Context awareness: Uses topic hints and past conversation context to improve transcription accuracy and help maintain consistency.
Accent robustness: Improves handling of regional accents, non-native accents, dialects, and varied speaking styles.
Whispering: Improves recognition of quiet or low-volume speech, including whispered commands and private dictation.
Live captioning and accessibility experiences: Generate real-time captions for meetings, events, media experiences, and assistive applications.
Contact center and voice workflows: Capture spoken details as conversations happen, supporting routing, quality review, summarization, and downstream automation.
Monitoring, analytics, and compliance workflows: Provide text visibility into ongoing spoken input so teams can analyze, review, and act on conversation data.

Also Available: GPT-realtime-2.1 and GPT-realtime-mini-2.1

gpt-realtime-2.1 and gpt-realtime-mini-2.1 are also available in Microsoft Foundry for developers building speech-to-speech applications. Unlike GPT-transcribe and GPT-live-transcribe, which return text, these models accept audio and generate audio for low-latency conversational experiences over the Realtime API. gpt-realtime-2.1 focuses on interaction quality and robustness, while gpt-realtime-mini-2.1 provides a smaller, faster, and more cost-efficient option for high-volume deployments.

Together with GPT-transcribe and GPT-live-transcribe, these realtime audio updates give developers more flexibility to build voice applications that need both accurate transcription and responsive spoken interaction, whether the experience is centered on capturing speech as text, responding with audio, or combining both patterns in a single workflow.

Use Cases by Model

GPT-transcribe

Use GPT-transcribe when the application needs accurate text transcripts from recorded, uploaded, or submitted audio. It is a strong fit for meeting and call transcription, media transcription, customer support intake, voicemail and message processing, quality review, compliance workflows, and domain-specific transcription where short utterances, structured alphanumeric details, specialized terminology, accents, background noise, code-mixed speech, or quiet audio can affect downstream accuracy.

GPT-live-transcribe

Use GPT-live-transcribe when the application needs live streaming transcription with low latency. It is designed for real-time captions, accessibility experiences, contact center transcription, voice-enabled workflows, live monitoring, operational dashboards, and agent-assist scenarios where spoken input needs to become text continuously as the interaction unfolds.

Pricing

The following pricing example shows Global Standard rates by model and modality. Rates for GPT-realtime-2.1 and GPT-realtime-mini-2.1 are listed per 1 million tokens. GPT-transcribe and GPT-live-transcribe are listed per audio hour.

Model	Deployment	Modality	Input	Cached Input	Output
GPT-realtime-2.1	Global Standard	Audio	$32.00	$0.40	$64.00
		Text	$4.00	$0.40	$24.00
		Image	$5.00	$0.50	--
GPT-realtime-mini-2.1	Global Standard	Audio	$10.00	$0.30	$20.00
		Text	$0.60	$0.06	$2.40
		Image	$0.80	$0.08	--
GPT-live-transcribe	Global Standard	Audio	--	--	$1.02/hour
GPT-transcribe	Global Standard	Audio	--	--	$0.27/hour

Getting Started

Choose GPT-transcribe when your application processes complete audio files asynchronously, or GPT-live-transcribe when it needs text continuously as speech arrives. Try the models in Microsoft Foundry, then use the resources below to explore the Realtime API, follow the audio quickstart, compare available models, and review Azure OpenAI in Foundry Models documentation.

For asynchronous transcription, submit a complete audio file to GPT-transcribe and process the returned transcript after the request completes. This pattern works well for recordings, voicemails, and uploaded media. For streaming transcription, open a Realtime API session with GPT-live-transcribe, send audio as it is captured, and handle incremental transcript events. This pattern supports live captioning and agent-assist experiences that need text during an active interaction. Refer to the linked quickstart and Realtime API documentation for current SDK setup, authentication, request schemas, and supported audio formats.

Explore Microsoft Learn documentation to learn more:

Executive AI Avatar Platform on Azure: Frontier Intelligence Meets Fireworks AI Open-Source

Gana_Chandrasekaran — Wed, 29 Jul 2026 14:00:00 GMT

The 2 a.m. Question Nobody Wants to Answer Twice

It is the night before open enrollment closes. An employee in another time zone has a question about HSA eligibility. No HR representative is available, and the FAQ PDF she received three months ago is nowhere to be found. She does not want to submit a ticket and wait two days—she needs an answer now, delivered by an experience that feels familiar, trustworthy, and connected to her organization.

The Executive AI Avatar Platform was built to close this gap. It provides a branded, always-available AI avatar that can engage through voice or text and respond using the organization’s approved tone, policies, and knowledge. It is more than a chatbot added to a help page. It combines a face, a voice, and an AI-powered conversational experience that can support an employee at 2 a.m. or engage an audience of 10,000 people during a live town hall—without requiring a team to be available around the clock.

That is the promise. The rest of this post explains how the platform is built and why the architecture does not rely on a single frontier model for every task. In an enterprise deployment, cost, control, performance, and flexibility matter just as much as model capability.

1. What Is the Executive AI Avatar Platform?

The Executive AI Avatar Platform is a full-stack conversational AI platform that enables organizations to create branded, conversational digital humans. These photorealistic or custom-trained avatars can represent executives, HR personas, or subject-matter experts and allow employees or customers to interact naturally through text or voice.

The platform uses a Next.js 15 frontend connected to a FastAPI backend, which orchestrates two primary Microsoft capabilities:

Azure Speech in Foundry Tools (Avatar) renders the avatar’s photorealistic, lip-synchronized video and speech in both batch-based scripted experiences and real-time conversational experiences.

Microsoft Foundry provides the deployment platform for the language models that generate the avatar’s responses. The architecture can use a frontier model such as GPT-4o or an open model such as MiniMax M2.5, deployed through Microsoft Foundry and served by Fireworks AI.

┌─────────────────────────────────────────────────────────┐
│ Next.js 15 Frontend   — Library · Studio · Interact     │
└───────────────────┬───────────────────────────────────────┘
                     │ REST / WebSocket
┌───────────────────▼───────────────────────────────────────┐
│ FastAPI Backend — /avatars /synthesis /speech /voice-live │
└───────────────────┬───────────────────────────────────────┘
                     │
        ┌────────────┴─────────────┐
        │                          │
Azure Speech in Foundry Tools Microsoft Foundry
(TTS Avatar — batch (GPT-4o realtime,
   & real-time rendering) MiniMax M2.5 via Fireworks AI, deployed through Microsoft Foundry)

An MCP server using Model Context Protocol, an open standard for connecting AI applications to external tools and data sources also wraps the platform API. This allows MCP-compatible clients, including Claude Desktop, GitHub Copilot in Visual Studio Code, and custom orchestration agents, to trigger avatar video generation without requiring each client to manage Azure credentials directly.

2. Use Cases: Live Interaction vs. Scripted Avatar

The platform supports two distinct interaction modes; each designed for different scenarios.

Live Interaction with Voice Live

Live Interaction enables a real-time spoken conversation. The user speaks naturally, and the avatar listens, processes the request, and responds with synchronized audio and video. Built-in turn detection allows the conversation to flow without requiring the user to press and hold a button to speak.

The experience is powered by the Voice Live API in Azure Speech in Foundry Tools and streamed through a persistent WebSocket connection, enabling low-latency, bidirectional communication between the user and the avatar.

Best suited for: live town halls, executive Q&A sessions, customer-support or help-desk kiosks, and other scenarios where responsiveness and a natural conversational experience are more important than extensive customization of the underlying reasoning model.

Scripted Avatar

This is a text-first interaction mode. The user enters a message or the system provides a predefined script and a language model generates the response. That text is then sent to Azure Speech in Foundry Tools, where text to speech (TTS) produces the avatar’s voice and lip-synchronized video.

The Web Real-Time Communication (WebRTC) session remains open throughout the interaction, so each new message requires only a lightweight text exchange followed by a speakTextAsync call, without establishing a new connection for every turn.

Best for: onboarding walkthroughs, policy explainers, FAQ bots, and scenarios where you want the flexibility to swap the underlying reasoning model based on cost, latency, or data-residency needs.

Model: MiniMax M2.5 via Fireworks AI, deployed through Microsoft Foundry, generates the text response. The response is then sent to Azure Speech in Foundry Tools to produce the avatar’s voice (TTS) and lip synchronization.

3. Models and Azure Services in the Stack

Capability	Service / Model	Used by
Real-time speech-to-speech reasoning	Voice Live API in Azure Speech in Foundry Tools, using GPT-4o or GPT-5 through Microsoft Foundry	Live Interaction tab
Text chat completion (avatar's "brain")	MiniMax M2.5 (FW-MiniMax-M2.5) via Fireworks AI, deployed through Microsoft Foundry	Scripted Avatar tab
Lip-synced video + speech rendering	Avatar in Azure Speech in Foundry Tools for batch and real-time synthesis	Both modes
Authentication	Microsoft Entra ID (DefaultAzureCredential) — no static API keys	All model calls

The key architectural decision is that the reasoning layer is pluggable. The backend exposes a single CHAT_MODEL_PROVIDER setting that routes each scripted-avatar chat completion to one of the supported model providers.

if settings.CHAT_MODEL_PROVIDER == "minimax":

# Open-model path: MiniMax M2.5 via Fireworks AI, # deployed through Microsoft Foundry

    client = get_minimax_client()
    response = await client.chat.completions.create(
        model=settings.MINIMAX_DEPLOYMENT, messages=messages, ...
    )
else:

# Frontier-model path: GPT-4o via Microsoft Foundry

    client = get_client()
    response = await client.chat.completions.create(
        model=settings.AZURE_OPENAI_DEPLOYMENT, messages=messages, ...
    )

Both model providers expose the same OpenAI-compatible chat.completions interface, allowing the rest of the pipeline including persona system prompts, conversation history, and avatar rendering to remain independent of the model generating the response. This interoperability makes the cost-and-performance trade-off described in the next section a simple configuration change rather than an architectural redesign.

4. The Trade-Off: Open Models via Fireworks AI vs. Frontier Models

Why route some workloads to an open model when GPT-4o is available through Microsoft Foundry? In practice, three considerations stand out.

Fast inference

Fireworks AI provides high-throughput, low-latency inference infrastructure for open models. For short, persona-specific responses—such as HR, IT, or policy-related answers—MiniMax M2.5 can provide a strong balance of latency, quality, and cost. Narrowly scoped tasks may not always require the full reasoning capabilities of a larger, general-purpose frontier model.

Fine-tuning and ownership of your own model & Intelligence

This is the strategic differentiator. Frontier models are typically optimized through prompt engineering, system instructions, retrieval, and supported customization capabilities. With an open model deployed through Microsoft Foundry and served by Fireworks AI, organizations can gain additional flexibility to:

Fine-tune the model using approved proprietary conversation data, policy content, or brand-specific examples.
Adapt domain-specific behavior, such as a precise HR-policy tone or specialized customer-support workflow.
Iterate on model behavior without depending entirely on a frontier model provider’s release cycle.
Retain greater control over the customized model and its long-term deployment strategy, subject to the model’s license and platform capabilities.

The open-model path is therefore not simply a lower-cost alternative. It can help organizations build a continuously improving intelligence asset tailored to their data, workflows, and brand.

The Other Side of the Trade-Off

Open models also come with trade-offs. Frontier models such as GPT-4o generally remain stronger for:

Complex, multi-step reasoning and ambiguous instruction following.
Broad knowledge without additional adaptation or grounding.
Advanced multimodal experiences, including the real-time audio interaction used by the Voice Live API.

That is why the platform supports both paths. Rather than choosing one model for every workload, the architecture allows each experience to use the model that best meets its requirements for capability, latency, cost, customization, and control.

The trade-off side

Frontier models such as GPT-4o still offer important advantages, particularly for:

Complex, multi-step reasoning and ambiguous instructions.
Broad knowledge without requiring additional fine-tuning or domain adaptation.
Advanced multimodal experiences, including the real-time audio reasoning used by the Voice Live API in Azure Speech in Foundry Tools—an area where many open models have not yet reached the same level of production readiness.

That is precisely why the platform supports both paths instead of selecting a single model for every workload. This allows each use case to be matched with the right balance of capability, cost, latency, customization, and control.

5. Cost Differences at Scale

Token costs can compound quickly when an avatar platform is deployed across an organization, supporting thousands of daily conversations across HR, IT, onboarding, and other employee-service scenarios. The comparison below uses publicly available list pricing from the Microsoft Foundry pricing page and Fireworks AI’s serverless pricing documentation, verified at the time of writing. Always confirm the latest pricing before building a business case, as rates and service tiers may change.

Model	Input ($ / 1M tokens)	Cached input ($ / 1M tokens)	Output ($ / 1M tokens)
Microsoft Foundry GPT-4o (Global)	$2.50	$1.25	$10.00
Microsoft Foundry GPT-5 (Global)	$1.25	$0.13	$10.00
Fireworks MiniMax (Standard serverless tier)	$0.30	$0.06	$1.20
Fireworks MiniMax (Priority serverless tier)	$0.45	$0.09	$1.80

Worked example — 50M input tokens + 10M output tokens in a month (a realistic volume for an org-wide FAQ/onboarding avatar handling tens of thousands of short conversations):

	Foundry GPT-4o	Foundry GPT-5	MiniMax M2.5 via Fireworks AI, deployed through Microsoft Foundry (Standard)
Input cost (50M tokens)	$125.00	$62.50	$15.00
Output cost (10M tokens)	$100.00	$100.00	$12.00
Total / month	$225.00	$162.50	$27.00
Relative to Foundry GPT-4o	100%	72%	~12%

At this volume, routing scripted-avatar text generation through MiniMax M2.5 via Fireworks AI, deployed through Microsoft Foundry, costs approximately 88% less than using Foundry GPT-4o for the text-generation portion of the pipeline. As usage grows to more than 500 million tokens per month across a large enterprise deployment, the difference can become a meaningful operating-budget consideration.

This comparison does not include avatar-rendering costs in Azure Speech in Foundry Tools, including text-to-speech and video synthesis. These costs apply separately, regardless of which language model generates the response.

The comparison also excludes live spoken sessions powered by the Voice Live API in Azure Speech in Foundry Tools using GPT-4o through Microsoft Foundry. Voice Live is priced separately and is not part of this comparison because MiniMax M2.5 supports the scripted text-generation path rather than the real-time audio interaction path.

Get Started

Everything described in this post—including dual-model routing, WebRTC-based avatar rendering, and MCP server integration—is available as open-source code that you can clone and run today.

GitHub repository:

# Clone the repository

git clone https://github.com/ganachan/ai-executive-avatar-platform.git

cd ai-executive-avatar-platform

# Configure environment variables

cp backend/.env.example backend/.env # Add your Azure resource details

cp frontend/.env.example frontend/.env

# Install and run the FastAPI backend

cd backend

pip install -r requirements.txt

uvicorn main:app --reload --port 8000

# In a second terminal, install and run the Next.js frontend

cd frontend

npm install

npm run dev

Prerequisites:

Node.js 18 or later
Python 3.11 or later
An Azure subscription with Avatar enabled in Azure Speech in Foundry Tools
A Microsoft Foundry resource with GPT-4o and MiniMax M2.5 deployed
Azure CLI authentication configured through az login for Microsoft Entra ID authentication; static API keys are not required by default

Switching the reasoning model for your deployment requires only a one-line configuration change in backend/config.py or backend/.env:

# Route scripted-avatar chat through MiniMax M2.5 via Fireworks AI, # deployed through Microsoft Foundry CHAT_MODEL_PROVIDER=minimax # ...or use GPT-4o via Microsoft Foundry CHAT_MODEL_PROVIDER=azure

Fork the repository, connect it to your own Azure resources, and customize the avatar personas, system prompts, knowledge sources, and fine-tuned models for your organization.

Summary: Open Models as Workflow Partners, Not Replacements

The key lesson from building this platform is not that open models outperform frontier models. It is that both belong in the same workflow, with each model routed to the tasks it handles best.

Use frontier models such as GPT-4o or GPT-5 through Microsoft Foundry when an interaction requires advanced reasoning, multimodal understanding, or real-time speech-to-speech fluency—such as the natural spoken conversations enabled by the Voice Live API in Azure Speech in Foundry Tools.

Use a fine-tuned open model such as MiniMax M2.5 via Fireworks AI, deployed through Microsoft Foundry, for high-volume, persona-specific, and cost-sensitive text interactions. These include FAQ and policy-related experiences in the Scripted Avatar mode, where organizations may also want greater control over model customization and the option to fine-tune using their approved data, subject to licensing and platform capabilities.

Because both model paths use the same OpenAI-compatible interface and are controlled through a single configuration setting, this is not merely a conceptual hybrid architecture. Switching between:

CHAT_MODEL_PROVIDER=minimax

and:

CHAT_MODEL_PROVIDER=azure

makes it possible to compare cost, latency, and response quality side by side using the same avatar, persona, and application workflow.

As avatar and agent platforms scale across the enterprise, this pattern—frontier models for complex interactions and customized open models for high-volume workloads—is likely to become a common architectural approach rather than an exception.

Built using Azure Speech in Foundry Tools, Microsoft Foundry, and Fireworks AI. Have questions about the architecture or want to explore the implementation? Share your questions in the comments or clone the repository linked above and try it with your own Azure resources.

Thank you, Binaka Sankaran, for volunteering to record the source video used to create the avatar and for helping us scale and accelerate this initiative.

Introducing Kimi K3 through Fireworks AI on Microsoft Foundry

davevoutila — Tue, 28 Jul 2026 17:34:17 GMT

Need to process entire codebases, lengthy legal documents, or hours of meeting transcripts in a single prompt? Kimi K3's 1 million-token context window—one of the largest available—is now deployable on Microsoft Foundry through Fireworks AI. Here's what that massive context means for your applications.

What makes Kimi K3 Different

Kimi K3 is an open-weight model from Moonshot AI featuring 2.8 trillion parameters—making it one of the largest open models available. Its 1 million-token context window means you can process approximately 750,000 words (roughly 10 full novels or an entire codebase) in a single request. Fireworks AI provides the optimized inference infrastructure, while Microsoft Foundry handles enterprise deployment and governance.

Use Cases

Full repository code analysis: Analyze entire codebases for security vulnerabilities or refactoring opportunities. For example, you could pass your entire Node.js application (package.json, all source files, and dependencies) in a single prompt and ask Kimi K3 to identify unused imports, potential SQL injection points, or suggest architectural improvements—no chunking or retrieval augmentation required.'
Long-document summarization: Process legal contracts, research papers, or compliance documents without chunking
Extended conversation memory: Build chatbots that remember entire conversation histories

Pricing and Deployment

Model Name	Deployment Type	Input/1M tokens	Output/1M tokens	Cached Input/1M tokens
FW Kimi K3	Data Zone	$3.30	$16.50	$0.33

Customers can deploy Kimi K3 in Microsoft Foundry via Fireworks, subject to applicable availability, deployment, and pricing terms.

Get Started Today

Deploy Kimi K3 through Fireworks in Foundry today.
Compare models: Browse all Foundry Models.
Learn more about Fireworks in Foundry

One agent, three runtimes: porting a CSA agent to Microsoft Scout and Foundry Local

joclemen — Tue, 28 Jul 2026 16:42:03 GMT

Most of my posts here are about Azure infrastructure lessons from customer engagements. This one is a little different — it's a real‑world engineering lesson from something I built to run my own practice.

In my role as a Senior Cloud Solution Architect (CSA), I'm part of a grass-roots organic development team for an internal persona‑driven productivity agent called CSA‑Sherpa. It runs my daily rhythm: a morning briefing, a running logbook of wins and blockers, pipeline and timekeeping summaries, and reporting/exports. It started life in the GitHub Copilot CLI. But over the last few months two things changed the ground under it: Microsoft Scout arrived as a managed cloud agent with native tooling, scheduling, and memory; and Foundry Local made it realistic to run a capable model entirely on‑device on a Copilot+ PC's NPU — no cloud round‑trip at all.

That raised a question I think a lot of people building agents will eventually ask:

If I designed the framework well, can I change how the model runs without rewriting the agent?

To find out, I stood the same agent up in three runtimes, then wrote a whitepaper and a comparison deck measuring what actually changed. This post explains:

How one shared, deterministic core made three very different runtimes comparable
What the three ports — Copilot CLI, Scout‑native, and Foundry Local (on‑device NPU) — actually took
What the analysis showed, and a simple decision framework for which runtime to use when

The part that stayed the same: a deterministic core

The whole exercise only works because all three implementations load the same behavioral core:

Agent definition — persona, behavioral rules, intent routing, workflow dispatch
Instructions — conventions, session bootstrap, change‑management rules
Skill library — one procedure file per workflow (morning briefing, logbook, pipeline, timekeeping, impact, ops, export…)
A deterministic validation contract — schema, formatting, and privacy validators plus a post‑save enforcement chain

That last point is the whole thesis: reliability belongs in code, not in the prompt. Rather than asking the model to "remember" to validate its output, a real gate (a validation step → a post‑save enforcement chain → index regeneration) enforces it every single run. This wasn't my idea in a vacuum — it follows the enterprise prompt‑engineering principles Kathiravan Thangavelu lays out in his article Prompt Engineering for Enterprise AI: Why Reliability Matters: keep deterministic logic in code, prefer schema‑driven / structured output over prompt‑enforced formatting, and replace "before you answer, verify that…" mental checklists with real machine validation. My validation gate is that principle in practice. And because that contract is identical across all three runtimes, I'm comparing three ways to execute one product — not three different products.

The deterministic payoff: faster and cheaper

Retrofitting those principles into the agent — moving work out of the model and into deterministic scripts — is the single change that paid off the most, on two axes at once:

Faster. Letting code (not the model) gather and aggregate history cut the average model round‑trips per workflow from ~8.7 to ~5.5 — roughly a third fewer turns. Fewer turns means less waiting on generation and less back‑and‑forth to finish a task.
Cheaper. The same change cut usage‑based cost ~24% — and, more importantly, held it flat as the logbook grew to hundreds of entries, because scripts carry the history the model used to re‑read every run.

That's the quiet lesson: the reliability work I did for correctness turned out to be the same work that made the agent quicker and less expensive. Determinism isn't a tax on speed — here it bought all three.

The work: three repositories, three runtimes

Everything below the core — runtime, data access, governance, file layout — is where the effort went.

1 · Mainline — Copilot CLI + MCP. The upstream, most feature‑complete build. Runs as a primary agent in the GitHub Copilot CLI on Claude Opus 4.8; data services are discovered through MCP. It carries the heaviest governance: a Spec Kit layer (spec‑driven‑development agents, a constitution + templates, and 50+ per‑feature spec artifacts gated at PR time) plus an add‑on framework. The richest architecture — and the most complex to operate.

2 · Scout‑native. A thin wrapper loads the exact same core onto Microsoft Scout — again on Claude Opus 4.8 — but data access is re‑platformed onto Scout's native tooling instead of MCP subprocesses. No broker to configure; native tools negotiate their own auth. It adds two things the CLI can't do as cleanly:

✅ Scheduled automations — my morning briefing fires automatically on weekday mornings
✅ Cross‑session memory in place of hand‑off files

The deterministic finalize gate stays fully intact.

3 · Foundry Local — on‑device NPU. The genuine outlier and the most involved port: a Python re‑implementation that runs the model — qwen2.5‑7b, an open ~7‑billion‑parameter model — 100% locally on the device's NPU (a Snapdragon X Elite Copilot+ PC) via Foundry Local's OpenAI‑compatible server. The agent loop, an MCP client, skill loading, and a distinct finalize pipeline all had to be rebuilt outside the CLI. The model never leaves the machine; only data connectors reach out when connected. The trade‑offs are real — modest throughput and a fixed context window — but so is the payoff: offline, private, near‑zero marginal cost.

The effort

This wasn't a weekend spike. Across the three code bases (plus a clean isolation clone I kept as an A/B baseline):

~340–380 commits per repository, three versions maintained in parallel
17 skills in each cloud build; 18 in the Foundry port
~37 scripts in the streamlined Scout build, up to ~97 in the governed Mainline build
A Spec Kit governance layer with 50+ feature specs on Mainline
A four‑part cost study and two written deliverables: an architecture whitepaper and a 20‑slide comparison deck

The analysis and reporting

The whitepaper and deck do two jobs. First, they document each runtime as a layered diagram — runtime, core, skills, scripting/validation, external services — so the differences are visible at a glance. Second, they convert the architecture fork into economics: a study that measured the actual token footprints of each repo and priced runs across billing models and hardware. The four dimensions: per‑skill cost, optimized‑vs‑out‑of‑the‑box, Copilot CLI vs Scout, and cloud vs local NPU.

By the numbers

The study priced measured token footprints at frontier‑model rates (treat the dollars as ±30% — the relative conclusions are far more robust than the absolute figures):

Per skill: roughly $0.6–$1.4 per run usage‑based — or a single flat "premium request" under request‑based billing
The determinism dividend: optimized, script‑driven skills cut model round‑trips ~8.7 → ~5.5 and usage‑based cost ~24% — and held cost flat as the logbook grew
Scout vs CLI: Scout ran ~37% cheaper across a five‑command session and consumed none of the premium‑request allowance
Cloud vs local: on‑device NPU inference came in 50–3,400× cheaper in cash than cloud — at the cost of throughput, context, and first‑pass reliability
A full active day (~4 runs) landed around a few dollars usage‑based

The headline isn't any single figure — it's the shape: cloud cents buy first‑pass reliability, on‑device near‑zero cost trades your time, and determinism makes either one cheaper and steadier.

What held up

The core is portable. The same agent, skills, and validation gate ran under all three runtimes. Good separation of concerns paid off.
Determinism pays three ways — faster, cheaper, and more reliable (detailed above). It was the highest‑leverage change I made.
Managed cloud wins the day job. Scout is the best daily driver: reliability gate intact, lower setup friction, scheduling + memory, and cheaper across a multi‑command session because it caches the bootstrap.
On‑device is strategic — but reliability is the tax. Local NPU inference is dramatically cheaper in cash. We ran an in‑depth test pass across every function and closed the gaps it surfaced — yet the smaller model that makes Foundry Local possible still hallucinates and drops instructions often enough on the first pass to matter. Each re‑run is nearly free in dollars, but it costs real time to catch and correct.
The winning pattern is hybrid. Draft and triage locally for ~nothing; escalate the correctness‑critical steps to cloud Opus 4.8, paying only where it buys first‑pass reliability.

Three runtimes, side by side

Figure: Three runtimes, one shared core. Only the top rows — runtime, model, data access, and governance — differ; the behavioral core, skill library, validation gate, and outputs are identical across all three.

Capability	Mainline (Copilot CLI)	Scout‑native	Foundry Local (NPU)
Runtime	Copilot CLI (cloud)	Scout (cloud, managed)	On‑device NPU
Model	Claude Opus 4.8	Claude Opus 4.8	qwen2.5‑7b (open, ~7B)
Data access	MCP	Native tools	MCP via local client
Governance	Spec Kit + PR gate	Behavioral rules	Behavioral rules
Scheduling + memory	❌	✅	❌
Runs fully offline	❌	❌	✅
Marginal cost / run	cloud per‑token	cloud per‑token (cheaper/session)	≈ free
Best for	Framework development	Daily production	Offline / privacy / bulk

When to use each

Daily CSA workflows → Scout‑native. Managed, cheaper across a session, reliable, and it doesn't burn your Copilot request allowance.
Building or versioning the framework → Mainline. Spec Kit governance and the add‑on system earn their keep here.
Offline, air‑gapped, or sensitive data → Foundry Local. 100% on‑device inference.
Bulk / high‑volume / non‑critical → Foundry Local. Zero marginal cost.
Must be right on the first pass → Cloud Opus 4.8. The cents are worth it.
Mixed, cost‑sensitive workload → Hybrid. Local draft → cloud escalate.

Closing Thoughts

The most useful reframe from this work: the three architectures aren't competitors — they're a portfolio. A managed cloud daily‑driver (Scout), a governed development platform (Mainline), and a sovereign on‑device runtime (Foundry Local). The job is to match the runtime to the task, not to crown one winner.

And the same lesson that applies to Azure infrastructure applies to agents: build reliability into the system, not into good intentions. Because CSA‑Sherpa keeps its guarantees in code, I could change the entire execution model underneath it — cloud CLI, managed cloud, on‑device NPU — and the agent still behaved the same way. That portability is the dividend of a deterministic design.

These workflows are genuinely complex, and that's exactly where the small model shows its limits: even after closing the gaps our testing surfaced, it still hallucinates and drops instructions often enough on the first pass to be a real cost. That's the honest trade‑off — near‑zero dollars, paid back in review‑and‑retry time — and it's why my recommendation lands on hybrid: let the small model draft where it's cheap and low‑risk, and escalate anything that has to be right the first time to cloud Opus 4.8.

I use the agent in Microsoft Scout daily, as part of my personal production process.

I did use AI to help draft and format this post — fittingly, the very agent it describes. The architecture, the analysis, and the conclusions are my own.

Thanks for reading.

Microsoft Foundry External MCP Server Traffic Routing via Corporate Firewall

edwinpantony415 — Tue, 28 Jul 2026 15:53:14 GMT

The customer would like to confirm whether traffic from an Azure AI Foundry agent to an external MCP server can be routed through a corporate firewall and whether this scenario is officially supported.

To validate this scenario, I configured the following in my lab:

Deployed an Azure AI Foundry resource using the Standard Agent Service with network injection.
Created a dedicated subnet for the Foundry Agent Service and delegated it to Microsoft.App/environments.
Associated a route table with the Foundry Agent subnet to route outbound traffic through Azure Firewall.
Configured the required application and network rules on Azure Firewall.

The Foundry agent is able to successfully retrieve data from the external MCP server. However, no corresponding traffic is visible in the Azure Firewall logs.

Could you please confirm whether outbound traffic from the Foundry agent to an external MCP server can be routed through Azure Firewall or a corporate firewall? Also, is this routing scenario officially supported?

Appreciate your support!

Claude Opus 5 is available today in Microsoft Foundry

amar_badal — Fri, 24 Jul 2026 17:21:19 GMT

Most AI handles one task well. The harder problem is the complex workflow that spans hours. Effective enterprise AI must be capable of managing complex workflows that require planning, adaptation, and persistence over extended periods.

We're excited to announce that Claude Opus 5, Anthropic's most advanced Opus model and the first Opus model in the fifth generation of Claude, is now available in Microsoft Foundry. Whether you are building agents, coding software, or automating enterprise workflows, Opus 5 is designed to work for hours, find paths around obstacles, navigate codebases like a senior engineer, and support deeper reasoning and higher accuracy. Combined with Microsoft Foundry's enterprise platform including Foundry evaluation tools, native security controls, governance, and scalable deployment, organizations can confidently build and operate the next generation of AI applications.

Navigates codebases like an engineer

Writing a good function isn't the hard part anymore. The hard part is understanding how a change ripples through a large repository, across files, across architecture decisions someone made years ago. Claude Opus 5 is Anthropic's strongest Opus model yet for that kind of work: it navigates large codebases with the judgment of an engineer, adjusts its approach as a project evolves, and can run for hours, even overnight, on a senior engineering task, working around obstacles instead of stalling on them.

Builds long-running agents that can work for hours

Many enterprise workflows cannot be completed in a single interaction. Research, software engineering, document review, and operational automation often require AI systems that can work continuously over multiple steps while responding intelligently to changing conditions.

Claude Opus 5 powers dependable long-running agents, recovering from errors, and reaching their objectives. Opus 5 plans a workflow, adapts the plan when something breaks, and holds context across tools and subagents to see it through. Combined with Microsoft Foundry Agent Service, developers can build reliable agents that automate sophisticated business processes while maintaining enterprise governance and observability.

Transforms enterprise knowledge work

From precise financial workflows to professional work that runs for days, , Opus 5 is built to handle the density of that work with deeper reasoning for long documents, higher accuracy for complex analysis, and raises the bar for the practical tasks that fill a workday. It is built for enterprise work, powering agents that run multi-day projects end-to-end, and follows instructions precisely, stays in scope, and produces professional-grade spreadsheets, slides, and docs.

Brings deeper reasoning to financial workflows

Financial analysis cannot tolerate shortcuts. Opus 5 is built to read a dense filing, hold a chart or table in context alongside it, and carry that thread across an entire transaction or reporting cycle, not just the page in front of it. That's the reasoning investment research, corporate finance, risk analysis, and due diligence work depend on.

Understands documents, charts, and visual information

Enterprise work is increasingly visual, not just text. Critical information lives in scanned PDFs, dashboards, invoices, technical diagrams, and screenshots of systems that may never have been formally documented. Claude Opus 5 brings advanced vision capabilities to the Opus class, enabling AI to accurately understand and reason over visual content rather than simply extract text from it. Combined with Foundry IQ, which connects agents to documentation and process knowledge already embedded across your organization, AI agents can securely access, interpret, and act on enterprise data in context. Together, these capabilities provide the foundation for intelligent document processing, enterprise search, and a new generation of multimodal applications.

Automates work across multiple applications

Claude Opus 5 is also a strong model for computer use. By combining high-resolution vision with deep reasoning, the model can interact with software interfaces and complete workflows that span multiple applications. It can navigate interfaces and execute multi-step tasks requiring planning and judgment. This opens new opportunities for automating repetitive business operations while reducing manual effort.

Enterprise-ready AI with Microsoft Foundry

Claude Opus 5 is Zero Data Retention compatible. Powerful models alone are not enough for production AI. Organizations also need platform capabilities to build, evaluate, secure, and scale enterprise applications.

With Opus 5 in Microsoft Foundry, developers can:

Build AI applications using Azure-native authentication, networking, billing, and governance.

Evaluate model quality and compare prompts, models, and workflows before deployment.

Ground AI applications with enterprise data and knowledge using Microsoft IQ.

Monitor performance and improve agents through continuous evaluation.

Deploy production AI with the security, compliance, and operational controls enterprises expect.

Microsoft Foundry gives organizations the complete platform needed to move from experimentation to production with confidence.

Opus 5 is available now in Microsoft Foundry. Start building: AI Model Catalog | Microsoft Foundry Models

Post-Stream Refinement is now generally available in Microsoft Foundry

SolarRezaei — Thu, 23 Jul 2026 21:29:47 GMT

When we introduced Post-Stream Refinement in public preview earlier this year, it closed the oldest trade-off in real-time speech: you could finally keep instant streaming results and get a highly accurate final transcript, with no penalty to first-token latency. A second recognition pass runs in parallel with streaming and replaces each final segment with a more accurate version once the utterance completes.

Today, Post-Stream Refinement reaches general availability for Azure AI Speech in Microsoft Foundry, backed by a production SLA. Just as important, it now ships with the capabilities production transcription actually depends on: diarization to preserve who said what, phrase lists for your product names and domain vocabulary, and a much wider footprint of 19 locales across 22 Azure regions.

Everything you already know about Post-Stream Refinement still applies. The real-time contract is unchanged, your partial results stream exactly as before, and you enable refinement by setting a single property on your existing SpeechConfig. What changes at GA is that the refined transcript is now production-grade and speaker-aware. 📖 Read the Documentation

What's new at general availability

If you have already used Post-Stream Refinement in preview, here is exactly what changes at GA, and what stays the same. The streaming path and SDK contract are untouched; the refinement pass is now production-ready and gains speaker and vocabulary features.

What's new at GA versus the public preview release. Same real-time streaming contract; production-ready refinement with new speaker and vocabulary features.

How Post-Stream Refinement works

Real-time and final results serve different needs. Partial results must appear quickly so captions, voice interfaces, and agent turn-taking stay responsive. Final results need enough context to support storage, search, summarization, and business workflows. Post-Stream Refinement runs both at once: a fast streaming pass and a deeper refinement pass over the same audio, in parallel.

Both passes run in parallel — partial results stream immediately; final results are refined with broader audio context.

Because the two passes share one input stream, enabling refinement does not require a second transcription job or a separate client pipeline. Your existing recognition events and partial-result handling stay exactly as they are.

Speaker attribution with diarization

New at GA, diarization is supported on the Post-Stream Refinement path, so the refined final transcript keeps its speaker labels. That makes the release a strong fit for meetings, contact centers, interviews, and any workflow where the transcript needs to identify who spoke, not just what was said. The refinement pass improves the wording, including proper nouns and named entities, while every utterance stays attributed to the right speaker.

Phrase lists for your vocabulary

Phrase lists let the recognizer prioritize the names and terms that matter to your application: product catalogs, medical and technical vocabulary, organization names, and acronyms that general speech models might not recognize consistently. At GA you can pair phrase lists with refinement so the second pass has both broad audio context and your domain vocabulary to draw on, which is where the largest accuracy gains on named entities show up.

Quality impact

In internal testing and partner evaluations across supported locales, Post-Stream Refinement reduced final-transcript word error rate by double-digit relative percentages compared with standard real-time transcription, with the largest gains on the hardest content: long utterances, proper nouns, and domain-specific speech. Pairing phrase lists with refinement improves named-entity accuracy further. Partial-result latency is unchanged; only the final transcript is refined.

Relative improvement versus standard real-time transcription. Gains vary by locale, acoustic conditions, and content type.

The refined final result may add a small amount of latency to the final segment because refinement happens after the segment audio is received. Partial results are unaffected.

Supported languages and regions

General availability supports 19 locales. You declare one locale per session, so the service is tuned to the language you expect. Alongside the Tier-1 languages, GA adds Indic locales, including Bengali, Marathi, Punjabi, and Telugu.

19 supported locales at GA. Declare one locale per session.

Post-Stream Refinement is generally available in 22 Azure regions across the Americas, Europe, and Asia Pacific.

Generally available in 22 Azure regions across the Americas, Europe, and Asia Pacific.

Proven at Microsoft scale

The technology behind Post-Stream Refinement already powers meeting transcription and Microsoft 365 Copilot experiences in Microsoft Teams, serving millions of users across meetings, webinars, and live events every day. General availability brings the same quality bar to every Azure AI Speech customer through a supported SDK integration, not a research prototype.

Preview customers across industries, including automotive, consumer electronics, and aviation, reported positive gains in transcription quality, with the clearest improvements on the hardest content: proper nouns, long-form speech, and domain-specific audio. Several are now moving those workloads into production on the GA release.

Get started

Enabling Post-Stream Refinement is a small configuration change on your existing SpeechConfig. You will need:

Speech SDK 1.50 or later. Earlier versions do not support the refinement path.
A Speech resource in one of the supported regions listed above.
The session locale you expect, set on the recognizer.

Set the post-processing option to PostRefinement. The example below also shows the optional phrase list for your domain vocabulary.

import azure.cognitiveservices.speech as speechsdk speech_config = speechsdk.SpeechConfig( subscription="YourSpeechKey", region="YourSpeechRegion") # Declare one locale for the session speech_config.speech_recognition_language = "en-US" # 1) Refine the final transcript (Post-Stream Refinement) speech_config.set_property( speechsdk.PropertyId.SpeechServiceResponse_PostProcessingOption, "PostRefinement") audio_config = speechsdk.AudioConfig(use_default_microphone=True) recognizer = speechsdk.SpeechRecognizer( speech_config=speech_config, audio_config=audio_config) # 2) (Optional) Phrase list for names, acronyms, and domain terms phrase_list = speechsdk.PhraseListGrammar.from_recognizer(recognizer) for term in ["Contoso", "Fabrikam", "Foundry", "OAuth"]: phrase_list.addPhrase(term)

Your existing recognition events and partial-result handling remain unchanged. For speaker attribution, enable diarization through the established real-time diarization path; refinement applies to the final transcript while speaker labels are preserved.

Choose the right release for your workload

Post-Stream Refinement now has two paths. They are the same product family with a different feature boundary, so match the path to what your customer needs.

	Monolingual PSR — generally available	Multilingual PSR — public preview
Language selection	One locale declared per session	Automatic detection and code-switching in a single stream (open-range, no locale declared)
Supported locales	19 locales, including Indic bn / mr / pa / te	25 languages / 29 locales, auto-detected
Azure regions	22 Azure regions across the Americas, Europe, and Asia Pacific	6 Azure regions
Phrase lists & diarization	Supported	Only diarization is supported

Working across languages? If a single stream needs to handle multiple languages or code-switching without a declared locale, use Multilingual Post-Stream Refinement, now in public preview. For a known session locale with phrase lists and diarization, monolingual GA is the right path.

Try Post-Stream Refinement Today

Turn on higher-accuracy, language-aware transcription in your Azure AI Speech applications with a single configuration change.

📖 Read the Documentation

We would love your feedback. Try Post-Stream Refinement in your applications and tell us how it improves your transcription quality.

For the first time, real-time transcription goes multilingual

SolarRezaei — Thu, 23 Jul 2026 21:29:27 GMT

When we introduced Post-Stream Refinement earlier this year, it closed the oldest gap in real-time speech: you could finally get instant streaming results and a highly accurate final transcript, with no latency penalty. But it kept one hard requirement — you had to tell the service, up front, which single language to expect. Real-world speech does not work that way. People code-switch mid-sentence, product and brand names cross languages, and a global app serves users who simply speak differently from one session to the next.

Today we remove that requirement. Multilingual Post-Stream Refinement enters public preview for Azure AI Speech in Microsoft Foundry, and for the first time ever a single real-time stream can transcribe multiple languages in one session — the spoken language is detected automatically, no locale is declared in advance, and the final transcript is refined for accuracy. Everything you already know about Post-Stream Refinement still applies; what changes is that the refinement pass itself is now multilingual. 📖 Read the Documentation

What's New in This Release

If you have already used Post-Stream Refinement, here is exactly what changes with the multilingual preview — and what stays the same:

What's new: the same real-time contract, now multilingual. One multilingual model runs the refinement pass, adding automatic multi-language detection, code-switching within a single utterance, open-range setup with no candidate list, and included diarization — while first-token streaming latency stays unchanged.

Quality Impact

In internal testing and partner evaluations across Tier-1 locales, multilingual Post-Stream Refinement reduced word error rate (WER) by approximately 10% relative on average, with double-digit relative reductions on the hardest cases — long utterances, proper nouns, and multilingual or code-switched speech. Partial-result latency is unchanged; only the final transcript is refined.

Gains are relative reductions versus the standard real-time model and vary by language, acoustic conditions, and content type. The refined final result may add a small amount of latency to the final segment; partial results are unaffected.

Measured quality gains (relative improvement vs. standard real-time transcription): about 10% relative reduction in average word error rate across Tier-1 locales; double-digit relative reduction on long utterances; double-digit relative reduction on proper nouns and named entities; no change to partial-result (first-token) latency. Relative reductions in word error rate; public preview, exact figures vary by locale and audio.

Supported Languages and Regions

The public preview supports 15 Tier-1 locales. Because language is detected automatically, a single stream can contain any mix of them:

Supported languages: 25 auto-detected languages spanning 29 market locales — any mix of which can appear in a single real-time stream without declaring a locale up front.

Available in these Azure regions:

Availability: multilingual Post-Stream Refinement is live in six Azure regions at public preview — East US, West US, North Europe, Central India, Southeast Asia, and Japan East.

Real-World Impact

Preview customers across industries — including travel, consumer electronics, automotive, aviation, and media — have reported positive gains in transcription quality. Customers testing multilingual and domain-specific audio have observed the clearest improvements on the hardest content: proper nouns, code-switching, and long-form speech. Several are actively validating the feature on their own audio ahead of general availability.

Get Started

Enabling multilingual Post-Stream Refinement is a small configuration change on your existing SpeechConfig. You will need:

Speech SDK 1.50 or later. Earlier versions do not support the multilingual path.
A Speech resource in one of the supported regions listed above.
Auto-detect language configuration (open range) so the service identifies the language from the audio — no candidate list required.

Set the post-processing option to PostRefinement and pass an open-range AutoDetectSourceLanguageConfig when you create the recognizer. Here is a complete, copy-paste Python example, including the optional end-of-utterance detection line:

import azure.cognitiveservices.speech as speechsdk speech_config = speechsdk.SpeechConfig( subscription="YourSpeechKey", region="YourSpeechRegion") # 1) Refine the final transcript (Post-Stream Refinement) speech_config.set_property( speechsdk.PropertyId.SpeechServiceResponse_PostProcessingOption, "PostRefinement") # 2) Multilingual auto-detect - no candidate language list needed auto_detect_config = speechsdk.languageconfig.AutoDetectSourceLanguageConfig() audio_config = speechsdk.AudioConfig(use_default_microphone=True) recognizer = speechsdk.SpeechRecognizer( speech_config=speech_config, auto_detect_source_language_config=auto_detect_config, audio_config=audio_config)

💡 Tip: Refinement matters most for applications that store or process the final transcript — meeting notes, call analytics, compliance archives, AI summarization. If you only use partial results for a live display and discard them, your real-time UX (already fast) is unchanged, while any final transcript you keep improves.

Try Multilingual Post-Stream Refinement Today

Turn on higher-accuracy, language-aware transcription in your Azure AI Speech applications with a single configuration change. Available now in public preview in Microsoft Foundry.

📖 Read the Documentation

We would love your feedback. Try Post-Stream Refinement in your applications and tell us how it improves your transcription quality.

Introducing MAI-Image-2.5 Pro and MAI-Voice-2 Flash in Microsoft Foundry

Naomi Moneypenny — Thu, 23 Jul 2026 20:41:12 GMT

One of the core principles in Microsoft Foundry is giving customers flexibility to choose the right model for the right job. As we've continued to expand the Microsoft AI (MAI) model family, we've heard a consistent theme from developers and enterprises: they want more choice based on their specific workload requirements. Some need the highest possible quality, fidelity, and consistency for professional production workflows. Others prioritize responsiveness and cost-efficiency for frequent, real-time responses.

Today, we're introducing two new additions to the MAI portfolio:

MAI-Image-2.5 Pro, designed for customers who need maximum visual fidelity and creative control
MAI-Voice-2 Flash, built for low-latency voice applications where every millisecond matters.

Together, these models extend the MAI family with specialized options that help developers optimize for the jobs they need to do. Let’s dive in.

MAI-Image-2.5 Pro: Built for professional creative workflows

MAI-Image-2.5 Pro is our newest high-fidelity image generation model, designed for scenarios where visual accuracy, consistency, and creative control are critical. The model delivers stronger object consistency, improved alignment with creative intent, and enhanced visual reasoning and world knowledge, making it the ideal choice when quality and fidelity matter more than throughput.

As generative AI moves from experimentation to production, creative teams increasingly need models that can reliably generate assets that meet professional standards without extensive manual editing. MAI-Image-2.5 Pro was built to address those needs. Here are some of the scenarios it best fits:

Campaign Hero Assets & Multi-Frame Storytelling. Creative agencies and marketing teams can create polished campaign visuals while maintaining consistent products, characters, and brand elements across every asset. Whether producing launch campaigns, social media variants, retail signage, or digital advertising, Image-2.5 Pro helps ensure visual consistency throughout the customer journey.
Product Photography & E-Commerce Catalogs. Retailers and consumer brands can generate high-quality product imagery with accurate rendering of packaging, labels, materials, and reflections. The model's improved object consistency helps maintain identical product representation across multiple angles, color variations, and lifestyle settings.
Storyboarding & Pre-Visualization. Film studios, game developers, and creative production teams can rapidly develop visual concepts while maintaining consistency across characters, environments, props, and scenes. Enhanced visual reasoning enables more accurate interpretation of camera direction, lighting, and staging instructions.
Regulated Industry Content Creation. Organizations in healthcare, financial services, manufacturing, and other regulated industries can generate imagery that requires greater real-world accuracy and domain understanding, reducing the effort needed to correct inaccuracies before customer-facing use

When should customers use MAI-Image-2.5 Pro vs. MAI-Image-2.5?

Both models deliver high-quality image generation capabilities, but they are optimized for different priorities.

Choose MAI-Image-2.5 Pro when:

You need maximum image fidelity and creative quality.
Object consistency across multiple images is critical.
Your workflow depends on visual reasoning, world knowledge, and close adherence to creative direction.
You are producing professional marketing, advertising, product design, or enterprise creative assets.

MAI-Voice-2 Flash: Real-time voice experiences at scale

We're also introducing MAI-Voice-2 Flash, a new low-latency text-to-speech model that extends MAI-Voice-2 with faster response times and greater cost efficiency across more than 15 supported languages.

As voice becomes a critical interface for AI applications, responsiveness is increasingly important to the end-user experience. Whether a customer is speaking with an AI-powered support agent, interacting with a voice assistant, or navigating a self-service phone system, long pauses can make experiences feel slow and unnatural. MAI-Voice-2 Flash was built to address those scenarios. Here are some of the scenarios on when to choose MAI-Voice-2 Flash:

Call Center Agents: Customer support organizations can generate spoken responses in real time, minimizing delays between conversation turns and enabling more natural customer interactions. Low latency helps improve customer experiences while supporting AI-powered service, support, and sales workflows.
Conversational Voice Assistants: Developers can build voice-enabled copilots, assistants, and intelligent applications that respond nearly instantly. The result is a more fluid, natural conversation that feels interactive rather than turn-based.
Interactive Voice Response (IVR) Systems: Organizations can modernize traditional phone systems with dynamic AI-generated speech that responds contextually to customer requests while maintaining a responsive user experience.

When should customers use MAI-Voice-2 Flash vs. MAI-Voice-2?

The distinction between the two voice models comes down to whether customers are optimizing for voice identity or real-time responsiveness.

Choose MAI-Voice-2 Flash when:

Low latency is a primary requirement.
You are building conversational assistants, IVR systems, or call center experiences.
Users expect immediate spoken responses as part of a live interaction.
Cost efficiency and responsiveness are more important.

Get started today in Microsoft Foundry

MAI-Image-2.5 Pro is available through Microsoft Foundry, providing developers with access to Microsoft's latest advancements in image generation. Pricing starts at $5 per 1M tokens for text input, and $106 per 1M tokens for image output.

MAI-Voice-2 Flash is available through Azure Speech, allowing customers to leverage Azure Speech's enterprise-grade reliability, scalability, and ecosystem while benefiting from Microsoft's latest voice technology. Pricing starts at $15 per 1M characters.

Grounding Copilot Studio Agents with Azure AI Search and Foundry IQ

BrittanyPugh — Wed, 22 Jul 2026 18:00:00 GMT

An employee opens the HR agent and asks, "How much PTO do I accrue each month?" A few minutes later, someone else asks, "Where is the official code of ethics policy?" Those sound like the same problem. They are not. The first person needs a grounded answer they can understand. The second person needs a link to the right document quickly, without interpretation. If you design for one experience, the other one feels broken.

That is usually where knowledge-agent projects start to get messy. “Grounding” can sound like one switch you turn on, but in practice it is a spectrum: from zero-code classic search, to agentic retrieval over a knowledge base, to a forced-grounding agent that synthesizes answers when synthesis is required. The easier way to think about it is this: who is doing the retrieval work, and what does the user need back?

This post walks through five working retrieval patterns for an “Ask HR” agent built on Copilot Studio, Azure AI Search, and Foundry IQ. Each one is running code in the companion sample repo: foundry-copilot-hr-policy-knowledge. Each has a clear “use this when,” and the five patterns share the same reusable knowledge base so you can layer them on without re-indexing. By the end, you should have a decision tree you can reuse for your own knowledge source, whether that is HR policy, product docs, or support runbooks.

Scope: companion sample for learning and experimentation, not production-ready deployment. Review the Azure Well-Architected Framework for reliability, security, cost, and operational hardening before you ship.

The scenario: one index, many front doors

Here is the setup. The sample answers employee questions from a small corpus of internal HR policy documents: PTO accrual, hiring rules, code of ethics, blood-borne pathogen procedures, and dozens more. Underneath every pattern is one foundation: an Azure AI Search index named hr-policy-index, populated by an indexer and skillset that chunk and vectorize the documents.

Patterns A, C, and the Hosted Agent query that index directly. Patterns A2 and B add a Foundry IQ knowledge base named hr-knowledge-base on top of the same index for agentic retrieval. That layering is the part to pay attention to. The retrieval assets stay separate from the orchestration layer, so you can start with the simplest pattern, prove value quickly, and move to a more capable one later without re-indexing.

Two questions that decide everything

Before we get into the patterns, it helps to define the two retrieval terms I use throughout the rest of the post:

Classic search, index-first retrieval: one hybrid (keyword + vector) query against an Azure AI Search index, ranked and returned. Fast and predictable.
Agentic retrieval, the knowledge base plans multiple sub-queries from the user's question, runs them in parallel, re-ranks, and merges the results before the agent composes an answer. Higher quality on complex, multi-part questions.

If you want the fuller picture of how these two approaches map to retrieval-augmented generation, the Azure AI Search team's RAG and generative AI overview walks through the trade-offs and uses a similar HR/PTO example.

Once those terms are clear, the decision tree comes down to three practical questions:

Q1: Do users need an answer or are they really trying to find the right document?
If they just need the document, stay on the locator path. If they need the policy explained or summarized, move into the answer-synthesis path.

QL: Is the content in a citation-friendly knowledge base?
For example, SharePoint content or Azure AI Search content with a reliable blob_url. If yes, Copilot Studio can usually handle this with native citation cards in Pattern A. If not, use Pattern C with the dual tool /api/lookup path so the agent can return the exact document link.

Q2: Do you actually need an LLM agent in the middle?
If the answer is no, keep it simple: use classic search or agentic retrieval over the knowledge base. If the answer is yes, move into the agent path.

QK: For that non-agent path, is classic index search enough, or do you need agentic KB retrieval?
Classic search points to Pattern A. Agentic retrieval over the knowledge base points to Pattern A2.

Q3: If you need an agent, do you want Foundry to run the request loop, or do you need to self-host it?
If Foundry can manage the runtime, use Pattern B. If you need the request loop in your own container, use the Hosted Agent.

That is the decision tree in plain terms: Q1 decides whether this is a document-locator experience or an answer-synthesis experience. Q2 decides whether you need an LLM agent at all. Q3 is only about where the agent runs, either Foundry or your container. It does not change the front door; Copilot Studio can still be the user-facing experience.

How the sample repo is organized

The repo follows the same flow as the post. Start with docs/DataPipelineAndTesting.md to understand how the HR policy corpus is indexed, tested, and validated. Use docs/RetrievalPatterns.md as the decision model for choosing between classic search, agentic retrieval, forced grounding, and hosted runtime options. Then use the pattern-specific docs when you are ready to wire each path.

For Copilot Studio patterns, docs/CopilotStudioIntegration.md maps to Pattern A, while docs/CopilotStudioHybridExample.md maps to Pattern C and the dual-tool locator flow. For the more advanced agent paths, docs/FoundryAgentArchitecture.md covers Pattern B and the hosted agent architecture. docs/Distribution-M365-Teams.md shows how the agent can be distributed through Microsoft 365 and Teams once the retrieval pattern is working.

The rest of the post is that tree, one branch at a time.

Pattern A: Direct index (classic search, zero agent code)

Start here. Copilot Studio queries hr-policy-index directly through its built-in Knowledge action. No custom agent code runs in the answer path. The sample only owns the index, skillset, and indexing pipeline.

Populate the index (server-side indexer + skillset handles chunking and vectorization):

uv run python scripts/index_knowledge_base_integrated_vectorization.py # Builds hr-policy-index; a client-side alternative exists for dev/test

What you get: very low latency in the sample, roughly 1-2 seconds, no LLM cost in the retrieval path, and native citation cards. When the source documents carry a blob_url or metadata_storage_path, Copilot Studio can render a click-through card straight to the document. For many "where is the policy?" questions, that may be enough.

The honest limitation: Pattern A is still classic search. It does not force synthesis. If Copilot Studio generates an answer from retrieved snippets, it may paraphrase a policy in a way that is close, but not precise enough. For HR policy, that matters. If exact wording matters, that is your sign to step up to Pattern B.

Pattern A2: Copilot Studio meets Foundry IQ (agentic retrieval, no prompt agent)

This is the pattern I would look at when you want better retrieval quality without taking on the overhead of a full prompt agent. In the Copilot Studio new agent experience preview, an agent connects directly to a Foundry IQ knowledge base through Microsoft IQ, with no Foundry prompt agent in between. You reuse the same hr-knowledge-base on top of the same hr-policy-index (one command: python -m src.agents.create_foundry_agent), but retrieval is now agentic: the knowledge base plans sub-queries, retrieves in parallel, reranks, and hands merged results to the agent.

Wiring it takes a few clicks in Copilot Studio (step-by-step on Microsoft Learn):

Build → Microsoft IQ → Foundry IQ → Create new connection
Choose Microsoft Entra ID Integrated authentication
Select hr-knowledge-base
Add to agent

A2 is worth the upgrade from A for two reasons. First, you get agentic-retrieval quality without having to build, deploy, or maintain a prompt agent. The knowledge base becomes the reusable asset you improve in Microsoft Foundry, not something you keep reworking inside each Copilot Studio agent. Second, when configured with Microsoft Entra ID Integrated authentication, retrieval can return ACL-trimmed results per user. Each person sees content based on their access.

Foundry IQ knowledge bases can also inherit enterprise-readiness controls such as customer-managed keys, network isolation, and Entra ID. A single knowledge base can also federate across multiple knowledge sources in parallel.

Use A2 when you want stronger hybrid retrieval quality without taking on the overhead of operating a full agent.

Pattern B: Foundry Agent Service with forced grounding

When answers need to be synthesized and grounded, publish a prompt agent to Microsoft Foundry with Foundry Agent Service. In the sample, the agent uses an MCPTool pointing at the knowledge-base endpoint, with tool_choice="required" so the model retrieves policy chunks before answering.

# src/agents/hr_policy_agent.py (excerpt) agent = PromptAgentDefinition( model=model_deployment_name, # e.g. gpt-5-mini instructions=HR_POLICY_INSTRUCTIONS, tools=[mcp_tool], # KB MCP endpoint tool_choice="required", # require retrieval before answering )

Invoke it through the OpenAI client the project hands you:

client = project.get_openai_client() response = client.responses.create( input="How does PTO accrue for a new hire?", extra_body={"agent_reference": {"name": agent_name}}, )

What you get: synthesized answers with grounding and inline [Policy XXXX - Title] citations, all from a single SDK call on a managed runtime. The trade-off: synthesis takes longer. In the sample, answers take roughly 10-14 seconds versus 1-2 seconds for classic search. For policy explanations, that extra time can be worth it because the user gets a composed, grounded answer instead of a list of snippets.

Pattern C: Dual-tool routing for deterministic document locators

Some questions do not need an essay; they just need the right URL, fast. Pattern C lets Copilot Studio route per turn:

"Where is the PTO policy?" → POST /api/lookup, a deterministic endpoint with no LLM, roughly 1-2 seconds, returning the document URL verbatim in the answer body.
"How many PTO hours do I accrue?" → hand off to Pattern A or B for a synthesized answer.

POST /api/lookup { "query": "PTO policy" } → 200 OK { "policy_id": "12345", "title": "Types of Leave: Paid Time Off (PTO)", "blob_url": "https://.../12345-pto.pdf" }

Reach for Pattern C when native citations are not enough. For example, use it when you need fast locator responses, the URL printed directly in the answer body, deterministic and auditable output, or support for a source that is not citation-friendly. The endpoint lives at src/backend/main.py:/api/lookup, with its contract in copilot/openapi-lookup-v2.json.

Hosted Agent: the same agent on your own runtime

If you need to own the request loop, custom authentication, side-car services, or infrastructure that stays inside your boundary, run the agent yourself. The Hosted Agent is the self-hosted version of the same idea: a container built on Microsoft Agent Framework with FoundryChatClient. It supports both classic and agentic retrieval through one environment variable:

RETRIEVAL_MODE	Strategy	Retrieval type
tool (default)	Custom @tool search_hr_policies (hybrid + semantic)	Classic search
context-semantic	Built-in AzureAISearchContextProvider before each turn	Classic search
context-agentic	AzureAISearchContextProvider over hr-knowledge-base	Agentic retrieval

The context-* modes use Agent Framework’s out-of-the-box RAG context provider. Retrieval runs automatically before each model call with standardized context and citation prompts, so the agent does not have to call a search tool explicitly. That gives the self-hosted path parity with the managed Foundry path across both retrieval types. Copilot Studio can still be the front door. Q3 in the decision tree is really about where the request loop runs, not who greets the user.

Choosing a pattern

Pattern	Orchestrator	Retrieval	Latency (sample)	Best for
A	Copilot Studio	Classic	~1-2 s	Start here, native citations, no agent code
A2	Copilot Studio → Foundry IQ	Agentic	~2-4 s	Agentic quality, no agent to maintain
B	Foundry Agent Service	Classic/agentic via MCP	~10-14 s	Forced-grounding synthesis in Foundry
C	Copilot Studio (router)	None for lookup	~1-2 s	Deterministic, verbatim document locators
Hosted	Agent Framework container	Classic + agentic	~10-14 s	Self-hosted runtime, custom auth

A simple way to read the table: start at A, move to A2 when you want agentic retrieval without operating an agent, choose B when each answer needs to be synthesized and grounded in Foundry, add C for high-volume locator traffic, and pick the Hosted Agent when you need the runtime on your own infrastructure. These are not mutually exclusive. A mature agent often routes locator queries to C and content questions to A2 or B.

What's next?

Try it: clone the sample and follow Steps 1-3 of the walkthrough to stand up Pattern A, provision hr-knowledge-base, connect Copilot Studio, and ask a question in minutes.
Go agentic: wire the same knowledge base into the Copilot Studio new agent experience via Foundry IQ (Pattern A2) and compare answer quality side by side.
Learn more: explore agentic retrieval in Azure AI Search, Foundry IQ, and Microsoft Agent Framework.
Adapt it: swap the HR policy corpus for your own product docs, support runbooks, or internal knowledge source, then compare Pattern A, A2, and B against the same user questions.
Use the repo-doc map: start with docs/RetrievalPatterns.md for the decision model, docs/CopilotStudioIntegration.md for Pattern A, docs/CopilotStudioHybridExample.md for Pattern C, docs/FoundryAgentArchitecture.md for Pattern B and Hosted Agent, and docs/DataPipelineAndTesting.md for ingestion and validation.

My recommendation: start simple, prove the index works, and move up the stack only when the use case needs it. Some questions need a trusted link. Others need a grounded explanation. A strong architecture supports both without forcing every request through the same path.

References

Copilot Studio + Foundry IQ

Foundry IQ / knowledge layer

Azure AI Search, retrieval

Microsoft Foundry Agent Service (Pattern B)

Foundry Agent Service overview

Microsoft Agent Framework (Hosted Agent)

Governance

Azure Well-Architected Framework

Related Microsoft Foundry blog posts