Microsoft Foundry Blog articles

Securing On-Device AI: Evaluating Foundry Local Models with AI Red Teaming Agent

mhadiputro — Thu, 30 Jul 2026 14:00:00 GMT

On-Device AI Inference Changes the Architecture, Not the Risk

As organizations transform into Frontier Firms and integrate AI into more applications, cloud-hosted inference is not always the optimal architectural choice. Some applications operate with limited connectivity, require low-latency responses, or involve data that organizations prefer to process locally. As AI usage scales, token economics also becomes an architectural consideration as organizations look for greater predictability and control over cloud inference cost. By running AI inference directly on the device, applications can eliminate cloud round trip, support offline operations, avoid cloud token costs, and provide greater control over data during processing.

However, moving AI inference onto the device changes the application architecture, not the model’s exposure to behavioral risks. On-device language models can still encounter adversarial inputs intended to bypass safeguards, manipulate instructions, or elicit unsafe and unexpected responses. This makes systematic security and safety evaluation, including AI red teaming, an important part of adopting on-device AI.

AI red teaming evaluates an AI system from an adversarial perspective to uncover safety and security weaknesses before they are encountered in production. While functional testing asks whether a model works as intended under expected conditions, red teaming examines how it behaves when adversarial user intentionally attempts to make it fail or misbehave.

This article shows how organizations can use Microsoft technology to run AI on end-user devices while systematically evaluate model behavior against safety and security risks. To make this practical, I provide a sample solution that development teams can use to automate repeatable evaluations across models.

Running and Evaluating On-Device AI Security Risks

Microsoft addresses these on-device AI challenges with complementary solutions: Foundry Local runs end-to-end AI inference on the device, while AI Red Teaming Agent helps developers evaluate model behavior before release.

Foundry Local (generally available since April 2026), is a cross-platform runtime for integrating local language models directly into applications. It runs on Windows, macOS, and Linux, with SDKs for Python, JavaScript, C#, and Rust. The runtime manages the model lifecycle within the application: downloading, caching, loading, inference, and unloading.

Foundry Local is designed for single-user inference embedded within an application, rather than multi-user model serving through server-based inference stack. It provides a curated catalog of hardware-optimized models, while allowing developers to extend beyond the catalog using Bring Your Own Model (BYOM) in the ONNX format. To learn more, refer to the Foundry Local site.

AI Red Teaming Agent complements the solution by enabling developers to evaluate the safety and security risks in on-device language models before release. It combines Microsoft’s open-source Python Risk Identification Tool (PyRIT), with Foundry Risk and Safety Evaluations to generate adversarial probes, assess attack-response pairs, calculate metrics such as Attack Success Rate, and produce detailed reports.

The AI Red Teaming Agent supports adversarial testing of cloud models hosted in Microsoft Foundry, while its local scan workflow (in Public Preview) extends this capability to on-device models. Using the azure-ai-evaluation[redteam] package, developers can provide a model configuration or application callback as the scan target. The sample solution that follows uses this callback pattern to evaluate a model running through Foundry Local.

Try It Yourself: Red-Teaming a Foundry Local model

To make this article practical, I provided a sample solution in the foundry-local-eval GitHub repository. The sample solution lets you select and download a model from the Foundry Local curated catalog, runs it entirely on your machine, and evaluate its responses to adversarial prompts using the Azure AI Evaluation red teaming SDK. The Azure Developer CLI (azd) provisions the required Azure resources, helping you move from a fresh clone to a scored risk report seamlessly.

The sample uses a hybrid architecture, in which the target model runs locally through Foundry Local, while the Azure AI Evaluation red teaming SDK uses Microsoft Foundry project to generate adversarial prompts and asses the model’s responses. Model inference remains on the device, while attack generation and risk scoring use Azure services.

The scan probes the model across four risk categories: violence, hate and unfairness, sexual content, and self-harm. It applies a range of attack strategies and reports key measurements: Attack Success Rate (ASR) that you can track across models and tune your safeguards against. For more details, such as prerequisites, configuration, and step-by-step run instructions, refer to the repository.

Final Thoughts: Building On-Device AI with Trust from the Start

As organizations progress toward becoming Frontier Firms, bringing AI into more applications must be accompanied by deliberate governance and evaluation. On-device AI is no exception. In this article, I explored how Microsoft technologies can help organizations deploy AI models to end-user devices and conduct systematic, automated safety and security evaluations through AI red teaming. The sample solution provides a practical reference implementation that development teams can use as a starting point.

By incorporating repeatable red teaming into the development lifecycle, teams can compare results across candidate models, strengthen safeguards before deployment, and treat trust as a design requirement. As applications with on-device AI continue to evolve, this evaluation discipline can help organizations innovate with greater confidence while keeping safety and security central to their Frontier Firm transformation journey.

Introducing GPT-transcribe and GPT-live-transcribe in Microsoft Foundry

Dave_Jacobs — Wed, 29 Jul 2026 23:07:34 GMT

A transcription model hears “account number 8-4-7-2” but returns “account number eighty-four seventy-two.” A single error can break a downstream automation workflow. Developers building voice applications need transcription models that can handle real-world audio conditions, natural speech patterns, and business-critical details, including codes, dates, addresses, account numbers, mixed-language conversations, specialized terminology, and quiet or low-volume speech.

GPT-transcribe and GPT-live-transcribe do just that and are available in Microsoft Foundry today. Two updates to the audio model family designed to improve automatic speech recognition across asynchronous transcription and live streaming scenarios.

Built for More Accurate Transcription in Real-World Audio

GPT-transcribe is the highest accuracy ASR model from Open AI, designed for asynchronous speech-to-text transcription of completed audio files and batch workloads. It accepts audio input and returns text output, making it a strong fit for workflows that process recorded, uploaded, or submitted audio, including meeting recordings, voicemails, and media files.

GPT-live-transcribe is designed for low-latency streaming transcription through the Realtime API. It supports real-time audio input and text output, helping developers build live experiences where speech needs to be transcribed continuously as audio arrives. This model also introduces “tunable latency” where developers can adjust the latency/accuracy trade-off for streaming. It is a strong fit for live captions, voice assistants, contact center workflows, accessibility experiences, field service applications, real-time intake, and monitoring systems.

Together, these models give developers transcription options in Microsoft Foundry for stored audio and live voice interactions. Their text output can support downstream workflows such as search, summarization, routing, analytics, automation, and quality review.

What’s New in Both Models

The features of the new transcription models focus on improving transcription quality in real-world audio environments where speech can be brief, noisy, accented, quiet, domain-specific, or mixed across languages. Key capabilities include:

Background noise: Helps isolate speech in noisy environments so transcription quality can remain more reliable when audio conditions are not controlled.
Short utterances: Improves recognition of brief commands, confirmations, interruptions, and clipped speech that can be difficult to capture accurately.
Alphanumeric perception: Strengthens transcription of IDs, codes, phone numbers, dates, addresses, account numbers, and mixed letter-number sequences.
Domain terminology understanding: Improves recognition of specialized vocabulary used in product, workflow, industry, and business-process contexts.
Codemix: Improves understanding when speakers switch between languages within a conversation or utterance.
Context awareness: Uses topic hints and past conversation context to improve transcription accuracy and help maintain consistency.
Accent robustness: Improves handling of regional accents, non-native accents, dialects, and varied speaking styles.
Whispering: Improves recognition of quiet or low-volume speech, including whispered commands and private dictation.
Live captioning and accessibility experiences: Generate real-time captions for meetings, events, media experiences, and assistive applications.
Contact center and voice workflows: Capture spoken details as conversations happen, supporting routing, quality review, summarization, and downstream automation.
Monitoring, analytics, and compliance workflows: Provide text visibility into ongoing spoken input so teams can analyze, review, and act on conversation data.

Also Available: GPT-realtime-2.1 and GPT-realtime-mini-2.1

gpt-realtime-2.1 and gpt-realtime-mini-2.1 are also available in Microsoft Foundry for developers building speech-to-speech applications. Unlike GPT-transcribe and GPT-live-transcribe, which return text, these models accept audio and generate audio for low-latency conversational experiences over the Realtime API. gpt-realtime-2.1 focuses on interaction quality and robustness, while gpt-realtime-mini-2.1 provides a smaller, faster, and more cost-efficient option for high-volume deployments.

Together with GPT-transcribe and GPT-live-transcribe, these realtime audio updates give developers more flexibility to build voice applications that need both accurate transcription and responsive spoken interaction, whether the experience is centered on capturing speech as text, responding with audio, or combining both patterns in a single workflow.

Use Cases by Model

GPT-transcribe

Use GPT-transcribe when the application needs accurate text transcripts from recorded, uploaded, or submitted audio. It is a strong fit for meeting and call transcription, media transcription, customer support intake, voicemail and message processing, quality review, compliance workflows, and domain-specific transcription where short utterances, structured alphanumeric details, specialized terminology, accents, background noise, code-mixed speech, or quiet audio can affect downstream accuracy.

GPT-live-transcribe

Use GPT-live-transcribe when the application needs live streaming transcription with low latency. It is designed for real-time captions, accessibility experiences, contact center transcription, voice-enabled workflows, live monitoring, operational dashboards, and agent-assist scenarios where spoken input needs to become text continuously as the interaction unfolds.

Pricing

The following pricing example shows Global Standard rates by model and modality. Rates for GPT-realtime-2.1 and GPT-realtime-mini-2.1 are listed per 1 million tokens. GPT-transcribe and GPT-live-transcribe are listed per audio hour.

Model	Deployment	Modality	Input	Cached Input	Output
GPT-realtime-2.1	Global Standard	Audio	$32.00	$0.40	$64.00
		Text	$4.00	$0.40	$24.00
		Image	$5.00	$0.50	--
GPT-realtime-mini-2.1	Global Standard	Audio	$10.00	$0.30	$20.00
		Text	$0.60	$0.06	$2.40
		Image	$0.80	$0.08	--
GPT-live-transcribe	Global Standard	Audio	--	--	$1.02/hour
GPT-transcribe	Global Standard	Audio	--	--	$0.27/hour

Getting Started

Choose GPT-transcribe when your application processes complete audio files asynchronously, or GPT-live-transcribe when it needs text continuously as speech arrives. Try the models in Microsoft Foundry, then use the resources below to explore the Realtime API, follow the audio quickstart, compare available models, and review Azure OpenAI in Foundry Models documentation.

For asynchronous transcription, submit a complete audio file to GPT-transcribe and process the returned transcript after the request completes. This pattern works well for recordings, voicemails, and uploaded media. For streaming transcription, open a Realtime API session with GPT-live-transcribe, send audio as it is captured, and handle incremental transcript events. This pattern supports live captioning and agent-assist experiences that need text during an active interaction. Refer to the linked quickstart and Realtime API documentation for current SDK setup, authentication, request schemas, and supported audio formats.

Explore Microsoft Learn documentation to learn more:

Executive AI Avatar Platform on Azure: Frontier Intelligence Meets Fireworks AI Open-Source

Gana_Chandrasekaran — Wed, 29 Jul 2026 14:00:00 GMT

The 2 a.m. Question Nobody Wants to Answer Twice

It is the night before open enrollment closes. An employee in another time zone has a question about HSA eligibility. No HR representative is available, and the FAQ PDF she received three months ago is nowhere to be found. She does not want to submit a ticket and wait two days—she needs an answer now, delivered by an experience that feels familiar, trustworthy, and connected to her organization.

The Executive AI Avatar Platform was built to close this gap. It provides a branded, always-available AI avatar that can engage through voice or text and respond using the organization’s approved tone, policies, and knowledge. It is more than a chatbot added to a help page. It combines a face, a voice, and an AI-powered conversational experience that can support an employee at 2 a.m. or engage an audience of 10,000 people during a live town hall—without requiring a team to be available around the clock.

That is the promise. The rest of this post explains how the platform is built and why the architecture does not rely on a single frontier model for every task. In an enterprise deployment, cost, control, performance, and flexibility matter just as much as model capability.

1. What Is the Executive AI Avatar Platform?

The Executive AI Avatar Platform is a full-stack conversational AI platform that enables organizations to create branded, conversational digital humans. These photorealistic or custom-trained avatars can represent executives, HR personas, or subject-matter experts and allow employees or customers to interact naturally through text or voice.

The platform uses a Next.js 15 frontend connected to a FastAPI backend, which orchestrates two primary Microsoft capabilities:

Azure Speech in Foundry Tools (Avatar) renders the avatar’s photorealistic, lip-synchronized video and speech in both batch-based scripted experiences and real-time conversational experiences.

Microsoft Foundry provides the deployment platform for the language models that generate the avatar’s responses. The architecture can use a frontier model such as GPT-4o or an open model such as MiniMax M2.5, deployed through Microsoft Foundry and served by Fireworks AI.

┌─────────────────────────────────────────────────────────┐
│ Next.js 15 Frontend   — Library · Studio · Interact     │
└───────────────────┬───────────────────────────────────────┘
                     │ REST / WebSocket
┌───────────────────▼───────────────────────────────────────┐
│ FastAPI Backend — /avatars /synthesis /speech /voice-live │
└───────────────────┬───────────────────────────────────────┘
                     │
        ┌────────────┴─────────────┐
        │                          │
Azure Speech in Foundry Tools Microsoft Foundry
(TTS Avatar — batch (GPT-4o realtime,
   & real-time rendering) MiniMax M2.5 via Fireworks AI, deployed through Microsoft Foundry)

An MCP server using Model Context Protocol, an open standard for connecting AI applications to external tools and data sources also wraps the platform API. This allows MCP-compatible clients, including Claude Desktop, GitHub Copilot in Visual Studio Code, and custom orchestration agents, to trigger avatar video generation without requiring each client to manage Azure credentials directly.

2. Use Cases: Live Interaction vs. Scripted Avatar

The platform supports two distinct interaction modes; each designed for different scenarios.

Live Interaction with Voice Live

Live Interaction enables a real-time spoken conversation. The user speaks naturally, and the avatar listens, processes the request, and responds with synchronized audio and video. Built-in turn detection allows the conversation to flow without requiring the user to press and hold a button to speak.

The experience is powered by the Voice Live API in Azure Speech in Foundry Tools and streamed through a persistent WebSocket connection, enabling low-latency, bidirectional communication between the user and the avatar.

Best suited for: live town halls, executive Q&A sessions, customer-support or help-desk kiosks, and other scenarios where responsiveness and a natural conversational experience are more important than extensive customization of the underlying reasoning model.

Scripted Avatar

This is a text-first interaction mode. The user enters a message or the system provides a predefined script and a language model generates the response. That text is then sent to Azure Speech in Foundry Tools, where text to speech (TTS) produces the avatar’s voice and lip-synchronized video.

The Web Real-Time Communication (WebRTC) session remains open throughout the interaction, so each new message requires only a lightweight text exchange followed by a speakTextAsync call, without establishing a new connection for every turn.

Best for: onboarding walkthroughs, policy explainers, FAQ bots, and scenarios where you want the flexibility to swap the underlying reasoning model based on cost, latency, or data-residency needs.

Model: MiniMax M2.5 via Fireworks AI, deployed through Microsoft Foundry, generates the text response. The response is then sent to Azure Speech in Foundry Tools to produce the avatar’s voice (TTS) and lip synchronization.

3. Models and Azure Services in the Stack

Capability	Service / Model	Used by
Real-time speech-to-speech reasoning	Voice Live API in Azure Speech in Foundry Tools, using GPT-4o or GPT-5 through Microsoft Foundry	Live Interaction tab
Text chat completion (avatar's "brain")	MiniMax M2.5 (FW-MiniMax-M2.5) via Fireworks AI, deployed through Microsoft Foundry	Scripted Avatar tab
Lip-synced video + speech rendering	Avatar in Azure Speech in Foundry Tools for batch and real-time synthesis	Both modes
Authentication	Microsoft Entra ID (DefaultAzureCredential) — no static API keys	All model calls

The key architectural decision is that the reasoning layer is pluggable. The backend exposes a single CHAT_MODEL_PROVIDER setting that routes each scripted-avatar chat completion to one of the supported model providers.

if settings.CHAT_MODEL_PROVIDER == "minimax":

# Open-model path: MiniMax M2.5 via Fireworks AI, # deployed through Microsoft Foundry

    client = get_minimax_client()
    response = await client.chat.completions.create(
        model=settings.MINIMAX_DEPLOYMENT, messages=messages, ...
    )
else:

# Frontier-model path: GPT-4o via Microsoft Foundry

    client = get_client()
    response = await client.chat.completions.create(
        model=settings.AZURE_OPENAI_DEPLOYMENT, messages=messages, ...
    )

Both model providers expose the same OpenAI-compatible chat.completions interface, allowing the rest of the pipeline including persona system prompts, conversation history, and avatar rendering to remain independent of the model generating the response. This interoperability makes the cost-and-performance trade-off described in the next section a simple configuration change rather than an architectural redesign.

4. The Trade-Off: Open Models via Fireworks AI vs. Frontier Models

Why route some workloads to an open model when GPT-4o is available through Microsoft Foundry? In practice, three considerations stand out.

Fast inference

Fireworks AI provides high-throughput, low-latency inference infrastructure for open models. For short, persona-specific responses—such as HR, IT, or policy-related answers—MiniMax M2.5 can provide a strong balance of latency, quality, and cost. Narrowly scoped tasks may not always require the full reasoning capabilities of a larger, general-purpose frontier model.

Fine-tuning and ownership of your own model & Intelligence

This is the strategic differentiator. Frontier models are typically optimized through prompt engineering, system instructions, retrieval, and supported customization capabilities. With an open model deployed through Microsoft Foundry and served by Fireworks AI, organizations can gain additional flexibility to:

Fine-tune the model using approved proprietary conversation data, policy content, or brand-specific examples.
Adapt domain-specific behavior, such as a precise HR-policy tone or specialized customer-support workflow.
Iterate on model behavior without depending entirely on a frontier model provider’s release cycle.
Retain greater control over the customized model and its long-term deployment strategy, subject to the model’s license and platform capabilities.

The open-model path is therefore not simply a lower-cost alternative. It can help organizations build a continuously improving intelligence asset tailored to their data, workflows, and brand.

The Other Side of the Trade-Off

Open models also come with trade-offs. Frontier models such as GPT-4o generally remain stronger for:

Complex, multi-step reasoning and ambiguous instruction following.
Broad knowledge without additional adaptation or grounding.
Advanced multimodal experiences, including the real-time audio interaction used by the Voice Live API.

That is why the platform supports both paths. Rather than choosing one model for every workload, the architecture allows each experience to use the model that best meets its requirements for capability, latency, cost, customization, and control.

The trade-off side

Frontier models such as GPT-4o still offer important advantages, particularly for:

Complex, multi-step reasoning and ambiguous instructions.
Broad knowledge without requiring additional fine-tuning or domain adaptation.
Advanced multimodal experiences, including the real-time audio reasoning used by the Voice Live API in Azure Speech in Foundry Tools—an area where many open models have not yet reached the same level of production readiness.

That is precisely why the platform supports both paths instead of selecting a single model for every workload. This allows each use case to be matched with the right balance of capability, cost, latency, customization, and control.

5. Cost Differences at Scale

Token costs can compound quickly when an avatar platform is deployed across an organization, supporting thousands of daily conversations across HR, IT, onboarding, and other employee-service scenarios. The comparison below uses publicly available list pricing from the Microsoft Foundry pricing page and Fireworks AI’s serverless pricing documentation, verified at the time of writing. Always confirm the latest pricing before building a business case, as rates and service tiers may change.

Model	Input ($ / 1M tokens)	Cached input ($ / 1M tokens)	Output ($ / 1M tokens)
Microsoft Foundry GPT-4o (Global)	$2.50	$1.25	$10.00
Microsoft Foundry GPT-5 (Global)	$1.25	$0.13	$10.00
Fireworks MiniMax (Standard serverless tier)	$0.30	$0.06	$1.20
Fireworks MiniMax (Priority serverless tier)	$0.45	$0.09	$1.80

Worked example — 50M input tokens + 10M output tokens in a month (a realistic volume for an org-wide FAQ/onboarding avatar handling tens of thousands of short conversations):

	Foundry GPT-4o	Foundry GPT-5	MiniMax M2.5 via Fireworks AI, deployed through Microsoft Foundry (Standard)
Input cost (50M tokens)	$125.00	$62.50	$15.00
Output cost (10M tokens)	$100.00	$100.00	$12.00
Total / month	$225.00	$162.50	$27.00
Relative to Foundry GPT-4o	100%	72%	~12%

At this volume, routing scripted-avatar text generation through MiniMax M2.5 via Fireworks AI, deployed through Microsoft Foundry, costs approximately 88% less than using Foundry GPT-4o for the text-generation portion of the pipeline. As usage grows to more than 500 million tokens per month across a large enterprise deployment, the difference can become a meaningful operating-budget consideration.

This comparison does not include avatar-rendering costs in Azure Speech in Foundry Tools, including text-to-speech and video synthesis. These costs apply separately, regardless of which language model generates the response.

The comparison also excludes live spoken sessions powered by the Voice Live API in Azure Speech in Foundry Tools using GPT-4o through Microsoft Foundry. Voice Live is priced separately and is not part of this comparison because MiniMax M2.5 supports the scripted text-generation path rather than the real-time audio interaction path.

Get Started

Everything described in this post—including dual-model routing, WebRTC-based avatar rendering, and MCP server integration—is available as open-source code that you can clone and run today.

GitHub repository:

# Clone the repository

git clone https://github.com/ganachan/ai-executive-avatar-platform.git

cd ai-executive-avatar-platform

# Configure environment variables

cp backend/.env.example backend/.env # Add your Azure resource details

cp frontend/.env.example frontend/.env

# Install and run the FastAPI backend

cd backend

pip install -r requirements.txt

uvicorn main:app --reload --port 8000

# In a second terminal, install and run the Next.js frontend

cd frontend

npm install

npm run dev

Prerequisites:

Node.js 18 or later
Python 3.11 or later
An Azure subscription with Avatar enabled in Azure Speech in Foundry Tools
A Microsoft Foundry resource with GPT-4o and MiniMax M2.5 deployed
Azure CLI authentication configured through az login for Microsoft Entra ID authentication; static API keys are not required by default

Switching the reasoning model for your deployment requires only a one-line configuration change in backend/config.py or backend/.env:

# Route scripted-avatar chat through MiniMax M2.5 via Fireworks AI, # deployed through Microsoft Foundry CHAT_MODEL_PROVIDER=minimax # ...or use GPT-4o via Microsoft Foundry CHAT_MODEL_PROVIDER=azure

Fork the repository, connect it to your own Azure resources, and customize the avatar personas, system prompts, knowledge sources, and fine-tuned models for your organization.

Summary: Open Models as Workflow Partners, Not Replacements

The key lesson from building this platform is not that open models outperform frontier models. It is that both belong in the same workflow, with each model routed to the tasks it handles best.

Use frontier models such as GPT-4o or GPT-5 through Microsoft Foundry when an interaction requires advanced reasoning, multimodal understanding, or real-time speech-to-speech fluency—such as the natural spoken conversations enabled by the Voice Live API in Azure Speech in Foundry Tools.

Use a fine-tuned open model such as MiniMax M2.5 via Fireworks AI, deployed through Microsoft Foundry, for high-volume, persona-specific, and cost-sensitive text interactions. These include FAQ and policy-related experiences in the Scripted Avatar mode, where organizations may also want greater control over model customization and the option to fine-tune using their approved data, subject to licensing and platform capabilities.

Because both model paths use the same OpenAI-compatible interface and are controlled through a single configuration setting, this is not merely a conceptual hybrid architecture. Switching between:

CHAT_MODEL_PROVIDER=minimax

and:

CHAT_MODEL_PROVIDER=azure

makes it possible to compare cost, latency, and response quality side by side using the same avatar, persona, and application workflow.

As avatar and agent platforms scale across the enterprise, this pattern—frontier models for complex interactions and customized open models for high-volume workloads—is likely to become a common architectural approach rather than an exception.

Built using Azure Speech in Foundry Tools, Microsoft Foundry, and Fireworks AI. Have questions about the architecture or want to explore the implementation? Share your questions in the comments or clone the repository linked above and try it with your own Azure resources.

Thank you, Binaka Sankaran, for volunteering to record the source video used to create the avatar and for helping us scale and accelerate this initiative.

Introducing Kimi K3 through Fireworks AI on Microsoft Foundry

davevoutila — Tue, 28 Jul 2026 17:34:17 GMT

Need to process entire codebases, lengthy legal documents, or hours of meeting transcripts in a single prompt? Kimi K3's 1 million-token context window—one of the largest available—is now deployable on Microsoft Foundry through Fireworks AI. Here's what that massive context means for your applications.

What makes Kimi K3 Different

Kimi K3 is an open-weight model from Moonshot AI featuring 2.8 trillion parameters—making it one of the largest open models available. Its 1 million-token context window means you can process approximately 750,000 words (roughly 10 full novels or an entire codebase) in a single request. Fireworks AI provides the optimized inference infrastructure, while Microsoft Foundry handles enterprise deployment and governance.

Use Cases

Full repository code analysis: Analyze entire codebases for security vulnerabilities or refactoring opportunities. For example, you could pass your entire Node.js application (package.json, all source files, and dependencies) in a single prompt and ask Kimi K3 to identify unused imports, potential SQL injection points, or suggest architectural improvements—no chunking or retrieval augmentation required.'
Long-document summarization: Process legal contracts, research papers, or compliance documents without chunking
Extended conversation memory: Build chatbots that remember entire conversation histories

Pricing and Deployment

Model Name	Deployment Type	Input/1M tokens	Output/1M tokens	Cached Input/1M tokens
FW Kimi K3	Data Zone	$3.30	$16.50	$0.33

Customers can deploy Kimi K3 in Microsoft Foundry via Fireworks, subject to applicable availability, deployment, and pricing terms.

Get Started Today

Deploy Kimi K3 through Fireworks in Foundry today.
Compare models: Browse all Foundry Models.
Learn more about Fireworks in Foundry

Claude Opus 5 is available today in Microsoft Foundry

amar_badal — Fri, 24 Jul 2026 17:21:19 GMT

Most AI handles one task well. The harder problem is the complex workflow that spans hours. Effective enterprise AI must be capable of managing complex workflows that require planning, adaptation, and persistence over extended periods.

We're excited to announce that Claude Opus 5, Anthropic's most advanced Opus model and the first Opus model in the fifth generation of Claude, is now available in Microsoft Foundry. Whether you are building agents, coding software, or automating enterprise workflows, Opus 5 is designed to work for hours, find paths around obstacles, navigate codebases like a senior engineer, and support deeper reasoning and higher accuracy. Combined with Microsoft Foundry's enterprise platform including Foundry evaluation tools, native security controls, governance, and scalable deployment, organizations can confidently build and operate the next generation of AI applications.

Navigates codebases like an engineer

Writing a good function isn't the hard part anymore. The hard part is understanding how a change ripples through a large repository, across files, across architecture decisions someone made years ago. Claude Opus 5 is Anthropic's strongest Opus model yet for that kind of work: it navigates large codebases with the judgment of an engineer, adjusts its approach as a project evolves, and can run for hours, even overnight, on a senior engineering task, working around obstacles instead of stalling on them.

Builds long-running agents that can work for hours

Many enterprise workflows cannot be completed in a single interaction. Research, software engineering, document review, and operational automation often require AI systems that can work continuously over multiple steps while responding intelligently to changing conditions.

Claude Opus 5 powers dependable long-running agents, recovering from errors, and reaching their objectives. Opus 5 plans a workflow, adapts the plan when something breaks, and holds context across tools and subagents to see it through. Combined with Microsoft Foundry Agent Service, developers can build reliable agents that automate sophisticated business processes while maintaining enterprise governance and observability.

Transforms enterprise knowledge work

From precise financial workflows to professional work that runs for days, , Opus 5 is built to handle the density of that work with deeper reasoning for long documents, higher accuracy for complex analysis, and raises the bar for the practical tasks that fill a workday. It is built for enterprise work, powering agents that run multi-day projects end-to-end, and follows instructions precisely, stays in scope, and produces professional-grade spreadsheets, slides, and docs.

Brings deeper reasoning to financial workflows

Financial analysis cannot tolerate shortcuts. Opus 5 is built to read a dense filing, hold a chart or table in context alongside it, and carry that thread across an entire transaction or reporting cycle, not just the page in front of it. That's the reasoning investment research, corporate finance, risk analysis, and due diligence work depend on.

Understands documents, charts, and visual information

Enterprise work is increasingly visual, not just text. Critical information lives in scanned PDFs, dashboards, invoices, technical diagrams, and screenshots of systems that may never have been formally documented. Claude Opus 5 brings advanced vision capabilities to the Opus class, enabling AI to accurately understand and reason over visual content rather than simply extract text from it. Combined with Foundry IQ, which connects agents to documentation and process knowledge already embedded across your organization, AI agents can securely access, interpret, and act on enterprise data in context. Together, these capabilities provide the foundation for intelligent document processing, enterprise search, and a new generation of multimodal applications.

Automates work across multiple applications

Claude Opus 5 is also a strong model for computer use. By combining high-resolution vision with deep reasoning, the model can interact with software interfaces and complete workflows that span multiple applications. It can navigate interfaces and execute multi-step tasks requiring planning and judgment. This opens new opportunities for automating repetitive business operations while reducing manual effort.

Enterprise-ready AI with Microsoft Foundry

Claude Opus 5 is Zero Data Retention compatible. Powerful models alone are not enough for production AI. Organizations also need platform capabilities to build, evaluate, secure, and scale enterprise applications.

With Opus 5 in Microsoft Foundry, developers can:

Build AI applications using Azure-native authentication, networking, billing, and governance.

Evaluate model quality and compare prompts, models, and workflows before deployment.

Ground AI applications with enterprise data and knowledge using Microsoft IQ.

Monitor performance and improve agents through continuous evaluation.

Deploy production AI with the security, compliance, and operational controls enterprises expect.

Microsoft Foundry gives organizations the complete platform needed to move from experimentation to production with confidence.

Opus 5 is available now in Microsoft Foundry. Start building: AI Model Catalog | Microsoft Foundry Models

Post-Stream Refinement is now generally available in Microsoft Foundry

SolarRezaei — Thu, 23 Jul 2026 21:29:47 GMT

When we introduced Post-Stream Refinement in public preview earlier this year, it closed the oldest trade-off in real-time speech: you could finally keep instant streaming results and get a highly accurate final transcript, with no penalty to first-token latency. A second recognition pass runs in parallel with streaming and replaces each final segment with a more accurate version once the utterance completes.

Today, Post-Stream Refinement reaches general availability for Azure AI Speech in Microsoft Foundry, backed by a production SLA. Just as important, it now ships with the capabilities production transcription actually depends on: diarization to preserve who said what, phrase lists for your product names and domain vocabulary, and a much wider footprint of 19 locales across 22 Azure regions.

Everything you already know about Post-Stream Refinement still applies. The real-time contract is unchanged, your partial results stream exactly as before, and you enable refinement by setting a single property on your existing SpeechConfig. What changes at GA is that the refined transcript is now production-grade and speaker-aware. 📖 Read the Documentation

What's new at general availability

If you have already used Post-Stream Refinement in preview, here is exactly what changes at GA, and what stays the same. The streaming path and SDK contract are untouched; the refinement pass is now production-ready and gains speaker and vocabulary features.

What's new at GA versus the public preview release. Same real-time streaming contract; production-ready refinement with new speaker and vocabulary features.

How Post-Stream Refinement works

Real-time and final results serve different needs. Partial results must appear quickly so captions, voice interfaces, and agent turn-taking stay responsive. Final results need enough context to support storage, search, summarization, and business workflows. Post-Stream Refinement runs both at once: a fast streaming pass and a deeper refinement pass over the same audio, in parallel.

Both passes run in parallel — partial results stream immediately; final results are refined with broader audio context.

Because the two passes share one input stream, enabling refinement does not require a second transcription job or a separate client pipeline. Your existing recognition events and partial-result handling stay exactly as they are.

Speaker attribution with diarization

New at GA, diarization is supported on the Post-Stream Refinement path, so the refined final transcript keeps its speaker labels. That makes the release a strong fit for meetings, contact centers, interviews, and any workflow where the transcript needs to identify who spoke, not just what was said. The refinement pass improves the wording, including proper nouns and named entities, while every utterance stays attributed to the right speaker.

Phrase lists for your vocabulary

Phrase lists let the recognizer prioritize the names and terms that matter to your application: product catalogs, medical and technical vocabulary, organization names, and acronyms that general speech models might not recognize consistently. At GA you can pair phrase lists with refinement so the second pass has both broad audio context and your domain vocabulary to draw on, which is where the largest accuracy gains on named entities show up.

Quality impact

In internal testing and partner evaluations across supported locales, Post-Stream Refinement reduced final-transcript word error rate by double-digit relative percentages compared with standard real-time transcription, with the largest gains on the hardest content: long utterances, proper nouns, and domain-specific speech. Pairing phrase lists with refinement improves named-entity accuracy further. Partial-result latency is unchanged; only the final transcript is refined.

Relative improvement versus standard real-time transcription. Gains vary by locale, acoustic conditions, and content type.

The refined final result may add a small amount of latency to the final segment because refinement happens after the segment audio is received. Partial results are unaffected.

Supported languages and regions

General availability supports 19 locales. You declare one locale per session, so the service is tuned to the language you expect. Alongside the Tier-1 languages, GA adds Indic locales, including Bengali, Marathi, Punjabi, and Telugu.

19 supported locales at GA. Declare one locale per session.

Post-Stream Refinement is generally available in 22 Azure regions across the Americas, Europe, and Asia Pacific.

Generally available in 22 Azure regions across the Americas, Europe, and Asia Pacific.

Proven at Microsoft scale

The technology behind Post-Stream Refinement already powers meeting transcription and Microsoft 365 Copilot experiences in Microsoft Teams, serving millions of users across meetings, webinars, and live events every day. General availability brings the same quality bar to every Azure AI Speech customer through a supported SDK integration, not a research prototype.

Preview customers across industries, including automotive, consumer electronics, and aviation, reported positive gains in transcription quality, with the clearest improvements on the hardest content: proper nouns, long-form speech, and domain-specific audio. Several are now moving those workloads into production on the GA release.

Get started

Enabling Post-Stream Refinement is a small configuration change on your existing SpeechConfig. You will need:

Speech SDK 1.50 or later. Earlier versions do not support the refinement path.
A Speech resource in one of the supported regions listed above.
The session locale you expect, set on the recognizer.

Set the post-processing option to PostRefinement. The example below also shows the optional phrase list for your domain vocabulary.

import azure.cognitiveservices.speech as speechsdk speech_config = speechsdk.SpeechConfig( subscription="YourSpeechKey", region="YourSpeechRegion") # Declare one locale for the session speech_config.speech_recognition_language = "en-US" # 1) Refine the final transcript (Post-Stream Refinement) speech_config.set_property( speechsdk.PropertyId.SpeechServiceResponse_PostProcessingOption, "PostRefinement") audio_config = speechsdk.AudioConfig(use_default_microphone=True) recognizer = speechsdk.SpeechRecognizer( speech_config=speech_config, audio_config=audio_config) # 2) (Optional) Phrase list for names, acronyms, and domain terms phrase_list = speechsdk.PhraseListGrammar.from_recognizer(recognizer) for term in ["Contoso", "Fabrikam", "Foundry", "OAuth"]: phrase_list.addPhrase(term)

Your existing recognition events and partial-result handling remain unchanged. For speaker attribution, enable diarization through the established real-time diarization path; refinement applies to the final transcript while speaker labels are preserved.

Choose the right release for your workload

Post-Stream Refinement now has two paths. They are the same product family with a different feature boundary, so match the path to what your customer needs.

	Monolingual PSR — generally available	Multilingual PSR — public preview
Language selection	One locale declared per session	Automatic detection and code-switching in a single stream (open-range, no locale declared)
Supported locales	19 locales, including Indic bn / mr / pa / te	25 languages / 29 locales, auto-detected
Azure regions	22 Azure regions across the Americas, Europe, and Asia Pacific	6 Azure regions
Phrase lists & diarization	Supported	Only diarization is supported

Working across languages? If a single stream needs to handle multiple languages or code-switching without a declared locale, use Multilingual Post-Stream Refinement, now in public preview. For a known session locale with phrase lists and diarization, monolingual GA is the right path.

Try Post-Stream Refinement Today

Turn on higher-accuracy, language-aware transcription in your Azure AI Speech applications with a single configuration change.

📖 Read the Documentation

We would love your feedback. Try Post-Stream Refinement in your applications and tell us how it improves your transcription quality.

For the first time, real-time transcription goes multilingual

SolarRezaei — Thu, 23 Jul 2026 21:29:27 GMT

When we introduced Post-Stream Refinement earlier this year, it closed the oldest gap in real-time speech: you could finally get instant streaming results and a highly accurate final transcript, with no latency penalty. But it kept one hard requirement — you had to tell the service, up front, which single language to expect. Real-world speech does not work that way. People code-switch mid-sentence, product and brand names cross languages, and a global app serves users who simply speak differently from one session to the next.

Today we remove that requirement. Multilingual Post-Stream Refinement enters public preview for Azure AI Speech in Microsoft Foundry, and for the first time ever a single real-time stream can transcribe multiple languages in one session — the spoken language is detected automatically, no locale is declared in advance, and the final transcript is refined for accuracy. Everything you already know about Post-Stream Refinement still applies; what changes is that the refinement pass itself is now multilingual. 📖 Read the Documentation

What's New in This Release

If you have already used Post-Stream Refinement, here is exactly what changes with the multilingual preview — and what stays the same:

What's new: the same real-time contract, now multilingual. One multilingual model runs the refinement pass, adding automatic multi-language detection, code-switching within a single utterance, open-range setup with no candidate list, and included diarization — while first-token streaming latency stays unchanged.

Quality Impact

In internal testing and partner evaluations across Tier-1 locales, multilingual Post-Stream Refinement reduced word error rate (WER) by approximately 10% relative on average, with double-digit relative reductions on the hardest cases — long utterances, proper nouns, and multilingual or code-switched speech. Partial-result latency is unchanged; only the final transcript is refined.

Gains are relative reductions versus the standard real-time model and vary by language, acoustic conditions, and content type. The refined final result may add a small amount of latency to the final segment; partial results are unaffected.

Measured quality gains (relative improvement vs. standard real-time transcription): about 10% relative reduction in average word error rate across Tier-1 locales; double-digit relative reduction on long utterances; double-digit relative reduction on proper nouns and named entities; no change to partial-result (first-token) latency. Relative reductions in word error rate; public preview, exact figures vary by locale and audio.

Supported Languages and Regions

The public preview supports 15 Tier-1 locales. Because language is detected automatically, a single stream can contain any mix of them:

Supported languages: 25 auto-detected languages spanning 29 market locales — any mix of which can appear in a single real-time stream without declaring a locale up front.

Available in these Azure regions:

Availability: multilingual Post-Stream Refinement is live in six Azure regions at public preview — East US, West US, North Europe, Central India, Southeast Asia, and Japan East.

Real-World Impact

Preview customers across industries — including travel, consumer electronics, automotive, aviation, and media — have reported positive gains in transcription quality. Customers testing multilingual and domain-specific audio have observed the clearest improvements on the hardest content: proper nouns, code-switching, and long-form speech. Several are actively validating the feature on their own audio ahead of general availability.

Get Started

Enabling multilingual Post-Stream Refinement is a small configuration change on your existing SpeechConfig. You will need:

Speech SDK 1.50 or later. Earlier versions do not support the multilingual path.
A Speech resource in one of the supported regions listed above.
Auto-detect language configuration (open range) so the service identifies the language from the audio — no candidate list required.

Set the post-processing option to PostRefinement and pass an open-range AutoDetectSourceLanguageConfig when you create the recognizer. Here is a complete, copy-paste Python example, including the optional end-of-utterance detection line:

import azure.cognitiveservices.speech as speechsdk speech_config = speechsdk.SpeechConfig( subscription="YourSpeechKey", region="YourSpeechRegion") # 1) Refine the final transcript (Post-Stream Refinement) speech_config.set_property( speechsdk.PropertyId.SpeechServiceResponse_PostProcessingOption, "PostRefinement") # 2) Multilingual auto-detect - no candidate language list needed auto_detect_config = speechsdk.languageconfig.AutoDetectSourceLanguageConfig() audio_config = speechsdk.AudioConfig(use_default_microphone=True) recognizer = speechsdk.SpeechRecognizer( speech_config=speech_config, auto_detect_source_language_config=auto_detect_config, audio_config=audio_config)

💡 Tip: Refinement matters most for applications that store or process the final transcript — meeting notes, call analytics, compliance archives, AI summarization. If you only use partial results for a live display and discard them, your real-time UX (already fast) is unchanged, while any final transcript you keep improves.

Try Multilingual Post-Stream Refinement Today

Turn on higher-accuracy, language-aware transcription in your Azure AI Speech applications with a single configuration change. Available now in public preview in Microsoft Foundry.

📖 Read the Documentation

We would love your feedback. Try Post-Stream Refinement in your applications and tell us how it improves your transcription quality.

Introducing MAI-Image-2.5 Pro and MAI-Voice-2 Flash in Microsoft Foundry

Naomi Moneypenny — Thu, 23 Jul 2026 20:41:12 GMT

One of the core principles in Microsoft Foundry is giving customers flexibility to choose the right model for the right job. As we've continued to expand the Microsoft AI (MAI) model family, we've heard a consistent theme from developers and enterprises: they want more choice based on their specific workload requirements. Some need the highest possible quality, fidelity, and consistency for professional production workflows. Others prioritize responsiveness and cost-efficiency for frequent, real-time responses.

Today, we're introducing two new additions to the MAI portfolio:

MAI-Image-2.5 Pro, designed for customers who need maximum visual fidelity and creative control
MAI-Voice-2 Flash, built for low-latency voice applications where every millisecond matters.

Together, these models extend the MAI family with specialized options that help developers optimize for the jobs they need to do. Let’s dive in.

MAI-Image-2.5 Pro: Built for professional creative workflows

MAI-Image-2.5 Pro is our newest high-fidelity image generation model, designed for scenarios where visual accuracy, consistency, and creative control are critical. The model delivers stronger object consistency, improved alignment with creative intent, and enhanced visual reasoning and world knowledge, making it the ideal choice when quality and fidelity matter more than throughput.

As generative AI moves from experimentation to production, creative teams increasingly need models that can reliably generate assets that meet professional standards without extensive manual editing. MAI-Image-2.5 Pro was built to address those needs. Here are some of the scenarios it best fits:

Campaign Hero Assets & Multi-Frame Storytelling. Creative agencies and marketing teams can create polished campaign visuals while maintaining consistent products, characters, and brand elements across every asset. Whether producing launch campaigns, social media variants, retail signage, or digital advertising, Image-2.5 Pro helps ensure visual consistency throughout the customer journey.
Product Photography & E-Commerce Catalogs. Retailers and consumer brands can generate high-quality product imagery with accurate rendering of packaging, labels, materials, and reflections. The model's improved object consistency helps maintain identical product representation across multiple angles, color variations, and lifestyle settings.
Storyboarding & Pre-Visualization. Film studios, game developers, and creative production teams can rapidly develop visual concepts while maintaining consistency across characters, environments, props, and scenes. Enhanced visual reasoning enables more accurate interpretation of camera direction, lighting, and staging instructions.
Regulated Industry Content Creation. Organizations in healthcare, financial services, manufacturing, and other regulated industries can generate imagery that requires greater real-world accuracy and domain understanding, reducing the effort needed to correct inaccuracies before customer-facing use

When should customers use MAI-Image-2.5 Pro vs. MAI-Image-2.5?

Both models deliver high-quality image generation capabilities, but they are optimized for different priorities.

Choose MAI-Image-2.5 Pro when:

You need maximum image fidelity and creative quality.
Object consistency across multiple images is critical.
Your workflow depends on visual reasoning, world knowledge, and close adherence to creative direction.
You are producing professional marketing, advertising, product design, or enterprise creative assets.

MAI-Voice-2 Flash: Real-time voice experiences at scale

We're also introducing MAI-Voice-2 Flash, a new low-latency text-to-speech model that extends MAI-Voice-2 with faster response times and greater cost efficiency across more than 15 supported languages.

As voice becomes a critical interface for AI applications, responsiveness is increasingly important to the end-user experience. Whether a customer is speaking with an AI-powered support agent, interacting with a voice assistant, or navigating a self-service phone system, long pauses can make experiences feel slow and unnatural. MAI-Voice-2 Flash was built to address those scenarios. Here are some of the scenarios on when to choose MAI-Voice-2 Flash:

Call Center Agents: Customer support organizations can generate spoken responses in real time, minimizing delays between conversation turns and enabling more natural customer interactions. Low latency helps improve customer experiences while supporting AI-powered service, support, and sales workflows.
Conversational Voice Assistants: Developers can build voice-enabled copilots, assistants, and intelligent applications that respond nearly instantly. The result is a more fluid, natural conversation that feels interactive rather than turn-based.
Interactive Voice Response (IVR) Systems: Organizations can modernize traditional phone systems with dynamic AI-generated speech that responds contextually to customer requests while maintaining a responsive user experience.

When should customers use MAI-Voice-2 Flash vs. MAI-Voice-2?

The distinction between the two voice models comes down to whether customers are optimizing for voice identity or real-time responsiveness.

Choose MAI-Voice-2 Flash when:

Low latency is a primary requirement.
You are building conversational assistants, IVR systems, or call center experiences.
Users expect immediate spoken responses as part of a live interaction.
Cost efficiency and responsiveness are more important.

Get started today in Microsoft Foundry

MAI-Image-2.5 Pro is available through Microsoft Foundry, providing developers with access to Microsoft's latest advancements in image generation. Pricing starts at $5 per 1M tokens for text input, and $106 per 1M tokens for image output.

MAI-Voice-2 Flash is available through Azure Speech, allowing customers to leverage Azure Speech's enterprise-grade reliability, scalability, and ecosystem while benefiting from Microsoft's latest voice technology. Pricing starts at $15 per 1M characters.

Grounding Copilot Studio Agents with Azure AI Search and Foundry IQ

BrittanyPugh — Wed, 22 Jul 2026 18:00:00 GMT

An employee opens the HR agent and asks, "How much PTO do I accrue each month?" A few minutes later, someone else asks, "Where is the official code of ethics policy?" Those sound like the same problem. They are not. The first person needs a grounded answer they can understand. The second person needs a link to the right document quickly, without interpretation. If you design for one experience, the other one feels broken.

That is usually where knowledge-agent projects start to get messy. “Grounding” can sound like one switch you turn on, but in practice it is a spectrum: from zero-code classic search, to agentic retrieval over a knowledge base, to a forced-grounding agent that synthesizes answers when synthesis is required. The easier way to think about it is this: who is doing the retrieval work, and what does the user need back?

This post walks through five working retrieval patterns for an “Ask HR” agent built on Copilot Studio, Azure AI Search, and Foundry IQ. Each one is running code in the companion sample repo: foundry-copilot-hr-policy-knowledge. Each has a clear “use this when,” and the five patterns share the same reusable knowledge base so you can layer them on without re-indexing. By the end, you should have a decision tree you can reuse for your own knowledge source, whether that is HR policy, product docs, or support runbooks.

Scope: companion sample for learning and experimentation, not production-ready deployment. Review the Azure Well-Architected Framework for reliability, security, cost, and operational hardening before you ship.

The scenario: one index, many front doors

Here is the setup. The sample answers employee questions from a small corpus of internal HR policy documents: PTO accrual, hiring rules, code of ethics, blood-borne pathogen procedures, and dozens more. Underneath every pattern is one foundation: an Azure AI Search index named hr-policy-index, populated by an indexer and skillset that chunk and vectorize the documents.

Patterns A, C, and the Hosted Agent query that index directly. Patterns A2 and B add a Foundry IQ knowledge base named hr-knowledge-base on top of the same index for agentic retrieval. That layering is the part to pay attention to. The retrieval assets stay separate from the orchestration layer, so you can start with the simplest pattern, prove value quickly, and move to a more capable one later without re-indexing.

Two questions that decide everything

Before we get into the patterns, it helps to define the two retrieval terms I use throughout the rest of the post:

Classic search, index-first retrieval: one hybrid (keyword + vector) query against an Azure AI Search index, ranked and returned. Fast and predictable.
Agentic retrieval, the knowledge base plans multiple sub-queries from the user's question, runs them in parallel, re-ranks, and merges the results before the agent composes an answer. Higher quality on complex, multi-part questions.

If you want the fuller picture of how these two approaches map to retrieval-augmented generation, the Azure AI Search team's RAG and generative AI overview walks through the trade-offs and uses a similar HR/PTO example.

Once those terms are clear, the decision tree comes down to three practical questions:

Q1: Do users need an answer or are they really trying to find the right document?
If they just need the document, stay on the locator path. If they need the policy explained or summarized, move into the answer-synthesis path.

QL: Is the content in a citation-friendly knowledge base?
For example, SharePoint content or Azure AI Search content with a reliable blob_url. If yes, Copilot Studio can usually handle this with native citation cards in Pattern A. If not, use Pattern C with the dual tool /api/lookup path so the agent can return the exact document link.

Q2: Do you actually need an LLM agent in the middle?
If the answer is no, keep it simple: use classic search or agentic retrieval over the knowledge base. If the answer is yes, move into the agent path.

QK: For that non-agent path, is classic index search enough, or do you need agentic KB retrieval?
Classic search points to Pattern A. Agentic retrieval over the knowledge base points to Pattern A2.

Q3: If you need an agent, do you want Foundry to run the request loop, or do you need to self-host it?
If Foundry can manage the runtime, use Pattern B. If you need the request loop in your own container, use the Hosted Agent.

That is the decision tree in plain terms: Q1 decides whether this is a document-locator experience or an answer-synthesis experience. Q2 decides whether you need an LLM agent at all. Q3 is only about where the agent runs, either Foundry or your container. It does not change the front door; Copilot Studio can still be the user-facing experience.

How the sample repo is organized

The repo follows the same flow as the post. Start with docs/DataPipelineAndTesting.md to understand how the HR policy corpus is indexed, tested, and validated. Use docs/RetrievalPatterns.md as the decision model for choosing between classic search, agentic retrieval, forced grounding, and hosted runtime options. Then use the pattern-specific docs when you are ready to wire each path.

For Copilot Studio patterns, docs/CopilotStudioIntegration.md maps to Pattern A, while docs/CopilotStudioHybridExample.md maps to Pattern C and the dual-tool locator flow. For the more advanced agent paths, docs/FoundryAgentArchitecture.md covers Pattern B and the hosted agent architecture. docs/Distribution-M365-Teams.md shows how the agent can be distributed through Microsoft 365 and Teams once the retrieval pattern is working.

The rest of the post is that tree, one branch at a time.

Pattern A: Direct index (classic search, zero agent code)

Start here. Copilot Studio queries hr-policy-index directly through its built-in Knowledge action. No custom agent code runs in the answer path. The sample only owns the index, skillset, and indexing pipeline.

Populate the index (server-side indexer + skillset handles chunking and vectorization):

uv run python scripts/index_knowledge_base_integrated_vectorization.py # Builds hr-policy-index; a client-side alternative exists for dev/test

What you get: very low latency in the sample, roughly 1-2 seconds, no LLM cost in the retrieval path, and native citation cards. When the source documents carry a blob_url or metadata_storage_path, Copilot Studio can render a click-through card straight to the document. For many "where is the policy?" questions, that may be enough.

The honest limitation: Pattern A is still classic search. It does not force synthesis. If Copilot Studio generates an answer from retrieved snippets, it may paraphrase a policy in a way that is close, but not precise enough. For HR policy, that matters. If exact wording matters, that is your sign to step up to Pattern B.

Pattern A2: Copilot Studio meets Foundry IQ (agentic retrieval, no prompt agent)

This is the pattern I would look at when you want better retrieval quality without taking on the overhead of a full prompt agent. In the Copilot Studio new agent experience preview, an agent connects directly to a Foundry IQ knowledge base through Microsoft IQ, with no Foundry prompt agent in between. You reuse the same hr-knowledge-base on top of the same hr-policy-index (one command: python -m src.agents.create_foundry_agent), but retrieval is now agentic: the knowledge base plans sub-queries, retrieves in parallel, reranks, and hands merged results to the agent.

Wiring it takes a few clicks in Copilot Studio (step-by-step on Microsoft Learn):

Build → Microsoft IQ → Foundry IQ → Create new connection
Choose Microsoft Entra ID Integrated authentication
Select hr-knowledge-base
Add to agent

A2 is worth the upgrade from A for two reasons. First, you get agentic-retrieval quality without having to build, deploy, or maintain a prompt agent. The knowledge base becomes the reusable asset you improve in Microsoft Foundry, not something you keep reworking inside each Copilot Studio agent. Second, when configured with Microsoft Entra ID Integrated authentication, retrieval can return ACL-trimmed results per user. Each person sees content based on their access.

Foundry IQ knowledge bases can also inherit enterprise-readiness controls such as customer-managed keys, network isolation, and Entra ID. A single knowledge base can also federate across multiple knowledge sources in parallel.

Use A2 when you want stronger hybrid retrieval quality without taking on the overhead of operating a full agent.

Pattern B: Foundry Agent Service with forced grounding

When answers need to be synthesized and grounded, publish a prompt agent to Microsoft Foundry with Foundry Agent Service. In the sample, the agent uses an MCPTool pointing at the knowledge-base endpoint, with tool_choice="required" so the model retrieves policy chunks before answering.

# src/agents/hr_policy_agent.py (excerpt) agent = PromptAgentDefinition( model=model_deployment_name, # e.g. gpt-5-mini instructions=HR_POLICY_INSTRUCTIONS, tools=[mcp_tool], # KB MCP endpoint tool_choice="required", # require retrieval before answering )

Invoke it through the OpenAI client the project hands you:

client = project.get_openai_client() response = client.responses.create( input="How does PTO accrue for a new hire?", extra_body={"agent_reference": {"name": agent_name}}, )

What you get: synthesized answers with grounding and inline [Policy XXXX - Title] citations, all from a single SDK call on a managed runtime. The trade-off: synthesis takes longer. In the sample, answers take roughly 10-14 seconds versus 1-2 seconds for classic search. For policy explanations, that extra time can be worth it because the user gets a composed, grounded answer instead of a list of snippets.

Pattern C: Dual-tool routing for deterministic document locators

Some questions do not need an essay; they just need the right URL, fast. Pattern C lets Copilot Studio route per turn:

"Where is the PTO policy?" → POST /api/lookup, a deterministic endpoint with no LLM, roughly 1-2 seconds, returning the document URL verbatim in the answer body.
"How many PTO hours do I accrue?" → hand off to Pattern A or B for a synthesized answer.

POST /api/lookup { "query": "PTO policy" } → 200 OK { "policy_id": "12345", "title": "Types of Leave: Paid Time Off (PTO)", "blob_url": "https://.../12345-pto.pdf" }

Reach for Pattern C when native citations are not enough. For example, use it when you need fast locator responses, the URL printed directly in the answer body, deterministic and auditable output, or support for a source that is not citation-friendly. The endpoint lives at src/backend/main.py:/api/lookup, with its contract in copilot/openapi-lookup-v2.json.

Hosted Agent: the same agent on your own runtime

If you need to own the request loop, custom authentication, side-car services, or infrastructure that stays inside your boundary, run the agent yourself. The Hosted Agent is the self-hosted version of the same idea: a container built on Microsoft Agent Framework with FoundryChatClient. It supports both classic and agentic retrieval through one environment variable:

RETRIEVAL_MODE	Strategy	Retrieval type
tool (default)	Custom @tool search_hr_policies (hybrid + semantic)	Classic search
context-semantic	Built-in AzureAISearchContextProvider before each turn	Classic search
context-agentic	AzureAISearchContextProvider over hr-knowledge-base	Agentic retrieval

The context-* modes use Agent Framework’s out-of-the-box RAG context provider. Retrieval runs automatically before each model call with standardized context and citation prompts, so the agent does not have to call a search tool explicitly. That gives the self-hosted path parity with the managed Foundry path across both retrieval types. Copilot Studio can still be the front door. Q3 in the decision tree is really about where the request loop runs, not who greets the user.

Choosing a pattern

Pattern	Orchestrator	Retrieval	Latency (sample)	Best for
A	Copilot Studio	Classic	~1-2 s	Start here, native citations, no agent code
A2	Copilot Studio → Foundry IQ	Agentic	~2-4 s	Agentic quality, no agent to maintain
B	Foundry Agent Service	Classic/agentic via MCP	~10-14 s	Forced-grounding synthesis in Foundry
C	Copilot Studio (router)	None for lookup	~1-2 s	Deterministic, verbatim document locators
Hosted	Agent Framework container	Classic + agentic	~10-14 s	Self-hosted runtime, custom auth

A simple way to read the table: start at A, move to A2 when you want agentic retrieval without operating an agent, choose B when each answer needs to be synthesized and grounded in Foundry, add C for high-volume locator traffic, and pick the Hosted Agent when you need the runtime on your own infrastructure. These are not mutually exclusive. A mature agent often routes locator queries to C and content questions to A2 or B.

What's next?

Try it: clone the sample and follow Steps 1-3 of the walkthrough to stand up Pattern A, provision hr-knowledge-base, connect Copilot Studio, and ask a question in minutes.
Go agentic: wire the same knowledge base into the Copilot Studio new agent experience via Foundry IQ (Pattern A2) and compare answer quality side by side.
Learn more: explore agentic retrieval in Azure AI Search, Foundry IQ, and Microsoft Agent Framework.
Adapt it: swap the HR policy corpus for your own product docs, support runbooks, or internal knowledge source, then compare Pattern A, A2, and B against the same user questions.
Use the repo-doc map: start with docs/RetrievalPatterns.md for the decision model, docs/CopilotStudioIntegration.md for Pattern A, docs/CopilotStudioHybridExample.md for Pattern C, docs/FoundryAgentArchitecture.md for Pattern B and Hosted Agent, and docs/DataPipelineAndTesting.md for ingestion and validation.

My recommendation: start simple, prove the index works, and move up the stack only when the use case needs it. Some questions need a trusted link. Others need a grounded explanation. A strong architecture supports both without forcing every request through the same path.

References

Copilot Studio + Foundry IQ

Foundry IQ / knowledge layer

Azure AI Search, retrieval

Microsoft Foundry Agent Service (Pattern B)

Foundry Agent Service overview

Microsoft Agent Framework (Hosted Agent)

Governance

Azure Well-Architected Framework

Related Microsoft Foundry blog posts

Design the Network Before You Deploy: Best Practices for Microsoft Foundry Standard Agents BYOVNet

Priya_Kedia — Wed, 22 Jul 2026 16:00:00 GMT

Ask any enterprise team standing up Microsoft Foundry in production and you'll hear the same non-negotiable: the agent can't run on the open internet. The moment it touches proprietary data, internal APIs, or regulated workloads, security wants it inside the company's own virtual network — behind private endpoints, a central firewall, and controlled DNS. That configuration — a Foundry Standard Agent injected into a Bring-Your-Own (BYO) VNet — is the topology most large organizations actually ship to production. It's also where first deployments quietly stall.

The reason is counterintuitive. Deploying the agent is, in the end, just a Bicep or Terraform template — the easy 20%. The hard 80% is the network design you lock in before you run it, and it's unforgiving in two ways the template isn't. Several of the choices are irreversible: outbound network injection and the agent subnet can't be changed after deployment, and subnets can't be resized in place — a wrong call on day one means a redeploy, not a tweak. And the real lead time is organizational, not technical: firewall change requests, IP allocation from central IPAM, NSG approvals, and Private DNS ownership in a hub-and-spoke model each take days to weeks to clear. Discover a requirement late and you're blocked on a ticket, not a template.

This guide is written to prevent that. It's for the people who own the groundwork — the platform / landing-zone architect and the network security team, with the AI platform lead who runs the deployment as the second reader. The goal is simple: that you can walk into your first planning meeting already knowing every subnet, CIDR, DNS zone, NSG rule, and firewall FQDN you need to request. Get the subnet layout, the Private DNS design, and the firewall allow-list right, and the template becomes the formality it should be.

Every enterprise conversation I have about Foundry Standard Agents in a Bring-Your-Own (BYO) VNet opens with the same handful of questions. They're the right questions — and each one is a design decision, not a troubleshooting step:

How many subnets do I need, and what does each one look like?
What CIDR should I reserve for the agent subnet?
Which FQDNs must I allow on my central firewall?
Where do the Private DNS zones live?
And how do I prove it's all correct before the first agent call?

The sections below answer each one as a decision you make up front. Get the subnet layout, the Private DNS design, and the firewall allow-list right, and the template becomes the formality it should be.

Scope: Microsoft Foundry Standard setup (account + project + capability host) injected into a customer VNet, optionally fronted by API Management, optionally behind a hub-and-spoke Azure Firewall with central Private DNS.

Who is this for?

Reader	What they own	What they get from this guide?
Cloud platform / landing-zone architect	Subnets, IPAM, Private DNS, hub-and-spoke topology	The exact subnet, CIDR, and DNS design to request and provision
Network security lead	NSGs, Azure Firewall policy, UDRs	The firewall allow-list and NSG rule set to approve in one pass
AI platform lead / solution architect	The Foundry deployment itself	A complete, precise ask to hand the network team on day one

How to use this guide: the pre-deploy checklist in section 7 is the deliverable. Sections 1–6 are the reasoning behind each checkbox — read them once, then work from the checklist.

1. Co-locate everything in one region

There is no hard platform requirement forcing all the pieces of a Foundry Standard deployment to live in the same Azure region. But once you inject the agent runtime into your VNet, spreading resources across regions costs you two things you don't want to pay for:

Cross-region data-transfer charges — if Foundry is in region A but Cosmos, Search, Storage, or your model endpoint sits in region B, every agent call traverses inter-region VNet peering, which is billed per GB in both directions. Keep everything in one region and that traffic stays local — no inter-region peering, no per-GB data-transfer bill, and a materially lower cloud spend.
Latency — agent runs are chatty (thread reads, tool calls, vector lookups, model calls). Adding 40–100 ms per hop across regions is felt immediately in user-facing responses.

Rule of thumb: the Foundry account, project, capability host, model deployments, dependent stores (Cosmos, Storage, Search), APIM (if used), the VNet, and the jumpbox should all live in the same region. Use paired regions only for DR — not for day-1 architecture.

2. Subnet structure — plan the layout before you touch a template

Before you copy the table — your SKU and landing zone change the shape of it. The layout below is the recommended pattern for production workload (Premium APIM + a jumpbox), but two choices materially move subnets in or out of it:

APIM SKU drives whether you need one subnet or two. Standard v2 uses VNet integration for outbound and a separate Private Endpoint for inbound — two distinct subnets with two distinct roles (apim-outbound-subnet plus the inbound PE on pe-subnet). Premium (classic) supports full VNet injection, where a single delegated subnet carries both ingress and egress — you get higher throughput, multi‑region, availability zones, and one less subnet to plan for, at a higher price point. Pick the SKU first; the subnet count follows.

The jumpbox subnet is landing‑zone dependent. If your enterprise landing zone already gives operators direct line of sight into the spoke — ExpressRoute, a site‑to‑site VPN — you don't need a dedicated jumpbox-subnet here at all. Add it when you're onboarding a partner team, remote contractors, or anyone who doesn't already have private routing into the VNet and needs a controlled entry point for nslookup / portal validation.

The most common mistake I see is treating the VNet as a single flat subnet. Foundry Standard + APIM has hard constraints that push you to at least four purpose-built subnets:

Subnet	Purpose	Delegation	Notes
agent-subnet	Foundry capability-host injects NICs here for the agent runtime	Microsoft.App/environments (required)	Must be empty of Private Endpoints — delegation and PE NICs cannot coexist
pe-subnet	Private Endpoints for Cosmos, Storage, Search, Foundry account, and APIM inbound	None	PE subnets cannot be delegated
apim-outbound-subnet	APIM VNet integration — the egress path from APIM to your private backends	Microsoft.Web/serverFarms (required)	Dedicated to a single APIM instance; cannot be shared
jumpbox-subnet	VM(s) for RDP/SSH access, nslookup validation, private portal browsing	None	Keep it separate from pe-subnet — different NSG posture, different UDR needs, and you don't want VM churn eating PE address space

What about extra workloads in the same VNet? The table above is the Foundry + APIM baseline. If you host anything else alongside — MCP servers, custom APIs, background workers — give each its own dedicated subnet. It cannot share agent-subnet (exclusive to one Foundry account) or pe-subnet (can't be delegated), and its delegation depends on the host: Container Apps → Microsoft.App/environments (a separate environment from Foundry's), Container Instances → Microsoft.ContainerInstance/containerGroups, App Service / Functions with VNet integration → Microsoft.Web/serverFarms, AKS / plain VMs → no delegation.

Recommended APIM topology — Premium v2, PE for inbound + integration for outbound

APIM has several supported networking modes, but for a production Foundry setup where you need to isolate both the inbound gateway and outbound calls to private backends, the pattern that gives you the cleanest topology is:

SKU: API Management Premium v2. Premium v2 gives you availability zones, higher throughput, workspaces, dedicated compute, and the highest scale ceiling — things you want on the production path for an AI Gateway sitting in front of Foundry. Standard v2 supports the same networking model, but Premium v2 is what you deploy when this API surface is a first-class production dependency.
Inbound = Private Endpoint on pe-subnet. Create a private endpoint for APIM against the Microsoft.ApiManagement/service Gateway sub-resource. This registers into privatelink.azure-api.net automatically and gives you a private inbound IP without any subnet delegation. Because PE subnets can't be delegated anyway, this is exactly where APIM's inbound belongs — next to your other PEs.
Outbound = VNet integration into apim-outbound-subnet. Enable Premium v2 VNet integration so APIM's calls to your backends leave through a dedicated subnet delegated to Microsoft.Web/serverFarms. From here, APIM reaches Foundry, Cosmos, an internal HR API, etc. via their PEs — traffic never touches the internet.

Why the delegations matter

Microsoft.App/environments on the agent subnet formally hands the subnet over to the Container Apps managed environment fabric that Foundry's capability-host runs on. Delegation grants the platform permission to provision and manage its own infrastructure inside the subnet on your behalf (NICs, load balancer, health probes, required routing policies) and makes the subnet exclusive to that service — no PEs, no VMs, no other workloads can share it. Without the delegation, capabilityHosts PUT fails subnet validation.
Microsoft.Web/serverFarms on the APIM outbound subnet is required for Premium v2 (or Standard v2) VNet integration. The subnet is dedicated — a single APIM instance owns it and it cannot be shared with any other Azure resource. Also make sure the Microsoft.Web resource provider is registered in the subscription.
Delegations are not fungible: you cannot repurpose the agent subnet as a PE subnet later, because a subnet cannot simultaneously host delegated services and Private Endpoints. And per official Foundry guidance, outbound network injection cannot be changed after deployment — pick the agent subnet carefully because moving it later means redeploying Foundry.

Agent subnet sizing. The Foundry Agent Service networking doc is explicit on this: the recommended size of the delegated agent subnet is /24 (256 addresses), precisely because the subnet is delegated to Microsoft.App/environments and the Container Apps runtime consumes IPs as it scales out. /27 is the API‑enforced hard floor and is fine for smaller, non‑production deployments — PoCs, dev/test, or low‑traffic internal agents where you know the scale ceiling. Treat `/27` as a lower bound the platform accepts, not a size to deploy production with.

On top of that, you cannot resize a subnet in place — if you run out, moving is destructive. Provision /24 from day one; it costs you nothing extra and removes an entire class of future scaling incidents.

APIM outbound subnet sizing. Premium v2 VNet integration requires a minimum of /27 and recommends /24 for scale-out headroom — same rationale, err on the generous side.

3. Hub-and-spoke with Azure Firewall — model the traffic before you write rules

If you sit behind a centralized Azure Firewall with a UDR forcing 0.0.0.0/0 through it, every hostname the runtime touches must be in your application rule collection. Before deploying, it helps to model the traffic in two buckets:

In-VNet, service-to-service traffic — Foundry runtime → Cosmos / Storage / Search / your model endpoint / APIM. Because you're deploying with private endpoints in front of every one of those services, this traffic is outbound to a private IP inside the VNet, not outbound to a public FQDN. It also does not traverse the firewall by default — the /32 system route Azure injects for each PE beats your 0.0.0.0/0 UDR, so PE traffic goes straight to the PE NIC. Your NSG rules should reflect that: allow agent-subnet → pe-subnet on 443, and you're done for the data plane. No firewall application rule needed for it.
Outbound to Microsoft-managed control-plane FQDNs — Entra token acquisition, managed identity, container image pulls, evaluations telemetry, etc. This is what the firewall application rule collection is really for.

Source addresses to whitelist on the firewall

agent-subnet — the runtime originates almost every control-plane call from here.
jumpbox-subnet — if you provisioned a dedicated jumpbox-subnet to reach the platform for `nslookup`, portal browsing, and troubleshooting.

FQDNs — start with the documented baseline

The official Foundry private-link doc publishes this minimum allow-list for Foundry with virtual network injection:

Scenario	FQDNs / Service Tag	Why
Agents	.identity.azure.net, login.microsoftonline.com, .login.microsoftonline.com, *.login.microsoft.com (or the AzureActiveDirectory service tag)	Required for the Azure Container App delegation that hosts the agent runtime — Entra + managed identity
Evaluations & Traces	AzureMachineLearning service tag, settings.sdk.monitor.azure.com	Evaluators catalog + shipping results to Application Insights Prefer the service tag over a raw `*.blob.core.windows.net` wildcard — it scopes egress to just the Microsoft‑managed storage endpoints Foundry actually needs, without opening every blob account in Azure.

Scenario

FQDNs / Service Tag

Why

Agents

*.identity.azure.net, login.microsoftonline.com, *.login.microsoftonline.com, *.login.microsoft.com (or the AzureActiveDirectory service tag)

Required for the Azure Container App delegation that hosts the agent runtime — Entra + managed identity

Evaluations & Traces

AzureMachineLearning service tag, settings.sdk.monitor.azure.com

Evaluators catalog + shipping results to Application Insights

Prefer the service tag over a raw `*.blob.core.windows.net` wildcard — it scopes egress to just the Microsoft‑managed storage endpoints Foundry actually needs, without opening every blob account in Azure.

Start there. If your setup is only Foundry + PEs + a firewall, that list is usually enough.

FQDNs — additions you'll typically need in practice

If you're also running APIM (Standard v2 outbound integration), pulling container images, or using Application Insights end-to-end, expect to add:

Category	FQDNs	When you need it
Azure Resource Manager	management.azure.com, *.management.azure.com	Runtime ARM calls (role assignment reads, resource metadata)
Microsoft Graph	graph.microsoft.com	Directory / user context in some agent scenarios
Container image pulls	mcr.microsoft.com, .data.mcr.microsoft.com, .azurecr.io	Cap-host pulls its runtime image
APIM internal telemetry	.prod.warm.ingest.monitor.core.windows.net, .prod.hot.ingest.monitor.core.windows.net, *.prod.microsoftmetrics.com, azureprofiler.trafficmanager.net	Only if firewall Deny logs show APIM's outbound subnet trying to reach Geneva/monitoring endpoints — in most Premium v2 setups this stays on the Microsoft backbone and never hits the firewall
Application Insights ingestion	.in.applicationinsights.azure.com, .monitor.azure.com	If your app or APIM sends telemetry to App Insights

Note : Wildcards are the pragmatic starting point; PEs on the premium SKUs are the hardening step for regulated / high‑sensitivity environments where a broad IP pool on the firewall is a compliance finding. Prefer premium SKUs with dedicated Private Endpoints over wide wildcards on the firewall. Several services on this list only expose public multi‑tenant endpoints on their lower tiers, which is why you end up whitelisting broad wildcards like `*.data.mcr.microsoft.com`, `*.monitor.azure.com`, or `*.in.applicationinsights.azure.com` — each one opens a large pool of shared Microsoft IPs to your egress. The premium tiers give you a way out:

Why *.blob.core.windows.net/AzureMachineLearning Service Tag shows up even when you don't use blob storage: several Microsoft-managed services (App Insights, Foundry evaluators, Container Apps image cache) round-trip through their own storage accounts. The official allow-list already includes it for Evaluations & Traces.

Design for iteration

Enable Azure Firewall diagnostic logs from day one and target a Log Analytics workspace. You'll want to filter Deny events by source subnet + target FQDN to close any remaining gaps in your allow-list — but you'll be iterating on a short delta, not building the list from scratch.

4. Private DNS — the highest-value decision to get right first

More BYO-VNet Foundry incidents trace back to DNS than to any other single component — which is exactly why it's worth designing first. Two rules will save you most of them:

Rule 1: Never have duplicate Private DNS zones

If privatelink.blob.core.windows.net exists in both your hub and your spoke, VNet resolution becomes non-deterministic — you'll intermittently resolve to the wrong private IP (or worse, to 0.0.0.0). Pick one home for each zone and delete duplicates.

Rule 2: In a hub-and-spoke, PDZs belong in the hub

Create each of the seven privatelink.* zones once, in the hub.
Set every spoke VNet's DNS servers to the firewall private IP (Azure Firewall with DNS Proxy enabled). Spoke query → firewall DNS proxy → Azure DNS (168.63.129.16) resolved in the hub context → hub-linked PDZ → returns the PE IP. No PDZ link on the spoke is needed for this to work.

The seven zones Foundry Standard needs: privatelink.services.ai.azure.com, privatelink.openai.azure.com, privatelink.cognitiveservices.azure.com, privatelink.search.windows.net, privatelink.blob.core.windows.net, privatelink.documents.azure.com, privatelink.azure-api.net

Force traffic through the firewall: VNet peering alone will not send spoke egress through the hub firewall — Azure routes prefer the peering directly. You need a UDR on every spoke subnet with a 0.0.0.0/0 route pointing at the firewall private IP. Without the UDR, your FQDN allow-list is enforced on nothing.

5. Network Security Groups — the minimum rule set

NSGs are the layer where a single missing outbound rule silently blocks the data plane — everything provisions cleanly, but nothing talks. Think of NSGs in the same two-bucket model from section 3 — service-to-service inside the VNet, and outbound control plane.

Allow outbound HTTPS (TCP 443) from agent-subnet and apim-outbound-subnet to the pe-subnet CIDR (or, if you prefer, to the relevant service tags — AzureCosmosDB, Storage.<region>, AzureCognitiveSearch, AzureAD). This covers the runtime's calls to Cosmos, Storage, Search, and the Foundry / model endpoints via their private endpoints.
Allow APIM's outbound dependencies — at minimum outbound TCP 443 to the Storage and AzureKeyVault service tags from apim-outbound-subnet, per the APIM outbound integration NSG reference. Missing these is the classic "APIM went 'Failed' in the portal" symptom.
Allow RDP/SSH from the firewall subnet (not from the internet) on the jumpbox-subnet, and use firewall DNAT to expose it — no public IPs on VMs.
Default-deny outbound to internet on the agent, PE, and APIM subnets; let only the firewall route escape via UDR. This ensures every FQDN call goes through your allow-list, not around it.

6. Validate before you write a single line of agent code

Before you call agents.create() from your app, prove the network is right. From your jumpbox:

1. DNS resolves privately from the spoke
nslookup <foundry-account>.services.ai.azure.com
nslookup <search-name>.search.windows.net
nslookup <cosmos-name>.documents.azure.com
nslookup <apim-name>.azure-api.net

Each should return a 10.x / 192.168.x IP (your PE), not a public IP.

If the jumpbox resolves privately but the agent subnet does not, your PDZ VNet links or your UDR are wrong. The agent subnet must resolve the same private IPs as the jumpbox — otherwise the runtime will egress to public IPs and get 403'd by your PE-only firewall on Cosmos.

The same source-IP principle applies to firewall rules: whatever subnets you validated from must be listed as sourceAddresses in the application rule collection, alongside every FQDN they need to reach.

7. A pre-deploy checklist

Walk this list before you kick off the deployment — every item is cheaper to fix now than after capabilityHosts PUT is in flight.

Agent subnet delegated to Microsoft.App/environments — sized /24 per Microsoft's official recommendation (/27 is the API-enforced hard floor, not a target).
Decided whether agent tools run behind the VNet — and picked the matching template (standard vs. tools-behind-VNet) accordingly.
APIM inbound is via a Private Endpoint on pe-subnet and APIM outbound is via VNet integration into a dedicated apim-outbound-subnet delegated to Microsoft.Web/serverFarms, sized /24 (min /27).
The privatelink.* PDZs (including privatelink.azure-api.net) exist once (in the hub, if hub-and-spoke) and are linked to the hub VNet only; spokes resolve them via the firewall DNS proxy.
Spoke VNet DNS = firewall private IP; firewall policy has DNS proxy enabled.
A UDR on every spoke subnet forces 0.0.0.0/0 → firewall.
Firewall application rules include the documented Foundry baseline (§3) plus additions for your extras (container pulls, App Insights), with the agent and jumpbox subnets as source addresses. (Only add apim-outbound-subnet if Deny logs show traffic from it.)
NSGs allow outbound 443 from agent + APIM outbound subnets to the PE subnet CIDR (or relevant service tags), and APIM's Storage / AzureKeyVault outbound rules per the APIM docs.
nslookup from both jumpbox and agent subnet returns private IPs for Foundry, Cosmos, Storage, Search, APIM.

Closing thought — decide first, deploy second

Foundry Standard on a BYO VNet isn't harder than any other network-isolated Azure workload. It's just the intersection of five product surfaces — Foundry, APIM, Cosmos, Search, and Storage — each with its own private-networking quirks. What makes it feel hard is deploying first and discovering the constraints second.

Reverse that order and it's straightforward. Before you run a single template:

Lock the subnet layout and delegations — the agent subnet is irreversible, so size it /24 from day one.
Design Private DNS once, in the hub , and force spoke egress through the firewall with UDRs.
Assemble the firewall allow-list and NSG rules as a change request your network team can approve in one pass.
Validate DNS from both the jumpbox and the agent subnet before any agent code exists.
Run the whole thing from the pre-deploy checklist — that list is the deliverable; everything above is the reasoning behind it.

Get those right and the template is a formality. Get them wrong and you're redeploying. That's the whole reason to think network-first.

Where to go next

Official Microsoft sample templates (Bicep + Terraform) for the exact setup described here — the microsoft-foundry/foundry-samples repo, infrastructure/ folder: https://github.com/microsoft-foundry/foundry-samples/tree/main/infrastructure
- 15-private-network-standard-agent-setup — Standard Agent with end-to-end network isolation (tools not behind the VNet).
- 16-private-network-standard-agent-apim-setup — the APIM-fronted topology.
- 19-private-network-agent-tools — use this when agent tools must also stay inside the VNet.
Foundry Agent Service limits, supported tools, regions, and troubleshooting — what's allowed, what isn't, which tools and models are supported, and how to handle limit/quota errors end to end: https://learn.microsoft.com/en-us/azure/foundry/agents/concepts/limits-quotas-regions
Set up private networking for Foundry Agent Service (Bicep / Terraform / azd walkthrough) and the companion Deep dive into Foundry Agent Service networking for subnet sizing and IP allocation: How to configure network isolation for Microsoft Foundry - Microsoft Foundry | Microsoft Learn, https://learn.microsoft.com/en-us/azure/foundry/agents/how-to/virtual-networks

Choosing the Right Starting Point for Enterprise AI Agents with Copilot Studio and Microsoft Foundry

PavanKalyanChilakalapalli — Tue, 21 Jul 2026 18:00:00 GMT

Choosing the Right Starting Point for Enterprise AI Agents with Copilot Studio and Microsoft Foundry

Enterprise AI conversations often begin with a familiar question: Should this agent be built with Microsoft Copilot Studio or Microsoft Foundry?

While the question is common, it can be misleading if it frames the platforms as competing choices. In practice, Copilot Studio and Microsoft Foundry serve different builders, development styles, and operating models. The more useful question is:

Which starting point best fits the team building the agent today while providing a path to scale as requirements evolve?

Some teams need a guided, low-code environment that accelerates delivery. Others need a code-first platform with greater control over architecture, deployment, security, and operations. The right choice depends less on the agent itself and more on who is building it, how it will be managed, and what level of control is required.

This article provides a practical framework for evaluating Microsoft Copilot Studio and Microsoft Foundry when planning enterprise AI agents.

Start with the team and operating model

Before selecting a platform, architects should answer three questions:

Who is building and maintaining the agent?
Business users, IT admins, pro-makers, developers, architects, and AI engineers have different requirements and tools.
How does the team work today?
Do they prefer guided authoring experiences or software engineering practices such as source control, CI/CD, testing, and release management?
What level of control is required?
Does the solution benefit from a managed experience, or does it require ownership of runtime, networking, telemetry, and deployment?

Agent classification remains useful:

Knowledge agent
Helps users find, summarize, and reason over trusted information.
Workflow agent
Helps users complete business processes such as approvals, ticket creation, or status tracking.
Autonomous or multi-agent system
Coordinates multiple tools, systems, models, or agents to complete more complex outcomes.

However, platform selection should consider more than complexity. Builder persona, authoring model, governance requirements, and operational ownership often have a bigger impact on long-term success.

Where Copilot Studio is a strong starting point

Microsoft Copilot Studio is a strong starting point when organizations need to create agents quickly through a guided, low-code experience.

It is especially useful for business users, IT admins, and pro-makers who want to build and improve agents without directly managing infrastructure.

Choose Copilot Studio when the scenario requires:

Business-led or IT-led ownership
Low-code or guided authoring
Rapid time to value
Microsoft 365 and Teams experiences
Business process automation
Managed deployment and operations
Ongoing iteration by business stakeholders
Minimal dependence on software engineering resources

Typical examples include HR assistants, onboarding agents, policy assistants, employee self-service agents, and departmental workflow solutions.

Where Microsoft Foundry is a strong starting point

Microsoft Foundry is a strong starting point when organizations need a code-first, Azure-native environment for building, deploying, and governing AI applications and agents.

Foundry is especially relevant for developers, architects, and AI engineers who build agents as software products and require deeper ownership of architecture, deployment, evaluation, and operations.

Choose Microsoft Foundry when the scenario requires:

Developer-led or architecture-led ownership
Code-first development
Custom orchestration and integrations
Advanced model selection and lifecycle management
Multi-agent or long-running workflows
Advanced evaluation, tracing, and observability
GitHub, Visual Studio Code, and DevOps integration
Customer-controlled Azure environments
Private networking and secure enterprise boundaries
Data residency, compliance, and operational governance requirements

For teams that already build software through repositories, pull requests, automated testing, and deployment pipelines, Foundry is often the most natural starting point.

Copilot Studio and Foundry together

Most organizations eventually use both platforms.

A common pattern is using Copilot Studio to provide a business-facing experience while Foundry powers custom orchestration, advanced reasoning, enterprise integrations, evaluation, or operational requirements.

Rather than treating platform selection as a one-time decision, organizations can start where a team moves fastest today and expand as requirements mature.

Decision framework: Choose the right starting point

Decision Factor	Copilot Studio	Microsoft Foundry
Primary builder	Business users, IT admins, pro-makers	Developers, architects, AI engineers
Authoring model	Guided, low-code experiences	Code-first development
Runtime ownership	Managed SaaS experience	Customer-controlled Azure resources
Networking requirements	Managed environment	Private networking and secure boundaries
Development lifecycle	Configuration-driven	Source control, CI/CD, automated testing
Customization	Standard business workflows	Custom orchestration and integrations
Observability	Basic monitoring needs	Advanced tracing and evaluation
Best fit	Business-facing agents and employee experiences	Developer-led AI applications and advanced agent systems

Practical scenarios

Scenario 1: HR policy assistant

An HR team needs an internal agent that answers questions about policies, benefits, onboarding, and leave processes using SharePoint content and internal documentation.

Recommended starting point: Copilot Studio

The ownership model is business-led, content changes frequently, and rapid deployment is valuable.

Scenario 2: IT helpdesk triage agent

An IT team needs an agent to answer support questions, create tickets, check ticket status, and escalate unresolved issues.

Recommended starting point: Copilot Studio or hybrid

Copilot Studio works well for conversational experiences and workflow automation. If requirements expand into custom orchestration, advanced retrieval, or engineering-owned services, Foundry can extend the solution.

Scenario 3: Enterprise architecture advisor

An architecture team wants an agent that reviews requirements, recommends architecture patterns, generates design artifacts, and identifies risks.

Recommended starting point: Microsoft Foundry

This scenario requires structured outputs, reasoning, orchestration, evaluation, and integration into engineering workflows.

Scenario 4: Multi-agent claims processing system

An insurance organization wants to automate document understanding, fraud assessment, policy validation, customer communication, and escalation processes.

Recommended starting point: Microsoft Foundry, with Copilot Studio where user-facing experiences are required

The solution involves orchestration, monitoring, governance, and multiple interacting components that benefit from a code-first platform.

Interoperability and governance at scale

Enterprise AI programs rarely operate within a single platform. Agents need to interact with tools, workflows, business systems, and other agents across the organization.

Architects should plan for interoperability across four layers:

Tools and connectors – APIs, plugins, and enterprise systems.
Agents – collaboration between specialized agents.
Workflows – integration with business processes and automation.
Channels – Microsoft 365, Teams, Copilot experiences, and custom applications.

Governance is equally important. Organizations should define ownership, identity controls, data boundaries, evaluation processes, monitoring, and lifecycle management before scaling agent adoption.

Agent 365 and related governance capabilities can help organizations maintain visibility and oversight across agents built using Copilot Studio, Microsoft Foundry, and other Microsoft AI experiences.

Common mistakes to avoid

Mistake 1: Treating Copilot Studio and Foundry as competing platforms

Most organizations use both over time. The key question is not which product wins, but which starting point best fits the team, operating model, and governance requirements.

Mistake 2: Choosing only based on agent complexity

Complexity matters, but it should not be the only factor. Builder persona, ownership model, deployment approach, and operational control requirements often have a greater impact on the platform decision.

Mistake 3: Delaying evaluation and governance

Enterprise AI agents require evaluation, monitoring, security, and governance from the beginning. Defining quality, safety, reliability, and operational metrics early reduces production risk.

Teams often begin by asking whether Copilot Studio or Microsoft Foundry is the better product. In practice, successful organizations focus less on platform competition and more on matching the right platform to the right team. Copilot Studio accelerates business-led experiences, while Microsoft Foundry enables developer-led AI engineering. The strongest strategy is creating a framework that allows both to evolve together as requirements mature.

Conclusion

Copilot Studio and Microsoft Foundry are complementary platforms that support different builder personas, development styles, and operating models.

Copilot Studio is a strong starting point for business-led teams that need a managed, low-code experience and rapid time to value. Microsoft Foundry is a strong starting point for developers and architects who require code-first development, secure enterprise boundaries, advanced orchestration, and greater operational control.

Rather than standardizing on a single platform, organizations should adopt a decision framework that helps each team start in the right place and evolve as requirements mature.

Next steps

If you are planning an enterprise AI initiative:

Identify the primary builder persona — business user, IT admin, pro-maker, developer, architect, or AI engineer.
Define the operating model — guided authoring, low-code configuration, code-first engineering, or hybrid delivery.
Assess security, control, and governance requirements.
Plan evaluation and monitoring before production deployment.

Then choose the starting point that best aligns with the team’s needs today while preserving flexibility to expand as the organization’s AI estate grows.

Agents can learn with Memory in Microsoft Foundry Agent Service

jianant — Thu, 16 Jul 2026 16:00:00 GMT

Recurring Failure

We are entrusting AI agents with an increasing range of tasks, from preparing briefs to scheduling meetings, and from filing tax returns to building and debugging production systems. While their capabilities continue to improve rapidly, the experience remains frustrating in a familiar way: the agent keeps making the same mistakes.

A simple example: your agent runs python main.py and hits a ModuleNotFoundError because the project uses uv for dependency management. After several failed attempts, it discovers uv run python main.py works. But next week? Same error, same wasted cycles. With Procedural Memory, the agent learns once and remembers: "In uv-managed projects, always use uv run.

This behavior is not an edge case. Agents rediscover workflows, repeat errors, and waste tokens and time because they do not learn from past experience.

Limits of Existing Memory

Existing memory systems persist information such as facts, user preferences, and conversation history. But it is not sufficient to distill what the agent should do in a given situation. As a result, agents struggle to execute workflows reliably, follow implicit constraints, or avoid repeating failures. The missing piece is not more information—it is procedural knowledge.

Below is an example of learnings that a shopping assistant agent has acquired from past trajectories. It has two fields: Context and Action. Context defines the triggering condition when the learning is applicable, and Action specifies the action pattern that should be followed. When the agent later encounters a similar situation, this learning allows it to filter out unavailable options, reduce distraction, and prevent erroneous actions.

{

"Context": "when a user wants to exchange a product for a different variant",

"Action": "check variants for availability and only present variants that are currently available",

}

Uneven Learning Signal

Not all experiences are equally valuable for learning. As foundation models become more capable, many successful patterns simply reflect existing knowledge, while failures and near-failures are a richer source of improvement.

These high-signal cases expose weaknesses in decision-making, such as incorrect assumptions, missed validations, implicit constraint violations, or premature termination. Prioritizing such experiences helps improve agent reliability rather than merely reinforcing behaviors the model already performs well.

Learning Without Labels

The research gap: Research has shown that agent skills can be distilled from past trajectories. However, bringing these approaches into production remains challenging.

Why production is different: Unlike benchmarks with labeled trajectories, real-world learning signals are buried in action sequences, tool outputs, and user feedback—noisy, incomplete, and often unreliable.

Toward Procedural Memory

Memory in Foundry Agent Service (Create and Use Memory - Microsoft Foundry | Microsoft Learn - explore the documentation to get started) now supports procedural memory. It captures learning signals using LLM-as-a-judge, produces valuable and actionable learnings, and stores them in a database. When an agent comes across tasks that are similar to the context of a learning, the learning gets retrieved and injected into the working context of the agent, which guides the agent to perform the task more efficiently, reliably and accurately.

We conduct systematic analysis on microsoft/STATE-Bench: Benchmark AI Agents on Enterprise Workflows (enterprise workflow tasks like customer support. See Introducing STATE-Bench: A benchmark for AI agent memory | Microsoft Open Source Blog for more information.) and sierra-research/tau2-bench: τ-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (tool-agent-user interactions in domains like retail and airline support). Results show consistent improvements across benchmarks.

Benchmark	Metric	Without PM	With PM	Δ
STATE-Bench	Pass¹	53.3%	58.3%	+5.0%
STATE-Bench	Pass⁵	32.7%	37.3%	+4.6%
Tau-Bench	Pass¹	75.1%	79.8%	+4.7%
Tau-Bench	Pass⁵	49.1%	55.4%	+6.3%

PM stands for procedural memory.
Pass¹: The average success rate across tasks, where each task is attempted up to 5 times. It measures best-effort capability — can the agent eventually solve the task?
Pass⁵: The percentage of tasks that succeed consistently across all 5 attempts. It measures stability and reliability — can the agent solve the task consistently?

Future Directions

Learning from Population-Level Patterns

Individual trajectories are often noisy and unreliable. By aggregating and analyzing similar trajectories, systems can identify consistently effective behaviors and uncover recurring failure modes, enabling more robust and generalizable learning.

Memory Promotion, Demotion, and Continuous Feedback

Procedural memories should evolve with changing environments rather than remain static. Continuous feedback can promote useful memories, demote outdated ones, and keep the memory base aligned with current workflows and user needs.

Procedural Memory Consolidation

As memories accumulate, redundancy and fragmentation can degrade retrieval quality. Consolidating similar memories into higher-level abstractions reduces noise and transforms accumulated experiences into reusable behavioral knowledge.

Get Started with Foundry Memory

Ready to give your agents persistent procedural memory?

Try it now:

- Create and Use Memory - Microsoft Foundry | Microsoft Learn Get started with Foundry Memory documentation

- azure.ai.projects.operations.BetaMemoryStoresOperations class | Microsoft Learn Explore the Foundry Memory API reference.

Go deeper:

- Browse the STATE-Bench repository to test your own agents.

Join the conversation:

- Keep the conversation going in Microsoft Foundry on GitHub and Discord Channel

- Follow [Microsoft Foundry] @MicrosoftFoundry for updates on Foundry Memory GA

Have questions? Drop them in the comments below—we'd love to hear how you're thinking about agent memory in your workflows.'

Choose the right speech and voice model

Anil_Korkut — Thu, 16 Jul 2026 14:00:00 GMT

Choose the right speech and voice model in Microsoft Foundry

The practical approach to selecting the right model is simple: do not ask which speech model is best. Ask what the speech system must guarantee. Once you know whether the job is conversation, translation, transcription, diarization, synthesis, or managed voice-agent orchestration, the model choice becomes much clearer.

Choosing the wrong speech model usually does not fail in a benchmark. It fails in production: a customer interrupts and the agent keeps talking, a meeting transcript has no speaker labels, a translation workflow adds too much delay, or a text-to-speech pipeline sounds good but cannot use the voice your product needs.

Microsoft Foundry now gives builders several speech and voice paths across Azure OpenAI, Voice Live API, Azure Speech in Foundry Tools, and Microsoft AI (MAI) speech models. This guide helps you choose the right surface for the job before you commit to region, quota, pricing, or architecture assumptions.

TL;DR: pick by job, not by model name

Use Azure OpenAI Realtime API when you want direct, model-native speech-to-speech control over WebSocket, WebRTC, or SIP.
Use Voice Live API when you want a managed voice-agent stack with speech recognition, model reasoning, text-to-speech, turn detection, interruption handling, voices, tools, and optional avatars.
Use Azure Speech realtime or batch speech-to-text (STT) when the transcript is the artifact of record.
Use a diarize or Azure Speech batch when speaker labels matter (Who said what).
Use Azure Speech TTS, Custom Neural Voice, gpt-4o-mini-tts, or MAI-Voice-2 when voice quality, identity, language coverage, or long-form narration matters.
Use mai-transcribe-1.5 when world-class, low-cost multilingual STT with phrase/entity biasing is useful and diarization is not required.
Validate region, deployment type, lifecycle, quota, and price in your tenant before treating any model choice as final.

A quick glossary

STT: speech-to-text, also called transcription.
TTS: text-to-speech, also called speech synthesis.
VAD: voice activity detection, used to decide when a speaker has started or stopped talking.
SIP: Session Initiation Protocol, commonly used for telephony integration.
PTU: provisioned throughput units, a reserved-capacity deployment option where supported.

Start with the user experience

The first decision is not which model is smartest. It is what the system must produce and how fast it must respond.

If a user speaks to an agent and expects the agent to listen, reason, be interrupted, and speak back in real time, you are building a speech-to-speech system. Choose between direct Azure OpenAI Realtime API and managed Voice Live API.

If a user uploads or records audio and needs a transcript, you are building STT. Use transcription APIs or Azure Speech batch, not a conversational voice model.

If text goes in and speech comes out, you are building TTS. Use a TTS surface, not a speech-to-speech model.

If speech must be translated while the speaker is still talking, you are building live speech translation. That is different from file translation to English with Whisper.

If the app can wait for a normal request/response turn, turn-based audio chat is often simpler than a realtime session.

Scenario map

Interactive experiences

Scenario	Start with	Use when	Watch for
Turn-based audio chat	gpt-audio, gpt-audio-mini, gpt-audio-1.5	The user records or sends audio, then waits for a response	No live barge-in; app owns turns and playback. Not ideal for real-time and latency sensitive scenarios
Composable voice chatbot	Azure Speech STT + Azure OpenAI or Foundry model + Azure Speech TTS	You want separate control over transcript, reasoning, and voice	More orchestration; app owns interruption and latency
Direct realtime voice agent	gpt-realtime, gpt-realtime-mini, gpt-realtime-1.5, gpt-realtime-2	You want model-native speech-to-speech over WebSocket, WebRTC, or SIP. Low latency, fast results.	Persistent session; app owns session behavior
Managed realtime voice agent	Voice Live API	You want STT, model reasoning, TTS, turn detection, voices, tools, and avatars in one surface. Advanced features like; noise reduction, enhanced turn detection	Separate quota/pricing; requires persistent WebSocket or WebRTC session; SIP not directly supported. Not ideal for strict word for word speech
Live translation	gpt-realtime-translate or Azure Speech Translation	The speaker is still talking while translation happens	Use openai/v1/realtime/translations?model=gpt-realtime-translate
Live captions	gpt-realtime-whisper or Azure Speech realtime STT	You need source-language text during a live session, streaming transcription	Not a durable diarized transcript by itself

Speech artifacts and workflows

Scenario	Start with	Use when	Watch for
File transcription	gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper, mai-transcribe-1.5	Audio is uploaded or bounded	File-size limits, preview status, diarization needs
Diarization	gpt-4o-transcribe-diarize or Azure Speech batch	Speaker labels matter	Not a Realtime API model; long files often fit batch better
Domain-specific STT	Azure Speech realtime STT or custom speech	Domain terms, accents, language detection, or punctuation matter	Feature support varies by region and API
Text-to-speech	gpt-4o-mini-tts, tts, tts-hd, Azure Speech TTS	Input is text and output is speech	TTS capabilities vary significantly across OpenAI TTS, Azure Speech TTS,
Branded or expressive voice	Azure Speech neural/HD voices, Custom Neural Voice, MAI-Voice-2	Voice identity, style, pronunciation, or long-form narration matters	Custom voice and voice prompting require access controls and consent
Privacy-sensitive or on-prem speech	Azure Speech batch storage and Speech containers	Data retention, local processing, or compliance dominates	Verify container support, pricing, and compliance scope

How the main surfaces work

Azure OpenAI Realtime API: direct model-native voice

Use the Azure OpenAI Realtime API when you want low-latency speech-to-speech and your app can manage the live session. The GA WebSocket path is:

wss://{resource}.openai.azure.com/openai/v1/realtime?model={deployment}

WebRTC uses /openai/v1/realtime/client_secrets and /openai/v1/realtime/calls. SIP call control uses /openai/v1/realtime/calls/{call_id}/.... Do not add date-based api-version parameters to the GA /openai/v1/realtime path. The Realtime API supports up to 32,000 input tokens and 4,096 output tokens.

Connection method	Use case	Latency	Best for
WebRTC	Client-side applications	~100 ms	Web apps, mobile apps, browser-based experiences
WebSocket	Server-to-server	~200 ms	Backend services, batch processing, custom middleware
SIP	Telephony integration	Varies	Call centers, IVR systems, phone-based applications

Choose realtime path when you need tight control over audio buffering, session state, function calls, and telephony. Choose Voice Live instead when you want Azure Speech voices, semantic turn detection, custom speech, custom voice, avatar output, or broader model selection without wiring those pieces yourself.

Voice Live API: managed voice agents

Voice Live API is the managed speech-to-speech surface in Azure Speech in Foundry Tools. It combines STT, model reasoning, TTS, turn detection, interruption handling, noise suppression, echo cancellation, voices, avatars, and function calling behind one realtime interface.

The WebSocket endpoint shape is:

wss://{resource}.services.ai.azure.com/voice-live/realtime?api-version=2026-04-10&model={model}

Older resources can use:

wss://{resource}.cognitiveservices.azure.com/voice-live/realtime?api-version=2026-04-10&model={model

Voice Live can also connect to Microsoft Foundry Agent Service with agent_id and project_id parameters. Microsoft Entra authentication is recommended. API keys are also supported.

A typical Voice Live session starts with session.update:

{ "type": "session.update", "session": { "instructions": "You are a concise customer support voice agent.", "modalities": ["text", "audio"], "turn_detection": { "type": "azure_semantic_vad", "silence_duration_ms": 500, "interrupt_response": true }, "input_audio_noise_reduction": { "type": "azure_deep_noise_suppression" }, "input_audio_echo_cancellation": { "type": "server_echo_cancellation" }, "voice": { "name": "en-US-AvaNeural", "type": "azure-standard" } } }

Voice Live is useful for contact centers, automotive assistants, accessibility experiences, public-service agents, learning companions, and any app where conversation quality depends on turn-taking as much as model intelligence.

Voice Live supports a broad model set: GPT-Realtime, GPT-4o, GPT-4.1, GPT-5 family models, Phi preview models, azure-realtime, Agent Service, and BYOM patterns.

Key tradeoffs:

Voice Live has separate quota and pricing from Azure OpenAI deployment tables.
Pricing is tiered by selected model: Pro, Basic, and Lite.
Custom speech, custom voice, and avatar features can add training, hosting, or usage charges.
Custom voice and avatar access can be limited or approval-gated.
WebRTC support is preview.
SIP is documented for Azure OpenAI Realtime API, but not directly supported in Voice Live, yet.

Azure Speech: the production speech service layer

Azure Speech in Foundry Tools is more than a model list. It is a service layer for realtime STT, batch STT, TTS, translation, custom speech, custom voice, privacy controls, and containers.

Use Azure Speech when the speech artifact matters: a durable transcript, a brand voice, a tuned recognizer, a controlled storage boundary, or a Speech SDK workflow.

For STT, Azure Speech supports realtime streaming and batch patterns. It can also support capabilities such as language detection, phrase hints, custom speech models, punctuation, capitalization, keyword spotting, pronunciation assessment, and diarization. Exact diarization limits and feature combinations should be verified in the current Speech docs for the selected API and region.

For TTS, Azure Speech is the broader production path for voice catalog depth, locale coverage, pronunciation control, and branded voice. Custom Neural Voice lets teams create a unique voice under limited-access controls.

For privacy-sensitive workloads, Azure Speech can help define the data boundary. Batch jobs can use customer-controlled Azure Storage for input and output. Speech containers can support customer-managed or on-premises processing for supported features. Validate the exact feature, container image, compliance scope, and region before making regulated production commitments.

Model and surface quick reference

Model or surface	What it is	Best use
Azure Speech realtime STT	Speech service streaming STT	Live captions and tuned recognition
Azure Speech batch transcription	Speech service async STT	Large files, durable transcripts, timestamps, diarization workflows
Azure Speech Translation	Speech service translation	Speech-to-text or speech-to-speech translation workflows
Azure Speech TTS / Custom Neural Voice / HD voices	Speech service synthesis	Branded voices, broad locales, long-form assets
Speech containers	Customer-managed Speech runtime	On-premises or controlled-boundary processing where supported
mai-transcribe-1.5	Microsoft AI multilingual STT	Top-tier STT with phrase/entity biasing and style options (mai-transcribe-1 is the earlier version)
MAI-Voice-2	Microsoft AI expressive multilingual TTS	Long-form, expressive, or voice-prompted speech where preview terms fit. (MAI-Voice-1 is the earlier version)
MAI-Voice-2-Flash	Roadmap/watch item	Design production low-latency TTS around it after SKU, endpoint, price, and regions are public

Capability Matrix

Model name	Description	Good for	Example use cases	Input types	Output types	Limitations	Max request tokens	Pricing
mai-transcribe-1.5	Microsoft AI (MAI) first-party speech-to-text model in the Azure Speech LLM Speech API (Foundry Tools) — not an Azure OpenAI model. Multilingual transcription model with entity biasing for names, brands, and industry terms; supports 43 languages and excels in noisy environments.	High-accuracy multilingual batch transcription where domain-term biasing and readable output matter; efficient long-audio STT.	Meeting/call transcription, media & podcast transcription, multilingual voice-note transcription, domain (medical/legal/brand) STT via phrase lists.	Audio file < 300 MB (WAV, MP3, FLAC; ~up to 2 hrs); optional phraseList, locales, transcribeStyle.	Text transcript (readability-optimized or verbatim).	Public preview	file-based — input limited to < 300 MB per file (~2 hrs); token limit will be published.	$0.36 / hour
gpt-4o-mini-tts (2025-12-15)	TTS model powered by GPT-4o mini with voice-style guidance; newer snapshot of gpt-4o-mini-tts.	Newer instruction-controlled speech (styled delivery).	Styled speech, assistant voice, narration.	Text (+ natural-language style "instructions").	Audio (mp3, wav, opus, aac, flac, pcm).	not a transcription/chat model.	4,096 characters	Global: text input $0.60, audio output $12.00; Data Zone: text input $0.66, audio output $13.20; Regional: text input $0.66–$0.792, audio output $13.20–$15.84 ( / 1M tokens)
gpt-4o-mini-tts (2025-03-20)	Small, fast, steerable text-to-speech model powered by GPT-4o mini (Azure OpenAI / OpenAI) — you can instruct not just what is said but how (tone, emotion, accent, pace).	Low-cost, expressive, controllable speech generation at scale; developer-friendly TTS with voice-style prompting; branded/styled speech assets when natural-language style instructions help.	Voiceovers and narration, spoken replies in voice agents, read-aloud/accessibility, IVR prompts, customer-service audio; branded speech-asset pipeline (style-controlled).	Text (+ natural-language style "instructions").	Audio (mp3, wav, opus, aac, flac, pcm).	Not a transcription/chat model; no verbatim-synthesis guarantee	4,096 characters	Global: text input $0.60, audio output $12.00; Data Zone: text input $0.66, audio output $13.20; Regional: text input $0.66–$0.792, audio output $13.20–$15.84 ( / 1M tokens)
gpt-audio (2025-08-28)	Audio-capable GPT chat model for turn-based spoken conversation over Chat Completions (not the full-duplex Realtime API); accepts and returns audio and/or text.	Asynchronous/turn-based voice chat and audio generation without building a realtime session.	Voice Q&A bots, audio analysis + spoken summary, spoken-language tutoring, request/response IVR; audiobook generation.	Text + audio.	Text + audio.	Not the Realtime API — no live barge-in/interruption; turn-based latency; ~20 MB audio-file limit in quickstart; use Chat Completions (not Responses) for audio chat.	128,000 context; up to 16,384 output tokens.	Text input $2.50, text output $10.00; audio input $40.00, audio output $80.00 / 1M tokens
gpt-audio-mini (2025-10-06)	Smaller, ~3× cheaper sibling of gpt-audio for turn-based audio chat; trades some capability for lower cost/latency.	High-volume or cost-sensitive turn-based audio chat; simplest path for a voice-enabled chat feature — user records audio, submits it, and gets an audio answer after a normal turn (simpler than a realtime session).	Large-scale voice bots, budget audio Q&A, lightweight spoken assistants; audio generation.	Text + audio.	Text + audio.	Same non-Realtime constraints as gpt-audio; lower capability; do not use this path if users expect live interruption/barge-in; older mini snapshot	128,000 context (gpt-audio family); ~16,384 output	Text input $0.60, text output $2.40; audio input $10.00, audio output $20.00 / 1M tokens
gpt-4o-audio-preview (2024-12-17)	Preview GPT-4o-family audio model for audio and text generation (turn-based, over Chat Completions).	Higher-quality turn-based audio chat.	Voice Q&A, spoken assistant responses, audio understanding, audio response generation.	Text, audio, text + audio; no image/video.	Text, audio, text + audio; no image/video.	Preview; not strict STT/TTS only; not ideal for batch transcripts, diarization, or low-latency interruption; structured outputs, fine-tuning, and predicted outputs not supported.	128,000 context; up to 16,384 output tokens.	Global: text input $2.50, text output $10, audio input $40, audio output $80 (/1M Tokens)
gpt-4o-mini-audio-preview (2024-12-17)	Preview audio model for audio and text generation (smaller/lower-cost, turn-based).	Lower-cost/faster turn-based audio chat.	Audio Q&A, short voice assistant replies, app-generated spoken responses.	Text, audio, text + audio; no image/video.	Text, audio, text + audio; no image/video.	Preview; not optimized for live barge-in.	128,000 context; up to 16,384 output tokens.	Global: text input $0.15, text output $0.60, audio input $10, audio output $20 (/1M Tokens)
gpt-audio-1.5 (2026-02-23)	Newer generation of the audio chat model with native audio-in/audio-out over Chat Completions; improved quality over gpt-audio.	Higher-quality turn-based multimodal (audio + text) chat and audio generation; good starting point for a voice-enabled chat feature (record → submit → audio reply after a normal turn).	Voice assistants, audio summarization, turn-based spoken translation, audio chat with tool/function calling; more natural audio output; audio generation.	Text + audio.	Text + audio.	Turn-based (not Realtime — no live barge-in); higher audio-token pricing; not for experiences that require live interruption.	128,000 context; up to 16,384 output tokens.	Text input $2.50, text output $10.00; audio input $32.00, audio output $64.00 / 1M tokens
gpt-realtime (2025-08-28)	GA speech-to-speech Realtime model for low-latency, full-duplex voice over WebSocket/WebRTC/SIP (/openai/v1/realtime); native audio in and out.	Production real-time voice agents needing natural, low-latency, interruptible conversation; general direct realtime speech-to-speech.	Live customer-support voice agents, real-time assistants, telephony bots (SIP), interactive voice experiences; contact-center agents (directly or as a Voice Live model).	Audio + text (image where supported).	Audio + text.	Requires a persistent realtime session and app-side orchestration; fewer composition options than managed Voice Live; not a strict transcript/TTS tool; do not use a speech-to-speech model as the meeting source-of-record transcript.	32,000 context; up to 4,096 output tokens.	Global: text input $4.00, cached text input $0.40, text output $16.00; audio input $32.00, cached audio input $0.40, audio output $64.00; image input $5.00, cached image input $0.50 / 1M tokens
gpt-realtime-mini (2025-10-06)	Cost-efficient sibling of gpt-realtime for low-latency speech-to-speech; cheaper per token.	High-volume or cost-sensitive real-time voice agents; lower-cost realtime voice experiences.	Scaled voice bots, lightweight realtime assistants, cost-sensitive live caption-and-respond flows; low-latency voice streaming.	Audio + text (image where supported).	Audio + text.	Lower capability than gpt-realtime; realtime session orchestration still required; older mini snapshot	32,000 context; up to 4,096 output tokens	Global: text input $0.60, cached text input $0.06, text output $2.40; audio input $10.00, cached audio input $0.30, audio output $20.00; image input $0.80, cached image input $0.08/ 1M tokens
gpt-realtime-mini (2025-12-15)	Cost-efficient sibling of gpt-realtime for low-latency speech-to-speech; newer snapshot with quality updates.	Lower-cost realtime voice with newer quality updates.	Voice agents needing lower cost and latency; low-latency voice streaming.	Audio + text (image where supported).	Audio + text.	Requires persistent realtime session; realtime orchestration required.	32,000 context; up to 4,096 output tokens.	gpt-realtime-mini-2025-12-15 Global: text input $0.60, cached text input $0.06, text output $2.40; audio input $10.00, cached audio input $0.30, audio output $20.00; image input $0.80, cached image input $0.08 / 1M tokens
gpt-realtime-1.5 (2026-02-23)	Realtime speech-to-speech model variant (production/GA) with audio/text (and image) input and audio/text output over WebRTC/WebSocket/SIP.	Production real-time voice agents needing enterprise governance (RBAC, logging, policy); newer full realtime voice experiences; newer direct realtime voice.	Enterprise voice assistants, live support, real-time translation apps, telephony agents; conversational agents, tool-driven voice flows.	Audio + text + image (input).	Audio + text.	Realtime session/orchestration required; function calling supported but (per one source) structured outputs / fine-tuning not, do not use as the source-of-record transcript.	32,000 context; up to 4,096 output tokens.	Global: text input $4.00, cached text input $0.40, text output $16.00; audio input $32.00, cached audio input $0.40, audio output $64.00; image input $5.00, cached image input $0.50 / 1M tokens
gpt-realtime-2 (2026-05-07)	Speech-to-speech Realtime model with built-in reasoning (adjustable reasoning.effort) and response phases (commentary preamble vs final_answer); audio in → audio out.	Real-time voice needing stronger instruction-following/reasoning and lower perceived latency via preambles; route complex resolution here when deeper multi-step reasoning matters (e.g., escalations in a multilingual support flow); complex voice support, multi-step voice assistants.	Complex voice agents/copilots, reasoning-heavy support flows, tool-announcing voice assistants.	Audio, text, image.	Audio, text (multi-item responses with commentary + final_answer phases).	**Public preview; stricter instruction-following may need prompt rephrasing; some features may be constrained; do not use as the source-of-record transcript	256,000	Global: audio input $32.00, cached audio input $0.40, audio output $64.00; text input $4.00, cached text input $0.40, text output $24.00; image input $5.00, cached image input $0.50 / 1M tokens
gpt-realtime-translate	Purpose-built model for continuous, real-time audio translation —(Different languages) processes live streams without chunking and emits translated speech + transcript as speech unfolds. Global Standard in Foundry.	Minimal-latency live translation that keeps pace with the speaker; multilingual customer support with live translated speech plus a translated transcript.	Live multilingual events/conferences/broadcasts, cross-language support calls, multilingual voice interfaces, live media localization, international meetings; multilingual customer support (route complex resolution to gpt-realtime-2 when deeper reasoning matters).	Audio (live stream).	Translated speech (audio) + translated transcript (text) in the target language.	Realtime API only; Azure exposes no distinct specialized endpoint path (uses generic Realtime); pair with gpt-realtime-whisper for source captions (or Azure Speech realtime STT for the source transcript); for durable compliance transcripts/diarization/word-level timestamps use Azure Speech batch	32,000 context; up to 4,096 output tokens.	$2.04 / hour; equivalently $0.034 / minute
gpt-realtime-whisper	Streaming transcription model that converts live audio to text in real time (Same languages); runs in parallel with speech-to-speech/translation models. Global Standard in Foundry.	Low-latency live captions/subtitles and source-language transcript during live audio; source transcript to run alongside gpt-realtime-translate in multilingual support (Azure Speech realtime STT is an alternative).	Live event captioning, compliance/quality monitoring transcripts, live-stream analytics, source captions paired with gpt-realtime-translate; live meeting captions.	Audio (live stream).	Text (streaming transcript).	Text only (not diarization/file STT); Realtime API only; for the durable meeting transcript use Azure Speech batch (large files, diarization, word-level timestamps).	32,000 context; up to 4,096 output tokens.	$1.02 / hour; equivalently $0.017 / minute; distinct from plain whisper-1
gpt-4o-transcribe (2025-03-20)	Preview GPT-4o-based speech-to-text model for high-accuracy file and request transcription via /audio/transcriptions.	Accurate prerecorded/file transcription with GPT-4o quality.	Meeting/interview transcription, media transcription, voice-note transcription, transcripts feeding downstream LLMs; captions; analysis.	Audio (+ optional text prompt).	Text transcript.	Preview; file-based (not live speech-to-speech); ~25 MB file limit; no diarization; not for deterministic word-for-word synthesis; for regulated/on-prem needs, start from the data boundary — consider Azure Speech batch (customer-controlled storage) or Speech containers.	16,000 context / 2,000 output tokens; request also bounded by ~25 MB audio file.	Global: audio input $6.00, text input $2.50, text output $10.00; Data Zone: audio input $6.60, text input $2.75, text output $11.00; Regional: audio input $6.60–$7.92, text input $2.75–$3.30, text output $11.00–$13.20 ( / 1M tokens). Docs equivalent: $0.36 / hour
gpt-4o-mini-transcribe (2025-03-20)	Preview smaller, faster, lower-cost GPT-4o transcription model for file/request STT.	Budget / high-throughput file transcription where speed and cost matter.	Bulk transcription, budget live-caption input, voice-note transcription, cost-sensitive STT pipelines.	Audio (+ optional text prompt).	Text transcript.	Preview; file-based; ~25 MB file limit; lower accuracy than full gpt-4o-transcribe; no diarization.	16,000 context / 2,000 output tokens; request also bounded by ~25 MB audio file.	Global: audio input $3.00, text input $1.25, text output $5.00; Data Zone: audio input $3.30, text input $1.375, text output $5.50; Regional: audio input $3.30–$3.96, text input $1.375–$1.65, text output $5.50–$6.60 / 1M tokens. Docs equivalent: $0.18 / hour
gpt-4o-mini-transcribe (2025-12-15)	Preview STT model powered by GPT-4o mini with improved transcription accuracy and robustness (newer snapshot).	Faster, lower-cost STT.	Captions, transcripts, high-volume audio files.	Audio (+ optional text prompt).	Text transcript.	Preview; ~25 MB file limit; no diarization.	16,000 context / 2,000 output tokens; request also bounded by ~25 MB audio file.	Global: audio input $3.00, text input $1.25, text output $5.00; Data Zone: audio input $3.30, text input $1.375, text output $5.50; Regional: audio input $3.30–$3.96, text input $1.375–$1.65, text output $5.50–$6.60 / 1M tokens. Docs equivalent: $0.18 / hour
whisper	General-purpose STT model via Azure OpenAI for multilingual transcription and English translation (/audio/transcriptions, /audio/translations).	Robust multilingual transcription and non-English → English translation of prerecorded audio.	Multilingual transcription (~100 languages), translating foreign-language audio to English text, subtitle generation, batch STT.	Audio file.	Text (transcript; or English translation).	25 MB file limit; no Azure OpenAI diarization; processes ~30-sec windows; legacy Standard/Regional deployment; no speaker labels; for diarization/word-level timestamps or durable/regulated transcripts, use Azure Speech batch or gpt-4o-transcribe-diarize.	25 MB file limit per request	whisper-1: $0.36 / hour; Retail plain Whisper Global: $0.36 / hour; Regional: $0.396–$0.4752 / hour. Keep separate from gpt-realtime-whisper
gpt-4o-transcribe-diarize (2025-10-15)	Preview GPT-4o transcription model that adds speaker diarization — a transcript with speaker-attributed segments (Speaker 1/2…). Request/file-based.	File transcription where "who said what" matters; meeting transcript/minutes for bounded files when its limits fit.	Meeting minutes with speakers, interview/podcast transcripts, call-center QA, multi-party conversation analysis.	Audio file.	Text transcript with speaker-labeled segments (speaker metadata).	Preview; (not Realtime API; no realtime/streaming diarization); generic speaker labels; best with few, clear speakers; no prompts, logprobs, or timestamp granularities; ~25 MB file limit; for large files or word-level timestamps use Azure Speech batch instead.	~25 MB file limit; underlying text context up to 128K but bounded by file limits.	Text input $2.50, text output $10.00; audio input $6.00, audio output N/A / 1M tokens
tts	Azure OpenAI's standard (preview) text-to-speech deployment name (OpenAI analog: tts-1); converts text to natural speech, optimized for speed.	General-purpose, real-time-friendly TTS at standard quality.	Voiceovers, spoken responses, narration, accessibility read-aloud; IVR prompts, short assistant replies, notifications.	Text.	Audio (mp3, opus, aac, flac, wav, pcm).	Preview; legacy path; standard (non-HD) quality; legacy Standard/Regional deployment; 4,096-char input; fewer newer style controls	4,096 characters	$15.00 / 1M characters
tts-hd	HD variant (preview) of the Azure OpenAI TTS deployment (OpenAI analog: tts-1-hd) — higher-quality, more natural speech, optimized for quality, slightly higher latency.	Higher-fidelity narration / branded audio where quality outweighs latency; for true branded voice identity, language coverage, or long-form quality, consider Azure Speech Custom Neural Voice, HD voices, or MAI-Voice-2.	Audiobooks, premium voiceovers, high-quality narration, brand voice assets; polished customer-facing audio.	Text.	Audio (mp3, opus, aac, flac, wav, pcm).	Preview; slower and costlier than speed-optimized TTS; legacy Standard/Regional; 4,096-char input.	4,096 characters	$30.00 / 1M characters
gpt-4o-realtime-preview (2025-06-03)	Preview GPT-4o realtime model for low-latency speech-in/speech-out conversations; superseded by the GA gpt-realtime family and on the retirement path.	Legacy/preview realtime voice agents — new work should use gpt-realtime (GA).	Live support agents, voice assistants, realtime translators, WebRTC apps, WebSocket audio flows, SIP/telephony demos.	Audio, text, image (through Realtime conversation items).	Audio + text (transcript/events).	superseded by gpt-realtime	32,000 context; up to 4,096 output tokens.	TBD
gpt-4o-realtime-preview (2024-12-17)	Preview GPT-4o realtime model for low-latency speech-in/speech-out conversations (older snapshot).	Early realtime voice prototypes.	Voice agents, live assistants, WebRTC/WebSocket audio flows.	Audio, text, image (through Realtime conversation items).	Audio + text (transcript/events).	Preview/legacy-watch; older preview snapshot; not recommended for production and does not follow the standard Azure OpenAI lifecycle; superseded by gpt-realtime GA.	16,000 context; up to 4,096 output tokens.	Global: text input $5.00, text output $20.00, audio input $40.00, audio output $80.00
gpt-4o-mini-realtime-preview (2024-12-17)	Smaller preview GPT-4o realtime model for cost / lower-latency audio processing.	Lower-cost preview voice-agent experiments.	Simple voice assistants, prototypes, browser/mobile realtime audio.	Audio, text, image (through Realtime conversation items).	Audio + text (transcript/events).	128,000 input / 4,096 output, but Realtime API docs say 32,000 input / 4,096 output	32,000 context (operational; model table lists 128,000); up to 4,096 output tokens.	Global: text input $0.60, text output $2.40, audio input $10.00, audio output $20.00
MAI-Voice-2	Microsoft AI (MAI) expressive multilingual TTS; 18+ languages.	Long-form, expressive, or voice-prompted speech.	Long-form content, emotion/style speech, multi-speaker scenarios, approved voice prompting, branded/expressive voice assets where voice identity, style, pronunciation, or long-form narration matters.	Text; approved voice prompting/reference clips (5–60 seconds recommended).	Audio.	Preview terms; custom voice and voice prompting require access controls	5–60 second recommended reference clips for voice prompting.	$22.00 / 1M characters.
MAI-Voice-2-Flash	Ultra-low-latency / low-cost TTS	Faster/lower-cost speech use cases Speech / TTS — real-time variant	Future low-latency TTS planning.	Not found.	Not found.	SKU, endpoint, price, and regions are TBA	TBA	Lower cost than MAI-Voice-2 but not announced yet

Transcription, diarization, and captions

For files, use /audio/transcriptions rather than a speech-to-speech model. The core Azure OpenAI choices are whisper, gpt-4o-transcribe, gpt-4o-mini-transcribe, and gpt-4o-transcribe-diarize.

Use gpt-4o-transcribe when accuracy matters most, and gpt-4o-mini-transcribe when cost matters more. Use gpt-4o-transcribe-diarize when a bounded file needs speaker-aware output. It is not a Realtime API model.

Use Azure Speech batch when files are long, files exceed synchronous limits, word-level timestamps matter, or mature diarization workflows are required. This is also the right place to evaluate customer-controlled storage and retention.

Use gpt-realtime-whisper or Azure Speech realtime STT for live captions. Use Azure Speech realtime STT when the product needs phrase hints, custom speech tuning, language detection, punctuation, or other Speech service features.

Use mai-transcribe-1.5 when preview multilingual STT with phrase/entity biasing is useful and diarization is not required. Current research records 43 languages, WAV/MP3/FLAC input under 300 MB, optional language forcing, phrase lists, and a transcribeStyle choice for readable or verbatim output.

Text-to-speech and voice assets

If text goes in and speech comes out, start with a TTS surface.

Use gpt-4o-mini-tts when style instructions are important. It supports instructions for accent, emotion, intonation, speed, and similar output qualities. OpenAI docs list voices such as alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse, marin, and cedar.

Use legacy tts and tts-hd only with care.

Use Azure Speech TTS or Custom Neural Voice when voice identity, broad locale support, pronunciation control, or brand voice matters. Use Voice Live when speech output is part of a realtime agent. Use MAI-Voice-2 when expressive quality, long-form content, emotion/style, multi-speaker scenarios, or approved voice prompting matter more than ultra-low latency.

MAI-Voice-2 has 18+ languages and 5-60 second recommended reference clips.

For enterprise design, validate five things in the target tenant: region, deployment type, lifecycle, quota, and price. For Azure Speech features, also validate privacy posture, storage ownership, container support, and compliance scope.

Practical architecture patterns

Multilingual customer support

Use gpt-realtime-translate for live translated speech and translated transcript. Run gpt-realtime-whisper or Azure Speech realtime STT when you also need the original source transcript. Route complex resolution to gpt-realtime-2 when deeper reasoning matters. Use Azure Speech batch for durable compliance transcripts, diarization, or word-level timestamps.

Choose Voice Live when the agent needs managed turn detection, noise suppression, echo cancellation, Azure Speech voices, custom voice, function calling, or avatar output in one surface.

Contact-center voice agent

Use Voice Live API with GPT-Realtime, GPT-4o, GPT-4.1, GPT-5 family, Phi, or azure-realtime models. Configure semantic VAD, barge-in, noise suppression, echo cancellation, voice output, and tool calls for account lookup or workflow actions.

Use WebSocket for server-to-server integration. Evaluate WebRTC preview for browser or mobile client audio. Validate default quota, model tier, custom voice/avatar access, and region.

Meeting transcript and summary

Do not use a speech-to-speech model as the source of record. For live captions, use gpt-realtime-whisper or Azure Speech realtime STT. For the durable transcript, use Azure Speech batch when files are large, diarization is needed, or word-level timestamps matter. Use gpt-4o-transcribe-diarize for bounded files when its limits fit.

Voice-enabled chat feature

If the user records audio, submits it, and receives an audio answer after a normal turn, start with gpt-audio-mini or gpt-audio-1.5. It is simpler than a realtime session. Do not use this path if users expect live interruption.

Branded speech asset pipeline

Use gpt-4o-mini-tts when style instructions are useful. Use Azure Speech TTS, Custom Neural Voice, HD voices, or MAI-Voice-2 when voice identity, language coverage, pronunciation control, or long-form quality matters. Validate consent, access controls, voice availability, and pricing before launch.

Regulated transcript or on-prem speech

Start from the data boundary, not the model name. Azure Speech batch can keep input and output files in customer-controlled Azure Storage. Speech containers can support customer-managed or on-premises processing for supported features. Validate container availability, feature coverage, region, pricing, and compliance scope.

Next steps

If you are choosing a speech architecture now:

Prototype the transport first. Build the smallest WebSocket, WebRTC, Speech SDK streaming, REST transcription, TTS, container, or batch proof of concept that matches the user interaction.
Validate availability in your tenant. Check model region, deployment type, lifecycle, quota, price, resource type, and compliance posture.
Test the failure mode that matters. For voice agents, test interruption and noisy audio. For transcripts, test diarization and timestamps. For TTS, test pronunciation, voice identity, and long-form quality.

Source-of-truth docs

Get started

Answers You Can Trust: Grounding Enterprise Agents with Foundry IQ

Paulo_Lacerda — Wed, 15 Jul 2026 20:00:00 GMT

Enterprise AI has a trust problem, and it is not about fluency. A model will answer almost any question with confidence, whether or not it actually knows. The moment a legal or support team catches it inventing a policy that does not exist, they stop trusting it for anything that matters. What makes an enterprise agent useful is not how well it writes. It is whether every answer comes from your own knowledge and shows where it came from.

That shifts the hard part of the work from the model to the grounding, and the industry now treats it that way. The trajectory is clear: by 2030, universal semantic layers will be treated as critical infrastructure, alongside data platforms and cybersecurity. The shared, governed layer of meaning that lets an agent answer in your business terms is moving from nice-to-have to non-negotiable foundation. The differentiator is no longer the base model. It is how well the system is grounded in what only your organization knows, and how tightly that grounding is governed.

Why Ungrounded AI Fails Enterprises

A raw model answers from a fixed snapshot of public text. That is fine for general knowledge and useless for your policies, your contracts, and your product specifics, none of which were in its training data. Ask it something particular to your business and it still answers, just from the nearest pattern it has seen. That is how you get a confident paragraph citing a policy you never wrote.

The failure has three parts, and they compound. Coverage: the model does not have your internal knowledge, so instead of admitting the gap it answers from its training and hallucinates, a fluent, confident paragraph that happens to be wrong. Traceability: even when it happens to be right, nothing points to a source, so no one can verify it and no auditor will accept it. Permissions: a system that pools every document into one searchable pile will surface a salary letter or an unreleased contract to whoever asks, because the model has no idea who is on the other side.

Grounding fixes all three at once, and it is more than pasting documents into a prompt. Done properly it means retrieving the right passages from your live corpus, answering only from them, showing the citation, and enforcing each user's permissions at the moment of retrieval.

What Grounding Actually Takes

Fixing all three takes a dedicated knowledge layer, the part of the system that retrieves the right passages from your content and hands the model only what it can cite. GPT-RAG is an open-source solution accelerator you deploy in your own Azure subscription, and its knowledge layer is Microsoft Foundry IQ. When a user asks a question, the agent does not recall an answer from training. It runs agentic retrieval over your own content.

Agentic retrieval breaks the question into parts, searches your sources in parallel, reranks the candidates for relevance, and answers only from the passages that come back, each carrying a citation. That is what separates it from a basic search-and-stuff pipeline, and it is what makes an answer both grounded and verifiable.

One question becomes focused subqueries, runs in parallel across your sources, and is reranked so the model answers only from the top cited passages

A Concrete Example: One Question, One Cited Answer

Take the question straight from the demo. A signed-in user opens the agent and asks: “In which model year did VW introduce the electronic fuel injection system?”

A large model might guess this correctly from something it read on the public web, and that guess is exactly the dependency we want to remove. In the demo the answer lives in a specific engineering document we control, and the whole point is that the agent answers from that document rather than from whatever the model happens to remember. So instead of guessing, it retrieves the relevant passage and composes a short, direct answer: VW introduced the system on its 1975 models. Next to the answer sits a blue citation, vw-fuel-system.pdf, the exact source the answer was built from.

Two things happened there that a plain search box cannot do. The answer is grounded, so it comes from your document rather than the model's memory, and it is cited, so the reader can open the source document behind it and confirm it. It is also permission-aware: retrieval ran as that specific user, so the document was only in play because they were allowed to see it. Change the user and the same question can return a different set of sources, or none, with no special prompting.

One question, answered end to end, grounded in the source and shown with its citation.

What Foundry IQ Actually Does

Foundry IQ is the knowledge layer that sits behind your agents. Instead of every application building and maintaining its own retrieval stack, the chunking, the indexing, and a separate connector for each source, you define a Knowledge Base once and reuse it. It runs on Azure AI Search, and Microsoft ships it as a first-party, SLA-backed retrieval service, so you inherit a supported engine rather than hand-rolled plumbing. As of Build 2026, Foundry IQ knowledge bases are generally available.

A Knowledge Base connects one or more knowledge sources, and those sources come in two flavors. Indexed sources are your content pre-processed into a searchable index. Point the Knowledge Base at your corpus, the files in Blob Storage and the content indexed in AI Search, and that corpus becomes the ground truth the agent answers from. Remote sources are live systems the Knowledge Base queries at retrieval time instead of indexing up front. Because sources are configuration and not code, extending what the agent knows is a matter of connecting another source, not rebuilding the app.

Those remote sources go beyond documents. The same Knowledge Base can connect a Fabric ontology that captures how your business entities relate, Fabric data agents that answer from live systems, and Work IQ for a person's Microsoft 365 context, so structured, up-to-the-minute signals ground the agent alongside your indexed files.

When you are ready to wire this up yourself, GPT-RAG's grounding sources overview is the operator guide. It walks through each supported source kind and the concrete steps to enable it on your Knowledge Base.

The intelligence is in how it retrieves. Basic retrieval throws your raw question at a single index and hopes the top hits are relevant. Agentic retrieval treats the question as a task: it decomposes a complex query, searches the sources in parallel, reranks the candidates by relevance, and hands the model a focused, well-ordered set of passages to work from. Microsoft continues to ship quality improvements to this path, and it is the layer that turns “the model said so” into “here is the source.”

Permissions are enforced at query time. The agent forwards the signed-in user's identity to retrieval as an on-behalf-of token, so the Knowledge Base trims results to what that person is allowed to see before the model receives anything. This is document-level security enforced on the retrieval side, not a filter the model could quietly skip. Two people can ask the identical question and get answers built from two correctly scoped views of the same corpus.

Beyond Documents: Where Work IQ and Fabric IQ Fit

Foundry IQ is one member of a family that Microsoft groups under Microsoft IQ, its intelligence layer for connecting knowledge to agents. Three grounding domains matter for an enterprise agent, because they answer three different kinds of question. The split is what separates an agent that knows “what is our refund policy” from one that also knows “what is on my calendar” and “what did we sell last quarter.”

Foundry IQ grounds on your authoritative content: the policies, product documentation, and reference material you curate into a corpus. It is the shared organizational source of truth, and it is what the demo above runs on.

Work IQ grounds on how your people actually work. It reaches into Microsoft 365, the mail, meetings, chats, files, and the org graph, to answer questions about a person's own work, like “what meetings do I have about Project Falcon” or “who owns the billing integration.” Every result is trimmed to what that signed-in user is already permitted to see.

Fabric IQ grounds on the state of the business. It sits over governed analytics in Microsoft Fabric and OneLake and answers questions about your numbers and your business entities, like “what were Q3 sales by region” or “how is Contoso related to Shipment 8842.” The answer comes back tied to a live semantic model, not a spreadsheet someone exported last month.

These are complementary, not competing, and you have more than one way to bring them into your agent. The choice comes down to how tightly a domain needs to sit inside your Knowledge Base retrieval.

Reach for the Foundry IQ path when you want one grounded answer that fuses everything at once. Along with your documents, a Knowledge Base can connect a Fabric ontology, Fabric data agents, and Work IQ as additional knowledge sources. The agent then makes a single retrieval call across content, structured business context, live systems, and a person's Microsoft 365 work, all with the same citations and on-behalf-of security. This is the right default when documents are the anchor and the other signals should co-retrieve with them.

Reach for Fabric IQ as a tool call when analytics is a separate question, not part of the same retrieval. Fabric IQ is a first-party grounding tool inside the Microsoft Foundry Agent Service. The agent decides to invoke it when the user asks something quantitative, and it queries governed analytics in OneLake alongside what the Knowledge Base returns. Use this when the analytics answer stands on its own and shouldn't be blended into every document lookup.

Reach for Work IQ via Agent-to-Agent (A2A) when you want to delegate a task rather than pull raw context. The agent hands off a natural-language request like “summarize my recent emails about Project Contoso,” and Work IQ handles retrieval, reasoning, and response synthesis against the user's Microsoft 365 data on their behalf. Use this when Work IQ's own reasoning over M365 is the answer you want, not just its retrieved snippets.

Foundry IQ stays the central knowledge layer for your content. Fabric IQ and Work IQ (both in public preview) extend that with analytics and workplace context when a use case calls for it, and you can still point Foundry IQ at a single source when that is all you need.

The Azure Services Doing the Work

Underneath the interface, GPT-RAG is the orchestrator. It owns the whole flow: it authenticates the user, forwards their identity to retrieval, calls the Foundry IQ Knowledge Base, and shapes the grounded, cited answer that comes back. It leans on a few Azure services to do this, and one of them does most of the retrieval work.

Azure AI Search hosts the Foundry IQ Knowledge Base. It holds the connection to your sources, runs agentic retrieval, enforces per-user permissions on the query, and returns passages with the metadata that becomes citations. The grounding story rests on this component.

The orchestrator is the coordinator. It takes the user's question and identity, acquires the on-behalf-of token, and calls the retrieve API. The passages that come back become the only material the model is allowed to answer from, and each one carries a source label that turns into a clickable citation. It then calls a chat model through a Microsoft Foundry endpoint to compose the final answer from those passages and nothing else.

Microsoft Entra ID is what makes the security real rather than cosmetic. People sign in, and their identity is what flows through the on-behalf-of token into retrieval, so document-level trimming runs against a real principal. The application itself runs on Azure Container Apps, with its dependencies reached over the network rather than exposed to the public internet.

One retrieval path: GPT-RAG on Container Apps, the Foundry IQ Knowledge Base on Azure AI Search, and the IQ knowledge sources it grounds on

Built for the Enterprise

A grounded answer is worth little if the system holding your most sensitive content is not one your security team will approve. That is where most demos quietly stop.

GPT-RAG is built to deploy on the Azure AI Landing Zone, Microsoft's reference architecture for AI workloads under a Zero Trust posture.

Every dependency, from AI Search and Microsoft Foundry to storage, is reached through a private endpoint inside a virtual network. Public traffic enters through a gateway with a web application firewall, and outbound traffic is forced through a firewall. Nothing talks to the public internet unless you deliberately allow it.

Pair that network isolation with document-level security at retrieval and you get the property enterprises actually need: the perimeter is locked down, and even inside it, every answer still respects the permissions your files already carry.

Try It Yourself

Getting a grounded agent running is a single azd up. The command provisions the Azure resources, deploys the orchestrator and the UI, and hands you a working agent. Point the Knowledge Base at your own documents, sign in, and ask your first question. For production, deploy GPT-RAG on the Azure AI Landing Zone for the network-isolated topology. The deployment guide has the preflight checks and the step-by-step, from a basic deployment to the network-isolated one.

From a Cited Answer to a Trusted Agent

The payoff is not a single impressive answer. It is what changes when trust stops depending on the model's confidence and starts depending on evidence. When every answer arrives grounded in your content, cited to its source, and scoped to the person asking, the agent moves out of the demo category and into the small set of systems a compliance team can approve. Foundry IQ makes that shift possible, and Work IQ and Fabric IQ extend it from your documents to your work and your numbers.

Deploy GPT-RAG, point it at your corpus, and ask it something only your data can answer.

References

Implementing Multi‑Agent Orchestration in Microsoft Copilot Studio

LeoMa — Wed, 15 Jul 2026 18:35:54 GMT

As organizations move beyond simple Q&A copilots, a common challenge quickly emerges: single agents struggle to handle multi-step, cross-domain scenarios. In our work with customers using Microsoft Copilot Studio, this shows up when copilots are expected to combine reasoning, real-time data, and domain-specific rules, such as troubleshooting workflows, aggregating system insights, or synthesizing information from multiple business areas.

Multi-agent orchestration addresses this challenge by allowing copilots to delegate work to specialized agents while keeping a single decision layer. In this post, we’ll share a practical orchestrator–specialist pattern implemented in Microsoft Copilot Studio, based on real customer engagements. We’ll walk through how the architecture works, why teams adopt it, and how to validate agent collaboration using built-in tools like the Activity Map.

Why Multi‑Agent Orchestration Matters

Teams typically adopt orchestrator patterns in Copilot Studio because they want:

Composability — break down complex tasks into smaller, specialized agents (finance, HR, ITSM, etc.)
Governance — each specialist agent is owned by its respective domain team, enabling safer lifecycle management
Reusability — specialist agents can be consumed by multiple copilots without re-creating logic
Observability & control — the orchestrator acts as a single decision layer, with activity maps showing which agent was invoked
Enterprise scalability — enables teams to scale to dozens of agents while maintaining consistent guardrails

How the Architecture Works

A typical multi‑agent setup in Copilot Studio uses:

One orchestrator agent: Routes user intent and synthesizes final answers
Multiple specialist agents: Each provides a single domain capability
Agent Actions: The mechanism that allows one agent to call another cleanly

In our demo:

Role	Agent	Responsibility
Orchestrator	Life Assistant Agent	Intent classification, routing, synthesis
Specialist	Weather Agent	Weather lookup, forecast, location-specific data
Specialist	Travel Assistant Agent	Trip suggestions, planning, transportation info

Step 1 - Create Specialized Agents ("Skills")

Weather Agent
1. Open your Copilot Studio agent
2. Go to the Knowledge tab
3. Select Add knowledge, choose Public websites

4. Provide the website url which is used to query the weather information(ex - www.weather.com/)

5. Provide the description accordingly

Note: You can add more websites if required.

Travel Assistant Agent

The setup steps of this agent is similar to the Weather agent and the only difference is that we need to provide traveling related websites which is used to provide trip suggestions, planning and transportation info.

Step 2 - Create Orchestrator Agent

Life Assistant Agent

1. Open your Copilot Studio agent
2. Go to the Agents tab
3. Select Add an agent, choose the agent created in step 1
4. Provide the name and description

5. Follow same above steps to add another agent

6. In the overview page add instructions of how to intelligently delegates tasks to its specialist sub‑agents

Step 3 - Test Steps (End-to-End Validation)

Open the Test copilot pane and ask for suggestions for your travelling destination, such as:

"I will travel to Beijing this weekend, provide me the travelling plan based on the weather forecast"

Expected result:

The Life Assistant acknowledges it needs weather + travel insights
Activity Map shows your orchestrator routing correctly before calling any sub‑agents
Weather data appears in the Life Assistant’s intermediate reasoning
Travel recommendations reflect weather conditions

Conclusion

Multi-agent orchestration in Microsoft Copilot Studio offers a practical way to build copilots that can reason, route, and collaborate across domains without becoming difficult to govern. By separating responsibilities between an orchestrator agent and domain-owned specialist agents, teams can scale functionality while maintaining clear ownership, reuse, and operational visibility.

From an enterprise perspective, this pattern mirrors how organizations already work: domain teams manage their expertise, while a central orchestration layer ensures consistent behavior and policy enforcement. For makers and developers, it demonstrates how far you can go using Copilot Studio’s native agent and action model without custom orchestration codes required. As copilots take on more complex, real-world scenarios, this architecture becomes a strong foundation rather than an advanced option.

Acknowledgements

We extend our sincere appreciation to the broader Cloud Accelerate Factory GCR team for their valuable contributions, insights, and close collaboration in validating this pattern across multiple customer engagements. Special thanks go to our AI Architects— Dr. HeZhang Dr. LongyuQi , JasonShao , and EthanTseng —as well as our PM partners, RayneJin and EmmaWang. Their thoughtful feedback and extensive field experience were instrumental in shaping and refining this guide.

Next steps

Microsoft Foundry Model Deployment Pricing Update

Chris Hoder — Mon, 27 Jul 2026 17:38:10 GMT

Running AI workloads in specific regions—whether for compliance, latency, or data residency—now comes with more options and clearer pricing. Starting September 1, 2026, Microsoft Foundry is updating pricing for EU Data Zone and Regional deployments outside the US, while also launching a new APAC Data Zone to serve customers across Asia-Pacific. Foundry continues to expand your options across cost, performance, and data residency. Because delivering AI within a specific geography at high availability costs more than a shared global pool, the updated pricing reflects both that investment and the compliance value these options provide.

TL;DR key changes: Global pricing stays the same. EU Data Zone increases 9% above Global. Regional deployments outside the US increase to 7-16% above Global. New APAC Data Zone launches at 20% above Global. Changes apply September 1, 2026

Pricing Update starting September 1, 2026

Running AI in a specific geography or country, at high availability, costs more than running in a shared global pool. To reflect that, pricing for Data Zone and Regional deployments outside of the United States is increasing on September 1, 2026*. Global pricing stays the same and remains your most cost-efficient choice.

Table 1: Pricing Update starting September 1, 2026

Deployment	Price vs. Global	What’s changing
Global	Same	No change
APAC Data Zone (new)	20% higher	Newly available
EU Data Zone	20% higher	Increasing by 9%
US Data Zone	10% higher	No change
Regional — US	10% higher	No change
Regional — outside the US	25% – 50% higher	Increasing by 7%-16% depending on you region

Table 2: Regional pricing effective September 1, 2026 (price relative to Global**)

New price vs. Global	Regions
+25%	Australia, India, Indonesia*, Malaysia, New Zealand**
+30%	Austria*, Belgium, Canada, Denmark**, Germany, Italy, Mexico, Poland, South Africa, Spain, Sweden, Switzerland, UAE
+35%	Japan, Korea, Taiwan***
+40%	France, Hong Kong, Israel***, Norway, Qatar, UK
+50%	Brazil, EU North, EU West, Singapore

*How the change applies:

Standard (pay as you go): the planned premiums apply only to models launched on or after September 1, 2026. Customers who stay on their current models see no price increase; the higher EU Data Zone and Regional premiums apply only when they move to a model launched on or after September 1, 2026.
Provisioned Throughput (PTU): the increase applies to all customers with EU Data Zone or Regional PTUs outside of the US.
- For PTU, prices relative to Global are for monthly and yearly reservations. Hourly PTU prices relative to Global might differ.

** For Data Zone and Regional deployments, price premiums are relative to Global pricing. Please see Azure pricing page for most current rates.

***Newly available regions at this price (no prior rate).

For exact, current rates by model and deployment type, see the Azure pricing page.

Learn more about Foundry model deployment types and regions here.

More Options across cost, performance, and data residency

Foundry’s core differentiator is choice. Foundry aims to be your platform of choice providing a single suite of offers that span not only models, performance tiers but also data processing requirements. This enables you to have control over your data all within the same, common code.

Here is where your requests can run:

Global — processing runs wherever Microsoft’s global infrastructure has capacity, built for agility and cost-efficiency.

Data Zone (US, EU, and now APAC — processing stays within the selected geography to support regional data residency requirements.

Regional — processing stays in-country for the strictest requirements.

How you run is a differentiator too. Choose Standard for flexible, agile, and economical scaling, Priority Processing for low-latency real-time apps, Provisioned Throughput (PTU) for predictable performance at scale, or Batch for high-volume jobs at the lowest cost of the four options. You can mix and match per workload. The instant model deployment experience is designed to reduce the setups friction so developers can go from model selection to first inference faster.

Foundry also keeps getting more efficient. The model router in Foundry Models matches each request to the most appropriate model, balancing quality, latency, and cost. Prompt Caching cuts redundant computation for recurring and long context interactions. PTU Spillover and quota optimization preserve service continuity through usage spikes. Together, these capabilities are designed to improve token efficiency, reduce compute cost, and shift more tuning into the platform rather than manual work.

Learn how to optimize model deployment and performance in Microsoft Foundry.

Choosing where AI processing happens

Where your AI requests are processed can affect both latency and compliance. Many organizations have data residency requirements that shape where they can run AI. Data Zone and Regional deployments let you keep processing within a chosen geography or country while using the same models and tooling available elsewhere on Azure. You remain responsible for determining whether a given configuration meets your regulatory obligations; Foundry provides deployment options that can support those requirements.

As Hongsoo Kim, Chief Data and AI Officer at Toss, put it, Foundry keeps “data processing regionally anchored while accessing advanced AI models at scale”- giving the confidence to “accelerate AI innovation responsibly.”

Build your next AI workload on Foundry

Whatever the workload calls for — lowest cost, low latency, an SLA, or strict data residency — there’s a Foundry deployment that fits. Start building today in Microsoft Foundry. For more information about deployment types in Microsoft Foundry, please read the learn documentation.

Frontier Tuning - A shift from classic fine-tuning

Laurentran — Thu, 09 Jul 2026 16:00:00 GMT

Your AI agents are burning through tokens and budget because they're relearning your business context on every single task. Frontier Tuning changes that equation. Instead of injecting thousands of context tokens at runtime, you teach the model your workflows once, and it learns from your organization's data. Here's how this new approach to post-training, announced at Build 2026, can make your agentic systems both smarter and cheaper to run.

Why the guidance is shifting: The economics have changed

Traditional guidance has previously steered AI practitioners to reserve fine-tuning for very specific domain workflows. Before jumping straight to fine-tuning a model, the best practice was to first exhausting prompt engineering, RAG approaches, and context engineering. Historically, fine-tuning's training and hosting costs were more expensive than they were worth, and fine-tuning was often overkill.

However, for agentic workflows, generic models don't understand a customer's terminology or task sequences, so RAG injects the context without teaching behavior every time at runtime. For complex workflows that require thousands of tokens per task, injecting context at runtime adds up quickly.. Now, the economics have inverted, so by tuning models to be task, organization, and line-of-business specific, performance and efficiency improve. Fine-tuning now can be a cost-reduction lever depending on token consumption. When your models are Frontier Tuned with your data, their performance naturally enables less token consumption through more optimal task solving.

In summary:

Before: Fine-tuning training and inference costs were usually not justified when RAG and context engineering could provide strong results
Now: For high-volume agentic workflows, tuning costs are often cheaper than repeated context injection
Result: Frontier tuning becomes both a cost-reduction lever and a performance add

Reinforcement fine-tuning vs. classic fine-tuning

There is now massive value in learning process and workflows in our agentic systems. Reinforcement learning is based on a rewards system, and when we apply Reinforcement Fine-Tuning (RFT), we reward model behavior for landing the correct sequence in an agentic workflow (think tool calls within traces). Historically, Reinforcement Learning excels at sequence and process-based scenarios. For example, think game-playing with complex games like GO or Chess. The system understands the current state and determines the next most optimal move based on that state. Now extend this concept to agentic workflows, where the agent has the current context (or state) and determines the next most optimal move (tool call, action, etc.) based on it's context. This is a shift from our former method to apply Supervised Fine-Tuning (SFT) with supervised data (i.e. ground truth labeled data of inputs and good outputs). With classic fine-tuning, we are only enabling the model to understand strong token outputs given a user prompt based on that labeled data, without considering an optimal sequence of actions that RFT enables our models to learn.

	Supervised Fine-Tuning (SFT)	Reinforcement Fine-Tuning (RFT)
Learns from	Labeled input/output pairs	Traces of actions and outcomes
Optimizes for	Correct outputs	Optimal action sequences
Best for	Single-turn Q&A, classification	Multi-step agentic workflows
Analogy	Determining correct answers	Learning to play chess

Example: Customer Service Agent Optimization

Consider a customer service agent that handles refund requests. Without tuning, each interaction requires injecting your refund policy (500 tokens), product catalog context (1,200 tokens), and customer history patterns (800 tokens) - roughly 2,500 tokens per request. At 10,000 daily requests, that's 25M context tokens monthly.

With Frontier Tuning, the model learns your refund workflows from historical traces. It knows when to check inventory, when to escalate, and how to apply your specific policies. Context injection drops to ~400 tokens per request for customer-specific details only, which is an 84% reduction in token consumption.

When to Frontier Tune?

Consider Frontier Tuning when:

You have workflow data: Traces of tool calls, decisions, and evaluation signals from your agentic systems
You can define success: Clear evaluation rubrics for what optimal task completion looks like
The economics make sense: Your token consumption is high enough that tuning costs less than repeated inference

Get Started with Frontier Tuning

Ready to explore Frontier Tuning for your agentic workflows?

Explore the fundamentals: Microsoft Frontier Tuning - understand the technical capabilities and how it works
Nominate your use case: Request early access - tell us about your agentic workflow and nominate it for the preview
Check out the MAI Blog: Microsoft AI Blog - get the latest news, stories, and perspectives from Microsoft AI
Experiment with the MAI models: Microsoft AI Model Playground - explore the models without a subscription
Not sure if Frontier Tuning is right for you? Start by instrumenting your agentic workflows to capture traces. You'll need this data regardless, and it will help you assess whether the economics make sense for your scenario.

Authored by the Americas Markets and Industries Office of the CTO

Monitoring & Observability in Microsoft Foundry Part 2: Configuration and Operations

Kalaivanan — Thu, 09 Jul 2026 14:00:00 GMT

Part 1 of this series made the case that AI observability is not just another dashboard problem. Once an agent is in production, the question is no longer can it respond; the question is whether it continues to respond accurately, safely, efficiently, and consistently as models evolve, prompts change, retrieval indexes drift, and real-world traffic exposes edge cases you never saw in test. That is why Microsoft Foundry’s observability story is built around three connected disciplines evaluation, monitoring, and tracing rather than traditional uptime metrics alone.

This article is the practical half of that story. It stays in the Foundry (new) experience throughout and focuses on the configuration and operational patterns that actually matter once you are moving agents toward production:

connecting Application Insights
enabling continuous evaluation
reading the monitoring dashboard like an operator
using traces to debug quality regressions
wiring alerts into Azure Monitor
adding quality gates to CI/CD
enforcing guardrails and compliance
integrating Defender for Cloud and Microsoft Purview and keeping a close eye on cost, quota, and token usage across the estate.

Portal note: Everything below assumes the Foundry (new) portal. If you are not seeing Build, Operate, and Admin in the current navigation, stop and switch to the new experience before you begin.

Start with the telemetry backbone: connect Application Insights

Every meaningful observability workflow in Foundry starts with one dependency: Azure Monitor Application Insights. It is the storage and correlation layer for the telemetry that powers monitoring, tracing, and the production-facing quality views. If you skip this step, the rest of the experience becomes partial at best and invisible at worst.

In the current portal experience, the cleanest place to verify that your project is wired correctly is Operate > Admin > Connected resources. That is where you confirm whether an Application Insights resource is already attached to the project, and where you add one if it is missing. Some agent-specific experiences also surface connection entry points elsewhere, but Connected resources is the most reliable control-plane view when you want to validate the configuration before you go any further.

In practice, the setup is straightforward. Open your project, go to Operate > Admin, select your Project, look for the tab Connected resources, and verify that an Application Insights resource is present. If not, add one there and keep it in the same region as the rest of your project resources where possible. Once that connection exists, Foundry has a place to emit and correlate the telemetry that the rest of this article depends on.

One operational detail is worth calling out early: the people who will investigate traces or query logs later need access to the monitoring resources themselves. In other words, connecting the resource is only half the job; the second half is making sure the operations or engineering team can actually read what Foundry writes there.

From connection to continuous evaluation

Part 1 described evaluation has to move from an occasional pre-release ritual to an always-on production discipline. The portal now supports that model directly: once your project is connected, you can turn on continuous evaluation so sampled production traffic is scored over time instead of only during offline test runs.

The natural entry point is the agent’s monitoring surface. Open Build > Agents, select the agent you want to operate, and then go to Monitor. This is where the platform brings together operational metrics and evaluation signals for the agent you actually care about, rather than forcing you to jump between disconnected tools. If your tenant has the built-in assistant enabled, Ask AI (Preview) can also summarize what the dashboard is showing and highlight abnormal behavior for the time range you have selected.

When you open the monitor settings, you are not just flipping a switch, you are deciding what “good” means for your workload. For a general conversational agent, the right starting point is usually coherence and relevance. For a RAG-heavy agent, groundedness and retrieval quality quickly become non-negotiable because they separate model behavior problems from retrieval problems. For safety or policy sensitive workloads, the relevant safety dimensions belong in the first wave as well, not as an afterthought once you have shipped.

The portal also forces a real-world tradeoff that Part 1 already warned about: sampling. Continuous evaluation is valuable precisely because it runs on live traffic, but it is not free. A sensible production posture is to begin with a modest sample, see how noisy or stable the results are in your environment, and then raise or lower the rate based on risk, volume, and budget.

One practical point that often gets missed: if you are enabling continuous evaluation against production traffic, the project’s managed identity must have the permissions required to execute those evaluation calls.

Read the dashboard like an operator, not a tourist

The Agent Monitoring Dashboard is easy to mistake for a status page. It is more useful than that. Used well, it becomes the place where you correlate quality, latency, failures, token consumption, and if you have the right workflows enabled, safety findings in one operational view backed by Azure Monitor data.

That matters because production AI rarely fails in a single dimension. A rise in latency might be harmless if quality stays steady and the cause is obvious. A flat latency curve can still hide a serious regression if groundedness or retrieval quality starts slipping. Token usage can expose cost drift long before a finance alert ever fires. The dashboard is valuable precisely because it puts those signals side by side so you can see whether the problem is quality-only, operations-only, or a combination of both.

In the current portal, you can reach that view from Build > Agents > Monitor for agent-centric investigation, and from Operate > Overview when you want a broader control-plane perspective across the subscription. Microsoft’s current documentation for Ask AI (Preview) explicitly points to both of those surfaces: Build > Models or Agents > Monitor for workload-specific analysis and Operate > Overview for estate-level analysis.

This is also where the most useful operator habit emerges: use metrics to spot the problem, but do not stop there. A low groundedness score or a rising error rate should trigger a trace investigation, not just a dashboard screenshot for later discussion. The dashboard is the front door, not the full diagnosis.

When a score drops, tracing tells you why

Monitoring tells you something is changing. Tracing tells you what actually happened inside a run. That distinction is what turns observability into an operational system rather than a collection of charts. Microsoft’s tracing model is built on OpenTelemetry, and the goal is not simply to log a request, it is to capture the execution path across model calls, tool invocations, retrieval steps, orchestration decisions, and the timings and tokens associated with each stage.

For Foundry-managed experiences, a good portion of that visibility is available once the telemetry backbone is connected. For custom agents and mixed stacks, the current guidance puts the emphasis on making sure the agent is registered and observable from the control plane, then sending standards-based telemetry into the same monitoring backend. Operationally, that means Foundry becomes the builder-facing surface while Azure Monitor remains the store, query, and correlation layer underneath.

The most important technical correction to many older tracing examples is the authentication and wiring model. The current guidance favors using the Application Insights connection string instead of older configurations that relied on the instrumentation key as a standalone setting, and the cleanest Python example today uses the AzureAIOpenTelemetryTracer from the LangChain integration path with that connection string explicitly provided.

from langchain.agents import create_agent from langchain_azure_ai.callbacks.tracers import AzureAIOpenTelemetryTracer application_insights_connection_string = "..." tracer = AzureAIOpenTelemetryTracer( connection_string=application_insights_connection_string, enable_content_recording=True, ) def get_weather(city: str) -> str: """Get weather for a given city.""" return f"It's always sunny in {city}!" agent = create_agent( model="openai:gpt-5.1", tools=[get_weather], system_prompt="You are a helpful assistant", ).with_config({"callbacks": [tracer]})

For teams using other frameworks, the broader principle is the same even if the code differs: emit OpenTelemetry spans and send them to the Application Insights resource already attached to your Foundry project. That keeps traces, quality signals, and operational telemetry in the same backend instead of fragmenting them across different tools and teams.

There is one operational caution here that should never be buried in a footnote: traces can capture prompts, responses, tool arguments, and other sensitive runtime content. In enterprise environments, trace retention, RBAC, and content hygiene matter just as much as trace completeness. If your prompts or tool payloads can contain secrets or personal data, fix that before you scale tracing, not after.

Use Foundry to discover issues, and Azure Monitor to operationalize them

Foundry is an excellent place to see that something is wrong. Azure Monitor is where you should generally act on it. That distinction creates a durable operating model because it keeps AI-specific insight in the portal while letting SRE and platform teams use the same alerting, routing, and automation surfaces they already trust for the rest of their workloads.

A practical alerting strategy does not start with every possible metric. It starts with a short list of conditions that justify operator intervention. For most agents, that means some combination of:

sustained latency increases,
run failure spikes,
abnormal token consumption,
and material drops in quality or safety signals that represent user-visible risk.

Foundry’s own monitoring surfaces can help you decide which thresholds matter, but the production-grade enforcement usually belongs in Azure Monitor alerts and action groups attached to the connected Application Insights resource. That is the pattern Microsoft’s observability guidance consistently points toward: the portal helps you understand the signal, while Azure Monitor helps you route and automate the response.

CI/CD quality gates: make quality a release decision

The real test of whether observability is part of engineering culture is whether it shows up in CI/CD. If evaluation only happens after deployment, it is already too late for many regressions. The better pattern is simple: run a repeatable evaluation set during the build or release process, publish the results, and fail the promotion if the thresholds that matter to the business are missed. That is how you turn “quality” from a conversation into a gate.

Because task and action names can change over time, the most durable approach for publication is not to hard-code a potentially short-lived marketplace extension name. Instead, keep the pipeline pattern stable and call your evaluation script explicitly. That gives readers something they can implement regardless of whether they use GitHub Actions or Azure DevOps and regardless of how the surrounding wrapper evolves.

A simple GitHub Actions pattern looks like this:

name: foundry-evaluation-gate on: pull_request: push: branches: [ main ] permissions: id-token: write contents: read jobs: evaluate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: azure/login@v2 with: client-id: ${{ secrets.AZURE_CLIENT_ID }} tenant-id: ${{ secrets.AZURE_TENANT_ID }} subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} - run: | python -m pip install --upgrade pip pip install -r requirements.txt - run: | python ci/run_foundry_evals.py \ --project-endpoint "${{ secrets.FOUNDRY_PROJECT_ENDPOINT }}" \ --dataset "tests/evalset.json" \ --output "artifacts/eval-results.json" - run: | python ci/assert_eval_thresholds.py artifacts/eval-results.json

The principle matters more than the wrapper: use the same evaluator definitions across development, CI, and production monitoring so a “good” score means the same thing wherever you see it. That consistency was one of the core ideas in Part 1, and it is what makes evaluation an operating system instead of a one-off test harness.

Guardrails and compliance belong in the operating loop

Once an agent is live, “responsible AI” has to move out of slideware and into operations. That is what the Compliance workspace in the control plane is for. In Part 1, the compliance story was framed around policy posture, guardrails, and remediation across the estate. In practice, that means Operate > Compliance becomes part of your operating routine, not something you visit only when audit season arrives.

The value of this workspace is not just visibility. It is that it lets you compare what your organization expects:guardrail controls, security posture, data-governance integration against what individual assets are actually doing. The policy view tells you where you are out of compliance. The asset views help you see which deployments are drifting. The remediation flow closes the loop before policy violations become real incidents.

Use Operate > Compliance to review policy posture, identify violations, compare required controls with actual deployment settings, and remediate the drift. The exact button labels can change over time; the operating motion does not.

Extend the view: Defender for Cloud and Purview

Foundry’s built-in controls are necessary, but enterprise deployments often need the larger Microsoft security and governance ecosystem around them. That is where Defender for Cloud and Microsoft Purview extend the story from application behavior into security posture, threat detection, audit, and data governance.

From an operational perspective, Defender for Cloud helps security teams see posture recommendations and AI-adjacent threats such as suspicious prompt activity or exposed configurations in the places they already work. Purview extends the governance model further by helping organizations treat prompts and responses as governed data, subject to the same kinds of audit, classification, and security workflows that apply elsewhere in the enterprise.

The right place to review those integrations in the current portal is the security and compliance experience under Operate, especially the views focused on posture and governance. Availability and exact onboarding flows can vary by tenant, role, and licensing, so what matters most for an enterprise is the operational principle: if your organization relies on Defender or Purview elsewhere, AI workloads should not become the exception.

Cost, quota, and token usage: keep them in the same conversation as quality

One of the easiest mistakes in AI operations is to discuss quality, safety, and cost as if they were separate systems. They are not. Token usage affects spend, spend influences sampling choices, and sampling choices shape what you can see about quality in production. Foundry’s Operate surfaces are useful precisely because they let you see those pressures in one operating loop rather than across unrelated tools.

The three views that matter most are still the ones Part 1 highlighted. Operate > Overview gives you the estate-level operational summary, including token and cost trends. Operate > Assets gives you the searchable inventory view where heavy consumers and unhealthy assets become obvious. Operate > Quota tells you how deployment capacity is being consumed and whether you are heading toward avoidable throttling or scaling friction.

What has to be explicit in production is the cost model around observability itself. Monitoring does not introduce an additional Foundry platform charge, but tracing still depends on the underlying Azure Monitor / Application Insights storage and retention footprint, and evaluation adds execution cost because more model work is being done. In other words, you are not only monitoring your agents, you are also operating a telemetry system. Treat it that way.

What good looks like in production

The biggest shift between Part 1 and Part 2 is this: once the features are enabled, observability becomes a habit. Good teams do not just turn on dashboards. They decide which signals matter, which thresholds justify intervention, where human review belongs, and how those decisions show up in daily operations and release engineering.

In practice, that usually means a handful of operating principles:

Do not monitor everything at once. Start with the few signals that map directly to user harm or operational pain.
Use the same evaluation logic across environments. Quality should not mean one thing in CI and another in production.
Keep traces useful, not reckless. Rich traces are valuable until they start capturing content you should never have stored.
Treat cost as a first-class signal. Token usage and trace volume are not background details; they are operational constraints.
Revisit the configuration periodically. The platform is evolving quickly, and what was preview-only a few months ago may already be operationally relevant in your tenant now.

Conclusion

Part 1 detailed that production AI requires a different kind of observability because AI systems fail in different ways. Part 2 is where that argument becomes operational. Once you connect Application Insights, enable continuous evaluation, use the monitoring dashboard to spot change, follow traces to root cause, route the right signals into Azure Monitor alerts, and bring compliance, security, and cost into the same operating loop, you stop treating quality and safety as release-time paperwork and start treating them as runtime disciplines.

Foundry’s observability model gives builders, operators, and governance teams a shared system for answering the same question from different angles: Is this agent still doing the right thing in production, and how do we know?

If Part 1 was the architectural blueprint, Part 2 is the operational playbook. Put together, they describe the shift every serious AI team eventually has to make: from “the agent works” to “the agent stays reliable, explainable, and governable in production.”

Foundry IQ is now in Copilot Studio: Bring your enterprise data to every agent conversation

Angie-Silva-Pereyra — Wed, 08 Jul 2026 21:24:32 GMT

Your AI agents are only as smart as the data they can access

Enterprise teams building Copilot agents face a familiar challenge: connecting agents to organizational knowledge without compromising security, governance, or answer quality. Foundry IQ solves this by providing a unified knowledge layer that sits between your data sources and your agent—delivering 54% better response relevance while keeping enterprise compliance built in.

What Foundry IQ Brings to Your Copilot Agent

Enterprise organizations need a specific set of capabilities to safely deploy AI agents in production, security, compliance, governance, and operational reliability. Foundry IQ is built to meet exactly those requirements. With Foundry IQ, you can create Knowledge Bases. These Knowledge Bases come with enterprise readiness built in, such as Customer-Managed Keys (CMK) for data encryption, ACLs for fine-grained access control, network isolation, Microsoft Entra ID integration, compliance with standards such as FedRAMP and SOC2 and others, the full set of capabilities any organization needs to deploy AI agents safely in production. Learn more about Foundry IQ (Azure AI Search) security capabilities → https://learn.microsoft.com/en-us/azure/search/search-security-overview

When you connect a Knowledge Base to your Copilot agent, it leverages the advanced retrieval capabilities that Foundry IQ brings, including agentic retrieval (automatic query planning that federates across multiple knowledge sources in parallel), iterative query planning and semantic ranking (relevance scoring that goes far beyond keyword matching). The result is a 54% average improvement in response relevance compared to traditional RAG approaches, more accurate, grounded answers for your users (For more details: Foundry IQ: Improve recall by up to 54% with knowledge bases | Microsoft Community Hub)

For Developers & End Users

For Developers

Today, developers can select one of their existing Foundry IQ Knowledge Bases to connect to a Copilot agent, giving their organization's users access to enterprise data safely, with permissions and governance already built in. This ensures that each user only sees the data they are authorized to access, and that every answer returned is grounded in the right, relevant content, no additional configuration needed on the user side

For End Users

End users get access to their organization's enterprise data safely and without any extra work on their end, no need to worry about permissions, data sources, or configurations. Just ask, and your Copilot agent returns accurate, grounded answers with inline citations, from the right sources, respecting what each user is allowed to see.

What Do I Need to Do? Step-by-Step

Prerequisites

Before getting started, make sure you have:

An active Microsoft Foundry connection
At least one Knowledge Base already created in Foundry IQ
An agent created in the new experience. Learn more in Create an agent

1.- Select Microsoft IQ in your Copilot agent settings

- Open your agent in Copilot Studio.
- Select the Build tab.
- In the components panel, select Microsoft IQ to open the Add Microsoft IQ dialog.
- Navigate to your Copilot agent configuration and select the Microsoft IQ option.

2.- Select Foundry IQ

3.- Select your Foundry IQ connection

Choose the Foundry IQ connection you want to use for this agent.

4.- Select a Knowledge Base

Pick the Knowledge Base you want your agent to use as its knowledge source.

5.- Get your agent ready with instructions

6.- Chat with your data

Your agent is now grounded in your enterprise Knowledge Base. Now you can start a conversation and explore your data!

Get Started: Try Foundry IQ in Copilot Studio: Connect to Foundry IQ from an agent (preview) - Microsoft Copilot Studio (new experience) | Microsoft Learn. Learn more about Foundry IQ retrieval capabilities: https://aka.ms/FoundryIQ

Before You Ship Your Agent: A Five-Step Path to Evaluations You Can Trust

amah — Wed, 08 Jul 2026 18:41:30 GMT

Authors: Caroline Yeh, Ali Mahmoudzadeh, Morteza Ziyadi

If you've shipped an agent to production, or you're close, you've already made a quiet bet: that the evals you ran during development reflect what happens when real users interact with the real system. That bet is harder to win than it looks. Not because evaluation is unsolved, but because agents break in ways single-turn metrics can't see. A task silently stalls mid-session. A factual claim slips through ungrounded. A conversation scores 4.2 on coherence while the user abandons it on turn three. None of that shows up as an error. None of it registers in a per-turn score. And if the test harness generating those scores has its own calibration problems, like a simulator that coaches your agent, a trace sampler that keeps pulling the same three request types, or an evaluator with a 20% flip rate, then you're not measuring your agent's quality. You're measuring your eval stack's optimism.

This post is for the engineers and applied scientists responsible for that bet. The ones writing the eval pipelines, choosing the judge models, deciding which traces go into the fine-tuning corpus, and fielding the question "is this agent ready to ship?" on every release cycle.

Over the past two weeks, the Microsoft Foundry observability team published five research studies on how to measure agents and, just as importantly, how to measure the tools doing the measuring. Today, we're pulling them into one prescriptive path.

Before you ship your agent, make sure you have:

Multi-turn evaluated: session-level quality scored across the whole conversation, not averaged across turns
A defined rubric: what "good" means for your task, your agent, your success criteria, not a generic catalog entry
Simulated users: test conversations that behave like real users, scored with a validated rubric
Sampled production traces: the right traces for breadth of coverage, not just the most frequent ones
Benchmarked: a standardized, reproducible yardstick for your deployment or agent

These aren't a one-time gate. They're a loop you re-run on every meaningful prompt, model, or tool change. Here's what each step does, why it matters to the engineer on the hook for agent quality, and where to go deep.

Why evaluation is harder for agents than it was for models

If you came up evaluating LLMs on fixed benchmarks or RAG pipelines on retrieval metrics, agents require a shift in how you think about measurement and it's worth naming explicitly before the five steps.

With a model, the unit of analysis is a response. With an agent, the unit of analysis is a session: a multi-turn conversation in which the agent asks clarifying questions, calls tools, retries when something fails, and stitches the whole thing into an outcome. A session can have every individual response score well on relevance, fluency, and faithfulness and still be unfinished, off-policy, or quietly hallucinated. Averaging turn-level scores doesn't recover those properties, and it can actively mislead you and the session that failed your user reads as a pass.

This creates a second-order problem specific to ML engineers: any tool you use to score your agent is itself a system that can be wrong. A weak evaluator that always outputs "pass" will flatter your agent. A judge with high run-to-run variance will generate phantom regressions on every rerun. A simulator that coaches your agent when it misses a step hides the exact failures you're trying to measure. If you don't instrument the evaluation stack with the same rigor you'd apply to the agent, you end up with a measurement system that's less reliable than your intuition.

The implication is straightforward: evaluation is recursive. You have to measure the evaluator, not just the agent. Every study below does exactly that; accuracy, reliability, and cross-judge robustness, so you know which evaluators to trust as hard gates, which to use as triage signals, and where the eval stack itself needs tuning before you can trust what it's telling you.

Step 1 Multi-turn evaluate

Four session-level evaluators: Task Completion, Customer Satisfaction, Groundedness, and Conversation Coherence, each targeting a distinct property of the whole conversation.

Start here because it defines the problem every other step is helping you solve: you need to score the session, not the turn. Each of the four evaluators targets a property that can only be assessed across the full conversation, and the design is intentionally complementary, an agent can pass one and fail another, and that gap is diagnostic, not redundant.

Task Completion measures if the user’s task is successfully and completely accomplished by the end of the session.

Customer Satisfaction shows how satisfied would the user be with the agent’s performance across the session.

Groundedness checks if the agent’s factual claims across all turns are supported by the conversation’s grounding sources.

Conversation Coherence verifies that the conversation flows logically across turns; state maintained, no contradictions, sensible progression.

The study validated all four with benchmark datasets, paired validity and reliability metrics across four independent runs, and six judge models. Two headline findings for the engineer making deployment decisions:

Task Completion and Customer Satisfaction reach strong enough agreement with ground truth and low enough run-to-run variance to use as single-pass scores in release gating so that you don't need four reruns to trust the verdict.

Groundedness is the exception. It's the hardest of the four, and the wrong tool for a hard release gate. Smaller judges don't just shift the mean, they collapse the hallucinated cluster in the score distribution, missing the minority-class failures the metric exists to catch. Deploy it as an aggregate trend signal with a frontier reasoning judge, route low-scoring sessions to human review, and track it over time.

Run these evaluators on your test set before each release. Gate on Task Completion and CSAT. Trend on Groundedness. Use the gaps between evaluators as the diagnostic signal.

Read the study Evaluating Multi-Turn Agents: A Quality Study of Microsoft Foundry’s Multi-Turn Evaluators | Microsoft Community Hub

Step 2 Define the rubric

Auto-generated, task-specific rubric evaluators that turn "what good looks like for your agent" into a weighted score with per-dimension explanations which is adaptive, context-aware, and reusable across development iterations.

This is where most evaluation stacks fall short, and where the opportunity is greatest. Generic evaluators measure generic properties. Task Completion tells you whether the job got done. CSAT tells you whether the user was satisfied. But neither captures the success criteria that are specific to your agent, your domain, and your policies.

"Verify the customer's identity before issuing a refund." "Confirm the alert is active before paging the on-call engineer." "Flag deviations from the contract template before drafting negotiation language." These aren't properties you can buy off the shelf. They live in your system prompt, your task definition, your policy documentation. A generic rubric can't see them. A fixed catalog can't anticipate them. And without them, your evaluations are measuring something real but just not the thing your users actually care about.

The auto-generated rubric evaluator is designed to close exactly this gap. Give it your agent definition and examples — the context you already have — and it generates a task-specific rubric with weighted dimensions and per-case explanations you can reuse across iterations. Unlike fixed multi-turn evaluators that apply the same criteria to every agent, the rubric adapts to what your agent is actually supposed to do. Change your agent's task scope, update its policy, expand its tool set. Generate a new rubric, and your evaluation pipeline stays current without manual re-authoring.

In validation, generated rubrics aligned with trusted reference signals at the case level (scores on individual samples), and ranked candidate agents consistently with ground-truth oracles (correlation of agents rankings between grounf-truth and evaluations). Not only the discriminative quality of the generated rubrics was high, but also a large fraction of expert-authored rubrics are automatically generated. Rubrics show high reliability, reruns produce consistent verdicts and good separability, the evaluator's ability to tell one candidate agent from another is decisive.

This is the step that makes your evaluation specific rather than generic, adaptive rather than fixed, and grounded in your agent's actual task rather than a one-size catalog. Run multi-turn evaluators to know whether the session succeeded. Run the rubric to know whether your agent did its specific job correctly and to understand the dimensions where it fell short.

Read the study: Auto-Generated Rubric Evaluators: Building Context-Aware Evaluators for AI Agents | Microsoft Community Hub

Step 3 Simulate users

USR-8: eight orthogonal metrics for scoring a user simulator that separate behavioral correctness from prose quality — so you can trust the test conversations generating your scores.

User simulators let you push an agent through hundreds of plausible conversations between every code push, without waiting for real users. The catch, which matters directly to anyone who owns the eval pipeline: a simulator that sounds polished can quietly inflate your agent's scores; a simulator that coaches your agent when it misses a step hides the regressions you're trying to catch. Neither failure mode shows up in the agent's score sheet.

USR-8 is an eight-metric rubric specifically for scoring user-simulator output, designed around a distinction that matters to the engineer, not just the researcher: behavior and style are separate axes. Clarity, relevance, steering (the no-coaching guard rail), responsiveness, consistency, and persona fidelity describe what the simulator does. Realism describes how it sounds. They move independently. Across 1,200 scored conversations in three domains, airline customer service, SRE incident triage, and legal contract review, the Foundry simulator scored at or near ceiling on every behavioral metric; realism was the one gap, and a focused prompt revision closed it with no regression elsewhere.

The key design decision worth internalizing as an eval engineer: most of what makes a simulator good or bad lives in the prompt policy, not the orchestration code. A prompt swap between two harnesses closed most of the measured gap on every per-conversation metric. If your simulator is underperforming, rewrite the prompt before you replace the framework.

Read the study How to Score a User Simulator: Introducing USR-8 | Microsoft Community Hub

Step 4 Sample production traces

MinHash farthest-first diversity sampling: server-side, zero extra per-token cost, selecting for coverage of the input space rather than a mirror of production frequencies.

Once your agent is in production, you have more traces than you can evaluate and the subset you pick shapes everything downstream: what your eval catches, what your fine-tuning learns, what your rubric generation sees. The default instinct is uniform random sampling. It's statistically unbiased and mirrors your production distribution accurately. It's also dominated by the handful of common request patterns that make up the bulk of your traffic, and systematically under-represents the rare prompts, unusual tool-call sequences, and edge cases that are the most important inputs for both evaluation and fine-tuning.

Intelligent sampling uses MinHash signatures and farthest-first traversal to select a subset that covers as much of the input space as possible, including the long tail a uniform sample buries. It runs server-side with no LLM or embedding calls — zero additional per-token cost, completing in under a minute on typical trace pools. On the primary validation dataset (5,000 WildChat traces, 100 selected), diversity sampling produced higher lexical diversity and larger vocabulary than a uniform-random baseline. An LLM judge preferred the diversity-sampled subset most of the time for training and evaluation use. Fine-tuning on a diversity-sampled corpus converged faster with comparable held-out generation quality.

Reach for diversity sampling when breadth of behavior drives downstream quality — evaluation suites, rubric generation, fine-tuning corpora. Reach for a uniform sample when you need faithful production-frequency representation — latency profiling, error-rate estimation, distribution-level SLA checks.

Read the study Intelligent sampling in Microsoft Foundry: the science behind selecting better production traces | Microsoft Community Hub

Step 5 Benchmark

Standardized open-source benchmarks run against your own model deployment or agent — with your judge model, your configuration, and side-by-side comparison across runs.

Public leaderboard scores tell you how a model performs in general. They don't tell you how your deployment performs on the reasoning, math, or domain-knowledge questions that matter for your application or whether it regressed after a prompt or model-version change. That's a different measurement, and it requires running the benchmark yourself, against your own deployment, under the same conditions you plan to compare.

Benchmarks in Microsoft Foundry (preview) make that a first-class part of the development workflow. You select from a catalog of well-known suites, reasoning, math, science, and truthfulness (GPQA Diamond, BIG-Bench Hard, MuSR, TruthfulQA, BBEH, AIME 2025, and more), each with its own curated dataset, task category, and built-in scorer. Run them from the portal wizard or the REST API, compare results side by side in the evaluation group view, and pin a stable judge model and benchmark version so the score is a reproducible yardstick you can trust across releases — not a one-time number tied to a single configuration.

The practical pattern for release gating: pick one or two reasoning-heavy benchmarks (GPQA Diamond or MuSR are good defaults), target your agent, run it on every meaningful change to the underlying model, system prompt, or tool set, and treat the delta as the signal — not the absolute score.

Read the study Benchmarks in Microsoft Foundry (preview): Standardized model and agent quality checks | Microsoft Community Hub

The thread running through all five

The key design decision across these studies: we measured every evaluator the way we ask you to measure your agent. Benchmark datasets. Paired validity and reliability metrics. Multiple judge models per evaluator so we could separate "evaluator works" from "this particular judge works." That methodology produced a result that's more useful than a single accuracy number: for each evaluator, you now know which axis to trust it on, which judge tier it requires, and whether the caveats live in the score quality or only in the threshold placement.

That distinction matters operationally. CSAT with a smaller judge produces good scores at a miscalibrated threshold, a fixable problem that can be solved by re-tuning the cutoff on your own data. Groundedness with a smaller judge produces genuinely weaker scores, not a fixable problem, a judge-replacement decision. Accuracy tables look the same. The per-judge ablation is what tells you which category of caveat you're dealing with.

What makes this approach different

Session-level measurement, not turn-level averages. The four multi-turn evaluators measure properties that only exist across a full conversation. No amount of single-turn metric aggregation recovers task completion or cross-turn grounding.

Task-specific rubrics, not one-size-fits-all criteria. The auto-generated rubric evaluator builds evaluation criteria from your agent's own context — adaptive to your task, your domain, and your policies — rather than applying a fixed catalog to every agent regardless of what it actually does.

Validated evaluators, not just provided ones. Every evaluator above ships with a published quality study — accuracy, reliability, and cross-judge robustness — so you know where to trust it as a hard gate and where to treat it as a triage signal before you wire it into your CI pipeline.

Coverage over representativeness, by design. Diversity sampling deliberately surfaces the long tail your uniform sample buries — the rare inputs and edge cases that evaluation and fine-tuning need most and random sampling systematically skips.

Simulator quality is a first-class metric. USR-8 gives you a rubric for the test harness itself. If the simulator isn't measured, the scores it produces aren't trustworthy — and you won't know in which direction they're wrong.

One workflow, linked to the traces. Multi-turn evaluation, rubric generation, simulation, trace sampling, and benchmarking live in the same observability surface, against the same traces — not five disconnected tools requiring five separate data pipelines.

Getting started

Start by reading our Evaluating AI Agents Practical Gide with Microsoft Foundry for holistic view and approach. Then run the path in order, keeping it as a loop on every meaningful change.

Step	Do this	Go deeper
1. Multi-turn evaluate	Run the four session-level evaluators on your test set before each release; gate on Task Completion / CSAT, trend on Groundedness, use gaps between evaluators as diagnostic signal	Evaluating Multi-Turn Agents: A Quality Study of Microsoft Foundry’s Multi-Turn Evaluators \| Microsoft Community Hub
2. Define the rubric	Generate a task-specific rubric from your agent definition and examples; review against known-good and known-bad cases before wiring into CI	Auto-Generated Rubric Evaluators: Building Context-Aware Evaluators for AI Agents \| Microsoft Community Hub
3. Simulate users	Choose your simulator philosophy (realistic foil vs. helpful tester) first, then score the simulator with USR-8 before trusting its output for agent evaluation	How to Score a User Simulator: Introducing USR-8 \| Microsoft Community Hub
4. Sample traces	Enable Application Insights, open the Traces tab, and create a dataset from production traces with diversity sampling — especially before fine-tuning or rubric generation	Intelligent sampling in Microsoft Foundry: the science behind selecting better production traces \| Microsoft Community Hub
5. Benchmark	From Build > Evaluations > Create, start with a small benchmark (AIME 2025 or GPQA Diamond) against your deployment or agent.	Benchmarks in Microsoft Foundry (preview): Standardized model and agent quality checks \| Microsoft Community Hub

Start building: ai.azure.com

Watch the Build session (BRK252): aka.ms/build26-BRK252

Read the Build recap: From observability to ROI for AI agents on any framework

Join the community: aka.ms/ai/discord

The teams shipping agents that hold up in production aren't the ones running the most evals. They're the ones who know which evals to trust, which to treat as signals, and which parts of their measurement stack to fix before they can believe any of the numbers. These five steps don't guarantee a perfect agent. They help ensure that when something breaks, the failure shows up in your data — not in your users' experience.

That's what evaluation is actually for.