foundry agent service
40 TopicsVoice agents in Microsoft Foundry Agent Service now generally available
People want to talk to AI as naturally as they talk to each other, with instant responses in their preferred language and voices that sounds Human-like. For organizations, this opens a new way to serve customers, with conversations that feel personal at any scale. Voice agents in Microsoft Foundry Agent Service, now generally available, give teams one enterprise-grade platform to build, deploy, operate and continuously improve those experiences. Over the past year, Microsoft has made significant strides in bringing this vision to life. At Ignite 2025, Voice Live API became generally available, giving developers a unified API for natural, real-time voice conversations with any AI agent. At Build 2026, we introduced Voice Live integration with Microsoft Foundry prompt agents and hosted agents, making it easier to build voice-enabled AI experiences. Today’s release brings together advanced speech capabilities and comprehensive agent lifecycle management in a unified, enterprise-grade platform. Developers can now build, deploy, operate, and continuously optimize intelligent, speech-to-speech conversational agents, accelerating the journey from prototype to production. We're also announcing significant advancements to Speech in Microsoft Foundry: more accurate speech recognition, more natural and expressive voices, and expanded choices across languages, voices, and avatars. Together, these innovations empower organizations to create richer, more human-like voice experiences for customers around the world. Voice agents in Microsoft Foundry Agent Service Voice agents in Microsoft Foundry Agent Service are now generally available. It's an enterprise-grade platform for developers to build Speech to speech conversational voice agents by providing the leading Speech capabilities and supporting the full agent's lifecycle — from building and deployment to operations and continuous optimization. The general availability of voice agents in Microsoft Foundry Agent Service introduces a comprehensive set of capabilities designed to help developers build more natural, intelligent, and reliable voice experiences, while simplifying development and operations at enterprise scale. End-to-end voice agent lifecycle management Going beyond model APIs, Voice Agent builds on the Microsoft Foundry agent platform to help developers build, deploy, and operate enterprise-ready voice agents through a unified platform. Build and connect: Developers can create voice agents in minutes, connect them to enterprise knowledge through Foundry IQ, and integrate with rich tools through server-side MCP and client-side function calling, apply guardrails to help keep conversations safe and aligned with business policies, with private-network support for supported enterprise resources and integrations—helping teams control access to sensitive systems and data. Observe, evaluate, and improve: Built-in voice-native tracing, monitoring, and evaluation make conversations inspectable and quality measurable. For example, if a customer experiences a slow response during a billing inquiry, developers can inspect the turn-by-turn trace to determine whether the delay came from speech recognition, the model, a billing-system tool call, or speech synthesis—and replay the conversation when transcript and audio capture are enabled. Connect to existing channels with easy setup: Deploy voice agents on the web using audio or text input / output or connect them to existing telephony and contact-center infrastructure—including Teams Phone, Twilio, and Genesys—without rebuilding your customer experience. Bring AI-powered conversations to the channels customers already use, while managing agents through a unified platform. More natural, expressive, and intelligent voice conversations Deliver engaging, real-time conversations with greater flexibility in speech models and voices. Expanded choice of speech-to-speech models: Alongside existing GPT Realtime models, we’re introducing new options for more natural voice conversations, giving enterprises greater flexibility to balance conversational experience, agentic capabilities, and cost. GPT Live 1: Delivers full-duplex, highly natural, fluid conversations—from follow-up questions to complex, multi-step requests. Beyond delegating to specialized models, it can route tasks to Foundry prompt agents, and hosted agents across agent platforms, bringing their specialized knowledge, tools, and workflows into the conversation without breaking its flow. This lets enterprises combine an engaging voice experience with agents that handle work behind the scenes, for experiences where conversational richness and task completion both matter. Microsoft Realtime Flash 1: Combines low-cost serving with competitive conversational quality and agentic capabilities across typical voice-agent domains, including customer support, appointment scheduling, and routine service requests. It supports experiences where agents need to understand customer intent, sustain a natural conversation, and use tools to take action—not simply answer questions. Its cost-efficient approach helps enterprises move beyond small pilots toward the economics required for millions of calls per day, keeping operating costs manageable as adoption grows. The model is now in preview with Voice Live Lite tier pricing, offering an affordable path to scaling voice automation. Below is our internal test result on τ³-Voice benchmark. Microsoft Realtime Flash 1 achieved the highest reported average score in internal testing: 70.9 versus GPT-realtime-2.1’s 63.7—a 7.2-point improvement. Gains were strongest in telecom (+16.6 points) and retail (+3.5 points), while airline performance was lower (−6.0 points). Against GPT-realtime-2.1-Mini, the average improved by 39.2 points. More expressive voices: New HD V3 voices bring natural, real-time speech with greater expressiveness, optimized for customer service scenarios. Expanded language coverage includes English, Chinese, Filipino, Indonesian, Malay, and Indic languages. Developers can also control speaking styles and expressions using simple instruction tags, without learning complex markup languages. Enhanced conversation quality: Improved noise suppression helps voice agents stay focused on the primary speaker, reducing interference from television audio and background voices. Combined with more accurate turn-taking and end-of-speech detection, azure_semantic_vad v2 helps distinguish genuine interruptions from brief acknowledgments and speech directed at others—reducing unnecessary interruptions and premature responses for smoother, more natural full-duplex conversations than previous version. Below table reports our test results on Full-Duplex-Bench v1.5, a public benchmark that tests how a voice model behaves when someone else talks while it is speaking. Each column is the share of test cases where the model did the right thing (0 to 1). The total score is the plain average of the four columns. As you can see, the total score improved from 46.3% to 72.2%, a 25.9-percentage-point gain, with the largest improvements in handling speech directed at others and background voices. Customized voice and avatar experiences Create differentiated, branded experiences through customizable voices and avatars. Organizations can use their own data to develop custom voices that reflect their brand identity. We're also introducing VASA-2 Photo Avatar, which transforms a single photo into a lifelike, upper-body talking avatar. Building on VASA-1, VASA-2 delivers more natural movements and greater control over facial expressions, enabling richer and more engaging visual conversations. Flexible multi-agent orchestration for complex voice agents experiences Voice agents can seamlessly collaborate with Foundry prompt agents and hosted agents through flexible multi-agent patterns. Developers can choose the architecture that best fits their applications: Delegation to sub-agents: Voice agents handle real-time conversations while delegating complex reasoning and specialized tasks to backend agents. This keeps voice interactions responsive and low latency without sacrificing reasoning capabilities. Inline sub-agents: Developers can orchestrate multiple specialized voice agents in a graph-based workflow, each with its own instructions, tools, and voice. Agents can seamlessly transition between tasks during a conversation, improving accuracy, controllability, and cost efficiency. Hosted agents as conversation engines: Voice agents manage real-time speech input and output while communicating with hosted agents through bidirectional text streaming. This allows developers to preserve existing agent logic while adding advanced conversational voice capabilities. Together, these patterns make it easier to build sophisticated voice experiences that combine real-time responsiveness with powerful reasoning and enterprise workflows. Simplified development and enterprise integration Accelerate voice agent development with a streamlined developer experience and ready-to-use integrations. Unified developer tooling: Azure Developer CLI (AZD) and GitHub Copilot support help developers quickly build, test, and deploy voice agents using familiar development workflows. Ready-to-use templates: More than 40 industry-specific agent templates accelerate development across customer support and other business scenarios. Native telephony integration: Built-in Session Initiation Protocol (SIP) support enables voice agents to connect directly with existing contact center and telephony infrastructure, making it easier to bring AI-powered conversations into established customer service workflows. Since the public preview launch, thousands of customers have adopted voice agents across a wide range of scenarios, including customer service, automotive, education, human resources, personal assistants, public services, and more. Hear from customers already building voice agents in Foundry: "As a leading airline, Gulf Air is proud to work with Microsoft to advance how we use AI across our business. Through our collaboration, we have migrated and enhanced our existing voice agents in Microsoft Foundry, bringing voice, enterprise knowledge and intelligent tools together in a more integrated experience. This enables us to innovate faster and deliver even better experiences for our teams and customers." Ahmed AbdulJalil Ebranhim, Manager, Digital Innovation & Process Management, Gulf Air “The Hera companion is a voice agent built on Microsoft Foundry. What began as a proof of concept is now an operational experience. Anyone can talk to Hera: she listens and answers in real time. And she doesn’t just answer, she acts. Ask for a tour, and she turns the spacecraft to show you. Switch to Italian halfway through, and she keeps going. But a great voice isn’t enough; she has to be right. With voice agents in Foundry, Foundry IQ grounding and safety controls come together in one place, so every answer is grounded in ESA’s own mission knowledge and checked against Microsoft’s safety controls and our own rules." Daniele Montemaggiori, IT Cloud Services Architect, European Space Agency (ESA) "We tested multiple AI vendors and voice pipelines, and none delivered the customer experience we wanted. Voice agents in Microsoft Foundry finally made AI calls feel like a conversation instead of a phone tree - callers can interrupt, pause to think and speak naturally, and the agent keeps up. With a regional bank customer, our AI agent now handles 30% of calls from end to end. When a caller needs a person, it hands off to a human expert in Microsoft Teams on the same call, with full context." Matt Landis, CEO & Founder, Landis Technologies What’s new in Speech in Microsoft Foundry - More accurate, natural speech models Beyond powering voice agents described above, customers use Speech in Microsoft Foundry for live captions, meeting and call transcription and translation, voiceover for videos and e-learning, and reading content aloud. At Ignite 2026, we are making it more accurate, more natural and offering more choice across three core capabilities: speech to text, text to speech and avatars, and speech translation. Speech to text: understand more languages, faster and more accurately Speech to text converts streaming and file-based audio into high-quality transcripts for voice agents, live captions, contact center analytics, and meeting transcription. These scenarios increasingly involve many languages, speakers who switch languages mid-conversation, and large volumes of recorded audio. Our latest updates, all now generally available, address these needs across real-time transcription, Fast Transcription, and containers. Real-time transcription now offers native multilingual recognition across 50+ languages, with more on the way, so you no longer have to specify the language up front. Results arrive in about 100 milliseconds, with availability in select regions. Multichannel support transcribes two audio channels, such as agent and customer, separately and detects the language on each. For single-language audio, post-stream refinement now spans all language locales. To learn more, see our latest blogs on multichannel transcription and multilingual transcription. Fast Transcription and LLM Speech are now faster than ever, processing audio at a 100+ speed factor across 100+ multilingual languages, including expanded Indic language support. Mixed-language audio can now be transcribed in a single call. LLM Speech ranks among the leaders on Hugging Face's Open ASR Leaderboard and holds the top spot on the multilingual leaderboard, underscoring its quality across languages. Read more in our latest blog post and on the public documentation. Microsoft AI's MAI-Transcribe-2 is now available as an additional model option. On-Prem Containers now support Fast Transcription, letting you transcribe audio files on-premises with GPU inference through the same API you use in the cloud. Learn more Text to speech (TTS) and TTS avatar: more natural voices, more choice, more lifelike avatars Microsoft text to speech voices and avatars power many scenarios beyond voice agents mentioned in section 1 above. In addition, natural and expressive voices are required for voiceover, e-learning and read-aloud content, and other apps need voices that sound clear and engaging. At Ignite 2026, we are delivering two updates: a better and broader set of Neural HD standard voices, and the new VASA-2 photo avatar. New update for Neural HD standard voices: Besides the Neural HD v3 voices enhancement covered in voice agent scenario, we also extend our Neural HD standard voice offering with the Partner Voices (Public Preview), which brings HD voices built by partners such as DataoceanAI and Telus into Microsoft Foundry. Partner voices give customers more choices to find the right one for both their voice agents and content creation scenarios. Locale Voice Name Sample Script Audio Link En-CA TTS-354-F1 Oh no, I completely understand how that must feel—it can be really frustrating when things don’t go the way you expect. But don’t worry, you’re not alone. I’ll be with you every step of the way and make sure we get this sorted out quickly. Let’s take it one step at a time, okay? Ko-KR TTS-038 모두 50대 이상 중년층 고객이 가장 많이 증가한 것으로 나타 났다. Zh-CN TTS-373-F231 哎呀,我不是故意让你失望的!这样吧:如果你想要古怪又有趣、充满笑声的体验,就去看蓝人乐团。如果你想要令人惊叹又充满魔力的表演,太阳马戏团是最佳选择。两者都会让你的第一次约会难忘,所以无论选择哪一个都不会错。 New VASA-2 photo avatar (Public Preview): Avatar brings a face to voice agent and empowers content creation scenarios such as marketing video or training materials. VASA-2 photo avatar is specially designed to enrich the voice agent experience to be more engaged and personal. In Ignite 2026, we bring VASA-2 turns a single photo into a lifelike, upper-body talking avatar, an important upgrade from the talking head of VASA-1. It also moves more naturally and gives you more control over facial expressions to fit your scenario. Learn more. Speech translation: talk across languages in your own voice Language should not stand between people. Live Interpreter API, our real-time speech-to-speech translation service, now runs on a new LLM-based model that translates speech while people are still talking. Breakthrough translation quality: 30% fewer translation errors than the previous production model. Fast response: Translates end to end in under 3 seconds, on par with professional human interpreters. Your voice in every language: With Personal Voice, Live Interpreter API keeps the speaker's voice, style and tone in each language, so conversations feel natural and personal. Learn more Customers are already using Live Interpreter for contact center agent assist, on phones and smart devices such as earbuds and AI glasses, and in international meetings. Get started today Build voice agents: Getting started only takes a few steps: Open the Microsoft Foundry portal go to Build, Agents and New agent, select “Voice”. Choose a voice-optimized model and configure your agent's instructions. Connect knowledge sources and tools as needed for your scenario. Test conversations directly in the built-in voice playground. Deploy to your desired channels and monitor performance using built-in observability and evaluation capabilities. Explore Speech in Microsoft Foundry. Use the Speech playgrounds in Microsoft Foundry Portal to try Speech transcription, voices, avatars and Speech translation. When you’re ready to integrate speech into an application, start with the Speech SDK and Speech in Microsoft Foundry documentation.654Views0likes0CommentsYour Agents Are Running. But Are They Safe and Compliant?
An AI agent can perform perfectly and still violate your organization’s policies. Picture an enterprise running hundreds of agents across customer service, financial transactions, and internal workflows. Every operational dashboard looks healthy: tasks are completing and targets are being met. But are those agents using approved models? Accessing only authorized tools? Running the required safety protections? And when one violates a requirement, can an administrator tell which policy failed, which asset is affected, and what to change? An operationally healthy fleet isn’t necessarily a safe or compliant one. As organizations move from experimenting with individual agents to running AI across business-critical operations, it’s no longer enough to define policies and trust each team to implement them. Enterprises need to enforce the controls that matter at runtime, verify those controls actually work, and confirm, across the fleet, that they’re in place. That’s what Microsoft Foundry Safety brings together. It’s the home for the capabilities that keep agents inside the boundaries you set: Foundry Guardrails: safety controls a developer configures and Foundry applies at runtime. agent controls: the managed policy engine underneath Guardrails, enforcing deterministic rules, classifiers, and LLM-based judgments across frameworks and clouds. Safety Rubric Evaluators: spec-driven evaluations that measure whether a control actually holds, before and after you ship. AI Red Teaming Agent: adversarial probing that finds failures before an attacker does. Compliance dashboard: the new experience that confirms, across your entire fleet, that these controls are assigned, in place, and holding. Each of these has its own story. Governing hundreds or thousands of agents is a different challenge than building one. The new Manage experience in Microsoft Foundry brings fleet inventory, operational telemetry, cost, and compliance into a single view. The Foundry Governance blog walks through Fleet, Operations, and Cost in depth. This post, however, focuses on the capabilities that tells you whether your fleet is safe and compliant. Compliance: from requirements to action At the center of this experience is Foundry Manage → Compliance, bringing together relevant governance signals from Azure Policy, Microsoft Defender for Cloud, and Microsoft Purview. Four complementary views help administrators investigate their AI estate: Compliance view What it helps answer Policy Which organizational policies apply, and where are violations occurring? Asset Is a particular agent, model deployment, or toolbox compliant? Security posture Which security recommendations require attention? Data security & governance What Purview coverage and data-security findings are relevant to my AI assets? Understand compliance from the policy or the asset Administrators and developers come to Compliance with different questions. An administrator may start with an organization-wide requirement and ask which resources violate it, across a subscription or resource group. A developer is more likely to start with a particular agent and ask which policy it failed, why, and what needs to change. The Policy and Asset views support these complementary workflows. Through its integration with Azure Policy, Microsoft Foundry brings organizational governance into the context of your AI estate. Administrators can define and assign policies at the appropriate scope, review compliance status, identify violating assets, and investigate policy requirements. These capabilities span model deployments, agent configurations, tools, network egress, and safety controls, as outlined below: Policy category What organizations can govern Models Restrict deployments to approved models or publishers and enforce model eligibility requirements, such as excluding preview models. Agents Govern approved agent types and trusted Hosted Agent container images. Tools Govern tool usage and connections, including approved tool types, MCP endpoints, and connectors. Network egress Control which outbound destinations hosted agents are permitted to access. Safety controls Establish requirements for safety guardrails, such as harmful-content filtering, jailbreak protection, indirect-attack protection, and custom controls. For example, an organization might establish a policy restricting AI deployments to an approved set of models. In Manage → Compliance → Policy, an administrator can review the assigned policy, examine its scope and configured parameters, understand whether it is compliant, and identify the exact model deployments that violate the requirement. They can also inspect the policy definition, enforcement effect, and available evaluation details to understand why an asset is noncompliant. For changes to assignments or exemptions, Foundry links to the relevant Azure Policy management experience. Audit and Deny provide different levels of control. Audit identifies noncompliant resources without blocking operations. Where supported, Deny prevents new noncompliant create or update operations, while existing resources that violate the policy remain flagged until remediated. The effects available depend on the specific policy definition. This gives administrators a clearer view of not just which policies exist, but where they apply, how they are configured, and which resources require attention. The Asset view provides the complementary perspective. Starting from a flagged customer-support agent, a developer or administrator can examine the relevant policy evaluations and identify the cause: perhaps a disallowed model deployment or a missing required protection. The agent may be operating successfully from an application perspective while remaining noncompliant. Operational health and policy compliance are not the same thing. And knowing a violation exists is only useful if you can act on it. Foundry connects the affected asset to the requirement and the relevant management experience, helping teams determine whether to update a configuration, enable a protection, or follow an Azure Policy workflow. From compliance requirements to safety controls enforced at runtime Configuration compliance is essential, but agents introduce another challenge: an agent can be configured correctly and still attempt an action that violates a business rule. Imagine a financial-services agent offering a new credit product to a customer who has disclosed financial hardship, or an agent attempting a transaction above its authorized limit. These are runtime behavioral requirements that depend on what the agent is doing in the moment. That's why we're bringing more flexible safety controls into the Foundry developer experience. Define safety controls once. Apply them automatically at runtime. With Microsoft Foundry Guardrails, developers can configure safety controls through Build → Guardrails or programmatically through the API. These controls include supported built-in safety protections, such as harmful-content filtering and prompt-attack protection. The architecture is also evolving to support more expressive organization-specific requirements through deterministic business rules and LLM-based judgments. Once a guardrail policy is configured and assigned to a supported agent, Foundry automatically applies its supported controls at runtime. Developers don't need to build a separate policy-evaluation and enforcement integration for the supported Foundry-hosted execution path. The runtime invokes the applicable protections at supported intervention points and applies their decisions as part of execution. Behind this experience are our investments in the open-source Agent Control Specification (ACS), Agent Hooks, and the managed agent controls service. Agent Hooks defines where the runtime can intervene; ACS defines the policy evaluation contract; the managed service evaluates supported policies; and the Foundry runtime enforces the returned decisions. For example, consider an insurance agent processing a policyholder's request to change a beneficiary. An organization might want to require verified identity context before the update, use an LLM judge to detect possible coercion requiring escalation, and apply built-in classifiers to relevant content risks. These are illustrative policy patterns, with availability depending on the supported controls and intervention points. Developers define the boundaries. Foundry handles runtime application of assigned, supported guardrails. Azure Policy complements this experience by establishing and auditing supported organizational requirements; an Azure Policy assignment is not a prerequisite for a properly assigned guardrail to operate at runtime. For more on the underlying architecture and cross-framework integration, see agent controls announcement. Check that the control actually works: Safety Rubric Evaluators Enforcing a control and verifying it holds are two different things. Safety Rubric Evaluators (preview), built on our open-source ASSERT work, turn a plain-language safety policy plus your agent’s own definition and traces into a structured set of Allowed and Disallowed behaviors, then score conversations on two independent metrics, whether the agent did something disallowed, and whether it refused something legitimate. Measuring those separately matters: the easiest way to drive unsafe outputs to zero is to make an agent refuse everything ambiguous, and a single blended score would hide that over-refusal. Teams can generate a rubric from a policy, review and version it in the Evaluator Catalog, and re-run it before and after a mitigation to confirm a fix reduced risk without breaking legitimate user workflows. Together, the pieces form a loop: Guardrails and agent controls enforce policy at runtime, Safety Rubric Evaluators and AI Red Teaming measure whether it holds, and Compliance confirms coverage across the fleet. (Read the Safety Rubric Evaluators deep dive →) Security and data governance belong in the same view Compliance doesn't stop at resource policies and runtime guardrails. An AI resource can meet its configuration requirements and still have security weaknesses or data-exposure risks. Security posture: Microsoft Defender for Cloud The Security posture view surfaces relevant Defender for Cloud recommendations for supported resources. For example, an administrator may discover a recommendation to disable local key-based authentication on a Foundry resource, see which AI resource is affected, and navigate to Defender for deeper investigation and remediation. Data security and governance: Microsoft Purview The Data security & governance view brings relevant Microsoft Purview information into Foundry, including supported coverage information, agent interactions, and prioritized data-security findings. As agents retrieve documents and process sensitive records, administrators can investigate relevant data-protection signals in the context of their AI estate, then use Purview for specialized management and remediation. From compliance signals to action An administrator may begin in Fleet or the Policy view to assess an organization-wide requirement, identify noncompliant assets, and navigate to Azure Policy for an assignment change or exemption. A developer may begin with an agent in the Asset view, inspect the specific violation, and update its model, tool configuration, or guardrails. Both benefit from seeing the requirement and the affected asset in the same Foundry management experience. If an agent is experiencing errors or elevated latency, Operations provides relevant telemetry. If the concern is infrastructure security, teams can investigate in Defender. If it involves sensitive enterprise information, Purview provides its specialized data-security workflows. These systems retain distinct responsibilities for policy evaluation, runtime enforcement, and evidence. What changes is that teams no longer have to begin their investigations with disconnected information scattered across multiple systems. Foundry connects the AI assets organizations build with the policies, protections, and governance insights they need to operate them responsibly. Get started Building an agent that works is only the beginning. The next challenge is building an AI fleet your organization can confidently operate and govern. With Microsoft Foundry Safety, we’re bringing the capabilities that keep agents safe into one place: Guardrails and agent controls to enforce policy at runtime, Safety Rubric Evaluators and AI Red Teaming to verify it holds, and the new Compliance experience to confirm coverage across the fleet. Together with the broader Manage experience — fleet inventory, operational monitoring, and cost — these help organizations move from defining policies to enforcing, verifying, and confirming them. Ready to explore? Start in Microsoft Foundry: Manage your AI fleet: Discover your agents, models, and other AI assets across supported scopes, and understand fleet-level activity and health. Explore Compliance: Navigate to Manage → Compliance to investigate organizational policies, noncompliant assets, security recommendations, and data-governance insights. Understand Azure Policy: Learn how to establish organizational requirements, manage policy assignments, and review compliance results. Configure runtime safety with Foundry Guardrails: Navigate to Build → Guardrails or use the supported API to define and assign safety controls that Foundry automatically applies to supported agents at runtime.201Views0likes0CommentsFrom Agents to Digital Coworkers with Foundry autonomous agents
We are already building agents that reason, retrieve knowledge, call tools, and execute multistep workflows. Some assist people directly. Others run in the background. Increasingly, multiple agents collaborate to get a job done. So what makes a Foundry autonomous agents different? Is it another form of automation? Another autonomous agent? An agent with more tools, better memory, or a task that runs longer? Those capabilities can exist in other agents too. "The more important shift is from an agent that performs work to a digital coworker with its own identity, a standing role in the organization, and a manager accountable for its operation." That changes more than what the agent can do. It changes the operating model around it. A different class, not just a more capable agent We often describe agents by how they are implemented: prompts, custom code, tools, skills, orchestration, models, or managed runtimes. We also describe them by what they can do: reason, plan, remember, retrieve information, use tools, or act autonomously. Foundry autonomous agents introduce another useful distinction: "Who is acting?" Microsoft Learn overview describes three agent types: Agent type Who acts? Where it fits Assistive agent An agent acting on behalf of a signed-in user, within that user's permissions Personal assistance, such as preparing a meeting brief Background service agent An agent using its own app-only permissions, without an agent user account Backend automation, such as responding to an infrastructure alert Foundry autonomous agents An agent with both an agent identity and an agent user account, acting as itself in Microsoft 365 A digital coworker coordinating work with a team These three agent types also represent different operating models for how agents participate in an organization. Autonomy alone does not define a Foundry autonomous agent. The more important distinction is that a Foundry autonomous agent can operate under its own organizational identity rather than inheriting a user's identity to perform its work. That distinction becomes particularly important when an agent moves from helping an individual complete a task to taking on an ongoing responsibility for a team. Its own place in the organization hierarchy Every Foundry agent has a Microsoft Entra agent identity. A Foundry autonomous agent additionally has an Entra agent user account. The two serve different purposes. The agent identity secures the infrastructure and resources used to run the Foundry autonomous agent. The agent user account gives the digital coworker its presence in Microsoft 365 and a place within the organization. This introduces a different collaboration model. Instead of interacting with software that always operates through someone else's identity, a team can work with a digital coworker under its own name. Authorized actions can be associated with that account rather than whichever teammate happens to be signed in. This is particularly valuable in two scenarios: Group work, where there is no obvious person whose identity the agent should borrow. Event-driven work, where the agent may need to respond even when no person is actively signed in. The important distinction is that this is not simply a background service with more automation. The Foundry autonomous agent becomes an organizational participant in Microsoft 365, with its own identity, defined responsibilities, and a manager accountable for its operation. Build once, hire multiple times The second major difference is the developer and deployment model. With Foundry autonomous agent, developers publish a blueprint that teams can use to create their own instances. Think of the blueprint as defining the digital role: Its purpose and instructions, Capabilities and tools, Expected behavior, Supporting platform infrastructure, And the technical boundaries within which its instances operate. Each hired instance then has its own identity, agent user account, assignment, and independently granted access. Imagine a Release Manager blueprint that knows how to track commitments, generate summaries, and work with project information. One team can hire an instance for its release work. Another can hire an instance for a different program. Both digital coworkers share the underlying implementation. They do not automatically share access to each other's business resources. Capability belongs to the blueprint. Business access belongs to the instance. That separation is essential. Adding a SharePoint tool does not automatically grant access to a SharePoint site. Likewise, administrative consent to an API capability does not, by itself, authorize access to every business resource behind it. This allows organizations to reuse a blueprint without hard-coding one team's projects, credentials, or data into the shared implementation. From an agent endpoint to a digital team member The third difference is how people work with the agent over time. A traditional agent experience often starts with an interaction: Ask a question. Invoke a workflow. Get a response. A digital coworker can instead have a standing responsibility that continues across individual interactions. Consider a digital Workstream Manager. Its job is not simply to answer questions about a project. Its responsibility might be to maintain visibility into commitments and outstanding work. A teammate asks it to track a proposal review. The digital coworker records the owner and expected date. Later, the manager asks what remains open, and the digital coworker prepares a workstream summary from the work it has been tracking. The reviewed Workstream Manager sample provides a concrete starting point for exploring this model: persistent work items, manager-controlled teammate access, on-demand summaries, and Microsoft 365 tools for authorized information sources. The important idea here is continuity of responsibility. Instead of treating every interaction as an isolated request, the digital coworker performs a defined role across the team's work. An agent can complete a request. A digital coworker can hold a bounded responsibility over time. Accountability Matters as Much as Autonomy Autonomy gets much of the attention in conversations about AI agents. But as agents become organizational participants, accountability becomes equally important. The Foundry autonomous agent makes this operating model explicit. Different organizational roles participate at different stages: Developers build, publish, and evolve the blueprint and its agent behavior. Azure administrators help provision and govern the supporting Azure resources. Identity and Microsoft 365 administrators review the permissions and organizational access the agents require. Hiring managers hire and onboard instances for their teams. Managers oversee those instances within their assigned responsibilities. Tenant administrators and security teams maintain governance across the broader agent estate. Onboarding establishes the practical boundaries of a particular digital coworker: who can interact with it, which conversations or events it responds to, whom it can communicate with, which resources it can access, and which information sources can ground its work. The manager then oversees that instance while developers continue improving the shared blueprint and administrators govern the broader fleet. This is more than assigning an owner to a software deployment. The manager is accountable for whether the digital coworker should continue performing its role, whether its scope should change, and when it should stop operating. That leads to an important design principle: Autonomy should increase only when identity, authorization, observability, and accountability increase with it. Building the Coworker Is Only Half the Architecture Microsoft Foundry provides the pro-code path for building the behavior of these digital coworkers: custom agent logic, model integration, tools, persistence, orchestration, and evaluation. But building the agent is only part of the architecture. Enterprise adoption also requires controls around identity, permissions, inventory, security, compliance, monitoring, and lifecycle management. Microsoft Agent 365 complements the development model by providing enterprise controls around the agent estate and integrating with capabilities across Microsoft Entra, Defender, and Purview. This distinction matters because designing an effective digital coworker is not only about deciding what it can do. We also need to determine: Who it is, What it can access, Who manages it, Where it can operate, How its activity is observed, and when its authority should end. Network isolation, permissions, data boundaries, telemetry, evaluation, and lifecycle controls therefore become architectural concerns, not afterthoughts. The goal is to control not only how the digital coworker works, but also where, for whom, and under whose authority it works. Start With a Role, Not with Autonomy Foundry autonomous agent can support individuals, teams, and broader organizational processes. But organizations should not start by asking: How autonomous can we make this agent? A more useful question is: What responsibility could we give a digital coworker that has a clear purpose, bounded access, measurable outcomes, and a named manager? Start with one role. Give it explicit responsibilities. Define what it can and cannot access. Determine what requires human approval. Establish how its manager will observe, coach, and eventually offboard it. Then expand its responsibilities and autonomy as confidence grows. The move from agents to digital coworkers is not about making every agent autonomous. It is about giving the right agents a defined identity, a bounded responsibility, and a place in how an organization gets work done. Get Started If you are exploring Foundry autonomous agent, a practical path is: 1. Clone the repository and open the sample folder. These PowerShell commands select the source snapshot reviewed for this article: git clone https://github.com/microsoft-foundry/foundry-samples.git Set-Location foundry-samples git checkout f960ccb05b003fa274e9b5903f9d39ac1228eb65 Set-Location samples\python\autopilot-agents\hello-world Follow the Hello World README for Python 3.11 or later locally, PowerShell 7, Azure CLI 2.80 or later, Azure Developer CLI 1.32 or later, and the azure.ai.agents extension 1.0.0-beta.16 or later. The declared hosted runtime is Python 3.13. Keep the parent autopilot-agents\scripts directory: provisioning and session updates use shared helpers. Before provisioning, confirm a supported hosted-agent region, a Responses-compatible model with quota, resource-management permissions, and Foundry project access. Arrange a tenant administrator for blueprint approval. Sign-in and provisioning do not grant the developer missing permissions; Microsoft 365 licensing and tenant app policies are separate items that would need to be addressed. 2. Define the behavior in Python In Hello World's agent\app.py, the message handler calls the configured model through the Foundry project. This is an excerpt from its existing respond function, not a standalone application: if context.activity.text: response = await project.get_openai_client().responses.create( model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"], input=context.activity.text, ) await send_reply(context, response.output_text) The same file constructs ActivityAgentServerHost with digital_worker=True and registers the Teams handlers. The preview SDK provides the activity-message endpoint, readiness endpoint, hosting, and storage lifecycle. Setting that flag selects the Autopilot identity model; it does not define a useful role or grant business-data access. For richer behavior, the sibling samples separate the responsibilities: File, relative to autonomous agents What it implements hello-world\agent\activity_routing.py Selects supported direct, group, and tagged channel messages foundry-toolbox\agent\toolbox.py Discovers and validates allowed tools; uses Microsoft Agent Framework to handle model-selected calls meeting-assistant\agent\app.py Registers meeting handlers and obtains Work IQ tokens using agent-user authorization meeting-assistant\agent\meeting_lifecycle.py Coordinates meeting start, end, transcript readiness, and persisted occurrence state meeting-assistant\agent\meeting_analysis.py Extracts explicit agenda items and validates structured closure classifications and evidence references 3. Separate deployment configuration from tool configuration Deployment and tool definitions have different jobs. Hello World's azure.yaml describes the hosted service and Microsoft 365 publication. Selected fields are shown below; retain the complete source file when deploying: services: hello-world-autopilot: host: azure.ai.agent project: . language: python codeConfiguration: runtime: python_3_13 entryPoint: main.py kind: hosted protocols: - protocol: activity version: 2.0.0 activity: digitalWorkerType: m365 publish: publishScope: tenant canRespondWithoutMention: true appVersion: 1.0.0 agentDisplayName: Hello World Autopilot The full file includes descriptions, developer and privacy URLs, environment settings, and resource sizing. Review the publication metadata before using it for your organization. canRespondWithoutMention is a publication setting, not a replacement for routing logic: Hello World still requires mentions for channel messages. For a concrete tool manifest, Foundry Toolbox supplies this separate toolbox.yaml: name: public-docs description: Microsoft Learn documentation and Bing-backed public web search. tools: - type: mcp server_label: learn server_url: https://learn.microsoft.com/api/mcp allowed_tools: - microsoft_docs_search - microsoft_docs_fetch - microsoft_code_sample_search require_approval: never - type: web_search name: web_search description: Search the public web for current information with source links. Microsoft Agent Framework discovers those functions and handles model-selected calls. The sample validates a read-only tool allowlist; require_approval: never is specific to this public-information example, not a recommendation for tools that write or access sensitive data. When deploying that sibling sample, follow its README to install azure.ai.toolboxes, create public-docs from toolbox.yaml, and save its MCP endpoint as TOOLBOX_ENDPOINT before azd deploy. Merely adding the YAML file does not create or connect a Toolbox. Public Web Search incurs additional charges and has separate data-handling boundaries; use public questions only. Meeting Assistant uses a different connection: Work IQ MCP, with the delegated WorkIQAgent.Ask permission declared in its azure.yaml and agent-user authorization configured at runtime. Follow its README for the newer agent extension requirement, 1.0.0-beta.18 or later, and a model supporting structured outputs. It also needs tenant consent, transcript API access, applicable licensing, and a Work IQ spending policy. 4. Package and deploy the Python code directly After completing the Hello World README's setup, run the following from its folder containing azure.yaml: $tenantId = "<tenant-id>" az login --tenant $tenantId azd auth login --tenant-id $tenantId azd env new my-hello-world azd provision azd deploy azd ai agent show This agent will also show up in Microsoft Foundry Portal 5. Publish the blueprint, then approve and hire an instance Review Hello World's activity.publish metadata, then submit the deployed agent to Microsoft 365: azd ai agent publish Deployment, publication, approval, and hiring are separate steps. An AI Administrator or Global Administrator approves the requested blueprint in the Microsoft 365 admin center and verifies its presence in the Agent 365 registry. A licensed user can then create an instance in Teams through Apps > Agents for your team, subject to tenant app policies. For Hello World, start with a direct message: "Say hello and tell me what you can help with." Check group-chat delivery and tagged channel messages separately. For Meeting Assistant, use synthetic meeting data to check agenda extraction, event delivery, transcript access, and evidence quality; its behavior is not a Hello World feature. For code-only changes, run azd deploy again. Existing sessions can remain on an older sandbox; the sample provides an explicit update helper: References New to autonomous agent? Start with the overview. Focus on the identity model and the distinction between a reusable blueprint and a hired instance. Ready to build? Start with the Python Hello World walkthrough. Deploy the code, inspect the hosted agent in Foundry, publish the blueprint, obtain approval, and create an instance in Teams. Then choose Foundry Toolbox for public-information grounding or Meeting Assistant for an event-driven team responsibility. Use each sample's own README and configuration together; their tools and prerequisites differ. Planning governance? Review the lifecycle and Agent 365 integration guide. Identify the Azure administrator, developer, tenant approver, and instance manager before deployment. Check current regional support, permissions, and licensing; the autonomous agent quickstart currently requires Frontier enrollment and an available Agent 365 Frontier license for each hired instance. What is an autopilot in Microsoft Foundry? - agent types, identity, blueprints, and collaboration patterns. Autopilot lifecycle in Microsoft Foundry - roles, hiring, onboarding, and accountability. Microsoft Agent 365 integration with Foundry - governance integration and hosted-agent telemetry requirements. Microsoft Agent 365 overview - product name, capabilities, and current product availability. Build your first autopilot - current prerequisites and documented publishing flow. foundry-samples/samples/python/foundry-autopilot-agent at main · microsoft-foundry/foundry-samples Happy Reading!178Views0likes0CommentsBest practices for building agentic AI applications with Foundry Agent Service
This blog was co-authored by Maira Ladeira Tanke, Mark Roy and Vivek Bhadauria Building an agent has become surprisingly easy. Building an agentic application that is reliable, secure, scalable, and doesn't cost a fortune is still hard. Over the last couple of years, agents have moved very quickly from prototypes to real applications. At the same time, the definition of an agent has expanded considerably. We now have agents that use tools, skills, maintain memory, communicate with other agents, execute code, interact with enterprise systems, and take actions on behalf of users. All these capabilities are useful, but they also make it easier to over-engineer an application. This is the first post in a series about building agentic applications with Microsoft Foundry. Foundry Agent Service brings together the tools to build agents, run them in production, and deliver them to users through channels such as Microsoft Teams and Microsoft 365 Copilot. We will cover topics such as evaluation and optimization, security, context management, and scale in much more detail in future posts. For this first post, however, we step back and look at some of the basic engineering principles that we believe are important regardless of the use case. Some of them sound obvious. In practice, they are also very easy to get wrong. Start with the business problem Agentic AI is very popular at the moment, but an agent is only as useful as the challenge it is solving. Before creating an agent, define the problem you are trying to solve. Agentic AI allows applications to deal with situations where the steps required to accomplish a task cannot necessarily be defined in advance. An agent can reason about the information available to it, decide which tools it needs, use their results to determine what to do next, and adapt its behavior along the way. However, flexibility comes at a cost. Every time you introduce reasoning where deterministic application logic would have been enough, you are adding latency, cost, and nondeterminism. Sometimes that is exactly the tradeoff you want to make. Sometimes it isn't. Math is a simple example. LLMs have become much better at math, but if your application already has all the inputs necessary to calculate a value, there is very little reason to spend tokens asking a model to calculate it. Code is faster, cheaper, and deterministic. The same is true for many validation rules, transformations, authorization checks, and predefined business logic. This doesn't mean those operations can't be part of an agentic workflow. They absolutely can. It means the agent can orchestrate them instead of replacing them. A principle we like to use is: if no reasoning is required, an agent is probably not required either. Microsoft Foundry gives us a lot of flexibility in how to build agents. We can define prompt agents through instructions and tools, use the Microsoft Copilot harness (currently in preview) for more complex, multi-step work, or bring our own code and orchestration with hosted agents, while leveraging any agent framework (LangGraph, Microsoft Agent Framework, and more). If you need to bring voice interactions to your application, voice agents for prompt and hosted agents make it simple. Toolbox lets us share tools across agents, and the choice of models means we can pick one that fits the task rather than design everything around a single model. This flexibility is valuable precisely because not every workload should have the same architecture. The objective shouldn't be to build the most agentic solution possible. It should be to build the simplest solution that solves the problem well. Define success early This is another step that is surprisingly easy to skip, especially during a proof of concept. Before spending too much time optimizing prompts, tools, models, or architecture, define what a successful agent means for your use case. For a customer support scenario, success could mean reducing the average time it takes to resolve a case. It could mean increasing case deflection without negatively affecting customer satisfaction. For an insurance claim resolution agent, it might mean reducing the amount of manual work required to complete a process. Those are very different goals, and they result in very different evaluation strategies. Technical metrics are important as well. We want to understand latency, token consumption, tool failures, answer quality, groundedness, and many other characteristics of the system. But a technically impressive agent is not necessarily a useful one. Those same success criteria should guide model selection. Compare candidate models on representative tasks from your application: can they produce grounded answers, choose the right tools, and complete the task within your latency and cost constraints? Choose the model that meets those requirements, rather than defaulting to the most capable model for every task. Re-run those evaluations before switching models. This becomes especially important when we try to calculate ROI. If we didn't establish the baseline and what we expected the agent to improve, it becomes very difficult to demonstrate later that the application created value. Define these measures early and keep them close to the application throughout its lifecycle. They will eventually become the foundation for your evaluations and production monitoring. Don't build it alone There is another tool available to us that we think developers sometimes forget about when building agents: other agents. Coding agents have become extremely capable. Whether you use Github Copilot, OpenAI Codex, Anthropic Claude Code, or one of the many other tools available today, they can help with a significant part of the work involved in building an agentic application. And you should use them. We don't mean asking a coding agent to blindly build your entire application from a one-line description and hoping for the best. The engineering principles in this post still apply. You need to understand the problem you are solving, make architectural decisions, define your success criteria, and validate what gets built. But there is very little value in manually writing every piece of boilerplate when a coding agent can help you scaffold the project, integrate APIs, create tests, troubleshoot an authentication issue, or iterate on your agent implementation. This gets particularly interesting when the coding agent understands your target platform. Bring Foundry into the workflow you already use. Foundry Toolkit for VS Code, Azure Developer CLI, and GitHub Copilot integrations support local development, scaffolding, debugging, and deployment. These tools give coding agents platform-specific guidance and give developers ways to work with Foundry from their terminal and development environment, carrying the same agent implementation toward production. The important part for us is that coding agents should not be treated as fancy autocomplete anymore. Give them the context they need and access to the appropriate skills and developer tools. Let them help you inspect your project, implement changes, run tests, evaluate results, and iterate. Interestingly, many of the recommendations in this post apply to your coding agent too. Give it vague instructions and little context, and you will probably spend your afternoon correcting it. Give it clear requirements, the right tools, platform-specific guidance, and a way to validate its work, and the experience is very different. You are building agents. You might as well use agents to help you do it. If you want to get started, Microsoft's guide for setting up the Microsoft Foundry development environment covers the Foundry developer tooling and how to configure your environment. Be very clear about what you expect from an agent Ever seen an agent get things completely wrong? It happens often. Not because the underlying models aren't capable, but because we tend to expect them to understand information that we never actually supplied, like you would do with your co-worker. The difference is your co-worker has months, if not years of context working with you. An agent only has the context available to it. If the instructions are ambiguous, the agent has to resolve that ambiguity. If two instructions conflict, it must decide which one to follow. If a tool has a vague description, the LLM model has to guess when that tool should be used. The more guessing we introduce, the less predictable the application becomes. Take a tool described simply as: def get_customer_data(): # Gets customer information. ... There is technically nothing wrong with that description, but there is a lot the model needs to figure out. What customer information does it return? When should it be called? What identifier does it expect? Should it be used for transactions as well? What about support cases? We can remove most of that uncertainty ourselves: def get_customer_profile(customer_id): """Returns the customer's account profile, including: - name, - account tier, - region and - account creation date. Use this tool when account profile information is required. Do not use it to retrieve transactions or support cases. """ ... This is a small example, but the same idea applies across a complete agentic application. Instructions, tool definitions, knowledge sources, memory, skills, and even descriptions of other agents are ultimately information we are providing to a model so that it can make a decision. The clearer that information is, the less work the model has to do figuring out what we intended. Longer isn't automatically better either. Adding another three pages to the system prompt every time the agent makes a mistake will eventually create other problems. That leads us directly into context management. Pay attention to what goes into the context When an agent invokes an LLM, a surprising amount of information can end up contributing to that request. There are the agent instructions and the user's input, but potentially also conversation history, tool definitions, retrieved documents, memory, skills, previous tool results, and information coming from other agents. All of that competes for context. Too little context and the model doesn't have enough information to do the job. Too much context and you are paying to process information that might not even be relevant to the current task. Worse, useful information can become harder to identify among everything else you provided. This is also why simply adding more tools isn't always an improvement. Each tool needs a description and input schema that the model can understand, so a growing toolset gives the agent more information to navigate. In Foundry, tool search retrieves relevant tools instead of loading every definition into each request, while skills supply reusable procedures when needed. A toolbox provides a shared, governed endpoint; tool search is what addresses the context cost of presenting the whole catalog. The same applies to conversation history and memory. Persisting something doesn't automatically mean it should be provided to the model on every subsequent request. This is why we think context management needs to be treated as an architectural concern, not just a prompt engineering problem. You need to understand what information is reaching the model, why it is there, how often it is being sent, and whether it is actually improving the outcome. And once applications become more complex, understanding this by looking at the final response becomes almost impossible. Doing it by hand gets tedious quickly, which is where insights and optimization come in later in this post. Which brings us to one of our favorite topics. Make your agent observable from day one Agentic applications can fail in many creative ways. Maybe the model picked the wrong tool. Maybe it picked the correct tool but passed the wrong parameters. Perhaps retrieval returned poor information. Perhaps retrieval returned too much information. Maybe the system prompt contains two instructions that made perfect sense independently but contradict each other when combined. From the user's perspective, all of those problems often look exactly the same: The agent gave me a bad answer. And good luck debugging that from the output alone. This is why we strongly recommend investing in tracing and observability early, including during development. When you can inspect the actual execution of the agent, the conversation changes considerably. Instead of guessing why the agent failed, you can see which tools were called, what was retrieved, where time was spent, and what happened throughout the execution. OpenTelemetry (OTEL) traces are particularly useful here because agentic applications are still distributed applications. The model invocation is important, but so are the tool calls, retrieval operations, downstream services, and other agents involved in completing the request. A trace can show that an answer was bad because retrieval introduced irrelevant context, that latency increased because the agent made five unnecessary calls, or that a tool description caused the model to choose the wrong operation. These are all things we can fix, but only when we can see what the agent is doing. Foundry's tracing helps developers inspect this behavior and understand what happens inside their agentic applications. Insights in Foundry (currently in preview) goes a step further: it analyzes production traces and groups recurring behavior into insights, each with representative traces and a likely cause, so you aren't reading traces one at a time. Once you can observe the behavior, you can start systematically measuring whether it is getting better. Build evaluation into the development lifecycle Evaluations shouldn't be something you run once before going to production. Every meaningful change to an agent has the potential to change its behavior. Change the system instructions and something else might regress. Add a tool and the agent now has another possible choice. Change the model and assumptions you made during development might no longer hold. Change retrieval and the model receives different information. This means you want an evaluation dataset early in development. It doesn't have to contain thousands of examples. Start with representative scenarios that are important to the application. Include the happy paths, but also include ambiguity, unusual phrasing, expected tool calls, cases where the agent should ask for clarification, and cases where it should not take an action at all. You don't have to build that first dataset by hand. If your agent already emits traces, Foundry can convert them into an evaluation dataset using intelligent sampling, which picks a representative set instead of a random slice. The same traces can also seed a rubric evaluator. If you don't have production traces yet, Foundry can also generate a synthetic dataset from your agent's instructions, a prompt, or a reference document such as a policy or spec, which covers prelaunch scenarios and edge cases. The two complement each other: traces show real user behavior, synthetic data covers what users haven't asked yet. The success criteria we defined earlier become useful here. Some evaluations will be technical, such as task completion, tool selection, groundedness, latency, or token consumption. Others can be much closer to the business outcome. Foundry gives us a few ways to turn those criteria into evaluations. The built-in evaluators cover common measures such as relevance, groundedness, task completion, tool call accuracy, and safety risks, so you get a first signal without writing anything. For criteria specific to your application, rubric evaluators use an LLM as the judge to score responses against weighted criteria you define, and you can auto-generate the first version of the rubric from your agent's instructions. Custom evaluators cover the rest: code-based (a Python function for deterministic checks such as format or length), prompt-based (your own judge prompt), and endpoint-based (an HTTP endpoint you host, useful when scoring logic lives in your own infrastructure). Rubric and custom evaluators are currently in preview. As the application evolves, the dataset should evolve with it. Production failures that matter are particularly good candidates for future regression tests. Evaluation also has a practical limitation: it costs money. Running every evaluator against every interaction in production can quickly become expensive. This is why an evaluation strategy matters just as much as having evaluations. Match the evaluation scope to the stage of development. During development, run smaller benchmarks frequently. Before important releases, run larger test sets through your CI/CD pipelines. In production, evaluate representative samples. The aim is enough data to understand behavior and identify regressions, not every evaluator on every request. Foundry brings evaluation into the same development lifecycle as the agent. Combined with agent insights, that gives us a way to move from diagnosing behavior to testing improvements. The agent optimizer uses that evaluation evidence to generate and rank candidate instructions, skills, tool descriptions, and model selections for prompt and hosted agents, which takes over much of the manual tuning that context management otherwise needs. We still need to assess quality, latency, and token consumption together: a version that scores better does not automatically cost less to run. There is enough here for an entire post, so we will explore evaluation and agent optimization separately in this series. Be deliberate about identity and isolation Personalization is important, but it comes with an equally important requirement: my experience with an agent can be different from yours, but my data should never accidentally become part of yours. That sounds obvious. Implementing it correctly requires thinking about several different concepts. Conversation state, session isolation, memory, user identity, agent identity, and authorization are related, but they aren't interchangeable. Memory, for example, can make an agent much more useful. The agent might remember preferences or information that allows it to provide a better experience over time. Foundry's managed memory, currently in preview, provides persistence without requiring developers to provision an external memory database. But managing the infrastructure doesn't answer the governance questions: whose information is it, where is it stored, how is it scoped, when should it be retrieved, and how long should it be retained? Those decisions still belong in the application design. Sessions introduce another boundary. In hosted agent architectures in Foundry, session isolation provides an execution boundary for a workload. Prompt-based experiences have their own mechanisms for maintaining conversation state. In either case, developers need to be deliberate about where state lives and what is shared (and can't be shared) between interactions. Identity becomes especially important when agents start using tools. Suppose we ask an agent to retrieve a document from a system we use at work. The easiest implementation might be to give the agent a highly privileged application identity capable of reading everything. It is probably not the implementation we want. Microsoft Entra ID gives us the capabilities to do this differently. One pattern you will encounter is On-Behalf-Of, usually shortened to OBO. At a very high level, OBO allows an application to call another service using the user's delegated identity. If you are using an agent to access information on your behalf, the downstream system can still make an authorization decision based on who you are and what you are allowed to access. Foundry supports this through user delegation, and the configuration guide shows how to set it up. You don't need to understand every OAuth detail to appreciate why that matters. If you cannot access a document, adding an LLM between you and the document shouldn't suddenly give you access to it. Foundry provides security building blocks, but they still need to be configured for the application. Microsoft Entra Agent ID gives agents an enterprise identity, and the Foundry runtime honors Agent 365 governance actions. Private connectivity, tool authentication and runtime guardrails help control access and behavior, with support varying by component and configuration. Security, identity, memory, and isolation get complicated quickly, so we will examine those boundaries and tradeoffs in a follow-on post. Until then, the identity and access overview is a good map of where each decision sits, and agent identity concepts explains how Foundry uses Entra Agent ID. Keep application logic in your application Earlier we mentioned math as an example of something that doesn't necessarily belong inside an LLM invocation. The same principle extends much further. Validation, transformations, deterministic routing, permission checks, sorting, filtering, and established business rules are operations that traditional software is very good at. Use it. An agent is part of an application. It doesn't replace the application. If a user asks an ambiguous question that requires interpreting intent and deciding which system can answer it, that's a good place for reasoning. Once the system knows exactly which API to invoke and has received structured data back, many subsequent operations are better handled by code. This distinction is useful for reliability, but it can also have a significant impact on latency and cost. A few unnecessary LLM calls probably won't matter during a demo. At production volume, they matter a lot. There is one nuance here. Telling an LLM not to calculate something directly doesn't mean an LLM can't help solve the problem. LLMs are also very good at generating code. For some workloads, generating code and executing it in an appropriate controlled environment can be a much better architecture than asking the model to reason through every intermediate calculation in tokens. This is especially interesting for data analysis and tasks where the requested computation itself is dynamic. Foundry gives us an off-the-shelf Code Interpreter tool that acts as a secure sandbox for this approach. The decision is not whether the model is capable of doing something. In many cases, it is. The more useful engineering question is whether a model invocation is the right component of the application to do it. Plan for scale: within an agent, and across your organization A successful proof of concept eventually faces two different scale problems: one agent serving thousands of users, and an organization creating thousands of agents. The first is about production capacity and reliability. The second is about reuse, ownership, and governance. An agent that works very well for ten people needs to continue working when thousands of users invoke it concurrently. At that point you start thinking about throughput, latency, quotas, model capacity, downstream dependencies, rate limits, connection management, resiliency, and cost. Agent runtimes can help with part of that problem, but your agent is only one component in the architecture. Its models, tools, data sources, APIs, and other dependencies all need to support the level of traffic you expect. This is also where capabilities such as AI Gateway become interesting. As AI usage grows, organizations need mechanisms for managing access to models, controlling and observing traffic, enforcing policies, and understanding consumption across applications. Cost governance needs explicit ownership and limits: set budgets for each application or team, track consumption against them, and define who responds when spending approaches those limits. Azure Cost Management remains the system of record for spend. AI Gateway provides project-scoped tokens-per-minute limits and total token quotas; those are token controls, not native dollar-budget enforcement in Agent Service. Decide in advance how budget alerts and request controls will work together, including when to alert, throttle, or stop requests, rather than waiting for the bill to reveal a problem. And there is another kind of scale that we find equally important: what happens when the problem isn't one agent serving thousands of users, but an organization creating thousands of agents? The requirements change. Different teams may independently create similar tools. Every agent might implement authentication differently. Tool integrations get duplicated. Agents need owners and lifecycle management. Developers need ways to discover capabilities that already exist. Organizations need governance without preventing teams from moving quickly. Tools should become reusable building blocks rather than implementations hidden inside individual agents. This is one of the reasons we find approaches such as reusable Foundry tooling and MCP particularly interesting. A well-designed capability can be exposed once and consumed by multiple agents instead of every team independently rebuilding the same integration. The agents themselves can also become reusable. A specialized agent built by one team might provide useful capabilities to another agent or application. Multi-agent communication makes it possible to compose those capabilities instead of creating one enormous agent that knows how to do everything. Distribution is another part of this problem. Users shouldn't necessarily need to visit a separate application for every agent an organization creates. Depending on the use case, agents may need to appear in applications, websites, Microsoft Teams, Microsoft 365 Copilot, or other channels users already work in. Foundry Agent Service provides a publishing path into Microsoft Teams and Microsoft 365 Copilot, with Agent 365 integration for tenant-wide governance. That connects the engineering decision about where an agent runs with the adoption question of where people will actually use it. Once organizations reach this point, the problem is no longer how to build an agent. The problem is how to provide an agentic platform that allows many teams to build agents without every team independently solving identity, tooling, observability, evaluation, governance, integration, and deployment. We are going to spend much more time on that problem later in the series. And this is just the start The goal isn’t to add more AI at every step. It’s to build applications that solve real problems and keep improving them as we learn. These principles are a starting point; their value comes from putting them into practice. In subsequent posts, we’ll get hands-on with evaluation and optimization, security and identity, context management, and scaling, exploring what Microsoft Foundry provides, the architectural tradeoffs, and how to turn what we learn into better applications, and more business value. Authors Maira Ladeira Tanke Maira is a Principal Software Engineer for Agentic AI at Microsoft Core AI, focused on helping customers successfully adopt Microsoft Foundry Agents. Mark Roy Mark Roy is a Principal Software Engineer at Microsoft working on Microsoft Foundry and AI agents, passionate about helping developers get agents out of the demo stage and into production. Vivek Bhadauria Vivek Bhadauria is a Partner Software Engineer at Microsoft focused on building AI agents, platforms, and optimization loops.1.5KViews5likes0CommentsBuild a dual-mode voice agent in Microsoft Foundry with native subagents
Voice agents are changing how companies interact with customers. From service questions to technical support, the expectation is the same: a natural conversation that leads to useful, reliable answers. But responding quickly and resolving a complex request are not always the same task. Some questions can be answered directly. Others require information from backend systems, specialized analysis, or several steps to determine what should happen next. The challenge is to support that deeper work without putting the conversation on hold, or giving an answer before the necessary information is available. In this blog, I will walk through a dual-mode voice-agent pattern in Microsoft Foundry. A frontline agent manages the real-time conversation, while a backend agent handles specialized work. We will explore how these responsibilities work together, how the frontline can continue interacting while backend processing runs, and how to configure and verify the experience. The pattern is broadly applicable, with particular relevance to telecom customer service and support. You will learn how to: Separate real-time conversation from specialized backend work. Configure a native voice subagent Understand how Voice Agents build on Voice Live Test delayed responses, interruptions, missing information, and backend failures. How voice agents work Before we look at our full workflow. Lets have a look at the voice experience in Microsoft Foundry. A voice application needs to carry audio, manage the conversation, and connect that conversation to useful information or actions. Those responsibilities work together, but they are not the same layer. A browser or contact-center connector handles audio capture and playback The voice runtime manages streaming and turn-taking Instructions, knowledge, and authorized tools determine what the agent can say and do. In Microsoft Foundry, Voice Live provides the real-time speech foundation. Voice Agents build on it with a named, versioned agent definition and capabilities such as tools, tracing, and evaluation. The diagram below shows the underlying voice pipeline. Audio arrives from the client Before anything else, Voice Live cleans up that audio. It uses conversational enhancements that can reduce background noise and echo and help detect when a speaker has finished a turn. The exact combination depends on the selected model, pipeline, and audio settings; not every session uses all the processing stages shown The conversation can then either follow a native speech-to-speech path. A model in the GPT Realtime family processes spoken input directly and produces a spoken response. Supported configurations can also use Azure Speech voices for output, so choosing a realtime model does not mean we must use its native output voice. Alternatively, a cascaded pipeline converts speech to text, sends that text to a language model or agent, and synthesizes the response as speech. This separates the recognition, conversation model, and voice choices. Voice Live can manage that pipeline without us writing the streaming and turn-taking loop ourselves. In either path, useful answers may also require knowledge sources or external tools. These connections give the agent access to information and actions beyond the conversation itself. Configure the required permissions and safety controls for the selected model and application; a tool or content-safety box in the diagram is not a substitute for those decisions. On the return path, speech streams back to the client for playback. Where supported, Azure Speech provides standard and HD voices, with custom voice subject to access requirements. These choices describe how the spoken conversation runs. The dual-mode pattern adds a different separation: the frontline owns that conversation, while a backend agent handles a focused task. It does not require two speech-to-speech models, and the backend does not speak directly to the caller. How the frontline and backend work together To keep the conversation responsive while specialized work runs, we separate the caller-facing interaction from the backend task. Each agent has a clear responsibility: The frontline agent is responsible for the interaction. It listens, takes turns, answers supported questions, and explains what happens next. The backend agent has a different job: work through a focused task, such as reviewing incident evidence or recommending a troubleshooting sequence. The important part is that delegation does not have to take the caller out of the conversation. The frontline can remain active while the backend works. In the native subagent pattern, this behavior is configured through the agent definitions, instructions, and response policy rather than a separate routing model The following diagram shows how those responsibilities fit together: The voice client captures and streams the caller’s audio: The client connects the caller to the frontline voice agent, sends microphone audio, and plays returned speech. It can also display transcripts and runtime events. The client can send audio and receive events concurrently. The frontline voice agent decides whether to answer directly or delegate: The frontline owns the live conversation. It handles straightforward requests within its configured knowledge and permissions without invoking the backend. When a request needs specialized work, the frontline uses its instructions and the configured subagent’s description to determine when to delegate. Direct answers return through the voice client without a backend invocation. Foundry invokes the configured backend subagent. The voice agent’s subagent_config references a backend in the same Foundry project by name. Foundry invokes that backend for the delegated task. The frontline retains ownership of the voice conversation; the caller is not transferred to the backend. The backend performs the task and uses its tools. A prompt or hosted agent follows its own instructions and can retrieve data, call APIs, or execute supported functions. Access to service information, incident records, or diagnostics must be explicitly configured and authorized. Meanwhile, the frontline keeps the voice interaction active. Its response_policy controls an immediate acknowledgment, gap-filling speech, and supported progress updates. The basic sample enables only immediate_ack=True; additional behavior requires configuration. The frontline can receive further caller input, but it must not invent backend findings while waiting. Further caller input does not automatically update an in-flight backend task. Results and supported progress return through Foundry. The backend returns its output to the frontline through managed delegation. Where supported and enabled, progress can also inform the conversation before completion. The diagram’s return arrow does not mean every backend automatically streams intermediate findings. The frontline presents the result to the caller. Following its instructions, it explains the findings, communicates uncertainty, and identifies appropriate next actions. Speech streams back through the client for playback. The backend does not speak directly to the caller, and its output is not necessarily identical to the spoken response. Note: The GPT Live samples demonstrate an alternative implementation of the conversation/backend split. They use VoiceAgentLiveDefinition and live.connect(), with direct Responses or managed prompt-agent delegation. The design goal is responsiveness without sacrificing depth. A fast acknowledgment improves perceived waiting time; it does not shorten the underlying investigation or guarantee an accurate answer. Measure conversational responsiveness and backend task quality separately. Decide when the frontline should ask for help With the flow in place, the next question is when the frontline should ask for help. Define clear boundaries in the frontline agent’s instructions and the subagent’s capability description. Rule 1: Answer directly when the frontline has sufficient grounded information and permission to respond. Rule 2: Delegate when the backend owns the required capability. This might include combining information from several systems, assessing service impact, investigating symptoms, or developing a troubleshooting plan. Rule 3: Escalate when the request exceeds the automated workflow’s boundaries. Define what should happen when an action requires human approval, evidence remains insufficient, or a tool failure prevents progress. Implement and test the human-handoff or failure path; instructions alone do not provide it. From the caller’s perspective, this should feel like one conversation, not a handoff between agents. The frontline listens, clarifies the request, and brings in specialized help when needed. The caller does not need to understand which backend agent or tool is doing the work. When backend processing begins, an enabled acknowledgment can make that transition clear: “Let me check that for you.” If gap-filling speech or supported progress updates are configured, the frontline can keep the caller informed while the task runs. Those updates should reflect what is actually known, without implying that a check has completed or a result has been confirmed. Once the backend returns, the frontline explains the outcome in language suited to the conversation. That might be a confirmed finding, a recommended next step, or a clear statement that more information is needed. The goal is not to fill every pause with speech. It is to keep the caller oriented while specialized work happens in the background, and to make the eventual answer useful, understandable, and grounded in evidence. Build the native voice-subagent workflow Now that we have defined the conversation flow, let’s configure the agents behind it. We will use native voice subagents throughout this walkthrough: a VoiceAgentDefinition for the frontline, a prompt agent for the backend, and subagent_config to connect them. Voice subagents are currently in preview. Microsoft does not recommend features in preview for production workloads. Step 1: Give the backend one focused responsibility Start with the work the frontline should delegate. A useful backend has a clear responsibility. The subagent can be a Foundry prompt agent or a hosted agent in the same project. For our example here we will create a prompt agent that can analyze information supplied to it. It can be connected to knowledge bases and different tools to help. You can also create a subagent in the Microsoft Foundry portal or deploy a code-based hosted agent. Keep the instructions specific to the backend’s actual capabilities. If it can only analyze supplied information, say so. If you later add an incident lookup, update the instructions and capability description to reflect that change. Instructions guide model behavior; they do not enforce permissions or guarantee an output schema. Restrict access in the tools and services themselves, and validate any structured fields your application relies on. Separate read-only investigation from actions that change a customer account or service. Step 2: Connect the backend to the frontline Next, we can create the caller-facing voice-based agent. We can create this using the SDK. In its instructions we should explain when to delegate to the backend agent and we should add the backend agent to its subagent_config. In the subagent_config, you identify the backend agent by name and describe when the voice-based agent should delegate a request to it in the agent_capabilities, which describes the specialist’s responsibility. You can use the following code to spin up such an agent (Also see: Use a subagent in a Voice-based agent - Microsoft Foundry | Microsoft Learn) import os from azure.ai.projects import AIProjectClient from azure.ai.projects.models import ( RealtimeAudioFormatsAudioPcm, VoiceAgentAudioConfig, VoiceAgentAudioInputConfig, VoiceAgentAudioOutputConfig, VoiceAgentDefinition, VoiceAgentInputTranscription, VoiceAgentServerVadTurnDetection, VoiceAgentSubagent, VoiceAgentSubagentConfig, VoiceAgentSubagentResponsePolicy, ) from azure.identity import DefaultAzureCredential endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"] model = os.environ.get("FOUNDRY_VOICE_MODEL", "gpt-realtime-2.1") definition = VoiceAgentDefinition( model_type="managed", model=model, instructions=( "You are a real-time customer support assistant for service incidents. " "Handle straightforward factual and support-process questions directly. " "For complex troubleshooting, SLA-risk triage, root-cause hypotheses, " "or multi-step remediation planning, call forward_to_subagent for " "the configured telco support agent. " "After the subagent returns, provide a concise, customer-safe summary " "and next actions. " "Do not present uncertain reasoning as confirmed fact." ), store=True, output_modalities=["audio"], audio=VoiceAgentAudioConfig( input=VoiceAgentAudioInputConfig( format=RealtimeAudioFormatsAudioPcm(rate=24000), turn_detection=VoiceAgentServerVadTurnDetection( threshold=0.5, prefix_padding_ms=300, silence_duration_ms=1000, ), transcription=VoiceAgentInputTranscription( model="azure-speech", ), ), output=VoiceAgentAudioOutputConfig( format=RealtimeAudioFormatsAudioPcm(rate=24000), voice="en-US-JennyNeural", voice_type="azure-standard", ), ), subagent_config=VoiceAgentSubagentConfig( subagents=[ VoiceAgentSubagent( agent_name="[BACKEND AGENT NAME]", agent_capabilities=( "Performs deeper SLA-risk triage and troubleshooting " "reasoning, including structured cause hypotheses " "and remediation planning." ), response_policy=VoiceAgentSubagentResponsePolicy( immediate_ack=True, ), ) ], ), ) with ( DefaultAzureCredential() as credential, AIProjectClient( endpoint=endpoint, credential=credential, allow_preview=True, ) as project_client, ): voice_agent = project_client.agents.create_version( agent_name="frontline-voice-agent", definition=definition, description="Voice frontline with subagent delegation", ) print( f"Created voice agent '{voice_agent.name}', " f"version {voice_agent.version}" ) While you currently can't link these subagents through the portal You can see the subagents linked and its instructions in the portal by clicking on the YAML button. Step 3: Configure what the caller hears while waiting With the basic connection in place, we can decide how the frontline should support the caller during backend work. Use the optional response_policy object to control what the voice agent says while it waits for a subagent: immediate_ack: Specifies whether the voice agent immediately acknowledges that it forwarded the request. ack_instructions Provides instructions for generating the immediate acknowledgment. enable_delta_progress Emits progress incrementally instead of waiting for the subagent invocation to finish. The default is false. gap_filling_interval Sets the period without subagent content or user input before a gap-filling response. Use a value from 5 through 120 seconds. gap_filling_instructions Provides instructions for speech generated while the voice agent waits for progress. progress_instructions Provides instructions for summarizing streamed subagent progress as speech. progress_update_interval Sets the minimum time between spoken progress updates. Use a value from 5 through 300 seconds. Step 4: Start a conversation with the voice agent With the frontline and its subagent configured, we can now test the spoken experience. The quickest starting point is the Foundry playground: open the project containing both agents, go to Build → Agents, select frontline-voice-agent, and open it in the playground. Start a voice session and allow microphone access. This tests the saved agent configuration without building a local audio client. For a custom application, we can use the shared native microphone client linked here (Native voice client: dependencies, launch commands, and event handling) or the complete client in the Microsoft Learn walkthrough. Configure it with the same Foundry project endpoint and the name created in Step 2: frontline-voice-agent. The client connects through beta.voice_agents.realtime.connect(), sends microphone audio, and receives speech and events concurrently. The example uses 24 kHz PCM audio and plays the returned audio chunks as they arrive. Follow the setup and readiness sequence of the client we choose rather than combining snippets from different clients. Voice activity detection identifies speech boundaries; the frontline instructions and subagent configuration still determine when to delegate. Start with a general support question, then ask for analysis that should use the backend. Check the execution evidence as well as the spoken answer: an acknowledgment alone does not prove that the subagent ran. In our current configuration, only immediate_ack is enabled, so additional gap filling or progress narration should not be assumed. Use headphones to prevent speaker audio from feeding back into the microphone. Then introduce a delayed response, a caller correction, missing evidence, and a backend failure. Check how the frontline responds and what the execution trace shows. Test interruptions separately: stopping audio playback does not prove that the backend task was canceled. From a microphone to a contact-center call The microphone client helps us validate the agent. Connecting it to a contact center adds another layer: a channel integration that bridges call audio and call-control events to the voice runtime. The division between frontline and backend remains useful, but a working local conversation does not automatically provide telephony routing or human transfer. The Azure-Samples Call Center Voice Agent Accelerator documents browser testing and multiple telephony providers, including Genesys Cloud, Azure Communication Services, and Twilio. It provides a reference for the channel-integration layer. There are also call-center-specific accelerators e.g. This Genesys Voice Live Connector sample. This integration option uses Genesys Cloud AudioConnector and a connector service between Genesys and Azure Voice Live. The connector handles the channel protocol and audio conversion; the agent handles the conversation and its configured tools. Conclusion A dual-mode voice agent separates responsibilities that operate on different timelines. The frontline keeps the conversation clear and responsive; the backend performs specialized work. Native subagent delegation connects them without handing the caller over to another speaking agent. The value is not constant speech or an instant answer to every request. It is an interaction that stays understandable while the necessary work happens, then delivers a response grounded in what that work actually established. Next Steps Ready to go deeper? Use a subagent in a Voice-based agent (preview) – Follow the end-to-end walkthrough to create a backend subagent, configure delegation, and test the spoken experience. Configure a voice agent – Explore model selection, audio settings, turn detection, voices, and agent behavior. Native voice-subagent samples – Explore working examples that connect a frontline voice agent to prompt or hosted backend agents. Voice Live overview – Learn about the underlying speech capabilities, supported models, and regional availability. Genesys Cloud Voice Live Connector – Explore a sample audio bridge between Genesys Cloud and Azure Voice Live, with a simulator for local testing. Call Center Voice Agent Accelerator – Explore deployment guidance and reference integrations for browser-based conversations and supported telephony providers.235Views0likes0CommentsFrom Python Agents to Copilot Harness : Putting Microsoft Foundry to the Test
You have built the agents. Written the Python. Connected the data. Worked out how the pieces talk to each other. Then the conversation shifts to harnesses. The prospect of convergence around the GitHub Copilot harness across GitHub, Microsoft Foundry and Copilot Studio (CPS) raises a very practical question: What happens to the pro-code agents I already built? Do I rewrite them, wrap them, or keep them? Rather than debate it in the abstract, I wanted to give it a shot. I already had a Microsoft Agent Framework (MAF) demo with Python agents, Fabric sales data and product knowledge. Could I keep the useful business workflows, remove the custom agent wrappers, and let a managed harness do more of the work? That experiment became the demo behind this blog. Not another empty chat box, but an app to ask business questions, inspect actual tool activity, follow the evidence and create a brief you can revise and download. This is a practical Foundry experiment and the question I wanted to answer was: how much of this particular agent application can move into the harness, and what still belongs in my code? TL;DR: I moved a Python/MAF research demo to a Foundry-managed Copilot harness to test whether it could handle the same business questions. Native research and source connections reduced the agent-side orchestration I had to write, while the app still owns authentication, streaming and secure access. Downloadable briefs are an extra experiment, not part of the original migration. Focused tests were promising, but do not establish complete feature parity or production readiness. The Starting Point: Pro-Code Agents The starting point is olivierb123/microsoftiq-lead-agent-demo. It used four hosted Agent Framework wrappers and application-controlled fan-out. Python agent code and custom source adapters handled the integrations; the application determined which agents to call and assembled their results. That approach makes the orchestration explicit. It also puts the responsibility for hosting the wrappers, maintaining source adapters and coordinating the workflow in the application. For this experiment with harness, I kept the business scenario but changed the execution model. Instead of recreating those wrappers inside another service, I configured one Foundry-managed Microsoft Copilot harness agent with instructions, native capabilities and connected sources. Responsibility Pro-Code Demo Harness Experiment Workflow orchestration Application-controlled agent calls Managed agent selects tools and can delegate Research Hosted wrappers and custom adapters Native web search and page fetching Business data Application source integration Native Fabric connection Product knowledge Application retrieval integration Foundry IQ knowledge-base retrieval through native MCP Working output Application-managed result assembly Conversation workspace and published artifacts Application code UI plus agent orchestration UI, authentication, streaming and session/download authorization This repository contains the migrated implementation. The original pro-code demo remains separate and was not overwritten. A Practical Question, Not a Tool Demonstration The scenario is a fictional sales team working with FieldForge product documentation. A useful assistant should help answer questions such as: How many opportunities are open in the expanded synthetic demo, what is their total and probability-weighted value, and how do our internal forecasting definitions explain those numbers? That request needs two different kinds of evidence: structured opportunity data and the actual internal document defining the forecast. It should not require the user to name a connector or prescribe a tool sequence. A more advanced request asks for a South territory brief combining sales performance, priority opportunities, product capabilities and official construction heat-exposure guidance. The user can then request a revision or a downloadable Markdown artifact in the same conversation. These examples exercise the harness as a working environment, not just a text-generation endpoint. However, they are requests to explore, not promises that every tool or subagent will run on every turn. Why a Local Web App? I deliberately kept the test experience in a local web app. I wanted to make it easy to ask a question, try a follow-up and inspect what happened, without turning the experiment into a web-hosting project first. The React workbench and Node.js backend run locally; the agent and connected sources run in the cloud. After signing in, I can choose a sample question, watch the streamed response, inspect Activity and Evidence, and request a revision or download in the same conversation. I do not need to reconstruct a raw SDK response to see whether a source was actually called. That separation also makes the migration easier to understand. The web app is the test surface, not the agent runtime. There is no local Python agent server hiding behind it: the backend sends the request to Foundry and handles the application's authentication, streaming and session-owned downloads. For this first experiment, the backend is intentionally loopback-only. Hosting the application publicly is a separate engineering task, not a prerequisite for testing the harness against a useful scenario. The Architecture The configured agent uses microsoft_copilot_preview and gpt-5-mini. The agent definition lives in harness/azure.yaml, rather than a Python agent server. The application, backend and data scripts use JavaScript/Node.js. There is no Python application runtime, pip setup or virtual environment required for this version. That is not a zero-code migration: the application still owns sign-in, token validation, streaming, conversation ownership and secure downloads. The backend does not execute the native source calls a second time. Foundry executes them; the application observes their returned results and presents them in Activity and Evidence. What It Looks Like in the Foundry Portal The local workbench is only one view of the experiment. Foundry also supports hosted pro-code agents: I can package custom agent code, including a Python/MAF implementation, and its dependencies in a container and let Foundry manage the hosting. I still own the agent implementation and its orchestration; using Python does not mean I must run the server myself. This experiment instead uses a managed-harness prompt agent. I configure its instructions, capabilities and source connections, while Foundry supplies the harness execution environment and native tool orchestration. Both approaches can run in Foundry. The distinction is not simply local versus cloud, or Python versus no Python: it is how much agent runtime and orchestration I implement versus delegate to the managed harness. The application's UI, authentication and session/download authorization remain my responsibility in this demo. My microsoftiq-copilot agent and its configured tools/connections make the change tangible. The configuration ties together the native harness capabilities, Fabric access and product-knowledge retrieval shown in the architecture diagram. No Web IQ Wrapper in This Version The Web IQ integration from the earlier demo is not carried forward. External research uses the harness's native web search to discover sources and its Web capability to read relevant pages. This supports the research portion of the scenario without maintaining a separate Web IQ agent wrapper. It is not a claim that native web research reproduces every Web IQ behavior or data contract. Fabric and product knowledge are optional grounding sources, not stages that every question must pass through. A public-research question can stay a public-research question. Private business data must not be sent to public web tools. More Direct Integration with Fabric and Foundry IQ For me, the integration improvement is the ownership boundary: the harness uses configured native connections instead of calling application adapters that obtain credentials and query sources themselves. Fabric supplies structured business facts. The native Fabric connection uses the signed-in user's context to access a dedicated data agent over imported semantic models. There is no app-only Fabric credential fallback. The models contain demonstration data, not live CRM records. Foundry IQ supplies document evidence. The product-knowledge MCP connection reaches a knowledge base backed by Azure AI Search. Query planning and semantic retrieval return extractive evidence, including stored document metadata, for the harness to use in its answer. This connection uses the Foundry project's managed identity; it does not establish per-user document ACL enforcement. The distinction matters. A supportingDocId in an opportunity row is a relationship reference, not proof that the corresponding document has been retrieved. Fabric cannot substitute for document content, titles or update dates from the knowledge base. Calling this integration "better" means more direct native integration with clearer source responsibilities. I have not established a quality, latency or cost improvement over the MAF implementation, nor certified genuinely different-user authorization. What the Experiment Actually Showed Focused tests showed that the harness could retrieve sales figures, find internal documents, research the web and continue a conversation. That supported the core research workflows we wanted to carry over from the MAF demo. The most revealing test asked for sales figures together with the internal document explaining how the forecast should be interpreted. Some early runs got the numbers right but did not retrieve the requested document. The answer sounded convincing, but part of its explanation lacked verified support. After clarifying the agent's instructions and which source should answer each part of the question, one test run successfully retrieved evidence from both sources. This used synthetic demo data, not live customer information. The lesson: connecting the tools is only the start. You still need to test whether the agent uses the right sources and whether those sources actually support its answer. Beyond the Migration: Trying Workspace Files and Briefs Once the core question-answering workflows were working in focused tests, I wanted to try something we had not implemented in the MAF demo: turning an answer into a working document. Why save a brief? Not every answer needs one. But a territory review is a useful example: instead of leaving the analysis in a chat response, I can request a Markdown document to keep, review or hand off, then ask for a revision in the same conversation. The file is an optional deliverable, not a prerequisite for retrieving sales data or product evidence. This is where the harness's conversation workspace and native file capabilities become interesting. I can explore creating, reading back and revising an artifact in the managed environment without rebuilding those agent-side tools in a Python wrapper. MAF could support the same kind of workflow with appropriate tools and code; this is an additional capability in this demo, not something exclusive to a harness. Publication adds a separate step when the user actually wants a download. A workspace file is not automatically downloadable: the workbench registers it only after a successful native upload returns a real file ID, and only the owning unexpired session can download it. The application still implements those access checks. The migration question is therefore "can I keep answering the useful business questions?" The extension question is "what else can I build on the harness's native capabilities?" Briefs help explore the second question. Actual tool results, not assistant narration, establish whether creation, revision or publication occurred. There are still operational boundaries. The app is loopback-only, the harness is a public preview, inference and retrieval incur costs, and workspace state is not durable application storage. Refreshing the browser or restarting the backend loses local session mappings. Production hosting, retention and broader authorization need separate work. Shell and external business-system writes remain disabled. The Takeaway So, what happened to my Python agents? For this scenario, their hosted wrappers were no longer needed. The business questions stayed; their execution moved into a managed harness, and the remaining application code focused on the experience and its security boundaries. For this practical research scenario, that shifts the work from maintaining Python wrappers and fixed fan-out toward configuring the agent, defining source responsibilities, building the user experience and verifying evidence. Fabric and Foundry IQ fit into that environment directly, while native web research replaces the earlier Web IQ path for the tasks explored here. When I Would Still Use MAF Deterministic Routing and Local Hybrid Execution: MAF allows you to deploy hybrid topologies where small, local models (like MAI-Decision / Jev) handle rapid, deterministic triage and tool-gating, reserving expensive cloud-hosted frontier models for complex reasoning. A managed harness forces all routing into its proprietary cloud runtime, stripping away your ability to optimize latency and compute token economics. Absolute Control over the Orchestration Loop: The Copilot Harness uses an abstracted reasoning engine to decide when to activate a skill or invoke a tool. Native Toolchain Continuity and Hosting: MAF agents operate as standard C# or Python applications. Granular State and Semantic Memory: Managed harnesses provide a generalized shared memory layer. MAF allows you to build custom memory providers (like custom Azure SQL or Cosmos DB schemas) Unrestricted Interoperability Mechanics: While the Copilot Harness can consume Agent-to-Agent (A2A) protocols and Model Context Protocol (MCP) servers, building the actual server logic in MAF gives you the power to manipulate the raw protocol mechanics. If you need to manage custom message headers, inject highly specific tokens for platforms like SAP or ServiceNow, or alter payload structures mid-flight, MAF provides the unabstracted access required to do so. For this demo, the harness was a useful way to move beyond fixed fan-out and put research, connected evidence and artifacts into one managed environment. For another application, keeping its MAF agents may be the better decision. My takeaway is simple: start with a real task, move the responsibilities that fit the harness, and test the boundary instead of assuming the migration works. What's Next This experiment answered one question—now I'm curious about others: How does the harness perform under sustained load? What happens when Fabric semantic models grow to production scale? Can the same approach work for agents that need to write back to business systems? If you're exploring similar migrations, I'd be interested to hear what worked and what didn't. Try the Project The code is available at olivierb123/microsoftiq-copilot-harness-demo. Choose your path: Run the demo locally → Start with the README and setup checklist Explore the agent configuration → See harness/azure.yaml for the Foundry agent definition Understand the data model → Review the synthetic data documentation Compare with the original MAF approach → Check out the original pro-code demo Questions or feedback? Open an issue on the repo or share your own migration experience in the comments below.372Views2likes0CommentsTurn enterprise knowledge into a personalized voice and avatar agent in minutes with Voice Agent
A customer asks how a product works, then follows up: “Can you give me an example?” They need a useful explanation based on your product documentation, without searching through pages of information. Now imagine that conversation delivered in a voice and with a face that represent your brand. Voice Agent in Foundry Agent Service brings three things together in one real-time conversation: your knowledge through Foundry IQ, your voice through Personal Voice, and your face through a custom photo avatar. Once these assets are prepared, combining them takes a few minutes in the Foundry portal: create a Voice Agent, connect the knowledge, select the voice and avatar, and start talking. You don't need to build speech, avatar or app code to try it: the playground handles all of that. Why Voice Agent Minutes, not a project. With your knowledge base, voice and avatar ready, setup is four steps in the portal, with no code required to try the experience. Answers grounded in your content. Foundry IQ connects the agent to your documents, and you can instruct it to say when the materials don't contain an answer instead of guessing. A natural conversation. Users speak in real time and ask follow-up questions that build on conversation context. With interruptions enabled in a supported voice configuration, users can also interrupt the agent mid-answer. Build the identity once, update the knowledge anytime. Refreshing your documents does not require recreating the voice or avatar, and the same prepared assets can be reused across agents. Give your knowledge a voice and a recognizable face Personalization is most useful when it helps someone complete a task. A common pattern brings together your knowledge, your authorized identity, and a Voice Agent that makes both conversational. Here are some places to start: Scenario Knowledge to connect Voice and avatar Product concierge on your website or in a store kiosk Product manuals and approved FAQs Your brand spokesperson Employee onboarding and HR help Policies, handbooks and benefits guides A familiar HR partner or team lead Internal IT helpdesk Runbooks and troubleshooting guides Your support team's representative Training and e-learning tutor Course materials The course instructor Museum, tourism or campus guide Exhibit and visitor information A curator or guide Expert “office hours” Talks, FAQs and published content The expert, with their consent In each case, users can move from an overview to a concrete example at their own pace, in the same conversation, instead of navigating long documents or restarting their search. These are starting points to adapt to your audience and approved use, not customer deployments. Watch the conversation, not just a talking avatar The demo below uses a Theodore Roosevelt knowledge base to show three connected interactions with Voice Agent in the Foundry playground. The historical topic is a compact example; replace it with your own knowledge to apply the same pattern to any of the scenarios above. Ask the Voice Agent What it shows What were Theodore Roosevelt's main contributions to conservation? A grounded answer from the connected knowledge base Could you give one specific example and explain why it mattered? A follow-up that builds on the conversation context What was Microsoft's closing stock price yesterday? Knowing its limits: the agent says the available materials do not contain the answer instead of guessing The voice and avatar in this demo are AI-generated. Theodore Roosevelt is the subject of the knowledge base, not the identity portrayed by the assistant. The point is not simply to animate a response. It is to let a user get an explanation, explore it through a follow-up and understand when the connected knowledge cannot help. The demo uses a platform-provided voice (Andrew Dragon HD) and a preset photo avatar (Marcel); with Personal Voice and a custom photo avatar, the conversation works the same way, with your own identity. From prepared assets to a conversation in four steps Start in a supported Foundry project with these resources prepared, or reuse ones you already have: Knowledge: a Foundry IQ knowledge base created from your authorized documents. Try a sample question to preview the results. Voice: a Personal Voice created from a short, clean voice sample and the speaker's recorded consent statement. Appearance: a custom photo avatar created from an authorized photo and a consent video of the person pictured. You can also start right away with platform-provided voices and avatars and personalize later. The Voice Agent setup guide walks through preparing each resource, including the consent and eligibility requirements, for both the portal and Python. 1. Create a Voice Agent In the Foundry portal, open your project and go to Agents → Build an agent. Enter a name, select Voice as the interaction mode, choose the appropriate use case, and select Create agent and open playground. Review the model and default instructions for your scenario. 2. Connect the knowledge that helps users complete their task In the agent configuration, expand Knowledge, select Add, and choose your prepared Foundry IQ knowledge base from Available knowledge bases. Give the agent a clear purpose, such as explaining product features from approved documentation, and keep spoken answers concise. For example: Use the connected knowledge base for factual answers. Give a brief, direct answer and use conversation context for follow-up questions. If the available materials do not contain the answer, say so rather than guessing. This gives the conversation a useful boundary: the knowledge supplies the answer, and the voice and avatar determine how it is presented. 3. Choose how the agent sounds and appears Under Voice, select your Personal Voice (Custom → Personal). Enable Avatar and choose your custom photo avatar (Custom → Photo avatar). Preview the selections to confirm they match the intended identity and are available for your project's resource and region. This is where the experience becomes recognizable: the same product knowledge can be delivered by your brand spokesperson or your trainer, and you can swap the knowledge or the identity independently. 4. Save and start talking Select Save, then Start session, and allow browser microphone access. Ask about your content, follow up naturally (for example, “Can you give me an example?”), and try a question outside your documents to check whether the agent acknowledges when the information is unavailable. That's it: your knowledge, voice and avatar are working together in a live conversation, right in the playground, with no local SDK setup. Check the answers against your source documents before sharing the agent with users. Personalize with permission Use voice, likeness and documents you are authorized to use, complete the required consent and access approvals, and disclose synthetic voice and avatar output. Personal Voice and custom photo-avatar capabilities have eligibility requirements; available models, regions and preview features vary. The setup guide links to the applicable requirements. Review audience access and data handling before using enterprise content, and validate the experience for your intended deployment. Build your first Voice Agent Start with one focused knowledge collection, one authorized identity and three useful questions. Voice Agent in Foundry Agent Service turns those building blocks into a conversation that people can hear, see and follow up on. Ready to build it? Follow the step-by-step setup guide to prepare your knowledge base, Personal Voice and photo avatar, then bring them together in a Voice Agent through the Foundry portal or Python. For more documentation, quickstarts and samples, explore the Microsoft Voice Agent repository. Your knowledge, your voice, your face: one conversation, a few clicks away.554Views1like0Comments