Best practices for building agentic AI applications with Foundry Agent Service
This blog was co-authored by Maira Ladeira Tanke, Mark Roy and Vivek Bhadauria Building an agent has become surprisingly easy. Building an agentic application that is reliable, secure, scalable, and doesn't cost a fortune is still hard. Over the last couple of years, agents have moved very quickly from prototypes to real applications. At the same time, the definition of an agent has expanded considerably. We now have agents that use tools, skills, maintain memory, communicate with other agents, execute code, interact with enterprise systems, and take actions on behalf of users. All these capabilities are useful, but they also make it easier to over-engineer an application. This is the first post in a series about building agentic applications with Microsoft Foundry. Foundry Agent Service brings together the tools to build agents, run them in production, and deliver them to users through channels such as Microsoft Teams and Microsoft 365 Copilot. We will cover topics such as evaluation and optimization, security, context management, and scale in much more detail in future posts. For this first post, however, we step back and look at some of the basic engineering principles that we believe are important regardless of the use case. Some of them sound obvious. In practice, they are also very easy to get wrong. Start with the business problem Agentic AI is very popular at the moment, but an agent is only as useful as the challenge it is solving. Before creating an agent, define the problem you are trying to solve. Agentic AI allows applications to deal with situations where the steps required to accomplish a task cannot necessarily be defined in advance. An agent can reason about the information available to it, decide which tools it needs, use their results to determine what to do next, and adapt its behavior along the way. However, flexibility comes at a cost. Every time you introduce reasoning where deterministic application logic would have been enough, you are adding latency, cost, and nondeterminism. Sometimes that is exactly the tradeoff you want to make. Sometimes it isn't. Math is a simple example. LLMs have become much better at math, but if your application already has all the inputs necessary to calculate a value, there is very little reason to spend tokens asking a model to calculate it. Code is faster, cheaper, and deterministic. The same is true for many validation rules, transformations, authorization checks, and predefined business logic. This doesn't mean those operations can't be part of an agentic workflow. They absolutely can. It means the agent can orchestrate them instead of replacing them. A principle we like to use is: if no reasoning is required, an agent is probably not required either. Microsoft Foundry gives us a lot of flexibility in how to build agents. We can define prompt agents through instructions and tools, use the Microsoft Copilot harness (currently in preview) for more complex, multi-step work, or bring our own code and orchestration with hosted agents, while leveraging any agent framework (LangGraph, Microsoft Agent Framework, and more). If you need to bring voice interactions to your application, voice agents for prompt and hosted agents make it simple. Toolbox lets us share tools across agents, and the choice of models means we can pick one that fits the task rather than design everything around a single model. This flexibility is valuable precisely because not every workload should have the same architecture. The objective shouldn't be to build the most agentic solution possible. It should be to build the simplest solution that solves the problem well. Define success early This is another step that is surprisingly easy to skip, especially during a proof of concept. Before spending too much time optimizing prompts, tools, models, or architecture, define what a successful agent means for your use case. For a customer support scenario, success could mean reducing the average time it takes to resolve a case. It could mean increasing case deflection without negatively affecting customer satisfaction. For an insurance claim resolution agent, it might mean reducing the amount of manual work required to complete a process. Those are very different goals, and they result in very different evaluation strategies. Technical metrics are important as well. We want to understand latency, token consumption, tool failures, answer quality, groundedness, and many other characteristics of the system. But a technically impressive agent is not necessarily a useful one. Those same success criteria should guide model selection. Compare candidate models on representative tasks from your application: can they produce grounded answers, choose the right tools, and complete the task within your latency and cost constraints? Choose the model that meets those requirements, rather than defaulting to the most capable model for every task. Re-run those evaluations before switching models. This becomes especially important when we try to calculate ROI. If we didn't establish the baseline and what we expected the agent to improve, it becomes very difficult to demonstrate later that the application created value. Define these measures early and keep them close to the application throughout its lifecycle. They will eventually become the foundation for your evaluations and production monitoring. Don't build it alone There is another tool available to us that we think developers sometimes forget about when building agents: other agents. Coding agents have become extremely capable. Whether you use Github Copilot, OpenAI Codex, Anthropic Claude Code, or one of the many other tools available today, they can help with a significant part of the work involved in building an agentic application. And you should use them. We don't mean asking a coding agent to blindly build your entire application from a one-line description and hoping for the best. The engineering principles in this post still apply. You need to understand the problem you are solving, make architectural decisions, define your success criteria, and validate what gets built. But there is very little value in manually writing every piece of boilerplate when a coding agent can help you scaffold the project, integrate APIs, create tests, troubleshoot an authentication issue, or iterate on your agent implementation. This gets particularly interesting when the coding agent understands your target platform. Bring Foundry into the workflow you already use. Foundry Toolkit for VS Code, Azure Developer CLI, and GitHub Copilot integrations support local development, scaffolding, debugging, and deployment. These tools give coding agents platform-specific guidance and give developers ways to work with Foundry from their terminal and development environment, carrying the same agent implementation toward production. The important part for us is that coding agents should not be treated as fancy autocomplete anymore. Give them the context they need and access to the appropriate skills and developer tools. Let them help you inspect your project, implement changes, run tests, evaluate results, and iterate. Interestingly, many of the recommendations in this post apply to your coding agent too. Give it vague instructions and little context, and you will probably spend your afternoon correcting it. Give it clear requirements, the right tools, platform-specific guidance, and a way to validate its work, and the experience is very different. You are building agents. You might as well use agents to help you do it. If you want to get started, Microsoft's guide for setting up the Microsoft Foundry development environment covers the Foundry developer tooling and how to configure your environment. Be very clear about what you expect from an agent Ever seen an agent get things completely wrong? It happens often. Not because the underlying models aren't capable, but because we tend to expect them to understand information that we never actually supplied, like you would do with your co-worker. The difference is your co-worker has months, if not years of context working with you. An agent only has the context available to it. If the instructions are ambiguous, the agent has to resolve that ambiguity. If two instructions conflict, it must decide which one to follow. If a tool has a vague description, the LLM model has to guess when that tool should be used. The more guessing we introduce, the less predictable the application becomes. Take a tool described simply as: def get_customer_data(): # Gets customer information. ... There is technically nothing wrong with that description, but there is a lot the model needs to figure out. What customer information does it return? When should it be called? What identifier does it expect? Should it be used for transactions as well? What about support cases? We can remove most of that uncertainty ourselves: def get_customer_profile(customer_id): """Returns the customer's account profile, including: - name, - account tier, - region and - account creation date. Use this tool when account profile information is required. Do not use it to retrieve transactions or support cases. """ ... This is a small example, but the same idea applies across a complete agentic application. Instructions, tool definitions, knowledge sources, memory, skills, and even descriptions of other agents are ultimately information we are providing to a model so that it can make a decision. The clearer that information is, the less work the model has to do figuring out what we intended. Longer isn't automatically better either. Adding another three pages to the system prompt every time the agent makes a mistake will eventually create other problems. That leads us directly into context management. Pay attention to what goes into the context When an agent invokes an LLM, a surprising amount of information can end up contributing to that request. There are the agent instructions and the user's input, but potentially also conversation history, tool definitions, retrieved documents, memory, skills, previous tool results, and information coming from other agents. All of that competes for context. Too little context and the model doesn't have enough information to do the job. Too much context and you are paying to process information that might not even be relevant to the current task. Worse, useful information can become harder to identify among everything else you provided. This is also why simply adding more tools isn't always an improvement. Each tool needs a description and input schema that the model can understand, so a growing toolset gives the agent more information to navigate. In Foundry, tool search retrieves relevant tools instead of loading every definition into each request, while skills supply reusable procedures when needed. A toolbox provides a shared, governed endpoint; tool search is what addresses the context cost of presenting the whole catalog. The same applies to conversation history and memory. Persisting something doesn't automatically mean it should be provided to the model on every subsequent request. This is why we think context management needs to be treated as an architectural concern, not just a prompt engineering problem. You need to understand what information is reaching the model, why it is there, how often it is being sent, and whether it is actually improving the outcome. And once applications become more complex, understanding this by looking at the final response becomes almost impossible. Doing it by hand gets tedious quickly, which is where insights and optimization come in later in this post. Which brings us to one of our favorite topics. Make your agent observable from day one Agentic applications can fail in many creative ways. Maybe the model picked the wrong tool. Maybe it picked the correct tool but passed the wrong parameters. Perhaps retrieval returned poor information. Perhaps retrieval returned too much information. Maybe the system prompt contains two instructions that made perfect sense independently but contradict each other when combined. From the user's perspective, all of those problems often look exactly the same: The agent gave me a bad answer. And good luck debugging that from the output alone. This is why we strongly recommend investing in tracing and observability early, including during development. When you can inspect the actual execution of the agent, the conversation changes considerably. Instead of guessing why the agent failed, you can see which tools were called, what was retrieved, where time was spent, and what happened throughout the execution. OpenTelemetry (OTEL) traces are particularly useful here because agentic applications are still distributed applications. The model invocation is important, but so are the tool calls, retrieval operations, downstream services, and other agents involved in completing the request. A trace can show that an answer was bad because retrieval introduced irrelevant context, that latency increased because the agent made five unnecessary calls, or that a tool description caused the model to choose the wrong operation. These are all things we can fix, but only when we can see what the agent is doing. Foundry's tracing helps developers inspect this behavior and understand what happens inside their agentic applications. Insights in Foundry (currently in preview) goes a step further: it analyzes production traces and groups recurring behavior into insights, each with representative traces and a likely cause, so you aren't reading traces one at a time. Once you can observe the behavior, you can start systematically measuring whether it is getting better. Build evaluation into the development lifecycle Evaluations shouldn't be something you run once before going to production. Every meaningful change to an agent has the potential to change its behavior. Change the system instructions and something else might regress. Add a tool and the agent now has another possible choice. Change the model and assumptions you made during development might no longer hold. Change retrieval and the model receives different information. This means you want an evaluation dataset early in development. It doesn't have to contain thousands of examples. Start with representative scenarios that are important to the application. Include the happy paths, but also include ambiguity, unusual phrasing, expected tool calls, cases where the agent should ask for clarification, and cases where it should not take an action at all. You don't have to build that first dataset by hand. If your agent already emits traces, Foundry can convert them into an evaluation dataset using intelligent sampling, which picks a representative set instead of a random slice. The same traces can also seed a rubric evaluator. If you don't have production traces yet, Foundry can also generate a synthetic dataset from your agent's instructions, a prompt, or a reference document such as a policy or spec, which covers prelaunch scenarios and edge cases. The two complement each other: traces show real user behavior, synthetic data covers what users haven't asked yet. The success criteria we defined earlier become useful here. Some evaluations will be technical, such as task completion, tool selection, groundedness, latency, or token consumption. Others can be much closer to the business outcome. Foundry gives us a few ways to turn those criteria into evaluations. The built-in evaluators cover common measures such as relevance, groundedness, task completion, tool call accuracy, and safety risks, so you get a first signal without writing anything. For criteria specific to your application, rubric evaluators use an LLM as the judge to score responses against weighted criteria you define, and you can auto-generate the first version of the rubric from your agent's instructions. Custom evaluators cover the rest: code-based (a Python function for deterministic checks such as format or length), prompt-based (your own judge prompt), and endpoint-based (an HTTP endpoint you host, useful when scoring logic lives in your own infrastructure). Rubric and custom evaluators are currently in preview. As the application evolves, the dataset should evolve with it. Production failures that matter are particularly good candidates for future regression tests. Evaluation also has a practical limitation: it costs money. Running every evaluator against every interaction in production can quickly become expensive. This is why an evaluation strategy matters just as much as having evaluations. Match the evaluation scope to the stage of development. During development, run smaller benchmarks frequently. Before important releases, run larger test sets through your CI/CD pipelines. In production, evaluate representative samples. The aim is enough data to understand behavior and identify regressions, not every evaluator on every request. Foundry brings evaluation into the same development lifecycle as the agent. Combined with agent insights, that gives us a way to move from diagnosing behavior to testing improvements. The agent optimizer uses that evaluation evidence to generate and rank candidate instructions, skills, tool descriptions, and model selections for prompt and hosted agents, which takes over much of the manual tuning that context management otherwise needs. We still need to assess quality, latency, and token consumption together: a version that scores better does not automatically cost less to run. There is enough here for an entire post, so we will explore evaluation and agent optimization separately in this series. Be deliberate about identity and isolation Personalization is important, but it comes with an equally important requirement: my experience with an agent can be different from yours, but my data should never accidentally become part of yours. That sounds obvious. Implementing it correctly requires thinking about several different concepts. Conversation state, session isolation, memory, user identity, agent identity, and authorization are related, but they aren't interchangeable. Memory, for example, can make an agent much more useful. The agent might remember preferences or information that allows it to provide a better experience over time. Foundry's managed memory, currently in preview, provides persistence without requiring developers to provision an external memory database. But managing the infrastructure doesn't answer the governance questions: whose information is it, where is it stored, how is it scoped, when should it be retrieved, and how long should it be retained? Those decisions still belong in the application design. Sessions introduce another boundary. In hosted agent architectures in Foundry, session isolation provides an execution boundary for a workload. Prompt-based experiences have their own mechanisms for maintaining conversation state. In either case, developers need to be deliberate about where state lives and what is shared (and can't be shared) between interactions. Identity becomes especially important when agents start using tools. Suppose we ask an agent to retrieve a document from a system we use at work. The easiest implementation might be to give the agent a highly privileged application identity capable of reading everything. It is probably not the implementation we want. Microsoft Entra ID gives us the capabilities to do this differently. One pattern you will encounter is On-Behalf-Of, usually shortened to OBO. At a very high level, OBO allows an application to call another service using the user's delegated identity. If you are using an agent to access information on your behalf, the downstream system can still make an authorization decision based on who you are and what you are allowed to access. Foundry supports this through user delegation, and the configuration guide shows how to set it up. You don't need to understand every OAuth detail to appreciate why that matters. If you cannot access a document, adding an LLM between you and the document shouldn't suddenly give you access to it. Foundry provides security building blocks, but they still need to be configured for the application. Microsoft Entra Agent ID gives agents an enterprise identity, and the Foundry runtime honors Agent 365 governance actions. Private connectivity, tool authentication and runtime guardrails help control access and behavior, with support varying by component and configuration. Security, identity, memory, and isolation get complicated quickly, so we will examine those boundaries and tradeoffs in a follow-on post. Until then, the identity and access overview is a good map of where each decision sits, and agent identity concepts explains how Foundry uses Entra Agent ID. Keep application logic in your application Earlier we mentioned math as an example of something that doesn't necessarily belong inside an LLM invocation. The same principle extends much further. Validation, transformations, deterministic routing, permission checks, sorting, filtering, and established business rules are operations that traditional software is very good at. Use it. An agent is part of an application. It doesn't replace the application. If a user asks an ambiguous question that requires interpreting intent and deciding which system can answer it, that's a good place for reasoning. Once the system knows exactly which API to invoke and has received structured data back, many subsequent operations are better handled by code. This distinction is useful for reliability, but it can also have a significant impact on latency and cost. A few unnecessary LLM calls probably won't matter during a demo. At production volume, they matter a lot. There is one nuance here. Telling an LLM not to calculate something directly doesn't mean an LLM can't help solve the problem. LLMs are also very good at generating code. For some workloads, generating code and executing it in an appropriate controlled environment can be a much better architecture than asking the model to reason through every intermediate calculation in tokens. This is especially interesting for data analysis and tasks where the requested computation itself is dynamic. Foundry gives us an off-the-shelf Code Interpreter tool that acts as a secure sandbox for this approach. The decision is not whether the model is capable of doing something. In many cases, it is. The more useful engineering question is whether a model invocation is the right component of the application to do it. Plan for scale: within an agent, and across your organization A successful proof of concept eventually faces two different scale problems: one agent serving thousands of users, and an organization creating thousands of agents. The first is about production capacity and reliability. The second is about reuse, ownership, and governance. An agent that works very well for ten people needs to continue working when thousands of users invoke it concurrently. At that point you start thinking about throughput, latency, quotas, model capacity, downstream dependencies, rate limits, connection management, resiliency, and cost. Agent runtimes can help with part of that problem, but your agent is only one component in the architecture. Its models, tools, data sources, APIs, and other dependencies all need to support the level of traffic you expect. This is also where capabilities such as AI Gateway become interesting. As AI usage grows, organizations need mechanisms for managing access to models, controlling and observing traffic, enforcing policies, and understanding consumption across applications. Cost governance needs explicit ownership and limits: set budgets for each application or team, track consumption against them, and define who responds when spending approaches those limits. Azure Cost Management remains the system of record for spend. AI Gateway provides project-scoped tokens-per-minute limits and total token quotas; those are token controls, not native dollar-budget enforcement in Agent Service. Decide in advance how budget alerts and request controls will work together, including when to alert, throttle, or stop requests, rather than waiting for the bill to reveal a problem. And there is another kind of scale that we find equally important: what happens when the problem isn't one agent serving thousands of users, but an organization creating thousands of agents? The requirements change. Different teams may independently create similar tools. Every agent might implement authentication differently. Tool integrations get duplicated. Agents need owners and lifecycle management. Developers need ways to discover capabilities that already exist. Organizations need governance without preventing teams from moving quickly. Tools should become reusable building blocks rather than implementations hidden inside individual agents. This is one of the reasons we find approaches such as reusable Foundry tooling and MCP particularly interesting. A well-designed capability can be exposed once and consumed by multiple agents instead of every team independently rebuilding the same integration. The agents themselves can also become reusable. A specialized agent built by one team might provide useful capabilities to another agent or application. Multi-agent communication makes it possible to compose those capabilities instead of creating one enormous agent that knows how to do everything. Distribution is another part of this problem. Users shouldn't necessarily need to visit a separate application for every agent an organization creates. Depending on the use case, agents may need to appear in applications, websites, Microsoft Teams, Microsoft 365 Copilot, or other channels users already work in. Foundry Agent Service provides a publishing path into Microsoft Teams and Microsoft 365 Copilot, with Agent 365 integration for tenant-wide governance. That connects the engineering decision about where an agent runs with the adoption question of where people will actually use it. Once organizations reach this point, the problem is no longer how to build an agent. The problem is how to provide an agentic platform that allows many teams to build agents without every team independently solving identity, tooling, observability, evaluation, governance, integration, and deployment. We are going to spend much more time on that problem later in the series. And this is just the start The goal isn’t to add more AI at every step. It’s to build applications that solve real problems and keep improving them as we learn. These principles are a starting point; their value comes from putting them into practice. In subsequent posts, we’ll get hands-on with evaluation and optimization, security and identity, context management, and scaling, exploring what Microsoft Foundry provides, the architectural tradeoffs, and how to turn what we learn into better applications, and more business value. Authors Maira Ladeira Tanke Maira is a Principal Software Engineer for Agentic AI at Microsoft Core AI, focused on helping customers successfully adopt Microsoft Foundry Agents. Mark Roy Mark Roy is a Principal Software Engineer at Microsoft working on Microsoft Foundry and AI agents, passionate about helping developers get agents out of the demo stage and into production. Vivek Bhadauria Vivek Bhadauria is a Partner Software Engineer at Microsoft focused on building AI agents, platforms, and optimization loops.Build a dual-mode voice agent in Microsoft Foundry with native subagents
Voice agents are changing how companies interact with customers. From service questions to technical support, the expectation is the same: a natural conversation that leads to useful, reliable answers. But responding quickly and resolving a complex request are not always the same task. Some questions can be answered directly. Others require information from backend systems, specialized analysis, or several steps to determine what should happen next. The challenge is to support that deeper work without putting the conversation on hold, or giving an answer before the necessary information is available. In this blog, I will walk through a dual-mode voice-agent pattern in Microsoft Foundry. A frontline agent manages the real-time conversation, while a backend agent handles specialized work. We will explore how these responsibilities work together, how the frontline can continue interacting while backend processing runs, and how to configure and verify the experience. The pattern is broadly applicable, with particular relevance to telecom customer service and support. You will learn how to: Separate real-time conversation from specialized backend work. Configure a native voice subagent Understand how Voice Agents build on Voice Live Test delayed responses, interruptions, missing information, and backend failures. How voice agents work Before we look at our full workflow. Lets have a look at the voice experience in Microsoft Foundry. A voice application needs to carry audio, manage the conversation, and connect that conversation to useful information or actions. Those responsibilities work together, but they are not the same layer. A browser or contact-center connector handles audio capture and playback The voice runtime manages streaming and turn-taking Instructions, knowledge, and authorized tools determine what the agent can say and do. In Microsoft Foundry, Voice Live provides the real-time speech foundation. Voice Agents build on it with a named, versioned agent definition and capabilities such as tools, tracing, and evaluation. The diagram below shows the underlying voice pipeline. Audio arrives from the client Before anything else, Voice Live cleans up that audio. It uses conversational enhancements that can reduce background noise and echo and help detect when a speaker has finished a turn. The exact combination depends on the selected model, pipeline, and audio settings; not every session uses all the processing stages shown The conversation can then either follow a native speech-to-speech path. A model in the GPT Realtime family processes spoken input directly and produces a spoken response. Supported configurations can also use Azure Speech voices for output, so choosing a realtime model does not mean we must use its native output voice. Alternatively, a cascaded pipeline converts speech to text, sends that text to a language model or agent, and synthesizes the response as speech. This separates the recognition, conversation model, and voice choices. Voice Live can manage that pipeline without us writing the streaming and turn-taking loop ourselves. In either path, useful answers may also require knowledge sources or external tools. These connections give the agent access to information and actions beyond the conversation itself. Configure the required permissions and safety controls for the selected model and application; a tool or content-safety box in the diagram is not a substitute for those decisions. On the return path, speech streams back to the client for playback. Where supported, Azure Speech provides standard and HD voices, with custom voice subject to access requirements. These choices describe how the spoken conversation runs. The dual-mode pattern adds a different separation: the frontline owns that conversation, while a backend agent handles a focused task. It does not require two speech-to-speech models, and the backend does not speak directly to the caller. How the frontline and backend work together To keep the conversation responsive while specialized work runs, we separate the caller-facing interaction from the backend task. Each agent has a clear responsibility: The frontline agent is responsible for the interaction. It listens, takes turns, answers supported questions, and explains what happens next. The backend agent has a different job: work through a focused task, such as reviewing incident evidence or recommending a troubleshooting sequence. The important part is that delegation does not have to take the caller out of the conversation. The frontline can remain active while the backend works. In the native subagent pattern, this behavior is configured through the agent definitions, instructions, and response policy rather than a separate routing model The following diagram shows how those responsibilities fit together: The voice client captures and streams the caller’s audio: The client connects the caller to the frontline voice agent, sends microphone audio, and plays returned speech. It can also display transcripts and runtime events. The client can send audio and receive events concurrently. The frontline voice agent decides whether to answer directly or delegate: The frontline owns the live conversation. It handles straightforward requests within its configured knowledge and permissions without invoking the backend. When a request needs specialized work, the frontline uses its instructions and the configured subagent’s description to determine when to delegate. Direct answers return through the voice client without a backend invocation. Foundry invokes the configured backend subagent. The voice agent’s subagent_config references a backend in the same Foundry project by name. Foundry invokes that backend for the delegated task. The frontline retains ownership of the voice conversation; the caller is not transferred to the backend. The backend performs the task and uses its tools. A prompt or hosted agent follows its own instructions and can retrieve data, call APIs, or execute supported functions. Access to service information, incident records, or diagnostics must be explicitly configured and authorized. Meanwhile, the frontline keeps the voice interaction active. Its response_policy controls an immediate acknowledgment, gap-filling speech, and supported progress updates. The basic sample enables only immediate_ack=True; additional behavior requires configuration. The frontline can receive further caller input, but it must not invent backend findings while waiting. Further caller input does not automatically update an in-flight backend task. Results and supported progress return through Foundry. The backend returns its output to the frontline through managed delegation. Where supported and enabled, progress can also inform the conversation before completion. The diagram’s return arrow does not mean every backend automatically streams intermediate findings. The frontline presents the result to the caller. Following its instructions, it explains the findings, communicates uncertainty, and identifies appropriate next actions. Speech streams back through the client for playback. The backend does not speak directly to the caller, and its output is not necessarily identical to the spoken response. Note: The GPT Live samples demonstrate an alternative implementation of the conversation/backend split. They use VoiceAgentLiveDefinition and live.connect(), with direct Responses or managed prompt-agent delegation. The design goal is responsiveness without sacrificing depth. A fast acknowledgment improves perceived waiting time; it does not shorten the underlying investigation or guarantee an accurate answer. Measure conversational responsiveness and backend task quality separately. Decide when the frontline should ask for help With the flow in place, the next question is when the frontline should ask for help. Define clear boundaries in the frontline agent’s instructions and the subagent’s capability description. Rule 1: Answer directly when the frontline has sufficient grounded information and permission to respond. Rule 2: Delegate when the backend owns the required capability. This might include combining information from several systems, assessing service impact, investigating symptoms, or developing a troubleshooting plan. Rule 3: Escalate when the request exceeds the automated workflow’s boundaries. Define what should happen when an action requires human approval, evidence remains insufficient, or a tool failure prevents progress. Implement and test the human-handoff or failure path; instructions alone do not provide it. From the caller’s perspective, this should feel like one conversation, not a handoff between agents. The frontline listens, clarifies the request, and brings in specialized help when needed. The caller does not need to understand which backend agent or tool is doing the work. When backend processing begins, an enabled acknowledgment can make that transition clear: “Let me check that for you.” If gap-filling speech or supported progress updates are configured, the frontline can keep the caller informed while the task runs. Those updates should reflect what is actually known, without implying that a check has completed or a result has been confirmed. Once the backend returns, the frontline explains the outcome in language suited to the conversation. That might be a confirmed finding, a recommended next step, or a clear statement that more information is needed. The goal is not to fill every pause with speech. It is to keep the caller oriented while specialized work happens in the background, and to make the eventual answer useful, understandable, and grounded in evidence. Build the native voice-subagent workflow Now that we have defined the conversation flow, let’s configure the agents behind it. We will use native voice subagents throughout this walkthrough: a VoiceAgentDefinition for the frontline, a prompt agent for the backend, and subagent_config to connect them. Voice subagents are currently in preview. Microsoft does not recommend features in preview for production workloads. Step 1: Give the backend one focused responsibility Start with the work the frontline should delegate. A useful backend has a clear responsibility. The subagent can be a Foundry prompt agent or a hosted agent in the same project. For our example here we will create a prompt agent that can analyze information supplied to it. It can be connected to knowledge bases and different tools to help. You can also create a subagent in the Microsoft Foundry portal or deploy a code-based hosted agent. Keep the instructions specific to the backend’s actual capabilities. If it can only analyze supplied information, say so. If you later add an incident lookup, update the instructions and capability description to reflect that change. Instructions guide model behavior; they do not enforce permissions or guarantee an output schema. Restrict access in the tools and services themselves, and validate any structured fields your application relies on. Separate read-only investigation from actions that change a customer account or service. Step 2: Connect the backend to the frontline Next, we can create the caller-facing voice-based agent. We can create this using the SDK. In its instructions we should explain when to delegate to the backend agent and we should add the backend agent to its subagent_config. In the subagent_config, you identify the backend agent by name and describe when the voice-based agent should delegate a request to it in the agent_capabilities, which describes the specialist’s responsibility. You can use the following code to spin up such an agent (Also see: Use a subagent in a Voice-based agent - Microsoft Foundry | Microsoft Learn) import os from azure.ai.projects import AIProjectClient from azure.ai.projects.models import ( RealtimeAudioFormatsAudioPcm, VoiceAgentAudioConfig, VoiceAgentAudioInputConfig, VoiceAgentAudioOutputConfig, VoiceAgentDefinition, VoiceAgentInputTranscription, VoiceAgentServerVadTurnDetection, VoiceAgentSubagent, VoiceAgentSubagentConfig, VoiceAgentSubagentResponsePolicy, ) from azure.identity import DefaultAzureCredential endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"] model = os.environ.get("FOUNDRY_VOICE_MODEL", "gpt-realtime-2.1") definition = VoiceAgentDefinition( model_type="managed", model=model, instructions=( "You are a real-time customer support assistant for service incidents. " "Handle straightforward factual and support-process questions directly. " "For complex troubleshooting, SLA-risk triage, root-cause hypotheses, " "or multi-step remediation planning, call forward_to_subagent for " "the configured telco support agent. " "After the subagent returns, provide a concise, customer-safe summary " "and next actions. " "Do not present uncertain reasoning as confirmed fact." ), store=True, output_modalities=["audio"], audio=VoiceAgentAudioConfig( input=VoiceAgentAudioInputConfig( format=RealtimeAudioFormatsAudioPcm(rate=24000), turn_detection=VoiceAgentServerVadTurnDetection( threshold=0.5, prefix_padding_ms=300, silence_duration_ms=1000, ), transcription=VoiceAgentInputTranscription( model="azure-speech", ), ), output=VoiceAgentAudioOutputConfig( format=RealtimeAudioFormatsAudioPcm(rate=24000), voice="en-US-JennyNeural", voice_type="azure-standard", ), ), subagent_config=VoiceAgentSubagentConfig( subagents=[ VoiceAgentSubagent( agent_name="[BACKEND AGENT NAME]", agent_capabilities=( "Performs deeper SLA-risk triage and troubleshooting " "reasoning, including structured cause hypotheses " "and remediation planning." ), response_policy=VoiceAgentSubagentResponsePolicy( immediate_ack=True, ), ) ], ), ) with ( DefaultAzureCredential() as credential, AIProjectClient( endpoint=endpoint, credential=credential, allow_preview=True, ) as project_client, ): voice_agent = project_client.agents.create_version( agent_name="frontline-voice-agent", definition=definition, description="Voice frontline with subagent delegation", ) print( f"Created voice agent '{voice_agent.name}', " f"version {voice_agent.version}" ) While you currently can't link these subagents through the portal You can see the subagents linked and its instructions in the portal by clicking on the YAML button. Step 3: Configure what the caller hears while waiting With the basic connection in place, we can decide how the frontline should support the caller during backend work. Use the optional response_policy object to control what the voice agent says while it waits for a subagent: immediate_ack: Specifies whether the voice agent immediately acknowledges that it forwarded the request. ack_instructions Provides instructions for generating the immediate acknowledgment. enable_delta_progress Emits progress incrementally instead of waiting for the subagent invocation to finish. The default is false. gap_filling_interval Sets the period without subagent content or user input before a gap-filling response. Use a value from 5 through 120 seconds. gap_filling_instructions Provides instructions for speech generated while the voice agent waits for progress. progress_instructions Provides instructions for summarizing streamed subagent progress as speech. progress_update_interval Sets the minimum time between spoken progress updates. Use a value from 5 through 300 seconds. Step 4: Start a conversation with the voice agent With the frontline and its subagent configured, we can now test the spoken experience. The quickest starting point is the Foundry playground: open the project containing both agents, go to Build → Agents, select frontline-voice-agent, and open it in the playground. Start a voice session and allow microphone access. This tests the saved agent configuration without building a local audio client. For a custom application, we can use the shared native microphone client linked here (Native voice client: dependencies, launch commands, and event handling) or the complete client in the Microsoft Learn walkthrough. Configure it with the same Foundry project endpoint and the name created in Step 2: frontline-voice-agent. The client connects through beta.voice_agents.realtime.connect(), sends microphone audio, and receives speech and events concurrently. The example uses 24 kHz PCM audio and plays the returned audio chunks as they arrive. Follow the setup and readiness sequence of the client we choose rather than combining snippets from different clients. Voice activity detection identifies speech boundaries; the frontline instructions and subagent configuration still determine when to delegate. Start with a general support question, then ask for analysis that should use the backend. Check the execution evidence as well as the spoken answer: an acknowledgment alone does not prove that the subagent ran. In our current configuration, only immediate_ack is enabled, so additional gap filling or progress narration should not be assumed. Use headphones to prevent speaker audio from feeding back into the microphone. Then introduce a delayed response, a caller correction, missing evidence, and a backend failure. Check how the frontline responds and what the execution trace shows. Test interruptions separately: stopping audio playback does not prove that the backend task was canceled. From a microphone to a contact-center call The microphone client helps us validate the agent. Connecting it to a contact center adds another layer: a channel integration that bridges call audio and call-control events to the voice runtime. The division between frontline and backend remains useful, but a working local conversation does not automatically provide telephony routing or human transfer. The Azure-Samples Call Center Voice Agent Accelerator documents browser testing and multiple telephony providers, including Genesys Cloud, Azure Communication Services, and Twilio. It provides a reference for the channel-integration layer. There are also call-center-specific accelerators e.g. This Genesys Voice Live Connector sample. This integration option uses Genesys Cloud AudioConnector and a connector service between Genesys and Azure Voice Live. The connector handles the channel protocol and audio conversion; the agent handles the conversation and its configured tools. Conclusion A dual-mode voice agent separates responsibilities that operate on different timelines. The frontline keeps the conversation clear and responsive; the backend performs specialized work. Native subagent delegation connects them without handing the caller over to another speaking agent. The value is not constant speech or an instant answer to every request. It is an interaction that stays understandable while the necessary work happens, then delivers a response grounded in what that work actually established. Next Steps Ready to go deeper? Use a subagent in a Voice-based agent (preview) – Follow the end-to-end walkthrough to create a backend subagent, configure delegation, and test the spoken experience. Configure a voice agent – Explore model selection, audio settings, turn detection, voices, and agent behavior. Native voice-subagent samples – Explore working examples that connect a frontline voice agent to prompt or hosted backend agents. Voice Live overview – Learn about the underlying speech capabilities, supported models, and regional availability. Genesys Cloud Voice Live Connector – Explore a sample audio bridge between Genesys Cloud and Azure Voice Live, with a simulator for local testing. Call Center Voice Agent Accelerator – Explore deployment guidance and reference integrations for browser-based conversations and supported telephony providers.From Python Agents to Copilot Harness : Putting Microsoft Foundry to the Test
You have built the agents. Written the Python. Connected the data. Worked out how the pieces talk to each other. Then the conversation shifts to harnesses. The prospect of convergence around the GitHub Copilot harness across GitHub, Microsoft Foundry and Copilot Studio (CPS) raises a very practical question: What happens to the pro-code agents I already built? Do I rewrite them, wrap them, or keep them? Rather than debate it in the abstract, I wanted to give it a shot. I already had a Microsoft Agent Framework (MAF) demo with Python agents, Fabric sales data and product knowledge. Could I keep the useful business workflows, remove the custom agent wrappers, and let a managed harness do more of the work? That experiment became the demo behind this blog. Not another empty chat box, but an app to ask business questions, inspect actual tool activity, follow the evidence and create a brief you can revise and download. This is a practical Foundry experiment and the question I wanted to answer was: how much of this particular agent application can move into the harness, and what still belongs in my code? TL;DR: I moved a Python/MAF research demo to a Foundry-managed Copilot harness to test whether it could handle the same business questions. Native research and source connections reduced the agent-side orchestration I had to write, while the app still owns authentication, streaming and secure access. Downloadable briefs are an extra experiment, not part of the original migration. Focused tests were promising, but do not establish complete feature parity or production readiness. The Starting Point: Pro-Code Agents The starting point is olivierb123/microsoftiq-lead-agent-demo. It used four hosted Agent Framework wrappers and application-controlled fan-out. Python agent code and custom source adapters handled the integrations; the application determined which agents to call and assembled their results. That approach makes the orchestration explicit. It also puts the responsibility for hosting the wrappers, maintaining source adapters and coordinating the workflow in the application. For this experiment with harness, I kept the business scenario but changed the execution model. Instead of recreating those wrappers inside another service, I configured one Foundry-managed Microsoft Copilot harness agent with instructions, native capabilities and connected sources. Responsibility Pro-Code Demo Harness Experiment Workflow orchestration Application-controlled agent calls Managed agent selects tools and can delegate Research Hosted wrappers and custom adapters Native web search and page fetching Business data Application source integration Native Fabric connection Product knowledge Application retrieval integration Foundry IQ knowledge-base retrieval through native MCP Working output Application-managed result assembly Conversation workspace and published artifacts Application code UI plus agent orchestration UI, authentication, streaming and session/download authorization This repository contains the migrated implementation. The original pro-code demo remains separate and was not overwritten. A Practical Question, Not a Tool Demonstration The scenario is a fictional sales team working with FieldForge product documentation. A useful assistant should help answer questions such as: How many opportunities are open in the expanded synthetic demo, what is their total and probability-weighted value, and how do our internal forecasting definitions explain those numbers? That request needs two different kinds of evidence: structured opportunity data and the actual internal document defining the forecast. It should not require the user to name a connector or prescribe a tool sequence. A more advanced request asks for a South territory brief combining sales performance, priority opportunities, product capabilities and official construction heat-exposure guidance. The user can then request a revision or a downloadable Markdown artifact in the same conversation. These examples exercise the harness as a working environment, not just a text-generation endpoint. However, they are requests to explore, not promises that every tool or subagent will run on every turn. Why a Local Web App? I deliberately kept the test experience in a local web app. I wanted to make it easy to ask a question, try a follow-up and inspect what happened, without turning the experiment into a web-hosting project first. The React workbench and Node.js backend run locally; the agent and connected sources run in the cloud. After signing in, I can choose a sample question, watch the streamed response, inspect Activity and Evidence, and request a revision or download in the same conversation. I do not need to reconstruct a raw SDK response to see whether a source was actually called. That separation also makes the migration easier to understand. The web app is the test surface, not the agent runtime. There is no local Python agent server hiding behind it: the backend sends the request to Foundry and handles the application's authentication, streaming and session-owned downloads. For this first experiment, the backend is intentionally loopback-only. Hosting the application publicly is a separate engineering task, not a prerequisite for testing the harness against a useful scenario. The Architecture The configured agent uses microsoft_copilot_preview and gpt-5-mini. The agent definition lives in harness/azure.yaml, rather than a Python agent server. The application, backend and data scripts use JavaScript/Node.js. There is no Python application runtime, pip setup or virtual environment required for this version. That is not a zero-code migration: the application still owns sign-in, token validation, streaming, conversation ownership and secure downloads. The backend does not execute the native source calls a second time. Foundry executes them; the application observes their returned results and presents them in Activity and Evidence. What It Looks Like in the Foundry Portal The local workbench is only one view of the experiment. Foundry also supports hosted pro-code agents: I can package custom agent code, including a Python/MAF implementation, and its dependencies in a container and let Foundry manage the hosting. I still own the agent implementation and its orchestration; using Python does not mean I must run the server myself. This experiment instead uses a managed-harness prompt agent. I configure its instructions, capabilities and source connections, while Foundry supplies the harness execution environment and native tool orchestration. Both approaches can run in Foundry. The distinction is not simply local versus cloud, or Python versus no Python: it is how much agent runtime and orchestration I implement versus delegate to the managed harness. The application's UI, authentication and session/download authorization remain my responsibility in this demo. My microsoftiq-copilot agent and its configured tools/connections make the change tangible. The configuration ties together the native harness capabilities, Fabric access and product-knowledge retrieval shown in the architecture diagram. No Web IQ Wrapper in This Version The Web IQ integration from the earlier demo is not carried forward. External research uses the harness's native web search to discover sources and its Web capability to read relevant pages. This supports the research portion of the scenario without maintaining a separate Web IQ agent wrapper. It is not a claim that native web research reproduces every Web IQ behavior or data contract. Fabric and product knowledge are optional grounding sources, not stages that every question must pass through. A public-research question can stay a public-research question. Private business data must not be sent to public web tools. More Direct Integration with Fabric and Foundry IQ For me, the integration improvement is the ownership boundary: the harness uses configured native connections instead of calling application adapters that obtain credentials and query sources themselves. Fabric supplies structured business facts. The native Fabric connection uses the signed-in user's context to access a dedicated data agent over imported semantic models. There is no app-only Fabric credential fallback. The models contain demonstration data, not live CRM records. Foundry IQ supplies document evidence. The product-knowledge MCP connection reaches a knowledge base backed by Azure AI Search. Query planning and semantic retrieval return extractive evidence, including stored document metadata, for the harness to use in its answer. This connection uses the Foundry project's managed identity; it does not establish per-user document ACL enforcement. The distinction matters. A supportingDocId in an opportunity row is a relationship reference, not proof that the corresponding document has been retrieved. Fabric cannot substitute for document content, titles or update dates from the knowledge base. Calling this integration "better" means more direct native integration with clearer source responsibilities. I have not established a quality, latency or cost improvement over the MAF implementation, nor certified genuinely different-user authorization. What the Experiment Actually Showed Focused tests showed that the harness could retrieve sales figures, find internal documents, research the web and continue a conversation. That supported the core research workflows we wanted to carry over from the MAF demo. The most revealing test asked for sales figures together with the internal document explaining how the forecast should be interpreted. Some early runs got the numbers right but did not retrieve the requested document. The answer sounded convincing, but part of its explanation lacked verified support. After clarifying the agent's instructions and which source should answer each part of the question, one test run successfully retrieved evidence from both sources. This used synthetic demo data, not live customer information. The lesson: connecting the tools is only the start. You still need to test whether the agent uses the right sources and whether those sources actually support its answer. Beyond the Migration: Trying Workspace Files and Briefs Once the core question-answering workflows were working in focused tests, I wanted to try something we had not implemented in the MAF demo: turning an answer into a working document. Why save a brief? Not every answer needs one. But a territory review is a useful example: instead of leaving the analysis in a chat response, I can request a Markdown document to keep, review or hand off, then ask for a revision in the same conversation. The file is an optional deliverable, not a prerequisite for retrieving sales data or product evidence. This is where the harness's conversation workspace and native file capabilities become interesting. I can explore creating, reading back and revising an artifact in the managed environment without rebuilding those agent-side tools in a Python wrapper. MAF could support the same kind of workflow with appropriate tools and code; this is an additional capability in this demo, not something exclusive to a harness. Publication adds a separate step when the user actually wants a download. A workspace file is not automatically downloadable: the workbench registers it only after a successful native upload returns a real file ID, and only the owning unexpired session can download it. The application still implements those access checks. The migration question is therefore "can I keep answering the useful business questions?" The extension question is "what else can I build on the harness's native capabilities?" Briefs help explore the second question. Actual tool results, not assistant narration, establish whether creation, revision or publication occurred. There are still operational boundaries. The app is loopback-only, the harness is a public preview, inference and retrieval incur costs, and workspace state is not durable application storage. Refreshing the browser or restarting the backend loses local session mappings. Production hosting, retention and broader authorization need separate work. Shell and external business-system writes remain disabled. The Takeaway So, what happened to my Python agents? For this scenario, their hosted wrappers were no longer needed. The business questions stayed; their execution moved into a managed harness, and the remaining application code focused on the experience and its security boundaries. For this practical research scenario, that shifts the work from maintaining Python wrappers and fixed fan-out toward configuring the agent, defining source responsibilities, building the user experience and verifying evidence. Fabric and Foundry IQ fit into that environment directly, while native web research replaces the earlier Web IQ path for the tasks explored here. When I Would Still Use MAF Deterministic Routing and Local Hybrid Execution: MAF allows you to deploy hybrid topologies where small, local models (like MAI-Decision / Jev) handle rapid, deterministic triage and tool-gating, reserving expensive cloud-hosted frontier models for complex reasoning. A managed harness forces all routing into its proprietary cloud runtime, stripping away your ability to optimize latency and compute token economics. Absolute Control over the Orchestration Loop: The Copilot Harness uses an abstracted reasoning engine to decide when to activate a skill or invoke a tool. Native Toolchain Continuity and Hosting: MAF agents operate as standard C# or Python applications. Granular State and Semantic Memory: Managed harnesses provide a generalized shared memory layer. MAF allows you to build custom memory providers (like custom Azure SQL or Cosmos DB schemas) Unrestricted Interoperability Mechanics: While the Copilot Harness can consume Agent-to-Agent (A2A) protocols and Model Context Protocol (MCP) servers, building the actual server logic in MAF gives you the power to manipulate the raw protocol mechanics. If you need to manage custom message headers, inject highly specific tokens for platforms like SAP or ServiceNow, or alter payload structures mid-flight, MAF provides the unabstracted access required to do so. For this demo, the harness was a useful way to move beyond fixed fan-out and put research, connected evidence and artifacts into one managed environment. For another application, keeping its MAF agents may be the better decision. My takeaway is simple: start with a real task, move the responsibilities that fit the harness, and test the boundary instead of assuming the migration works. What's Next This experiment answered one question—now I'm curious about others: How does the harness perform under sustained load? What happens when Fabric semantic models grow to production scale? Can the same approach work for agents that need to write back to business systems? If you're exploring similar migrations, I'd be interested to hear what worked and what didn't. Try the Project The code is available at olivierb123/microsoftiq-copilot-harness-demo. Choose your path: Run the demo locally → Start with the README and setup checklist Explore the agent configuration → See harness/azure.yaml for the Foundry agent definition Understand the data model → Review the synthetic data documentation Compare with the original MAF approach → Check out the original pro-code demo Questions or feedback? Open an issue on the repo or share your own migration experience in the comments below.
Turn enterprise knowledge into a personalized voice and avatar agent in minutes with Voice Agent
A customer asks how a product works, then follows up: “Can you give me an example?” They need a useful explanation based on your product documentation, without searching through pages of information. Now imagine that conversation delivered in a voice and with a face that represent your brand. Voice Agent in Foundry Agent Service brings three things together in one real-time conversation: your knowledge through Foundry IQ, your voice through Personal Voice, and your face through a custom photo avatar. Once these assets are prepared, combining them takes a few minutes in the Foundry portal: create a Voice Agent, connect the knowledge, select the voice and avatar, and start talking. You don't need to build speech, avatar or app code to try it: the playground handles all of that. Why Voice Agent Minutes, not a project. With your knowledge base, voice and avatar ready, setup is four steps in the portal, with no code required to try the experience. Answers grounded in your content. Foundry IQ connects the agent to your documents, and you can instruct it to say when the materials don't contain an answer instead of guessing. A natural conversation. Users speak in real time and ask follow-up questions that build on conversation context. With interruptions enabled in a supported voice configuration, users can also interrupt the agent mid-answer. Build the identity once, update the knowledge anytime. Refreshing your documents does not require recreating the voice or avatar, and the same prepared assets can be reused across agents. Give your knowledge a voice and a recognizable face Personalization is most useful when it helps someone complete a task. A common pattern brings together your knowledge, your authorized identity, and a Voice Agent that makes both conversational. Here are some places to start: Scenario Knowledge to connect Voice and avatar Product concierge on your website or in a store kiosk Product manuals and approved FAQs Your brand spokesperson Employee onboarding and HR help Policies, handbooks and benefits guides A familiar HR partner or team lead Internal IT helpdesk Runbooks and troubleshooting guides Your support team's representative Training and e-learning tutor Course materials The course instructor Museum, tourism or campus guide Exhibit and visitor information A curator or guide Expert “office hours” Talks, FAQs and published content The expert, with their consent In each case, users can move from an overview to a concrete example at their own pace, in the same conversation, instead of navigating long documents or restarting their search. These are starting points to adapt to your audience and approved use, not customer deployments. Watch the conversation, not just a talking avatar The demo below uses a Theodore Roosevelt knowledge base to show three connected interactions with Voice Agent in the Foundry playground. The historical topic is a compact example; replace it with your own knowledge to apply the same pattern to any of the scenarios above. Ask the Voice Agent What it shows What were Theodore Roosevelt's main contributions to conservation? A grounded answer from the connected knowledge base Could you give one specific example and explain why it mattered? A follow-up that builds on the conversation context What was Microsoft's closing stock price yesterday? Knowing its limits: the agent says the available materials do not contain the answer instead of guessing The voice and avatar in this demo are AI-generated. Theodore Roosevelt is the subject of the knowledge base, not the identity portrayed by the assistant. The point is not simply to animate a response. It is to let a user get an explanation, explore it through a follow-up and understand when the connected knowledge cannot help. The demo uses a platform-provided voice (Andrew Dragon HD) and a preset photo avatar (Marcel); with Personal Voice and a custom photo avatar, the conversation works the same way, with your own identity. From prepared assets to a conversation in four steps Start in a supported Foundry project with these resources prepared, or reuse ones you already have: Knowledge: a Foundry IQ knowledge base created from your authorized documents. Try a sample question to preview the results. Voice: a Personal Voice created from a short, clean voice sample and the speaker's recorded consent statement. Appearance: a custom photo avatar created from an authorized photo and a consent video of the person pictured. You can also start right away with platform-provided voices and avatars and personalize later. The Voice Agent setup guide walks through preparing each resource, including the consent and eligibility requirements, for both the portal and Python. 1. Create a Voice Agent In the Foundry portal, open your project and go to Agents → Build an agent. Enter a name, select Voice as the interaction mode, choose the appropriate use case, and select Create agent and open playground. Review the model and default instructions for your scenario. 2. Connect the knowledge that helps users complete their task In the agent configuration, expand Knowledge, select Add, and choose your prepared Foundry IQ knowledge base from Available knowledge bases. Give the agent a clear purpose, such as explaining product features from approved documentation, and keep spoken answers concise. For example: Use the connected knowledge base for factual answers. Give a brief, direct answer and use conversation context for follow-up questions. If the available materials do not contain the answer, say so rather than guessing. This gives the conversation a useful boundary: the knowledge supplies the answer, and the voice and avatar determine how it is presented. 3. Choose how the agent sounds and appears Under Voice, select your Personal Voice (Custom → Personal). Enable Avatar and choose your custom photo avatar (Custom → Photo avatar). Preview the selections to confirm they match the intended identity and are available for your project's resource and region. This is where the experience becomes recognizable: the same product knowledge can be delivered by your brand spokesperson or your trainer, and you can swap the knowledge or the identity independently. 4. Save and start talking Select Save, then Start session, and allow browser microphone access. Ask about your content, follow up naturally (for example, “Can you give me an example?”), and try a question outside your documents to check whether the agent acknowledges when the information is unavailable. That's it: your knowledge, voice and avatar are working together in a live conversation, right in the playground, with no local SDK setup. Check the answers against your source documents before sharing the agent with users. Personalize with permission Use voice, likeness and documents you are authorized to use, complete the required consent and access approvals, and disclose synthetic voice and avatar output. Personal Voice and custom photo-avatar capabilities have eligibility requirements; available models, regions and preview features vary. The setup guide links to the applicable requirements. Review audience access and data handling before using enterprise content, and validate the experience for your intended deployment. Build your first Voice Agent Start with one focused knowledge collection, one authorized identity and three useful questions. Voice Agent in Foundry Agent Service turns those building blocks into a conversation that people can hear, see and follow up on. Ready to build it? Follow the step-by-step setup guide to prepare your knowledge base, Personal Voice and photo avatar, then bring them together in a Voice Agent through the Foundry portal or Python. For more documentation, quickstarts and samples, explore the Microsoft Voice Agent repository. Your knowledge, your voice, your face: one conversation, a few clicks away.
Building a Self-Updating RAG System: Handling Document Uploads, Updates and Deletions
The Hidden Challenge of Production RAG Imagine your organization has deployed an AI assistant to answer questions about employee policies. An employee asks, “How many days of annual leave am I entitled to?” The assistant responds confidently: “15 days.” There is just one problem. The company updated its policy last week. Employees are now entitled to 20 days. The updated PDF was uploaded successfully. The storage service is healthy. The search index is responding. The language model is generating answers without errors. Yet the answer is wrong. This is a different kind of production failure. Nothing appears to be broken at the infrastructure level. The failure lies in the relationship between the source document and the information the AI retrieves. Retrieval-Augmented Generation (RAG) systems depend on external knowledge to ground their responses. But that knowledge is not static. Policies change, contracts get revised, product documentation evolves, and outdated documents are removed. When the underlying documents change, the associated chunks, embeddings, metadata, and search-index entries must be managed accordingly. Otherwise, the system can continue generating plausible answers from information that is no longer valid. Building a RAG pipeline is a retrieval problem. Keeping it correct as data changes is a data lifecycle engineering problem. In this article, we will design a self-updating RAG system that handles document creation, updates, and deletion while addressing asynchronous processing, version consistency, failed jobs, and stale retrieval results. The goal is not simply to index documents automatically. It is to make sure that the right version of the right document is available to the right user when a query is executed. Understand the Problem Before Designing the Architecture A basic RAG pipeline is relatively straightforward: Load a document. Extract its text. Split the text into smaller chunks. Generate an embedding for each chunk. Store the chunks and embeddings in a searchable index. Retrieve relevant chunks when a user asks a question. Pass the retrieved context to a language model to generate an answer. This works well when the document collection is relatively static. The difficulty begins when the source data changes after indexing. Consider three common scenarios. Scenario A: A document is uploaded A new policy is added to the knowledge base. The system must process it and make its content searchable. Until indexing finishes, the document may not be available to the chatbot. Scenario B: A document is updated The original policy allows 15 days of annual leave. A revised policy allows 20 days. If the new version is indexed without correctly deactivating the old version, both versions may appear in search results. The language model may then select the wrong passage or combine conflicting information. Scenario C: A document is deleted An administrator removes a document from the application. However, deleting the original PDF from storage does not automatically delete the corresponding records from Azure AI Search. The content may remain searchable until its indexed chunks are removed. Azure AI Search document deletion itself can also take time to be reflected in index statistics. Microsoft Learn: Delete documents in Azure AI Search These scenarios reveal an important design principle: The source document, its metadata, the background processing jobs, and the search index must be treated as parts of one logical lifecycle—even though they are separate systems. We need a design that can handle failures between these systems without exposing outdated or unauthorized content. Design the System Around Clear Responsibilities Rather than making the upload API responsible for every operation, we can separate the system into a document management path, an asynchronous processing path, and a query path. The Proposed Architecture The design uses the following components: Component Responsibility Azure Blob Storage Stores the original document files Azure Cosmos DB Maintains document metadata, versions, and lifecycle state Azure Service Bus Queues ingestion and deletion jobs Background worker Extracts text, creates chunks, generates embeddings, and updates the index Microsoft Foundry Provides the models used for embedding generation and response generation Azure AI Search Stores searchable document chunks, metadata, and vectors FastAPI Exposes document management and chat APIs Microsoft Agent Framework Supports agentic query workflows when multi-step reasoning or tool use is required The background worker can be implemented as a separate Python service or another suitable compute component. It does not have to be part of the FastAPI process. Why separate these responsibilities? Suppose a user uploads a 100-page contract. The application may need to extract text, split thousands of characters into chunks, generate embeddings, and send multiple indexing requests. Performing all of that synchronously inside the upload request increases latency and makes the API vulnerable to timeouts. Instead, FastAPI accepts the request, persists the document and its metadata, and arranges for the work to be processed asynchronously. The worker performs the expensive operations independently. This allows ingestion to scale separately from query traffic and makes retries easier to manage. The query path remains separate: it retrieves eligible, relevant chunks and uses them to ground the model's response. Establish a source of truth Azure Cosmos DB should hold the authoritative lifecycle metadata in this design. Azure AI Search is the retrieval index, not the authority on whether a document is currently active. For example, a document record might look like this: { "id": "policy-123", "documentId": "policy-123", "currentVersion": 2, "indexedVersion": 1, "status": "PROCESSING", "blobPath": "policies/leave-policy.pdf" } Here, version 2 is the desired version, but version 1 is still the last successfully indexed version. The application must not assume that the latest upload has already become searchable simply because the metadata record has been updated. This distinction between the desired version and the indexed version is central to reliable lifecycle management. Implementing Document Uploads A reliable upload operation involves more than storing a file and sending a message to a queue. It must establish a durable record of the document, identify the version being processed, and ensure that background processing can be retried safely. Step 1: Accept and persist the document When a client uploads a document, FastAPI validates the request, checks the user's permissions, and stores the original file in Azure Blob Storage. The application then creates or updates the document metadata in Cosmos DB. At this stage, the document can be marked as PROCESSING until the ingestion pipeline completes. Step 2: Schedule background processing The system creates an ingestion job containing enough information for the worker to locate the source and determine exactly which version to process. For example: { "operationId": "op-789", "operation": "INGEST", "documentId": "policy-123", "documentVersion": 2, "blobPath": "policies/leave-policy-v2.pdf" } The worker should process the specified version rather than blindly reading whatever file happens to exist at a mutable storage path. For systems that overwrite the same blob path, use blob versioning, an immutable version-specific path, or another mechanism that ensures the worker reads the intended content. Step 3: Extract, chunk, and embed The worker downloads the appropriate document and extracts its text. The extracted text is divided into chunks. Each chunk should carry identifiers that connect it to its parent document and version. For example: { "id": "policy-123-v2-chunk-001", "documentId": "policy-123", "documentVersion": 2, "chunkNumber": 1, "content": "Employees are entitled to 20 days of annual leave." } The worker generates an embedding for each chunk and writes the chunk, its metadata, and its vector to Azure AI Search. The exact index schema depends on the retrieval strategy, embedding dimensions, filtering requirements, and whether hybrid search is used. Step 4: Mark the version as indexed After the worker confirms that all required indexing operations succeeded, it updates the document metadata to record the indexed version and processing status. For example: { "id": "policy-123", "currentVersion": 2, "indexedVersion": 2, "status": "ACTIVE" } This update must be conditional on the document still being at version 2. If a newer version has already been uploaded, the older job must not mark itself as the current version. Why idempotency matters Distributed systems can deliver the same job more than once. A worker may finish indexing but fail before acknowledging the queue message. The message can then be delivered again. If every attempt creates new chunk IDs, duplicate content may accumulate in the index. A practical solution is to derive chunk IDs deterministically from the document ID, version, and chunk identifier. Reprocessing the same version then targets the same index records instead of creating a new set each time. This does not eliminate every failure mode, but it makes repeated indexing much safer. Azure Service Bus supports duplicate detection for eligible tiers, but queue-level duplicate detection is not a substitute for idempotent worker logic. Microsoft Learn: Duplicate detection Updating Documents Without Serving Stale Content Document updates are the most interesting part of the lifecycle because they introduce a consistency problem. Imagine the current leave policy is version 1, allowing 15 days. An administrator uploads version 2, allowing 20 days. The worker starts processing version 2, but embedding generation takes time. Meanwhile, users continue querying the application. What should those users see? There are two common strategies. The right choice depends on whether the system prioritizes availability or strict freshness. Strategy A: Make the new version active after indexing succeeds In this approach, the existing active version remains available while the new version is processed. The workflow is: Create version 2 and mark it as processing. Extract text and generate embeddings for version 2. Index all required version 2 chunks. Verify indexing success. Atomically update the authoritative metadata to make version 2 active. Clean up version 1's indexed chunks asynchronously. This minimizes the time during which the document is unavailable. However, it means that version 1 can remain available while version 2 is being processed. That may be acceptable for some knowledge bases but inappropriate for policies that must become effective immediately. Strategy B: Disable the old version as soon as an update is accepted For high-sensitivity documents, the application can immediately mark the old version as ineligible for retrieval. It then processes the new version and activates it only after successful indexing. The trade-off is that the document may temporarily have no eligible version. The application should return a clear message or use an approved fallback rather than silently serving obsolete information. Prevent old jobs from overwriting new versions Consider this sequence: Version 2 starts processing. Version 3 is uploaded before version 2 finishes. Version 3 finishes indexing first. Version 2 finishes later and attempts to update the metadata. Without version checks, the late version 2 job could incorrectly reactivate stale content. The worker must therefore use conditional updates or optimistic concurrency checks against the authoritative document version. A job may complete its indexing work, but it must not activate its version unless it is still eligible to do so. Handle the lack of a cross-service transaction Cosmos DB and Azure AI Search do not participate in a single distributed transaction in this architecture. A metadata update can succeed while an indexing request fails. An indexing request can succeed while the subsequent metadata update fails. The system should account for these partial failures using: Explicit lifecycle states. Conditional version updates. Retryable and idempotent operations. Periodic reconciliation between metadata and indexed records. The important engineering principle is to make incomplete operations detectable and recoverable, rather than assuming every multi-step workflow succeeds atomically. Deleting Documents Safely Deletion requires special treatment because removing a document from the source does not automatically remove its indexed content. More importantly, waiting for physical index cleanup before preventing retrieval creates a window in which content that should no longer be available may still be returned. Step 1: Make the document ineligible for retrieval When an authorized deletion request arrives, the application updates the authoritative metadata to indicate that the document is being deleted or has been logically deleted. The retrieval path must check that status before using retrieved content. For sensitive documents, this check must happen before the content is added to the LLM context. Merely hiding a document in the user interface is insufficient. Step 2: Schedule physical cleanup The application enqueues a deletion job that identifies the document and all indexed records associated with it. The worker queries or otherwise identifies the corresponding chunk keys and deletes those records from Azure AI Search. The deletion operation should be idempotent. Retrying a deletion for an already-removed chunk should not cause the workflow to fail unnecessarily. Step 3: Verify cleanup The worker checks whether the targeted records have been removed and records the cleanup result. If an operation fails, the job should be retried according to a defined policy. Jobs that continue failing should be sent to a dead-letter queue for investigation and controlled recovery. Azure AI Search supports deleting indexed documents by their document keys. Its documentation also describes how to verify deletion and why index statistics may not update immediately. Microsoft Learn: Delete documents A simplified FastAPI deletion endpoint The following example illustrates the API boundary. The helper functions represent application-specific implementations. from fastapi import FastAPI, HTTPException app = FastAPI() @app.delete("/documents/{document_id}") async def delete_document(document_id: str): document = await get_document_metadata(document_id) if document is None: raise HTTPException( status_code=404, detail="Document not found" ) # Must enforce authorization and use a # concurrency-safe lifecycle transition. await mark_document_deleting(document_id) # The job must be published reliably. await enqueue_deletion_job( document_id=document_id, document_version=document["currentVersion"] ) return { "documentId": document_id, "status": "DELETING" } This is an illustrative example, not a complete production implementation. In particular, the metadata transition and message publication must be coordinated reliably. One approach is the transactional outbox pattern: write the lifecycle change and an outbox event in the same Cosmos DB transactional batch when the data model and partitioning allow it, then publish the event asynchronously. The publisher can retry without losing the intent to schedule cleanup. Deletion must also account for in-flight queries Suppose a query retrieves a document's chunks just before the deletion request is processed. The query might still hold those chunks in memory even after the document is marked deleted. For stricter deletion guarantees, the application needs a defined policy for in-flight requests, cache invalidation, and authorization checks immediately before content is sent to the model. If the requirement is that no response may contain content after a deletion takes effect, that guarantee must explicitly cover requests already in progress. Physical cleanup and retrieval eligibility are separate responsibilities. Both matter. Coordinate Lifecycle Changes Reliably Once ingestion and deletion run asynchronously, the queue becomes part of the system's correctness model. A production worker should not assume that each message is delivered exactly once or that messages for a particular document always finish in order. Design for at-least-once processing A practical worker lifecycle is: Receive a message using a lock-based delivery mode. Validate the document ID, version, and operation. Check whether the operation is still eligible to run. Perform idempotent indexing or deletion. Update lifecycle metadata conditionally. Acknowledge the message only after successful processing. If the worker fails before acknowledging the message, the message may be delivered again. That is expected behavior and should be handled by the design. Add retry and dead-letter policies Not all failures should be treated equally. Transient network errors or throttling may succeed after a delay. Invalid documents, unsupported file formats, and persistent schema mismatches may require manual investigation. The system should use bounded retries with appropriate backoff, record meaningful failure details, and route repeatedly failing messages to a dead-letter queue. Use reconciliation as a safety net Even with retries, the system needs a way to detect inconsistencies that escaped the normal workflow. A reconciliation job can compare authoritative metadata against indexed document versions and lifecycle states. It can identify cases such as: A document marked active whose current version is not fully indexed. Chunks belonging to an obsolete version. Search records for a logically deleted document. An operation that remained in a processing state beyond its expected duration. Reconciliation should repair only states it can safely determine. It should not reactivate a version simply because its chunks exist in the index. For applications using Azure Cosmos DB change feed, document metadata changes can also drive incremental background processing. The change feed captures inserts and updates in its standard latest-version mode; capturing deletions requires selecting and configuring an appropriate supported mode or using explicit lifecycle events. Microsoft Learn: Cosmos DB change feed Where Microsoft Agent Framework Fits A common mistake is to assume that every part of an agentic RAG application should be implemented as an AI agent. Document uploads, version transitions, embedding generation, and index deletion are predictable operations with explicit success and failure conditions. These are generally best implemented with conventional application logic and background workers. Microsoft Agent Framework becomes useful when the query itself requires multi-step reasoning, tool selection, or orchestration. For example, a user might ask: “Compare the current leave policy with the previous version and explain the changes that affect employees.” An agent could use approved tools to retrieve the current policy and the permitted historical version, compare the relevant passages, and generate an evidence-based explanation. The application must still control which documents and versions are eligible for retrieval. The agent should not independently decide whether a deleted document is authorized for use. This distinction is important: deterministic workflows protect data integrity; agentic workflows help solve tasks that benefit from reasoning and tool use. Microsoft Agent Framework provides both agent and workflow capabilities, allowing developers to choose an appropriate orchestration model for the task. Microsoft Agent Framework documentation Production Readiness: What Should We Measure? A pipeline that successfully indexes a document once is not necessarily production-ready. The system needs observable lifecycle states and measurable recovery behavior. Area What to validate Ingestion Can a failed upload or indexing job be retried safely? Idempotency Does processing the same document version twice create duplicate records? Version consistency Can a delayed job reactivate an older version? Deletion Can logically deleted content enter the LLM context? Authorization Are tenant and document-level permissions enforced during retrieval? Recovery Can reconciliation detect and repair partial failures? Observability Can operators identify stuck jobs, indexing failures, and cleanup delays? Performance How long does it take for an accepted update to become searchable? Useful operational metrics include ingestion success rate, queue age, indexing latency, retry counts, dead-letter queue depth, time spent in each lifecycle state, and deletion cleanup latency. Also test race conditions deliberately. Upload a new version while an earlier job is still running. Delete a document while a query is in flight. Simulate an indexing failure after metadata has been updated. Restart a worker after it has indexed content but before it acknowledges the message. These tests expose the problems that a happy-path demonstration will not reveal. Bringing It All Together Keeping a RAG system accurate is not simply a matter of re-running embeddings whenever a file changes. It requires a deliberate lifecycle design that connects source documents, versioned metadata, asynchronous processing, search-index operations, and retrieval-time authorization. Azure Blob Storage can preserve the source documents, Cosmos DB can track authoritative lifecycle state, Service Bus can distribute background work, and Azure AI Search can provide retrieval. Microsoft Foundry supplies the models, while Microsoft Agent Framework can support the parts of the application that genuinely benefit from agentic orchestration. The architecture becomes reliable when it assumes that failures will happen: messages may be repeated, jobs may finish out of order, index operations may partially succeed, and cleanup may take time. Version checks, idempotent operations, explicit lifecycle states, reliable event publication, and reconciliation turn those failure modes into manageable engineering problems. The objective is not merely to keep the index populated. It is to ensure that every answer is grounded in information that is current, eligible, and authorized for the user who asked the question. Further Reading Azure AI Search: Delete documents Azure Cosmos DB: Change feed Azure Service Bus: Duplicate detection Microsoft Agent Framework I'd be interested to hear from engineers building production RAG systems: how do you coordinate document updates and deletions across your source of truth and search index?Implementing Next Best Action (NBA) in Foundry with Microsoft-Decision-1
Imagine a sales rep opening their account list at the start of the day. One customer wants a product demo. Another has a larger potential deal, but its requirements may not match what your product can do. A third has asked you not to contact them for another week. A construction permit looks promising, but you are not even sure the applicant is the company in your CRM. You do not need another summary of the list. You need to know: what should I do next? That is the starting point for this demo: a next best action workbench for sales reps, built with Microsoft Foundry. It compares rules, GPT-4o, and Microsoft-Decision-1 against the same evidence. The accounts and evidence are synthetic; the hosted model responses are real. The interesting part is not which model wins. It is how the sales question changes, and what that reveals about the kind of decision support we need. A Bigger Deal Is Not Always the Next Call The first scenario puts four fictional sales motions in front of the rep: Apex: a $76,000 expansion. The buyer has requested a dispatch demo, and the capability exists today. Summit: a $210,000 cross-sell. The buyer wants capabilities that are not all established as currently supported. Coastal: a $158,000 expansion. The customer has explicitly paused sales contact until October 23. A similar-name permit applicant: a $3.1 million construction project, but an unverified account match and unknown software opportunity value. Sorting by the largest number would miss the point. A construction budget is not software revenue. A large opportunity does not justify promising a roadmap feature. A territory's quota pressure does not override a customer's request not to be contacted. So "next best action" is really two questions: which account deserves attention, and what kind of attention is appropriate? Contact now, clarify requirements, defer, and request human review are different outcomes. Doing nothing yet can be the correct one. Some of those decisions should not be left to a model. In this application, code enforces contact restrictions, evidence requirements, and identity checks before any model recommendation is accepted. Coastal's pause is a boundary, not a factor a model can trade against deal size. The unverified applicant needs review, not a confident sales approach. With those boundaries in place, we can compare how each method handles the remaining judgment. First, Try the Simple Answer The first version uses a transparent scoring formula: software opportunity value, territory quota shortfall, a requested meeting, and whether the buyer needs an unavailable capability. In the recorded routine comparison, all three methods agreed on the ranking and next actions. Rules needed no model tokens and were the simplest adequate implementation for that case. That is a useful outcome. If the relevant facts are already explicit and a few conditions capture the decision, adding a model may add latency and cost without adding useful capability. But consider what happens when "meeting requested" stops telling the whole story. The Same Meeting Request Can Mean Different Things The second scenario is a separate authored version of the synthetic evidence, not a claim that these accounts changed in real time. Apex now has the larger seller-entered estimate, but the meeting is exploratory: there is no established buying process or approved budget. Summit has a smaller funded pilot, completed requirements review, and near-term rollout readiness. Both have requested a meeting. Both need supported capabilities. To the fixed formula, those meeting requests look the same. It ranks Apex first. To a rep reading the context, a discovery invitation and a funded pilot are not equivalent signals. We could extend the rules with readiness fields and more conditions. That may be the right choice. But this is also where a different kind of model becomes interesting: one that interprets the evidence and returns a bounded decision rather than a paragraph. Where Decision Models Fit Models such as TypeSafe AI's Jev have brought attention to this decision-first approach. Supply the situation, define the possible answers, and receive structured choices or scores that software can use directly. TypeSafe calls this a System One model. Microsoft-Decision-1 brings that approach to Microsoft Foundry. Instead of asking it to write a recommendation, this application asks two questions for each eligible account: what action fits, and how much priority should it receive? The application defines the available actions and the priority scale. Decision-1 does not generate a written rationale. It returns a selected action and a score. Its API also supports yes/no probability questions. Jev is context for the emerging category, not a model tested in this comparison. In the recorded contextual runs, both Decision-1 and GPT-4o ranked Summit first, while rules ranked Apex first. One earlier run also showed a useful distinction: GPT-4o suggested outreach for Apex; Decision-1 suggested researching it further. The takeaway is not that models understand sales better than rules. It is narrower: the context changed the models' recommendation in a way this particular formula could not express. That is a reason to evaluate contextual judgment, not proof of better revenue outcomes. These are authored teaching cases, and "Summit first" is the intended case criterion, not a measured commercial result. Choosing the Account Is Only the Beginning Now change the question again. It is no longer just "should we pursue the funded pilot?" It becomes: Can we propose a rollout that fits the customer's deadline and budget? A selected action is no longer the complete answer. The rep needs a proposal: phases, dates, costs, and remaining budget. That is the third scenario. All methods receive the same operational facts, but only the generative adapter is implemented to produce a plan. Rules and Decision-1 remain explicitly decision-only. GPT-4o uses a calculation tool for the budget, then returns a structured schedule and proposal. The successful recorded plan finished on October 22, cost $23,200, and left $300 of budget. Application code independently checked phase order, business-day durations, non-overlap, costs, deadline assessment, and supplied references. The path to that result matters. Earlier model attempts produced correct schedules but incorrect costs, and the application rejected them. The fix was to provide an arithmetic tool, not weaken the checks or substitute a prewritten answer. This is a role for generation alongside deterministic tools. It is not a claim that only an LLM can solve a schedule: a purpose-built scheduler could solve these fixed constraints too. It shows why choosing an action and producing a usable plan are different tasks. The Use Case Gives Us the Framework We started with one sales question, but uncovered three kinds of work: What the rep needs A mechanism to consider What still belongs in code An action from explicit, structured facts Rules The conditions, formula, and permitted actions. A contextual choice within known options A bounded decision model, compared with a structured-output LLM Evidence checks, action boundaries, and output validation. A proposed plan or other generated artifact A generative LLM with tools, or a purpose-built solver where appropriate Calculations, constraint checks, and approval before action. This is why I find "rules, bounded decisions, and generation" more useful here than simply "deterministic versus non-deterministic." Repeatability alone does not tell us whether a recommendation is useful or correct. The required output and the cost of an error tell us more. These mechanisms can coexist in one workflow. Keep hard boundaries deterministic. Use contextual judgment where it adds value. Use generation when you need a generated artifact. Microsoft Agent Framework coordinates the comparison: What This Demo Does, and Does Not, Show The comparison is deliberately small and inspectable. Every method receives the same prepared evidence; expected teaching outcomes stay out of model input. In the recorded decision-only runs, Decision-1 had lower measured branch time and estimated token cost than GPT-4o. Those are single-run observations, not a general speed, quality, or savings benchmark. The planning case performs additional work and is not a like-for-like latency comparison. The scenarios use a fixed simulation time of October 15, 2026, at 12:00 UTC. The primary demo retrieves versioned synthetic snapshots from Azure AI Search, not live CRM or live IQ-agent responses. It does not turn historical records into fresh facts. The workbench opens editable synthetic drafts; it sends no outreach and makes no CRM changes. "Best" therefore means a suggested next action under the supplied evidence and rubric, not a proven optimal action or a prediction of sales success. A valid response shape or evidence reference does not establish a correct judgment. Production use would need representative held-out evaluation, verified access controls, and appropriate human approval. The practical starting point is simple: take a decision your users already make, define the available actions, identify the non-negotiable boundaries, and compare the simplest adequate implementation with the alternatives. Let the use case earn the complexity. Try the Workbench The decision-lead-agent-demo repository contains the local workbench and the three scenarios. Follow the local setup instructions to reproduce the comparison with approved model deployments and credentials. Hosted calls can incur charges. Start with the routine case, where all three may agree. Then inspect what changes in the contextual case, and what extra work the planning task requires. The question to carry into your own application is not "where can I add AI?" It is "what does the next action require that my current approach cannot provide?" Technical Walkthrough: The Code Behind the Comparison The story above does not require reading the implementation. For builders, the following sections retain the exact rules and invocation code, pinned to commit d1a16a5. Microsoft Agent Framework runs each method against an isolated copy of the same prepared snapshot. Shared checks restrict actions before the decision and validate the result afterward. Invalid results are rejected rather than repaired and reported as successes. Snapshot fingerprints verify comparable inputs; evidence-reference checks verify membership, not the truth of a model's judgment. 1. The Rules Baseline The baseline is intentionally transparent. It weights capped software opportunity value at 30%, proportional territory quota shortfall at 20%, a requested meeting at 30%, and no unavailable-capability requirement at 20%. It scales the result to 0-3. Here is the complete rules runner from the implementation at the demo commit. The territory-specific branch supports the separate live research mode; the synthetic comparison uses the opportunity formula. async def rules_runner(snapshot: dict[str, Any]) -> dict[str, Any]: suggestions = [] for candidate in snapshot["candidates"]: signals, policy = candidate["buyingSignals"], candidate["policy"] performance = candidate["territoryPerformance"] gap = max(0, performance["quota"] - performance["revenue"]) / performance["quota"] score = ( 30 * min((candidate["opportunityAmount"] or 0) / 250000, 1) + 20 * min(gap, 1) + 30 * (signals["meetingRequested"] is True) + 20 * (signals["requiresUnavailableCapability"] is False) ) if snapshot.get("scope") == "territory-prospecting": score = 100 * min(gap, 1) action = "outreach" if signals["requiresUnavailableCapability"] is False and signals["meetingRequested"] is True else "research" if not {"outreach", "research"}.intersection(policy["allowedActions"]): action, score = ("defer" if "defer" in policy["allowedActions"] else "cannot_tell"), 0 suggestions.append({ "candidateId": candidate["candidateId"], "action": action, "priority": round(score * 3 / 100, 4), "evidenceIds": [item["evidenceId"] for item in candidate["evidence"]] + [item["evidenceId"] for item in snapshot["productEvidence"]], "basis": (f"Territory research only: proportional quota shortfall {gap:.2f}; no inferred buyer intent." if snapshot.get("scope") == "territory-prospecting" else f"Rules: value {min((candidate['opportunityAmount'] or 0) / 250000, 1):.2f}; quota gap {gap:.2f}; meeting {signals['meetingRequested']}; unavailable capability {signals['requiresUnavailableCapability']}."), }) return validate_recommendations({"status": "ok", "model": "Deterministic rules v1", "suggestions": suggestions, "usage": None, "inferenceCostUsd": 0}, snapshot) These weights are demonstration heuristics, not validated sales economics. Their advantage is that their behavior is easy to inspect. Their limitation is equally concrete: the formula does not read whether a meeting is exploratory or follows an approved, funded pilot. That limitation is not inherent to all rules systems. We could add structured readiness fields or more rules. The question is whether maintaining that logic is preferable to using a model for the contextual part. 2. Invoke Microsoft-Decision-1 Decision-1 is not a drop-in model-name change on the chat endpoint. It uses a different request contract: POST <resource-origin>/providers/microsoft/v1/systemone The request contains the deployment name, shared state, and questions. The following is the complete adapter method from the model adapters. Its surrounding class supplies the HTTP transport; policy_exclusions, ACTIONS, and token_cost are repository helpers. async def decision1(self, snapshot): rubric = snapshot.get("rubric", RUBRIC) priority_criteria = (["No research priority or review required", "Low research priority", "Moderate research priority", "High research priority"] if snapshot.get("scope") == "territory-prospecting" else ["Not actionable or review required", "Exploratory", "Timely but uncertain", "Ready and timely"]) questions, suggestions = {}, policy_exclusions(snapshot) for candidate in snapshot["candidates"]: candidate_id = candidate["candidateId"] allowed = candidate["policy"]["allowedActions"] if not {"outreach", "research"}.intersection(allowed): continue questions[f"action_{candidate_id}"] = { "type": "choice", "instructions": f"{rubric} Choose the next action for {candidate_id}.", "criteria": {action: ACTIONS[action] for action in allowed}, } questions[f"priority_{candidate_id}"] = { "type": "score", "instructions": f"{rubric} Rate attention priority for {candidate_id}.", "criteria": priority_criteria, } if not questions: return {"status": "ok", "model": "No model call: all candidates policy-gated", "suggestions": suggestions, "usage": None, "inferenceCostUsd": 0} response = await self.transport( os.environ["DECISION1_ENDPOINT"].rstrip("/") + "/providers/microsoft/v1/systemone", {"model": os.environ["DECISION1_DEPLOYMENT"], "state": snapshot, "questions": questions}, ) answers = response["answers"] for candidate in snapshot["candidates"]: candidate_id = candidate["candidateId"] if not {"outreach", "research"}.intersection(candidate["policy"]["allowedActions"]): continue action_answer, score_answer = answers[f"action_{candidate_id}"], answers[f"priority_{candidate_id}"] if action_answer.get("type") != "choice" or score_answer.get("type") != "score": raise ValueError("Unexpected Decision-1 answer type") suggestions.append({"candidateId": candidate_id, "action": action_answer["choice"], "priority": score_answer["score"], "evidenceIds": supplied_ids(candidate, snapshot), "basis": "Supplied evidence references; Decision-1 provides no written rationale."}) return {"status": "ok", "suggestions": suggestions, "usage": response.get("usage"), "model": response.get("model", os.environ["DECISION1_DEPLOYMENT"]), "inferenceCostUsd": token_cost(response.get("usage"), "DECISION1")} Two details matter. First, each question identifies the candidate in its instructions, not only in its dictionary key. Second, action criteria come from the application's policy-allowed set. Related action and priority questions share one request and one state. The four score levels run from 0 to 3; a returned score can fall between levels. The demo consumes the selected action and score, not the full probability distributions. The evidence IDs attached here are supplied references, not model-generated citations or a written explanation. 3. Invoke GPT-4o With Structured Output The LLM receives the same snapshot and qualitative rubric, but uses Chat Completions with a strict JSON schema. The basic comparison asks for decisions, not a persuasive paragraph. That gives the LLM a bounded task too, rather than making it generate an explanation that Decision-1 cannot provide. These are exact excerpts from HostedModels.llm, not a standalone script. The method derives eligible_ids from policy and defines this schema: item_schema = { "type": "object", "additionalProperties": False, "properties": {"candidateId": {"type": "string", "enum": eligible_ids}, "action": {"type": "string", "enum": ["outreach", "research", "cannot_tell"]}, "priority": {"type": "number"}, "evidenceIds": {"type": "array", "items": {"type": "string"}}}, "required": ["candidateId", "action", "priority", "evidenceIds"], } schema = {"type": "object", "additionalProperties": False, "properties": {"suggestions": {"type": "array", "items": item_schema}}, "required": ["suggestions"]} For the decision-only task, plan_instruction is "Do not generate an explanation." The payload and endpoint are: payload = { "model": os.environ["LLM_DEPLOYMENT"], "messages": [{"role": "system", "content": rubric + f" Return JSON suggestions ONLY for these eligible candidate IDs: {json.dumps(eligible_ids)}. The application supplies excluded candidates separately. Use candidateId, action, priority and evidenceIds. " + plan_instruction}, {"role": "user", "content": json.dumps(snapshot)}], "response_format": {"type": "json_schema", "json_schema": {"name": "seller_priorities", "strict": True, "schema": schema}}, } endpoint = os.environ["LLM_ENDPOINT"].rstrip("/") + "/openai/v1/chat/completions" After any optional planning-tool interaction, the actual call and parsing are: response = await self.transport(endpoint, payload) responses.append(response) result = json.loads(response["choices"][0]["message"]["content"]) The schema constrains response structure. Shared application validation still checks policy, candidate coverage, score bounds, and evidence references. Structured output is not a substitute for those checks. 4. Run the Demo Locally To try the story yourself, clone the demo repository and open it in WSL. You need Node.js/npm compatible with the project, Python 3.13, and uv. For the full comparison, you also need access to existing GPT-4o and Microsoft-Decision-1 deployments and an Azure AI Search index containing the synthetic snapshots. Starting the local app does not provision those services. Configure your environment. Create a root .env using the repository's .env.example as a template. Set these values for your own resources: Setting What to provide LLM_ENDPOINT / LLM_DEPLOYMENT Your GPT-4o resource origin and deployment name. DECISION1_ENDPOINT / DECISION1_DEPLOYMENT Your Decision-1 resource origin and deployment name. DECISION_RECORDS_SEARCH_ENDPOINT Your Azure AI Search service endpoint. DECISION_RECORDS_SEARCH_INDEX decision-demo-snapshots-v1, populated with the repository's canonical records. Model endpoints must be HTTPS resource origins, not project URLs or complete API paths. Sign in with a local Azure developer identity permitted to invoke both models and read the Search documents; the backend uses that identity, not the simulated UI persona. No credentials belong in the browser configuration. Follow the snapshot-seeding instructions to populate your own index before comparing. Setting its name alone does not create or load it. The seeding tool offers a credential-free preview; uploading requires approved access and index/document-write permissions. The focused three-tab UI retrieves these stored synthetic records and does not silently substitute local fixtures if retrieval fails. You do not need to deploy the separate IQ agents for this path. Start the comparison backend. From the repository root in a WSL terminal: uv venv --python 3.13 .venv uv pip install --python .venv/bin/python -r agent/requirements-workflow.txt npm ci npm run workflow The backend listens on http://127.0.0.1:8008. Restart it after changing environment settings. Start the workbench. In a second WSL terminal, from the same repository root: npm run dev Open http://localhost:5175/. If that port is occupied, run npm run dev -- --port 5176 instead and open http://localhost:5176/. Use the development server, not vite preview: the development server forwards comparison requests to the local backend. Walk through the three scenarios. Select a scenario tab, inspect the account context, then press Compare approaches. Switching tabs does not itself call a model. Start with the routine case, move to the discovery-versus-readiness case, and finish with the rollout plan. Compare the returned actions and ranks, then inspect branch times, token estimates, and the plan's independent checks. If a branch is unavailable or fails, treat that as an incomplete comparison, not a model recommendation. Hosted inference and Search can incur charges. The deployment used for the recorded demo permits one Decision-1 request per minute, so allow at least a minute between comparisons on that deployment; your own limits may differ. This is a local development app: keep the backend on loopback and do not expose it publicly. 5. Measurement Details The repository records these hosted browser observations from October 16, 2026. Each comparison confirmed matching prepared snapshot fingerprints across branches. Scenario Observed result Branch time / estimated inference token cost Routine, 06:06 UTC All methods agreed on ranks and actions. Rules: 0.13 ms / $0; GPT-4o: 3,364.12 ms / $0.003690; Decision-1: 2,570.10 ms / $0.000110418. Contextual, 06:07 UTC Both models ranked Summit first; rules ranked Apex first. GPT-4o: 6,509.38 ms / $0.003795; Decision-1: 3,619.02 ms / $0.000115248. Decision plus plan, 06:04 UTC Tool-assisted GPT-4o plan passed independent checks. Other adapters supplied no plan. GPT-4o: 7,386.28 ms / $0.016510, including both requests. These are single-run observations on deliberately authored synthetic cases, not latency distributions, a held-out accuracy study, or a general savings claim. A future run can produce different timings or decisions. GPT-4o is the existing general-purpose baseline, not an exhaustive search for the best alternative model. Costs use returned token usage and configured retail rates. Cached GPT-4o input is priced separately without double-counting. Estimates exclude Search, IQ retrieval, hosting, taxes, and contract-specific adjustments; $0 for rules means no inference token charge, not zero operating cost. Microsoft's launch pricing lists Decision-1 at $0.042 per million input tokens with free output. Recheck current pricing, deployment availability, and preview lifecycle before reproducing the experiment. Before relying on model scores, evaluate representative held-out cases, paraphrases, and option ordering. Measure abstentions, invalid outputs, failures, latency distributions, and cost per successfully validated task. A priority score is not a probability of closing a sale, and matching actions or rankings does not imply calibrated score equivalence. For model setup and the current API contract, see Deploy and use Microsoft-Decision-1 in Microsoft Foundry.
Building Permission-Aware Hosted Agents for SharePoint in Foundry Agent Service
Organizations increasingly publish agents to Microsoft Teams and Microsoft 365 Copilot that answer questions using internal knowledge sources. SharePoint is a very common source for this information; where many times access is different for every user. A finance employee, HR employee, and project manager may all ask the same question but have access to different documents. When an agent is hosted in Foundry Agent Service, the agent itself runs under its own identity. This creates an important challenge: how can retrieval respect the permissions of the person asking the question rather than the permissions of the agent? This post shows how to do that with built-in capabilities: a Foundry IQ knowledge base with a remote SharePoint knowledge source, connected to the agent through the Foundry toolbox. Foundry IQ is the knowledge layer for agents, built on Azure AI Search. A knowledge base is its top-level resource: it orchestrates retrieval across one or more knowledge sources, which are connections to indexed or remote content. In this solution the only knowledge source is SharePoint, queried live as the signed-in user through the On-Behalf-Of (OBO) pattern, and the agent reaches the knowledge base through a toolbox. Why agent identity is not enough Hosted agents run as their own agent identity. When Foundry Agent Service runs your Microsoft Agent Framework agent as a hosted agent, it authenticates as the agent, not as the person chatting with it. SharePoint permissions belong to users. Two people asking the same question may be allowed to see different documents. So grounded answers (RAG) need per-user retrieval. If retrieval runs as the agent, SharePoint can't check the asking user's access. Results are either empty or not filtered for that user. Therefore the user's token has to reach the retrieval step. Changing the model or the prompt can't fix this. The architecture has to carry the user's identity all the way to SharePoint. This is the key architectural principle behind this solution. The hosted agent keeps using its own identity for model inference, while retrieval runs with the signed-in user's identity. What you'll build A hosted agent, published to Microsoft Teams, that answers from one or more SharePoint sites. Answers limited to what each user can open in SharePoint, using On-Behalf-Of (identity passthrough) retrieval. Clickable citations back to the source documents. Built on out-of-the-box Foundry IQ and Foundry toolbox capabilities: no custom server to host, and no client secrets in the retrieval path. The complete sample is on GitHub: SharePoint knowledge base sample The sample repository contains the setup scripts and agent code used in this walkthrough. Before we run them, let’s trace how the user’s identity flows through the architecture and why it matters for permission trimming. Note: Remote SharePoint knowledge sources in Foundry IQ were in preview at the time this article was written. How it works The agent runs as itself. Retrieval runs as the user. Three pieces carry the user's identity end to end: FoundryToolbox in the agent sends the per-request call ID with each tool call, so Foundry knows which user the call belongs to. A UserEntraToken connection passes that user's Microsoft Entra ID token to the Foundry IQ knowledge base on Azure AI Search. This is identity passthrough (On-Behalf-Of): every request downstream runs as the signed-in user. A remote SharePoint knowledge source queries SharePoint live through the Microsoft 365 Copilot Retrieval API as that user, scoped to the site paths you choose. Nothing is copied into an index. Why not Work IQ? Work IQ is a good choice when an agent should reason over a user’s broader Microsoft 365 work context. This pattern is intended for scenarios where the agent should answer from specific SharePoint sites, treating those sites as a defined knowledge boundary. Use Work IQ when broad Microsoft 365 context is appropriate. Use this pattern when retrieval should be limited to selected SharePoint content. The right choice depends on the scope and behavior your scenario requires. Before building the sample, let’s check the services and permissions the setup requires. Before you start You need a Foundry project with a model deployment, an Azure AI Search service and a SharePoint site containing the content you want the agent to use.The Search service must be in a region that supports agentic retrieval, in the same Microsoft Entra tenant as Microsoft 365. Make sure the following permissions are in place: Role Required permissions Setup user Search Service Contributor on the Azure AI Search service, and rights to create connections and toolboxes in the Foundry project End users Search Index Data Reader on the Search service, Foundry Agent Consumer on the agent, a Microsoft 365 Copilot licence, and access to the documents in SharePoint With the concepts established, let's build the end-to-end solution. Build it in four steps Clone the sample repository before running the setup scripts. The commands below assume you are running them from the repository root: git clone https://github.com/Heenanl/hosted-agent-sharepoint-knowledge-base.git Set-Location hosted-agent-sharepoint-knowledge-base python -m pip install -r setup/requirements.txt Copy-Item setup/.env.example setup/.env Fill in setup/.env with your Foundry project, Azure AI Search, and SharePoint knowledge source values. The repository README contains the complete configuration and deployment instructions. The steps below highlight the key settings and decisions. 1. Create the knowledge base Run setup/create_knowledge_base.py . It creates a remote SharePoint knowledge source and a knowledge base. The setting that limits retrieval to your site is: "filterExpression": "Path:\"https://<TENANT>.sharepoint.com/sites/<SITE>/\"" 2. Connect it to Foundry with the user's identity Run setup/create_connection.py . It registers the knowledge base's MCP endpoint as a Foundry connection. The setting that matters most in the whole architecture is: "authType": "UserEntraToken" With UserEntraToken , Foundry forwards the signed-in user's token instead of a shared identity, so SharePoint can apply that user's permissions. Then run setup/create_toolbox.py . It wraps the connection in a toolbox that exposes only knowledge_base_retrieve . 3. Build the agent The agent itself is small. The key line is tools=FoundryToolbox(credential) : agent = Agent( client=FoundryChatClient(project_endpoint=endpoint, model=model, credential=credential), instructions=INSTRUCTIONS, tools=FoundryToolbox(credential), ) Two instructions in the agent's prompt made a visible difference in Teams: Cite every source as a markdown link, using the Title and uri fields the tool returns. If the tool returns nothing relevant, say that the available sources do not contain the answer. Do not guess. The first gives users clickable citations. The second stops the model from filling gaps when a user can't see a document. 4. Deploy and publish to Teams Follow the sample README to initialize the agent with azd ai agent init , set the required environment values, and deploy it with azd up . After deployment, publish the agent to Microsoft Teams and Microsoft 365 Copilot. The repository includes a publishing script, including support for Foundry projects on a private network. Users do not need a separate, tool-specific consent flow. Prove the permission trimming A good answer doesn't prove the right permissions were used. Test with two users and one document that only one of them can open: User SharePoint access Result in Teams A Can open the document Answer from the document, with a citation link B Cannot open the document "The available sources do not contain the answer." In our testing, both users' requests reached the knowledge base, but only user A got content from the document, because the Microsoft 365 Copilot Retrieval API ran as each user. Run the test in separate conversations, and repeat it after any permission change. Common pitfalls Using a managed identity for the connection The retrieval runs as the agent's managed identity, so SharePoint has no user to check permissions against. In our testing, a ProjectManagedIdentity connection failed with x-ms-query-source-authorization is invalid . Use UserEntraToken . Forgetting Search Index Data Reader Because each user's own token reaches Azure AI Search, every user needs this role, not just you. Otherwise the user can reach the agent, but retrieval fails for them. Assign the role to a security group once, replacing SEARCH_SERVICE , RESOURCE_GROUP and $GROUP_ID with your own values: $SEARCH_ID = az search service show -n SEARCH_SERVICE -g RESOURCE_GROUP --query id -o tsv az role assignment create --assignee $GROUP_ID --role 'Search Index Data Reader' --scope $SEARCH_ID Using a custom MCP client instead of FoundryToolbox A hand-built MCP client that attaches only the agent's credential drops the per-request call ID, which links a tool call to its user. In our testing with another OAuth tool, every user ended up running as whoever had consented first. FoundryToolbox fixed it. Building a custom MCP proxy A custom On-Behalf-Of (OBO) proxy can work, but you then own its hosting, secrets, token checks and patching. The knowledge base gives the same per-user retrieval with built-in components. Once the setup and permission checks are complete, use the sample repository to try the full flow in your own environment. The links below provide more detail on the Foundry IQ and toolbox building blocks. Get started Try it: clone the sample and follow the README end to end. Learn the building blocks: Create a remote SharePoint knowledge source Connect a Foundry IQ knowledge base to Foundry Agent Service Toolbox authentication What is a Foundry toolbox? Use a toolbox with a hosted agent Validate before rollout: use the two-user isolation checklist. Share your scenario: open an issue on the repo. Acknowledgements Thanks to Mahya Gheini and Linda Li from the Microsoft Foundry product team for their guidance on toolboxes and knowledge bases.
