ai agents
126 TopicsMicrosoft Retriever 1 brings sub-second agentic retrieval to Foundry IQ
Microsoft Retriever 1 is an agentic search model trained by Microsoft and now in preview in Foundry IQ. It cuts typical knowledge base retrieval latency by more than 6x, raises search evidence recall by 8–11 points, and lets agents use a third fewer input tokens. It needs no provisioning or reindexing.138Views0likes0CommentsKnowledge graphs in Foundry IQ improve evidence recall with fewer tokens
Knowledge graphs in Foundry IQ, now in public preview, add two capabilities. Entities answered 93.5% of structured queries correctly versus 26.1% for RAG, using 64% fewer tokens. Topics improved evidence recall on thematic questions by up to 13.2 points over vector search.74Views0likes0CommentsFoundry IQ: Introducing knowledge graphs, engineered for agents
Your applications already help people find business information by keyword, field and facet. Agents use that same information to get work done: checking a policy in SharePoint, finding a supplier contract, or working out which products a new regulation affects. Retrieving the right information from your business data takes expertise and the right building blocks. Without them, agents repeatedly search for and piece together the same facts and relationships, adding time and cost. Foundry IQ helps developers connect unstructured business content to agents. Today we’re announcing grounding in SharePoint documents with permissions enforced, knowledge graphs that connect facts across those documents, and a specialized retrieval model that plans each search. Together, they are built to get an agent to the right answer in fewer searches. Foundry IQ is part of Microsoft IQ Platform, the enterprise context management system that captures, governs and reuses the business context your apps and agents need. Microsoft IQ brings together four sources of context: Work IQ for how people work, Fabric IQ for governed business data, Web IQ for the web, and Foundry IQ for your unstructured data. What’s new Today we’re announcing the following capabilities in Foundry IQ: Knowledge graphs, public preview: Map entities during indexing using your ontology, or bring an existing graph, then query connections in natural language or GQL with source citations. Get started with the Foundry IQ plugin. SharePoint knowledge source with permissions, generally available: Index your SharePoint content into Foundry IQ, and each agent sees only what the person using it can see with user Entra group and SharePoint site group permissions. Foundry IQ Serverless, generally available in December 2026: Usage-based pricing that scales to zero when idle, with nothing to size up front and no commitments. General availability adds a production SLA, higher limits and more regions. Microsoft Retriever 1, public preview: A specialized retrieval model inside Foundry IQ, built to plan each search at lower cost, with no frontier model to deploy for the retrieval plan. Knowledge graphs: connect the dots across your documents Knowledge graphs in Foundry IQ give agents two ways to understand your business: entities connect the facts across your documents, and topics organize your content by theme. As it indexes your documents, Foundry IQ uses your ontology to map how your documents, contacts and information relate. It records the source of every connection. Entities connect facts that sit in different documents, and each connection keeps its source. Topics organize the same content by theme for broad questions. Topics answer the broad questions, such as which themes run across every contract. Flag a question as broad, and retrieval draws on the whole collection. Knowledge graphs are available in public preview: Entities: Add a graph configuration with your ontology to a knowledge source you already use, or bring a graph you already have. Query through the knowledge base in natural language, or the graph directly in GQL. There’s no graph database to build and no second pipeline to run. Topics: Topics and subtopics identified automatically from your existing embeddings, with no ontology required, and organized into a sitemap of your corpus. Topics can also propose a candidate ontology for you to review. Install the Foundry IQ plugin to get started with knowledge graphs. Microsoft Discovery grounds R&D workflows with Foundry IQ Knowledge graphs in Foundry IQ are a key component of Bookshelf, the knowledge experience in Microsoft Discovery. With Foundry IQ integration, Discovery customers can now leverage a wider variety of data sources, such as SharePoint and Work IQ, to ground their R&D workflows with high-quality scientific reasoning. John Link, Partner Product Manager, Microsoft Discovery & Quantum Find the answer in fewer steps Every search an agent runs is a tool call you pay for in tokens. An agent without your knowledge in reach searches, then searches again. Foundry IQ plans each search for the agent, and it’s built to get agents to the right answer in fewer searches. Grounding your agents is simpler, too. Foundry IQ is built into every new Foundry project, powered by serverless, so you can connect your enterprise knowledge and start building grounded agents without provisioning or managing a separate resource. Microsoft Retriever 1, public preview: Built to plan each search at lower cost. It runs on your existing index, with no frontier model to deploy for the retrieval plan. Foundry IQ built into every new Foundry project, public preview in December 2026: It’s now easier to ground agents with your documents, using a managed Foundry IQ resource provided in every Foundry project. Foundry IQ handles retrieval and indexing, while Foundry sets up and manages the underlying service for you. Foundry IQ Serverless, generally available in December 2026: Usage-based pricing that scales to zero when idle, with nothing to size up front and no commitments. General availability adds a production SLA, higher limits and more regions. Sign up to get started with a managed Foundry IQ resource. Connect to your enterprise content ecosystem Your knowledge is distributed across various source in different formats. Foundry IQ brings it to your agents in Foundry, Microsoft IQ and third party agents, while honoring document and access permissions. Your sources Work IQ SharePoint knowledge source, generally available: Index your SharePoint content through the Microsoft 365 indexer, with its permissions enforced at query time. More knowledge sources, generally available: Azure SQL Database and file upload knowledge sources are generally available, Fabric IQ semantic models and OneLake shortcuts and mirrored items are public preview. Semantic chunking and image descriptions, generally available: Azure Document Intelligence and Content Understanding in Foundry split documents by meaning and make the images in them searchable. Your permissions Source permissions, generally available: Foundry IQ enforces SharePoint user, Entra group and site group permissions, ADLS Gen2 ACLs and RBAC, and Azure Blob container-level RBAC under the caller’s identity, for supported and configured sources. The same controls apply when the knowledge base is reached over MCP. OneLake item-level permissions, public preview: Results from OneLake respect each item’s permissions under the caller’s identity. Purview sensitivity-label enforcement, public preview: Label policy is evaluated when the agent reads, and the call fails closed when policy can’t be verified. OneStream empowers finance teams with better work context and Foundry IQ Frontier Finance teams are rethinking their decisioning and workflows around AI, and they need to work in Excel, Copilot, and other tools of choice to do that, without giving up governance or auditability. Our integration with Foundry IQ means Finance teams have greater work context to their AI answers, while still being grounded in the financial intelligence from OneStream that make those answers auditable and contextualized. Together with Microsoft, we’re unlocking powerful, purpose-built AI that works where Finance teams work, and that creates answers a CFO can stand behind. Tom Shea, Founder and CEO, OneStream Innovating with Microsoft Foundry Here’s how customers are using Foundry IQ to ground their agents in their own business knowledge. McKinsey & Company grounds their AI operating system with Foundry IQ and Fabric IQ AI needs more than access to data. AI needs to understand how a business works. McKinsey’s EcliptOS platform brings together Microsoft Fabric IQ and Microsoft Foundry IQ to connect business data with enterprise knowledge. We build on that foundation with McKinsey’s industry expertise, codified into impact recipes, and our transformation capabilities to turn business context into persistent business value. Brian Elliott, McKinsey Senior Partner and Co-Lead of EcliptOS, McKinsey & Company Rockwell Automation powers troubleshooting assistant with Foundry IQ When a machine goes down on the night shift, our technicians can’t always reach an expert, so the answer they get has to be one they can trust. With Foundry IQ, our troubleshooting assistant draws on each plant’s own manuals, historical downtime records and expert insights, and links every step to its source. When the evidence isn’t there, it says so instead of guessing. That’s what gives our people the confidence to act. Simon Lee, Senior Manufacturing Engineer, Rockwell Automation Get started today The easiest way to explore is through the Microsoft Foundry portal. Read the documentation and get started with the Foundry IQ plugin. Watch Foundry IQ's Microsoft Ignite 2026 sessions in person or on-demand: BRK297 - The shift from RAG to context: Grounding agents with Foundry IQ BRK307 - Foundry IQ: Knowledge graphs for better enterprise context BRK300 - Building agentic retrieval: From DIY to Foundry IQ INV641 - Harnessing your IQ with a continuously improving agent platform200Views1like0CommentsBest practices for building agentic AI applications with Foundry Agent Service
This blog was co-authored by Maira Ladeira Tanke, Mark Roy and Vivek Bhadauria Building an agent has become surprisingly easy. Building an agentic application that is reliable, secure, scalable, and doesn't cost a fortune is still hard. Over the last couple of years, agents have moved very quickly from prototypes to real applications. At the same time, the definition of an agent has expanded considerably. We now have agents that use tools, skills, maintain memory, communicate with other agents, execute code, interact with enterprise systems, and take actions on behalf of users. All these capabilities are useful, but they also make it easier to over-engineer an application. This is the first post in a series about building agentic applications with Microsoft Foundry. Foundry Agent Service brings together the tools to build agents, run them in production, and deliver them to users through channels such as Microsoft Teams and Microsoft 365 Copilot. We will cover topics such as evaluation and optimization, security, context management, and scale in much more detail in future posts. For this first post, however, we step back and look at some of the basic engineering principles that we believe are important regardless of the use case. Some of them sound obvious. In practice, they are also very easy to get wrong. Start with the business problem Agentic AI is very popular at the moment, but an agent is only as useful as the challenge it is solving. Before creating an agent, define the problem you are trying to solve. Agentic AI allows applications to deal with situations where the steps required to accomplish a task cannot necessarily be defined in advance. An agent can reason about the information available to it, decide which tools it needs, use their results to determine what to do next, and adapt its behavior along the way. However, flexibility comes at a cost. Every time you introduce reasoning where deterministic application logic would have been enough, you are adding latency, cost, and nondeterminism. Sometimes that is exactly the tradeoff you want to make. Sometimes it isn't. Math is a simple example. LLMs have become much better at math, but if your application already has all the inputs necessary to calculate a value, there is very little reason to spend tokens asking a model to calculate it. Code is faster, cheaper, and deterministic. The same is true for many validation rules, transformations, authorization checks, and predefined business logic. This doesn't mean those operations can't be part of an agentic workflow. They absolutely can. It means the agent can orchestrate them instead of replacing them. A principle we like to use is: if no reasoning is required, an agent is probably not required either. Microsoft Foundry gives us a lot of flexibility in how to build agents. We can define prompt agents through instructions and tools, use the Microsoft Copilot harness (currently in preview) for more complex, multi-step work, or bring our own code and orchestration with hosted agents, while leveraging any agent framework (LangGraph, Microsoft Agent Framework, and more). If you need to bring voice interactions to your application, voice agents for prompt and hosted agents make it simple. Toolbox lets us share tools across agents, and the choice of models means we can pick one that fits the task rather than design everything around a single model. This flexibility is valuable precisely because not every workload should have the same architecture. The objective shouldn't be to build the most agentic solution possible. It should be to build the simplest solution that solves the problem well. Define success early This is another step that is surprisingly easy to skip, especially during a proof of concept. Before spending too much time optimizing prompts, tools, models, or architecture, define what a successful agent means for your use case. For a customer support scenario, success could mean reducing the average time it takes to resolve a case. It could mean increasing case deflection without negatively affecting customer satisfaction. For an insurance claim resolution agent, it might mean reducing the amount of manual work required to complete a process. Those are very different goals, and they result in very different evaluation strategies. Technical metrics are important as well. We want to understand latency, token consumption, tool failures, answer quality, groundedness, and many other characteristics of the system. But a technically impressive agent is not necessarily a useful one. Those same success criteria should guide model selection. Compare candidate models on representative tasks from your application: can they produce grounded answers, choose the right tools, and complete the task within your latency and cost constraints? Choose the model that meets those requirements, rather than defaulting to the most capable model for every task. Re-run those evaluations before switching models. This becomes especially important when we try to calculate ROI. If we didn't establish the baseline and what we expected the agent to improve, it becomes very difficult to demonstrate later that the application created value. Define these measures early and keep them close to the application throughout its lifecycle. They will eventually become the foundation for your evaluations and production monitoring. Don't build it alone There is another tool available to us that we think developers sometimes forget about when building agents: other agents. Coding agents have become extremely capable. Whether you use Github Copilot, OpenAI Codex, Anthropic Claude Code, or one of the many other tools available today, they can help with a significant part of the work involved in building an agentic application. And you should use them. We don't mean asking a coding agent to blindly build your entire application from a one-line description and hoping for the best. The engineering principles in this post still apply. You need to understand the problem you are solving, make architectural decisions, define your success criteria, and validate what gets built. But there is very little value in manually writing every piece of boilerplate when a coding agent can help you scaffold the project, integrate APIs, create tests, troubleshoot an authentication issue, or iterate on your agent implementation. This gets particularly interesting when the coding agent understands your target platform. Bring Foundry into the workflow you already use. Foundry Toolkit for VS Code, Azure Developer CLI, and GitHub Copilot integrations support local development, scaffolding, debugging, and deployment. These tools give coding agents platform-specific guidance and give developers ways to work with Foundry from their terminal and development environment, carrying the same agent implementation toward production. The important part for us is that coding agents should not be treated as fancy autocomplete anymore. Give them the context they need and access to the appropriate skills and developer tools. Let them help you inspect your project, implement changes, run tests, evaluate results, and iterate. Interestingly, many of the recommendations in this post apply to your coding agent too. Give it vague instructions and little context, and you will probably spend your afternoon correcting it. Give it clear requirements, the right tools, platform-specific guidance, and a way to validate its work, and the experience is very different. You are building agents. You might as well use agents to help you do it. If you want to get started, Microsoft's guide for setting up the Microsoft Foundry development environment covers the Foundry developer tooling and how to configure your environment. Be very clear about what you expect from an agent Ever seen an agent get things completely wrong? It happens often. Not because the underlying models aren't capable, but because we tend to expect them to understand information that we never actually supplied, like you would do with your co-worker. The difference is your co-worker has months, if not years of context working with you. An agent only has the context available to it. If the instructions are ambiguous, the agent has to resolve that ambiguity. If two instructions conflict, it must decide which one to follow. If a tool has a vague description, the LLM model has to guess when that tool should be used. The more guessing we introduce, the less predictable the application becomes. Take a tool described simply as: def get_customer_data(): # Gets customer information. ... There is technically nothing wrong with that description, but there is a lot the model needs to figure out. What customer information does it return? When should it be called? What identifier does it expect? Should it be used for transactions as well? What about support cases? We can remove most of that uncertainty ourselves: def get_customer_profile(customer_id): """Returns the customer's account profile, including: - name, - account tier, - region and - account creation date. Use this tool when account profile information is required. Do not use it to retrieve transactions or support cases. """ ... This is a small example, but the same idea applies across a complete agentic application. Instructions, tool definitions, knowledge sources, memory, skills, and even descriptions of other agents are ultimately information we are providing to a model so that it can make a decision. The clearer that information is, the less work the model has to do figuring out what we intended. Longer isn't automatically better either. Adding another three pages to the system prompt every time the agent makes a mistake will eventually create other problems. That leads us directly into context management. Pay attention to what goes into the context When an agent invokes an LLM, a surprising amount of information can end up contributing to that request. There are the agent instructions and the user's input, but potentially also conversation history, tool definitions, retrieved documents, memory, skills, previous tool results, and information coming from other agents. All of that competes for context. Too little context and the model doesn't have enough information to do the job. Too much context and you are paying to process information that might not even be relevant to the current task. Worse, useful information can become harder to identify among everything else you provided. This is also why simply adding more tools isn't always an improvement. Each tool needs a description and input schema that the model can understand, so a growing toolset gives the agent more information to navigate. In Foundry, tool search retrieves relevant tools instead of loading every definition into each request, while skills supply reusable procedures when needed. A toolbox provides a shared, governed endpoint; tool search is what addresses the context cost of presenting the whole catalog. The same applies to conversation history and memory. Persisting something doesn't automatically mean it should be provided to the model on every subsequent request. This is why we think context management needs to be treated as an architectural concern, not just a prompt engineering problem. You need to understand what information is reaching the model, why it is there, how often it is being sent, and whether it is actually improving the outcome. And once applications become more complex, understanding this by looking at the final response becomes almost impossible. Doing it by hand gets tedious quickly, which is where insights and optimization come in later in this post. Which brings us to one of our favorite topics. Make your agent observable from day one Agentic applications can fail in many creative ways. Maybe the model picked the wrong tool. Maybe it picked the correct tool but passed the wrong parameters. Perhaps retrieval returned poor information. Perhaps retrieval returned too much information. Maybe the system prompt contains two instructions that made perfect sense independently but contradict each other when combined. From the user's perspective, all of those problems often look exactly the same: The agent gave me a bad answer. And good luck debugging that from the output alone. This is why we strongly recommend investing in tracing and observability early, including during development. When you can inspect the actual execution of the agent, the conversation changes considerably. Instead of guessing why the agent failed, you can see which tools were called, what was retrieved, where time was spent, and what happened throughout the execution. OpenTelemetry (OTEL) traces are particularly useful here because agentic applications are still distributed applications. The model invocation is important, but so are the tool calls, retrieval operations, downstream services, and other agents involved in completing the request. A trace can show that an answer was bad because retrieval introduced irrelevant context, that latency increased because the agent made five unnecessary calls, or that a tool description caused the model to choose the wrong operation. These are all things we can fix, but only when we can see what the agent is doing. Foundry's tracing helps developers inspect this behavior and understand what happens inside their agentic applications. Insights in Foundry (currently in preview) goes a step further: it analyzes production traces and groups recurring behavior into insights, each with representative traces and a likely cause, so you aren't reading traces one at a time. Once you can observe the behavior, you can start systematically measuring whether it is getting better. Build evaluation into the development lifecycle Evaluations shouldn't be something you run once before going to production. Every meaningful change to an agent has the potential to change its behavior. Change the system instructions and something else might regress. Add a tool and the agent now has another possible choice. Change the model and assumptions you made during development might no longer hold. Change retrieval and the model receives different information. This means you want an evaluation dataset early in development. It doesn't have to contain thousands of examples. Start with representative scenarios that are important to the application. Include the happy paths, but also include ambiguity, unusual phrasing, expected tool calls, cases where the agent should ask for clarification, and cases where it should not take an action at all. You don't have to build that first dataset by hand. If your agent already emits traces, Foundry can convert them into an evaluation dataset using intelligent sampling, which picks a representative set instead of a random slice. The same traces can also seed a rubric evaluator. If you don't have production traces yet, Foundry can also generate a synthetic dataset from your agent's instructions, a prompt, or a reference document such as a policy or spec, which covers prelaunch scenarios and edge cases. The two complement each other: traces show real user behavior, synthetic data covers what users haven't asked yet. The success criteria we defined earlier become useful here. Some evaluations will be technical, such as task completion, tool selection, groundedness, latency, or token consumption. Others can be much closer to the business outcome. Foundry gives us a few ways to turn those criteria into evaluations. The built-in evaluators cover common measures such as relevance, groundedness, task completion, tool call accuracy, and safety risks, so you get a first signal without writing anything. For criteria specific to your application, rubric evaluators use an LLM as the judge to score responses against weighted criteria you define, and you can auto-generate the first version of the rubric from your agent's instructions. Custom evaluators cover the rest: code-based (a Python function for deterministic checks such as format or length), prompt-based (your own judge prompt), and endpoint-based (an HTTP endpoint you host, useful when scoring logic lives in your own infrastructure). Rubric and custom evaluators are currently in preview. As the application evolves, the dataset should evolve with it. Production failures that matter are particularly good candidates for future regression tests. Evaluation also has a practical limitation: it costs money. Running every evaluator against every interaction in production can quickly become expensive. This is why an evaluation strategy matters just as much as having evaluations. Match the evaluation scope to the stage of development. During development, run smaller benchmarks frequently. Before important releases, run larger test sets through your CI/CD pipelines. In production, evaluate representative samples. The aim is enough data to understand behavior and identify regressions, not every evaluator on every request. Foundry brings evaluation into the same development lifecycle as the agent. Combined with agent insights, that gives us a way to move from diagnosing behavior to testing improvements. The agent optimizer uses that evaluation evidence to generate and rank candidate instructions, skills, tool descriptions, and model selections for prompt and hosted agents, which takes over much of the manual tuning that context management otherwise needs. We still need to assess quality, latency, and token consumption together: a version that scores better does not automatically cost less to run. There is enough here for an entire post, so we will explore evaluation and agent optimization separately in this series. Be deliberate about identity and isolation Personalization is important, but it comes with an equally important requirement: my experience with an agent can be different from yours, but my data should never accidentally become part of yours. That sounds obvious. Implementing it correctly requires thinking about several different concepts. Conversation state, session isolation, memory, user identity, agent identity, and authorization are related, but they aren't interchangeable. Memory, for example, can make an agent much more useful. The agent might remember preferences or information that allows it to provide a better experience over time. Foundry's managed memory, currently in preview, provides persistence without requiring developers to provision an external memory database. But managing the infrastructure doesn't answer the governance questions: whose information is it, where is it stored, how is it scoped, when should it be retrieved, and how long should it be retained? Those decisions still belong in the application design. Sessions introduce another boundary. In hosted agent architectures in Foundry, session isolation provides an execution boundary for a workload. Prompt-based experiences have their own mechanisms for maintaining conversation state. In either case, developers need to be deliberate about where state lives and what is shared (and can't be shared) between interactions. Identity becomes especially important when agents start using tools. Suppose we ask an agent to retrieve a document from a system we use at work. The easiest implementation might be to give the agent a highly privileged application identity capable of reading everything. It is probably not the implementation we want. Microsoft Entra ID gives us the capabilities to do this differently. One pattern you will encounter is On-Behalf-Of, usually shortened to OBO. At a very high level, OBO allows an application to call another service using the user's delegated identity. If you are using an agent to access information on your behalf, the downstream system can still make an authorization decision based on who you are and what you are allowed to access. Foundry supports this through user delegation, and the configuration guide shows how to set it up. You don't need to understand every OAuth detail to appreciate why that matters. If you cannot access a document, adding an LLM between you and the document shouldn't suddenly give you access to it. Foundry provides security building blocks, but they still need to be configured for the application. Microsoft Entra Agent ID gives agents an enterprise identity, and the Foundry runtime honors Agent 365 governance actions. Private connectivity, tool authentication and runtime guardrails help control access and behavior, with support varying by component and configuration. Security, identity, memory, and isolation get complicated quickly, so we will examine those boundaries and tradeoffs in a follow-on post. Until then, the identity and access overview is a good map of where each decision sits, and agent identity concepts explains how Foundry uses Entra Agent ID. Keep application logic in your application Earlier we mentioned math as an example of something that doesn't necessarily belong inside an LLM invocation. The same principle extends much further. Validation, transformations, deterministic routing, permission checks, sorting, filtering, and established business rules are operations that traditional software is very good at. Use it. An agent is part of an application. It doesn't replace the application. If a user asks an ambiguous question that requires interpreting intent and deciding which system can answer it, that's a good place for reasoning. Once the system knows exactly which API to invoke and has received structured data back, many subsequent operations are better handled by code. This distinction is useful for reliability, but it can also have a significant impact on latency and cost. A few unnecessary LLM calls probably won't matter during a demo. At production volume, they matter a lot. There is one nuance here. Telling an LLM not to calculate something directly doesn't mean an LLM can't help solve the problem. LLMs are also very good at generating code. For some workloads, generating code and executing it in an appropriate controlled environment can be a much better architecture than asking the model to reason through every intermediate calculation in tokens. This is especially interesting for data analysis and tasks where the requested computation itself is dynamic. Foundry gives us an off-the-shelf Code Interpreter tool that acts as a secure sandbox for this approach. The decision is not whether the model is capable of doing something. In many cases, it is. The more useful engineering question is whether a model invocation is the right component of the application to do it. Plan for scale: within an agent, and across your organization A successful proof of concept eventually faces two different scale problems: one agent serving thousands of users, and an organization creating thousands of agents. The first is about production capacity and reliability. The second is about reuse, ownership, and governance. An agent that works very well for ten people needs to continue working when thousands of users invoke it concurrently. At that point you start thinking about throughput, latency, quotas, model capacity, downstream dependencies, rate limits, connection management, resiliency, and cost. Agent runtimes can help with part of that problem, but your agent is only one component in the architecture. Its models, tools, data sources, APIs, and other dependencies all need to support the level of traffic you expect. This is also where capabilities such as AI Gateway become interesting. As AI usage grows, organizations need mechanisms for managing access to models, controlling and observing traffic, enforcing policies, and understanding consumption across applications. Cost governance needs explicit ownership and limits: set budgets for each application or team, track consumption against them, and define who responds when spending approaches those limits. Azure Cost Management remains the system of record for spend. AI Gateway provides project-scoped tokens-per-minute limits and total token quotas; those are token controls, not native dollar-budget enforcement in Agent Service. Decide in advance how budget alerts and request controls will work together, including when to alert, throttle, or stop requests, rather than waiting for the bill to reveal a problem. And there is another kind of scale that we find equally important: what happens when the problem isn't one agent serving thousands of users, but an organization creating thousands of agents? The requirements change. Different teams may independently create similar tools. Every agent might implement authentication differently. Tool integrations get duplicated. Agents need owners and lifecycle management. Developers need ways to discover capabilities that already exist. Organizations need governance without preventing teams from moving quickly. Tools should become reusable building blocks rather than implementations hidden inside individual agents. This is one of the reasons we find approaches such as reusable Foundry tooling and MCP particularly interesting. A well-designed capability can be exposed once and consumed by multiple agents instead of every team independently rebuilding the same integration. The agents themselves can also become reusable. A specialized agent built by one team might provide useful capabilities to another agent or application. Multi-agent communication makes it possible to compose those capabilities instead of creating one enormous agent that knows how to do everything. Distribution is another part of this problem. Users shouldn't necessarily need to visit a separate application for every agent an organization creates. Depending on the use case, agents may need to appear in applications, websites, Microsoft Teams, Microsoft 365 Copilot, or other channels users already work in. Foundry Agent Service provides a publishing path into Microsoft Teams and Microsoft 365 Copilot, with Agent 365 integration for tenant-wide governance. That connects the engineering decision about where an agent runs with the adoption question of where people will actually use it. Once organizations reach this point, the problem is no longer how to build an agent. The problem is how to provide an agentic platform that allows many teams to build agents without every team independently solving identity, tooling, observability, evaluation, governance, integration, and deployment. We are going to spend much more time on that problem later in the series. And this is just the start The goal isn’t to add more AI at every step. It’s to build applications that solve real problems and keep improving them as we learn. These principles are a starting point; their value comes from putting them into practice. In subsequent posts, we’ll get hands-on with evaluation and optimization, security and identity, context management, and scaling, exploring what Microsoft Foundry provides, the architectural tradeoffs, and how to turn what we learn into better applications, and more business value. Authors Maira Ladeira Tanke Maira is a Principal Software Engineer for Agentic AI at Microsoft Core AI, focused on helping customers successfully adopt Microsoft Foundry Agents. Mark Roy Mark Roy is a Principal Software Engineer at Microsoft working on Microsoft Foundry and AI agents, passionate about helping developers get agents out of the demo stage and into production. Vivek Bhadauria Vivek Bhadauria is a Partner Software Engineer at Microsoft focused on building AI agents, platforms, and optimization loops.1.3KViews5likes0CommentsBuild a dual-mode voice agent in Microsoft Foundry with native subagents
Voice agents are changing how companies interact with customers. From service questions to technical support, the expectation is the same: a natural conversation that leads to useful, reliable answers. But responding quickly and resolving a complex request are not always the same task. Some questions can be answered directly. Others require information from backend systems, specialized analysis, or several steps to determine what should happen next. The challenge is to support that deeper work without putting the conversation on hold, or giving an answer before the necessary information is available. In this blog, I will walk through a dual-mode voice-agent pattern in Microsoft Foundry. A frontline agent manages the real-time conversation, while a backend agent handles specialized work. We will explore how these responsibilities work together, how the frontline can continue interacting while backend processing runs, and how to configure and verify the experience. The pattern is broadly applicable, with particular relevance to telecom customer service and support. You will learn how to: Separate real-time conversation from specialized backend work. Configure a native voice subagent Understand how Voice Agents build on Voice Live Test delayed responses, interruptions, missing information, and backend failures. How voice agents work Before we look at our full workflow. Lets have a look at the voice experience in Microsoft Foundry. A voice application needs to carry audio, manage the conversation, and connect that conversation to useful information or actions. Those responsibilities work together, but they are not the same layer. A browser or contact-center connector handles audio capture and playback The voice runtime manages streaming and turn-taking Instructions, knowledge, and authorized tools determine what the agent can say and do. In Microsoft Foundry, Voice Live provides the real-time speech foundation. Voice Agents build on it with a named, versioned agent definition and capabilities such as tools, tracing, and evaluation. The diagram below shows the underlying voice pipeline. Audio arrives from the client Before anything else, Voice Live cleans up that audio. It uses conversational enhancements that can reduce background noise and echo and help detect when a speaker has finished a turn. The exact combination depends on the selected model, pipeline, and audio settings; not every session uses all the processing stages shown The conversation can then either follow a native speech-to-speech path. A model in the GPT Realtime family processes spoken input directly and produces a spoken response. Supported configurations can also use Azure Speech voices for output, so choosing a realtime model does not mean we must use its native output voice. Alternatively, a cascaded pipeline converts speech to text, sends that text to a language model or agent, and synthesizes the response as speech. This separates the recognition, conversation model, and voice choices. Voice Live can manage that pipeline without us writing the streaming and turn-taking loop ourselves. In either path, useful answers may also require knowledge sources or external tools. These connections give the agent access to information and actions beyond the conversation itself. Configure the required permissions and safety controls for the selected model and application; a tool or content-safety box in the diagram is not a substitute for those decisions. On the return path, speech streams back to the client for playback. Where supported, Azure Speech provides standard and HD voices, with custom voice subject to access requirements. These choices describe how the spoken conversation runs. The dual-mode pattern adds a different separation: the frontline owns that conversation, while a backend agent handles a focused task. It does not require two speech-to-speech models, and the backend does not speak directly to the caller. How the frontline and backend work together To keep the conversation responsive while specialized work runs, we separate the caller-facing interaction from the backend task. Each agent has a clear responsibility: The frontline agent is responsible for the interaction. It listens, takes turns, answers supported questions, and explains what happens next. The backend agent has a different job: work through a focused task, such as reviewing incident evidence or recommending a troubleshooting sequence. The important part is that delegation does not have to take the caller out of the conversation. The frontline can remain active while the backend works. In the native subagent pattern, this behavior is configured through the agent definitions, instructions, and response policy rather than a separate routing model The following diagram shows how those responsibilities fit together: The voice client captures and streams the caller’s audio: The client connects the caller to the frontline voice agent, sends microphone audio, and plays returned speech. It can also display transcripts and runtime events. The client can send audio and receive events concurrently. The frontline voice agent decides whether to answer directly or delegate: The frontline owns the live conversation. It handles straightforward requests within its configured knowledge and permissions without invoking the backend. When a request needs specialized work, the frontline uses its instructions and the configured subagent’s description to determine when to delegate. Direct answers return through the voice client without a backend invocation. Foundry invokes the configured backend subagent. The voice agent’s subagent_config references a backend in the same Foundry project by name. Foundry invokes that backend for the delegated task. The frontline retains ownership of the voice conversation; the caller is not transferred to the backend. The backend performs the task and uses its tools. A prompt or hosted agent follows its own instructions and can retrieve data, call APIs, or execute supported functions. Access to service information, incident records, or diagnostics must be explicitly configured and authorized. Meanwhile, the frontline keeps the voice interaction active. Its response_policy controls an immediate acknowledgment, gap-filling speech, and supported progress updates. The basic sample enables only immediate_ack=True; additional behavior requires configuration. The frontline can receive further caller input, but it must not invent backend findings while waiting. Further caller input does not automatically update an in-flight backend task. Results and supported progress return through Foundry. The backend returns its output to the frontline through managed delegation. Where supported and enabled, progress can also inform the conversation before completion. The diagram’s return arrow does not mean every backend automatically streams intermediate findings. The frontline presents the result to the caller. Following its instructions, it explains the findings, communicates uncertainty, and identifies appropriate next actions. Speech streams back through the client for playback. The backend does not speak directly to the caller, and its output is not necessarily identical to the spoken response. Note: The GPT Live samples demonstrate an alternative implementation of the conversation/backend split. They use VoiceAgentLiveDefinition and live.connect(), with direct Responses or managed prompt-agent delegation. The design goal is responsiveness without sacrificing depth. A fast acknowledgment improves perceived waiting time; it does not shorten the underlying investigation or guarantee an accurate answer. Measure conversational responsiveness and backend task quality separately. Decide when the frontline should ask for help With the flow in place, the next question is when the frontline should ask for help. Define clear boundaries in the frontline agent’s instructions and the subagent’s capability description. Rule 1: Answer directly when the frontline has sufficient grounded information and permission to respond. Rule 2: Delegate when the backend owns the required capability. This might include combining information from several systems, assessing service impact, investigating symptoms, or developing a troubleshooting plan. Rule 3: Escalate when the request exceeds the automated workflow’s boundaries. Define what should happen when an action requires human approval, evidence remains insufficient, or a tool failure prevents progress. Implement and test the human-handoff or failure path; instructions alone do not provide it. From the caller’s perspective, this should feel like one conversation, not a handoff between agents. The frontline listens, clarifies the request, and brings in specialized help when needed. The caller does not need to understand which backend agent or tool is doing the work. When backend processing begins, an enabled acknowledgment can make that transition clear: “Let me check that for you.” If gap-filling speech or supported progress updates are configured, the frontline can keep the caller informed while the task runs. Those updates should reflect what is actually known, without implying that a check has completed or a result has been confirmed. Once the backend returns, the frontline explains the outcome in language suited to the conversation. That might be a confirmed finding, a recommended next step, or a clear statement that more information is needed. The goal is not to fill every pause with speech. It is to keep the caller oriented while specialized work happens in the background, and to make the eventual answer useful, understandable, and grounded in evidence. Build the native voice-subagent workflow Now that we have defined the conversation flow, let’s configure the agents behind it. We will use native voice subagents throughout this walkthrough: a VoiceAgentDefinition for the frontline, a prompt agent for the backend, and subagent_config to connect them. Voice subagents are currently in preview. Microsoft does not recommend features in preview for production workloads. Step 1: Give the backend one focused responsibility Start with the work the frontline should delegate. A useful backend has a clear responsibility. The subagent can be a Foundry prompt agent or a hosted agent in the same project. For our example here we will create a prompt agent that can analyze information supplied to it. It can be connected to knowledge bases and different tools to help. You can also create a subagent in the Microsoft Foundry portal or deploy a code-based hosted agent. Keep the instructions specific to the backend’s actual capabilities. If it can only analyze supplied information, say so. If you later add an incident lookup, update the instructions and capability description to reflect that change. Instructions guide model behavior; they do not enforce permissions or guarantee an output schema. Restrict access in the tools and services themselves, and validate any structured fields your application relies on. Separate read-only investigation from actions that change a customer account or service. Step 2: Connect the backend to the frontline Next, we can create the caller-facing voice-based agent. We can create this using the SDK. In its instructions we should explain when to delegate to the backend agent and we should add the backend agent to its subagent_config. In the subagent_config, you identify the backend agent by name and describe when the voice-based agent should delegate a request to it in the agent_capabilities, which describes the specialist’s responsibility. You can use the following code to spin up such an agent (Also see: Use a subagent in a Voice-based agent - Microsoft Foundry | Microsoft Learn) import os from azure.ai.projects import AIProjectClient from azure.ai.projects.models import ( RealtimeAudioFormatsAudioPcm, VoiceAgentAudioConfig, VoiceAgentAudioInputConfig, VoiceAgentAudioOutputConfig, VoiceAgentDefinition, VoiceAgentInputTranscription, VoiceAgentServerVadTurnDetection, VoiceAgentSubagent, VoiceAgentSubagentConfig, VoiceAgentSubagentResponsePolicy, ) from azure.identity import DefaultAzureCredential endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"] model = os.environ.get("FOUNDRY_VOICE_MODEL", "gpt-realtime-2.1") definition = VoiceAgentDefinition( model_type="managed", model=model, instructions=( "You are a real-time customer support assistant for service incidents. " "Handle straightforward factual and support-process questions directly. " "For complex troubleshooting, SLA-risk triage, root-cause hypotheses, " "or multi-step remediation planning, call forward_to_subagent for " "the configured telco support agent. " "After the subagent returns, provide a concise, customer-safe summary " "and next actions. " "Do not present uncertain reasoning as confirmed fact." ), store=True, output_modalities=["audio"], audio=VoiceAgentAudioConfig( input=VoiceAgentAudioInputConfig( format=RealtimeAudioFormatsAudioPcm(rate=24000), turn_detection=VoiceAgentServerVadTurnDetection( threshold=0.5, prefix_padding_ms=300, silence_duration_ms=1000, ), transcription=VoiceAgentInputTranscription( model="azure-speech", ), ), output=VoiceAgentAudioOutputConfig( format=RealtimeAudioFormatsAudioPcm(rate=24000), voice="en-US-JennyNeural", voice_type="azure-standard", ), ), subagent_config=VoiceAgentSubagentConfig( subagents=[ VoiceAgentSubagent( agent_name="[BACKEND AGENT NAME]", agent_capabilities=( "Performs deeper SLA-risk triage and troubleshooting " "reasoning, including structured cause hypotheses " "and remediation planning." ), response_policy=VoiceAgentSubagentResponsePolicy( immediate_ack=True, ), ) ], ), ) with ( DefaultAzureCredential() as credential, AIProjectClient( endpoint=endpoint, credential=credential, allow_preview=True, ) as project_client, ): voice_agent = project_client.agents.create_version( agent_name="frontline-voice-agent", definition=definition, description="Voice frontline with subagent delegation", ) print( f"Created voice agent '{voice_agent.name}', " f"version {voice_agent.version}" ) While you currently can't link these subagents through the portal You can see the subagents linked and its instructions in the portal by clicking on the YAML button. Step 3: Configure what the caller hears while waiting With the basic connection in place, we can decide how the frontline should support the caller during backend work. Use the optional response_policy object to control what the voice agent says while it waits for a subagent: immediate_ack: Specifies whether the voice agent immediately acknowledges that it forwarded the request. ack_instructions Provides instructions for generating the immediate acknowledgment. enable_delta_progress Emits progress incrementally instead of waiting for the subagent invocation to finish. The default is false. gap_filling_interval Sets the period without subagent content or user input before a gap-filling response. Use a value from 5 through 120 seconds. gap_filling_instructions Provides instructions for speech generated while the voice agent waits for progress. progress_instructions Provides instructions for summarizing streamed subagent progress as speech. progress_update_interval Sets the minimum time between spoken progress updates. Use a value from 5 through 300 seconds. Step 4: Start a conversation with the voice agent With the frontline and its subagent configured, we can now test the spoken experience. The quickest starting point is the Foundry playground: open the project containing both agents, go to Build → Agents, select frontline-voice-agent, and open it in the playground. Start a voice session and allow microphone access. This tests the saved agent configuration without building a local audio client. For a custom application, we can use the shared native microphone client linked here (Native voice client: dependencies, launch commands, and event handling) or the complete client in the Microsoft Learn walkthrough. Configure it with the same Foundry project endpoint and the name created in Step 2: frontline-voice-agent. The client connects through beta.voice_agents.realtime.connect(), sends microphone audio, and receives speech and events concurrently. The example uses 24 kHz PCM audio and plays the returned audio chunks as they arrive. Follow the setup and readiness sequence of the client we choose rather than combining snippets from different clients. Voice activity detection identifies speech boundaries; the frontline instructions and subagent configuration still determine when to delegate. Start with a general support question, then ask for analysis that should use the backend. Check the execution evidence as well as the spoken answer: an acknowledgment alone does not prove that the subagent ran. In our current configuration, only immediate_ack is enabled, so additional gap filling or progress narration should not be assumed. Use headphones to prevent speaker audio from feeding back into the microphone. Then introduce a delayed response, a caller correction, missing evidence, and a backend failure. Check how the frontline responds and what the execution trace shows. Test interruptions separately: stopping audio playback does not prove that the backend task was canceled. From a microphone to a contact-center call The microphone client helps us validate the agent. Connecting it to a contact center adds another layer: a channel integration that bridges call audio and call-control events to the voice runtime. The division between frontline and backend remains useful, but a working local conversation does not automatically provide telephony routing or human transfer. The Azure-Samples Call Center Voice Agent Accelerator documents browser testing and multiple telephony providers, including Genesys Cloud, Azure Communication Services, and Twilio. It provides a reference for the channel-integration layer. There are also call-center-specific accelerators e.g. This Genesys Voice Live Connector sample. This integration option uses Genesys Cloud AudioConnector and a connector service between Genesys and Azure Voice Live. The connector handles the channel protocol and audio conversion; the agent handles the conversation and its configured tools. Conclusion A dual-mode voice agent separates responsibilities that operate on different timelines. The frontline keeps the conversation clear and responsive; the backend performs specialized work. Native subagent delegation connects them without handing the caller over to another speaking agent. The value is not constant speech or an instant answer to every request. It is an interaction that stays understandable while the necessary work happens, then delivers a response grounded in what that work actually established. Next Steps Ready to go deeper? Use a subagent in a Voice-based agent (preview) – Follow the end-to-end walkthrough to create a backend subagent, configure delegation, and test the spoken experience. Configure a voice agent – Explore model selection, audio settings, turn detection, voices, and agent behavior. Native voice-subagent samples – Explore working examples that connect a frontline voice agent to prompt or hosted backend agents. Voice Live overview – Learn about the underlying speech capabilities, supported models, and regional availability. Genesys Cloud Voice Live Connector – Explore a sample audio bridge between Genesys Cloud and Azure Voice Live, with a simulator for local testing. Call Center Voice Agent Accelerator – Explore deployment guidance and reference integrations for browser-based conversations and supported telephony providers.199Views0likes0CommentsFoundry Toolbox Lifecycle: Deploy and Promote
Foundry Toolbox Lifecycle: Deploy and Promote For developers looking to manage Microsoft Foundry tools across agents, this article presents one recommended pattern: define each Toolbox in its own azure.yaml , deploy it separately from project infrastructure and agents, and pass environment-specific Bicep outputs through GitHub Actions. Why Manage a Foundry Toolbox Separately? Organizations are integrating agent development into established software development lifecycles, raising questions about how teams govern and reuse tools. In this post, a “tool” means an external capability an agent invokes such as web search, a tool exposed by a Model Context Protocol (MCP) server, or a third-party application. Tools let agents retrieve live data or take actions in external systems. How can teams avoid rebuilding the same integrations? Microsoft Foundry Toolbox offers a shared way to manage tools. This post explores how to structure Toolbox ownership and deployment so a reusable collection doesn’t become a Toolshed. What Is a Microsoft Foundry Toolbox? A Toolbox in Foundry is a versioned collection of tools exposed through a single MCP-compatible endpoint. It is created in the context of a Foundry Project, and agents connect to that endpoint to use its tools. Microsoft’s Toolbox overview describes a way to define shared tools once and expose them through a single MCP-compatible endpoint. It centralizes tool access, credentials, governance, and versioning so teams can reuse approved tools across agents without rebuilding each integration. Instead of embedding a separate Web IQ integration in every Finance agent, teams can configure Web IQ once in a shared Toolbox. Each agent then connects to the Toolbox’s MCP endpoint and invokes the tools it exposes. The Toolbox and its project connections provide the shared configuration and credentials. Should a Toolbox Be Deployed with an Agent? Foundry examples show toolboxes created through SDKs and consumed by agents. The key design decision is which deployment lifecycle owns toolbox version changes. When multiple agents share a toolbox, manage its versions separately from any one agent’s deployment. A Foundry Project can contain zero or more agents. Agent deployments should be idempotent and managed independently. An agent’s availability should not depend on another agent. A Toolbox has its own lifecycle and versioning. A Toolbox lives outside any single agent, within a Foundry Project. Creating a toolbox with the Python SDK does not, by itself, make the toolbox part of an agent; the SDK call creates a version in the Foundry Project. The coupling arises when the agent’s deployment pipeline also owns and publishes that toolbox. To keep the lifecycles independent, manage toolbox versions through a project-level pipeline and have agents consume the toolbox endpoint. This avoids redeploying an unrelated agent for toolbox changes and avoids making shared tool availability depend on one agent’s lifecycle. Provision the Project, Then Deploy the Toolbox A Foundry project is provisioned as an Azure resource (`Microsoft.CognitiveServices/accounts/projects`), while toolbox versions are created through the Foundry project API. In this deployment pattern, we need to provision the project and any required connections first, then create and deploy the toolbox using the project endpoint. See the toolbox version API reference for details. The Foundry project and its project-level role-based access control (RBAC) and connections are infrastructure prerequisites that can be managed with infrastructure as code (IaC). The toolbox version is then created in a separate deployment step using the project endpoint, with environment-specific values supplied as Bicep outputs. This keeps the Toolbox alongside the project’s lifecycle without coupling it to any one agent. Where to Store the Toolbox Configuration Like the Foundry Project, a Toolbox is shared across agents and should follow the project’s lifecycle rather than an individual agent’s lifecycle. Store its definition alongside the project’s IaC and include its deployment in the project-level infrastructure workflow. Create a separate folder and `azure.yaml` for each Toolbox. This gives each Toolbox its own deployment step and `azd` environment, so teams can deploy, update, and version Toolboxes independently without coupling their release cycles. This post uses that structure to make each Toolbox’s lifecycle explicit. Foundry Toolbox Versioning and Promotion Foundry stores each Toolbox version as an immutable snapshot of its configuration. Changes create a new version rather than modifying an existing one. The first version becomes the default automatically; subsequent versions must be promoted explicitly. See Foundry Toolbox version managementfor details. Agents using the Toolbox’s stable consumer endpoint receive the default version. Developers can use a version specific endpoint to validate changes before promotion. Promoting a version changes what the consumer endpoint serves without requiring a rebuild and deploy of the agents code. To roll back, promote a previously retained version. Foundry provides these versioning and promotion capabilities, but the team still decides how changes are tested, approved, retained, and retired. A version captures Toolbox configuration not a frozen copy of the external services it calls—so rolling back the Toolbox does not necessarily reverse downstream service changes. i.e. changes that were made to tools of a custom downstream MCP server. Teams can remove older or unwanted versions after confirming they aren’t needed for rollback or used through version specific endpoints. If multiple versions exist, promote another version before deleting the current default. When using `azd`, deleting the only remaining version also removes the Toolbox. One important connection to our example: its `postdeploy` hook promotes immediately. We should distinguish that sample behavior from a production process that validates a candidate version before promotion. Deploy a Foundry Toolbox with GitHub Actions In this example, the Toolbox is deployed as a separate step in the same GitHub Actions workflow, after the Bicep deployment provisions the Foundry Project. This lets the workflow populate the Toolbox’s `azd` environment with the infrastructure outputs before deployment. You can also split the deployments across jobs or workflows, provided the Toolbox step receives the required outputs and runs after its prerequisites are ready. Prerequisites Foundry Project and region: Create the Foundry Project and any project connections required by the Toolbox before deployment. Confirm that the target region supports the Toolbox’s tool types and models using the tool support by region and model table Identity and access: Configure GitHub Actions to authenticate to Azure using OpenID Connect (OIDC), as in the linked sample. The identity that creates or updates Toolbox versions needs the Foundry User role on the project (Toolbox prerequisites). If the same workflow deploys Bicep, that identity also needs permission to deploy the template’s resources and create its role assignments at the applicable scopes. Tooling: Install Azure Developer CLI 1.37.0+, as required by the unified `microsoft.foundry` extension. Install the extension with `azd ext install microsoft.foundry`, following the Foundry extension installation guidance. The linked sample pins `azd` 1.37.1 and installs `azure.ai.toolboxes` separately. Configuration Provide the project endpoint, environment name, location, and every downstream endpoint or connection name referenced by the Toolbox’s `azure.yaml`. Create the required Foundry project connections before deploying the Toolbox. Configure credentials on those connections or in an approved secret store, not in Bicep outputs; see Toolbox authentication. - name: Deploy and promote the toolbox working-directory: foundation/toolboxes/fibey-toolbox env: AZURE_ENV_NAME: ${{ steps.infrastructure.outputs.AZURE_ENV_NAME }} FOUNDRY_PROJECT_ENDPOINT: ${{ steps.infrastructure.outputs.FOUNDRY_PROJECT_ENDPOINT }} AZURE_AI_PROJECT_ENDPOINT: ${{ steps.infrastructure.outputs.AZURE_AI_PROJECT_ENDPOINT }} run: | azd deploy fibey-toolbox --no-prompt The infrastructure deployment should expose the values the Toolbox needs as outputs—in this case, Bicep outputs. The linked sample uses `AZURE_ENV_NAME`, `AZURE_LOCATION`, `FOUNDRY_PROJECT_ENDPOINT`, and `AZURE_AI_PROJECT_ENDPOINT` for the deployment context. Its Toolbox configuration also consumes `INVENTORY_MCP_URL`, `WORK_ORDERS_API_URL`, `KB_MCP_URL`, and `KB_CONNECTION_NAME`. Expose the corresponding values for each target environment and region. The Toolbox’s `azure.yaml` references the imported values, so the same definition can be reused across environments and regions. For each target, provide that target’s infrastructure outputs instead of manually changing endpoint values, and confirm that the required services and tools are supported in the target region. The example references `steps.infrastructure.outputs` because the infrastructure and Toolbox deployment steps run in the same GitHub Actions job. If they run in separate jobs, expose the required values as job outputs and consume them through `needs`. This keeps the deployments connected without tying their release cycles together: infrastructure provides the deployment context, and the Toolbox consumes it. Keep credentials in managed identities or approved secret stores rather than ordinary deployment outputs. The complete workflow demonstrates authentication, infrastructure outputs, and environment initialization. Its extension-install step currently uses `azure.ai.toolboxes`; replace that step with `azd ext install microsoft.foundry` when following the current Learn guidance above. Version promotion is handled by the `postdeploy` hook in the Toolbox’s `azure.yaml`. Verify Toolbox Deployment and Promotion The sample’s `postdeploy` hook promotes the deployed version to the default and checks that promotion succeeded. Redeploying an unchanged Toolbox definition does not create a new version. To inspect the versions and confirm which one is default, run: azd ai toolbox versions list fibey-toolbox The default version is marked with `*`. This confirms which version is default, but not that its tools work. For runtime verification, follow the Toolbox verification guidance to initialize MCP and list tools using the version-specific endpoint. Test a tool call only with a safe, read-only operation. The developer needs the Foundry User role on the project; prompt-agent and OAuth scenarios can require additional permissions for the agent identity or end user. Troubleshooting If deployment or promotion fails, check that the deployment identity has the required project role, that the project endpoint and Toolbox configuration values were supplied, and that the required project connections and tools are available in the target region. Conclusion Toolbox provides a shared way to manage tools across agents without tying their lifecycle to any one agent. Keeping toolbox definitions alongside your Foundry Project infrastructure, while deploying them as a separate step, lets teams update and version tools independently while retaining a clear dependency on the project. As your organization adopts this pattern, consider who owns each toolbox, how changes are tested and promoted, and who can access its tools. That governance will help keep a useful Toolbox from becoming a Toolshed.276Views0likes0CommentsFrom Python Agents to Copilot Harness : Putting Microsoft Foundry to the Test
You have built the agents. Written the Python. Connected the data. Worked out how the pieces talk to each other. Then the conversation shifts to harnesses. The prospect of convergence around the GitHub Copilot harness across GitHub, Microsoft Foundry and Copilot Studio (CPS) raises a very practical question: What happens to the pro-code agents I already built? Do I rewrite them, wrap them, or keep them? Rather than debate it in the abstract, I wanted to give it a shot. I already had a Microsoft Agent Framework (MAF) demo with Python agents, Fabric sales data and product knowledge. Could I keep the useful business workflows, remove the custom agent wrappers, and let a managed harness do more of the work? That experiment became the demo behind this blog. Not another empty chat box, but an app to ask business questions, inspect actual tool activity, follow the evidence and create a brief you can revise and download. This is a practical Foundry experiment and the question I wanted to answer was: how much of this particular agent application can move into the harness, and what still belongs in my code? TL;DR: I moved a Python/MAF research demo to a Foundry-managed Copilot harness to test whether it could handle the same business questions. Native research and source connections reduced the agent-side orchestration I had to write, while the app still owns authentication, streaming and secure access. Downloadable briefs are an extra experiment, not part of the original migration. Focused tests were promising, but do not establish complete feature parity or production readiness. The Starting Point: Pro-Code Agents The starting point is olivierb123/microsoftiq-lead-agent-demo. It used four hosted Agent Framework wrappers and application-controlled fan-out. Python agent code and custom source adapters handled the integrations; the application determined which agents to call and assembled their results. That approach makes the orchestration explicit. It also puts the responsibility for hosting the wrappers, maintaining source adapters and coordinating the workflow in the application. For this experiment with harness, I kept the business scenario but changed the execution model. Instead of recreating those wrappers inside another service, I configured one Foundry-managed Microsoft Copilot harness agent with instructions, native capabilities and connected sources. Responsibility Pro-Code Demo Harness Experiment Workflow orchestration Application-controlled agent calls Managed agent selects tools and can delegate Research Hosted wrappers and custom adapters Native web search and page fetching Business data Application source integration Native Fabric connection Product knowledge Application retrieval integration Foundry IQ knowledge-base retrieval through native MCP Working output Application-managed result assembly Conversation workspace and published artifacts Application code UI plus agent orchestration UI, authentication, streaming and session/download authorization This repository contains the migrated implementation. The original pro-code demo remains separate and was not overwritten. A Practical Question, Not a Tool Demonstration The scenario is a fictional sales team working with FieldForge product documentation. A useful assistant should help answer questions such as: How many opportunities are open in the expanded synthetic demo, what is their total and probability-weighted value, and how do our internal forecasting definitions explain those numbers? That request needs two different kinds of evidence: structured opportunity data and the actual internal document defining the forecast. It should not require the user to name a connector or prescribe a tool sequence. A more advanced request asks for a South territory brief combining sales performance, priority opportunities, product capabilities and official construction heat-exposure guidance. The user can then request a revision or a downloadable Markdown artifact in the same conversation. These examples exercise the harness as a working environment, not just a text-generation endpoint. However, they are requests to explore, not promises that every tool or subagent will run on every turn. Why a Local Web App? I deliberately kept the test experience in a local web app. I wanted to make it easy to ask a question, try a follow-up and inspect what happened, without turning the experiment into a web-hosting project first. The React workbench and Node.js backend run locally; the agent and connected sources run in the cloud. After signing in, I can choose a sample question, watch the streamed response, inspect Activity and Evidence, and request a revision or download in the same conversation. I do not need to reconstruct a raw SDK response to see whether a source was actually called. That separation also makes the migration easier to understand. The web app is the test surface, not the agent runtime. There is no local Python agent server hiding behind it: the backend sends the request to Foundry and handles the application's authentication, streaming and session-owned downloads. For this first experiment, the backend is intentionally loopback-only. Hosting the application publicly is a separate engineering task, not a prerequisite for testing the harness against a useful scenario. The Architecture The configured agent uses microsoft_copilot_preview and gpt-5-mini. The agent definition lives in harness/azure.yaml, rather than a Python agent server. The application, backend and data scripts use JavaScript/Node.js. There is no Python application runtime, pip setup or virtual environment required for this version. That is not a zero-code migration: the application still owns sign-in, token validation, streaming, conversation ownership and secure downloads. The backend does not execute the native source calls a second time. Foundry executes them; the application observes their returned results and presents them in Activity and Evidence. What It Looks Like in the Foundry Portal The local workbench is only one view of the experiment. Foundry also supports hosted pro-code agents: I can package custom agent code, including a Python/MAF implementation, and its dependencies in a container and let Foundry manage the hosting. I still own the agent implementation and its orchestration; using Python does not mean I must run the server myself. This experiment instead uses a managed-harness prompt agent. I configure its instructions, capabilities and source connections, while Foundry supplies the harness execution environment and native tool orchestration. Both approaches can run in Foundry. The distinction is not simply local versus cloud, or Python versus no Python: it is how much agent runtime and orchestration I implement versus delegate to the managed harness. The application's UI, authentication and session/download authorization remain my responsibility in this demo. My microsoftiq-copilot agent and its configured tools/connections make the change tangible. The configuration ties together the native harness capabilities, Fabric access and product-knowledge retrieval shown in the architecture diagram. No Web IQ Wrapper in This Version The Web IQ integration from the earlier demo is not carried forward. External research uses the harness's native web search to discover sources and its Web capability to read relevant pages. This supports the research portion of the scenario without maintaining a separate Web IQ agent wrapper. It is not a claim that native web research reproduces every Web IQ behavior or data contract. Fabric and product knowledge are optional grounding sources, not stages that every question must pass through. A public-research question can stay a public-research question. Private business data must not be sent to public web tools. More Direct Integration with Fabric and Foundry IQ For me, the integration improvement is the ownership boundary: the harness uses configured native connections instead of calling application adapters that obtain credentials and query sources themselves. Fabric supplies structured business facts. The native Fabric connection uses the signed-in user's context to access a dedicated data agent over imported semantic models. There is no app-only Fabric credential fallback. The models contain demonstration data, not live CRM records. Foundry IQ supplies document evidence. The product-knowledge MCP connection reaches a knowledge base backed by Azure AI Search. Query planning and semantic retrieval return extractive evidence, including stored document metadata, for the harness to use in its answer. This connection uses the Foundry project's managed identity; it does not establish per-user document ACL enforcement. The distinction matters. A supportingDocId in an opportunity row is a relationship reference, not proof that the corresponding document has been retrieved. Fabric cannot substitute for document content, titles or update dates from the knowledge base. Calling this integration "better" means more direct native integration with clearer source responsibilities. I have not established a quality, latency or cost improvement over the MAF implementation, nor certified genuinely different-user authorization. What the Experiment Actually Showed Focused tests showed that the harness could retrieve sales figures, find internal documents, research the web and continue a conversation. That supported the core research workflows we wanted to carry over from the MAF demo. The most revealing test asked for sales figures together with the internal document explaining how the forecast should be interpreted. Some early runs got the numbers right but did not retrieve the requested document. The answer sounded convincing, but part of its explanation lacked verified support. After clarifying the agent's instructions and which source should answer each part of the question, one test run successfully retrieved evidence from both sources. This used synthetic demo data, not live customer information. The lesson: connecting the tools is only the start. You still need to test whether the agent uses the right sources and whether those sources actually support its answer. Beyond the Migration: Trying Workspace Files and Briefs Once the core question-answering workflows were working in focused tests, I wanted to try something we had not implemented in the MAF demo: turning an answer into a working document. Why save a brief? Not every answer needs one. But a territory review is a useful example: instead of leaving the analysis in a chat response, I can request a Markdown document to keep, review or hand off, then ask for a revision in the same conversation. The file is an optional deliverable, not a prerequisite for retrieving sales data or product evidence. This is where the harness's conversation workspace and native file capabilities become interesting. I can explore creating, reading back and revising an artifact in the managed environment without rebuilding those agent-side tools in a Python wrapper. MAF could support the same kind of workflow with appropriate tools and code; this is an additional capability in this demo, not something exclusive to a harness. Publication adds a separate step when the user actually wants a download. A workspace file is not automatically downloadable: the workbench registers it only after a successful native upload returns a real file ID, and only the owning unexpired session can download it. The application still implements those access checks. The migration question is therefore "can I keep answering the useful business questions?" The extension question is "what else can I build on the harness's native capabilities?" Briefs help explore the second question. Actual tool results, not assistant narration, establish whether creation, revision or publication occurred. There are still operational boundaries. The app is loopback-only, the harness is a public preview, inference and retrieval incur costs, and workspace state is not durable application storage. Refreshing the browser or restarting the backend loses local session mappings. Production hosting, retention and broader authorization need separate work. Shell and external business-system writes remain disabled. The Takeaway So, what happened to my Python agents? For this scenario, their hosted wrappers were no longer needed. The business questions stayed; their execution moved into a managed harness, and the remaining application code focused on the experience and its security boundaries. For this practical research scenario, that shifts the work from maintaining Python wrappers and fixed fan-out toward configuring the agent, defining source responsibilities, building the user experience and verifying evidence. Fabric and Foundry IQ fit into that environment directly, while native web research replaces the earlier Web IQ path for the tasks explored here. When I Would Still Use MAF Deterministic Routing and Local Hybrid Execution: MAF allows you to deploy hybrid topologies where small, local models (like MAI-Decision / Jev) handle rapid, deterministic triage and tool-gating, reserving expensive cloud-hosted frontier models for complex reasoning. A managed harness forces all routing into its proprietary cloud runtime, stripping away your ability to optimize latency and compute token economics. Absolute Control over the Orchestration Loop: The Copilot Harness uses an abstracted reasoning engine to decide when to activate a skill or invoke a tool. Native Toolchain Continuity and Hosting: MAF agents operate as standard C# or Python applications. Granular State and Semantic Memory: Managed harnesses provide a generalized shared memory layer. MAF allows you to build custom memory providers (like custom Azure SQL or Cosmos DB schemas) Unrestricted Interoperability Mechanics: While the Copilot Harness can consume Agent-to-Agent (A2A) protocols and Model Context Protocol (MCP) servers, building the actual server logic in MAF gives you the power to manipulate the raw protocol mechanics. If you need to manage custom message headers, inject highly specific tokens for platforms like SAP or ServiceNow, or alter payload structures mid-flight, MAF provides the unabstracted access required to do so. For this demo, the harness was a useful way to move beyond fixed fan-out and put research, connected evidence and artifacts into one managed environment. For another application, keeping its MAF agents may be the better decision. My takeaway is simple: start with a real task, move the responsibilities that fit the harness, and test the boundary instead of assuming the migration works. What's Next This experiment answered one question—now I'm curious about others: How does the harness perform under sustained load? What happens when Fabric semantic models grow to production scale? Can the same approach work for agents that need to write back to business systems? If you're exploring similar migrations, I'd be interested to hear what worked and what didn't. Try the Project The code is available at olivierb123/microsoftiq-copilot-harness-demo. Choose your path: Run the demo locally → Start with the README and setup checklist Explore the agent configuration → See harness/azure.yaml for the Foundry agent definition Understand the data model → Review the synthetic data documentation Compare with the original MAF approach → Check out the original pro-code demo Questions or feedback? Open an issue on the repo or share your own migration experience in the comments below.336Views2likes0CommentsBuilding the next generation of multimodal AI experiences with fal in Microsoft Foundry
Video workloads do not all start with the same input or aim for the same result. A creative team might explore several interpretations of a written brief. A retailer might turn an approved product image into a short motion concept. An application developer might combine image generation and video generation into a workflow that takes an idea from description to frame to scene. Each scenario brings different requirements for visual quality, motion, creative control, turnaround time, and cost. All of this means that a multi-model solution is necessary to realize a full vision. Today, Microsoft and fal are helping developers find the right tools for their workloads. We are excited to announce that Kling, LTX, and MiniMax video models are rolling out to in public preview in Microsoft Foundry through our new partnership with fal. Available as a first-party Azure service, these text-to-video and image-to-video options give creators and developers more ways to utilize video generation with the experiences they want to build. The partnership expands the choices available in Microsoft Foundry while bringing video into the broader platform developers use to build AI applications. The goal is to let developers pick the right model for each step of an experience, then connect those steps into a working application. “fal has seen incredible adoption this year alone with generative media across all industries from advertising to marketing to gaming. Partnering with Microsoft allows both of us to truly enable the wide creative ecosystem and ensure that all AI solutions that lean on generative media, becomes a multi-model one.” - Burkay Gur, Cofounder Bringing fal’s generative media focus to enterprise builders fal is the full stack for generative media, built for creators and developers, with APIs for over 1500 models across image, video, audio, 3D, and generative worlds. As a truly encapsulating platform, fal provides inference, training, and infrastructure to run generative media workloads in production, while also providing innovative post trained models that pushes the boundaries on speed and quality. That specialization matters because generative media is both a creative and an engineering workload. A compelling demo is only the start. Production applications also need to handle real traffic, scale with demand, and balance output quality, latency, and cost. Through this partnership, developers will be able to access a selected set of video models in Microsoft Foundry, served by fal and sourced from the companies that build them. The initial portfolio gives teams a choice of models for different creative requirements instead of relying on one model for every task, with more models planned over time. A portfolio for different video experiences The initial portfolio spans Kling v3 Pro, LTX-2.5 Fast, LTX-2.5 Pro, MiniMax H3 Max, and MiniMax H3 Max Turbo. These models give developers different starting points for generating video from text and animating existing images. A team rapidly exploring creative concepts may prioritize speed and cost, while a team producing fewer, higher-fidelity assets may weigh output quality more heavily. Whether an application starts from text or an existing image, and what it needs in output quality and cost, also shapes the choice. Kling: From a still image to a directed scene Kling v3 Pro combines cinematic visuals and fluid motion with native audio generation and custom element support. Its image-to-video capabilities give developers control over how a starting image becomes a moving scene, a fit for applications that prioritize visual direction and motion. LTX: One model for fast drafts, one for final output For rapid iteration and previews, LTX-2.5 Fast is the speed-optimized option; for high-fidelity final output, LTX-2.5 Pro is the quality-optimized option. Both support text-to-video and image-to-video generation with synchronized audio. MiniMax: Another approach to video generation MiniMax H3 Max and MiniMax H3 Max Turbo are post-trained variants of MiniMax H3 focused on prompt adherence, aesthetics, and inference performance. For developers, they provide additional options to evaluate against an application’s creative brief, quality requirements, and performance needs. “When building with video models, the details matter: how closely the output follows your prompt, how well it preserves your reference images/videos/audios and how quickly you can try another idea. Kling gives developers control over detailed reference to video generation. LTX offers fast iteration and high-quality output through its Fast and Pro variants. H3 Max brings fal's post-training and inference optimizations to improve quality and speed. Having these models available through Microsoft Foundry gives developers more flexibility to choose what works best for the application they are building." - Gökay Aydoğan, Creative Engineer Built for enterprise video workloads Bringing video generation into an enterprise application requires more than choosing the right model. Teams need a platform where they can evaluate different options, build with the models that fit their requirements, and operate those experiences alongside the rest of their applications. The initial portfolio of video models is hosted in the US, and Microsoft Foundry is the first provider to offer a US-hosted Kling 3 endpoint. Key considerations for enterprise teams: Data residency: For organizations with data residency requirements, hosting location is one factor to weigh. Review each model's documented data processing and storage terms to confirm it meets your requirements. Model evaluation: Teams can assess quality, performance, and cost, then connect video generation with the other models and capabilities their applications need. Production readiness: As they move toward production, they can use the evaluation, optimization, governance, security, observability, and infrastructure capabilities supported for their selected offering. Pricing Video generation pricing varies by model and output resolution, giving teams options to balance creative requirements with their application’s budget. The table below shows pricing in USD per minute of generated video for LTX and MiniMax models, making it easier to compare costs across the available options. Model name Deployment type Format Resolution Price per video second (USD) MiniMax H3 Max Global Standard Text to video Image to video Reference to video 480p $0.05 768p $0.08 1080p $0.16 MiniMax H3 Max Turbo Global Standard Text to video Image to video 480p $0.025 768p $0.04 1080p $0.08 LTX-2.5 Fast Global Standard Text to video Image to video 720p $0.09 1080p $0.13 1440p $0.19 4K $0.30 LTX-2.5 Pro Global Standard Text to video Image to video 720p $0.12 1080p $0.17 Kling v3 Pro US Data Zone Text to video Image to video audio off $0.1232 Kling v3 Pro US Data Zone Text to video Image to video audio on $0.1848 Kling v3 Pro US Data Zone Text to video Image to video audio + voice control $0.2156 Pricing will be updated and listed on our Azure pricing page within the coming weeks. Get started Ready to start building? Join us at Ignite for BRK256: Video AI at Scale with fal on Microsoft Foundry. Naomi Moneypenny and Alexander Hughes from Microsoft will be joined by Burkay Gur and Lovis Odin from fal on stage to explore this partnership on Thursday, November 19 at 11:00 AM - 11:45 AM CT. Register today! Explore Kling v3 Pro, LTX-2.5 Fast, LTX-2.5 Pro, MiniMax H3 Max, and MiniMax H3 Max Turbo in Microsoft Foundry and find the model that fits your video experience. Explore the models in Microsoft Foundry Learn more about fal Read the quick start guide to using fal models with ComfyUI here If you are interested in testing out new models, check the Microsoft Foundry Model Releases Repository5.3KViews1like0CommentsImprove production agents with closed-loop evaluations in Microsoft Foundry
A production agent can lose accuracy or cost too much without a single code change. As live traffic grows, new user requests expose gaps in the agent instructions. At the same time, newer models can offer better quality or lower operating cost. Yet editing prompts by hand or swapping models in place risks breaking workflows that already work. The diagram uses v1 and v2 for the active baseline and candidate; Microsoft Foundry assigns the actual version numbers. Prerequisites and integration responsibilities Before implementing this workflow, verify these platform resources and application components: Microsoft Foundry resources: Deploy a hosted agent with the Agent Optimizer SDK. Connect Application Insights, deploy approved models, and grant Microsoft Entra ID permissions. Application team responsibilities: Supply the sandboxed tool test harness, reviewed outcome labels ( ground_truth ), the CI/CD release policy script, and session canary routing ( version_indicator ). How the scheduled loop runs end to end A scheduled GitHub Actions workflow or an Azure Monitor alert starts each cycle across Microsoft Foundry and GitHub Actions. On each run, the workflow reads recent continuous evaluation scores, Application Insights error metrics, and newly approved model deployments. When triage confirms an agent quality drop or a scheduled model evaluation, the workflow samples recent traces into a dataset for human label review. Next, Agent Optimizer searches across instructions, skills, tool descriptions, and models on your training and validation datasets. Your release automation applies the selected candidate and opens a pull request for engineer review. It then gates promotion on a held-out regression suite and updates .agent_configs/baseline/ after promotion. 1. Monitor live traffic with continuous evaluation Safe promotion starts with measuring the active production configuration, which serves as the baseline for every comparison against a proposed candidate. On live traffic, Microsoft Foundry continuous evaluation scores execution traces to track how quality, tool use, latency, and cost change over time: Tool output versus reply ( groundedness ). Checks whether the reply has support in the supplied context, flagging unsupported completion claims. Instructions versus actions ( task_adherence ). Checks whether the agent followed its system prompt rules during the conversation. Query versus tool calls ( tool_call_accuracy and intent_resolution ). Verifies that the agent selected the right tool and arguments for the user's request. Safety and trace rules ( indirect_attack and span checks). Flags injected instructions while span checks detect tool errors and repeated write calls. Triage alerts and set the optimization objective When a quality alert fires, check Application Insights traces and Azure Monitor metrics first. Separate an agent configuration issue from other operational changes before running an optimizer job: Backend or tool outages. Rising tool timeouts or API errors call for infrastructure recovery before you tune the agent. Traffic-mix shifts. A surge in a new request type may require new tools, policy rules, or test coverage before optimization. Evaluator or rubric updates. A changed grader prompt or threshold shifts scores even when agent behavior is unchanged. Confirm that a configuration change can address the issue, or start a run from a scheduled review or new model deployment. Define the objective up front: Optimization objective Primary target Mandatory requirements Improve quality Achieve a target score gain on failing task categories and meet the required quality floor Preserve scores in existing categories, pass all critical safety and business checks, and stay within cost and latency limits Reduce operating cost Lower token and model cost across the agent workload Meet the quality floor, stay within the allowed variation margin, pass all critical checks, and stay within latency limits 2. Turn production traces into isolated, executable datasets Once monitoring identifies what to improve, the next step is building the evaluation datasets for Agent Optimizer and CI/CD. Microsoft Foundry trace dataset generation ( project_client.datasets.begin_create_generation_job ) samples production traces from Application Insights into a versioned dataset registered in your project. When you set max_samples , Microsoft Foundry uses MinHash sampling to filter out repetitive messages and select diverse conversations. Pinning agent_version samples behavior from one known release while redacting private content ( redact_private_content=True ) by default. In a tool-using workflow such as an order refund agent, flagged traces often expose three failure patterns: Unsafe retries after a tool timeout. The agent calls a write tool such as submit_refund , and the call times out with outcome_unknown . The agent repeats the write request before it checks the first request's status. The backend must also prevent duplicate transactions. Premature completion claims on asynchronous work. The agent skips get_refund_status and reports a refund as completed while backend processing remains pending . Prompt injection inside untrusted tool data. The agent obeys injected text inside a customer return note ( untrusted_return_note ) and reports an unverified status. Figure 1. The agent checks the refund status after a timed-out submission. The tool sequence shows how it handles an unknown outcome. Bridge the gap from recorded traces to safe test execution A generated trace dataset captures the conversation history from production. Turning those recorded interactions into safe, repeatable evaluation cases requires verified expected outcomes and a sandboxed test backend. Keep all evaluation runs isolated from live payment APIs and customer databases: Review expected outcomes ( ground_truth ). Inspect each sampled trace and add the reviewed business outcome ( status , order_id , refund_id , and reason ) to ground_truth . Preserve the original captured response as the historical record. Isolate tool execution in a test harness. Bind each evaluation case to controlled tool doubles and a resettable test database or ledger snapshot. Every evaluation run executes fresh tool calls against this sandbox. Keep optimizer data separate from the release regression suite. Split reviewed cases into training and validation datasets ( 2a ) for Agent Optimizer, and reserve a held-out regression suite ( 2b ) for CI/CD release gating. Keep all requests from the same incident in one data split. Figure 2. This example dataset uses synthetic interactions to illustrate the production trace workflow. Reviewers verify expected outcomes and pair each case with isolated test state. 3. Configure eval.yaml and evaluators against fixed requirements With the datasets registered in Microsoft Foundry, define how each run is scored. Use eval.yaml to bind your baseline agent configuration, the training and validation datasets, evaluators, and approved model search space: name: Refund agent configuration search agent: name: refund-release-loop kind: hosted model: gpt-5.4-mini config: .agent_configs/baseline/metadata.yaml dataset: name: refund-loop-training version: "1" validation_dataset: name: refund-loop-validation version: "1" evaluators: - builtin.task_adherence - name: refund-state version: "3" options: eval_model: gpt-5.4-mini optimization_model: gpt-5.4 max_candidates: 4 optimization_config: model_search_space: - gpt-5.4-mini - gpt-5.4 Keep the held-out regression suite ( refund-loop-release ) in a separate CI/CD evaluation configuration so Agent Optimizer cannot access or overfit those cases during search. Keep release criteria independent of candidate instructions Prompt evaluators such as builtin.task_adherence help spot instruction drift, while Agent Optimizer rewrites instructions.md and SKILL.md to improve performance. Grade every candidate against fixed business requirements kept outside the editable agent configuration: Evaluator type What it checks on each fresh sandboxed run How it protects the release gate Semantic and rubric evaluators Grades the reply against fixed task-level criteria , mapped grounding context, and ground truth Allows better phrasing or model changes while enforcing business rules and grounded claims Custom code evaluator Inspects the tool call sequence, structured JSON output, and resulting sandbox ledger state Blocks duplicate write calls, skipped status checks, and mismatched transaction states To use builtin.groundedness on tool-supported claims, pass serialized tool outputs as grounding context and map that field in your evaluator inputs. Before testing, set your quality floor, target gain or cost savings, and allowed regression margin. Semantic evaluation permits harmless wording changes. The regression margin defines the largest acceptable quality loss. Assess sampling uncertainty separately. Microsoft Foundry custom code evaluators inspect the candidate's tool events and sandbox state snapshot directly. The following illustrative check blocks duplicate refund submissions and status claims that disagree with the verification tool or ledger: # Illustrative custom code evaluator logic submissions = [e for e in events if e["tool"] == "submit_refund"] if len(submissions) > 1: failures.append("repeated_submission") verifications = [e["result"] for e in events if e["tool"] == "get_refund_status"] claimed = output.get("status") if claimed in {"completed", "pending", "failed"}: if not verifications or verifications[-1].get("status") != claimed: failures.append("unverified_claim") if not ledger or ledger[-1].get("status") != claimed: failures.append("claim_state_mismatch") Because custom code evaluators run in an isolated environment without network access, the evaluation harness attaches the resulting sandbox state snapshot to each evaluation item. 4. Run Agent Optimizer on your evaluation dataset Once eval.yaml is ready, run Agent Optimizer from your agent project root: azd ai agent optimize The command detects eval.yaml automatically. Agent Optimizer is in preview; see the optimization quickstart for supported flags and regions. During the job, it scores .agent_configs/baseline/ , proposes up to max_candidates revisions from training tasks, and ranks candidates on validation_dataset : Optimization target Baseline input What Agent Optimizer changes Instruction tuning instructions.md Rewrites the system prompt with explicit constraints, step ordering, and output format rules Skill improvement skills/<name>/SKILL.md Refines domain skill descriptions, bodies, and activation criteria while keeping their purpose intact Tool optimization tools.json Clarifies function and parameter descriptions without changing the executable tool schema Model selection model_search_space in eval.yaml Tests candidate model deployments from the approved list in your Foundry project Load the selected instructions, skills, model, and tool descriptions at runtime with load_config() . Keep application logic, authorization, and tool contracts outside the optimization scope. For an illustrative submit_refund timeout, a candidate can update the refund skill and tool descriptions to check get_refund_status first. The configuration diff shows the intended changes, while evaluation provides evidence that the agent follows your business rules. Figure 3. The comparison ranks four candidates on validation runs. The selected candidate refines the skill and tool descriptions. 5. Gate the candidate in GitHub Actions CI/CD After you select a candidate in Agent Optimizer, apply its configuration to .agent_configs/ and open a pull request for review: azd ai agent optimize apply --candidate <candidate-id> In GitHub Actions, the workflow signs in to Azure with Microsoft Entra ID. It then runs a four-step release gate while live traffic stays on v1 : Deploy the candidate version ( v2 ). Run azd deploy --no-prompt to register the candidate container without changing the endpoint selector ( v1 ). Evaluate both pinned versions on the held-out suite ( 2b ). Run cloud evaluations on refund-loop-release for v1 and v2 (or azd ai agent eval run per version). Retrieve complete grades and resource metrics. Collect all evaluator scores, latency metrics, and model token counts from both completed runs. Enforce the release policy. Compare v2 against v1 using the objective from Step 1, and exit non-zero when the policy fails: Release check result Outcome on the held-out regression suite CI/CD gate action Quality objective pass Candidate clears the quality floor, meets the required quality gain after accounting for sampling uncertainty, passes all critical checks, and stays within resource limits Pass the release gate and permit canary verification Cost objective pass Candidate reduces measured model spend (using model token pricing), clears the quality floor, stays within the allowed regression margin, and passes all critical checks Pass the release gate and permit canary verification Insufficient gain or savings Candidate misses the target quality gain or measured cost reduction Keep the baseline active Regression or critical failure Candidate drops below the quality floor or triggers repeated_submission , unverified_claim , or claim_state_mismatch Fail the workflow and block release Separate evaluation job health from agent task performance. Every test case must return a complete evaluation record and all required evaluator scores; missing records or failed evaluator jobs always block release. Score recorded agent execution errors (such as a malformed response) as task failures. This lets a candidate repair a broken baseline case, such as an illustrative outcome_unknown timeout retry. Fixing one failure mode alone does not satisfy every release requirement. In Figure 4, the gate blocks promotion when the run shows category regressions, excess token usage, or incomplete baseline evidence. Figure 4. GitHub Actions deploys the candidate version, blocks release when the held-out regression suite fails, and skips endpoint promotion. 6. Run a canary release, promote, and prepare the next cycle Passing the held-out regression suite provides evidence that the candidate meets the requirements covered by those tests. Verify live traffic in four steps before full promotion: Smoke-test the deployed version. Send synthetic sessions with version_indicator set to v2 to verify container startup, credentials, and tool connections. Route canary sessions against a live control. Microsoft Foundry hosted endpoints send 100 percent of default traffic to v1 . Set your minimum sample count, observation window, and thresholds before starting. Route a slice of new sessions ( 5% , then 25% ) to v2 via version_indicator alongside matching v1 control traffic. Wait for settled outcomes. Hold the canary stage until v1 and v2 collect enough continuous evaluation scores, Azure Monitor metrics, and settled refund outcomes. Promote, hold, or roll back: Canary and post-deployment evidence Release action Candidate meets quality, latency, and settled outcome thresholds against live v1 control traffic during the canary window Promote v2 by switching the Microsoft Foundry endpoint version selector to v2 for all new default sessions Sample size is still below the minimum count or asynchronous transactions are still settling Hold the current canary percentage and continue collecting evidence Candidate quality falls below the required floor, exceeds the allowed regression margin against control, or triggers critical tool errors or Azure Monitor alerts Roll back routing for new sessions to v1 , stop or drain active v2 sessions, and reconcile affected records Plan for both session routing and backend state during a rollback. During a canary, clearing version_indicator sends new sessions to the default version ( v1 ). After promotion, rolling back requires switching the Microsoft Foundry endpoint selector back to v1 . Existing v2 sessions continue on v2 until you stop them explicitly (for example, with project_client.agents.stop_session ) or let them drain. For transactions already submitted during the canary window, use backend idempotency keys and compensating reversals to keep customer records consistent. Reset the baseline and refresh held-out cases for the next loop After promoting v2 to 100 percent of default traffic, your release automation records its configuration in .agent_configs/baseline/ as the new comparison baseline. Keep v1 available until all in-flight v1 sessions finish. To keep the held-out regression suite representative of current traffic, periodically add fresh reviewed production traces to both your training datasets and your release suite. This reduces the risk of repeated candidate selection gradually overfitting a single fixed test set. Next steps: Implement continuous evaluation and CI/CD gating Use these paths to set up the workflow in Microsoft Foundry: Enable continuous evaluation and trace datasets. Configure production monitoring on your deployed agent and convert production conversations into trace datasets. Create your evaluation suite and custom evaluators. Prepare eval.yaml and datasets for Agent Optimizer, add custom evaluators, and run cloud evaluations. Optimize and automate CI/CD releases. Follow the Agent Optimizer overview, the optimization quickstart, and the hosted agent CI/CD guide to gate deployments in GitHub Actions.521Views0likes0Comments