Blog Post

Microsoft Foundry Blog
8 MIN READ

External agents in Microsoft Foundry are now generally available

aprilk's avatar
aprilk
Icon for Microsoft rankMicrosoft
Nov 18, 2026

Built your agent somewhere else? Foundry can still help you measure and improve it.

External agents in Microsoft Foundry are now generally available. If your agent runs outside Foundry and sends OpenTelemetry traces to Application Insights, you can register it in Foundry and use those traces to observe and evaluate it, without moving or rebuilding the agent. Registration, the agent trace view, Agent Insights, and continuous evaluation of production traces are generally available. Other capabilities in this post are in preview and are labeled where they appear.

Why use production traces? Because they show you what your tests missed. Take Contoso Travel, a fictional company that runs its own trip-planning agent outside Foundry.* Before launch, the agent passed every test the team wrote. After launch, users asked about carry-on bag rules and planned trips with three or four cities. Neither was in the test set. That's normal: tests cover the scenarios your team thought of, and real users bring the rest.

This post follows Contoso as they connect the agent to Foundry, find problems in production, turn expert review into tests, and use those tests to measure and improve the agent.

Connect the agent you already have

Your agent keeps running where it is, on its existing endpoint. You send its traces to Application Insights, register it in Foundry, and Foundry reads the traces from there.

Before you begin, you need:

  • An Application Insights resource connected to your Foundry project.
  • An agent that emits OpenTelemetry traces following the GenAI semantic conventions, with a gen_ai.agent.id attribute on its invoke_agent span. Agents built with frameworks other than the one shown here can also connect if they emit compatible traces.
  • The Foundry User role on the project, and Monitoring Reader on the Application Insights resource to view traces.

For the full list of prerequisites, see Register external agents for observability and evaluation.

Contoso's agent is built with Microsoft Agent Framework in Python and runs on Azure Kubernetes Service. Connecting it takes three steps.

Step 1: Send traces to Application Insights

import os from agent_framework import Agent from agent_framework.openai import OpenAIChatCompletionClient from azure.identity import DefaultAzureCredential from microsoft.opentelemetry import use_microsoft_opentelemetry # Send traces, including prompts and responses, to Application Insights. use_microsoft_opentelemetry( enable_azure_monitor=True, azure_monitor_connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"], enable_sensitive_data=True, ) client = OpenAIChatCompletionClient( model=os.environ["AZURE_OPENAI_DEPLOYMENT"], azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], credential=DefaultAzureCredential(), ) agent = Agent( client, "You are Contoso Travel's trip-planning assistant...", id="contoso-trip-planner", # Emitted as gen_ai.agent.id. Register the agent with this ID in step 2. name="contoso-trip-planner", tools=[search_flights, search_hotels, policy_lookup], )

This uses the Microsoft OpenTelemetry distro. Call use_microsoft_opentelemetry once at startup, before you create the agent.

Decide early how you'll handle message content. enable_sensitive_data=True records prompts and responses in your traces. Evaluators that score what the agent said, such as quality and rubric evaluators, need that content. The tracing docs recommend turning content recording off in production unless you need it. Before you turn it on in production, decide what content you're allowed to capture, who can see it, how long you keep it, and whether to redact it first. For the full set of considerations, see Restrict access to sensitive content in Microsoft Foundry traces and Tracing and data handling.

Step 2: Register the agent in Foundry

In the Foundry portal, go to Build > Agents > New agent > Link external agent, then enter the agent name and its OpenTelemetry ID.

Or use the SDK (azure-ai-projects 2.11.0 or later):

import os from azure.ai.projects import AIProjectClient from azure.ai.projects.models import ExternalAgentDefinition from azure.identity import DefaultAzureCredential project = AIProjectClient( endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], credential=DefaultAzureCredential(), ) project.agents.create_version( agent_name="contoso-trip-planner", definition=ExternalAgentDefinition(otel_agent_id="contoso-trip-planner"), )

Step 3: Check that traces arrive

Send the agent a few requests, then open the agent's Traces tab in Foundry. Traces usually appear within a few minutes.

If traces don't show up, see Troubleshoot missing traces.

That's the setup. The rest of this post builds on those traces, and you can adopt each part on its own.

Find problems in production

With traces flowing, the next step is finding what's going wrong. Once an agent is live, there are too many conversations to read one by one, and bad answers often don't show up as errors in logs.

Start with the trace view. Filter by time range, status, or token count to narrow down the traces you want to review, then open a run. The trace view shows each run step by step, based on what your traces record, such as model calls, tool calls with their inputs and outputs, and messages. That context can help you investigate where a run went wrong, rather than seeing only the final result.

For a deeper look across many traces, use Agent Insights. You can run it on demand or periodically. It analyzes the agent's recent traces with a model you choose, then groups recurring issues and shows each one with its severity, the linked traces, and a proposed next step. Foundry doesn't have access to an external agent's code, so for external agents that proposal is general guidance for you to apply in your own code. Changing an insight's status records your decision. It doesn't change your agent.

Contoso's Agent Insights run flagged a recurring multi-city issue: the agent searched every pair of cities and then searched the same legs again. Around the same time, the support team saw complaints about carry-on answers. Filtering the traces to bag questions showed the agent quoting a size limit that was out of date. It answered from its instructions and never called policy_lookup, the tool that returns the current policy. Instead of reading traces one by one, the team used these signals to decide which traces to look at closely.

Turn human expert review into reusable tests

Agent Insights shows you where to look. Deciding whether an answer is actually wrong often takes someone who knows the business. An LLM judge can help score whether an answer is relevant and grounded, but a general-purpose LLM judge may not know that your airline partner changed its bag rules last month. Your domain experts do, but their judgment often ends up in a chat thread or a support ticket and doesn't make it back into testing.

In Foundry, a human expert's review can become a test you re-run. Foundry calls this human in the loop. 

Annotate the trace (preview). Reviewers can rate a run, add tags, leave a comment, and write the answer they expected. Contoso's support lead reviewed the carry-on traces, marked them as incorrect, and wrote the expected answer with the current limit. The support lead did the same for a few multi-city trips.

One useful practice, also recommended in the annotation docs, is to annotate good runs as well as bad ones, and to say in the comment why an answer is wrong. A score alone doesn't record the reason.

Turn annotations into test cases. Foundry can generate dataset rows from annotated traces. A generated row keeps the user's question and the reviewer's expected answer, and Foundry can add a task-level rubric (preview): pass/fail criteria for that specific case. For a carry-on question, one criterion checks that the agent calls policy_lookup before giving a bag rule. The team reviewed the rows, edited one that was too strict, and added them to their golden dataset: a small, reviewed set of test cases the team trusts and re-runs to compare agent versions.

Define shared quality criteria. Task-level rubrics check specific scenarios. A shared rubric applies the same criteria across an evaluation. Foundry can generate one from a description of the agent's job that you write, plus its production traces, and the team edits it to match their standards. Contoso added a criterion that policy answers must come from policy_lookup, so the carry-on lesson applies to fees and seat rules too. When they regenerate the rubric later, Foundry keeps the dimensions they edited and the version history.

As a result, the support lead's judgment is captured in evaluation artifacts the team can reuse, and the tests come from questions users actually asked.

Measure and improve

With a golden dataset and a shared rubric, Contoso can measure the agent the same way every time. That matters because quality can change after launch, for example when a tool's data changes or someone edits a prompt. To know whether a change helped, you need to measure the same way before and after.

You can also evaluate the agent continuously. Foundry scores a sample of its production traces at an interval you set, such as every hour. Intelligent sampling picks a varied set of traces rather than a random one. Use the shared rubric, built-in evaluators, or your own custom evaluators. Contoso added a custom evaluator that checks each answer against the current policy, the kind of fact a general-purpose judge wouldn't know. Each scheduled run records its scores, so you can track quality over time and set an alert in Azure Monitor when a score falls. Low scores are also a good place to look for the next traces to annotate.

With the problems identified, the team had two ways to improve the agent.

Fix it yourself. For carry-on bags, the cause was clear. The team updated the instructions to always call policy_lookup for bag and fee questions, then re-ran the golden dataset. In this example, the carry-on rows passed and the other rows still passed.

Learn from past traces with skill optimization (preview). Multi-city trips were harder to fix by hand, so the team tried skill optimization. It analyzes past traces, along with any annotations and evaluation results, and proposes a new or updated skill: a versioned folder of instructions, with a SKILL.md file and optional scripts and reference files, that an agent can load when it needs them. The team ran it on recent traces with a focus on multi-city planning. It returned findings and a proposed skill.

A successful run means the run finished. It doesn't mean the proposal is better. The team reviewed the findings and the proposed skill before trying it. Because the agent runs outside Foundry, they downloaded the skill into the agent's project, then re-ran the golden dataset to compare it with the current version. A skill ships only if the comparison shows it helps.

Either way, the check is the same: run the same golden dataset before and after and compare.

Put it together

Each piece works on its own, and they also connect into one loop. Here's the carry-on issue from start to finish:

  1. Detect: support complaints and a trace filter surface outdated carry-on answers.
  2. Review: the support lead marks the traces incorrect and writes the expected answer.
  3. Test: the annotated traces become rows in the golden dataset.
  4. Fix and re-run: the team updates the instructions and re-runs the golden dataset.
  5. Ship and monitor: Foundry continuously evaluates a sample of production traffic.

Steps 1 to 4 fix the issue in front of you. Over time, the team adds reviewed cases to the golden dataset and refines the shared rubric, so later evaluations include what they learned earlier.

Pre-launch testing still matters. Production can add scenarios those tests missed, including unexpected requests or misuse, and expert review helps you decide which ones should become test cases.

Start where you are

You don't need everything at once. Start at the top and go as far as you need:

StageWhat to do
ConnectLink your agent and check its traces
LearnFilter and review traces, run Agent Insights, and annotate traces that show the problems your team cares about
CaptureBuild a golden dataset and define evaluation criteria
MeasureContinuously evaluate a sample of production traces
ImproveTest each proposed change against the same golden dataset

Get started

Register your external agent, send it a few requests, and open its Traces tab. To try the full workflow, start from our external agent sample.

Reference

Preview features are subject to the Supplemental Terms of Use for Microsoft Azure Previews.

*Contoso Travel and all data and results in this post are fictional and for illustration only.

Updated Nov 18, 2026
Version 1.0
No CommentsBe the first to comment