Blog Post

Microsoft Foundry Blog
9 MIN READ

Observe and improve agents built outside Foundry

katelynrothney's avatar
Nov 18, 2026

Built your agent somewhere else? Foundry can still help you measure and improve it.

External agents in Microsoft Foundry are now generally available. If your agent runs outside Foundry and sends OpenTelemetry traces to Application Insights, you can register it in Foundry and use its traces to observe and evaluate it. You don't have to move or rebuild the agent. External agent registration, the agent trace view, Insights, and continuous evaluation of production traces are generally available. Other capabilities in this post are in preview and are labeled where they appear.

 

Before launch, you test the scenarios you expect. Once your agent is live, users bring scenarios you didn't plan for.

Take Contoso Travel, a fictional company that runs its own trip-planning agent outside Foundry.* 

Before launch, the agent passed every test the team wrote. After launch, users asked about carry-on bag rules and planned trips with three or four cities. Neither was in the test set.

Tests can only cover the scenarios your team thought of. Production shows you the ones it didn't.

This post follows Contoso as they connect the agent to Foundry, find problems in production, capture human expert judgment, and turn it into tests they can run again.

Connect the agent you already have

Connecting an external agent doesn't mean migrating it. Your agent keeps running where it is, on its existing endpoint. You send its traces to Application Insights and register it in Foundry, and Foundry reads the traces from there.

Before you begin, you need:

  • An Application Insights resource connected to your Foundry project.
  • An agent that emits OpenTelemetry traces following the GenAI semantic conventions, with gen_ai.agent.id and microsoft.foundry.project.id attributes on its spans. Agents built with frameworks other than the one shown here can also connect if they emit compatible traces.
  • The Foundry User role on the project, and Monitoring Reader on the Application Insights resource to view traces.

For the full list of prerequisites, see Register external agents for observability and evaluation.

Contoso's agent is built with Microsoft Agent Framework in Python and runs on Azure Kubernetes Service. Connecting it takes three steps.

Step 1: Send traces to Application Insights

import os

from microsoft.opentelemetry import use_microsoft_opentelemetry

from opentelemetry import trace

from opentelemetry.sdk.trace import SpanProcessor



use_microsoft_opentelemetry(

    enable_azure_monitor=True,

    azure_monitor_connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"],

    enable_sensitive_data=True,

    instrumentation_options={

        "agent-framework": {

            "enabled": True,

            # Emitted as gen_ai.agent.id. Register the agent with the same ID in step 2.

            "agent_id": "contoso-trip-planner",

            "agent_name": "contoso-trip-planner",

        },

    },

)



# Tag every span with the Foundry project, so the agent's Traces tab can find it.

class FoundryProject(SpanProcessor):

    def on_start(self, span, parent_context=None):

        span.set_attribute("microsoft.foundry.project.id", os.environ["FOUNDRY_PROJECT_RESOURCE_ID"])



trace.get_tracer_provider().add_span_processor(FoundryProject())

 

This uses the Microsoft OpenTelemetry distro. Run it once at startup, then create and run the agent as usual. Set FOUNDRY_PROJECT_RESOURCE_ID to your project's resource ID, in the form /subscriptions/<subscription>/resourceGroups/<group>/providers/Microsoft.CognitiveServices/accounts/<account>/projects/<project>.

Decide early how you'll handle message content. enable_sensitive_data=True records prompts and responses in your traces. Evaluators that score what the agent said, such as quality and rubric evaluators, need that content. The tracing docs recommend turning content recording off in production unless you need it. Before you turn it on in production, decide what content you're allowed to capture, who can see it, how long you keep it, and whether to redact it first. For the full set of considerations, see Restrict access to sensitive content in Microsoft Foundry traces and Tracing and data handling.

Step 2: Register the agent in Foundry

In the Foundry portal, go to Build > Agents > New agent > Link external agent, then enter the agent name and its OpenTelemetry ID.

Or use the SDK (azure-ai-projects 2.11.0 or later):

import os

from azure.ai.projects import AIProjectClient

from azure.ai.projects.models import ExternalAgentDefinition

from azure.identity import DefaultAzureCredential



project = AIProjectClient(

    endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],

    credential=DefaultAzureCredential(),

)



project.agents.create_version(

    agent_name="contoso-trip-planner",

    definition=ExternalAgentDefinition(otel_agent_id="contoso-trip-planner"),

)

Step 3: Check that traces arrive

  • Send the agent a few requests, then open the agent's Traces tab in Foundry. Traces usually appear within a few minutes.
  • If traces don't show up, see Troubleshoot missing traces.

That's the setup. The rest of this post builds on those traces, and you can adopt each part on its own.

Find problems in production

With traces flowing, the next step is finding what's going wrong. Once an agent is live, there are too many conversations to read one by one, and bad answers often don't show up as errors in logs.

Start with the trace view. Filter by time range, status, or model to narrow down the traces you want to review, then open a run. The trace view shows each run step by step, based on what your traces record, such as model calls, tool calls with their inputs and outputs, and messages. That context can help you investigate where a run went wrong, rather than seeing only the final result.

For a deeper look across many traces, use Insights. You can run it on demand or periodically. It analyzes the agent's recent traces with a GPT model you choose, then groups recurring issues and shows each one with its severity, the linked traces, and a proposed next step. Foundry doesn't have access to an external agent's code, so for external agents that proposal is general guidance for you to apply in your own code. Changing an insight's status records your decision. It doesn't change your agent.

Contoso's Insights run flagged two issues. First, some answers about carry-on bags gave a size limit that was out of date. The traces showed that the agent answered from its instructions and never called policy_lookup, the tool that returns the current policy. Second, on multi-city trips the agent searched legs in an inconsistent order and repeated searches it had already done. Instead of reading traces one by one, the team used these issues to decide which traces to look at closely.

Turn human expert review into reusable tests

Insights shows you where to look. Deciding whether an answer is actually wrong often takes someone who knows the business. An LLM judge can help score whether an answer is relevant and grounded, but a general-purpose LLM judge may not know that your airline partner changed its bag rules last month. Your domain experts do, but their judgment often ends up in a chat thread or a support ticket and doesn't make it back into testing.

In Foundry, a human expert's review can become a test you re-run. Foundry calls this human in the loop, and it works in three steps. Annotation and task-level rubrics are in preview. Dataset generation from traces and shared quality rubrics are generally available.

  1. Annotate the trace (preview). Reviewers can rate a run, add tags, leave a comment, and write the answer they expected. Contoso's support lead reviewed the carry-on traces from the insight, marked them as incorrect, and wrote the expected answer with the current limit. The support lead did the same for a few multi-city trips.

One useful practice, also recommended in the annotation docs, is to annotate good runs as well as bad ones, and to say in the comment why an answer is wrong. A score alone doesn't record the reason.

  1. Turn annotations into test cases. Foundry can generate dataset rows from annotated traces. A generated row keeps the user's question and the reviewer's expected answer, and Foundry can add a task-level rubric (preview): pass/fail criteria for that specific case. For a carry-on question, one criterion checks that the agent calls policy_lookup before giving a bag rule. The team reviewed the rows, edited one that was too strict, and added them to their golden dataset: a small, reviewed set of test cases the team trusts and re-runs to compare agent versions.
  2. Define shared quality criteria. Task-level rubrics check specific questions. A shared rubric applies the same criteria across an evaluation. Foundry can generate one from a description of the agent's job that you write, plus its production traces, and the team edits it to match their standards. Contoso added a criterion that policy answers must come from policy_lookup, so the carry-on lesson applies to fees and seat rules too. When they regenerate the rubric later, Foundry keeps the dimensions they edited and the version history.

As a result, the support lead's judgment is captured in evaluation artifacts the team can reuse, and the tests come from questions users actually asked.

Measure the same way before and after every change

With a golden dataset and a shared rubric, Contoso can measure the agent the same way every time. That matters because quality can change after launch, for example when a tool's data changes or someone edits a prompt. To know whether a change helped, you need to measure the same way before and after.

You can also evaluate the agent continuously. Foundry scores a sample of its production traces at an interval you set, such as every hour. Intelligent sampling picks a varied set of traces rather than a random one. Use the shared rubric, built-in evaluators, or your own custom evaluators. The Agent Monitoring Dashboard shows the scores over time, and you can set an alert in Azure Monitor for when a score drops. Low scores are also a good place to look for the next traces to annotate.

With the problems identified, the team had two ways to improve the agent.

  • Fix it yourself. For carry-on bags, the cause was clear. The team updated the instructions to always call policy_lookup for bag and fee questions, then re-ran the golden dataset. In this example, the carry-on rows passed and the other rows still passed.
  • Use agent optimizer. Agent optimizer can be used on external agents too. Multi-city trips were harder to fix by hand, so the team tried skill optimization (preview), which learns from past traces. It analyzes those traces, along with any annotations and evaluation results, and proposes a new or updated skill: a versioned folder of instructions, with a SKILL.md file and optional scripts and reference files, that an agent can load when it needs them. The team ran it on recent multi-city traces with a focus instruction: improve multi-city planning. It returned findings and a proposed skill version. For more on how it works, see [Improve production agents with agent optimizer in Microsoft Foundry].

A successful run means the run finished. It doesn't mean the proposal is better. The team reviewed the findings and the proposed skill, and decided to try it. Because the agent runs outside Foundry, they downloaded the skill into the agent's project, then re-ran the golden dataset to compare. In this example, the multi-city rows passed and the other rows still passed, so they shipped it.

Either way, the check is the same: run the same golden dataset before and after, and compare.

From flagged issue to shipped fix

Each piece works on its own, and they also connect into one loop. Here's the carry-on issue from start to finish:

  1. Detect. The Insights run flags outdated carry-on answers.
  2. Review. The support lead marks the traces incorrect and writes the expected answer.
  3. Test. The annotated traces become rows in the golden dataset.
  4. Fix and re-run. The team updates the instructions and re-runs the golden cont.
  5. Ship and monitor. Foundry continuously evaluates a sample of production traffic.

 

 

Both loops start from the same production traces. Steps 1 to 4 fix the issue in front of you. Over time, the team adds reviewed cases to the golden dataset and refines the shared rubric, so later evaluations include what they learned earlier.

Pre-launch testing still matters. Production can add scenarios those tests missed, including unexpected requests or misuse, and expert review helps you decide which ones should become test cases.

Start where you are

You don't need everything at once. Start at the top and go as far as you need:

Stage

What to do

Connect

Link your agent and check its traces

Learn

Filter and review traces, run Insights, and annotate traces that show the problems your team cares about

Capture

Build a golden dataset and define evaluation criteria

Measure

Continuously evaluate a sample of production traces

Improve

Test each proposed change against the same golden dataset

 

Get started

Reference

Preview features are subject to the Supplemental Terms of Use for Microsoft Azure Previews.

*Contoso Travel and all data and results in this post are fictional and for illustration only.

Updated Nov 18, 2026
Version 1.0
No CommentsBe the first to comment