Production AI is a moving target. Once you deploy an AI-powered agent, its behavior can shift without a single line of code being changed. As Microsoft's Foundry team states: "The hard part isn't getting the agent to work it's keeping it working". Microsoft addressed this challenge by making Evaluations, Monitoring, and Tracing in Microsoft Foundry generally available through Foundry Control Plane, deeply integrated with Azure Monitor so AI agent observability lives on the same operational plane as existing infrastructure telemetry.
This article, Part 1 of a two-part series covers why AI monitoring demands new approaches, what observability capabilities Microsoft Foundry provides, and how these fit into an enterprise's operational, security, and cost governance toolkit. Part 2 will follow with step-by-step configuration guidance.
Portal note: This article focuses exclusively on the Foundry (new) portal experience. Ensure the "New Foundry" toggle is enabled in your portal banner to confirm you are using the current experience.
Why Continuous AI Monitoring Is Critical
Traditional application performance monitoring (APM) - uptime, CPU/memory, request latency, error rates, remains necessary for AI services but is nowhere near sufficient. AI systems can fail in ways classic web apps never would: returning subtly wrong answers, leaking sensitive data in responses, or gradually degrading in quality over time. These failures are invisible to uptime metrics.
An AI system's behavior is a function of forces that change independently of your code
|
Force |
What Changes |
Impact |
|
Foundation model updates |
Providers ship new model versions continuously |
Output style, reasoning patterns, and edge-case handling can shift without any action on your part |
|
Prompt changes |
Team adjusts system or user prompts |
Can have nonlinear effects downstream, especially in multi-step agentic flows |
|
Retrieval pipeline drift |
Document indexes, databases, or knowledge bases evolve |
The context an agent sees at inference time today may differ from what it saw during testing. Stale or shifted data reduces accuracy |
|
Real-world query distribution |
Users submit queries never anticipated in test datasets |
Long-tail inputs surface failure modes invisible during development |
Most evaluation workflows are designed around a pre-deployment gate: build a test dataset, run evaluations, review scores, ship. That approach has real value, but it has a hard ceiling. The implication: evaluation must be continuous, not episodic. Quality signals are needed at development time, at every CI/CD commit, and continuously against live production traffic, all using the same evaluator definitions so results are comparable across environments.
Tradeoff to consider: Continuous evaluation of production traffic incurs additional model invocations (and thus cost) for the evaluator models. Architects must balance the sampling rate against the cost of missed regressions. Monitoring 100% of traffic may be cost-prohibitive, but sampling at too low a rate risks missing transient quality drops. A practical starting approach: configure a modest sampling rate (e.g., 5–10% of production queries) and adjust based on observed variance and budget.
Foundry's Three Pillars of AI Observability
Microsoft Foundry provides three core capabilities that work together across the full AI application lifecycle: Evaluation, Monitoring, and Tracing. All are deeply integrated with Azure Monitor, and reached general availability in March 2026.
Pillar 1: Continuous Evaluation "How good are my AI's outputs?"
Built-in Evaluators (GA): Foundry ships with standardized evaluators covering the most critical quality and safety dimensions for production agent systems:
|
Evaluator |
What It Measures |
Why It Matters |
|
Coherence & Relevance |
Whether responses are internally consistent and on-topic relative to the input |
Table-stakes signals for any conversational or task-completion agent |
|
Groundedness |
Whether the model's output is actually supported by retrieved context (not hallucinated from parametric memory) |
Leading indicator of hallucination risk in production; often invisible to human reviewers at scale |
|
Retrieval Quality |
Whether the retrieval step surfaced relevant context independently of generation quality |
Isolates root cause: is the model ignoring good context, or is the retrieval pipeline failing? |
|
Safety & Policy Alignment |
Whether outputs meet deployment policy requirements. content safety, topic restrictions, response format compliance |
Ensures deployed agents respect organizational and regulatory boundaries |
Groundedness deserves particular attention for RAG-based architectures. Groundedness failures can originate in two places: the model may be ignoring good context, or the retrieval pipeline may not be surfacing relevant context in the first place. By splitting groundedness and retrieval quality into separate signals, Foundry makes root-cause analysis far more tractable.
Evaluators run at three lifecycle stages:
- Local development - run evaluations inline as you iterate on prompts, retrieval config, or orchestration logic
- CI/CD pipelines - gate every commit against quality baselines; catch regressions before they reach production
- Production traffic monitoring - continuously evaluate sampled live traffic and surface trends over time
Because the evaluators are identical across all three contexts, a score in CI means the same thing as a score in production monitoring.
Custom Evaluators (Public Preview): Production agents often need to satisfy criteria specific to a domain, regulatory environment, or internal standard. Foundry supports two types of custom evaluators:
- LLM-as-a-Judge evaluators: Configure a prompt and grading rubric (e.g., a 1–5 scale with criteria for each level), then use a language model to apply that rubric at scale. This approach suits quality dimensions requiring reasoning or contextual judgment, whether a response appropriately acknowledges uncertainty, matches brand tone, or meets clinical documentation standards.
- Code-based evaluators: Python functions implementing any evaluation logic expressible programmatically. regex matching, schema validation, business rule checks, compliance assertions, or calls to external systems.
Custom and built-in evaluators compose naturally, running against the same traffic, producing results in the same schema, feeding into the same dashboards and alert rules.
Additional monitoring settings in the Agent Monitor tab:
|
Setting |
Status |
Purpose |
|
Continuous evaluation |
GA |
Runs evaluations on sampled agent responses. Configure evaluators and sample rate |
|
Scheduled evaluations |
Preview |
Runs evaluations on a schedule to validate performance against benchmarks |
|
Red team scans |
Preview |
Runs adversarial tests to detect risks such as data leakage or prohibited actions |
|
Alerts |
Preview |
Detects performance anomalies, evaluation failures, and security risks; configurable for latency, token usage, evaluation scores, or red team findings |
These settings are accessible via the gear icon on the Monitor tab for a given agent. To enable continuous evaluation rules, the project's managed identity must be assigned the Azure AI User role.
Pillar 2: Integrated Monitoring & Azure Monitor "Is my AI service performing well?"
All observability data produced by Foundry - evaluation results, traces, latency, token usage, and quality metrics are published directly to Azure Monitor. This deep integration unlocks four capabilities that siloed AI monitoring tools cannot match:
|
Capability |
Description |
|
Cross-stack correlation |
When a groundedness score drops, determine whether the cause is a model update, a retrieval pipeline issue, or an infrastructure problem. All from the same Application Insights workspace, in minutes rather than hours |
|
Unified alerting |
Configure Azure Monitor alert rules on any evaluation metric - trigger a PagerDuty incident when groundedness drops below threshold, send a Teams notification when safety violations spike, or create automated runbook responses when retrieval quality degrades |
|
Enterprise governance by default |
Azure Monitor's RBAC, retention policies, diagnostic settings, and audit logging apply automatically to all AI observability data. Organizations inherit the governance framework already built and approved |
|
Grafana integration |
For teams using Azure Managed Grafana, evaluation metrics flow into existing dashboards alongside other operational metrics. A single pane of glass for application health, infrastructure performance, and AI agent quality |
The Agent Monitoring Dashboard in the Foundry portal provides an AI-native view out of the box. Evaluation metric trends, safety threshold status, quality score distributions, and latency breakdowns. Everything in that dashboard is backed by Azure Monitor data, so SRE teams can always drill deeper.
How to access agent-level monitoring in the new portal: Navigate to the Build page using the top navigation, select the agent you want to examine, then select the Monitor tab to view operational, evaluation, and red-teaming data. The dashboard consists of summary cards at the top for high-level metrics and charts and graphs below for granular details, reflecting data for the selected time range.
Dashboard metric definitions
- Token usage: Token counts for agent traffic. High token usage might indicate verbose prompts or responses that could benefit from optimization.
- Latency: Response time for agent runs. Latency above 10 seconds might indicate model throttling, complex tool calls, or network issues.
- Run success rate: The percentage of runs that complete successfully. A rate below 95% warrants investigation into failed runs.
- Evaluation metrics: Scores produced by evaluators on sampled agent outputs. Scores vary by evaluator; review individual evaluator documentation for interpretation guidance.
- Red teaming results: Outcomes from scheduled red team scans, if enabled. Failed scans indicate potential security risks requiring remediation.
Monitoring data is stored in the connected Application Insights resource. Retention and billing follow your Application Insights configuration. The Agent Monitoring Dashboard reads telemetry from this connected resource. If Application Insights is not connected, no data appears.
Note: While the underlying Evaluations and Monitoring capabilities are GA, the portal's "View agent metrics" experience is currently labeled (preview). This means the underlying data pipeline and Azure Monitor integration are production-grade, but the portal UI may still evolve.
Fleet-wide visibility via Operate: For organizations managing multiple agents across projects, the Operate section provides subscription-wide oversight via the Overview pane (fleet health, active agents, cost trends, run completion rate, prevented behaviors) and the Assets pane (a unified searchable table of every agent, model, and tool with health scores, cost, alerts, and token usage). From any entry in the Assets table, you can drill directly into the agent's Evaluation or Monitoring tab for pre-deployment and post-deployment insights.
Pillar 3: End-to-End Tracing "What exactly happened during this AI request?"
A quality score tells you something is wrong. A trace tells you exactly where the failure occurred and what the agent actually did.
Foundry provides OpenTelemetry-based distributed tracing that follows each request through the entire agent system: model calls, tool invocations, retrieval steps, orchestration logic, and cross-agent handoffs. Traces capture the full execution path - inputs, outputs, latency at each step, tool call parameters and responses, and token usage.
The key design decision: evaluation results are linked directly to traces When you see a low groundedness score in your monitoring dashboard, you navigate directly to the specific trace that produced, no manual timestamp correlation, no separate trace ID lookup. The connection is made automatically. This is what makes the three pillars a system rather than isolated tools: evaluation identifies the problem, monitoring surfaces the trend, and tracing pinpoints the cause.
Framework auto-collection Foundry auto-collects traces across the frameworks agents are commonly built on:
- Microsoft Agent Framework
- Semantic Kernel
- LangChain and LangGraph
- OpenAI Agents SDK
For custom or less common orchestration frameworks, the Azure Monitor OpenTelemetry Distro provides an instrumentation path.
Cost consideration: Tracing stores telemetry in Azure Monitor Application Insights, which may incur costs based on data volume and retention settings. Monitoring data retention and billing follow your Application Insights configuration. Architects should factor this into capacity planning, particularly for high-throughput agent deployments.
Security, Compliance, and Responsible AI
Monitoring AI systems must extend beyond performance and quality to cover security and governance. Building AI workloads introduces unique security challenges:
|
Risk |
Description |
|
Prompt injection |
Malicious inputs crafted to manipulate model behavior or bypass safety controls |
|
Sensitive data leakage |
Models inadvertently exposing confidential information in responses |
|
Model misuse |
Unauthorized or abusive use of AI capabilities that violates organizational policies |
Microsoft Foundry, Defender for Cloud, and Microsoft Purview address these risks in combination:
|
Layer |
Tool |
Function |
|
Model Layer |
Foundry Guardrails |
Content filters, prompt shields, and abuse detection protect managed inference endpoints |
|
Threat Detection |
Defender for Cloud |
Security posture recommendations identify misconfigurations; threat protection for Foundry Tools detects jailbreak and user input attacks |
|
Data Governance |
Microsoft Purview |
Auditing, sensitive information type (SIT) classification, and DSPM for AI provide visibility and governance over prompt and response data |
|
Third-Party Runtime |
Palo Alto Prisma AIRS + Zenity (GA) |
Runtime security integrations for prompt injection, data leakage, and tool misuse detection |
Navigating Compliance in the New Portal
Foundry Control Plane centralizes compliance, inventory, observability, and security into one role-aware interface. The capabilities are organized into panes accessed by selecting Operate on the upper-right toolbar of the Foundry workspace. From Operate, you can monitor, govern, and optimize every agent, model, and deployment within your subscription.
The Compliance pane lets you define, apply, and continuously monitor guardrails and compliance policies across your AI resources. It provides a unified interface to operationalize responsible AI principles while helping ensure enterprise-grade safety and regulatory alignment. Specific capabilities include:
- Define and enforce protections through deep integrations with Azure Policy, Defender, and Microsoft Purview, ensuring identity, data, and threat safeguards work in concert.
- Apply versioned policies and track assignments to maintain full auditability and traceability across agents and environments.
- Monitor compliance posture in real time, surfacing noncompliant assets and enabling bulk remediation directly from Foundry Control Plane.
Within the Compliance pane, there are four tabs for navigation:
|
Tab |
Access Path |
Outcome |
|
Policies |
Operate > Compliance > Policies |
Review guardrail policies, check compliance, create or edit enforcement rules |
|
Assets |
Operate > Compliance > Assets |
Inspect individual model deployments, view policy violations, jump to remediation |
|
Guardrails |
Operate > Compliance > Guardrails |
Compare guardrail configurations across deployments, spot coverage gaps |
|
Security posture |
Operate > Compliance > Security posture |
Review Defender for Cloud recommendations, manage Microsoft Purview enablement |
Access the compliance workspace by selecting Operate on the toolbar, and then selecting Compliance on the left pane. Use the subscription and project filters to scope your view before switching tabs.
Visibility prerequisites: The compliance workspace is available only in the Foundry (new) portal. If you do not see the Compliance pane or its sub-tabs, verify: (1) the "New Foundry" toggle is enabled in the portal banner, (2) you have the required permissions, and (3) you are using the latest agent versions for full support of compliance features. Some items within the compliance documentation are marked (preview) and are in public preview. Permissions required: viewing compliance status needs no special permissions beyond project access; creating or editing guardrail policies requires Owner or Resource Policy Contributor at the subscription or resource group level; enabling Defender for Cloud requires Security Admin or Owner; configuring Microsoft Purview requires Azure AI Account Owner.
Guardrail policies let central teams mandate minimum guardrail controls (content filtering, abuse monitoring, and other safety measures) for model deployments across a subscription or within a resource group. Noncompliant guardrail policies display a "Violations detected" value in the Policy Compliance column. Selecting a noncompliant policy opens a pane comparing the asset's guardrail settings with the policy requirements; selecting "Fix now" opens the model deployment's guardrail configuration pane where you can adjust settings. Compliance status updates within a few minutes after saving changes.
Defender for Cloud integration: To get security posture recommendations from Defender for Cloud, enable it on your Azure subscription. To get threat protection alerts for jailbreak attacks based on risk detection in Foundry for user input attacks, enable threat protection for Foundry Tools specifically. Jailbreak attacks attempt to bypass AI safety measures by using carefully crafted prompts, and Foundry detects these attack patterns in user input. Recommendations and alerts cover AI workloads built on Foundry's managed inference endpoints. Detections for jailbreak and prompt attacks appear as Defender alerts and can be correlated with Microsoft Purview Audit records of AI interactions to support investigation and governance.
Microsoft Purview integration (preview): When enabled via the Security posture tab (Operate > Compliance > Security posture), Purview accesses, processes, and stores prompt and response data from Foundry apps and agents, including associated metadata. Supported scenarios include: Microsoft Purview Audit, sensitive information type classification, DSPM for AI analytics, Insider Risk Management, Communication Compliance, Data Lifecycle Management, and eDiscovery. Microsoft Purview Data Security Policies require Microsoft Entra ID user-context tokens to attach policies to user interactions. Integration with Microsoft Purview does not yet support network isolation.
Tradeoff: Enabling Defender for Cloud's AI workload protection and Purview's DSPM for AI introduces additional Azure service costs and configuration complexity. For non-regulated workloads with lower risk profiles, teams may start with Foundry's built-in guardrails alone and layer Defender/Purview as the deployment matures.
Cost and Quota Monitoring
Token consumption is the primary cost driver for generative AI workloads. Foundry's Agent Monitoring Dashboard surfaces token usage as a key metric, token counts for agent traffic in the selected time range. High token usage may indicate verbose prompts or responses that could benefit from optimization.
The Operate > Quota pane shows model deployments and how much quota each deployment consumes, providing insights into usage patterns and helping manage resources effectively. By default, the quota view displays only models with active deployments; turning on the Show all toggle reveals the full list of available models and regions, including models not yet deployed, useful for checking capacity before creating a deployment.
The Operate > Overview pane aggregates key operational metrics including cost trends and token usage across your AI fleet, allowing you to drill into anomalies or cost spikes through contextual charts and direct links to inventory, observability, or policy information. Similarly, the Assets pane lets you filter and sort by cost, token usage, and other attributes to quickly locate resource-intensive agents or models.
Foundry Control Plane also supports tracking rate limits, token usage, and cost anomalies to prevent inefficiency or abuse. By combining this in-portal visibility with Azure Monitor's alerting capabilities and Azure Cost Management's budgeting tools, teams can build proactive cost governance rather than reacting to unexpected bills.
Conclusion & What's Next
Microsoft Foundry's integrated observability, Evaluation, Monitoring, and Tracing, all generally available through Foundry Control Plane, combined with deep Azure Monitor integration and the Compliance workspace for guardrail governance, makes it possible to treat AI quality and safety as live operational signals rather than pre-deployment checkboxes.
The broader platform context reinforces why observability matters now: the Foundry Agent Service (GA) provides a production-ready runtime built on the Responses API, with enterprise security, private networking, and full tracing, all manageable through the Foundry Control Plane's fleet-wide visibility, compliance enforcement, and security capabilities. As organizations evolve from isolated copilots to autonomous multi-agent fleets, the need for unified oversight grows.
Prompt Optimizer (currently in public preview) adds a feedback-loop dimension: it analyzes existing prompts and applies structured prompt engineering techniques, clarifying ambiguous instructions, improving formatting for model comprehension, restructuring few-shot examples with paragraph-level explanations for every change it makes. Rather than producing a black-box "optimized" prompt, it shows you exactly what it changed and why. This closes the loop between monitoring insights ("groundedness is low") and prompt improvement ("here's what to change and why").
In Part 2, we get hands-on: connecting Application Insights to a Foundry project, instrumenting agents for tracing, enabling continuous evaluations, setting up Azure Monitor alerts for AI-specific metrics, and implementing guardrail policies. Complete with portal steps and code samples. Stay tuned.