eBPF-Powered Observability Beyond Azure: A Multi-Cloud Perspective with Retina
Kubernetes simplifies container orchestration but introduces observability challenges due to dynamic pod lifecycles and complex inter-service communication. eBPF technology addresses these issues by providing deep system insights and efficient monitoring. The open-source Retina project leverages eBPF for comprehensive, cloud-agnostic network observability across AKS, GKE, and EKS, enhancing troubleshooting and optimization through real-world demo scenarios.Building Production-Ready AI Agents in Microsoft Foundry: 10 Lessons Learned
A Real-World Scenario Imagine a customer support agent that answers invoice questions. During testing, everything works perfectly. But in production: The Finance API occasionally times out. The knowledge base contains outdated information. Tool calls fail during peak traffic. Token consumption rises unexpectedly. The result isn't a broken AI model. It's an unreliable system. This is where Microsoft Foundry becomes critical. Building production-ready agents requires grounding, observability, resiliency, governance, and continuous monitoring. New challenges appear: Challenge Impact Hallucinated responses Reduced trust Missing citations Verification issues Tool failures Broken workflows High token consumption Increased cost Latency spikes Poor user experience Limited monitoring difficult troubleshooting Security concerns Compliance risks The lesson was clear: Production readiness is not about making the agent smarter. It is about making the agent reliable. My Observation While experimenting with AI agents in Microsoft Foundry, I found that model selection was rarely the primary challenge. Most production issues stemmed from grounding quality, tool reliability, observability, and security controls. Addressing these operational concerns often had a greater impact on user trust than changing the underlying model. Production AI Agent Reference Architecture A production-ready AI agent typically includes several components beyond the language model itself. Layer Technology User Interface Web App, Teams, Copilot Agent Runtime Microsoft Foundry Agent Service Knowledge Layer Azure AI Search Foundation Model GPT-4o Tool Integration APIs and Functions Monitoring Azure Monitor Security Managed Identity and Key Vault Request Flow Lesson 1: Start with the Use Case, Not the Model One of the most common mistakes is beginning with model selection. Many teams ask: Which model should I use? Should I choose GPT-4o? Should I use a larger context window? A more important question is: What business problem are we solving? Before evaluating models, define: Users Business goals Success metrics Compliance requirements Operational constraints An enterprise support agent and a financial compliance agent may use the same model but require completely different architectures. Key Takeaway Successful AI projects start with business outcomes, not model benchmarks. Lesson 2: Grounding Matters More Than Prompting Prompt engineering helps. Grounding drives trust. Without access to reliable enterprise data, even advanced models can generate confident but incorrect responses. Grounding Sources Azure AI Search SharePoint documents Internal knowledge bases Structured enterprise data Approved policy repositories Example from azure.ai.projects import AIProjectClient # Configure grounding with Azure AI Search agent = client.agents.create_agent( model="gpt-4o", name="support-agent", tools=[ { "type": "azure_ai_search", "azure_ai_search": { "index_name": "support-kb", "endpoint": "https://your-search.search.windows.net" } } ] ) In production environments, grounding through Azure AI Search helps agents retrieve trusted enterprise information rather than relying solely on model knowledge. This significantly improves response accuracy and trustworthiness. Try it yourself: Configure Azure AI Search as a knowledge source in Microsoft Foundry and compare grounded versus non-grounded responses. Key Takeaway Reliable retrieval is usually more valuable than sophisticated prompting. Lesson 3: Design Tool Usage Carefully AI agents become powerful when they can interact with external tools. Examples include: CRM systems Databases APIs Ticketing systems Business applications However, every tool increase complexity. Ask yourself: When should the agent call the tool? What happens if the tool is unavailable? How should failures be handled? Example Failure Scenario +-----------------------------+ | User asks for invoice status | +-------------+---------------+ | v +-----------------------------+ | Agent calls Finance API | +-------------+---------------+ | v +-----------------------------+ | API timeout detected | +-------------+---------------+ | v +-----------------------------+ | Fallback response returned | +-----------------------------+ Example Failure Scenario: A production-ready agent should gracefully handle external service failures and return a fallback response instead of failing completely. Key Takeaway Design for failure before designing for capability. Lesson 4: Evaluate Before Deployment Many teams test only for response quality. Production agents require broader evaluation. Evaluation Area Why It Matters Accuracy Correct answers Grounding Quality Faithful responses Latency User experience Safety Risk reduction Cost Sustainability Tool Success Rate Reliability Evaluation should become part of every deployment pipeline. Key Takeaway You cannot improve what you do not measure. Lesson 5: Make Observability a First-Class Feature Observability is often neglected until something breaks. Unfortunately, production systems always encounter unexpected behavior. Track metrics such as: Metric Purpose Request Volume Demand tracking Average Latency Performance Token Usage Cost visibility Grounding Success Rate Quality Tool Failure Rate Reliability User Satisfaction Business value Setting Up Tracing in Foundry from azure.ai.projects import AIProjectClient from azure.monitor.opentelemetry import configure_azure_monitor # Configure Azure Monitor for tracing configure_azure_monitor( connection_string="InstrumentationKey=xxx" ) # Run the agent response = agent.run( thread_id=thread.id, instructions="..." ) Tracing provides visibility into how an AI agent processes requests, invokes tools, and generates responses. By integrating Azure Monitor, teams can track latency, identify failed tool calls, analyze token usage, and troubleshoot unexpected agent behavior. This observability is essential for operating AI agents reliably in production environments. Try it yourself: Enabling tracing and monitoring for a Microsoft Foundry agent using Azure Monitor. Example Dashboard +-----------------------------+ | Production Monitoring Dashboard | +-----------------------------+ | Requests Today | 5,120 | | Average Latency | 3.2s | | Grounding Success | 97% | | Tool Failure Rate | 1% | | Average Tokens | 2,800 | +-----------------------------+ Monitoring production metrics helps teams understand agent performance, reliability, and cost efficiency. Key indicators such as latency, grounding success rate, tool failure rate, and token consumption provide valuable insights into the operational health of an AI agent and enable proactive troubleshooting before users are impacted. Key Takeaway What you can't observe, you can't effectively operate or improve. Tracing is a critical capability for maintaining production-ready AI agents. Lesson 6: Monitor Token Consumption Token usage directly impacts cost. A highly successful agent can quickly become expensive if token growth is unmanaged. Common optimization techniques: Optimization Benefit Prompt Compression Lower token usage Response Caching Reduced model calls RAG Filtering Focused context Context Trimming Smaller requests Model Selection Cost control Key Takeaway Cost optimization should be planned from day one. Lesson 7: Build Security into the Design Security should never be an afterthought. Enterprise AI systems must enforce the same security boundaries as traditional applications. Recommended Controls Control Purpose Managed Identity Secure authentication Azure Key Vault Secret management RBAC Authorization Audit Logs Compliance Content Filtering Safety Private Endpoints Network Security Guiding Principle An AI agent should never access data beyond a user's permissions. Key Takeaway Security is a design requirement, not a deployment task. Lesson 8: Expect Tool Failures External dependencies inevitably fail. Production-ready agents should anticipate: API downtime Authentication failures Network interruptions Rate limiting Unexpected responses Recommended Strategy Key Takeaway async def call_tool_with_resilience(tool_name, params): try: result = await tool_client.execute( tool_name, params, timeout=5.0 ) return result except TimeoutError: return await cache.get_fallback( tool_name, params ) Example Outcome: Introducing retry and fallback logic can significantly improve reliability. By handling temporary API failures gracefully and returning cached responses, when necessary, agents can reduce user-facing errors and provide a more consistent experience. Lesson 9: Evaluate the Process, Not Just the Output A correct answer does not always mean the process was correct. In production environments, teams should evaluate not only the final response but also the steps the agent took to generate that response. An answer may appear accurate even when it was based on incorrect retrieval results, unnecessary tool calls, or incomplete citations. Review: Retrieval quality Tool execution Citation accuracy Security compliance Reasoning path User Query | v Retrieval | v Tool Execution | v Reasoning | v Response For example, an agent might generate the correct answer by chance, even though it retrieved irrelevant documents or used an inefficient workflow. Without evaluating the process, these hidden issues may go unnoticed until they affect reliability, compliance, or user trust. Key Takeaway Inspect the entire workflow, not just the final answer. Lesson 10: Think Like a Production Engineer As adoption grows, operational excellence becomes the differentiator. Ask questions such as: Can we troubleshoot failures? Can we measure business impact? Can we control costs? Can we scale safely? Can we govern usage? These questions become more important than model selection over time. Key Takeaway Production success comes from engineering discipline, not model size. Microsoft Foundry Features That Helped Improve Reliability Foundry Capability Production Benefit Agent Service Agent orchestration Knowledge Sources Grounded responses Evaluations Quality measurement Tracing Workflow visibility Model Catalog Model flexibility Safety Systems Risk mitigation These capabilities help teams move beyond proofs of concept and build solutions that are ready for real-world adoption. Production Readiness Checklist Before releasing an AI agent, verify: ✅ Business goals defined ✅ Grounding strategy implemented ✅ Security controls enabled ✅ Evaluation pipeline established ✅ Monitoring configured ✅ Failure handling tested ✅ Cost optimization reviewed ✅ Governance process defined ✅ User feedback loop available ✅ Deployment rollback strategy prepared Get Started with Production-Ready Agents If you're building AI agents using Microsoft Foundry, start by focusing on grounding, observability, security, and evaluation from day one. Suggested next steps: Build your first agent in Microsoft Foundry Configure Azure AI Search grounding Enable tracing and monitoring Evaluate agent quality before deployment Add resilience and fallback strategies Which of these 10 lessons has been most valuable in your own AI agent journey? Share your experiences and insights in the comments.Discover the Future of Data Engineering with Microsoft Fabric for Technical Students & Entrepreneurs
Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place. This makes it an ideal platform for technical students and entrepreneurial developers looking to streamline their data engineering and analytics workflows.Closing the AI Agent Governance Gap with Microsoft Foundry
Developers are shipping agents faster than security teams can catalog them. As organizations move beyond pilots and begin operating dozens or hundreds of agents, one question keeps coming up: how do we actually get visibility into our AI agents across our environment? In this article, we'll walk through how Azure services can help establish visibility, guardrails, and cost accountability across your AI estate. Governance for AI happens across four layers: Resources – who can create new AI resources Builders – who can develop and publish agents in a certain scope Behavior – how agent outputs are evaluated, monitored, and governed Dependencies – what models, tools, APIs, and MCP servers agents can interact with Most organizations already have governance controls for identities, networking, and compliance. The challenge isn't creating new controls. It's connecting existing controls into an operating model that works for AI agents. Below we walk through each area and go a bit deeper on how to close the gap. Setting up boundaries with Azure Policy First, let's start in the Azure portal with Azure Policy. Azure Policy lets you set guardrails on what can be deployed in your environment and flags or blocks anything that doesn't comply. For AI workloads, the built-in definitions range from limiting models that people in your organization can deploy to locking down the network through enabling private endpoints. Some policies you get started with: Foundry model deployments should only use approved models: Restricts deployments to models or publishers your organization has explicitly approved Foundry model deployments should meet eligibility requirements (preview): Applies rules based on model attributes like preview vs. GA status and distribution source Azure AI Services resources should have key access disabled: Makes Microsoft Entra ID the only entry point The full list of policies related to Azure AI Services are available here: List of built-in policy definitions - Azure Policy | Microsoft Learn Why this comes first: Policy checks resources before they are created, so it proactively keeps your environment aligned with your standards. Implementing role-based access control (RBAC) Once these boundaries are in place, the next step is RBAC. Setting RBAC up early ensures that people and identities building agents have the right scope for what they actually need to do. Foundry roles only apply when you authenticate using Microsoft Entra ID. If you're using key-based authentication instead, the key grants full access with without role restrictions. API keys are convenient for quick development usage but when moving towards production, Microsoft Entra ID is the preferred method. Roles can be assigned at three scopes: the Foundry resource, a Foundry project, or an individual agent itself. Below is an example of how different personas within organization can map to a certain scope for creating and building agents with Foundry. Here's how each role in the diagram compares, from least to most privileged: Role Privilege Level What it does in Microsoft Foundry Foundry Agent Consumer Least Interact with agent endpoints in a project. This is your least-privilege role for people who only need to use agents. Foundry User Low Grants reader access to the Foundry project, the Foundry resource, and data actions for your Foundry project. Least-privilege access role for developers building and testing agents. Foundry Project Manager Medium This role lets you perform management actions on Foundry projects, build and develop with projects, and conditionally assign the Foundry User role to other user principals. Foundry Account Owner Higher Grants full access to manage Foundry projects and resources, and lets you assign the Foundry User role to other user principals. Foundry Owner Highest Grants full access to manage Foundry projects and resources to build and develop with projects. This role can also assign the Foundry User, ACR, and monitoring roles to users in the environment. Source: Role-based access control for Microsoft Foundry - Microsoft Foundry | Microsoft Learn For the agent resources themselves, assign managed identities rather than API keys since it lowers the risk of having compromised credentials. Note if you're scripting RBAC permissions: these roles were recently renamed from Azure AI User, Azure AI Owner, Azure AI Account Owner, and Azure AI Project Manager. The role IDs and permissions didn't change, so use the role definition GUID in your code to avoid issues while the rename rolls out. Observability in Microsoft Foundry Governance requires more than access control. Organizations also need evidence of how agents are being used. Observability provides the audit trail needed to investigate incidents, understand usage patterns, and track costs. The Foundry Control Plane brings these observability and governance tools together in one place, alongside services like Azure Monitor, Microsoft Entra, Microsoft Purview, and Azure Policy. In the Foundry portal, tracing is a good starting point. Once you connect an Application Insights resource to your project, Foundry turns on tracing automatically so every run, including the ones you test in the playground, is logged. After it is completed, you can search by Response ID or Trace ID to see the conversation history, token usage, run steps, tool calls, and inputs and outputs between the user and the agent. For more granular queries, you can write KQL to dive into individual agent runs or use the prebuilt Grafana dashboards in Azure Monitor. With client-side tracing, you can also export traces to observability tools you may already use, such as Datadog or Jaeger. Note: Permissions required for viewing this telemetry requires the Log Analytics Reader role on the connected Application Insights resource, and Privileged Monitoring Data Reader on top of that if the underlying Log Analytics tables are protected. Alongside all of these monitoring features, every agent comes with content safety guardrails and evaluations let you test and optimize performance before and after you publish your agents. When agents get published to Microsoft Teams and Microsoft 365 Copilot, Microsoft 365 admins can approve usage requests. These requests can be further scoped to a limited group for pilot testing/department usage or the full organization. Adding an AI gateway Observability tells you what your agents are doing, but how do you actually control them? This is where Azure API Management comes in. Once you have more than one agent, model deployment, or multiple teams consuming them, you need a single enforcement point between the agents and the resources they call. Adding Azure API Management in front of Microsoft Foundry gives you: Rate limiting and load balancing across regions and model deployments Consistent authentication and quota policies for model and tool traffic Usage tracking per team or cost center so you can accurately charge back to different departments Governed access to your custom and remote MCP servers Note: When choosing MCP servers, start with trusted, enterprise supported sources (GitHub, Microsoft, internally developed servers, etc.) that have documented security controls, enterprise authentication, clear ownership, and least-privilege permissions. Treat community MCP servers as untrusted until they have undergone a formal security review and are verified by organizations. Adding an AI gateway completes the governance picture. Azure Policy governs what can be deployed. RBAC governs who can build and manage agents. Observability provides evidence of how agents behave in production. The AI gateway extends governance into runtime, controlling how agents interact with models, tools, and external systems. Combined, these layers help organizations move beyond simply building agents to operating them responsibly at scale. Extend governance across the wider estate A few directions to take this further: MCP registry in Azure API Center – As MCP usage grows, you can create an approved inventory of MCP servers and APIs that can be used across an organization. Microsoft Agent 365 – Microsoft's enterprise control plane for AI agents. It gives every agent its own Microsoft Entra Agent ID and published Foundry agents sync to its registry automatically. This gives your IT team one place to run access reviews, lifecycle policies, and owner attestation across every agent in the tenant, including shadow agents discovered outside Foundry. Copilot Studio – when you add an MCP tool, you can point it at the API Management URL instead of the direct remote endpoint to gain additional observability through the gateway. GitHub Copilot – you can apply the same AI gateway-fronted MCP registry, applied to the developer side. Microsoft Purview – data classification, DLP, audit, and AI interaction governance across the wider estate. Where to go next Looking for a quick start? Turn on the three Azure Policy definitions above in audit mode against a non-production subscription and see what gets flagged that is out of compliance. Ready to design the end-to-end pattern? Take a look at this Cloud Adoption Framework guidance on AI governance and governing Azure platform services for AI. Want to go deeper on agent observability? Start with these articles around Application Insights integrations with Foundry: Use Insights in Microsoft Foundry and Monitor AI Agents with Application Insights Governing AI agents doesn't require starting from scratch. The identity, policy, monitoring, and cost controls you already use for the rest of your Azure estate can extend to AI workloads. Start with one layer, connect the next, and build a governance foundation that grows with your AI adoption.Foundry Agent Service at Ignite 2025: Simple to Build. Powerful to Deploy. Trusted to Operate.
The upgraded Foundry Agent Service delivers a unified, simplified platform with managed hosting, built-in memory, tool catalogs, and seamless integration with Microsoft Agent Framework. Developers can now deploy agents faster and more securely, leveraging one-click publishing to Microsoft 365 and advanced governance features for streamlined enterprise AI operations.Observability for Multi-Agent Systems with Microsoft Agent Framework and Azure AI Foundry
Agentic applications are revolutionizing enterprise automation, but their dynamic toolchains and latent reasoning make them notoriously hard to operate. In this post, you'll learn how to instrument a Microsoft Agent Framework–based service with OpenTelemetry, ship traces to Azure AI Foundry observability, and adopt a practical workflow to debug, evaluate, and improve multi-agent behavior in production. We'll show how to wire spans around reasoning steps and tool calls (OpenAPI / MCP), enabling deep visibility into your agentic workflows. Who Should Read This? Developers building agents with Microsoft Agent Framework (MAF) in .NET or Python Architects/SREs seeking enterprise-grade visibility, governance, and reliability for deployments on Azure AI Foundry Why Observability Is Non-Negotiable for Agents Traditional logs fall short for agentic systems: Reasoning and routing (which tool? which doc?) are opaque without explicit spans/events Failures often occur between components (e.g., retrieval mismatch, tool schema drift) Without traces across agents ⇄ tools ⇄ data stores, you can't reproduce or evaluate behavior Microsoft has introduced multi-agent observability patterns and OpenTelemetry (OTel) conventions that unify traces across Agent Framework, Foundry, and popular stacks—so you can see one coherent timeline for each task. Reference Architecture Key Capabilities Agent orchestration & deployment via Microsoft Agent Framework Model access using Foundry’s OpenAI-compatible endpoint OpenTelemetry for traces/spans + attributes (agent, tool, retrieval, latency, tokens) Step-by-Step Implementation Assumption: This article uses Azure Monitor (via Application Insights) as the OpenTelemetry exporter, but you can configure other supported exporters in the same way. Prerequisites .NET 8 SDK or later Azure OpenAI service (endpoint, API key, deployed model) Application Insights and Grafana Create an Agent with OpenTelemetry (ASP.NET Core or Console App) Install required packages: dotnet add package Azure.AI.OpenAI dotnet add package Azure.Monitor.OpenTelemetry.Exporter dotnet add package Microsoft.Agents.AI.OpenAI dotnet add package Microsoft.Extensions.Logging dotnet add package OpenTelemetry dotnet add package OpenTelemetry.Trace dotnet add package OpenTelemetry.Metrics dotnet add package OpenTelemetry.Extensions.Hosting dotnet add package OpenTelemetry.Instrumentation.Http Setup environment variables: AZURE_OPENAI_ENDPOINT: https://<your_service_name>.openai.azure.com/ AZURE_OPENAI_API_KEY: <your_azure_openai_apikey> APPLICATIONINSIGHTS_CONNECTION_STRING: <your_application_insights_connectionstring_for_azuremonitor_exporter> Configure tracing once at startup: var applicationInsightsConnectionString = Environment.GetEnvironmentVariable("APPLICATIONINSIGHTS_CONNECTION_STRING"); // Create a resource describing the service var resource = ResourceBuilder.CreateDefault() .AddService(serviceName: ServiceName) .AddAttributes(new Dictionary<string, object> { ["deployment.environment"] = "development", ["service.instance.id"] = Environment.MachineName }) .Build(); // Setup OpenTelemetry TracerProvider var traceProvider = Sdk.CreateTracerProviderBuilder() .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(ServiceName)) .AddSource(SourceName) .AddSource("Microsoft.Agents.AI") .AddHttpClientInstrumentation() .AddAzureMonitorTraceExporter(options => { options.ConnectionString = applicationInsightsConnectionString; }) .Build(); // Setup OpenTelemetry MeterProvider var meterProvider = Sdk.CreateMeterProviderBuilder() .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(ServiceName)) .AddMeter(SourceName) .AddAzureMonitorMetricExporter(options => { options.ConnectionString = applicationInsightsConnectionString; }) .Build(); // Configure DI and OpenTelemetry var serviceCollection = new ServiceCollection(); // Setup Logging with OpenTelemetry and Application Insights serviceCollection.AddLogging(loggingBuilder => { loggingBuilder.SetMinimumLevel(LogLevel.Debug); loggingBuilder.AddOpenTelemetry(options => { options.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(ServiceName)); options.IncludeScopes = true; options.IncludeFormattedMessage = true; options.AddAzureMonitorLogExporter(exporterOptions => { exporterOptions.ConnectionString = applicationInsightsConnectionString; }); }); loggingBuilder.AddApplicationInsights( configureTelemetryConfiguration: (config) => { config.ConnectionString = Environment.GetEnvironmentVariable("APPLICATIONINSIGHTS_CONNECTION_STRING"); }, configureApplicationInsightsLoggerOptions: options => { options.TrackExceptionsAsExceptionTelemetry = true; options.IncludeScopes = true; }); }); Configure custom metrics and activity source for tracing: using var activitySource = new ActivitySource(SourceName); using var meter = new Meter(SourceName); // Create custom metrics var interactionCounter = meter.CreateCounter<long>("chat_interactions_total", description: "Total number of chat interactions"); var responseTimeHistogram = meter.CreateHistogram<double>("chat_response_time_ms", description: "Chat response time in milliseconds"); 2. Wire-up the AI Agent: // Create OpenAI client var endpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT"); var apiKey = Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY"); var deploymentName = "gpt-4o-mini"; using var client = new AzureOpenAIClient(new Uri(endpoint), new AzureKeyCredential(apiKey)) .GetChatClient(deploymentName) .AsIChatClient() .AsBuilder() .UseOpenTelemetry(sourceName: SourceName, configure: (cfg) => cfg.EnableSensitiveData = true) .Build(); logger.LogInformation("Creating Agent with OpenTelemetry instrumentation"); // Create AI Agent var agent = new ChatClientAgent( client, name: "AgentObservabilityDemo", instructions: "You are a helpful assistant that provides concise and informative responses.") .AsBuilder() .UseOpenTelemetry(SourceName, configure: (cfg) => cfg.EnableSensitiveData = true) .Build(); var thread = agent.GetNewThread(); logger.LogInformation("Agent created successfully with ID: {AgentId}", agent.Id); 3. Instrument Agent logic with semantic attributes and call OpenAI-compatible API: // Create a parent span for the entire agent session using var sessionActivity = activitySource.StartActivity("Agent Session"); Console.WriteLine($"Trace ID: {sessionActivity?.TraceId} "); var sessionId = Guid.NewGuid().ToString("N"); sessionActivity? .SetTag("agent.name", "AgentObservabilityDemo") .SetTag("session.id", sessionId) .SetTag("session.start_time", DateTimeOffset.UtcNow.ToString("O")); logger.LogInformation("Starting agent session with ID: {SessionId}", sessionId); using (logger.BeginScope(new Dictionary<string, object> { ["SessionId"] = sessionId, ["AgentName"] = "AgentObservabilityDemo" })) { var interactionCount = 0; while (true) { Console.Write("You (or 'exit' to quit): "); var input = Console.ReadLine(); if (string.IsNullOrWhiteSpace(input) || input.Equals("exit", StringComparison.OrdinalIgnoreCase)) { logger.LogInformation("User requested to exit the session"); break; } interactionCount++; logger.LogInformation("Processing interaction #{InteractionCount}", interactionCount); // Create a child span for each individual interaction using var activity = activitySource.StartActivity("Agent Interaction"); activity? .SetTag("user.input", input) .SetTag("agent.name", "AgentObservabilityDemo") .SetTag("interaction.number", interactionCount); var stopwatch = Stopwatch.StartNew(); try { logger.LogInformation("Starting agent execution for interaction #{InteractionCount}", interactionCount); var response = await agent.RunAsync(input); Console.WriteLine($"Agent: {response}"); Console.WriteLine(); stopwatch.Stop(); var responseTimeMs = stopwatch.Elapsed.TotalMilliseconds; // Record metrics interactionCounter.Add(1, new KeyValuePair<string, object?>("status", "success")); responseTimeHistogram.Record(responseTimeMs, new KeyValuePair<string, object?>("status", "success")); activity?.SetTag("interaction.status", "success"); logger.LogInformation("Agent interaction #{InteractionNumber} completed successfully in {ResponseTime:F2} seconds", interactionCount, responseTimeMs); } catch (Exception ex) { Console.WriteLine($"Error: {ex.Message}"); Console.WriteLine(); stopwatch.Stop(); var responseTimeMs = stopwatch.Elapsed.TotalSeconds; // Record error metrics interactionCounter.Add(1, new KeyValuePair<string, object?>("status", "error")); responseTimeHistogram.Record(responseTimeMs, new KeyValuePair<string, object?>("status", "error")); activity? .SetTag("response.success", false) .SetTag("error.message", ex.Message) .SetStatus(ActivityStatusCode.Error, ex.Message); logger.LogError(ex, "Agent interaction #{InteractionNumber} failed after {ResponseTime:F2} seconds: {ErrorMessage}", interactionCount, responseTimeMs, ex.Message); } } // Add session summary to the parent span sessionActivity? .SetTag("session.total_interactions", interactionCount) .SetTag("session.end_time", DateTimeOffset.UtcNow.ToString("O")); logger.LogInformation("Agent session completed. Total interactions: {TotalInteractions}", interactionCount); Azure Monitor dashboard Once you run the agent and generate some traffic, your dashboard in Azure Monitor will be populated as shown below: You can drill down to specific service / activity source / spans by applying relevant filters: Key Features Demonstrated OpenTelemetry instrumentation with Microsoft Agent framework Custom metrics for user interactions End-to-end Telemetry correlation Real time telemetry visualization along with metrics and logging interactions Further reading Introducing Microsoft Agent Framework Azure AI Foundry docs OpenTelemetry Aspire Demo with Azure OpenAI
