Monitor OpenAI Agents SDK with Application Insights

As AI agents become more prevalent in applications, monitoring their behavior and performance becomes crucial. In this blog post, we'll explore how to monitor the OpenAI Agents SDK using Azure Application Insights through OpenTelemetry integration.

Video provider requires cookies to play the video. Accept to continue or watch it directly on the provider's site.

Accept

Enhancing OpenAI Agents with OpenTelemetry

The OpenAI Agents SDK provides powerful capabilities for building agent-based applications. By default, the SDK doesn't emit OpenTelemetry data, as noted in GitHub issue #18. This presents an opportunity to extend the SDK's functionality with robust observability features.

Adding OpenTelemetry integration enables you to:

• Track agent interactions across distributed systems
• Monitor performance metrics in production
• Gain insights into agent behaviour
• Seamlessly integrate with existing observability platforms

Fortunately, the Pydantic Logfire SDK has implemented an OpenTelemetry instrumentation wrapper for OpenAI Agents. This wrapper allows us to capture telemetry data and propagate it to an OpenTelemetry Collector endpoint.

How It Works

The integration works by wrapping the OpenAI Agents tracing provider with a Logfire-compatible wrapper that generates OpenTelemetry spans for various agent activities:

• Agent runs
• Function calls
• Chat completions
• Handoffs between agents
• Guardrail evaluations

Each of these activities is captured as a span with relevant attributes that provide context about the operation.

Implementation Example

Here's how to set up the Logfire instrumentation in your application:

import logfire
from openai import AsyncAzureOpenAI
from agents import set_default_openai_client, set_tracing_disabled

# Configure your OpenAI client
azure_openai_client = AsyncAzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    api_version=os.getenv("AZURE_OPENAI_API_VERSION"),
    azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
    azure_deployment=os.getenv("AZURE_OPENAI_DEPLOYMENT")
)

# Set as default client and enable tracing
set_default_openai_client(azure_openai_client)
set_tracing_disabled(False)

# Configure OpenTelemetry endpoint
os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "http://0.0.0.0:4318/v1/traces"

# Configure Logfire
logfire.configure(
    service_name='my-agent-service',
    send_to_logfire=False,
    distributed_tracing=True
)

# Instrument OpenAI Agents
logfire.instrument_openai_agents()

Setting Up the OpenTelemetry Collector

To collect and forward the telemetry data to Application Insights, we need to set up an OpenTelemetry Collector. There are two ways to do this:

Option 1: Run the Collector Locally

Find the right OpenTelemetry Contrib Releases for your processor architecture at:  https://github.com/open-telemetry/opentelemetry-collector-releases/releases/tag/v0.121.0

Only Contrib releases will support Azure Monitor exporter.

./otelcol-contrib --config=otel-collector-config.yaml

Option 2: Run the Collector in Docker

docker run --rm \
   -v $(pwd)/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml \
   -p 4318:4318 \
   -p 55679:55679 \
   otel/opentelemetry-collector-contrib:latest

Collector Configuration

Here's a basic configuration for the OpenTelemetry Collector that forwards data to Azure Application Insights:

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"

exporters:
  azuremonitor:
    connection_string: "InstrumentationKey=your-instrumentation-key;IngestionEndpoint=https://your-region.in.applicationinsights.azure.com/"
    maxbatchsize: 100
    maxbatchinterval: 10s
  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [azuremonitor, debug]

What You Can Monitor

With this setup, you can monitor various aspects of your OpenAI Agents in Application Insights:

• Agent Performance: Track how long each agent takes to process requests
• Model Usage: Monitor which AI models are being used and their response times
• Function Calls: See which tools/functions are being called by agents
• Handoffs: Track when agents hand off tasks to other specialized agents
• Errors: Identify and diagnose failures in agent processing
• End-to-End Traces: Follow user requests through your entire system

Example Trace Visualisation

In Application Insights, you can visualise the traces as a hierarchical timeline, showing the flow of operations:

Known Issue: Span Name Display in Application Insights

When using LogFire SDK 3.8.1 with Application Insights, you might notice that span names appear as message templates (with regular expressions) instead of showing the actual agent or model names. This makes it harder to identify specific spans in the Application Insights UI.

Issue: In the current implementation of LogFire SDK's OpenAI Agents integration (source code), the message template is used as the span's name, resulting in spans being displayed with placeholders like {name!r} or {gen_ai.request.model!r} instead of actual values.

Temporary Fix

Until LogFire SDK introduces a fix, you can modify the /logfire/_internal/integrations/openai_agents.py file to properly format the span names. This is after pip install logfire, the file will usually be at venv/lib/python3.11/site-packages/logfire/_internal/integrations/openai_agents.py

Replace the span creation code around line 100:

Original code

logfire_span = self.logfire_instance.span(
    msg_template,
    **attributes_from_span_data(span_data, msg_template),
    **extra_attributes,
    _tags=['LLM'] * isinstance(span_data, GenerationSpanData),
)

Modified code with setting Span name as message

attributes = attributes_from_span_data(span_data, msg_template)
message = logfire_format(msg_template, dict(attributes or {}), NOOP_SCRUBBER)
logfire_span = self.logfire_instance.span(
    msg_template,
    _span_name=message,
    **attributes,
    **extra_attributes,
    _tags=['LLM'] * isinstance(span_data, GenerationSpanData),
)

This change formats the message template with actual values and sets it as the span name, making it much easier to identify spans in the Application Insights UI.

@dataclass
class LogfireSpanWrapper(LogfireWrapperBase[Span[TSpanData]], Span[TSpanData]):
    # ...
    def on_ending(self):
        # This is where to_agent gets updated, but too late for the span name
        # ...

Auto-Instrumentation for AKS

Azure Kubernetes Service (AKS) offers a codeless way to enable OpenTelemetry instrumentation for your applications. This approach simplifies the setup process and ensures that your OpenAI Agents can send telemetry data without requiring manual instrumentation.

How to Enable Auto-Instrumentation

To enable auto-instrumentation for Python applications in AKS, you can add an annotation to your pod specification:

annotations:
  instrumentation.opentelemetry.io/inject-python: 'true'

This annotation tells the OpenTelemetry Operator to inject the necessary instrumentation into your Python application.

For more details, refer to the following resources:

• Microsoft Learn: Codeless application monitoring for Kubernetes
• OpenTelemetry Docs: Automatic Instrumentation for Kubernetes

Built-in Managed OpenTelemetry Collector in Azure Container Apps

Azure Container Apps provides a built-in Managed OpenTelemetry Collector that simplifies the process of collecting and forwarding telemetry data to Application Insights. This eliminates the need to deploy and manage your own collector instance.

Setting Up the Managed Collector

When you enable the built-in collector, Azure Container Apps automatically sets the OTEL_EXPORTER_OTLP_ENDPOINT environment variable for your applications. This allows the Logfire SDK to send traces to the collector without any additional configuration.

Here's an example of enabling the collector in an ARM template:

{
  "type": "Microsoft.App/containerApps",
  "properties": {
    "configuration": {
      "dapr": {},
      "ingress": {},
      "observability": {
        "applicationInsightsConnection": {
          "connectionString": "InstrumentationKey=your-instrumentation-key"
        }
      }
    }
  }
}

For more information, check out these resources:

• Microsoft Learn: OpenTelemetry agents in Azure Container Apps
• Tech Community: How to monitor applications by using OpenTelemetry on Azure Container Apps

Conclusion

Monitoring OpenAI Agents with Application Insights provides valuable insights into your AI systems' performance and behavior. By leveraging the Pydantic Logfire SDK's OpenTelemetry instrumentation and the OpenTelemetry Collector, you can gain visibility into your agents' operations and ensure they're functioning as expected.

This approach allows you to integrate AI agent monitoring into your existing observability stack, making it easier to maintain and troubleshoot complex AI systems in production environments.

Resources

Implementation can be found at https://github.com/hieumoscow/azure-openai-agents

References:

• OpenAI Agents Python SDK
• GitHub Issue: OpenAI Agents Logging
• OpenTelemetry Collector Documentation
• Azure Application Insights Documentation
• Codeless application monitoring for Kubernetes
• OpenTelemetry Automatic Instrumentation for Kubernetes
• OpenTelemetry agents in Azure Container Apps
• How to monitor applications using OpenTelemetry on Azure Container Apps