Introduction to Monitoring Azure OpenAI Service

Brass Contributor

Monitoring Azure OpenAI : Generated using Microsoft DesignerMonitoring Azure OpenAI : Generated using Microsoft Designer


Monitoring data

Azure OpenAI collects monitoring data as:

  • Activity log

  • Alerts

  • Metrics

  • Diagnostic settings

  • Insights

Collection and routing

collections and routing.png
  • Platform metrics and the Activity log are collected automatically
  • Can be routed to other locations by using a diagnostic setting
  • Configure diagnostic settings in the Azure portal

Implement logging and monitoring for Azure OpenAI models - Architecture

Monitoring and logging Azure OpenAI Architecture.png


Diagram that shows an architecture that provides monitoring and logging for Azure OpenAI.


  • Client applications access Azure OpenAI endpoints to perform text generation (completions) and model training (fine-tuning).
  • Azure Application Gateway provides a single point of entry to Azure OpenAI models and provides load balancing for APIs.

  • Azure API Management enables security controls and auditing and monitoring of the Azure OpenAI models.
    a. In API Management, enhanced-security access is granted via Azure Active Directory (Azure AD) groups with subscription-based access permissions.
    b. Auditing is enabled for all interactions with the models via Azure Monitor request logging.
    c. Monitoring provides detailed Azure OpenAI model usage KPIs and metrics, including prompt information and token statistics for usage traceability.

  • API Management connects to all Azure resources via Azure Private Link. This configuration provides enhanced security for all traffic via private endpoints and contains traffic in the private network.

  • Multiple Azure OpenAI instances enable scale-out of API usage to ensure high availability and disaster recovery for the service.


    More details : Monitoring Azure OpenAI Service - Azure AI services | Microsoft Learn

0 Replies