Blog Post

Apps on Azure Blog
9 MIN READ

From AI Adoption to AI Governance - Using APIM as the Gateway for Azure AI Foundry

Abhishek_Mittal's avatar
Jul 13, 2026

Co-authored by Gaurav Jain (Senior Cloud Solution Architect @ Microsoft) and Abhishek Mittal (Cloud Solution Architect @ Microsoft)

 

Enterprises move through three phases of AI adoption: evaluating models, building apps and agents, and operationalizing them in production. The first two are easier to accelerate. The third is where governance becomes critical. Once multiple teams share an AI endpoint, leaders need clear answers to practical questions: which model consumed tokens, which team used them, and who is authorized to call it?

This post shows how to place Azure API Management (APIM) in front of Azure AI Foundry as an AI Gateway, turning a shared endpoint into a governed control point for per-model token visibility, chargeback, and budget alerts — with no changes to client code. It also shows where Azure Front Door and Web Application Firewall (WAF) fit in a secure AI Landing Zone.

The problem: AI adoption is outpacing AI governance

A common starting point is an Azure OpenAI resource running multiple models. The team already has operational telemetry, but governance needs a different view: per-model token usage for chargeback, budget alerts, and capacity planning, captured in one place.

Azure gives you rich resource-level telemetry out of the box, and that is exactly where we started:

  • Azure Monitor — Metrics blade: shows token usage split by model and deployment in near real time.
  • Diagnostic settings: stream the resource's metrics and request logs into Log Analytics (the AzureMetrics and AzureDiagnostics tables).

Azure Monitor provides useful resource-level telemetry, including metrics and diagnostic logs. A governance view needs something different: model identity and token counts correlated in a single record, so teams can build a per-model, month-to-date ledger for chargeback and alerting.

  • The AzureMetrics table carries the token totals, aggregated at the resource level.
  • The AzureDiagnostics logs carry the model and deployment name at the request level.

Each stream does its job well. Correlating them into one per-model, month-to-date ledger — and alerting on it — is a governance concern that sits above any single resource. Azure Monitor metric alerts, for instance, work on a rolling 24-hour window that maps cleanly onto a per-day token budget; a month-to-date, per-model chargeback ledger is simply a different shape of question — and a natural fit for a dedicated control point.

This transition is the focus of this post: moving from AI adoption to AI governance by introducing a control point where model identity and token usage are captured together by design. The natural home for that control point is an AI gateway — and we build it next with Azure API Management in front of Azure AI Foundry.

The pattern: APIM as the AI Gateway for Azure AI Foundry

The AI gateway in Azure API Management is a set of capabilities to secure, scale, monitor, and govern the AI models, agents, and tools behind your apps. It isn't a separate product — it extends the existing API Management gateway. As Microsoft's guidance puts it, as AI adoption matures the gateway helps you authenticate and authorize access to AI services, load balance across endpoints, monitor and log AI interactions, and manage token usage and quotas across multiple applications.

APIM becomes the governed front door for Azure AI Foundry. Clients continue calling an OpenAI-compatible endpoint; APIM authenticates to Foundry with a system-assigned managed identity, forwards the request, and emits per-model token telemetry to Azure Monitor and Application Insights. The result is per-model visibility without client-side changes.

Models behind the gateway

The gateway can front any model deployment in Azure AI Foundry — Azure OpenAI models, other Foundry models, or a mix — and the pattern is identical no matter which you run. For a concrete reference, the walkthrough in this post sits in front of two existing deployments:

Deployment

Model

Provisioned capacity (TPM)

gpt-4.1

gpt-4.1

500K

gpt-5

gpt-5

50K

Example deployments referenced throughout this post. Note the deliberate capacity gap — gpt-5 at 50K TPM versus 500K for gpt-4.1 — exactly the kind of asymmetry that makes per-model visibility a governance requirement, not a nice-to-have.

Architecture

Figure 1 — Architecture / component flow: consumers call one governed API; the inbound policy authenticates with a managed identity, resolves the model, and emits per-model token metrics to Azure Monitor and Application Insights.

The starting point (“before”): a pass-through without a usage signal

By default, APIM operates as a straightforward proxy. If you import a Foundry API and keep the default configuration, the policy simply selects the backend service:

<policies>
  <inbound>
    <base />
    <set-backend-service id="apim-generated-policy" backend-id="foundry-backend" />
  </inbound>
  <backend><base /></backend>
  <outbound><base /></outbound>
  <on-error><base /></on-error>
</policies>

A plain pass-through API. It forwards traffic faithfully — it simply doesn't surface a usage signal yet. This is our “before.”

Pass-through configuration works, but it does not distinguish traffic by model. All requests flow through the same stream, with no per-model chargeback signal, no capacity warning, and no clear view of which deployment is driving consumption. To govern the workload, the gateway must understand the traffic — not just relay it.

The governed gateway (“after”): a policy that sees every token

The custom inbound policy below is the heart of the pattern. It does four things in order:

  1. set-backend-service — select the Azure AI Foundry backend.
  2. authentication-managed-identity — obtain an Entra ID token for cognitiveservices.azure.com using the APIM system-assigned identity. No keys ever leave the gateway.
  3. set-variable deployment-id — resolve the model name from either the URL path or the request body (more on why below).
  4. azure-openai-emit-token-metric — emit prompt, completion, and total token counts to Azure Monitor, dimensioned by model.
<policies>
  <inbound>
    <base />
    <set-backend-service backend-id="foundry-backend" />
    <authentication-managed-identity resource="https://cognitiveservices.azure.com" />

    <!-- Resolve the model/deployment name. The Foundry Model Inference API
         (/models/chat/completions, /models/embeddings, /anthropic/v1/messages)
         passes it in the JSON body as "model". The Azure OpenAI-style surface
         (/openai/deployments/{name}/...) passes it in the URL path. Handle both. -->

    <set-variable name="deployment-id" value="@{
        var path = context.Request.Url.Path ?? "";
        var m = System.Text.RegularExpressions.Regex.Match(path, "/deployments/([^/?]+)");
        if (m.Success) { return m.Groups[1].Value; }
        try {
            var body = context.Request.Body?.As<JObject>(preserveContent: true);
            var model = body?["model"];
            if (model != null && !string.IsNullOrEmpty(model.ToString())) {
                return model.ToString();
            }
        } catch (Exception) { }
        return "unknown";
    }" />

    <!-- Emit token-usage metrics dimensioned by model, so consumption can be
         sliced per model in Azure Monitor / Application Insights. -->

    <azure-openai-emit-token-metric namespace="genai-tokens">
        <dimension name="ModelDeploymentName" value="@((string)context.Variables["deployment-id"])" />
        <dimension name="ModelName" value="@((string)context.Variables["deployment-id"])" />
        <dimension name="APIId" value="@(context.Api.Id)" />
        <dimension name="Subscription" value="@(context.Subscription?.Id ?? "none")" />
        <dimension name="Client IP" value="@(context.Request.IpAddress)" />
        <dimension name="Product ID" value="@(context.Product?.Id ?? "none")" />
    </azure-openai-emit-token-metric>
  </inbound>
  <backend>
    <base />
  </backend>
  <outbound>
    <base />
  </outbound>
  <on-error>
    <base />
  </on-error>
</policies> 

The full custom policy, applied at API scope. The highlighted value is the model-name resolution feeding a per-model token metric.

Request flow, end to end

Figure 2 traces a single chat request from top to bottom — from the caller, through the gateway's inbound policy, out to Azure AI Foundry, and into your telemetry.

Figure 2 — Per-request flow: authentication, model resolution, forwarding, and token-metric emission.

The request flow is straightforward: the client calls the APIM endpoint, the gateway selects the Foundry backend, authenticates with managed identity, resolves the model name, forwards the request, emits token metrics, and returns the response unchanged. Governance is added at the gateway without requiring client-side changes.

Why dual-shape model resolution matters

The azure-openai-emit-token-metric policy can emit usage, but it still needs a model dimension. Different API surfaces place the model name in different locations: Foundry Model Inference and Anthropic-style requests use the body, while Azure OpenAI-compatible calls use the URL path. The policy handles both shapes, so one gateway can govern all callers consistently.

Observability: per-model token visibility and chargeback

Metrics land in Azure Monitor / Application Insights under the namespace genai-tokens. The policy records Total Tokens, Prompt Tokens, and Completion Tokens, each tagged with Model Name, Model Deployment Name, API Id, APIM Product Subscription, Client IP, and Product ID. The data can then be queried directly. Per-model consumption over time:

customMetrics
| where name == "Total Tokens"
| where timestamp >= startofmonth(now())
| extend ModelName = tostring(customDimensions["ModelName"])
| summarize TotalTokens = sum(valueSum), Calls = sum(valueCount)
    by ModelName

Per-model token consumption (Application Insights customMetrics).

And a per-model, per- API product subscription view for chargeback:

customMetrics
| where name == "Total Tokens"
| where timestamp >= startofmonth(now())
| extend Model = tostring(customDimensions["ModelName"]),
         Sub   = tostring(customDimensions["Subscription"])
| summarize Tokens = sum(value) by Model, Sub, name
| order by Tokens desc

Chargeback: tokens by model and consuming subscription.

The gateway records model identity and token usage together, so the chargeback view is built in. The same signal supports dashboards, daily budget alerts, capacity planning, and cost allocation by subscription or product — without changing client code.

Turning the signal into a Cost Guardrail: a 24-hour token alert

Because the token totals now carry the model name, you can put a hard guardrail on spend. Wrap a query in an Azure Monitor log search alert rule that sums Total Tokens over the last 24 hours per model and returns only the deployments that breach a daily budget:

// Rolling 24-hour token-budget guardrail — returns any model over its daily cap 

let dailyTokenBudget = 50000; // max Total Tokens per model in a rolling 24h window 
customMetrics | where name == "Total Tokens" 
| where timestamp > ago(24h) 
 extend Model = tostring(customDimensions["ModelName"])
 | summarize TokensLast24h = sum(value) by Model 
| where TokensLast24h > dailyTokenBudget 
| extend OverBudgetBy = TokensLast24h - dailyTokenBudget 
| project Model, TokensLast24h, DailyBudget = dailyTokenBudget, OverBudgetBy 

A rolling 24-hour token-budget check. The alert rule fires whenever this query returns one or more rows.

Configure this as a scheduled log search alert: evaluate on a short cadence (for example, hourly over the trailing 24-hour window), set the alert logic to fire when the result count is greater than zero, and attach an action group that notifies the team through an email distribution list or Microsoft Teams channel. When any model crosses its rolling 24-hour token budget, the owning team is alerted, so overspend is detected within the day rather than at invoice time. Tune dailyTokenBudget per model, or add a single all-up cap, and translate token budgets into estimated daily cost ceilings to maintain continuous spend visibility.

Completing the picture: securing the AI Landing Zone with Front Door + WAF

APIM governs model usage; Azure Front Door with WAF governs public access. Placing WAF at the edge protects the AI endpoint from common web attacks, malicious bots, abusive callers, and unwanted source IP ranges before traffic reaches APIM or Foundry.

What Front Door + WAF adds

  • OWASP protection: The Azure-managed Microsoft_DefaultRuleSet_2.1 helps defend against OWASP Top 10 web attacks and known CVEs; Microsoft_BotManagerRuleSet_1.0 helps block malicious bots. Run the policy in prevention mode so offending requests are rejected with a 403, not just logged.
  • IP restriction and rate limiting: Custom WAF rules restrict access to known IP ranges and throttle abusive callers before they reach APIM or Foundry.
  • Global edge: Front Door terminates TLS at the edge and provides a single, DDoS-protected public entry point for the workload.

Defense in depth across the landing zone

Layered against the Azure AI Foundry landing zone baseline, the request path looks like this:

Defense in depth: WAF at the edge, governance at the gateway, isolation on the network, Foundry reachable over a private endpoint.

The baseline Azure AI Foundry landing zone reinforces every layer: private endpoints keep PaaS services (Foundry, Key Vault, Storage, AI Search) off the public internet; a system-assigned managed identity helps remove API keys; a hub-and-spoke topology routes egress through Azure Firewall; Azure Key Vault holds the Front Door TLS certificate; and Azure Policy enforces guardrails across the subscription. The gateway pattern from Sections 2–6 slots directly into this architecture as the governed control point for model traffic.

Outcomes and what to extend next

With APIM and the security edge in place, the shared endpoint supports per-model chargeback, capacity planning, zero client changes, and a stronger security baseline. The same gateway pattern can then be extended with token quotas, semantic caching, content safety, resiliency, and a unified model API (preview).

Closing thoughts

Moving from AI adoption to AI governance does not require re-architecting every app; it requires a control point. APIM in front of Azure AI Foundry provides that point: one policy turns token usage into a per-model governance signal, and Front Door with WAF provides a hardened edge. Start with visibility, then add quotas, safety, and resiliency as adoption scales.

References

  1. AI gateway capabilities in Azure API Management — Microsoft Learn
  2. Baseline Microsoft Foundry Chat Reference Architecture in an Azure Landing Zone - Azure Architecture Center
  3. Protect Azure OpenAI using Azure Web Application Firewall on Azure Front Door — Microsoft Learn
  4. LLM token limit policy — Microsoft Learn
  5. Emit token consumption metrics (llm-emit-token-metric) — Microsoft Learn
Updated Jul 14, 2026
Version 2.0
No CommentsBe the first to comment