With Azure API Management can you hide Try me function for API and limit product subscription count
With Azure APIM developer portal: Question 1: Can you hide the "Try me" button if the user is not logged in or does not have any API/Product subscription Question 2: Can we stop users from having multiple subscriptions to the same productFrom AI Adoption to AI Governance - Using APIM as the Gateway for Azure AI Foundry
Co-authored by Gaurav Jain (Senior Cloud Solution Architect @ Microsoft) and Abhishek Mittal (Cloud Solution Architect @ Microsoft) Enterprises move through three phases of AI adoption: evaluating models, building apps and agents, and operationalizing them in production. The first two are easier to accelerate. The third is where governance becomes critical. Once multiple teams share an AI endpoint, leaders need clear answers to practical questions: which model consumed tokens, which team used them, and who is authorized to call it? This post shows how to place Azure API Management (APIM) in front of Azure AI Foundry as an AI Gateway, turning a shared endpoint into a governed control point for per-model token visibility, chargeback, and budget alerts — with no changes to client code. It also shows where Azure Front Door and Web Application Firewall (WAF) fit in a secure AI Landing Zone. The problem: AI adoption is outpacing AI governance A common starting point is an Azure OpenAI resource running multiple models. The team already has operational telemetry, but governance needs a different view: per-model token usage for chargeback, budget alerts, and capacity planning, captured in one place. Azure gives you rich resource-level telemetry out of the box, and that is exactly where we started: Azure Monitor — Metrics blade: shows token usage split by model and deployment in near real time. Diagnostic settings: stream the resource's metrics and request logs into Log Analytics (the AzureMetrics and AzureDiagnostics tables). Azure Monitor provides useful resource-level telemetry, including metrics and diagnostic logs. A governance view needs something different: model identity and token counts correlated in a single record, so teams can build a per-model, month-to-date ledger for chargeback and alerting. The AzureMetrics table carries the token totals, aggregated at the resource level. The AzureDiagnostics logs carry the model and deployment name at the request level. Each stream does its job well. Correlating them into one per-model, month-to-date ledger — and alerting on it — is a governance concern that sits above any single resource. Azure Monitor metric alerts, for instance, work on a rolling 24-hour window that maps cleanly onto a per-day token budget; a month-to-date, per-model chargeback ledger is simply a different shape of question — and a natural fit for a dedicated control point. This transition is the focus of this post: moving from AI adoption to AI governance by introducing a control point where model identity and token usage are captured together by design. The natural home for that control point is an AI gateway — and we build it next with Azure API Management in front of Azure AI Foundry. The pattern: APIM as the AI Gateway for Azure AI Foundry The AI gateway in Azure API Management is a set of capabilities to secure, scale, monitor, and govern the AI models, agents, and tools behind your apps. It isn't a separate product — it extends the existing API Management gateway. As Microsoft's guidance puts it, as AI adoption matures the gateway helps you authenticate and authorize access to AI services, load balance across endpoints, monitor and log AI interactions, and manage token usage and quotas across multiple applications. APIM becomes the governed front door for Azure AI Foundry. Clients continue calling an OpenAI-compatible endpoint; APIM authenticates to Foundry with a system-assigned managed identity, forwards the request, and emits per-model token telemetry to Azure Monitor and Application Insights. The result is per-model visibility without client-side changes. Models behind the gateway The gateway can front any model deployment in Azure AI Foundry — Azure OpenAI models, other Foundry models, or a mix — and the pattern is identical no matter which you run. For a concrete reference, the walkthrough in this post sits in front of two existing deployments: Deployment Model Provisioned capacity (TPM) gpt-4.1 gpt-4.1 500K gpt-5 gpt-5 50K Example deployments referenced throughout this post. Note the deliberate capacity gap — gpt-5 at 50K TPM versus 500K for gpt-4.1 — exactly the kind of asymmetry that makes per-model visibility a governance requirement, not a nice-to-have. Architecture Figure 1 — Architecture / component flow: consumers call one governed API; the inbound policy authenticates with a managed identity, resolves the model, and emits per-model token metrics to Azure Monitor and Application Insights. The starting point (“before”): a pass-through without a usage signal By default, APIM operates as a straightforward proxy. If you import a Foundry API and keep the default configuration, the policy simply selects the backend service: <policies> <inbound> <base /> <set-backend-service id="apim-generated-policy" backend-id="foundry-backend" /> </inbound> <backend><base /></backend> <outbound><base /></outbound> <on-error><base /></on-error> </policies> A plain pass-through API. It forwards traffic faithfully — it simply doesn't surface a usage signal yet. This is our “before.” Pass-through configuration works, but it does not distinguish traffic by model. All requests flow through the same stream, with no per-model chargeback signal, no capacity warning, and no clear view of which deployment is driving consumption. To govern the workload, the gateway must understand the traffic — not just relay it. The governed gateway (“after”): a policy that sees every token The custom inbound policy below is the heart of the pattern. It does four things in order: set-backend-service — select the Azure AI Foundry backend. authentication-managed-identity — obtain an Entra ID token for cognitiveservices.azure.com using the APIM system-assigned identity. No keys ever leave the gateway. set-variable deployment-id — resolve the model name from either the URL path or the request body (more on why below). azure-openai-emit-token-metric — emit prompt, completion, and total token counts to Azure Monitor, dimensioned by model. <policies> <inbound> <base /> <set-backend-service backend-id="foundry-backend" /> <authentication-managed-identity resource="https://cognitiveservices.azure.com" /> <!-- Resolve the model/deployment name. The Foundry Model Inference API (/models/chat/completions, /models/embeddings, /anthropic/v1/messages) passes it in the JSON body as "model". The Azure OpenAI-style surface (/openai/deployments/{name}/...) passes it in the URL path. Handle both. --> <set-variable name="deployment-id" value="@{ var path = context.Request.Url.Path ?? ""; var m = System.Text.RegularExpressions.Regex.Match(path, "/deployments/([^/?]+)"); if (m.Success) { return m.Groups[1].Value; } try { var body = context.Request.Body?.As<JObject>(preserveContent: true); var model = body?["model"]; if (model != null && !string.IsNullOrEmpty(model.ToString())) { return model.ToString(); } } catch (Exception) { } return "unknown"; }" /> <!-- Emit token-usage metrics dimensioned by model, so consumption can be sliced per model in Azure Monitor / Application Insights. --> <azure-openai-emit-token-metric namespace="genai-tokens"> <dimension name="ModelDeploymentName" value="@((string)context.Variables["deployment-id"])" /> <dimension name="ModelName" value="@((string)context.Variables["deployment-id"])" /> <dimension name="APIId" value="@(context.Api.Id)" /> <dimension name="Subscription" value="@(context.Subscription?.Id ?? "none")" /> <dimension name="Client IP" value="@(context.Request.IpAddress)" /> <dimension name="Product ID" value="@(context.Product?.Id ?? "none")" /> </azure-openai-emit-token-metric> </inbound> <backend> <base /> </backend> <outbound> <base /> </outbound> <on-error> <base /> </on-error> </policies> The full custom policy, applied at API scope. The highlighted value is the model-name resolution feeding a per-model token metric. Request flow, end to end Figure 2 traces a single chat request from top to bottom — from the caller, through the gateway's inbound policy, out to Azure AI Foundry, and into your telemetry. Figure 2 — Per-request flow: authentication, model resolution, forwarding, and token-metric emission. The request flow is straightforward: the client calls the APIM endpoint, the gateway selects the Foundry backend, authenticates with managed identity, resolves the model name, forwards the request, emits token metrics, and returns the response unchanged. Governance is added at the gateway without requiring client-side changes. Why dual-shape model resolution matters The azure-openai-emit-token-metric policy can emit usage, but it still needs a model dimension. Different API surfaces place the model name in different locations: Foundry Model Inference and Anthropic-style requests use the body, while Azure OpenAI-compatible calls use the URL path. The policy handles both shapes, so one gateway can govern all callers consistently. Observability: per-model token visibility and chargeback Metrics land in Azure Monitor / Application Insights under the namespace genai-tokens. The policy records Total Tokens, Prompt Tokens, and Completion Tokens, each tagged with Model Name, Model Deployment Name, API Id, APIM Product Subscription, Client IP, and Product ID. The data can then be queried directly. Per-model consumption over time: customMetrics | where name == "Total Tokens" | where timestamp >= startofmonth(now()) | extend ModelName = tostring(customDimensions["ModelName"]) | summarize TotalTokens = sum(valueSum), Calls = sum(valueCount) by ModelName Per-model token consumption (Application Insights customMetrics). And a per-model, per- API product subscription view for chargeback: customMetrics | where name == "Total Tokens" | where timestamp >= startofmonth(now()) | extend Model = tostring(customDimensions["ModelName"]), Sub = tostring(customDimensions["Subscription"]) | summarize Tokens = sum(value) by Model, Sub, name | order by Tokens desc Chargeback: tokens by model and consuming subscription. The gateway records model identity and token usage together, so the chargeback view is built in. The same signal supports dashboards, daily budget alerts, capacity planning, and cost allocation by subscription or product — without changing client code. Turning the signal into a Cost Guardrail: a 24-hour token alert Because the token totals now carry the model name, you can put a hard guardrail on spend. Wrap a query in an Azure Monitor log search alert rule that sums Total Tokens over the last 24 hours per model and returns only the deployments that breach a daily budget: // Rolling 24-hour token-budget guardrail — returns any model over its daily cap let dailyTokenBudget = 50000; // max Total Tokens per model in a rolling 24h window customMetrics | where name == "Total Tokens" | where timestamp > ago(24h) extend Model = tostring(customDimensions["ModelName"]) | summarize TokensLast24h = sum(value) by Model | where TokensLast24h > dailyTokenBudget | extend OverBudgetBy = TokensLast24h - dailyTokenBudget | project Model, TokensLast24h, DailyBudget = dailyTokenBudget, OverBudgetBy A rolling 24-hour token-budget check. The alert rule fires whenever this query returns one or more rows. Configure this as a scheduled log search alert: evaluate on a short cadence (for example, hourly over the trailing 24-hour window), set the alert logic to fire when the result count is greater than zero, and attach an action group that notifies the team through an email distribution list or Microsoft Teams channel. When any model crosses its rolling 24-hour token budget, the owning team is alerted, so overspend is detected within the day rather than at invoice time. Tune dailyTokenBudget per model, or add a single all-up cap, and translate token budgets into estimated daily cost ceilings to maintain continuous spend visibility. Completing the picture: securing the AI Landing Zone with Front Door + WAF APIM governs model usage; Azure Front Door with WAF governs public access. Placing WAF at the edge protects the AI endpoint from common web attacks, malicious bots, abusive callers, and unwanted source IP ranges before traffic reaches APIM or Foundry. What Front Door + WAF adds OWASP protection: The Azure-managed Microsoft_DefaultRuleSet_2.1 helps defend against OWASP Top 10 web attacks and known CVEs; Microsoft_BotManagerRuleSet_1.0 helps block malicious bots. Run the policy in prevention mode so offending requests are rejected with a 403, not just logged. IP restriction and rate limiting: Custom WAF rules restrict access to known IP ranges and throttle abusive callers before they reach APIM or Foundry. Global edge: Front Door terminates TLS at the edge and provides a single, DDoS-protected public entry point for the workload. Defense in depth across the landing zone Layered against the Azure AI Foundry landing zone baseline, the request path looks like this: Defense in depth: WAF at the edge, governance at the gateway, isolation on the network, Foundry reachable over a private endpoint. The baseline Azure AI Foundry landing zone reinforces every layer: private endpoints keep PaaS services (Foundry, Key Vault, Storage, AI Search) off the public internet; a system-assigned managed identity helps remove API keys; a hub-and-spoke topology routes egress through Azure Firewall; Azure Key Vault holds the Front Door TLS certificate; and Azure Policy enforces guardrails across the subscription. The gateway pattern from Sections 2–6 slots directly into this architecture as the governed control point for model traffic. Outcomes and what to extend next With APIM and the security edge in place, the shared endpoint supports per-model chargeback, capacity planning, zero client changes, and a stronger security baseline. The same gateway pattern can then be extended with token quotas, semantic caching, content safety, resiliency, and a unified model API (preview). Closing thoughts Moving from AI adoption to AI governance does not require re-architecting every app; it requires a control point. APIM in front of Azure AI Foundry provides that point: one policy turns token usage into a per-model governance signal, and Front Door with WAF provides a hardened edge. Start with visibility, then add quotas, safety, and resiliency as adoption scales. References AI gateway capabilities in Azure API Management — Microsoft Learn Baseline Microsoft Foundry Chat Reference Architecture in an Azure Landing Zone - Azure Architecture Center Protect Azure OpenAI using Azure Web Application Firewall on Azure Front Door — Microsoft Learn LLM token limit policy — Microsoft Learn Emit token consumption metrics (llm-emit-token-metric) — Microsoft LearnControlling Tool Access with APIM MCP Gateway
If you've started working with MCP servers in GitHub Copilot, Claude, or any other agent host in an enterprise environment, you've probably hit a similar problem. You want to give your developers access to a useful MCP server, but not every tool it ships with. Maybe one tool is noisy and burns context for no good reason. Maybe one tool calls a paid API. Maybe one tool does something your security team is not comfortable with. The MCP server is all-or-nothing: install it and you get the lot. Most MCP servers don't give you a way to switch individual tools off. The MCP spec doesn't define one either. So if you want fine-grained control, you need to put something in front of the server that can see the JSON-RPC traffic and make decisions about it. That something is an MCP gateway, and Azure API Management (APIM) can do the job. What an MCP gateway actually is An MCP gateway sits between the agent (the client) and the MCP server (the backend) and inspects the MCP protocol traffic flowing between them. Because MCP is just JSON-RPC over HTTP or SSE, anything that can do reverse-proxy plus payload inspection can in theory act as a gateway. The interesting bit is what you do with that position in the network. If you've worked with API gateways before, the mental model is the same: you're putting a managed entry point in front of one or more backends, and using policy to control what gets through. The differences are that the protocol is JSON-RPC over a long-lived connection, and the things you're filtering aren't routes but tools, prompts, and resources. Out of the box, MCP gives you very little operational control. The protocol assumes a trusted one-to-one relationship between a client and a server. There's no built-in authentication story beyond what the transport gives you, no rate limiting, no audit trail beyond the agent's own logs, and no way to share one MCP server safely across multiple users or teams. A gateway is how you fix all of that without modifying the MCP server itself. MCP Gateway Functionality The tool access problem is not the only reason to put a gateway in front of an MCP server. The list looks a lot like the list of reasons you'd put any API behind a gateway, with a few MCP-specific twists: Centralised authentication and authorisation. Add Entra ID, mTLS, or scoped tokens in front of MCP servers that ship with little more than an API key. Rate limiting and quota. Stop a runaway agent loop from hammering a paid upstream API and racking up a bill in minutes. Logging, audit and observability. Capture which user invoked which tool with what arguments and ship it to Log Analytics or your SIEM. Network isolation. Keep developer machines off the public internet by fronting external MCP servers with private endpoints and fixed egress IPs. Sharing one MCP server across many clients. Turn a single-tenant MCP server into a multi-tenant front door with per-user identity and per-team limits. Policy and governance. Control which tools are exposed, redact fields, validate arguments, or transform responses before they reach the agent. Aggregating multiple MCP servers. Present one logical MCP endpoint that fans out to several backends so agent hosts have a single connection point. Failure handling and resilience. Add retries, circuit breakers, and caching of tools/list rather than have every agent host grow its own. Cost management. Put caps, alerts, and per-team chargeback on MCP servers that wrap something paid by the call. APIM as the gateway APIM gives you all of the above, and Microsoft has been adding MCP-specific support over the last few releases. The relevant bits for this post: There's a dedicated MCP servers section in APIM, separate from the standard APIs blade. This is where MCP-aware features live. You can register an existing MCP server as an external MCP server. APIM proxies the protocol traffic and applies policies to it. You can also expose a normal REST API as an MCP server, by adding an MCP layer on top of an existing API. The standard APIM policy engine works on MCP traffic. The shape of the problem The MCP protocol has two methods that matter here: tools/list is what the client calls when it connects. The server returns the catalogue of tools available, with names, descriptions, and input schemas. The agent uses this to decide what it can do. tools/call is what the client sends when the agent actually wants to invoke a tool. If you want to hide a tool, you have to deal with both. Filtering tools/list stops the agent ever knowing the tool exists, which is what you usually want. Blocking tools/call stops a determined client (or a tool the agent guessed at) from calling it directly. You need both to be confident the tool is genuinely off-limits. For the rest of this post I'll use the Microsoft Learn MCP server as the example, because it's public, useful, and ships with three tools: microsoft_docs_search microsoft_docs_fetch microsoft_code_sample_search Say I want to allow the first two and block the third. Here's how to do it. Setting up APIM as an MCP gateway I'm going to skip the bit where you provision APIM. The MCP-specific setup is: In your APIM instance, go to MCP Servers in the left-hand nav (it's its own section, not under APIs). Add a new MCP server. Choose External MCP server as the type. Point it at the Microsoft Learn MCP server endpoint. The transport will be HTTP or SSE depending on the upstream, which APIM handles for you. Save. APIM now proxies the MCP traffic. Configure the agent to point at the APIM-exposed URL instead of the upstream. At this point you have a working passthrough. The next step is the policy. Two ways to control tool access There are two patterns that work, and the right choice depends on how much maintenance you want to take on. Option A: Static allowlist You hardcode the list of tools APIM will expose. APIM intercepts tools/list and returns your fixed list, ignoring whatever the backend says. It also blocks tools/call for anything not on the list. Pros: predictable, doesn't read the response body, doesn't care what the upstream changes. Cons: you have to maintain the schemas yourself. If Microsoft adds a useful new tool to the Learn MCP server, you won't see it until you update the policy. If they change the input schema for an existing tool, you'll need to update that too. This is the option I'd reach for first if I wanted strict control and wasn't expecting the upstream to change much. Option B: Dynamic deny-list You let tools/list flow through, then rewrite the response on its way back out, removing the tools you don't want. You also block tools/call for those tools. Pros: lower maintenance. New tools appear automatically. Schema changes are picked up. Cons: this reads and rewrites context.Response.Body, and Microsoft's own guidance for MCP policies warns that response-body access can interfere with streaming. In practice, tools/list is non-streaming JSON-RPC and this works fine, but you need to test it carefully in your environment, particularly if your agent host is fussy about response handling. This method also requires you to keep on top of when new tools are added and add them to the list if you don't want them used, they will automatically be available to users if you do not. Pick this one when you're confident in the upstream and want to minimise toil. Option A: the allowlist policy Here's the full policy for the static allowlist. It goes into the MCP Server policy editor in APIM. You'll notice there's no <base /> in the sections. Some MCP Server policy editors don't include them, and the policy works fine without them. <policies> <inbound> <choose> <when condition="@{ var body = context.Request.Body.As<Newtonsoft.Json.Linq.JObject>(preserveContent: true); if (body == null) { return false; } var rpcMethod = (string)body["method"]; return string.Equals(rpcMethod, "tools/list", System.StringComparison.OrdinalIgnoreCase); }"> <return-response> <set-status code="200" reason="OK" /> <set-header name="Content-Type" exists-action="override"> <value>application/json</value> </set-header> <set-body>@{ var req = context.Request.Body.As<Newtonsoft.Json.Linq.JObject>(preserveContent: true); var id = req != null ? req["id"] : Newtonsoft.Json.Linq.JValue.CreateNull(); var tools = new Newtonsoft.Json.Linq.JArray( new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("name", "microsoft_docs_search"), new Newtonsoft.Json.Linq.JProperty("description", "Search official Microsoft/Azure documentation"), new Newtonsoft.Json.Linq.JProperty("inputSchema", new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("type", "object"), new Newtonsoft.Json.Linq.JProperty("properties", new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("query", new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("type", "string") )) )) )) ), new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("name", "microsoft_docs_fetch"), new Newtonsoft.Json.Linq.JProperty("description", "Fetch a Microsoft documentation page as markdown"), new Newtonsoft.Json.Linq.JProperty("inputSchema", new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("type", "object"), new Newtonsoft.Json.Linq.JProperty("properties", new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("url", new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("type", "string") )) )), new Newtonsoft.Json.Linq.JProperty("required", new Newtonsoft.Json.Linq.JArray("url")) )) ) ); var response = new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("jsonrpc", "2.0"), new Newtonsoft.Json.Linq.JProperty("id", id), new Newtonsoft.Json.Linq.JProperty("result", new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("tools", tools) )) ); return response.ToString(Newtonsoft.Json.Formatting.None); }</set-body> </return-response> </when> <when condition="@{ var body = context.Request.Body.As<Newtonsoft.Json.Linq.JObject>(preserveContent: true); if (body == null) { return false; } var rpcMethod = (string)body["method"]; var toolName = (string)body["params"]?["name"]; return string.Equals(rpcMethod, "tools/call", System.StringComparison.OrdinalIgnoreCase) && string.Equals(toolName, "microsoft_code_sample_search", System.StringComparison.OrdinalIgnoreCase); }"> <return-response> <set-status code="200" reason="OK" /> <set-header name="Content-Type" exists-action="override"> <value>application/json</value> </set-header> <set-body>@{ var req = context.Request.Body.As<Newtonsoft.Json.Linq.JObject>(preserveContent: true); var id = req != null ? req["id"] : Newtonsoft.Json.Linq.JValue.CreateNull(); var error = new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("jsonrpc", "2.0"), new Newtonsoft.Json.Linq.JProperty("id", id), new Newtonsoft.Json.Linq.JProperty("error", new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("code", -32601), new Newtonsoft.Json.Linq.JProperty("message", "Tool disabled by API Management policy: microsoft_code_sample_search") )) ); return error.ToString(Newtonsoft.Json.Formatting.None); }</set-body> </return-response> </when> </choose> </inbound> <backend /> <outbound /> <on-error /> </policies> It's a chunk of XML, but it's doing only three things. Here's what each part is for. The outer <choose> and the two <when> branches. This is just an if/else if. APIM peeks at the inbound JSON-RPC request and decides which branch to fire. The first branch handles tools/list. The second handles tools/call for the specific tool I want to block. If neither matches, nothing happens in the policy and the request flows through to the backend as normal. So tools/call for the allowed tools, plus initialize, ping, and anything else MCP throws at the gateway, all pass through untouched. The tools/list branch: synthesise the catalogue. When the agent asks for the tool list, APIM never forwards the call. Instead, the policy reads the id off the incoming request (so the response correlates correctly), builds a JArray containing only the tools I want to expose, with their names, descriptions, and inputSchema blocks, wraps that in a JSON-RPC result envelope, and returns it directly with <return-response>. The backend never sees the request. That's the bit that makes Option A bulletproof: there is no upstream behaviour that can leak through, because the upstream isn't involved. The tools/call branch: hard-block the disallowed tool. When the agent tries to call microsoft_code_sample_search directly (whether it guessed at it, cached it from a previous run, or someone added it to a config file), the policy short-circuits with a JSON-RPC error. The error code is -32601, which the spec defines as "Method not found", and the message says plainly that the tool was disabled by APIM. Well-behaved agents will surface that to the user and move on. Without this branch, hiding the tool from tools/list would only stop honest clients. A couple of incidental things worth knowing. The preserveContent: true argument on context.Request.Body.As<...>() is important: without it, reading the body consumes it and the rest of the pipeline gets nothing. And the OrdinalIgnoreCase comparisons are belt-and-braces: MCP method names are case-sensitive in the spec, but agents in the wild are inconsistent. Option B: the deny-list policy Same scenario, different approach. Let tools/list flow through to the backend and rewrite the response on the way out. Still block tools/call for the disallowed tool. ```xml <policies> <inbound> <choose> <when condition="@{ var body = context.Request.Body.As<Newtonsoft.Json.Linq.JObject>(preserveContent: true); if (body == null) { return false; } var rpcMethod = (string)body["method"]; var toolName = (string)body["params"]?["name"]; return string.Equals(rpcMethod, "tools/call", System.StringComparison.OrdinalIgnoreCase) && string.Equals(toolName, "microsoft_code_sample_search", System.StringComparison.OrdinalIgnoreCase); }"> <return-response> <set-status code="200" reason="OK" /> <set-header name="Content-Type" exists-action="override"> <value>application/json</value> </set-header> <set-body>@{ var req = context.Request.Body.As<Newtonsoft.Json.Linq.JObject>(preserveContent: true); var id = req != null ? req["id"] : Newtonsoft.Json.Linq.JValue.CreateNull(); var error = new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("jsonrpc", "2.0"), new Newtonsoft.Json.Linq.JProperty("id", id), new Newtonsoft.Json.Linq.JProperty("error", new Newtonsoft.Json.Linq.JObject( new Newtonsoft.Json.Linq.JProperty("code", -32601), new Newtonsoft.Json.Linq.JProperty("message", "Tool disabled by API Management policy: microsoft_code_sample_search") )) ); return error.ToString(Newtonsoft.Json.Formatting.None); }</set-body> </return-response> </when> </choose> </inbound> <backend /> <outbound> <choose> <when condition="@{ var req = context.Request.Body.As<Newtonsoft.Json.Linq.JObject>(preserveContent: true); if (req == null) { return false; } var rpcMethod = (string)req["method"]; return string.Equals(rpcMethod, "tools/list", System.StringComparison.OrdinalIgnoreCase); }"> <set-body>@{ var bodyText = context.Response.Body.As<string>(preserveContent: true); if (string.IsNullOrEmpty(bodyText)) { return bodyText; } var resp = Newtonsoft.Json.Linq.JObject.Parse(bodyText); var tools = resp["result"]?["tools"] as Newtonsoft.Json.Linq.JArray; if (tools == null) { return bodyText; } var filtered = new Newtonsoft.Json.Linq.JArray(); foreach (var t in tools) { var name = (string)t?["name"]; if (!string.Equals(name, "microsoft_code_sample_search", System.StringComparison.OrdinalIgnoreCase)) { filtered.Add(t); } } ((Newtonsoft.Json.Linq.JObject)resp["result"])["tools"] = filtered; return resp.ToString(Newtonsoft.Json.Formatting.None); }</set-body> </when> </choose> </outbound> <on-error /> </policies> This policy splits the work across inbound and outbound. The shape is different from Option A, but again it's only doing a few things. Inbound: block direct calls to the disallowed tool. This block is identical to the one in Option A. If the agent tries to invoke microsoft_code_sample_search directly, APIM returns the JSON-RPC -32601 error and the request never reaches the backend. Everything else (including tools/list and calls to the allowed tools) carries on through to the upstream MCP server. Backend: nothing custom. The empty <backend /> is deliberate. We want the upstream server to handle every request that wasn't blocked in inbound, including tools/list. APIM forwards it, the server returns its full catalogue, and we get a chance to rewrite the response on the way back. Outbound: filter the catalogue on the way out. This is the bit that does the dynamic work. The <when> condition checks the original request (not the response) to see if this was a tools/list call, because we only want to rewrite responses to that specific method. If it was, the policy reads the response body as a string, parses it as JSON, walks the result.tools array, builds a new array containing everything except the deny-listed tool, swaps it back into the response object, and writes the modified JSON back with <set-body>. For any other method, the response flows through untouched. The reason this option is more fragile than Option A is right there in that last paragraph: it reads and rewrites context.Response.Body. Microsoft's own guidance for MCP policies flags that response-body access can interfere with streaming. For non-streaming JSON-RPC like tools/list, this works fine in practice, but if you ever extend this pattern to filter streamed responses (resource updates, long-running tool calls), you'll need to think much harder about it. For the narrow case of trimming tools/list, it's a reasonable trade for the lower maintenance cost. Validating it works Once the policy is saved, hit the APIM-exposed MCP endpoint with the agent of your choice and check three things: tools/list returns only microsoft_docs_search and microsoft_docs_fetch. The third tool should not appear. tools/call for microsoft_code_sample_search returns a JSON-RPC error with code -32601. The two allowed tools still actually work end-to-end. Which one should you use Default to Option A (the static allowlist) unless you have a specific reason not to. It's the most predictable and it doesn't touch the response body, which keeps you well clear of the streaming caveat. You will have to update it when the upstream tool catalogue changes, but this does give you greater control of when you make those changes available and creates a conscious decision to allow them, or not. Reach for Option B when the upstream changes frequently, or when you'd rather block specific known-bad tools and let everything else through. Test it carefully and watch the agent's behaviour for any sign of streaming or framing issues. If you've got a fully internal MCP server you control, the cleanest answer is to fix the tool list at the source and skip the gateway changes altogether. But for third-party MCP servers, this is an approach you can apply without changing the underlying MCP server.You Can Build a Framework-Agnostic AI Gateway on Azure App Service — Here's How
The agent infrastructure conversation moved this year. In October 2025, AWS shipped Amazon Bedrock AgentCore — a managed agent runtime with per-session microVM isolation, built-in long-term memory, native MCP support, and an opinionated policy engine. A few months earlier, Cloudflare shipped its Agents SDK on top of Durable Objects, betting that edge-native stateful agents are the future. Both bets are real, both are interesting, and both arrive as closed, proprietary runtimes. So: what's Azure's answer? It's a question I've heard a couple times from architects in the last six months. The honest answer is that Azure already has the pieces. They don't ship as one product called AgentRuntime, and that's actually the point. Azure's pitch is composable: App Service + API Management + MCP, three services you already have access to, glued together with open standards. This post walks through a runnable sample of that composition. One App Service hosting both an agent (built with the Microsoft Agent Framework) and the stateless MCP server it calls, fronted by Azure API Management with the AI Gateway policy set — semantic caching, token rate limiting, per-subscription token emission for chargeback. One azd up deploys the lot. Repo: app-service-ai-gateway-mcp-apim-python. The headline claim is in the title. The point I actually want to make is the one underneath it: the framework is replaceable, the gateway is the contribution. Swap the Agent Framework module for Pydantic AI or LangGraph and the rest of the architecture is unchanged. That's what "run anything" means, made literal. The composable stack ┌────────────────────────────────────────────────┐ │ Azure API Management │ MCP / Agent ──┤ AI Gateway policies: │ client │ • llm-token-limit │ │ • llm-semantic-cache-lookup / store │ │ • llm-emit-token-metric │ │ • rate-limit-by-key (MCP API) │ └─────────────┬───────────────────┬──────────────┘ │ │ ┌────────────────▼──┐ ┌────────────▼──────────────┐ │ Azure OpenAI │ │ Azure App Service │ │ • chat model │ │ FastAPI app: │ │ • embedding │ │ • /mcp (stateless) │ │ model │ │ • /agent/chat │ └───────────────────┘ │ Managed identity → │ │ APIM (via subscription) │ └────────────┬──────────────┘ ▼ Application Insights (cloud_RoleInstance, APIM token metrics) Three observations that drive everything else: APIM is the only thing that talks to Azure OpenAI. The App Service agent doesn't have an AOAI key. It has an APIM subscription key. Every LLM call passes through the gateway, picks up the policies, and gets logged with consistent dimensions. That's where the governance part lives. The agent runtime is App Service. Linux, Python, FastAPI. Any language. Any framework. Pick your tool. We use Microsoft Agent Framework because it just GA'd and the API is clean, but the agent module is the easiest thing in the stack to swap. The MCP server is co-located with the agent. Same App Service, different route. The agent calls its own tools either in-process (fast path) or back out through APIM (so MCP traffic gets rate-limited and observed too). That choice is one environment variable. What the sample actually does The FastAPI app exposes three routes that matter: /mcp — a stateless HTTP MCP server (protocol revision 2025-11-25 ), implementing four tools: whoami , echo , lookup_fact , and summarize_app_service_doc . Any MCP client (Claude, VS Code, your own agent runtime) can connect. /agent/chat — a Microsoft Agent Framework agent that uses those same MCP tools as its tool set, and calls AOAI through APIM. /health and / — the boring but essential supporting cast (health check for App Service probes, status page showing the serving instance ID). Here's the agent definition. The key line is the endpoint: from agent_framework.openai import OpenAIChatCompletionClient client = OpenAIChatCompletionClient( azure_endpoint=os.environ["APIM_GATEWAY_URL"], # ← APIM, not AOAI model=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT"], api_version="2024-10-21", api_key=os.environ["APIM_SUBSCRIPTION_KEY"], default_headers={"Ocp-Apim-Subscription-Key": os.environ["APIM_SUBSCRIPTION_KEY"]}, ) agent = client.as_agent( name="AppServiceExpert", instructions=SYSTEM_INSTRUCTIONS, tools=build_tools(), ) That's it. The agent has no idea APIM exists. It thinks it's talking to AOAI. APIM is doing every interesting thing — auth, caching, throttling, metric emission — without the agent code knowing or caring. The policy that does the heavy lifting The AOAI API in APIM has one policy attached at the API scope. The full XML is in infra/apim/policies/aoai-policy.xml; here's the bones of it: <policies> <inbound> <base /> <authentication-managed-identity resource="https://cognitiveservices.azure.com" output-token-variable-name="aoai-token" /> <set-header name="Authorization" exists-action="override"> <value>@("Bearer " + (string)context.Variables["aoai-token"])</value> </set-header> <azure-openai-token-limit counter-key="@(context.Subscription?.Id ?? "anonymous")" tokens-per-minute="50000" estimate-prompt-tokens="true" /> <azure-openai-semantic-cache-lookup score-threshold="0.85" embeddings-backend-id="aoai-embeddings-backend" embeddings-backend-auth="system-assigned"> <vary-by>@(context.Subscription?.Id ?? "anonymous")</vary-by> </azure-openai-semantic-cache-lookup> <set-backend-service backend-id="aoai-backend" /> <azure-openai-emit-token-metric namespace="ai-gateway"> <dimension name="Subscription ID" value="@(context.Subscription?.Id ?? "anonymous")" /> <dimension name="API ID" value="@(context.Api.Id)" /> <dimension name="Operation ID" value="@(context.Operation.Id)" /> <dimension name="Client IP" value="@(context.Request.IpAddress)" /> </azure-openai-emit-token-metric> </inbound> <outbound> <base /> <azure-openai-semantic-cache-store duration="3600" /> </outbound> </policies> Four things are happening here that would otherwise be your problem: Auth to AOAI. APIM's managed identity holds the Cognitive Services OpenAI User role on the AOAI account. No keys. Token rate limiting. Each APIM subscription gets a tokens-per-minute budget. One runaway team can't starve everyone else. Semantic caching. The inbound policy embeds the prompt using the embedding deployment, queries the Redis-backed APIM cache for a vector match above the 0.85 threshold, and short-circuits the AOAI call on a hit. The outbound azure-openai-semantic-cache-store writes successful misses back. Per-call metric emission. Every call pushes PromptTokens , CompletionTokens , and TotalTokens to Application Insights as custom metrics tagged with the APIM subscription, the API, the operation, and the client IP. That's your chargeback dashboard, ready to query. The whole thing is XML. None of it is in your agent code. Deploying it azd auth login azd up azd up provisions a P0v3 App Service Plan with the web app and a staging slot, an AOAI account with gpt-4o-mini + text-embedding-3-small deployments, an APIM Developer SKU service with the two APIs and the policy XML wired up, an Azure Cache for Redis Basic C0 as the semantic-cache store, and a Log Analytics workspace + Application Insights. The postprovision hook fetches the APIM subscription key for the AI Gateway product and writes it into the App Service's APIM_SUBSCRIPTION_KEY setting (and the staging slot's, so slot swaps are clean). Be patient. Developer SKU APIM takes 30–45 minutes the first time. If you want to prototype faster, the sample supports Consumption SKU as a one-line flip: azd env set APIM_SKU Consumption azd provision Consumption provisions in about a minute and is great for sketching. Verify your specific policies are supported there before you ship it. Governing it like a grown-up The toy version of this post stops at "look, semantic cache works." The version your platform engineering lead wants to see goes further. Per-team chargeback. The token-emit policy tags every call with the APIM subscription ID. Issue one subscription per team, hand it over with a quota, and your monthly chargeback report is a KQL query: customMetrics | where timestamp > startofmonth(now()) | where name == "TotalTokens" | summarize Tokens=sum(valueSum) by Team=tostring(customDimensions["Subscription ID"]) | extend USD = Tokens * 0.00015 / 1000 // gpt-4o-mini blended rate | order by USD desc Content safety as a policy plug-in. Add an llm-content-safety block to the inbound policy and point it at an Azure AI Content Safety resource — every prompt and response gets moderated before reaching agents or end users. The sample doesn't deploy Content Safety by default (to keep the demo cost-free), but the README has the one-line bicep + one-block policy delta. Circuit breaker + multi-region failover. Add a second AOAI backend in a different region and an APIM backend pool, give the pool a circuit-breaker rule, and your agents inherit failover with zero code changes. Rate-limit MCP traffic too. The MCP API has its own policy with rate-limit-by-key , so a runaway agent can't pin the MCP server with a hot loop. None of these are gymnastics. They're one policy block each. The pattern is the same every time: write policy at the gateway, leave the agent code alone. Proving it works After azd up finishes, two checks. First, hit the agent endpoint: curl -sS -X POST "$(azd env get-value WEB_URI)/agent/chat" \ -H 'Content-Type: application/json' \ -d '{"message": "How does App Service horizontally scale an MCP server?"}' | jq You should see a reply that cites the instance ID (the agent calls whoami and summarize_app_service_doc to ground its answer) and a tool_calls array showing the agent's reasoning trace. Second, run the k6 load test: export BASE_URL="$(azd env get-value WEB_URI)" export APIM_SUBSCRIPTION_KEY="$(azd env get-value APIM_SUBSCRIPTION_KEY)" k6 run loadtest/k6-gateway.js The script hits /agent/chat with a small pool of semantically-similar prompts. After a 30-second warmup, the steady phase should report a cache-hit ratio above 30%: APIM AI Gateway — k6 summary ───────────────────────────── Cache hits : 412 Cache misses : 88 Hit ratio : 82.4% Cross-check in App Insights: ApiManagementGatewayLogs | where TimeGenerated > ago(15m) | where ApiId == "aoai" | extend cache = tostring(parse_json(ResponseHeaders)["x-llm-cache-status"]) | summarize count() by cache, bin(TimeGenerated, 1m) | render columnchart A solid bar of hits next to a smaller bar of misses is the gateway earning its keep. "Run anything" — the proof Here's the part where I cash the check the title wrote. The agent module is the easiest thing in this stack to replace. Three changes to ship the same demo on Pydantic AI: # requirements.txt - agent-framework-core==1.5.0 - agent-framework-openai==1.5.0 + pydantic-ai==0.4.0 # agent/agent.py from pydantic_ai import Agent from pydantic_ai.models.openai import OpenAIModel def build_agent(): model = OpenAIModel( "gpt-4o-mini", base_url=f"{os.environ['APIM_GATEWAY_URL']}/openai/deployments/gpt-4o-mini", api_key=os.environ["APIM_SUBSCRIPTION_KEY"], ) return Agent(model, system_prompt=SYSTEM_INSTRUCTIONS, tools=build_tools()) That's it. build_tools() returns the same list of async callables (Pydantic AI accepts plain Python functions as tools, same as Agent Framework). LangGraph works the same way — wire build_tools() into a ToolNode , point ChatOpenAI at the APIM gateway URL, done. Every APIM policy still fires. Every token metric still emits. Every cache hit still hits. The gateway is the boundary; the runtime above it is fungible. What AgentCore gets right I want to land this without spin. AgentCore's per-session microVM isolation is genuinely interesting — it's a stronger sandboxing story than running multiple agents in shared App Service workers, and it matters for multi-tenant SaaS where agents execute arbitrary user code or call third-party tools you don't fully trust. The managed long-term memory primitive is also a real convenience; Azure has the building blocks (Cosmos DB, AI Search, Cognitive Search) but they aren't pre-wired into a single "agent memory" API the way AgentCore's are. Where the App Service + APIM + MCP composition genuinely wins: Open standards. MCP is a public protocol with implementations across the industry. AgentCore's tool layer is AWS-native. No new runtime to learn. App Service is the same App Service. Your existing CI/CD, your existing security review, your existing monitoring all apply. Bring your own framework. Pydantic AI, LangGraph, Agent Framework, Semantic Kernel, AutoGen, CrewAI — they all work, because the App Service doesn't care what's running inside the container. Existing enterprise footprint. VNet integration, private endpoints, managed identity, deployment slots, sidecars, Easy Auth. None of it is new for App Service. You inherit a decade of platform work. The right framing isn't "Azure's answer to AgentCore." It's that Azure is making a different bet: that enterprises will value the composability of services they already trust over the convenience of a new proprietary runtime. For some, that bet is probably correct. For a few — multi-tenant agent marketplaces, untrusted code execution — AgentCore's isolation model is a better fit. Pick the one that matches your threat model. What's next If you ship the sample and want to compare notes, the repo is at app-service-ai-gateway-mcp-apim-python.Transforming Retry-After Headers in Azure APIM: A Step-by-Step Guide
In this blog post, you'll learn how to customize the Retry-After response header in Azure API Management (APIM) rate-limiting policies, enhancing your API's flexibility and user experience. While it does not delve into the specifics of the rate-limit or rate-limit-by-key policies, it provides a practical guide for altering the Retry-After header. For detailed information on the rate-limit policy, please visit Azure API Management policy reference - rate-limit | Microsoft Learn. Understanding Rate Limiting: Protecting Your API from Overuse Rate limiting is a technique used to control how often requests are made to a resource. It helps prevent excessive or abusive use and ensures the resource is available to all users. Rate limiting is often used to protect against denial-of-service (DoS) attacks, which aim to overwhelm a network or server with too many requests, making it unavailable to legitimate users. It can also limit the number of requests from individual users to prevent a single user or group from monopolizing the resource. Azure API Management Rate Limit Policies In Azure, access to APIs is controlled using the following API Management policies: rate-limit rate-limit-by-key The implementation of these policies is straightforward but somewhat limited and less flexible, in my opinion. The Default Retry-After Header: What You Need to Know In the Azure APIM rate-limit policy documentation, it is mentioned that once the client's requests are throttled, the service starts returning a response header containing the time interval (in seconds) after which the client should retry the request. The default name of the header is Retry-After, and this name can be customized. For example: Retry-After: 60 However, in one use case for a customer, there was a requirement to provide a timestamp instead of a time interval as a header value. For example: Retry-After: 2020-05-04T12:23:41.6181792Z To implement this, the header value needs to change, but this is something that the rate-limit policy does not support. Customizing the Retry-After Header The basis for changing the response header value lies in the on-error scope. You can implement a policy like the following: <inbound> <base> <rate-limit-by-key calls="1000" renewal-period="60" counter-key="@(context.Request.IpAddress)" increment-condition="@(context.Response.StatusCode == 200)" remaining-calls-variable-name="remainingCallsPerIP" retry-after-header-name="Retry-After" remaining-calls-header-name="Requests-Remaining" retry-after-variable-name="retryAfter"> </rate-limit-by-key></inbound> <on-error> <choose> <when condition="@(context.LastError.Reason == " ratelimitexceeded")"=""> <set-header name="Retry-After" exists-action="override"> <value>@(DateTime.UtcNow.AddSeconds(context.Variables.GetValueOrDefault<int>("retryAfter")).ToString("o"))</int></value> </set-header> </when> </choose> <base> </on-error> Please refer to APIM predefined errors for policies here: Error handling in Azure API Management policies | Microsoft Learn Here, the key point is that whenever the APIM rate limit is reached, an error occurs, which is then captured in the on-error scope. To set or override the response header only in rate-limiting scenarios, you need to filter using the RateLimitExceeded error reason. After that, the exact error value is determined by adding the current UTC timestamp with the value of the retryAfter variable in seconds. With this, you have now customized the Retry-After header with a timestamp instead of a time interval (in seconds). Conclusion In conclusion, customizing the Retry-After response header in Azure API Management can significantly enhance the flexibility and user experience of your API services. By leveraging the on-error scope and handling the RateLimitExceeded error, you can provide a more informative and user-friendly response to clients when rate limits are exceeded. This approach not only meets specific customer requirements but also demonstrates the adaptability of Azure APIM in handling various scenarios. With these steps, you can ensure that your API remains robust, efficient, and user centric.Exciting Updates Coming to Conversational Diagnostics (Public Preview)
Last year, at Ignite 2023, we unveiled Conversational Diagnostics (Preview), a revolutionary tool integrated with AI-powered capabilities to enhance problem-solving for Windows Web Apps. This year, we're thrilled to share what’s new and forthcoming for Conversational Diagnostics (Preview). Get ready to experience a broader range of functionalities and expanded support across various Azure Products, making your troubleshooting journey even more seamless and intuitive.
Azure Logic App workflow (Standard) Resubmit and Retry
Hello Experts, A workflow is scheduled to run daily at a specific time and retrieves data from different systems using REST API Calls (8-9). The data is then sent to another system through API calls using multiple child flows. We receive more than 1500 input data, and for each data, an API call needs to be made. During the API invocation process, there is a possibility of failure due to server errors (5xx) and client errors (4xx). To handle this, we have implemented a "Retry" mechanism with a fixed interval. However, there is still a chance of flow failure due to various reasons. Although there is a "Resubmit" feature available at the action level, I cannot apply it in this case because we are using multiple child workflows and the response is sent back from one flow to another. Is it necessary to utilize the "Resubmit" functionality? The Retry Functionality has been developed to handle any Server API errors (5xx) that may occur with Connectors (both Custom and Standard), including client API errors 408 and 429. In this specific scenario, it is reasonable to attempt retrying or resubmitting the API Call from the Azure Logic Apps workflow. Nevertheless, there are other situations where implementing the retry and resubmit logic would result in the same error outcome. Is it acceptable to proceed with the Retry functionality in this particular scenario? It would be highly appreciated if you could provide guidance on the appropriate methodology. Thanks -Sri
