You Can Build a Framework-Agnostic AI Gateway on Azure App Service — Here's How

The agent infrastructure conversation moved this year. In October 2025, AWS shipped Amazon Bedrock AgentCore — a managed agent runtime with per-session microVM isolation, built-in long-term memory, native MCP support, and an opinionated policy engine. A few months earlier, Cloudflare shipped its Agents SDK on top of Durable Objects, betting that edge-native stateful agents are the future. Both bets are real, both are interesting, and both arrive as closed, proprietary runtimes.

So: what's Azure's answer?

It's a question I've heard a couple times from architects in the last six months. The honest answer is that Azure already has the pieces. They don't ship as one product called AgentRuntime, and that's actually the point. Azure's pitch is composable: App Service + API Management + MCP, three services you already have access to, glued together with open standards.

This post walks through a runnable sample of that composition. One App Service hosting both an agent (built with the Microsoft Agent Framework) and the stateless MCP server it calls, fronted by Azure API Management with the AI Gateway policy set — semantic caching, token rate limiting, per-subscription token emission for chargeback. One azd up deploys the lot. Repo: app-service-ai-gateway-mcp-apim-python.

The headline claim is in the title. The point I actually want to make is the one underneath it: the framework is replaceable, the gateway is the contribution. Swap the Agent Framework module for Pydantic AI or LangGraph and the rest of the architecture is unchanged. That's what "run anything" means, made literal.

The composable stack

                 ┌────────────────────────────────────────────────┐
                 │           Azure API Management                 │
   MCP / Agent ──┤  AI Gateway policies:                          │
   client        │    • llm-token-limit                           │
                 │    • llm-semantic-cache-lookup / store         │
                 │    • llm-emit-token-metric                     │
                 │    • rate-limit-by-key (MCP API)               │
                 └─────────────┬───────────────────┬──────────────┘
                               │                   │
              ┌────────────────▼──┐   ┌────────────▼──────────────┐
              │  Azure OpenAI     │   │  Azure App Service        │
              │   • chat model    │   │   FastAPI app:            │
              │   • embedding     │   │     • /mcp (stateless)    │
              │     model         │   │     • /agent/chat         │
              └───────────────────┘   │   Managed identity →      │
                                      │   APIM (via subscription) │
                                      └────────────┬──────────────┘
                                                   ▼
                                        Application Insights
                                        (cloud_RoleInstance,
                                         APIM token metrics)

Three observations that drive everything else:

1. APIM is the only thing that talks to Azure OpenAI. The App Service agent doesn't have an AOAI key. It has an APIM subscription key. Every LLM call passes through the gateway, picks up the policies, and gets logged with consistent dimensions. That's where the governance part lives.
2. The agent runtime is App Service. Linux, Python, FastAPI. Any language. Any framework. Pick your tool. We use Microsoft Agent Framework because it just GA'd and the API is clean, but the agent module is the easiest thing in the stack to swap.
3. The MCP server is co-located with the agent. Same App Service, different route. The agent calls its own tools either in-process (fast path) or back out through APIM (so MCP traffic gets rate-limited and observed too). That choice is one environment variable.

What the sample actually does

The FastAPI app exposes three routes that matter:

• /mcp — a stateless HTTP MCP server (protocol revision 2025-11-25), implementing four tools: whoami, echo, lookup_fact, and summarize_app_service_doc. Any MCP client (Claude, VS Code, your own agent runtime) can connect.
• /agent/chat — a Microsoft Agent Framework agent that uses those same MCP tools as its tool set, and calls AOAI through APIM.
• /health and / — the boring but essential supporting cast (health check for App Service probes, status page showing the serving instance ID).

Here's the agent definition. The key line is the endpoint:

from agent_framework.openai import OpenAIChatCompletionClient

client = OpenAIChatCompletionClient(
    azure_endpoint=os.environ["APIM_GATEWAY_URL"],   # ← APIM, not AOAI
    model=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT"],
    api_version="2024-10-21",
    api_key=os.environ["APIM_SUBSCRIPTION_KEY"],
    default_headers={"Ocp-Apim-Subscription-Key": os.environ["APIM_SUBSCRIPTION_KEY"]},
)

agent = client.as_agent(
    name="AppServiceExpert",
    instructions=SYSTEM_INSTRUCTIONS,
    tools=build_tools(),
)

That's it. The agent has no idea APIM exists. It thinks it's talking to AOAI. APIM is doing every interesting thing — auth, caching, throttling, metric emission — without the agent code knowing or caring.

The policy that does the heavy lifting

The AOAI API in APIM has one policy attached at the API scope. The full XML is in infra/apim/policies/aoai-policy.xml; here's the bones of it:

<policies>
  <inbound>
    <base />
    <authentication-managed-identity resource="https://cognitiveservices.azure.com"
                                     output-token-variable-name="aoai-token" />
    <set-header name="Authorization" exists-action="override">
      <value>@("Bearer " + (string)context.Variables["aoai-token"])</value>
    </set-header>

    <azure-openai-token-limit
      counter-key="@(context.Subscription?.Id ?? "anonymous")"
      tokens-per-minute="50000"
      estimate-prompt-tokens="true" />

    <azure-openai-semantic-cache-lookup
      score-threshold="0.85"
      embeddings-backend-id="aoai-embeddings-backend"
      embeddings-backend-auth="system-assigned">
      <vary-by>@(context.Subscription?.Id ?? "anonymous")</vary-by>
    </azure-openai-semantic-cache-lookup>

    <set-backend-service backend-id="aoai-backend" />

    <azure-openai-emit-token-metric namespace="ai-gateway">
      <dimension name="Subscription ID" value="@(context.Subscription?.Id ?? "anonymous")" />
      <dimension name="API ID" value="@(context.Api.Id)" />
      <dimension name="Operation ID" value="@(context.Operation.Id)" />
      <dimension name="Client IP" value="@(context.Request.IpAddress)" />
    </azure-openai-emit-token-metric>
  </inbound>

  <outbound>
    <base />
    <azure-openai-semantic-cache-store duration="3600" />
  </outbound>
</policies>

Four things are happening here that would otherwise be your problem:

1. Auth to AOAI. APIM's managed identity holds the Cognitive Services OpenAI User role on the AOAI account. No keys.
2. Token rate limiting. Each APIM subscription gets a tokens-per-minute budget. One runaway team can't starve everyone else.
3. Semantic caching. The inbound policy embeds the prompt using the embedding deployment, queries the Redis-backed APIM cache for a vector match above the 0.85 threshold, and short-circuits the AOAI call on a hit. The outbound azure-openai-semantic-cache-store writes successful misses back.
4. Per-call metric emission. Every call pushes PromptTokens, CompletionTokens, and TotalTokens to Application Insights as custom metrics tagged with the APIM subscription, the API, the operation, and the client IP. That's your chargeback dashboard, ready to query.

The whole thing is XML. None of it is in your agent code.

Deploying it

azd auth login
azd up

azd up provisions a P0v3 App Service Plan with the web app and a staging slot, an AOAI account with gpt-4o-mini + text-embedding-3-small deployments, an APIM Developer SKU service with the two APIs and the policy XML wired up, an Azure Cache for Redis Basic C0 as the semantic-cache store, and a Log Analytics workspace + Application Insights. The postprovision hook fetches the APIM subscription key for the AI Gateway product and writes it into the App Service's APIM_SUBSCRIPTION_KEY setting (and the staging slot's, so slot swaps are clean).

Be patient. Developer SKU APIM takes 30–45 minutes the first time. If you want to prototype faster, the sample supports Consumption SKU as a one-line flip:

azd env set APIM_SKU Consumption
azd provision

Consumption provisions in about a minute and is great for sketching. Verify your specific policies are supported there before you ship it.

Governing it like a grown-up

The toy version of this post stops at "look, semantic cache works." The version your platform engineering lead wants to see goes further.

Per-team chargeback. The token-emit policy tags every call with the APIM subscription ID. Issue one subscription per team, hand it over with a quota, and your monthly chargeback report is a KQL query:

customMetrics
| where timestamp > startofmonth(now())
| where name == "TotalTokens"
| summarize Tokens=sum(valueSum)
  by Team=tostring(customDimensions["Subscription ID"])
| extend USD = Tokens * 0.00015 / 1000   // gpt-4o-mini blended rate
| order by USD desc

Content safety as a policy plug-in. Add an llm-content-safety block to the inbound policy and point it at an Azure AI Content Safety resource — every prompt and response gets moderated before reaching agents or end users. The sample doesn't deploy Content Safety by default (to keep the demo cost-free), but the README has the one-line bicep + one-block policy delta.

Circuit breaker + multi-region failover. Add a second AOAI backend in a different region and an APIM backend pool, give the pool a circuit-breaker rule, and your agents inherit failover with zero code changes.

Rate-limit MCP traffic too. The MCP API has its own policy with rate-limit-by-key, so a runaway agent can't pin the MCP server with a hot loop.

None of these are gymnastics. They're one policy block each. The pattern is the same every time: write policy at the gateway, leave the agent code alone.

Proving it works

After azd up finishes, two checks. First, hit the agent endpoint:

curl -sS -X POST "$(azd env get-value WEB_URI)/agent/chat" \
  -H 'Content-Type: application/json' \
  -d '{"message": "How does App Service horizontally scale an MCP server?"}' | jq

You should see a reply that cites the instance ID (the agent calls whoami and summarize_app_service_doc to ground its answer) and a tool_calls array showing the agent's reasoning trace.

Second, run the k6 load test:

export BASE_URL="$(azd env get-value WEB_URI)"
export APIM_SUBSCRIPTION_KEY="$(azd env get-value APIM_SUBSCRIPTION_KEY)"
k6 run loadtest/k6-gateway.js

The script hits /agent/chat with a small pool of semantically-similar prompts. After a 30-second warmup, the steady phase should report a cache-hit ratio above 30%:

APIM AI Gateway — k6 summary
─────────────────────────────
Cache hits   : 412
Cache misses : 88
Hit ratio    : 82.4%

Cross-check in App Insights:

ApiManagementGatewayLogs
| where TimeGenerated > ago(15m)
| where ApiId == "aoai"
| extend cache = tostring(parse_json(ResponseHeaders)["x-llm-cache-status"])
| summarize count() by cache, bin(TimeGenerated, 1m)
| render columnchart

A solid bar of hits next to a smaller bar of misses is the gateway earning its keep.

"Run anything" — the proof

Here's the part where I cash the check the title wrote. The agent module is the easiest thing in this stack to replace. Three changes to ship the same demo on Pydantic AI:

# requirements.txt
- agent-framework-core==1.5.0
- agent-framework-openai==1.5.0
+ pydantic-ai==0.4.0

# agent/agent.py
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel

def build_agent():
    model = OpenAIModel(
        "gpt-4o-mini",
        base_url=f"{os.environ['APIM_GATEWAY_URL']}/openai/deployments/gpt-4o-mini",
        api_key=os.environ["APIM_SUBSCRIPTION_KEY"],
    )
    return Agent(model, system_prompt=SYSTEM_INSTRUCTIONS, tools=build_tools())

That's it. build_tools() returns the same list of async callables (Pydantic AI accepts plain Python functions as tools, same as Agent Framework). LangGraph works the same way — wire build_tools() into a ToolNode, point ChatOpenAI at the APIM gateway URL, done.

Every APIM policy still fires. Every token metric still emits. Every cache hit still hits. The gateway is the boundary; the runtime above it is fungible.

What AgentCore gets right

I want to land this without spin. AgentCore's per-session microVM isolation is genuinely interesting — it's a stronger sandboxing story than running multiple agents in shared App Service workers, and it matters for multi-tenant SaaS where agents execute arbitrary user code or call third-party tools you don't fully trust. The managed long-term memory primitive is also a real convenience; Azure has the building blocks (Cosmos DB, AI Search, Cognitive Search) but they aren't pre-wired into a single "agent memory" API the way AgentCore's are.

Where the App Service + APIM + MCP composition genuinely wins:

• Open standards. MCP is a public protocol with implementations across the industry. AgentCore's tool layer is AWS-native.
• No new runtime to learn. App Service is the same App Service. Your existing CI/CD, your existing security review, your existing monitoring all apply.
• Bring your own framework. Pydantic AI, LangGraph, Agent Framework, Semantic Kernel, AutoGen, CrewAI — they all work, because the App Service doesn't care what's running inside the container.
• Existing enterprise footprint. VNet integration, private endpoints, managed identity, deployment slots, sidecars, Easy Auth. None of it is new for App Service. You inherit a decade of platform work.

The right framing isn't "Azure's answer to AgentCore." It's that Azure is making a different bet: that enterprises will value the composability of services they already trust over the convenience of a new proprietary runtime. For some, that bet is probably correct. For a few — multi-tenant agent marketplaces, untrusted code execution — AgentCore's isolation model is a better fit. Pick the one that matches your threat model.

What's next

If you ship the sample and want to compare notes, the repo is at app-service-ai-gateway-mcp-apim-python.