In May, we built an APIM-based AI Gateway for an App Service agent. Foundry can now create or associate that gateway directly—without changing the application architecture.
In May, I published a runnable sample that put Azure API Management in front of an AI agent on Azure App Service. APIM handled token limits, semantic caching, token metrics, and access to the model. The point was simple: keep the agent framework interchangeable and put the production controls at the gateway.
A recent Apps on Azure post, From AI Adoption to AI Governance, arrived at the same architectural conclusion from a platform-governance angle. It uses APIM to turn model identity and token consumption into a per-model chargeback signal for Microsoft Foundry workloads.
That is useful validation, but the more interesting update is in the product itself: Microsoft Foundry can now create or associate an APIM-based AI Gateway directly from its Admin console. The data path is still APIM. What changed is who can configure and govern it.
This post explains what that means for an App Service-hosted agent, what Foundry now manages, what still belongs in APIM, and what I would change in the sample we built earlier.
Update (August 2026): Azure API Management now offers a dedicated AI Gateway tier in public preview. This article describes Foundry’s integration with APIM-based gateways. The new tier adds an AI-focused portal and resource experience for models and MCP tools and is expected to integrate with Foundry separately. See the announcement.
The short version: same gateway, new control plane
The architecture is still familiar:
Client -> App Service agent -> APIM AI Gateway -> Foundry model
APIM remains the traffic gateway. It authenticates callers, applies policies, forwards requests, and emits telemetry. App Service remains the application runtime for the agent and its APIs.
The new piece is the Foundry control plane. From Operate > Admin console > AI Gateway, a platform team can now:
- Create a new APIM instance or associate a supported existing instance.
- Enable the gateway for individual Foundry projects.
- Apply independent token limits to projects sharing the gateway.
- Verify gateway traffic and inspect logs.
Once the gateway is enabled, separate Foundry experiences can also register custom agents and govern supported MCP tools. Those actions do not happen in the AI Gateway tab itself, but the gateway provides their governed traffic path.
That is a meaningful shift. In the original sample, the infrastructure owner created APIM, imported the model API, deployed policy XML, and wired the agent to the gateway. Foundry does not remove APIM or replace those advanced capabilities. It gives Foundry administrators a first-party path into the same architecture.
What Foundry now owns
Project onboarding
An AI Gateway is associated with a Foundry resource, then enabled for projects inside it. New projects can inherit the gateway automatically. Existing projects must be added explicitly.
This creates a useful governance boundary: multiple teams can share the same APIM instance without sharing one undifferentiated token pool. Each project can receive its own token ceiling. If a project exceeds its configured limit, the gateway returns 429 Too Many Requests while other projects continue using their allocations.
Foundry-level limits
Our earlier sample used an APIM token-limit policy keyed by an APIM subscription. That is still a valid pattern, especially when the consumer boundary is an application, business unit, or external customer.
The Foundry integration adds another natural counter key: the Foundry project. For organizations already organizing models and agents by project, limits can now follow that structure instead of being recreated independently in every client application.
Inventory for agents and tools
The AI Gateway experience is expanding beyond model endpoints in two distinct ways. Foundry can register custom agents running outside Foundry, including agents that use A2A as their communication protocol. Separately, AI Gateway can govern supported MCP tools. The current tools-governance preview is scoped to MCP; it does not cover every Foundry or OpenAPI tool type.
That matters for App Service because the runtime does not need to move. An agent can continue running on App Service while participating in Foundry's inventory, and its supported MCP traffic can flow through the gateway.
These experiences are still preview features, so I would treat them as a control-plane integration to evaluate, not a reason to redesign a working production agent.
What still belongs in APIM
Foundry makes the common path easier. It does not make the full APIM surface unnecessary.
I would still use APIM directly for:
- Custom policy expressions and organization-specific headers or claims.
- Semantic caching backed by Azure Managed Redis.
- Backend pools, priority routing, and circuit breakers.
- Private networking, multi-region gateways, and advanced topology decisions.
- Custom metrics and dimensions needed for internal chargeback.
The clean mental model is: Foundry manages the AI assets and common project guardrails; APIM manages the traffic contract and advanced gateway behavior.
Applying this to the App Service sample
The existing framework-agnostic AI Gateway sample hosts a Microsoft Agent Framework agent and an MCP server on one Linux App Service. The agent reaches its model through APIM, which applies token limits, semantic caching, token metrics, and managed-identity authentication.
Most of that architecture should stay exactly as it is.
1. Keep the App Service application unchanged
The agent should continue calling a stable gateway endpoint. It should not need to understand whether APIM was provisioned by Bicep, associated in Foundry, or managed by a central platform team. That separation was the reason to introduce the gateway in the first place.
2. Choose the APIM ownership model deliberately
Update (August 2026): Foundry can now associate eligible API Management instances on classic (v1) SKUs as AI Gateways, in addition to v2 instances. If you already operate a classic-tier gateway, check Use existing APIM before planning a migration. Association does not change that tier’s SLA, scaling, networking, or feature limitations.
For a new proof of concept, Foundry can create a Basic v2 APIM instance. For production, Microsoft recommends evaluating Standard v2 or Premium v2 based on throughput and networking requirements.
For an existing enterprise gateway, use the Foundry option to associate an APIM instance only after confirming that it meets the eligibility requirements: it must be in the same Microsoft Entra tenant and Azure subscription as the Foundry resource, you need the API Management Service Contributor role (or Owner), and the instance must use a currently supported APIM tier.
3. Add existing projects explicitly
Creating or associating the gateway at the Foundry resource level does not automatically enable every existing project. Add each existing project to the gateway, then verify its status is Enabled.
4. Decide where each limit belongs
Avoid layering limits without a clear ownership model. A Foundry project limit, an APIM subscription limit, and a model deployment TPM limit answer different questions:
| Control | Best boundary | Question it answers |
|---|---|---|
| Foundry project limit | Team or workload project | How much shared capacity can this project consume? |
| APIM policy limit | Subscription, user, tenant, or application | How much can this specific consumer use? |
| Model deployment quota | Backend deployment | What capacity exists at the model endpoint? |
Use all three when the boundaries are intentional. Otherwise, operators end up debugging a 429 without knowing which layer produced it.
5. Verify the data path
After enabling a project, make a model request and confirm that the APIM request metric increments. Microsoft also recommends checking the GatewayLogs table for a successful response and an API name matching the AI Gateway.
Then test the failure path. Set a deliberately small project limit, exceed it, and confirm that the gateway returns 429. A governance feature is not finished until the team has observed both the allowed and denied behavior.
Which path should you choose?
| Scenario | Recommended approach |
|---|---|
| New Foundry proof of concept | Create the AI Gateway from Foundry and start with project-level limits. |
| Existing v2 enterprise APIM | Associate the existing instance and preserve central networking and policy ownership. |
| Existing classic-tier APIM |
Associate the existing instance with Foundry if eligible; migrate only when production, networking, or scale requirements justify it. |
| Advanced routing or custom policy needs | Use Foundry for inventory and common guardrails, then manage the advanced behavior in APIM. |
| Strict isolation between project groups | Use separate Foundry resources and separate gateways rather than assuming one gateway per project. |
The architecture was right; the experience caught up
The most important takeaway is not that everyone should rebuild an AI Gateway in a new portal. It is that the composable architecture now has a more accessible control plane.
App Service still runs the agent. APIM still governs the traffic. Foundry now gives model and platform owners a direct way to connect projects, allocate capacity, and bring agents and tools into the same governance view.
If you already built the earlier sample, the application boundary does not need to change. Evaluate the Foundry integration, decide whether its project model matches your organization, and check the APIM tier before planning any migration. The gateway remains the contribution; Foundry now makes it easier to operate.