What is Multi-Model Chaos, and what cost and security challenges does it pose?
Multi-model chaos describes the sprawl that emerges when an organization rapidly adopts many large language and foundation models—OpenAI, Anthropic, Meta Llama, Mistral, and a long tail of open-source and fine-tuned variants—across teams and applications without any unifying control plane. Instead of a single governed entry point, each team wires its own keys, endpoints, SDKs, and prompts directly to whichever provider it prefers, leaving the enterprise with a fragmented, duplicated, and largely invisible AI estate. On the cost side, this fragmentation makes spend almost impossible to predict or contain, identical workloads run against premium models when cheaper ones would suffice, token consumption goes unmeasured, redundant calls and missing caching inflate bills, and finance teams have no consolidated view to attribute usage back to a team, product, or customer. On the security and governance side, the risks compound: API keys are scattered across code and config files, sensitive or regulated data flows to external endpoints with no data-loss prevention or residency guarantees, prompt-injection and jailbreak attempts go unmonitored, and there is no centralized authentication, rate limiting, auditing, or content filtering. The net effect is an uncontrolled attack surface and a compliance blind spot—precisely the conditions that motivate consolidating model access behind a governed AI gateway.
In short, multi-model chaos trades short-term speed for runaway costs and an unmanaged security risk, making a governed AI Gateway essential.
What is a Governed AI Gateway, and how do they help reduce cost and improve security?
A governed AI gateway is an enterprise control plane built on Azure API Management (APIM) that consolidates every model behind a single, governed endpoint. It unifies Azure OpenAI (the gpt-5.4 family) and Azure AI Foundry (open-source and partner models such as grok-4.3 and DeepSeek-V4-Pro), so consumers reach any of them through one consistent, policy-enforced entry point rather than a tangle of direct connections. Every backend is password-less, authenticated through managed identity, which eliminates scattered API keys. On top of this foundation, the gateway enforces per-consumer model permissions, token-based rate limits, and cost-based budget downgrade—automatically routing teams to more economical models as they approach their spend limits—all administered from a self-service Admin UI.
- One governed endpoint for every backend. Azure OpenAI and Azure AI Foundry (OSS and partner) models are bundled behind a single governance endpoint. Each backend is reachable only over a private endpoint with key authentication disabled, so APIM authenticates using its own managed identity—no model keys ever live on the gateway.
- Per-consumer governance, edited live in the Admin UI with no redeployment:
- Allowed models — a consumer can call only the models explicitly granted to it; anything else returns a 403.
- Rate limits — per-consumer TPM and token-quota tiers (small / medium / large), returning a 429 once exceeded.
- Cost budget — a daily USD spend limit; when it is exceeded, requests are automatically downgraded to a cheaper model along a configured ladder, including cross-backend downgrades (e.g. gpt → OSS or OSS → gpt).
- Self-service Admin UI (React + FastAPI, Entra ID login, gated to an admin group) to issue consumer keys, set model, limit, and budget policies, and review the usage dashboard and request logs.
- Built-in observability — per-call token metrics, broken down by consumer and model, stream to Application Insights, surfaced through the Admin UI's usage dashboard and a request / blocked & downgrade-event log.
- Flexible client authentication — an APIM subscription key by default, or an Entra ID JWT (client_auth_mode).
How is it different from APIM AI Gateway?
APIM already provides useful GenAI gateway primitives: token rate limiting, token-usage metrics, semantic caching, backend routing, endpoint import, authentication, authorization, and monitoring. The difference is that APIM enables the enforcement runtime and policy control point, but not the full operating model required to run a shared, multi-tenant AI platform across teams, models, and budgets.
Inside the policy pipeline, APIM remains the load-bearing layer: llm-token-limit enforces per-consumer token-per-minute and quota limits, llm-emit-token-metric streams token usage into our metrics namespace, and standard APIM capabilities handle endpoint exposure, access control, and platform monitoring.
The governed AI Gateway adds the governance layer APIM does not provide out of the box:
- Self-service onboarding — a platform team can issue or revoke consumer keys and manage access from the Admin UI, without raising a pull request or redeploying infrastructure.
- Per-consumer model entitlements — every consumer has an explicit allow-list of model deployments. The gateway calculates the effective allowed set per request and returns 403 when a caller asks for a model it is not entitled to use.
- Live configuration without redeployment — entitlements, rate tiers, token quotas, budgets, and downgrade levels live in the configuration store. A sync worker projects those values into APIM named values continuously, so operational changes can take effect without a terraform apply while the policy logic stays version-controlled in IaC.
- Managed-identity-only, private backends — key-based authentication is disabled on Azure OpenAI and Azure AI Foundry. APIM injects a managed identity token on every backend call, and the backends are reachable only over private endpoints.
- Cost-based downgrade across backends — when a consumer approaches its budget, the gateway can route to a cheaper model while preserving availability, including cross-backend downgrades between Azure OpenAI and Azure AI Foundry.
APIM’s AI gateway is the enforcement runtime while the governed AI Gateway is the platform operating model around it. APIM handles the gateway primitives extremely well, while our governance layer adds identity, self-service administration, entitlement management, live configuration, cost controls, and cross-model routing so teams can safely consume multiple models without creating new cost, security, or compliance sprawl.
Solution overview
Figure 1 shows the end-to-end architecture of the governed AI gateway. Client applications never talk to the models directly; instead, every request passes through Azure API Management, which acts as the single governed entry point that authenticates callers, applies per-consumer policy, and routes traffic privately to the appropriate model backend. Around this gateway sit the supporting planes for administration, identity, and observability, giving the organization one consistent place to control access, contain cost, and monitor usage across both Azure OpenAI and Azure AI Foundry models. This solution is also completely serverless.
Key components:
- Client / consumer applications — the apps and services that call for model inference, each identified by its own consumer key or Entra ID identity.
- Azure API Management (the gateway) — the single governance endpoint that handles authentication, allowed-model checks, rate limiting, and cost-based budget downgrade before any request reaches a model.
- Model backends — Azure OpenAI (the gpt-5.4 family) and Azure AI Foundry (OSS and partner models such as grok-4.3 and DeepSeek-V4-Pro), each reachable only over a private endpoint.
- Microsoft Entra ID — provides identity for both clients (optional JWT auth) and the gateway's own managed identity used to reach the backends without password credentials.
- Admin UI (React + FastAPI) — the self-service control plane for issuing consumer keys and setting model, rate-limit, and budget policies.
- Application Insights — collects per-call token metrics by consumer and model, powering the usage dashboard and request / blocked-event logs.
Figure1: Solution architecture diagram
Request flow
- Authenticate — a client calls the gateway with an APIM subscription key (or an Entra ID JWT) instead of any model key.
- Authorize the model — APIM checks whether the consumer is permitted to call the requested model; if not, it returns 403.
- Enforce limits — the gateway applies the consumer's TPM and token-quota tier, returning 429 when the limit is exceeded.
- Apply the cost budget — if the consumer's daily USD budget is exhausted, the request is automatically downgraded to a cheaper model along the configured ladder.
- Route to the backend — APIM forwards the request over a private endpoint, authenticating with its managed identity to Azure OpenAI or Azure AI Foundry.
- Return and record — the model response is returned to the client while per-call token metrics are emitted to Application Insights and surfaced in the Admin UI dashboard and logs.
Implement the solution
This section describes how to deploy the solution architecture.
In this post, you’ll perform the following tasks:
- Create APIM
- Create Cosmos DB
- Create Microsoft foundry with Gpt-5.4, Gpt-5.4-mini, DeepSeek-V4-Pro and Grok-4.3 deployed
- Create the Admin UI on container apps
- Create a consumer with an APIM subscription key on the Admin UI
- Integrate APIM endpoint with Github Copilot chat and Copilot CLI
- Create a budget and rate limit in the Admin UI
- Simulate and validate auto downgrade feature
Ensure that you have the following prerequisites deployed before moving to the next section
- An Azure subscription with model quota (Azure OpenAI and, optionally, Azure AI Foundry models).
- Tools: Git, Terraform ≥ 1.7, Azure CLI, and az login to the subscription. Container images are built remotely in Azure Container Registry, so Docker is not required.
- VScode and Copilot CLI
Deploy the Azure AI Gateway
Clone the repository from https://github.com/microsoft/apim-foundry-governance
git clone https://github.com/microsoft/apim-foundry-governance git checkout english
By default the solution deploys in koreacentral region. Export your custom variables if needed.
export location=eastus2 export backend-rg=rg-aigw-tfstate-dev-eastus2 export storage-prefix=staigwtfstate export state-key=ai-gateway-eus2.tfstate Bootstrap the Terraform state backend (once per subscription)
This creates an eastus2 resource group + storage account for remote state (Entra auth, public blob access blocked).
./scripts/bootstrap-backend.sh \
--location $location \
--backend-rg $backend-rg \
--storage-prefix $storage-prefix \
--state-key $state-keySet Terraform variables
cp infra/terraform.tfvars.example infra/terraform.tfvars
# Edit infra/terraform.tfvars: prefix, location, owner, cost_center, apim_publisher_*, budget_* Create the Gateway Core
On the first apply, leave worker_image and admin_ui_image empty (default ""). The images don't exist yet, and the worker Job / Admin UI app are count-gated on these variables.
cd infra
terraform init
# If you are moving an existing state from another backend, run `terraform init -migrate-state` instead.
terraform applyBuild and push the container images with app registrations
After the registry is created, build the worker and Admin UI images remotely (no local Docker needed).
acr=$(terraform output -raw registry_login_server)
reg=$(terraform output -raw registry_name)
az acr build --registry $reg --image config-sync-worker:latest ../app/config-sync-worker
az acr build --registry $reg --image admin-ui:latest ../app/admin-ui
The worker and Admin UI requires entra app registrations for a user to access the frontend. Create the admin security group, BFF API App registrations and SPA public-client app registrations.
./scripts/app-registration.sh
Enable the worker and Admin UI
From the output above, populate the image references and the three Entra variables from the prerequisites into infra/terraform.tfvars and apply again.
worker_image = "<registry_login_server>/config-sync-worker:latest"
admin_ui_image = "<registry_login_server>/admin-ui:latest"
admin_ui_public = true # external FQDN (still Entra-gated). false = VNet-only
admin_group_object_id = "<entra security group object id>"
bff_api_audience = "api://<bff app id>"
spa_client_id = "<spa app id>"
entra_tenant_id = "<tenant id>"CosmosDB Seed configuration
Cosmos is private with key auth disabled, so the initial config is seeded from a jumpbox inside the VNet. Default confguration of enable_jumpbox = true in infra/terraform.tfvars triggers Terraform to: provision the jumpbox VM, grant it’s managed identity the Cosmos DB Built-in Data Contributor role (scoped to the config container), and runs a VM run-command that seeds both documents automatically:
- Global config (id=global) — allowed models + token limits.
- Per-model pricing (id=pricing) — prompt/completion rates for cost-based budgeting.
To seed manually instead (jumpbox connected via Bastion), the same scripts can be run directly:
# Global allowed models + limits
./scripts/seed-cosmos-jumpbox.sh https://<cosmos-account>.documents.azure.com:443/
# Per-model pricing (for cost-based budgeting)
./scripts/seed-pricing-jumpbox.sh https://<cosmos-account>.documents.azure.com:443/ Access the AdminUI
Update the SPA with your containerapps url
spa_app_id="$(az ad app list --display-name "AI Gateway SPA" --query "[].appId" -o tsv)" # spa_client_id
fqdn=$(terraform output -raw admin_ui_fqdn) # run from infra/
oid=$(az ad app show --id "$spa_app_id" --query id -o tsv)
az rest --method PATCH \
--uri "https://graph.microsoft.com/v1.0/applications/$oid" \
--headers 'Content-Type=application/json' \
--body "{\"spa\":{\"redirectUris\":[\"https://$fqdn\"]}}"
Browse to the admin_ui_fqdn, which is also the container apps fqdn. You will need to login via EntraID (Users will need to be added to the Entra group for them to login). Go ahead and register the consumer with a name and issue the API key. The API key is the APIM subscription key and will only be shown once on the UI, so copy and paste it somewhere safe.
Figure2: AI Gateway Consumers and KeysNext, on the left hand tab, click on budgets. This will set the daily budget limit a user is allowed to consume in a day and is also where the model downgrade logic resides. For the purpose of demonstration, set a low budget of $1.8 and select the model priority that you want the downgrade to occur. In this case, gpt-5.4 will be used first, followed by gpt-5.4-mini, DeepSeek then Grok.
Figure3: AI Gateway BudgetsLastly, on the land hand tab, select Rate limits. This sets the amount of tokens a user can consume in a day. It is a daily limit and resets after 24 hours. Select the large tier.
Figure4: AI Gateway Rate LimitsBrowse to Dashboard, it shows you all the token information, request status codes and group them by consumer and model. You can also view the budget downgrade for a specific user.
Figure5: AI Gateway CaptionsIntegrate endpoint with github copilot chat in vscode
In VScode, type “Ctrl + Shift + p” and select “Chat: Manage Language Model”. Select Add Models and choose Azure.
Figure6: Add models toGithubCopilot ChatFollow through the prompts. It will create or edit a chatLanguageModels.json file. Your file should look like this. Take note that you will need to use the /vscode path.
[
{
"name": "Azure",
"vendor": "azure",
"models": [
{
"id": "gpt-5.4",
"name": "gpt-5.4 (APIM)",
"url": "https://<REPLACE WITH YOUR APIM ENDPOINT>.azure-api.net/vscode/openai/deployments/gpt-5.4/chat/completions?api-version=2025-01-01-preview",
"toolCalling": true,
"vision": true,
"maxInputTokens": 128000,
"maxOutputTokens": 16000,
"requestHeaders": {
"Ocp-Apim-Subscription-Key": "<REPLACE WITH YOUR SUBSCRIPTION KEY"
}
}
]
}
]Now select the gpt-5.4 (APIM) model and ask it a question.
Integrate endpoint with copilot cli
As copilot only accepts api-key headers, a separate api is used. Replace and export the following variables before using copilot cli.
export COPILOT_PROVIDER_TYPE="azure"
export COPILOT_PROVIDER_BASE_URL="<REPLACE WITH YOUR APIM ENDPOINT>"
export COPILOT_PROVIDER_API_KEY="<REPLACE WITH YOUR SUBSCRIPTION KEY>"
export COPILOT_MODEL="gpt-5.4"
export COPILOT_PROVIDER_AZURE_API_VERSION="2025-01-01-preview"
export COPILOT_PROVIDER_MODEL_ID="gpt-5.4"You should see a similar response.
Figure7: Integration of APIM to copilot cliSimulate downgrade feature
Continue to ask more questions to consume more tokens. Once it hits the 80% cost threshold, you should see that the tag has been switched to “Auto-switch level 1”, meaning it will downgrade to gpt-5.4-mini for future requests.
Figure8: AI Gateway Downgrade FeatureValidate by running this command in your terminal with your own endpoints and api-key.
curl -sS -i -X POST "https://<REPLACE>.azure-api.net/openai/deployments/gpt-5.4/chat/completions?api-version=2025-01-01-preview" -H "api-key: <REPLACE WITH API KEY>" -H "Content-Type: application/json" -d '{"messages":[{"role":"user","content":"hi"}],"max_completion_tokens":8}'
Inspect the headers, you should see that the downgrade level is 1 and the effective model is gpt-5.4-mini despite hitting the same endpoint of gpt-5.4.
Figure9: Model downgradeConclusion
This post started with the problem of multi-model chaos: teams moving quickly with different models, endpoints, SDKs, keys, quotas, and cost profiles, but without a common control plane resulting in ineffective cost control and potential security leaks with model API keys. The governed AI Gateway addresses that by putting Azure OpenAI and Azure AI Foundry behind a single APIM-based entry point, where access, limits, routing, identity, telemetry, and budget behavior can be applied consistently for every consumer.
We also walked through how the gateway is different from APIM’s native AI gateway capabilities. APIM provides the enforcement runtime and the GenAI policy primitives, such as token limits, token metrics, semantic caching, and backend routing. The governed AI Gateway builds the operating model around those primitives: self-service onboarding, per-consumer model entitlements, live configuration without redeployment, managed-identity-only private backends, per-call cost telemetry, and cost-based downgrade across model providers.
From there, we integrated the APIM endpoint with Github Copilot Chat and Copilot CLI, and validated the downgrade behavior when spend thresholds were reached. The result is not just an AI proxy, but a reusable enterprise pattern for running AI access as a governed platform: developers keep a simple model endpoint experience, while the platform team keeps control over security, cost, observability, and operational change.
Overall, this post helps organizations bring multi-model AI usage under one governed entry point, reducing sprawl across endpoints, keys, policies, and cost controls. It also gives platform teams centralized control over model access, rate limits, budgets, telemetry, and private backend access while preserving a simple endpoint experience for developers.
References
Microsoft