It doesn’t take long for AI agents to get out of hand.
In most enterprises, the first few agents are celebrated. A chatbot here. A document summarizer there. Then another team ships an agent that calls APIs. Someone else connects one to internal data. Within months, IT is staring at dozens—or hundreds—of autonomous systems running across subscriptions, regions, and tools.
At that point, the questions stop being about model quality and start being uncomfortable operational ones:
- Who owns this agent?
- What data can it access?
- What happens if it misbehaves?
- Why did it just consume half our monthly token budget in a day?
Developers can build an AI agent in minutes—the difficult part is understanding what agents are doing, how they perform, and whether they comply with organizational policy. Signals scatter across tools, context is lost, and governance becomes reactive.
This reference architecture exists to solve that problem.
It describes a multi‑region AI agent landing zone on Azure that treats agents as first‑class, governable workloads—provisioned automatically, constrained by policy, and observable from day one.
The architectural principle: separate control from execution
The design starts with a simple but non‑negotiable rule:
Control plane concerns must be separated from runtime concerns.
Azure landing zones already follow this model. Management groups, Azure Policy, and RBAC are global constructs. Workloads run in regions. This architecture applies the same discipline to AI agents.
- The runtime plane is where agents execute, models infer, and data flows—often in multiple Azure regions.
- The control plane is where identity, policy, safety, evaluation, and oversight live—independent of region.
This separation is what allows teams to scale agents without losing control.
Layer 1: Azure AI Gateway — governing every request
The first control layer sits directly in the request path.
The AI gateway in Azure API Management provides a policy‑enforcement and observability layer in front of AI models, agents, and tools. It is not a separate service—it extends Azure API Management.
Everything flows through it:
- Microsoft Foundry model deployments
- Azure AI Model Inference API endpoints
- OpenAI‑compatible third‑party models
- Self‑hosted models
- MCP servers and A2A agent APIs (preview)
What the gateway actually enforces
This layer is intentionally narrow and operational:
- Token quotas and rate limits
The llm-token-limit policy (GA) enforces tokens‑per‑minute or quota ceilings per consumer before requests reach the backend. This prevents one application—or one agent—from exhausting shared capacity. - Content safety at ingress
The llm-content-safety policy (GA) integrates Azure AI Content Safety to moderate prompts automatically. Unsafe requests never reach the model. - Traffic routing and resiliency
Azure API Management supports multi‑region gateway deployment (Premium tier). If a region fails, traffic routes to the next closest gateway automatically.
Token usage, prompts, and completions are logged to Azure Monitor and Application Insights using built‑in policies such as llm-emit-token-metric.
The gateway does not understand agent intent or business context. That is by design. It governs traffic, not behavior.
Layer 2: Azure AI Foundry Control Plane — governing behavior at scale
The second layer governs what agents do, not just how requests flow.
Azure AI Foundry Control Plane provides a unified management surface for AI agents, models, and tools across projects and subscriptions. It is designed specifically for agentic systems.
Foundry Control Plane is currently in public preview.
What Foundry Control Plane adds
- Fleet‑wide inventory
Every agent, model, and tool appears in a single, searchable view across projects. - Continuous evaluation on production traffic
Foundry runs evaluations that measure task adherence, groundedness, tool‑call accuracy, sensitive data exposure, and other agent‑specific risk dimensions. - Centralized guardrails
Policy is enforced across inputs, outputs, and tool interactions—not just prompts. Bulk remediation can be applied across the fleet. - Security integration
Foundry integrates with:- Microsoft Entra for agent identity (Entra Agent ID)
- Microsoft Defender for threat signals
- Microsoft Purview for data protection and compliance visibility
Foundry Control Plane also requires an AI Gateway to be configured for advanced governance scenarios—reinforcing the layered approach.
Layer 3: Microsoft Agent 365 — enterprise oversight, not just Azure oversight
The third layer exists because Azure governance alone is not enough.
Agents don’t just call APIs. They act on behalf of users. They access enterprise data. They operate inside Microsoft 365 workflows.
Microsoft Agent 365 is the tenant‑level control plane for AI agents. It brings agents under the same administrative model used for users and applications.
- Status: Frontier Preview
- General availability: May 1, 2026
Why this layer matters
Agent 365 introduces controls that Azure alone cannot provide:
- Agent registry
A single inventory of all agents in the tenant—including sanctioned and shadow agents. Unsanctioned agents can be quarantined. - Identity‑first access control
Every agent is issued an Entra agent ID. Conditional Access policies apply to agents the same way they do to users. - Human‑in‑the‑loop oversight
Agents surface in Microsoft 365 admin workflows, not just Azure portals. - Security and compliance
Defender and Purview extend threat detection and data protection policies to agent activity.
Agent 365 does not replace Foundry Control Plane. It complements it—connecting agent operations to enterprise identity, compliance, and productivity systems.
How the pieces work together
Individually, these services are powerful. The architecture works because they are deliberately layered.
External approval → automated provisioning
When a use case is approved in an external governance system, it triggers an Azure DevOps pipeline using the REST API.
That pipeline:
- Provisions subscriptions and resource groups
- Deploys Foundry projects
- Configures Azure API Management with AI Gateway policies
- Enables monitoring and logging
Governance is applied before the first request is made.
One policy model, many regions
Azure landing zones are region‑agnostic at the governance layer. This architecture follows that guidance.
- Policies and RBAC apply globally
- AI Gateway enforces limits locally in each region
- Runtime services scale region by region
Expanding to a new region does not introduce a new governance model—only new capacity.
A single operational view
Signals flow upward:
- AI Gateway emits traffic and usage metrics
- Foundry Control Plane correlates evaluations, guardrail enforcement, and security alerts
- Agent 365 aggregates tenant‑level identity, compliance, and threat signals
Operations teams no longer hunt across dashboards. They work from one prioritized view, with context intact.
What this architecture deliberately does not promise
This is a reference architecture, not a silver bullet.
It does not eliminate the need for:
- Clear agent ownership
- Business‑level approval processes
- Ongoing evaluation of agent usefulness
What it does provide is a foundation—one that lets organizations scale agentic AI without accepting chaos as the cost of innovation.
Closing thoughts
Agent sprawl is not a tooling failure. It’s an architectural one.
By separating control from execution, layering governance where it belongs, and aligning AI operations with existing Azure and Microsoft 365 control planes, this architecture gives enterprises a way to move fast without losing sight of what their agents are doing.
That’s the difference between experimentation—and production.
Co-Contributor:
Jorge Pena Alarcon-Sr. Cloud & AI Specialist
References (official Microsoft sources)
Microsoft