Enterprises are moving from AI experiments to AI in production. Copilots are in the flow of daily work, custom applications are built on foundation models, and autonomous agents are beginning to take actions on behalf of the business. As organizations combine human ambition with machine intelligence, trust becomes the defining requirement for scaling AI responsibly and realizing AI's full potential. 

That shift changes the governance question from "can we use AI responsibly?" to a harder one: "can we prove, continuously, that every AI system and agent in our estate is safe, compliant, observable, and accountable?" 

The organizations getting this right have stopped treating governance as a static document that lives in a policy binder. They treat it as an operating system: a connected set of policies, controls, telemetry, and evidence that runs alongside AI everywhere it operates. Governance defines what should happen. Observability and evaluations verify what is actually happening. Audit and response prove it, and feed what they learn back into policy. The loop, not any single control, is what makes governance real. 

Every control in this map ladders up to a principled foundation: Microsoft's Responsible AI Standard, whose six principles (fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability) set the bar these controls exist to meet. 

This post lays out a practical reference map for that operating system: the domains AI governance must cover, the cross-cutting runtime enforcement layer that operationalizes them, and the Microsoft services that support each one. 

The four pillars of AI governance

At its core, AI governance does four things. Skip any one of them and a gap opens up -policy without control is aspirational, control without visibility is blind, and visibility without proof cannot withstand an audit.

1. Policy - Define the rules, and who owns them. What's acceptable use, which use cases are approved, how each is classified by risk, and -critically -who is accountable for outcomes at each stage.
2. Control - Enforce the boundaries, proportionate to risk. Turn policy into guardrails, access decisions, and lifecycle gates that constrain systems at runtime -with tighter controls on higher-risk use cases.
3. Visibility - Observe behavior, including fairness and quality. Capture logs, metrics, traces, token consumption, tool calls, and runtime signals that reveal what AI is truly doing -not just latency and cost, but grounding, drift, and fairness signals.
4. Proof - Audit, prove, and improve. Produce the evidence to demonstrate compliance and transparency (explainability and traceability you can show regulators and customers), investigate incidents, and feed what you learn back into policy.

What AI governance covers: the domain map

A complete AI governance program spans nine core domains, with runtime enforcement operating as a cross-cutting layer across several of them. Use this taxonomy to map your own requirements to concrete controls:  

• Policy: Responsible AI policies, approval workflows, and guardrails.
• Data governance: Classification, sensitivity labels, DLP, retention, and lineage.
• Model governance: Validation, versioning, documentation, and change control.
• Observability: Logs, metrics, traces, token usage, and runtime signals.
• Evaluations: Quality, safety, grounding, drift, and agent task success.
• Security: Threat detection, posture management, proactive AI red teaming, and prompt-injection resistance.
• Identity & access: RBAC, least privilege, conditional access, and agent identities.
• Audit & compliance: Evidence, eDiscovery, legal hold, and reporting.
• Agent governance: Registry, lifecycle, policy enforcement, and fleet visibility.

Cross-cutting runtime enforcement: Runtime enforcement is not a separate governance program. It is the control layer that operationalizes policy, security, identity, data protection, observability, and agent governance while AI systems are running. It governs live interactions between users, agents, models, tools, APIs, MCP servers, A2A agent APIs, and enterprise systems through authentication, authorization, traffic controls, token governance, quotas, policy checks, and operational guardrails.

The Microsoft governance stack

Microsoft's approach is a composed platform rather than a single product — AI lifecycle, data governance, identity, security, observability, and audit working together:

1. Policy and the control plane: define and enforce

Governance starts with a place to define the rules and decide who owns them. This is where responsible AI and acceptable use policies are authored; where new use cases come in through intake, get classified by risk, and are approved or rejected; where you decide up front which decisions and agent actions require human sign-off versus running autonomously within guardrails; and where accountability is assigned across the deploy, update, and retire lifecycle. Enforcing those rules through runtime guardrails, security posture, agent-level controls, and API-level controls is the job of the sections that follow; the work here is defining what “acceptable” means before anything is switched on. 

Services that support it:

• Microsoft Foundry -Compliance workspace for authoring and housing responsible AI policies and use case approvals.
• Azure Policy -Resource level governance to codify those rules as enforceable baselines.
• Azure API Management (AI Gateway): Runtime enforcement for AI applications, models, agents, tools, and APIs, including authentication, authorization, token governance, quota management, and policy enforcement. 

(Agent specific policy and lifecycle controls via Microsoft Agent 365 are covered in Section 6.)

Practical takeaway: establish a governance baseline before AI and agent adoption scales -including deciding up front where a human stays in the loop. Retrofitting controls onto a sprawling estate is far harder than starting from one.

2. Data governance and compliance: Purview at the center

Most AI risk is ultimately data risk: the wrong information reaching a model, leaving through a response, or being retained without a policy. Microsoft Purview is the primary governance engine for AI interactions and enterprise data controls, and it works in three moves: 

• Discover - Identify AI usage, sensitive prompts and responses, and overall risk posture with DSPM for AI.
• Protect - Apply sensitivity labels, DLP, access boundaries, and retention policies.
• Prove - Generate audit trails, eDiscovery evidence, and compliance reporting.

If the question is “how do we govern the data flowing through AI?” - Purview is the anchor.

3. Model governance, safety, and evaluations: quality at build time and runtime

Evaluations are what make governance measurable, and they belong at two points in the lifecycle:

• Pre-deployment - Release gating on safety, grounding, quality, fairness, and task success before anything ships.
• Post-deployment - Runtime scoring for drift, quality thresholds, and harmful-output alerts once a system is live.

Services that support it: 

• Microsoft Foundry evaluators: quality, safety, RAG grounding, and agent task metrics 
• Azure AI Content Safety: detects unsafe or harmful content in prompts and responses. This is also integrated in Microsoft Foundry as guardrails that protect against unsafe content and direct and indirect jailbreak prompt injection attacks. 
• ASSERT (open source): Microsoft's open-source, policy-driven evaluation framework, short for Adaptive Spec-driven Scoring for Evaluation and Regression Testing. ASSERT turns your specific requirements and organizational policies into targeted, safety-focused test cases rather than generic benchmarks, and runs across frameworks such as LangChain, CrewAI, LiteLLM, and OpenAI, so evaluation is not locked to any single stack. ASSERT also pairs with runtime controls to close the loop: measure a failure rate, apply a control (see Agent Control Specification in section 6), then re-run the same evaluation to prove the rate dropped. That turns "we mitigated it" into a measured before-and-after. 

Evaluations are not separate from governance - they are what turn governance from a promise into measurable proof. 

4. Observability and runtime monitoring: know what is actually happening

Once AI is in production, you need to see its behavior in real time. Capture four kinds of signal: 

• Logs: prompts, responses, and decisions 
• Metrics: latency, token usage, quality scores, request volume, and quota consumption 
• Traces: agent reasoning paths, tool calls, model calls, and inter-service dependencies 
• Alerts: policy violations, drift, harmful outputs, and abnormal traffic patterns 

Microsoft Foundry provides purpose-built observability for AI applications and agents across three connected capabilities. Evaluation measures quality, safety, and reliability throughout development. Monitoring tracks deployed systems in real-world conditions. Tracing, built on OpenTelemetry standards, captures the execution flow of LLM calls, tool invocations, and agent decisions, so you can debug multi-step reasoning rather than guess at it, across frameworks including LangChain, LangGraph, the OpenAI Agents SDK, and the Microsoft Agent Framework. The Agent Monitoring Dashboard brings these signals together for production traffic: token usage, latency, run success rates, evaluation scores, and red-teaming results in one view, backed by Azure Monitor, Application Insights. From the same surface you can turn on continuous evaluation to score sampled live responses, scheduled evaluations to detect drift against benchmarks, scheduled red team scans to probe deployed agents for emerging risks, and alerts that fire when outputs fail quality thresholds or produce harmful content. 

Critically, this coverage is not limited to agents built on Foundry. Agents running elsewhere can be onboarded through the AI Gateway and instrumented with OpenTelemetry semantic conventions to send telemetry to the same Application Insights instance, so one dashboard, one evaluation pipeline, and one alerting surface cover the whole estate. 

The broader observability fabric 

Foundry observability composes with the rest of the platform: Azure Monitor, Application Insights, Log Analytics, Purview Audit, and Microsoft Agent 365 dashboards. Azure API Management (AI Gateway) extends visibility further by capturing AI traffic patterns, token consumption, API usage, policy violations, and agent-to-tool interactions, providing runtime insight into AI workloads at the network boundary. 

Practical takeaway: wire observability before launch, not after the first incident. Continuous evaluation on sampled production traffic is the cheapest early-warning system you can buy, and the trace data it rides on is the same evidence an auditor will ask for later. 

5. Security and identity governance: protect the AI estate

AI expands the attack surface, so governance has to include security and identity across the following:

• Threat protection -Prompt injection, jailbreak, data exfiltration, and unsafe actions -with Microsoft Defender for Cloud and Microsoft Defender XDR.
• Proactive adversarial testing -Find failures before attackers do, with the AI Red Teaming Agent in Microsoft Foundry for automated agent probing and PyRIT, Microsoft’s open-source red-teaming framework.
• Security posture -Misconfigurations, exposed endpoints, and risky resources -with Defender for Cloud and Azure Policy.
• Identity & access -Who can use AI, and what data, apps, and agents it can reach -with Microsoft Entra ID and RBAC.
• Runtime access enforcement: Who can call which model, tool, API, or MCP server, under which conditions, and with what level of throttling, inspection, and monitoring. 
• Secrets & network -Protecting keys and enforcing private access and network boundaries -with Azure Key Vault and Private Link.
• Device management-Discover, monitor, and govern unmanaged AI agents and the endpoints they run on -with Microsoft Intune and Microsoft Defender for Endpoint (MDE).

Cross-cutting runtime enforcement 

As AI systems increasingly rely on APIs, tools, MCP servers, enterprise applications, and agent-to-agent interactions, governance must extend beyond model build-time controls to the live interactions that connect those systems. Runtime enforcement helps operationalize the broader AI governance program by enforcing who can call what, under which conditions, at what scale, and with what level of monitoring. 

Services that support it :

• Azure API Management (AI Gateway): Centralized authentication, authorization, token controls, traffic shaping, quota enforcement, content safety policies, and policy governance for AI models, agents, tools, MCP servers, A2A agent APIs, and enterprise APIs. 
• Microsoft Entra ID: Identity and access control for applications, users, and agents. 
• Azure RBAC: Fine-grained authorization for AI resources and workloads. 

The mindset shift: treat agents as non-human identities with their own permissions, lifecycle, runtime access and monitoring.

6. Agent governance: the emerging control plane

Agents are a new kind of enterprise actor, and governing them takes two complementary pieces: a control plane to manage the whole fleet, and a portable control standard to enforce safety inside each agent's workflow.

Microsoft Agent 365: govern the agent fleet

Agents introduce a new problem - sprawl. As teams build agents on different platforms, organizations need one way to see and govern all of them. Microsoft Agent 365, now generally available, is the control plane for AI agents: it gives each agent its own identity and manages agents with the admin tools you already use - Microsoft Entra, Defender, Purview, Intune, and the Microsoft 365 admin center. It works across agents built on Microsoft, open-source, and third-party platforms, through five core capabilities:

• Registry - Discover and catalog every agent - including shadow agents - and quarantine the unsanctioned ones.
• Access control - Give each agent a unique Microsoft Entra Agent ID, and enforce least-privilege, risk-based access with policy templates.
• Visualization - Unified dashboards, telemetry, and alerts across the entire agent fleet.
• Interoperability - Works with agents from Microsoft, open-source, and third-party frameworks, alongside your Microsoft 365 apps and data.
• Security - Threat detection, data protection, and compliance for agents at scale, through Defender, Purview, and Entra.

Agent Control Specification (ACS): portable runtime controls

Governing agents also means enforcing controls consistently, no matter which framework an agent is built on. The Agent Control Specification (ACS) is an open industry standard, part of Microsoft's Agent Governance Toolkit, for placing deterministic safety and security controls at defined checkpoints in an agent's workflow. Just as the Model Context Protocol (MCP) standardized how agents connect to tools, and A2A standardized how agents communicate with each other, ACS aims to standardize how agents are governed: one portable control layer that any framework can adopt, with Microsoft providing reference implementations. 

• Control checkpoints: validation at defined intervention points across the agent's lifecycle, spanning input, model invocation, state, tool selection and execution, and output 
• Composable controls: deterministic checks (allow/deny rules, custom filters) alongside model-based checks (classifier endpoints, LLM judges), each placed exactly where it is needed 
• Portable and auditable: expressed as a declarative manifest with policy-as-code evaluation, so controls are versionable, auditable, and travel with the agent across any stack 

Together they cover both altitudes of agent governance: Agent 365 governs the fleet from the outside, while ACS enforces safety inside each agent's workflow. Those checkpoints can also route to a human for approval before a high-impact action -the runtime side of human-in-the-loop -while Agent 365 defines which agents and actions require sign-off.

The governance service map

Use this as a quick reference for matching each governance area to the Microsoft services that support it. Runtime enforcement is shown as a cross-cutting capability because it operationalizes several domains rather than replacing them: 

A practical path forward

You do not have to govern everything at once. A pragmatic sequence gets you to a defensible position quickly: 

• Start with scope: Inventory the AI apps, Copilots, agents, APIs, tools, MCP servers, and custom Microsoft Foundry workloads that exist today. 
• Classify risk: Determine what data is used, what actions AI can take, what systems it can reach, and which regulations apply. 
• Apply controls: Bring in Purview, Entra, Defender, Microsoft Foundry guardrails, Azure API Management AI Gateway policies, Agent 365 policies, and portable ACS checkpoints for agents. 
• Measure and prove: Add evaluations, including policy-driven ASSERT runs and adversarial testing with the AI Red Teaming Agent, plus observability, APIM runtime telemetry, audit evidence, and operational dashboards. 

Governance works best when it follows the business rather than leading it. This journey view shows how the controls layer in progressively starting from a high-value business use case and adding only what each next step requires, so governance accelerates safe delivery instead of gating it. The path runs through four stages: align on the use case and its value, stand up a safe foundation of minimum viable controls, build and validate quality and safety, then operate and scale -monitoring, auditing, enforcing runtime controls and governing agents in production.

 

 

Microsoft gives you the building blocks to govern AI as a live operating system - not a static policy document. To go deeper, explore Microsoft Foundry, Microsoft Purview, Microsoft Defender for Cloud, Microsoft Entra, Azure API Management and Microsoft Agent 365 -plus the open-source ASSERT evaluation framework and the Agent Control Specification (ACS) for portable, framework-agnostic agent controls. 

References

Policy & lifecycle

• Microsoft Foundry — https://learn.microsoft.com/en-us/azure/foundry/what-is-foundry
• Microsoft Foundry Compliance and Security- Manage compliance and security in Microsoft Foundry - Microsoft Foundry | Microsoft Learn
• Azure Policy — https://learn.microsoft.com/en-us/azure/governance/policy/overview
• Azure API Management AI Gateway - https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities
• Microsoft Agent 365 — https://learn.microsoft.com/en-us/microsoft-agent-365/overview

Data governance

• Microsoft Purview — https://learn.microsoft.com/en-us/purview/purview
• DSPM for AI (Data Security Posture Management) — https://learn.microsoft.com/en-us/purview/data-security-posture-management-learn-about
• DLP (Data Loss Prevention) — https://learn.microsoft.com/en-us/purview/dlp-learn-about-dlp
• Sensitivity labels — https://learn.microsoft.com/en-us/purview/sensitivity-labels

Model safety

• Azure AI Content Safety — https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview
• Prompt Shields — https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/jailbreak-detection

Evaluations

• Azure AI Foundry evaluators (observability & evaluation) — https://learn.microsoft.com/en-us/azure/foundry/concepts/observability
• ASSERT — official announcement (open-source agent evals; no standalone Learn page yet): https://devblogs.microsoft.com/foundry/build-2026-open-trust-stack-ai-agents/

Observability

• Azure Monitor — https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/overview
• Application Insights — https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview
• Log Analytics — https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-workspace-overview
• Purview Audit — https://learn.microsoft.com/en-us/purview/audit-solutions-overview

Security

• Microsoft Defender for Cloud — https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-cloud-introduction
• Microsoft Defender XDR — https://learn.microsoft.com/en-us/defender-xdr/microsoft-365-defender
• Microsoft Sentinel — https://learn.microsoft.com/en-us/azure/sentinel/sentinel-overview

Identity

• Microsoft Entra ID — https://learn.microsoft.com/en-us/entra/identity/
• Azure RBAC (role-based access control) — https://learn.microsoft.com/en-us/azure/role-based-access-control/overview
• Conditional Access — https://learn.microsoft.com/en-us/entra/identity/conditional-access/overview

Audit & compliance

• Purview Audit — https://learn.microsoft.com/en-us/purview/audit-solutions-overview
• eDiscovery — https://learn.microsoft.com/en-us/purview/edisc
• Compliance Manager — https://learn.microsoft.com/en-us/purview/compliance-manager

Agent governance

• Microsoft Agent 365 — https://learn.microsoft.com/en-us/microsoft-agent-365/overview
• Agent Control Specification (ACS) — https://microsoft.github.io/agent-governance-toolkit/packages/agent-control-specification/
• Agent Governance Toolkit — https://microsoft.github.io/agent-governance-toolkit/
• Purview for agents — https://learn.microsoft.com/en-us/purview/purview

Contributors:

This article is maintained by Microsoft. It was originally written by the following contributors.

1. Manasa Ramalinga | Senior Principal Cloud Solution Architect – US Customer Success 
2. Mehrnoosh Sameki | Principal PM Manager -AI Governance Product Team