Global enterprise multi-agent systems in production: Design and security (Part 1)

The work behind this series began with customers while hosted agents in Microsoft Foundry were still in private preview. Customers adopted the service because it took the platform work off their plates: running, scaling, open ecosystem and versioning agents, each with its own identity and built-in observability. Since then, that work has taken multiple enterprise multi-agent systems to production across many geographies.

Those customers weren't all at the same stage. Many were just starting with large language model (LLM) features and had concerns about quality assurance, so some of that work started before any code. For one customer with a business-to-business (B2B) product, the first step was helping their executives understand how an LLM feature would affect the product and the clients who use it. A white paper, presented and circulated to the product manager and leadership to address their concerns, set out a blueprint for LLM evaluation and assurance with global considerations. It covered the main risk areas, from hallucination and harmful output to privacy, prompt injection, reliability, and global regulation, and how to evaluate and mitigate each one.

One lesson from that work came early: a convincing response is where the engineering starts, not where it ends. Dependable systems make identity, state, recovery, and control boundaries explicit, and they prove each one holds. This post shares seven design and security practices from that work, each with a release check you can apply to your own system.

What hosted agents handle, and what you own

Hosted agents in Microsoft Foundry let you keep the agent framework you already use. The basic flow has four steps:

1. Build the agent with Microsoft Agent Framework, LangGraph, or your custom Python or C# code.
2. Run and test it locally.
3. Deploy it with the Azure Developer CLI (azd), which packages it as a container image for Foundry Agent Service. The Microsoft Agent Framework hosting guide walks through steps 2 and 3.
4. Call its endpoint from your application, or publish it to Microsoft Teams and Microsoft 365.

Foundry runs the agent on Microsoft-managed infrastructure and supplies what teams would otherwise build themselves: a dedicated Microsoft Entra agent identity and endpoint, isolated per-session sandboxes that scale to zero and resume with their files, immutable versions, Foundry tools through a Toolbox endpoint, and private networking. Agents built on the hosting protocol libraries emit OpenTelemetry traces to Application Insights by default, so tracing works from the first deployment. Part 2 shows how the Microsoft Foundry evaluation and improvement tools build on those traces.

Managed hosting changes where the code runs, not who owns the outcome. You still own application authorization, business effects, data handling, and evidence that the whole system behaves correctly. You don't need every practice in this series on day one. A single agent for one team in one region can start with the four steps above and add practices as you add agents, users, sensitive data, and geographies.

Feature availability: At the time of writing, several features in this series are in preview: the state store, resilient long-running execution, network egress controls, the tool call and tool response intervention points, spotlighting, Task Adherence, document-level access features in Azure AI Search, Microsoft Defender protection for Foundry agents, and Insights in Foundry. Hosted-agent optimization requires an allow-listed subscription. Preview features have no service-level agreement and aren't recommended for production workloads; generally available (GA) features are supported for production use. Check each feature's current status in its documentation before you rely on it.

The example behind these lessons

Across the implementations behind this series, adoption kept growing as teams applied hosted agents to more of the documented scenarios. A generalized example from those implementations runs through the series: an enterprise assistant that answers domain questions for business users. An orchestrator agent checks whether each request is in scope and delegates it to specialist agents. Those agents query governed enterprise data through tools, run sandboxed code, and draw on confidential internal knowledge that may inform an answer but must not be exposed directly. Long-running jobs outlast a single response, and users expect conversation history, visible progress, and cancellation that stops the work itself.

Figure 1. A generalized example drawn from these implementations, with the controls the lessons recommend.

Organization-specific details are generalized throughout. Each lesson takes one boundary in this example and turns it into a contract you can test before release.

What changes across geographies

• Each region is its own deployment. Give each one its own identity, quota, network, configuration, telemetry, and recovery evidence, and coordinate releases and recovery across them.
• Features and quotas differ by region. Verify feature availability and quotas region by region; deployments aren't interchangeable by default.
• Data residency is a design decision. Decide where business records, retrieved content, and diagnostic copies may be processed and stored. Multiple regions and managed hosting alone establish neither residency nor failover, so design and demonstrate both.

What you will learn

Next in this series: release and operations (Part 2), which closes with the improvement loop.

1. Start with ownership and the smallest sufficient architecture

Choose the simplest architecture that does the job, and give every part of it an owner. Azure Architecture Center guidance on artificial intelligence (AI) agent orchestration patterns favors the simplest sufficient option: a direct model call, one tool-enabled agent, or multiple agents. Each step up adds handoffs, model calls, distributed state, and failure combinations, so add an agent only when a simpler design can't do the job. Then give application ingress, orchestration, models, tools, knowledge ingestion, stores, and operational response each an owner and a failure contract. State and diagnostic data need their own ownership, access, retention, and deletion rules.

Figure 2. Ownership boundaries for a hosted-agent action. A generalized pattern, not a specific customer's architecture.

Match the orchestration pattern to the work:

Let models interpret requests and propose a plan, and use deterministic workflows for permitted data access, job authorization, approvals, and business transitions. Use an orchestrator that rejects out-of-scope requests as the first layer, and back it with tool and parameter validation, call limits, and timeouts in trusted code. In one implementation, the team's guardrail design put a planner agent first to reject out-of-scope requests, capped tool calls, set tool-call timeouts, and validated parameters passed between tools. They cross-checked the design against the Open Worldwide Application Security Project (OWASP) Top 10 for Large Language Model Applications.

Share one end-to-end budget for deadlines, model calls, tokens or spend, tool calls, retries, handoffs, fan-out, and concurrency. Nested agents draw from it instead of getting fresh allowances. Validate handoff schemas, and pass along the necessary provenance, classification, and remaining budget rather than every conversation.

Keep deployment, discovery, and authorization separate. Model Context Protocol (MCP) exposes a tool-service interface, and Microsoft Foundry Toolbox supplies its documented discovery and authentication path. Give trusted functions and model-generated code different execution boundaries.

Watch for:

• More agents add coordination, not inherent reliability.
• Neither MCP nor Toolbox deploys an agent, and discovering a tool grants no permission to use it.
• A sandbox for generated code is not another trusted application programming interface (API).

Check before release: Record simpler-design comparisons and an ownership matrix; demonstrate bounded termination and explicit degraded behavior.

Rejected alternative: Going multi-agent by default. Independent specialist ownership can help maintainability, but coordination must earn its latency, cost, and operational burden.

Further reading: AI agent orchestration patterns · Workflow-oriented multi-agent systems · Use Toolbox with hosted agents

2. Treat identity as a chain, not a single managed identity

Map every identity in a request's path before you assign a single permission. Separate five responsibilities; a principal is the identity acting on a resource, such as a user or application:

• The caller authenticating to the application.
• The deployment principal that pushes the image versus the principal that pulls it.
• The runtime accessing dependencies.
• The end user, when downstream work uses delegated permissions.
• Each tool service enforcing access to its records and actions.

Verify the effective principals for your deployment and network configuration against the hosted permissions reference. Record each image-pull workaround or investigative role assignment as a scoped exception, not as a least-privilege template.

Derive trusted user context from authenticated server-side information, and validate audience, scope, tenant, destination, expiry, and consent at the appropriate hop. The on-behalf-of (OBO) guidance separates user context, the backend's Microsoft Foundry invocation token, and the downstream delegated token. Validate incoming tokens intended for your backend's API, and treat downstream tokens as opaque unless you own their receiving API. Per the same guidance, deferred work must reacquire valid delegation through the trusted backend.

For example, a user's token arrives at your backend, which validates that the token was issued for your API. The backend exchanges it through OBO for a delegated token scoped to the downstream API. It then calls Foundry with its own workload token, passes the delegated token in a forwarded x-client-* header, and sends the user's object ID so Foundry associates the request with that user. The agent uses only the delegated token to call the downstream API. Forwarding the user's original token instead fails, because that token was issued for your API, not the downstream one.

Keep identity-bearing MCP clients, Toolbox clients, credentials, and associated resources request-scoped, with cleanup. The hosting guide warns that process-wide MCP connections can retain request identity, so pool connections only when identity isolation stays correct. Partition custom stores by trusted user and session context, and enforce ownership server-side.

Watch for:

• A user identifier is not a downstream delegated token, and forwarding headers alone doesn't perform the OBO exchange.
• Spoofable client headers. Reject them.
• Saving bearer-token headers for deferred work, or silently substituting app-only access.
• Platform user isolation and application-owned multiplexing are different: a shared sandbox doesn't give each user a separate filesystem or partition your custom caches.
• An index or partition key is not authorization. A middle tier that delegates user identity must still authorize each user's session routing, because delegation isn't a platform-enforced fence between the users it represents.

Check before release: With two differently entitled test users, probe history, files, retrieval, metadata, tool results, and delayed callbacks using guessed identifiers, reused connections, concurrent work, and resumed sessions. Verify intentional operator access separately.

Rejected alternative: One broadly privileged identity everywhere. Easier initial setup weakens least privilege, attribution, and incident containment.

Further reading: Hosted agent permissions · On-behalf-of flow · Hosting guide · Isolate sessions per user · Multiplex session users

3. Name every kind of state before promising persistence

Name every kind of state the application keeps, and give each one an owner before you promise persistence. An agent application has several state contracts, not one storage box:

Know what the platform keeps for you. Hosted sessions keep $HOME and uploaded files across turns and idle periods, and the state store gives you explicit, server-backed JavaScript Object Notation (JSON) persistence.

On the way to production, two questions came up again and again: was the conversation saved, and did the next turn load it? In one implementation, history was saved correctly, but the workflow adapter in use didn't load it on the next turn. So test them separately: a saved record proves persistence, and only a correct next turn proves hydration, the loading of saved state into the running application.

Check that the installed framework loads the intended conversation and checkpoint, preserves permitted partial output, and keeps tool messages intact. Treat a hosting-library upgrade as a possible protocol and state migration rather than a routine package change. Use the protocol-specific session binding, and test the path you chose, because published examples can differ.

Add a custom application index when you need labels, navigation, queries, or metadata such as conversation titles. Retire an existing database only when its replacement meets the actual requirements. Plan each lifecycle separately: compute idle timeout, session expiry, history retention, state retention, and deletion, plus replicas, backups, exports, and derived chunks. Definition, execution-state, and history stores need not share one expiry policy.

Watch for:

• The state store is not an automatic snapshot of random-access memory (RAM), variables, or the entire filesystem, and a framework session object is not a separate hosted sandbox.
• Matching first answers don't prove that local and hosted behavior are equivalent.
• Duplicated or lost tool messages after a resume.
• A response-chain identifier is not evidence of filesystem continuity, so don't equate history with sandbox identity.
• Adding a custom index only because you assume the platform offers no history or listing at all.
• A response's creation timestamp is not necessarily every conversation item's.
• Stopping a session, deleting a mapping, and physically erasing a record are different operations.

Check before release: For every used store, document ownership/lifecycle and test reopen, restart, version compatibility, deletion, and denied access.

Rejected alternative: Promising "persistent storage" as one thing. That simplicity hides the recovery and migration contracts each store needs.

Further reading: Hosted agents · State store · Manage hosted sessions

4. Design recovery, approval, and cancellation as explicit protocols

Design long-running work, approvals, and cancellation as explicit protocols, so a crash, retry, or stop request leaves a business outcome you can explain. Background execution, checkpointing, external-event continuation, and crash recovery are separate capabilities, each with explicit opt-in and mode-specific behavior. Task duration, leases, payloads, checkpoint retention, and recovery latency each have their own bounds.

Check how your framework fits in, too. The hosting guide describes different recovery behavior for native workflows, wrapped existing agents, and custom handlers, and steering (changing direction during a running turn) isn't available in Python. Pin the framework adapter version you test, and verify workflow, protocol, and steering support on the path your application uses.

Make recovery cover the business effect

For a long-running job, separate the business effect:

1. Persist authorized intent and a stable operation identifier.
2. Use an idempotent downstream action, one that doesn't repeat its business effect when retried, where supported.
3. Record its authoritative receipt and resulting state.
4. Reconcile uncertainty, and recheck authority, before retrying or compensating.

Write checkpoints before advancing progress markers, as the task-state guide advises. Work that outlives the request, such as background processing or a step resumed later, has no call ID to inherit. Capture the request's call ID, the opaque x-agent-foundry-call-id value that identifies the original caller, and pass it on each deferred state store item operation so those operations still act for that caller. It isn't a downstream bearer token, so don't treat it as one. Protect mutable conversations and checkpoints with supported conditional writes or serialization, and test stale entity tags (ETags) and lost updates.

Bind approval and continuation to authorized work

Bind any required approval to the exact action, arguments, resource, approver, version, and expiry. Reauthorize changed work, and stop denied or expired work before its effect. Enforce approval in the runtime: the Toolbox guide notes that approval metadata doesn't block a direct MCP tools/call.

Figure 3. Approval, execution, and uncertain-outcome recovery. A generalized flow, not a specific customer's implementation.

Authenticate callbacks, bind them to operations, handle duplicates and out-of-order arrival, and define behavior after expiry or cancellation. Choose a supported continuation path on evidence, and decide explicitly whether the continuation contract needs another service.

Agree the cancellation and history contract

In production, cancellation turned out to start in the application's own interface. A stop request or closed browser must reach the running work through the application API and software development kit (SDK). Even a working cancel path didn't settle how partial output should appear on the next turn. So define cancellation across the user interface, server workflow, tool, and remote job: how to prevent further effects, how to report terminal status, and which partial history to keep under the application's retention contract.

Watch for:

• An open stream is not durable work, and "long-running" is not unlimited.
• Recovery can reenter a handler in a new process without restoring its call stack, and checkpoints don't atomically commit external effects.
• Marking completion before the action can lose work; recording it afterward without downstream deduplication can repeat it after a crash.
• Neither an operation key nor an effect ledger (the record of attempted business actions and outcomes) guarantees exactly-once execution.
• A write conflict may follow an already committed remote effect. Concurrency protection neither commits that effect nor justifies blindly replaying it.
• A model's assertion is not an approval record.
• Adding another continuation service by default, or ruling one out by assumption.
• Cancellation can't undo a committed request, and stopping or deleting a session is not canceling one turn.

Check before release: In an authorized test environment, inject controlled failures around effects/checkpoints, stale writes, callbacks, approvals, disconnects, and cancellation during generation/tool work. Verify the business outcome and the next authorized turn's history against the agreed contract.

Rejected alternative: Treating a successful cancel response or a saved checkpoint as proof of recovery. Neither verifies what happened downstream.

Further reading: Long-running agent resilience · Recover long-running work · Hosting guide · Manage task state · State store · Use Toolbox with hosted agents

5. Validate the complete network and capacity envelope

Validate every network path and capacity limit before the first regional rollout, because each one fails independently. Map image build and push, platform image pull, client ingress, runtime egress, models, tools, retrieval, identity endpoints, telemetry, and recovery. For each path's effective principal, validate Domain Name System (DNS) resolution, routes, private endpoints, firewall rules, Transport Layer Security (TLS) trust, certificate rotation, and authorization.

The virtual-network guide and Azure Private Link guidance distinguish inbound access, outbound injection, and tool-specific paths. Some choices have to be made when the resources are created, so resolve conflicting guidance for your topology before committing.

Plan independent bounds: regional availability, session quotas, provisioned and resumed compute, subnet Internet Protocol (IP) addresses, model tokens or requests per minute or provisioned capacity, tool throughput, storage, and concurrency. Concurrent hosted-session quotas are shared across accounts and projects within a subscription and region. Provisioning or running compute counts against them, and stopped or deprovisioned sessions need quota when they resume. Across subscriptions, each subscription-and-region pair has its own quota envelope, capacity plan, configuration, and monitoring.

Absorb resume bursts, running-but-idle sessions, and fleet-wide demand with bounded admission and queues, finite retries with backoff and jitter, and circuit breakers. Queue-based load leveling still needs bounded consumers, or it moves the overload downstream. Treat egress policy as a release dependency: with sandbox egress controls in Microsoft Foundry, verify policy adoption through the documented version and session lifecycle.

Watch for:

• An inbound private endpoint doesn't privatize every dependency.
• Another project doesn't add hosted-session quota in the same subscription and region.
• Multiplied retry layers.
• Egress audit mode isn't inert: transforms and rewrites still modify traffic, and policy updates don't hot-reload into running sandboxes.

Check before release: Use bounded test load and a region/network/capacity matrix to verify throttling, quota exhaustion, DNS failure, denied destinations, and resume contention in each region; one validated region is not evidence for another.

Rejected alternative: Giving each customer its own project and calling it done. That can clarify ownership, but it leaves shared quotas, permissions, networking, cost, and fleet operations unsolved.

Further reading: Virtual networks · Private Link · Limits, quotas, and regions · Handle transient faults · Queue-based load leveling · Hosted agent guardrails and egress

6. Treat knowledge ingestion and authorization as a subsystem

Give knowledge ingestion and authorization an owner, and judge them by what each user can retrieve rather than by deployment status or document count. Own source selection, extraction, chunking, embedding versions, index schemas, metadata, access control list (ACL) propagation, freshness, revocation, deletion, and rebuilds. A generally visible guide and a restricted internal methodology must not gain equal access just because both are indexed.

Derive security filters in trusted logic, and require trusted user context at the application boundary. Azure AI Search document-level access guidance separates application security filters from native permission features, and authorization to call Azure AI Search differs from the intended end user's query identity. For indexed knowledge sources that require ingestionPermissionOptions, configure it at creation, as the agentic retrieval guide describes. As an application control, deny protected retrieval when required prerequisites are absent, and test missing-user and missing-ingestion-permission cases.

Propagate permissions to every chunk and keep them current, populating permission fields in child-index projections as well as parent mappings. Microsoft SharePoint ACL guidance distinguishes item-level changes from inherited site, library, list, or folder changes, which need an explicit permission refresh followed by an indexer run, so test the connector you use. Configure blob deletion detection before initial ingestion, and confirm the indexer observes deletion within retention, including after outages or backlog. Include revocation and deletion in recovery design, because rebuilds, restored sources, caches, evaluation copies, and backups can reintroduce content.

Treat retrieved documents, including internal ones, as untrusted data, not instructions. Preserve provenance and classification, and admit only authorized, necessary context. Spotlighting can mark lower-trust content where the model and API support it.

Separate permission to retrieve, to reason with information, and to disclose it. "May reason but not disclose" requires a different design from "may not retrieve." In the example, internal knowledge had to inform answers without being quoted verbatim or extracted in bulk. Retrieval permissions alone can't express that policy. Where supported responses include sensitivity-label metadata, preserve it through authorized handoffs and enforce recipient and output policy downstream.

Watch for:

• A filter string is not authentication, and a service credential or connector registration is no substitute for the user's query identity.
• Missing-token behavior varies across retrieval paths.
• A valid user token can't compensate for missing ingestion permissions.
• Indexer reset doesn't remove every orphan, and a deletion policy added later won't repair entries that are already orphaned.
• Spotlighting is neither authorization nor a confidentiality guarantee.
• Groundedness and citations establish neither confidentiality nor independent truth, and protected-material detection is not customer-specific intellectual-property protection or data loss prevention.
• Carrying a sensitivity label is not enforcement, and missing metadata is not permission to disclose. Response shapes vary by API and client.

Check before release: With non-production documents and test users, test retrieval utility, denied users, revocation, injected documents, stale chunks, deletion, and rebuilds, checking generated answers as well as search results.

Rejected alternative: Relying on regular-expression filters alone. They catch literal matches but miss paraphrases, fragmented content, and alternate tool paths.

Further reading: Document-level access · Query-time access control · Agentic retrieval and sensitivity labels · SharePoint ACLs · Reset and run indexers · Blob change and deletion detection · Prompt Shields and spotlighting · Protected material detection

7. Build a control-coverage matrix, not a product checklist

Build a control-coverage matrix that names, for every boundary, which control applies and what evidence proves it works. Content classification, action authorization, network restriction, and threat detection serve different purposes. The applicability matrix distinguishes model controls from agent guardrails and tool intervention points, so check each feature's conditions; a product name alone doesn't settle it. For each boundary, record inspected content, protocol and tool support, effective policy, mode, enforcement owner, failure behavior, and expected evidence:

Check protocol and payload handling

The hosted guardrail guide makes configuration consequences explicit:

• Omitted rai_config means no hosted content-safety guardrail; including it without a policy name selects Microsoft.DefaultV2. Separately called model APIs can have their own controls.
• An invalid policy reference can report active without filtering. For rai_policy_name, use the full Azure Resource Manager resource ID, and verify existence and enforcement.
• Custom invocations needs invocations_moderation that matches input format, output, and stream fields. Without it, the attachment is inert, and plain text sent to a JSON-expected configuration can bypass input screening.
• invocations_ws doesn't support this hosted content-safety moderation path.
• The path has documented limits: requests over 2 MB, unparseable JSON, or input paths that select no text can bypass screening; text beyond 10,000 characters per check is truncated; buffered responses over 1 MB are rejected; and stream selectors match exact, case-sensitive event types and top-level text fields, so a selector that doesn't match can miss text. An output shape outside the declared response mode can be rejected with a Hypertext Transfer Protocol (HTTP) 502 error. Test your payloads against them.

Handle the whole response sequence, because an HTTP 200 status and partial streamed content can precede a terminal content_filter error. Minimize and sanitize error bodies, since this path doesn't screen container error bodies or empty acknowledgments, and keep raw prompts, tool results, and secrets out of them.

Check the supported intervention-point tool list for your custom MCP calls and in-process functions. Use Prompt Shields, personally identifiable information (PII) filtering, groundedness, and Task Adherence for the different risks each one addresses, and enforce their signals. Separate detection false positives from correctly detected information that business policy permits.

Test the control matrix adversarially

Attack every row of the matrix before release. Run cloud red teaming against the intended agent version, and add targeted tests where built-in coverage misses your workload's risks: in the example, attempts to extract internal knowledge or push the orchestrator out of scope. Part 2's improvement loop covers red-teaming targets, taxonomy review, and safe test environments, alongside rubric evaluators, Agent Optimizer, and Insights. These tools test the boundaries, and the boundaries still need their own enforcement.

Watch for:

• Microsoft Defender for Cloud detection is not universal inline prevention. Microsoft Defender agent real-time protection has its own interoperability boundaries: the default rule audits, and custom blocking rules need a supported interoperability path.
• None of these content controls grants authority to retrieve or execute tools, and model-only checks don't automatically apply at agent and tool boundaries.
• PII output filtering is not a telemetry sanitizer, and a Task Adherence risk signal still needs enforcement.
• Removing required protection to improve a benchmark.
• A clean scan is evidence about the tested scope only, not proof of secure isolation, every tool path, or every regional deployment.

Check before release: Exercise safe allow/block test cases for invalid policies, selectors, payload limits, partial streams, errors, unsupported paths, and drift. Record effective settings and residual gaps.

Rejected alternative: Replacing several controls with one product that seems to cover them all. Layered controls can add latency and over-blocking, but remove one only with approved evidence that coverage stays intact.

Further reading: Guardrails overview · Hosted agent guardrails · Intervention points · Defender for Cloud AI threat protection · Defender real-time agent protection · Prompt Shields · PII filtering · Groundedness · Task Adherence · Cloud red teaming

Turn the boundaries into release checks

Map one representative user journey end to end, for the example from question to verified answer, and give every trust boundary and store an owner. Run the checks in an authorized test environment with non-production data, two differently entitled test users, knowledge sources with different disclosure permissions, and isolated test endpoints for the data and job services. In the example, agents interpret and plan, and trusted application logic enforces scope, data access, disclosure, action authorization, and any required approval. The job service owns receipts and results: an accepted submission is not a completed job. Add a specialist only if its responsibility justifies a separate handoff.

Use these five checks as release criteria, adapted to your workloads, and record the observed result for each against the exact candidate; the expected behavior is the bar to meet:

Start each check as Not run, and record the candidate and configuration, expected and observed behavior, safe evidence, and disposition. Resolve support or control gaps before relying on the design, and repeat affected checks when dependencies change.

The five checks put the opening lesson into practice: a convincing response is only the start, and each boundary has to hold under test. Once the design passes them, the next question is whether users get the exact release that passed. Part 2: release and operations picks up there, with releases, evaluation, telemetry, latency, cost, operational recovery, and the improvement loop.