A production agent should remember a user's language preference, an unresolved support issue, or a proven workflow. It should not quietly retain credentials, health details, financial data, prompt-injection payloads, or a stale instruction that changes how a tool behaves. The difference is not the model. It is the architecture around the model.

Microsoft Foundry Agent Service provides managed memory stores that can extract and retrieve user-profile, chat-summary, and procedural memory. It also supports item-level management, default time-to-live (TTL), and direct remember-or-forget behavior. Those primitives are useful—but a production design still needs application-level policy for scope, purpose, consent, provenance, conflict handling, observability, and deletion.

Implementation path  Build an identity-bound scope, separate retrieval and write-admission gates, expose user-visible lifecycle controls, and release only after privacy-aware tracing and stateful evaluation pass.

Figure 1. Governed memory state architecture for production AI agents in Microsoft Foundry. Retrieval passes through an authorization gate; candidate memories pass through a separate write-admission gate; lifecycle events flow to an evidence plane.

1. Start with a memory contract, not an API call

Before provisioning a store, write a one-page memory contract for the agent. It should name the business purpose, allowed memory classes, prohibited content, scope key, default retention, deletion promise, owners, and evidence required for an audit. This turns an ambiguous feature—“the agent remembers”—into a reviewable system behavior.

2. Partition state so one user—or purpose—cannot bleed into another

Foundry Memory uses a scope value to isolate collections of memory items. Treat that value as a security boundary. For an end-user agent, derive scope from an authenticated, tenant-qualified subject—not from an email address supplied in the prompt. For direct low-level API operations, send the scope explicitly. For an attached memory search tool, the platform can resolve the signed-in user when configured with the supported user placeholder.

A safe scope is stable enough to survive sessions but opaque enough that logs and operators do not receive a raw personal identifier. A keyed HMAC also prevents a malicious caller from guessing another user's scope.

SCOPE DERIVATION

4. Put a write-admission gate between conversation and durable state

Extraction is not authorization. A model can propose a candidate memory, but deterministic application policy must decide whether the candidate is eligible for persistence. This is where the architecture prevents accidental retention and memory poisoning.

A practical gate performs five checks: authenticated scope, allowed memory type, purpose and consent, sensitivity classification, and lifecycle metadata. It then deduplicates the candidate and resolves conflicts before calling the memory API. Store the reason for admission separately in an audit record; avoid placing operational metadata inside user-visible memory text.

APPLICATION WRITE-ADMISSION POLICY

 

For higher-risk agents, require a source allowlist for procedural memory. A procedure inferred from an untrusted webpage, a user-uploaded prompt, or a failed tool run should not become durable operating guidance. Promote procedures only from successful, evaluated trajectories—or through a human-reviewed workflow.

5. Retrieve selectively and label memory as untrusted context

More retrieved memory is not better memory. Retrieve profile facts once near the start of a session, retrieve summaries only when relevant to the current turn, and retrieve procedures only when the intent and tool contract match. Apply a minimum score, freshness rules, type allowlists, and a token budget before the content reaches the model.

POST-RETRIEVAL FILTER

 

6. Design remember, forget, inspect, and correct as product features

Governance becomes real when a user can see and change what the agent retains. Provide four explicit operations: remember a fact, forget a fact, list retained memories, and correct a memory. The UI should preview the normalized fact, its purpose, and its expected retention before committing it.

1. Authenticate the caller and derive the scope server-side.
2. Parse the request into a typed command; the model may assist, but the endpoint owns validation.
3. Apply the same sensitivity and purpose gate used for automatic extraction.
4. Use an idempotency key so retries do not create duplicate facts.
5. Write an audit event containing actor, scope hash, operation, policy version, outcome, and correlation ID.
6. For “forget,” delete or tombstone the target, invalidate caches, and verify that subsequent retrieval no longer returns it.

LIFECYCLE ENDPOINT PATTERN

 

The repository in this example is an application abstraction over the supported Foundry memory-management API. Keeping the contract in your application prevents preview SDK changes from leaking across the codebase and makes lifecycle tests deterministic.

7. Keep thread storage, memory, and enterprise records distinct

A thread is a conversation transcript. A memory is distilled context selected for future use. A system of record is authoritative business data. They can refer to one another, but they must not be collapsed into one store or one retention policy.

 

Organizations that require customer-owned data resources, customer-managed keys, or network isolation can use the standard Agent Service setup with their own Azure resources for supported thread and file workloads. Current Foundry Memory preview limitations must be evaluated separately; for example, the current documentation states that VNet integration is not supported for memory stores.

8. Use identity and least privilege all the way to runtime

Prefer managed identity in Azure and DefaultAzureCredential in code. Separate the identity that deploys infrastructure from the project identity that runs the agent. Grant roles at the narrowest practical scope and test denied paths—not only successful paths.

• Provisioning identity: can create or configure the project and required resources, but is not used by the runtime.
• Project managed identity: receives only the data-plane roles needed by the agent and its connected resources.
• Developers and operators: use separate human roles for authoring, support, and audit; do not share service credentials.
• End user: is authorized by the application. A valid model response never substitutes for an authorization decision.

CREDENTIAL PATTERN

 

9. Trace decisions without turning telemetry into a shadow memory store

Connect Foundry observability to Application Insights and emit a span for each memory decision. The trace should explain why memory was retrieved or rejected without copying full memory text. Record type, hashed scope, policy version, result count, latency, score distribution, and the policy outcome. Sample or redact content at the instrumentation boundary.

PRIVACY-AWARE TRACING

 

Build dashboards and alerts around outcomes: write rejection rate, retrieval-empty rate, stale-memory rate, forget-request completion time, cross-scope authorization failures, conflict frequency, token contribution, and the quality delta with memory enabled. A sudden rise in accepted procedural memories or a drop in write rejections can signal a policy regression.

10. Evaluate stateful behavior, not only single-turn answer quality

Memory changes behavior over time, so evaluation must span episodes. Create test users with known scopes, run multi-session trajectories, and compare behavior with memory on and off. Include adversarial attempts to poison memory, retrieve another user's state, revive expired content, or convert retrieved text into a privileged instruction.

 

For reliability, repeat the same stateful scenario multiple times. A memory system that succeeds once but fails under retries, concurrency, or delayed extraction is not production-ready.

11. Build an operational runbook before launch

The production team needs a runbook for memory—not just for the agent endpoint. Assign an owner and measurable objective for each lifecycle path.

Practical launch gate  Do not enable production writes until you can demonstrate isolation, prohibited-data rejection, user-visible deletion, trace redaction, and rollback in a staging environment with production-like identity and retention settings. 

12. Reference implementation sequence

A small team can implement the design incrementally. Keep the first release narrow: one agent, one purpose, one memory store, and one approved profile category. 

1.  Define the memory contract and threat model. Choose the first allowed memory category and the default TTL.

2.  Provision a dedicated store, chat model, and embedding model with pinned SDK and infrastructure versions.

3.  Implement authenticated scope derivation and negative isolation tests.

4.  Add retrieval with type, relevance, freshness, and token-budget filters. Delimit results as untrusted context.

5.  Add candidate extraction in shadow mode. Measure what would be accepted and rejected before enabling writes.

6.  Enable write admission for the narrow category, with consent/provenance evidence and idempotency.

7.  Expose inspect, correct, remember, and forget operations; verify deletion end-to-end.

8.  Connect traces and dashboards, then run stateful utility, isolation, lifecycle, and poisoning evaluations.

9.  Promote only when the runbook, rollback, and evidence package are complete.

 

Closing perspective

Agent memory earns trust when it is useful and bounded. Microsoft Foundry supplies managed memory primitives, lifecycle controls, identity integration, storage options, and observability hooks. The production architecture must connect those capabilities into a governed state machine: authorize the read, admit the write, constrain its influence, observe the decision, and honor deletion.

That approach changes the design question from “How do we make the agent remember more?” to “What state is this agent allowed to remember, for whom, for what purpose, for how long, and with what evidence?” The second question is the one production systems can safely answer.

Implementation notes and source references

1. https://learn.microsoft.com/en-us/azure/foundry/agents/how-to/memory-usage
2. https://learn.microsoft.com/en-us/azure/foundry/agents/concepts/what-is-memory
3. https://learn.microsoft.com/en-us/azure/foundry/agents/how-to/use-your-own-resources
4. https://learn.microsoft.com/en-us/azure/foundry/agents/concepts/standard-agent-setup