Building a Privacy-Aware Personal Agent with Microsoft Foundry Local

The project: useful assistance within a defined data boundary

The goal was to keep planning and private workspace access on the device while allowing selected, sanitised subtasks to receive cloud assistance. The cloud would see a restricted representation of the evidence. The local system would retain the original information and control the final answer.

I tested three kinds of work:

• Resolving current commercial terms when signed records, older agreements, and drafts disagree.
• Preparing a briefing by following a calendar event into related email and document evidence.
• Checking archive conditions and creating a local note when the task authorises it.

These tasks exercise behaviour beyond generating plausible text. An agent must identify the right source, preserve citations, follow permissions, and sometimes produce a persistent result. The project therefore combined an agent architecture with an evaluation framework that could check both answers and execution behaviour.

Building the local workflow

I organised the system around a structured workflow: coordination, planning, execution, finalisation, and validation. Microsoft Agent Framework provided the orchestration, while Foundry Local served the local language model.

The local coordinator decides whether a request needs a direct answer, clarification, or workspace access. For workspace requests, the planner produces typed subtasks with dependencies. The executor uses local tools to obtain evidence and carry out permitted actions. Some steps use a model; others, including retrieval and policy checks, are deterministic functions.

One deliberate choice was deterministic BM25-style document retrieval. Documents are split into overlapping chunks, and retrieved evidence includes file and line citations. This made the retrieval process reproducible and its provenance easier to inspect. It also avoided adding an embedding model as another variable in the experiment.

I screened models for planning, structured tool use, sanitisation, and finalisation, alongside latency and memory requirements. The frozen experiment used Qwen 2.5 7B through Foundry Local and DeepSeek V4 Pro for cloud reasoning, with thinking disabled. One local model served the different roles, reducing the operational complexity of keeping multiple models loaded.

The next challenge was adding cloud assistance without handing over the workspace.

How the privacy boundary works

Figure 1. The privacy workflow keeps the original request, raw tool results, and placeholder mapping on-device. “Untrusted” identifies the cloud as outside this project's trust boundary. This diagram summarises the workflow; execution and optional refinement occur per subtask.

Before considering a cloud call, the system sanitises a subtask's instruction and local result together. Presidio recognisers detect sensitive values, and an optional local model adds context-dependent detections. Identified values become stable typed placeholders, such as <PERSON_1> and <EMAIL_ADDRESS_1>. Processing the instruction and context together keeps those replacements consistent.

The mapping back to the original values stays local. A separate CloudTaskEnvelope contains only a task identifier, task kind, redacted instruction, redacted context, and behavioural constraints such as preserving citations. It has no field for raw tool objects or the placeholder mapping.

This division is important: the cloud adapter receives the envelope, rather than the complete local execution state.

The router then applies deterministic privacy checks. A sanitisation error, critical secret, unavailable cloud service, or exhausted escalation budget keeps the subtask local. Only after those checks does it consider whether cloud refinement is warranted. Hybrid uses a fixed threshold of 0.5 on a heuristic score based on local confidence and task or evidence characteristics. That score is not a calibrated probability.

After a cloud call, the system checks the response for problems such as raw-value leakage, invented placeholders, and detected personal information. Further guards check whether required fields or all existing citations have been lost. A failed call or rejected refinement falls back to the local result. Accepted refinement is combined with local evidence, and permitted placeholder restoration happens on-device.

This cycle runs before dependent subtasks continue. Cloud assistance can therefore affect subsequent reasoning, while workspace tools remain under local control.

The technical components

The implementation is written in Python. Each major component has a specific role:

The file tools enforce a separate sandbox. They reject paths outside the permitted workspace and restrict writes to new notes in a designated output directory, without overwriting existing files. Operational traces exclude prompts, document contents, tool results, and placeholder mappings. This extends the data-boundary design into debugging and observability.

Evaluating the complete system

I generated fictional workspaces with known facts and sensitive-value annotations. This made it possible to check what crossed the cloud boundary without exposing real personal records.

The benchmark contained 27 tasks: three task families, three complexity levels, and three cases per family–level combination. Complexity increased through additional sources, distractors, reasoning steps, and conflicts. Each task ran three times under five conditions, producing 405 executions.

The three primary conditions were Local, Hybrid, and Structured-cloud. Local made no cloud calls. Hybrid selectively escalated sanitised subtasks. Structured-cloud used the broad structured workflow with cloud models in its cognitive roles and access to raw synthetic evidence, providing a reference ceiling within this experiment.

Two diagnostic conditions added context. Cloud-escalated sent every privacy-eligible subtask for cloud refinement. Direct-cloud used an autonomous cloud tool loop rather than the fixed structured workflow.

These are comparisons of complete systems. Their differences include model placement, context, orchestration, and privacy controls. The experiment does not isolate model capability alone.

The primary evaluator checked facts, conflict resolution, citations, outcomes, and instruction following. It reported a graded quality score out of 100 and a stricter pass measure requiring full credit on every applicable item. Privacy and execution-policy gates remained separate from answer quality. I also used Azure AI Evaluation on the saved answers as a complementary semantic check.

Results: a measurable quality gain with additional latency

Quality scores average repetitions within each task, then average across the 27 tasks. Payload is an arithmetic mean. Structured-cloud deliberately sends raw synthetic evidence and does not meet the study's raw-data deployment restriction.

Hybrid improved on Local by 3.24 points, with a 95% task-bootstrap confidence interval of 0.84–5.78 points. It recovered 60.72% of the quality gap between Local and Structured-cloud while sending 18.89% of Structured-cloud's outbound payload volume. Payload volume measures bytes sent, not a percentage of privacy preserved.

The gain was larger at higher complexity: 1.23 points on Easy tasks, 3.48 on Medium, and 5.02 on Hard. Individual outcomes varied: Hybrid improved nine tasks, tied sixteen, and scored lower on two.

Figure 2. Quality and outbound payload across the five conditions. Filled markers indicate conditions that passed the declared raw-data rule; hollow markers identify references that cross that boundary. The vertical axis starts at 91.5, and kB denotes 1,000 bytes. The connecting line concerns quality and payload only, not latency or all deployment requirements.

The improvement came with a clear latency cost: Hybrid's median was 93.33 seconds, compared with 63.10 seconds locally. Its mean cloud inference cost was approximately US$0.000232 per task under the experiment's recorded pricing, compared with US$0.000758 for Structured-cloud. These figures exclude local hardware and energy costs and are historical experiment measurements, not current price estimates.

Cloud-escalated scored 95.44, close to Hybrid, despite sending more data. Direct-cloud scored 92.66. Those observations suggest that orchestration and the evidence available to a model deserve attention alongside cloud access. The small Hybrid–Cloud-escalated difference does not establish a general advantage for the routing policy.

All 405 executions returned non-empty answers. Local, Hybrid, and Cloud-escalated passed the declared raw-data rule, with no recorded raw-value or entity leakage in the evaluated cloud payloads. However, privacy success did not imply that every execution passed all other checks: Hybrid met the combined deployment gates in 78 of 81 executions because separate tool-policy/security failures remained.

Azure's assessment broadly supported the same quality ordering on the primary conditions. It evaluated the same answers, so this is agreement between evaluators, rather than an independent replication.

What I learned

The project changed how I think about evaluating an assistant. A fluent answer gives only a partial view of success. I also need to know which evidence it used, whether it followed permissions, and whether an intended action actually happened. Keeping those checks separate made failures easier to interpret.

I learned to make privacy requirements inspectable in code. A dedicated cloud contract, local restoration state, and deterministic routing guards gave me concrete interfaces to test. That approach also exposed the limitations of the guarantee: a perfect score under the synthetic leakage evaluator does not prove resistance to re-identification, cumulative disclosure, or every possible attack.

Reproducibility became a substantial part of the engineering work. Saved answers, versioned scoring, corpus hashes, and reset workspace state allowed analysis to be repeated without calling the models again. Inspecting disagreements between deterministic scoring and semantic evaluation also reinforced the need to examine individual records behind aggregate scores.

Supervision and acknowledgements

This was an individual research project. I developed the implementation and evaluation and carried out the analysis. Lee Stott at Microsoft and Gerard Gorman at Imperial College London supervised the project. I particularly thank Lee for his guidance throughout the work.

AI coding assistants supported development and review, as disclosed in my dissertation. I reviewed and verified their contributions and remain responsible for the design, implementation, and interpretation.

What comes next

The present evidence comes from a synthetic development corpus, three task families, one device, and one selected local–cloud model pair. There was no real-user study, and three repetitions give limited evidence about run-to-run variation.

The next step is evaluation on newly generated, unseen tasks and carefully controlled real-workspace tasks. Privacy testing should also examine adversarial inputs, re-identification, and disclosure accumulated over multiple interactions. Finally, experiments across model pairs and hardware could test whether calibrated routing and a more efficient local pipeline retain the quality gain with less waiting.

Potential applications include meeting preparation and document-based assistance in settings where raw records must remain on-device. Demonstrating usefulness in those settings will require both broader testing and feedback on latency, refusals, and restored answers.

Build and evaluate your own local-first workflow

This project showed that selective, sanitised cloud refinement can improve a local personal agent within a defined benchmark. The useful engineering outcome is an explicit boundary that can be inspected, alongside measurements that reveal its quality and performance trade-offs.

If you are exploring a similar design, begin with a small fictional workspace and one task whose correct result you can verify. Establish a local baseline, inspect the exact data proposed for cloud transfer, and measure the result after adding optional refinement.

These resources cover the technologies used in the project:

• Get started with Foundry Local: run a model locally and explore its lifecycle.
• Microsoft Agent Framework overview: learn about agents, tools, and explicit workflows.
• Presidio documentation: explore sensitive-information detection and de-identification.
• Evaluate AI agents with Azure AI Evaluation: examine semantic and agent-behaviour evaluation. This resource documents the classic SDK workflow used by the project; consult its applicability notes when building a new system.

Which part of your agent's workflow truly needs cloud reasoning, and what evidence would you need to justify sending it? Share your use case or questions in the comments.