PrivyDoc: Building a Zero-Data-Leak AI with Foundry Local & Microsoft's Agent Framework

MVP

Nov 11, 2025

I'm Shivam Goyal, a Microsoft MVP in AI with a deep passion for Artificial Intelligence and its power to solve complex, real-world problems. I'm constantly amazed by AI's ability to tackle intricate challenges, but I also see a huge one on the horizon: the clash between powerful AI analysis and the non-negotiable need for data privacy.

How can we use AI on our most sensitive documents without uploading them to the cloud?

This is the exact problem I set out to solve. In this article, I want to take you on a journey through the creation of PrivyDoc, a secure, on-device document intelligence solution I built to address this challenge head-on. We'll explore how it uses two powerful Microsoft technologies, Foundry Local and the Microsoft Agent Framework, to bring the AI to the data, not the other way around. Let's dive in!

The Core Problem: Data Privacy vs. AI Insights

Let's face it, for many organizations, the trade-off between powerful AI insights and data security is a non-starter. Think about the hurdles:

Data Privacy Concerns: You can't just upload confidential legal contracts or patient records to a third-party server.
Compliance Rules: Strict regulations like HIPAA or GDPR, plus internal data policies, often forbid data from ever leaving the local environment.
Air-Gapped Environments: Many secure facilities (think government, finance, or R&D labs) operate completely offline, making cloud-based tools useless.
Lack of Control: When your data is in the cloud, you lose control. You don't know how it's being processed, stored, or if it's being used for training.

PrivyDoc tackles these challenges by inverting the model. Instead of sending your documents to the AI, it brings the AI to your documents.

The Core Technologies: Foundry Local & Agent Framework

PrivyDoc's power comes from two key Microsoft technologies. One provides the secure, local brain (Foundry), and the other acts as the intelligent coordinator (Agent Framework).

1. Microsoft Foundry Local

Foundry Local is a toolkit that lets developers download, set up, and run powerful AI models (like Microsoft's Phi-3.5-mini) directly on a local machine. It provides an OpenAI-compatible API, making it simple to interact with, but its operation is 100% on-device.

Its Importance in PrivyDoc: This is the entire foundation of PrivyDoc's privacy promise. By using Foundry Local, PrivyDoc guarantees zero data transmission. The contents of a sensitive legal contract or medical record never leave the user's computer. This makes the tool "air-gap compatible," inherently compliant, and completely secure. It's the engine that performs the AI analysis locally.

2. Microsoft Agent Framework

The Microsoft Agent Framework is an open-source framework (a successor to projects like AutoGen and Semantic Kernel) for building and orchestrating multi-agent applications. Instead of relying on one giant AI model to do everything, it allows you to create a team of specialized AI "agents" that collaborate on a complex task.

Its Importance in PrivyDoc: Document analysis isn't one simple step. PrivyDoc uses the Agent Framework to build an intelligent pipeline. There's an agent to extract text, another to identify document sections, a third to perform Named Entity Recognition (NER), and a fourth to summarize and analyze sentiment. The Agent Framework manages this entire workflow, passing the document from one specialist agent to the next. This makes the analysis more robust, modular, and manageable.

How It Works: An Architecture of Privacy

PrivyDoc’s architecture is built on this "local-first" principle.

Document Upload & Text Extraction: The user uploads a PDF or DOCX file via the Chainlit-powered web interface. PrivyDoc first uses parsers to extract clean text.
Multi-Agent Pipeline (Agent Framework): The extracted text is handed off to the multi-agent pipeline orchestrated by the Microsoft Agent Framework.
- Section Extraction Agent: Identifies the document's logical sections.
- Entity Recognition (NER) Agent: Scans the text for entities like People, Organizations, Locations, and Dates.
- Summarize & Sentiment Agent: Generates concise summaries and determines the emotional tone.
Local AI Inference (Foundry Local): Each time an agent in the pipeline needs to "think" or perform an analysis task, it makes its request not to a cloud API, but to the Foundry Local service running on the user's machine.
Results Compilation: Finally, all this information is compiled and presented to the user in the web UI.

Key Features and Capabilities

Multi-Format Support: Processes both PDF and DOCX files locally.
AI-Powered Insights:
- Smart Summarization: Get concise overviews of entire documents or specific sections.
- Entity Recognition: Automatically extract people, organizations, locations, and dates.
- Sentiment Analysis: Analyze the emotional tone at both the document and section levels.
- Topic Classification: Automatically categorize documents by subject matter.
Security & Compliance:
- 100% Local Processing: Powered by Foundry Local, no data ever leaves your device.
- Air-Gap Compatible: Fully functional in offline environments after the initial model download.
- Analysis Traceability: All processing interactions are logged locally.
User-Friendly Interface:
- Web UI: A clean web interface built with Chainlit allows for easy file uploads, progress tracking, and result exploration.
- CLI: A command-line interface is also available for batch processing.
Flexible Export: Analysis results can be exported in Markdown, JSON, or CSV formats for use in other systems.

Tech Stack Summary

AI Engine (Local): Microsoft Foundry Local
AI Orchestration: Microsoft Agent Framework
AI Models: qwen2.5-0.5b (default), phi-3.5-mini, phi-4
Web UI: Chainlit
Language: Python 3.10+

Conclusion

PrivyDoc is a powerful example of how modern AI can be deployed in a secure, privacy-first way. By leveraging Microsoft Foundry Local, it smartly moves the compute to the data, not the other way around. By using the Microsoft Agent Framework, it builds a sophisticated, multi-step analysis pipeline that delivers deep insights.

For any field where document confidentiality is non-negotiable (like legal, healthcare, or finance), PrivyDoc provides a clear path to embrace AI intelligence without compromise.

If you'd like to know more about the project, discuss AI privacy, or collaborate, feel free to reach out to me on LinkedIn or check out the project on GitHub !