Architecture First: Drawing the Boundary for On-Premises Health RAG

A retrieval-augmented generation prototype can look convincing after an afternoon: load a few documents, create embeddings, retrieve passages, and ask a model to answer. A health-records system is harder for a reason that has little to do with prompt wording. Before retrieval quality matters, the team must be able to answer a more basic question: where can sensitive data travel?

This prototype starts there. Its primary invariant is that health-record data, prompts, vectors, retrieved evidence, conversations, model output, and operational telemetry stay on premises. There is no cloud inference path and no silent cloud fallback. If local inference is unavailable, model-dependent work degrades or fails explicitly.

The invariant is not a stylistic preference. A number of countries require, by law, that processing connected to the provision of primary or secondary health care be carried out through a server and data centre located inside the country, or that at least one serving copy of the data be held there. Kenya, where the research behind this series was carried out, is one such jurisdiction: it designates health care a strategic interest of the state and imposes exactly that requirement, permitting cross-border transfer of health data only on an adequacy finding or appropriate safeguards. A hosted chat model, a managed vector store, or a cloud reranking API is processing of record content, and it places that content outside the country. Keeping every stage on premises satisfies the strictest form of a residency rule by construction: no copy of the record set leaves the facility, so the transfer regime never has to be engaged. Jurisdictions differ in how far they go, but the shape recurs - where a residency duty applies, the deployment decision is settled before the architecture starts.

That decision shapes every component. It determines where credentials live, which process talks to databases, how streaming crosses the client boundary, and why the system separates operational clinical databases from its internal search store. It also keeps the prototype honest: “local model” is not enough if the browser holds a bearer token, a hosted reranker sees passages, or logs export prompts.

The series implementation uses Rust, but this pattern can be implemented with the Foundry Local SDK Reference in C#, JavaScript, Python, or Rust; Part 2 provides links for each language.

The other half of the constraint is the hardware. A referral hospital's compute is usually one commodity server already running an EMR, on a grid that is not guaranteed - in Kenya, 26% of medical facilities have no power connection at all, and only 15% of connected ones get uninterrupted electricity. “Keep it on premises” is only actionable if the premises can run it. Two things make that possible: small language models under ten billion parameters that are competitive on medical benchmarks, and an inference runtime that is not tied to one accelerator. Foundry Local resolves a model alias to a device-specific variant at load time across CPU, WebGPU/DirectML, CUDA and TensorRT on NVIDIA, OpenVINO on Intel GPU or NPU, QNN on Qualcomm NPU, and Vitis AI on AMD NPU - so the deployment target is “whatever the facility owns” rather than a required SKU. The evaluation behind this series ran generation on an integrated GPU within a 32 GB envelope.

 

The regulatory and requirements analysis summarised here is drawn from the research project behind this prototype: Gatimu, K. C. (2026), *An On-Premises Retrieval-Augmented Generation (RAG) System for Retrieving Health Records in Level 5 Hospitals in Kenya*, MSc research project, Kirinyaga University.

 

This is a working research prototype using synthetic data. The Desktop App is implemented; the Mobile App is initialized but not production-device validated. Core authentication, ingestion, retrieval, structured query, local generation, and streaming paths exist. The service-line agent roster and per-source schema binding described below are the accepted target design (specified in plans/new/) and are being implemented; the current build still ships the legacy agent kinds (auto, health_query, trends, patient_lookup, summarize, chat) and dev-schema-specific SQL templates. TLS, OS-backed token storage, strict certificate validation, complete audit governance, representative clinical evaluation, and production capacity certification remain open. This article is an architecture report, not clinical advice and not a compliance claim.

Problem statement

The system had to be buildable under a data-residency obligation: record content, and everything derived from it, must remain on a server inside the facility. That excludes the cloud-hosted retrieval and generation services most clinical RAG implementations assume, and it makes the data boundary a design requirement rather than a deployment detail.

An on-premises RAG project can still have an undefined data boundary. Teams may describe every database as a source of truth, let clients call APIs directly, or assume that choosing a local chat model prevents prompts, vectors, credentials, and telemetry from leaving approved systems.

That ambiguity creates concrete failure modes: a hosted fallback can leak sensitive context, a browser-held token can bypass intended policy, and a derived search copy can be mistaken for the operational clinical record. Product layers may also enforce different authorization or freshness rules, making answers difficult to interpret and audit.

The architecture must work without trusting client behavior or relying on cloud services. It must preserve the distinction between External Clinical Databases and the Internal DocumentDB Hybrid Store, keep policy in the server, and make every model, retrieval, streaming, and failure path obey the same on-premises boundary.

Success criteria

• Every sensitive-data flow has an explicit on-premises destination and owner.
• No stage - connector read, chunking, embedding, retrieval, reranking, generation, or telemetry - places a copy of record content outside the facility.
• External Clinical Databases remain authoritative; derived search state is labeled separately.
• The Desktop App and Mobile App cannot bypass server-side policy or directly hold the JWT.
• Model-dependent work fails or degrades explicitly without a cloud fallback.
• The same build runs on CPU-only, integrated-GPU, discrete-GPU, and NPU hosts, degrading in speed rather than failing.
• Structured and semantic answers retain distinct, inspectable provenance.

Begin with invariants, not products

A boundary-first design names rules that remain true even when frameworks or models change:

1. Sensitive data stays on premises. No prompt, source row, vector, citation, or generated answer is sent to a hosted model or retrieval service.
2. The server is the policy boundary. Authorization, source access, query validation, model routing, persistence, and resource limits are server responsibilities.
3. The client view is not trusted with credentials. The web-view layer renders data but does not hold the JWT or connect directly to server APIs.
4. Clinical systems remain authoritative. PostgreSQL, MySQL, and SQL Server retain the operational record. The internal store contains derived, searchable copies and application state.
5. Evidence types stay distinct. Live SQL answers preserve SQL and returned rows; internal aggregations preserve their validated specification and pipeline; semantic answers preserve citations.
6. Failure never widens trust. A failed structured path may fall back to local semantic retrieval, but it cannot bypass authorization or escape to cloud inference.
7. Agents are bound to functions, not table names. An agent owns hospital concepts - admissions, deliveries, bills - and a per-source binding decides which physical tables and columns supply them. No agent, router rule, or SQL template names a table directly.

The broader RAG solution design and evaluation guidance makes the same important point from another direction: a RAG system is a pipeline whose quality must be evaluated stage by stage. In a sensitive deployment, that pipeline must also be evaluated boundary by boundary.

The deployment topology

The product clients are called the Desktop App and Mobile App. They share a Tauri v2 shell, a Rust bridge, and a React/TypeScript web view. Those technologies matter to implementation, but they are not separate product tiers.

The diagram intentionally separates External Clinical Databases from the Internal DocumentDB Hybrid Store - documents + vectors + metadata/state. Collapsing them into one “database” box would hide a critical ownership distinction.

The external databases are systems of record. The server connects with least-privileged credentials, introspects schemas, reads bounded pages for ingestion, and can execute validated read-only SQL for exact questions. The internal DocumentDB store is application-owned. It holds copied chunks, vectors, schema cards, ingest state, conversations, settings, users, and audit events.

Why a bridge exists between the clients and server

A desktop application with a web view can accidentally recreate browser security problems inside a trusted-looking package. This prototype treats the web view as presentation, not as a secret store or network authority.

The React layer invokes typed Tauri commands. The local Rust bridge owns the server URL and bearer token, adds authentication to requests, consumes server-sent events, and emits typed application events. The web layer receives public user information and response content, but not the JWT itself.

This arrangement reduces accidental exposure through ordinary JavaScript state, developer tools, and frontend networking code. It does not magically secure the device. The current bridge persists its token in a plaintext plugin store, so moving credentials to Windows Credential Manager and Android Keystore-backed storage is a production requirement. The current web-view content security policy and Tauri capabilities also require hardening.

The lesson is broader than Tauri: put credentials and transport enforcement behind an interface the rendering layer cannot bypass. Then test that boundary as deliberately as any API.

One server owns policy and orchestration

The Rocket 0.5 server is the center of trust. It verifies stateless HS256 JWTs, reloads users to enforce token-version revocation, applies role and ownership checks, decrypts saved source credentials, validates structured operations, and limits expensive work.

Keeping this logic server-side matters because client-side route hiding is only user experience. A modified client must not gain broader data access. The server also owns conversation memory, so a client cannot inject an invented transcript as authoritative context.

The server is async under Tokio, but not unbounded. Process-local semaphores constrain generation, retrieval, global ingestion, and per-source ingestion. Saturation returns a retryable response instead of allowing every request to compete for model memory and database connections. Separate guards serialize native model lifecycle operations and prevent unloading a model during generation.

This is a practical prototype pattern: admission control belongs near the scarce local resource, not merely in the UI.

Split generation from retrieval models

Foundry Local is embedded in the Rocket process through foundry-local-sdk v1. On Windows, the project enables the winml feature. The native SDK handles local model discovery, acquisition, loading, hardware execution providers, and chat generation. There is no application-managed Foundry HTTP service and no cloud inference endpoint.

Delegating execution-provider selection to the SDK is what makes one build deployable across facilities. The server reports what it found, classified by accelerator - CPU; WebGPU/DirectML, CUDA, TensorRT, or OpenVINO on a GPU; QNN or Vitis AI on an NPU - and a task-aware router expresses each model role’s placement as an ordered device preference rather than a fixed device, so [Npu, Cpu] or [Gpu, Cpu] resolves to the best variant a given host can actually load. Part 2 covers the lifecycle and the traps; the architectural point is that hardware variability is absorbed at load time instead of forking the deployment.

The project currently defaults its generative roles to the qwen3-8b alias. Those roles include grounded chat, query rewriting, ambiguous route classification, structured planning and narration, SQL planning, memory compaction, and optional extraction or verification.

Embedding and reranking are deliberately separate. fastembed runs BGE-M3 to produce 1,024-dimensional vectors and runs bge-reranker-v2-m3 as the cross-encoder. Foundry Local SDK Reference covers the generation SDK used by this Rust project; Foundry Local does not create contentVector values or rerank retrieved passages here.

That division prevents a common documentation error: treating “the LLM” as if it performs every AI task. Embedding, reranking, and generation have different model compatibility, lifecycle, concurrency, and evaluation requirements.

Two answer paths, two kinds of truth

Not every question should be answered from a semantic top-k list. “Summarize the latest notes” and “How many encounters occurred by month?” require different execution.

The router therefore separates semantic and structured work, and the structured branch is a ladder whose every rung is decided by Rust:

Before routing, a model-free focus step resolves cheap references - “her”, “that ward”, “the same period” - from typed conversation state, so downstream stages see a self-contained question. The router is tiered: a regex gate for conversation, then a deterministic-first parse that tries to turn the question into a typed QuerySpec before any model is consulted, then lexical markers, then a small local classifier, and finally a single bounded clarifying question when the intent is clearly structured but a required slot is missing. A successful deterministic parse is a routing decision.

For structured work, the model proposes and Rust decides. One source is linked, and the candidate tables are restricted to the service line the question belongs to. The deterministic path binds the QuerySpec against that source’s schema binding and compiles dialect SQL; on a miss, the local model is asked to fill the same typed QuerySpec, not to write SQL, and a spec that does not bind is rejected before it reaches a database. Either way, the resulting SQL must parse as one read-only statement, use allow-listed tables, satisfy row and cost limits, and execute through a constrained connector with a timeout. The answer carries the rows and a provenance record of the exact path taken.

Bounded failure has a precise meaning: parse miss, bind error, validation error, connector error, timeout, or cost rejection. An empty result set on a list or grouping is a hit with an honest “none”; it does not trigger a fallback that would swap an exact answer for a fuzzy one. When a rung does fail, the next is a validated aggregation over ingested copies restricted to the same scope, and then semantic retrieval filtered to the same tables.

Semantic retrieval uses the last successfully ingested copy rather than claiming live freshness. It rewrites and expands the query locally, embeds it with BGE-M3, searches vector and lexical indexes, fuses ranks in Rust, deduplicates rows, reranks candidates, applies a relevance gate, and gives only bounded numbered passages to Foundry Local. Conversation memory may clarify the question, but it is not citable evidence.

This split preserves semantics. Exact counts should come from exact operations; narrative synthesis should cite retrieved records. The current build implements the older form of this ladder with legacy agent kinds and hardcoded templates; the version drawn above is the target specified in router v3 and the structured execution plan.

Agents follow the hospital, not the query

The first iteration of this prototype named its agents after query mechanics: Health Query counted, Trends bucketed, Summarize narrated. That taxonomy is invisible to the people who use a hospital. A midwife does not think “this is an aggregation question”; she thinks “how many deliveries did we have last night”. It also did not travel: every template and ownership rule was written against the development seed’s table names, so a hospital whose schema says PatientMaster and OB_Delivery would get nothing.

The target design separates two things that had been fused.

A fixed hospital ontology. The agent roster is thirteen service lines - Patient Chart, Front Desk, Ward Board, Emergency, Maternity, Theatre, Pharmacy, Diagnostics, Revenue, Quality & Safety, Workforce, Chronic Care, Facilities - plus Ask, the router-driven entry point. Each service line owns entity concepts (Admission, Delivery, Prescription, Bill, Incident), and concepts declare the column roles they need (a patient reference, an event time, a status, an amount). Ownership overlaps where the hospital’s does: Admission belongs to Ward Board, Emergency, Maternity, and Theatre alike. Patient, Encounter, Provider, Department, and DiagnosisCode are shared by everyone.

A per-source schema binding. When a source’s catalog is refreshed, a binder scores every table against every concept - normalized name tokens, column vocabulary, required roles, a BGE-M3 similarity between the table card and the concept description, and foreign-key shape - and assigns each column a role. Administrators can override any table’s concept or column’s role without touching code, and the binding is rebuilt. The result is one SchemaBinding per source that every downstream stage reads: the router’s lexicon, the deterministic SQL binder, the schema linker’s candidate list, the aggregation planner’s catalog, the retrieval filter, and the agent personas themselves.

Ownership here is a read allow-list, not a security boundary. It narrows the schema-linking search space from the whole database to the handful of tables a service line writes to - the single largest lever on structured-query accuracy - and it lets an agent answer “what can you do?” truthfully from its bound concepts. Role-based authorization stays where it was, in the server’s auth layer. A Maternity question that names the payments table gets a redirect to Revenue, not a join.

The portability argument is the point. What is stable across hospitals is the function: every hospital admits patients, books appointments, dispenses drugs, and sends bills. What varies is naming, language, and normalization. Binding the fixed ontology to the variable schema at catalog time means the same agents, grammar rules, and personas work on most hospitals’ databases, and a deployment that binds badly is fixed with an override rather than a release. The design, binder algorithm, and acceptance tests are specified in plan 01 and the product shape in Hospital agents and their data map.

Ingestion is publication, not bulk copying

A naive refresh deletes old data and starts writing new chunks. If embedding fails halfway through, search is left with a partial corpus. This prototype uses per-table generations instead.

Rows are read in bounded pages, explicitly excluded columns are removed, useful fields are projected into text, and text is chunked at approximately 384 whitespace-delimited words with 64-word overlap. BGE-M3 embeds batches locally. Every new chunk is written with a fresh generation ID and active: false.

Only after the table finishes does a transaction deactivate the old generation, activate the new one, and update the table’s active-generation pointer. Cleanup follows afterward. A failed refresh leaves the last-known-good generation searchable, while durable checkpoints support recovery work.

Microsoft’s RAG enrichment-phase guidance is useful context for chunking and enrichment choices. In this prototype, optional extraction annotates records; it does not redact PHI, and the current chunk-size setting is an approximation rather than tokenizer-accurate.

Streaming without crossing the boundary

The server streams citations and generated tokens with SSE. The Rust bridge consumes that stream and emits run-scoped Tauri events such as routed decisions, citations, tokens, errors, and completion. Tokens are JSON encoded before they enter SSE; otherwise leading spaces can be stripped and words can fuse in the client.

Each run has an identifier, allowing different conversations to execute concurrently and enabling client-side cancellation. Cancellation currently closes the client transport, but complete interruption of every already-running native model or database stage is not yet proven.

The useful architectural principle is that streaming does not require the browser layer to own network credentials. A trusted local adapter can preserve both authentication and incremental UX.

What “on premises” does - and does not - prove

Local processing narrows data movement. It does not automatically establish secure deployment, clinical validity, or regulatory compliance.

What it does establish is narrow, and worth stating precisely. Because retrieval, embedding, reranking, and generation all execute inside the facility, the architecture removes any reliance on cross-border transfer safeguards and satisfies an in-country processing requirement of the kind described at the top of this article by construction rather than by undertaking. That is an architectural property. It is also the only regulatory question locality answers: an impact assessment is still required wherever the processing counts as high-risk - which regulators generally take an AI-powered clinical system to be - and locality does nothing about an authenticated insider reading beyond clinical need.

Before real PHI is appropriate, the deployment still needs TLS between bridge and server, trusted DocumentDB certificates, OS-backed token storage, strict client capabilities and CSP, sanitized error responses, least-privileged source accounts, protected backups, retention rules, and a stronger audit posture. The current audit path is best effort rather than a tamper-evident ledger.

Model quality also needs representative evaluation. A synthetic benchmark can validate routing and deterministic SQL behavior without validating semantic retrieval or clinical answer quality. The prototype makes no unmeasured throughput claims and no assertion that a relevance threshold is clinically calibrated.

There is also an important scale caveat. Official guidance positions device-focused Foundry Local for hardware-constrained, primarily single-user inference. It is not a continuous-batching, multi-user server runtime. Embedding it in one Rocket process is a conscious prototype tradeoff, protected by bounded admission - not proof of production concurrency.

Do not confuse that SDK with Foundry Local on Azure Local. The latter is the enterprise-scale Arc/Kubernetes offering for managed, multi-node on-premises inference. It is not the device SDK used by this project.

Practical boundary review checklist

Before implementation, ask:

• Can the rendering layer reach the server or databases without the trusted bridge?
• Which process owns tokens, source credentials, authorization, and conversation history?
• Are operational sources clearly separate from derived search copies?
• Does every model and preprocessing stage run locally?
• Can unavailable local inference ever trigger a remote fallback?
• Are structured model outputs validated before execution?
• Does failed ingestion preserve a complete searchable generation?
• Are citations tied to the actual passages supplied to generation?
• Are logs, backups, vectors, and schema samples treated as sensitive data?
• Are capacity limits explicit and tested on target hardware?
• Is every table in every connected source claimed by at least one service line?
• Is the physical mapping from tables and columns to hospital concepts editable without a code change?

If any answer is vague, the architecture diagram is not finished.

Key takeaways

• Start an on-premises RAG design with data-flow and trust invariants, not model selection.
• Keep the Desktop App and Mobile App behind a bridge that owns credentials and transport.
• Make one server-side boundary authoritative for authorization, data access, validation, and admission.
• Separate External Clinical Databases from the Internal DocumentDB Hybrid Store.
• Treat generation, embedding, and reranking as distinct local model workloads.
• Route exact questions to guarded structured execution and narrative questions to cited semantic retrieval.
• Bind agents to hospital functions through a per-source schema binding, so the same roster and rules travel between databases.
• Publish indexed data by generation so refresh failures preserve the last-known-good corpus.
• Describe the prototype honestly: local is a deployment property, not a compliance or scale guarantee.

For implementation-level detail, see System Architecture Design, Data Architecture, Security, Authentication, and Audit, and RAG Patterns. The service-line and binding design is specified in plans/new/.

Learn more on Microsoft Learn

• What is Foundry Local?
• Get started with Foundry Local
• Design and evaluate a RAG solution
• Plan the RAG enrichment phase
• Understand Foundry Local on Azure Local for Arc/Kubernetes enterprise-scale deployments

Continue the series

Next: Part 2 - Getting Started with the Foundry Local Rust SDK

The code behind this series is open source: github.com/kevin-gatimu/onprem-health-rag. It is a synthetic-data research prototype, not a clinical or compliance-ready system — issues and corrections are welcome.