Evaluate Before Production: Evidence-Based Gates for On-Premises RAG

A working demo answers a question. A production decision explains which revision answered which test set, on which hardware, with which models and configuration, at what quality and latency, under which failures and concurrency.

That distinction matters for RAG. A fluent answer can conceal a retrieval miss. A perfect count can exercise deterministic SQL while proving nothing about semantic search. A load script or runbook can exist without proving capacity or recovery.

This synthetic prototype uses a strict evidence rule: plans, targets, code paths, and fixtures are not results. A claim becomes release evidence only when an artifact records the environment, revision, method, and outcome.

This series implements the system in Rust, but the evaluation pattern applies when using the Foundry Local SDK Reference from C#, JavaScript, Python, or Rust; Part 2 links to each language.

Evaluation is itself processing, and it is a common place for the boundary to leak. A judge model called over an API sees the question, the retrieved evidence, and the answer. A benchmark corpus mailed to a collaborator is a transfer of record content. A hosted dashboard exports whatever it charts. Where a country requires processing connected to primary or secondary health care to be carried out inside its borders - as in Kenya, the setting of the research behind this prototype - an evaluation harness inherits the same residency obligation as the system under test. The research handled that by scoring a synthetic corpus with a locally hosted judge, which is also why its results are reported as engineering evidence rather than clinical findings. Part 1 sets out the boundary; this article is about the evidence that has to be produced inside it.

Prototype scope: the repository contains engineering evidence, but it does not establish clinical efficacy, compliance, or production capacity.

Problem statement

A convincing demo, passing unit tests, and a written runbook do not establish that a RAG release is accurate, bounded, or recoverable. Each artifact can exercise a different slice, and none may record the revision, models, hardware, configuration, or failures behind an observed result.

This system has distinct routing, structured execution, retrieval, generation, refusal, and operational paths. One aggregate score can hide a broken stage; a deterministic SQL benchmark cannot validate semantic retrieval, and a brief load test cannot establish sustained capacity or restart behavior. Synthetic success also cannot support claims of clinical suitability or compliance.

Evaluation must create stage-specific datasets and oracles, measure composed behavior on named environments, inject realistic faults, and retain artifacts that distinguish plans from executed evidence. Release gates must remain closed when quality, capacity, security, or recovery evidence is missing. A completed data protection impact assessment belongs in that gate as well: high-risk processing generally requires one before deployment, and an AI system reading health records is treated as high-risk, so it is a precondition of deployment rather than a document written afterwards.

Success criteria

• Versioned datasets separately cover routing, SQL, retrieval, grounding, refusal, and authorization.
• Reports bind outcomes to source revision, models, execution providers, hardware, and configuration.
• Metrics expose per-stage and per-slice failures rather than only a global average.
• Capacity runs record concurrency, latency, errors, saturation, and sustained resource use.
• Fault drills demonstrate bounded failure and measured recovery for required dependencies.
• Release gates explicitly fail when required evidence is absent, stale, or below threshold.

Evaluate the system as several coupled systems

The application has at least five quality surfaces:

1. Routing: Did the question choose conversation, structured execution, semantic retrieval, or a bounded agent path?
2. Structured execution: Did deterministic or model-planned SQL return the exact expected rows under safety controls?
3. Retrieval: Did the relevant records appear, and how highly were they ranked?
4. Generation and refusal: Were claims supported by cited evidence, and did the system refuse when evidence was absent?
5. Operations: Did the system remain bounded and recoverable under concurrency, dependency failure, restart, and prolonged use?

A single aggregate score hides which layer failed. Evaluation should isolate each layer and then test the composed experience.

 

 

 

The test model keeps External Clinical Databases separate from the Internal DocumentDB Hybrid Store. Structured live-source queries and semantic retrieval therefore need different fixtures and oracles.

What the repository proves today

Deterministic fixture runner

eval/run.mjs is a local Node evaluator. It authenticates to the on-premises server and supports router, retrieval, and combined full suites, plus a five-case smoke mode. It can compare retrieval presets:

• full: hybrid search with reranking;
• naive: vector-only without reranking;
• no-rerank: hybrid without reranking; and
• fast: vector-only with reranking.

The current committed datasets are intentionally small and synthetic:

• 13 router cases across conversational, structured, and semantic decisions;
• 7 retrieval cases focused on medication and allergen records.

The router suite compares route, intent, and optional tier, then reports accuracy against a threshold. The retrieval suite finds an expected record-ID fragment in the returned passages and reports hit rate at six and mean reciprocal rank (MRR).

For N queries, reciprocal rank is:

 

 

 

where a miss contributes zero. Hit rate at k is the fraction of queries for which at least one expected item appears in the top k.

The runner writes timestamped JSON reports under ignored eval/reports/. Its --validate-only mode parses fixture files without contacting a server. Fixtures must never contain production PHI.

Continuous integration

The Windows CI workflow currently:

1. checks and tests the Rocket server;
2. installs Desktop App/Mobile App frontend dependencies;
3. runs TypeScript checking and a Vite production build; and
4. validates evaluator fixture syntax.

That is valuable regression protection, but its boundary must be stated precisely. CI does not start DocumentDB, Foundry Local, or source databases; it does not run live router/retrieval suites; it does not build the Tauri Rust bridge or Android package; and it does not prove hardware-dependent inference.

Dated manual evidence

A dated synthetic Auto benchmark records 10/10 strict passes for the A-series, including the A10 follow-up after a grounding defect was fixed. It reports 11 measured requests, no timeout, and latency values for that environment. Additional S-series joins have mixed run status, and a semantic S01 case is recorded as a failure caused by an upstream retrieval/enrichment gap.

The essential caveat is that all final A-series passes used deterministic SQL. That benchmark is evidence for the tested deterministic path. It is not evidence for the local SQL planner, semantic RAG quality, citation grounding, broad refusal behavior, or multi-user capacity.

This is how an evidence record should behave: preserve failures and “not run” states instead of rewriting history into a success narrative.

Build a quality matrix before adding a judge

A useful evaluation set labels the expected execution mode as well as the answer.

Exact SQL equality means result equality

String-equal SQL is usually the wrong oracle. Two safe queries can be semantically equivalent while differing in aliases, ordering syntax, or dialect. For structured questions, execute against a versioned synthetic snapshot and compare:

• column meaning;
• row values;
• row ordering when ordering is part of the question;
• numeric types and precision; and
• empty-result behavior.

Also test the guard independently: multiple statements, unlinked tables, writes, locks, dangerous functions, excessive limits, over-cost plans, and timeout paths must be rejected. Deterministic templates and local model proposals should pass through the same safety assertions.

Retrieval needs positive, hard-negative, and unanswerable cases

A seven-case set can catch regressions, but it cannot characterize a clinical retrieval system. Expand along realistic dimensions using synthetic or properly governed de-identified data:

• abbreviations, misspellings, synonyms, codes, and negation;
• common versus rare concepts;
• short identifiers and long narrative questions;
• multiple records for one patient;
• near-duplicate rows and conflicting dates;
• relevant lexical match with wrong semantics;
• relevant semantics with little lexical overlap; and
• questions unsupported by the corpus.

Track per-slice metrics, not only a global mean. A system can improve average MRR while regressing medication negation or patient-identifier retrieval.

No-answer is a first-class outcome

A score gate that refuses weak evidence is valuable only if both sides are measured.

• Refusal precision: among refusals, how many were truly unanswerable?
• Refusal recall: among unanswerable questions, how many were refused?

A threshold tuned only for recall may answer unsupported questions. A threshold tuned only for precision may refuse useful queries. Plot the tradeoff and choose policy based on harm, not aesthetics.

Citation groundedness requires claim-level review

A citation beside a paragraph does not prove that the citation supports every claim. Split an answer into atomic claims and label whether each claim is entailed by one or more cited passages. Track:

• supported claims divided by all verifiable claims;
• citations that actually support their attached claims;
• claims missing citations;
• citations outside the provided passage set; and
• whether the answer omits important evidence.

The prototype has an optional local post-stream verifier, but runtime verification is not a validated judge. It runs after the user has already received the answer, can fail open as “skipped,” and has no complete committed quality slice. Treat verifier output as a feature under evaluation, not ground truth.

Local judged evaluation: useful, but not implemented yet

There is no committed full local RAGAs harness or local LLM judge in this repository. That absence should remain explicit.

A future judged suite must keep prompts, answers, and evidence on premises. It should pin the judge model and version, use deterministic decoding where practical, archive the rubric, calibrate against human labels, and report disagreement. Metrics such as groundedness, completeness, utilization, and relevance can complement deterministic checks, but they should not replace record-ID or exact-result oracles.

Changing the judge model, execution provider, prompt, parser, or rubric invalidates direct score comparisons. Record each input.

Planned: prove the system is hospital-agnostic

Every benchmark above runs against one synthetic schema. That is a weakness the next design iteration makes explicit. Its central claim is that the agent roster is a fixed hospital ontology bound to any relational schema at catalog time (plan 01), and that deterministic SQL is compiled from a schema-independent QuerySpec rather than templates that name the development tables. A claim like that cannot be evaluated on the schema it was developed against.

The alt-hospital schema as a portability test

Plan 08 adds a second synthetic seed with a deliberately different naming convention and structure for the same domain: PascalCase PatientMaster with an MRN instead of patient_no; Visit with a department code foreign key; IPD_Admission with no length-of-stay column, so it must be computed; Booking whose status vocabulary says DNA rather than no_show; LabRes with a character abnormal flag instead of a boolean; Invoice and Receipt with the currency in the column name; roughly 25 tables in total, served through the second SQL dialect. Some service lines are legitimately unusable on it - there are no facilities tables - and the UI must hide them rather than show empty tabs.

The point is not that the alternative schema is realistic. It is that the same code, with no overrides, must bind it, route against it, and compile SQL for it. Overfitting to the development seed shows up as a gap between the two columns of the report.

New suites and fixtures

The evaluator would grow from router | retrieval | full to add --suite sql, --suite agents, and --suite flows, each runnable with --schema dev or --schema alt:

Binder expectations for both schemas and IR golden files live beside the Rust tests. The SSE consumer in the runner would capture routed, provenance, sql, rows, and suggestions so that reports can include per-rung latency percentiles and how often the model classifier had to be invoked at all.

Gates for the service-line design

These are targets the plans set, not results anyone has recorded. They are listed so that a future report can be judged against them:

• binder binds ≥ 22 of 25 alt-schema tables to the correct concept without overrides (and ≥ 62 of 64 on the dev seed);
• golden SQL suite ≥ 55/60 deterministic on the dev seed and ≥ 50/60 on the alt schema;
• router accuracy ≥ 0.95 on the extended fixture, with the A-series still 10/10;
• the model classifier is reached on ≤ 25 % of routed questions (deterministic-first is the latency strategy, so this is a quality and performance gate);
• deterministic answers p50 < 300 ms on the development host;
• conversation flows resolve references with zero model calls where the fixture marks them deterministic;
• a CI grep finds no development-seed table name in the ontology, IR, router, or agents modules outside tests and fixtures.

The last gate is unusual and worth copying: it is a structural test of the portability claim that runs without a database.

Feature-flag shadow rollout as an evaluation technique

The plans stage the change behind flags - the IR compiler and router v3 default off - and run each new component in shadow: the old path answers the user while the new path’s result is logged beside it and compared. The flag flips only when the shadow comparison meets its gate on real traffic against synthetic data, and the legacy path is deleted only after two green evaluation cycles. This is evaluation on the same synthetic workloads the fixtures do not cover, at no user-visible risk, and it produces the dated report a release gate demands. Treat it as a stage in the evidence ladder, not a deployment convenience.

Performance evidence needs a hardware identity

perf/load.mjs can authenticate, issue concurrent /chat requests, and report status counts plus p50/p95/p99 end-to-end latency. It is a useful probe, not a capacity certification. It does not measure process RSS, queue depth, GPU/NPU utilization, ingestion throughput, or recovery.

For each tested hardware profile, record at least:

revision: <commit>
host:
  os: <version>
  cpu: <model and cores>
  memory_gib: <value>
  accelerator: <model and driver>
models:
  chat: <exact variant>
  embedding: bge-m3
  reranker: bge-reranker-v2-m3
execution_providers: [<registered and used>]
corpus:
  rows: <count>
  chunks: <count>
  vector_index: <kind and parameters>
configuration:
  admission: <limits>
  retrieval: <top-k, rerank depth, score gate>
workload:
  request_mix: <structured/semantic/conversation>
  concurrency: <value>
results:
  status_counts: <map>
  ttft_ms: <p50/p95/p99>
  total_ms: <p50/p95/p99>
  peak_rss_mib: <value>
  quality_report: <artifact>

The repository has a runbook for 10/25/50-client ladders and an eight-hour soak, but no committed report proves those runs. It also lacks evidence for safe maximum concurrent generations/searches/ingestions, bounded memory after load, multi-million-row ingestion, or production vector-index tuning.

Foundry Local adds another boundary. The in-process device SDK is officially optimized for single-user device scenarios, not central multi-user serving. Admission can reject excess work, but it does not provide continuous batching or efficient shared-GPU scheduling. Before central production, benchmark a server runtime or evaluate the distinct Foundry Local on Azure Local Arc/Kubernetes option.

Fault injection turns design into evidence

Test failures at controlled boundaries with synthetic data:

• stop DocumentDB during retrieval, persistence, and ingestion cutover;
• make required indexes unavailable;
• unload or corrupt a cached local model;
• remove an execution provider or force CPU fallback;
• delay an external SQL source beyond timeout;
• return an over-cost query plan;
• interrupt ingestion between pages and during generation cutover;
• exhaust generation, retrieval, and ingestion permits;
• disconnect a Desktop App or Mobile App during SSE;
• restart the server during active work;
• rotate JWT and credential keys in a rehearsal environment; and
• expire or replace TLS certificates.

For each trial, define expected status, user behavior, persisted state, retry semantics, audit event, and recovery objective. “The process did not crash” is insufficient. Ingestion must preserve the previous active generation after a failed refresh and resume without publishing partial data.

A release gate should be executable and unforgiving

A release candidate should fail when evidence is missing. A practical gate for this prototype is:

Build and deterministic checks

• ☐ Server check and unit tests pass.
• ☐ Tauri Rust bridge check passes.
• ☐ TypeScript check and production build pass.
• ☐ Router and retrieval fixtures parse.
• ☐ No production PHI appears in fixtures, reports, logs, or CI artifacts.

Live quality

• ☐ Live router suite meets a versioned threshold on a representative expanded dataset.
• ☐ Retrieval hit rate, MRR, and per-slice thresholds pass.
• ☐ Structured cases match exact result oracles across supported SQL dialects.
• ☐ Follow-up/entity-carryover cases pass.
• ☐ No-answer precision and recall pass policy thresholds.
• ☐ Citation and claim groundedness pass human-calibrated thresholds.
• ☐ Local judged results, if required, include model/rubric versions and human agreement.

Security and authorization

• ☐ Unauthorized disclosure tests return zero disclosures.
• ☐ TLS, trusted certificates, secure token storage, CSP, sanitized 5xx responses, and production Android network policy are complete.
• ☐ Audit durability and record-level authorization requirements are complete and tested.

Performance and recovery

• ☐ 10/25/50 concurrency ladder is recorded on named hardware.
• ☐ Admission behavior and 429 rates are acceptable.
• ☐ Peak RSS, task growth, queueing, TTFT, and p95/p99 latency remain within approved bounds.
• ☐ Eight-hour soak completes without unbounded growth.
• ☐ Required fault-injection and restart/recovery drills pass.
• ☐ Backup restoration is demonstrated.
• ☐ Desktop App and Mobile App release paths are validated; Android remains a gate if included in scope.

Artifact completeness

• ☐ Every report contains commit, date, host, models, execution providers, corpus, configuration, workload, and result.
• ☐ Failures and not-run cases are retained.
• ☐ /metrics/summary, readiness, and relevant local logs are captured without prompt or record text.
• ☐ Exceptions have owners, expiry dates, and explicit approval.

Anti-patterns to avoid

“The demo passed, so RAG passed”

Determine which path ran. Deterministic SQL success does not validate retrieval or generation.

“The evaluator exists, so the threshold is achieved”

A script and a runbook are capabilities. Only a dated report is evidence.

“One average score is enough”

Aggregate metrics can hide dangerous slices. Report by intent, source, concept type, answerability, and hardware profile.

“The LLM judge is objective”

Judges have model, prompt, and parser variance. Calibrate them against human labels and retain deterministic oracles.

“Not run means no defect”

Not run means unknown. A required unknown fails the gate.

“Load testing on any machine proves capacity”

Local AI performance depends on exact model variants, drivers, execution providers, memory, corpus, and concurrency. Hardware identity is part of the result. This cuts both ways in a multi-site rollout. Because one build targets CPU, integrated and discrete GPU, and NPU hosts alike, a result measured on one accelerator says little about another - and a figure obtained on an integrated GPU is not evidence that accelerator provision is unnecessary, only that the envelope tested was sufficient for the workload tested.

Tradeoffs

Rigorous evaluation costs time and hardware access. Exact-result fixtures require curated snapshots. Human claim labeling is expensive. Fault injection interrupts shared environments. Soak tests slow release cadence.

The alternative is hidden uncertainty. The practical answer is a layered cadence: fast deterministic checks on every change, live synthetic smoke tests on integration builds, broader quality suites on release candidates, and hardware-specific capacity and recovery campaigns before deployment. Preserve a small stable regression set while growing representative challenge sets separately.

Prototype status

Implemented: local router/retrieval fixture runner, hit rate and MRR, configurable retrieval comparisons, timestamped reports, fixture-only CI validation, server tests, frontend checks/build, request summaries, process metrics, a basic load generator, and dated synthetic manual benchmark records.

Not implemented or not proven: a full local RAGAs/judged suite, comprehensive no-answer metrics, release-grade citation groundedness thresholds, an automated structured exact-result suite in eval/run.mjs, live evaluation in CI, bridge/Android CI coverage, completed 10/25/50 capacity reports, eight-hour soak, broad fault-injection evidence, larger-than-memory ingestion proof, and production hardware profiles. The alt-hospital schema, the sql/agents/flows suites, and the portability gates above are planned in plans/new and have no recorded result.

No production claim is justified without the missing artifact.

Key takeaways

1. Evaluate routing, structured execution, retrieval, generation, refusal, and operations separately before evaluating them together.
2. Exact result oracles and record IDs are stronger than prose similarity where they are available.
3. Measure both retrieval success and correct refusal.
4. Treat local model judges as calibrated instruments, not truth.
5. Attach hardware, model, execution-provider, corpus, and configuration identity to every performance result.
6. Fault injection and soak results are evidence; runbooks are only instructions.
7. If a required test was not run, the release gate fails.

Source deep dives

• Evaluation, production validation, and release
• Verified platform notes
• Operations, performance, and observability
• RAG patterns
• Routing, agents, and structured query
• Evaluator and evidence: eval/run.mjs, eval/data/router.jsonl, eval/data/retrieval.jsonl, eval/AUTO_BENCHMARK.md, and eval/QUESTIONS.md
• Planned suites and gates: Evaluation, rollout and documentation
• Load probe: perf/load.mjs

Learn more on Microsoft Learn

• Design and develop a RAG solution
• What is Foundry Local?
• Get started with Foundry Local
• Foundry Local SDK Reference - Rust on Windows
• Foundry Local on Azure Local overview - the enterprise Arc/Kubernetes option, not the in-process device SDK used by this prototype.

 

Series navigation: Previous: Part 9 - Hardening On-Premises RAG · End of series

 

The code behind this series is open source: github.com/kevin-gatimu/onprem-health-rag. It is a synthetic-data research prototype, not a clinical or compliance-ready system - issues and corrections are welcome.