Industrial maintenance often requires engineers to locate relevant information quickly across lengthy technical manuals, sometimes in environments where cloud access is limited or unsuitable. This project explores a fully local Retrieval-Augmented Generation (RAG) assistant built with Microsoft Foundry Local, and how its architecture evolved from multi-agent orchestration to deterministic-first Hybrid control.
Building the Local RAG System
The system was designed as a fully local RAG pipeline, with both retrieval and model inference running on the same machine. The knowledge base came from two equipment manuals covering pump and motor operation, maintenance, and troubleshooting. Rather than treating every page as plain text, I kept different types of information in forms that matched their structure, including paragraphs, Q&A entries, tables, procedures, and fault–cause–remedy troubleshooting records. The final corpus contained 165 retrievable records.
Each record was embedded locally with all-MiniLM-L6-v2 and stored in ChromaDB. At query time, the system retrieved the most relevant manual records and passed the selected evidence to Phi-4-mini, a small language model (SLM) running through Microsoft Foundry Local, for grounded response generation. Once the required models and resources were installed, the full retrieval and inference workflow could run on the local machine without depending on cloud inference.
I began with a simple Single-Agent RAG baseline. For each query, the system retrieved relevant manual evidence and used a single Phi-4-mini call to generate a grounded response. That gave me a clear reference point before I started adding separate model-driven stages for query understanding, retrieval, and reasoning.
The Experiment That Changed the Architecture
The next version split the workflow into three roles: Query Understanding, Retrieval, and Reasoning. All three reused the same local Phi-4-mini model through Microsoft Foundry Local, but with different prompts and responsibilities. I expected this separation to give the system more control over how queries were interpreted and how evidence moved through the pipeline. In practice, the extra stages added a noticeable cost, so I profiled the pipeline to see where the time was going.
The stage timings showed where the delay was coming from:
- Query Understanding: 15.469 s
- Evidence Selection: 11.557 s
- Reasoning: 8.251 s
- Vector search: 0.002 s
Vector search was only a tiny part of the total time. Most of the latency came from the stages that repeatedly called the local model. The bottleneck was repeated model inference, not vector search. I therefore started looking for decisions that could be handled reliably without another SLM call.
The end-to-end numbers matched the profile. The Single-Agent baseline averaged 23.1 seconds per query, compared with 33.0 seconds for the Initial Multi-Agent system, without a consistent improvement in answer quality. That shifted the focus of the project from making the pipeline more agentic to deciding where model reasoning was actually useful, and where simpler deterministic control would be enough.
A Deterministic-First Hybrid Design
The Hybrid redesign kept the same local retrieval stack and Phi-4-mini setup, but changed when the model was asked to make decisions. Clear, bounded routing decisions were handled deterministically first. Depending on the query, the system could proceed to retrieval, request clarification when required information was missing, or stop requests that depended on information outside the manuals. Phi-4-mini was used for query analysis only when these rules could not make a reliable decision.
In the 12-case final evaluation, Phi-4-mini was needed for query analysis in only three cases. The remaining routing decisions were handled deterministically.
The Hybrid pipeline also handled clearly multi-part requests differently. Instead of relying on a single retrieval query, the system could identify separate information needs and search them alongside the original request. For example, a two-part pump question could be handled like this:
Multi-part Query
│
▼
"What should I check before starting the
B114N pump, and what should I inspect
if it fails to prime?"
│
▼
Deterministic Decomposition
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
Original Query Pre-start Checks Failure-to-prime
Search Search Search
│ │ │
└──────────────────┼──────────────────┘
▼
Combine Ranked Results
│
▼
Retain Component Provenance
│
▼
Candidate EvidenceThe decomposition was deliberately conservative. It handled clear structures such as multiple questions, comparisons, or lists of requested facts, and did not split a query simply because it contained a comma or the word “and.”
Once candidate evidence had been retrieved, clear matches could be selected without another model call, while Phi-4-mini was used when the candidates required additional judgement.
Candidate Evidence
│
▼
Can deterministic rules
select the evidence?
/ \
Yes No
│ │
▼ ▼
Deterministic Selection Phi-4-mini
Evidence Selection
│ │
└───────┬────────┘
▼
Selected Evidence
│
▼
Evidence Guard
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Sufficient More evidence Insufficient
coverage already in the evidence
│ candidate pool │
▼ │ ▼
Response ▼ Stop
Generation Retain additional
evidence
│
▼
Re-check coverage
│
└──────► Evidence GuardThe Evidence Guard checked whether the selected records covered the information requested by the user. Supporting evidence already present in the retrieved candidate set could be added before checking coverage again. When the available evidence was still insufficient, the pipeline stopped instead of asking the model to fill the gap. The guard never triggered a new retrieval or SLM call.
The final response stage used the model selectively as well. Some bounded cases, including clarification, out-of-scope requests, insufficient evidence, and selected structured answers, could be handled without another model call. When synthesis was still needed, Phi-4-mini generated a grounded response from the selected evidence, while citation formatting remained deterministic.
What Changed?
I evaluated the three architectures on the same frozen 12-case test set, using answer quality and end-to-end latency as the main comparison points. The Final Hybrid system produced the lowest average latency while maintaining answer quality at roughly the same level as the Single-Agent baseline.
| Architecture | Quality score | Mean latency |
|---|---|---|
| Single-Agent RAG | 7.00 / 10 | 22.709 s |
| Initial Multi-Agent | 6.50 / 10 | 33.454 s |
| Final Hybrid | 7.08 / 10 | 16.905 s |
The Initial Multi-Agent design was the slowest of the three and did not produce a consistent quality improvement over the simpler baseline. The Hybrid redesign reversed that trend: average latency fell to 16.9 seconds while the overall quality score remained comparable to the baseline. For this prototype, adding more model-driven stages was therefore less useful than being selective about where model reasoning was actually needed.
The call counts show the same change. Across the 12 final cases, the Hybrid system made 12 stage-level SLM calls: three during query analysis, four during evidence selection, and five during reasoning. A pipeline that invoked the SLM at all three stages for every query would require 36 stage-level calls over the same 12 cases.
This was only a small 12-case evaluation, so I would not treat it as a final measure of system performance. What it did show was that cutting unnecessary SLM calls made the local pipeline faster without hurting the quality score in this test.
From Pipeline to Local Application
I connected the Final Hybrid backend to a lightweight PySide6 desktop interface. The application lets a user enter a maintenance question, view the generated answer, and inspect the supporting manual sources.
What I Learned
The biggest lesson from the project was that adding more model-driven stages did not automatically make the system better. The Initial Multi-Agent design gave each stage a clearer role, but the extra SLM calls increased latency without consistently improving answer quality.
I also found that good retrieval alone was not enough. The system still needed to decide whether the retrieved evidence actually covered the user’s request and whether another model call was necessary. This is where deterministic routing, evidence checks, and model reasoning worked well together.
By the end of the project, the question was no longer how many agents to add, but where an SLM call was actually useful. In this system, the best result came from combining simple deterministic control with model reasoning only where it was needed.
What Comes Next
The next step would be to test the system on a larger and more varied set of maintenance questions, including more unseen cases and feedback from engineers. I would also like to compare different local models and hardware configurations to see how response quality, latency, and resource use change in practice.
Beyond that, the same approach could be applied to a larger maintenance knowledge base or adapted to other equipment domains. One question I would like to explore further is whether the deterministic-first design still works as well when the corpus, query types, and deployment conditions become more varied.
Resources
