Getting Qwen3-8B running through Foundry Local was only the beginning. Once it became part of a diagnostic-ranking application, the questions were less about launching a model and more about what to do when it returned an incomplete ranking, displaced a correct result, or never saw the right candidate in the first place.
I explored those questions using MDDial, a template-constructed differential-diagnosis dialogue benchmark. The task was deliberately limited to ranking 12 known disease classes from fixed dialogues, not diagnosing patients in an open clinical setting. Model inference ran on a Windows laptop with an NVIDIA RTX 5060 Laptop GPU; the rest of the workflow combined deterministic processing with selective language-model calls.
1. Constrain First, Rerank Second
Qwen3-8B first extracts a structured symptom state from the dialogue, recording each finding's polarity and source reference. A deterministic normalization stage maps the findings to the benchmark symptom vocabulary. A pre-existing disease–symptom graph then scores the 12 diseases and produces an initial ranking and a smaller candidate set.
The routing rule is simple: a unique top graph score bypasses reranking; a tie at the top triggers another local Qwen3-8B call. The reranker can reorder the supplied candidates, but it cannot introduce a disease that was excluded earlier. The evaluated candidate policy uses Top-5 seeding, boundary-tie expansion, and a cap of 10 diseases.
Figure 1. The local LLM extracts findings and, when triggered, reranks graph-generated candidates. Deterministic components control normalization, candidate construction, routing, and acceptance of the output.
That acceptance step is structural, not clinical. The interface checks candidate membership, uniqueness, and completeness. It can complete a usable partial ranking, attempt recovery or retry after a malformed response, and apply a predefined fallback when no acceptable model ordering is obtained.
2. Did the Extra Structure Actually Help?
I tested three architectures on the same cached symptom representations from 264 MDDial dialogues, balanced across the 12 classes. Graph-only ranked all diseases deterministically. Direct-Qwen-12 gave the same model the full disease list, without graph ranking cues or candidate-specific graph evidence. Selective Hybrid used the graph candidate set and invoked Qwen only under the tie-triggered rule.
| Approach | Hit@1 | MRR |
|---|---|---|
| Graph-only | 43.2% | 0.591 |
| Direct-Qwen-12 | 59.5% | 0.700 |
| Selective Hybrid | 64.8% | 0.763 |
Direct-Qwen-12 was a strong baseline: the local model did considerably better than graph scoring alone. The Hybrid had the highest observed scores, although its 5.3-percentage-point Hit@1 advantage over Direct-Qwen-12 was not statistically conclusive. Its paired MRR advantage was +0.063, with a confidence interval above zero.
The graph also imposed a measurable candidate constraint. Final candidate coverage was 98.1%, with a mean candidate-set size of 7.83 out of 12 diseases. But looking only at the aggregate rankings would have hidden how those results arose.
3. Where the System Failed
The execution records showed several distinct outcomes. Among 211 cases sent to the Hybrid reranker, 65 changed from an incorrect Graph-only Top-1 result to a correct one; eight went from correct to incorrect. Another 54 remained correct and 84 remained incorrect. The routing rule bypassed reranking in 53 cases, 52 of which already had the correct Graph Top-1 result.
Five reference diagnoses were excluded before reranking could help. Three were in the expanded candidate set but removed by the ten-disease cap; two were missing before capping. These are candidate-construction failures, not reranker failures.
Figure 2. Candidate retention and routing are measured across all 264 dialogues; reranking transitions and interface outcomes use the 211 invoked cases. Counts describe different stages and should not be added across panels.
The interface created a third kind of observable outcome. Of the 211 reranker invocations, 140 ended with strict-valid output, 69 with partial recovery, and two with graph fallback. The logs distinguish those interface events from whether the accepted disease ranking was correct.
I also audited the links between graph evidence and the source dialogue. All 9,096 score-relevant candidate–symptom evidence relations in the primary evaluation satisfied the implemented source-link path. This is provenance of the recorded computation, not an explanation of Qwen's hidden reasoning or proof that the clinical interpretation is correct.
4. Choosing a Local Model Is a Systems Trade-off
To compare backends without changing the task, I ran Qwen3-8B, OLMo-3-7B-Instruct, and Phi-4-mini on 180 identical frozen reranking packets. The structured findings, candidate sets, prompts, and interface procedure were fixed; only the model backend changed.
Qwen3-8B performed best on ranking quality, reaching 61.7% Hit@1 and 0.744 MRR. OLMo and Phi each reached 52.8% Hit@1, with MRR values of 0.668 and 0.665 respectively.
Phi-4-mini surprised me. Although it ranked less accurately than Qwen, it returned a strict-valid ranking in all 180 cases and had a recorded median latency of 1.13 seconds, compared with 17.82 seconds for Qwen. These sequential runs were not a controlled model-speed benchmark, but the observed difference mattered when choosing a backend for a strict, latency-sensitive interface.
I retained Qwen for reranking because ranking quality was the priority in this prototype. The same architecture could favour a different local model if latency or output compliance mattered more.
5. What Local Deployment Does — and Does Not — Show
Foundry Local kept the evaluated inference path on the laptop and allowed different models to be tested within the same software workflow. The project also has frozen evaluation configurations, scripts, targeted deterministic tests, and deployment documentation, so the evaluated implementation can be inspected beyond the final result table.
The scope of the evidence remains narrow. MDDial dialogues are constructed from templates rather than collected from real clinical encounters, and the main 264-case evaluation comes from the benchmark's published training partition. The results do not establish performance on external patients, unseen diseases, or real clinical conversations. This is a working, closed-set ranking prototype—not an autonomous medical agent or a clinically validated diagnostic tool.
Locality is a deployment property. Clinical validity requires separate evidence.
6. What Comes Next
This project showed that a useful local model does not have to control the whole decision path. Direct-Qwen-12 was capable on its own, while the staged Hybrid made candidate exclusions, ranking changes, and interface recovery visible as separate events.
The next meaningful test would use a genuinely held-out or external dialogue dataset, with independent assessment of upstream symptom interpretation. That would show whether the same stage boundaries remain useful when the conversations and disease distribution are less controlled.
