Foundry Local with Rust: From Catalog Discovery to In-Process Streaming

The shortest path to local inference is not to begin with a web server, an agent framework, or a full RAG pipeline. Begin with a spike that proves five things on the target machine: the SDK initializes, the catalog is visible, an appropriate model alias resolves, model artifacts can be cached and loaded, and tokens stream from the local runtime.

That narrow test is especially valuable in Rust, where application logic, native runtime loading, async streams, execution providers, and model lifecycle can otherwise fail at the same time. Once the spike works, it can be embedded behind a deliberate application boundary.

This article uses the current device-focused Foundry Local SDK. The project that motivated it embeds foundry-local-sdk v1 directly in a Rocket process and enables the winml feature on Windows. Calls go to the native core in process: there is no cloud inference request and no separate application-managed Foundry HTTP service.

That absence of a cloud call is the requirement, not a property of the demo. Some countries’ data-protection laws require processing connected to primary or secondary health care to be carried out through a server and data centre located inside the country - Kenya, the setting of the research behind this series, is one example. A prompt is processing. An OpenAI-compatible endpoint that happens to accept the same request shape moves patient content to whichever jurisdiction hosts it, and the cross-border transfer rules then apply in full. Proving the native lifecycle in process - resolve, download, load, stream, unload - is how the pipeline stays inside a boundary that a residency obligation can be satisfied against. Part 1 sets out that boundary and the constraint behind it.

There is a deployment reason to care about this beyond tidiness. The target for this project is a hospital server that already runs an EMR, on modest and sometimes unreliable power, with no option to add a dedicated accelerator per site. What makes that tractable is that Foundry Local does not bind the application to one accelerator: it resolves a model alias to a device-specific variant at load time, so the same build runs on a CPU-only box, an integrated GPU, a discrete NVIDIA card, or an NPU, with no manual quantisation or model-format conversion. The spike below is how you find out which of those you actually have.

 

The project integration implements in-process initialization, catalog discovery, execution-provider registration, model download/load/unload, streaming generation, role-based model selection, busy guards, and GPU-residency controls. Its default generative alias is qwen3-8b. Hardware compatibility, sustained concurrency, cancellation through every native stage, and production capacity remain target-machine validation work. All test data is synthetic; this is not clinical advice or a compliance claim.

Problem statement

Many “local AI” prototypes still depend on a cloud or OpenAI-compatible endpoint, or launch a separate model process whose lifecycle is outside the application. Those shortcuts can obscure where prompts travel and postpone the hardware, packaging, and concurrency problems that determine whether local inference actually works on the target machine. Where a data-residency law covers the records in question, those shortcuts are not merely unclear about where prompts travel; they are the non-compliant path.

For a Rust application, the failure surface spans native SDK initialization, catalog resolution, model acquisition, execution-provider selection, loading, streaming, and shutdown. Adding HTTP orchestration before proving those steps makes runtime failures harder to isolate. Treating the generative runtime as the embedding service also couples two different model contracts.

The required pattern must prove and then embed the complete Foundry Local lifecycle in process, expose bounded streaming to async Rust, record hardware-specific resolution, and keep embedding and reranking under fastembed rather than the chat model.

Success criteria

• The Rust process initializes the native SDK and enumerates the local catalog.
• A catalog alias resolves, downloads, loads, streams tokens, and unloads on target hardware.
• Execution-provider and resolved-model details are observable for repeatable validation.
• Lifecycle operations are guarded against conflicting concurrent work.
• Embedding and reranking remain separate from Foundry Local generation.

Choose the correct Foundry Local

The name can refer to two different deployment shapes.

The SDK in this article is the device-oriented runtime described in the Foundry Local overview. A Rust application loads a platform-native core library and calls it directly. The SDK exposes model discovery, acquisition, execution-provider management, lifecycle operations, and chat APIs.

Foundry Local on Azure Local is different. It is an enterprise-scale, Arc-enabled Kubernetes deployment with operators, model resources, secured endpoints, and multi-node serving. It is not the device SDK embedded in this prototype.

There is also an optional local HTTP server in the device product, but the current Rust architecture does not use it. The native path is simpler:

 

 

Use the language that fits your application

This series uses Rust because its reference project is a Rust Rocket/Tauri system, but the architecture and patterns are language-independent. Foundry Local SDK supports C#, JavaScript, Python, and Rust; readers can reproduce the manager, catalog, model download, load, chat, stream, and unload lifecycle in the language that fits their application while preserving the same privacy, validation, storage, and retrieval boundaries. API syntax differs by language.

• Foundry Local SDK for Rust
• Foundry Local SDK for C#
• Foundry Local SDK for JavaScript
• Foundry Local SDK for Python

Prerequisites and dependency shape

Start with the official Foundry Local getting-started guide, then verify the Foundry Local SDK Reference for Rust because the API has changed from the legacy CLI-dependent SDK.

[dependencies]
  anyhow = "1"
  foundry-local-sdk = "1"
  tokio = { version = "1", features = ["full"] }
  tokio-stream = "0.1"

  [target.'cfg(windows)'.dependencies]
  foundry-local-sdk = { version = "1", features = ["winml"] }

The target-specific entry makes the intent explicit: use the WinML-backed package on Windows while retaining the cross-platform package definition elsewhere. A real workspace should use one dependency arrangement appropriate to its supported targets and lock the resolved versions.

Model and execution-provider downloads can require network access during setup. Local inference does not imply that every installation is born air-gapped. For disconnected operation, pre-stage the required runtime components, execution providers, and model artifacts under an approved software-supply process.

First inspect the catalog

Do not copy a model alias from a blog and assume it exists forever or has a suitable variant for the machine. Catalog contents and optimized variants can change. List the catalog first:

use foundry_local_sdk::{FoundryLocalConfig, FoundryLocalManager};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let manager = FoundryLocalManager::create(
        FoundryLocalConfig::new("catalog-spike"),
    )?;

    let models = manager.catalog().get_models().await?;
    for model in models {
        println!(
            "alias={} id={} context={:?} capabilities={:?}",
            model.alias(),
            model.id(),
            model.context_length(),
            model.capabilities(),
        );
    }

    Ok(())
}

Run this on the actual target hardware. Select a small chat-capable alias shown by that output. The official examples have used aliases such as qwen2.5-0.5b; the larger project currently defaults to qwen3-8b, but a smoke test should favor the smallest suitable catalog model rather than inherit an application production default.

An alias is not necessarily a concrete artifact. The catalog can resolve it to a hardware-specific variant. For repeatable deployments, record the alias, resolved variant ID, runtime version, execution provider, driver version, and machine profile used during validation.

A minimal streaming Rust spike

The following example performs the full lifecycle without adding an HTTP layer. Replace MODEL_ALIAS with an available small chat alias printed by the catalog listing.

use std::io::{self, Write};

use foundry_local_sdk::{
    ChatCompletionRequestMessage,
    ChatCompletionRequestSystemMessage,
    ChatCompletionRequestUserMessage,
    FoundryLocalConfig,
    FoundryLocalManager,
};
use tokio_stream::StreamExt;

const MODEL_ALIAS: &str = "qwen2.5-0.5b";

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let manager = FoundryLocalManager::create(
        FoundryLocalConfig::new("rust-foundry-local-spike"),
    )?;

    println!("Available catalog models:");
    for entry in manager.catalog().get_models().await? {
        println!("  {} ({})", entry.alias(), entry.id());
    }

    manager.download_and_register_eps(None).await?;

    let model = manager.catalog().get_model(MODEL_ALIAS).await?;
    println!("Selected {} -> {}", model.alias(), model.id());

    if !model.is_cached().await? {
        model
            .download(Some(|percent| {
                print!("\rDownloading: {percent:5.1}%");
                let _ = io::stdout().flush();
            }))
            .await?;
        println!();
    }

    if !model.is_loaded().await? {
        model.load().await?;
    }

    let messages: Vec<ChatCompletionRequestMessage> = vec![
        ChatCompletionRequestSystemMessage::from(
            "Answer briefly. Do not invent facts that are absent from the prompt.",
        )
        .into(),
        ChatCompletionRequestUserMessage::from(
            "Explain reciprocal rank fusion in two sentences.",
        )
        .into(),
    ];

    let client = model
        .create_chat_client()
        .temperature(0.2)
        .max_tokens(160);

    let mut stream = client.complete_streaming_chat(&messages, None).await?;
    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        if let Some(token) = chunk
            .choices
            .first()
            .and_then(|choice| choice.delta.content.as_deref())
        {
            print!("{token}");
            io::stdout().flush()?;
        }
    }
    println!();

    model.unload().await?;
    Ok(())
}

This is intentionally a spike, not production lifecycle code. It proves the native path, but a service should not unload a shared model after every request. It should retain a manager, load lazily or during warmup, coordinate concurrent lifecycle operations, and keep active models resident according to a measured policy.

Understand each lifecycle step

1. Create one manager for the process

FoundryLocalManager::create initializes the native runtime. The project wraps the returned process singleton in a FoundryManager owned by application state. Recreating it per request would blur cache, provider, and model-residency ownership.

The project optionally supplies a cache directory to FoundryLocalConfig. This detail matters when the CLI and embedded application use different process state: a model visible to one runtime configuration is not evidence that the other points at the same cache or has the same execution providers registered.

2. Register execution providers in the same core

On Windows, hardware detection and execution-provider registration are distinct facts. A GPU or NPU appearing in operating-system inventory does not prove that the embedded Foundry core can use it.

The project learned this operationally: provider setup performed by a CLI service did not automatically register those providers in the server’s embedded manager. The application therefore invokes provider registration against its own manager and surfaces both physical-device discovery and SDK registration state.

Registration may download components. Treat it as setup/warmup work, expose progress where useful, and validate usability by loading and running a model - not by trusting a hardware label alone.

It is worth knowing what the SDK can target, because this is the property that makes one build deployable across facilities with different hardware. The project classifies each reported provider into a coarse accelerator class so an operator can see what the machine offers:

Unknown provider names are classified as Other and reported with the raw string rather than guessed at, so a newer SDK does not silently look like missing hardware.

The practical consequence for a constrained site is that a CPU-only host still answers, from the same code path, more slowly. That is a very different failure mode from “unsupported hardware,” and it is what lets the smallest facility in a rollout run the same build as the largest.

3. Resolve an alias, then inspect the concrete variant

catalog().get_model(alias) gives the SDK room to choose an optimized variant. The larger prototype also supports pinned variant IDs and explicit device preferences. It examines variant IDs for CPU, GPU, or NPU markers and applies application policy before loading. Device preference is an ordered list rather than a single target: a role specified as [Npu, Cpu] takes an openvino-npu or qnn-npu variant where one exists and falls back to generic-cpu where it does not, and [Gpu, Cpu] prefers openvino-gpu, generic-gpu, or cuda-gpu before the same fallback. Placement therefore adapts to the host instead of the host having to satisfy the configuration.

Alias convenience and deployment reproducibility pull in opposite directions. During exploration, use an alias. During controlled release, retain enough evidence to reproduce the concrete artifact and hardware path.

4. Cache before load

is_cached, download, is_loaded, load, and unload are separate lifecycle concepts. Downloads are artifact acquisition; load creates runtime residency. The first operation can be concurrent in the project, while native load and unload calls are serialized because concurrent lifecycle operations caused process crashes in observed Windows testing.

This distinction also improves readiness. The prototype warms the local embedding stack and execution providers, but does not download a missing large chat model merely to make startup appear ready. It can preload an already-cached routed model.

5. Stream content incrementally

complete_streaming_chat returns model deltas. The server keeps the stream alive while forwarding content over SSE to the client bridge. It marks the concrete model busy for the full stream lifetime so an administrator action or LRU eviction cannot unload weights mid-generation.

Streaming changes failure semantics. Errors before stream creation can become ordinary HTTP errors. Errors after tokens begin need an explicit stream error event, and partial output should not be persisted as a successful assistant message.

Turn the spike into an application service

The project adds several layers around the minimal code:

A ModelSpec maps tasks such as chat, classification, rewrite, summarization, verification, and text-to-SQL planning to an alias and generation settings. All current defaults use qwen3-8b on GPU, which reduces model swapping but is still configuration, not a universal recommendation.

The service imposes a GPU-resident LRU cap. Before loading another GPU variant, it unloads the oldest non-busy resident. If every resident is active, it temporarily exceeds the cap rather than invalidate a running generation. CPU and NPU variants have separate treatment because they do not necessarily consume the same GPU budget.

A model call is also not trusted merely because it is local. Structured outputs are parsed and validated. SQL is checked independently for read-only shape, table allowlists, limits, cost, and timeout. Tool descriptions do not grant direct database execution; Rust owns the operation.

Keep embedding out of this integration boundary

Current Foundry Local releases have broader capabilities than earlier versions, including embedding APIs in official documentation. That does not mean an existing application should silently replace its embedding stack.

This project’s index contract is BGE-M3 through fastembed: 1,024 dimensions, the same prefix-free encoder for documents and queries, and a separate bge-reranker-v2-m3 cross-encoder. Existing vectors, search indexes, thresholds, and evaluation data depend on that contract. The project therefore uses the Foundry Local SDK Reference for Rust for chat and tool-oriented generation only.

Changing embedding models is a data migration, not a model dropdown. It requires regenerating vectors, rebuilding compatible indexes, recalibrating retrieval, and preventing mixed vector spaces.

Failure modes worth testing early

A successful “hello world” is necessary but incomplete. Test these cases on every target hardware class:

• catalog initializes but the chosen alias is absent;
• model is listed but no acceptable device variant exists;
• execution provider is detected but not registered;
• the only available provider is CPU, and latency rises accordingly;
• artifact download is interrupted or the cache is truncated;
• a cached model fails to load on the selected provider;
• two requests attempt model lifecycle operations concurrently;
• unload is requested while a stream is active;
• the client disconnects during generation;
• the local runtime is unavailable when the application starts;
• the machine cannot fit the selected model and context in memory.

The prototype handles a narrow corrupt-cache signature by deleting and downloading once more. Other load failures remain visible rather than triggering a destructive purge. Foundry initialization failure does not abort the entire Rocket server; non-model operations stay available and model-dependent routes return unavailable.

The honest server tradeoff

The official Foundry Local overview explicitly distinguishes device inference from server runtimes. The device SDK is optimized for hardware-constrained scenarios where one user primarily accesses the model. It does not provide the continuous batching, sophisticated multi-user queuing, or GPU-sharing behavior of a dedicated serving stack.

Embedding it in a Rocket process can still be a useful prototype choice: deployment is compact, inference stays local, and the Rust application controls the complete request path. But multiple Desktop App and Mobile App connections do not turn the runtime into production multi-tenant inference infrastructure.

The prototype therefore uses a small process-local generation semaphore, a finite acquisition deadline, model busy guards, and explicit readiness signals. Those controls prevent obvious overload; they do not establish throughput. Production decisions require named hardware, representative prompts and contexts, latency distributions, memory/device utilization, failure tests, and sustained concurrency measurements.

If enterprise-scale Arc/Kubernetes operations and multi-node serving are the requirement, evaluate Foundry Local on Azure Local as a different architecture rather than stretching the device SDK beyond its intended role.

Key takeaways

• Prove catalog discovery, provider registration, model caching/loading, and streaming before adding a web layer.
• Use the current foundry-local-sdk v1 API; enable winml on Windows.
• List the catalog on the target machine and choose an available small alias for the first spike.
• Treat alias resolution, artifact caching, and runtime loading as separate concerns.
• Keep one manager per process and coordinate native load/unload operations.
• Prevent unloading a model while an active stream uses it.
• Preserve the application’s embedding-model contract; generation and embeddings need not share a runtime.
• Bound concurrency and measure it honestly because device-focused Foundry Local is not a continuous-batching multi-user server stack.

For the full implementation, see Role of Foundry Local, Models, accelerators, and lifecycle, Verified Platform Notes, and the Rust module at ../onprem-rag-server/src/foundry/mod.rs.

Learn more on Microsoft Learn

• What is Foundry Local?
• Get started with Foundry Local
• Foundry Local SDK Reference - Rust on Windows
• Understand Foundry Local on Azure Local for Arc/Kubernetes enterprise-scale deployments

Continue the series

• Previous: Part 1 - Drawing the Boundary for On-Premises Health RAG
• Next: Part 2 - DocumentDB as an Internal Hybrid Store: Documents, Vectors, State, and Safe Publication

 

The code behind this series is open source: github.com/kevin-gatimu/onprem-health-rag. It is a synthetic-data research prototype, not a clinical or compliance-ready system — issues and corrections are welcome.