A practical guide for AI engineers building production agents that need privacy, low latency, and frontier capability — without choosing just one.
Why hybrid, and why now
If you build AI features today, you are caught between three forces. Users want low latency and strong privacy. Product teams want frontier reasoning capability. Finance teams want predictable cost. No single model satisfies all three. Run everything on a small on-device model and you bottleneck on complex questions. Send everything to a frontier cloud model and you pay for trivial requests, leak sensitive data across a network boundary, and add hundreds of milliseconds of latency to greetings.
The pragmatic answer is hybrid inference: a lightweight local model classifies every request first, simple or sensitive ones stay on the device, and only the genuinely hard or frontier-capability requests escalate to the cloud. Microsoft now ships both halves of that pattern as supported Python SDKs — foundry-local-sdk for on-device inference and azure-ai-projects for Microsoft Foundry cloud models. This post walks through a working reference implementation that combines them behind a single ask() call.
The full source is at github.com/leestott/fl-mixedmodel. It is Python-only, secretless by design, and ships with a Gradio diagnostics UI, a CLI demo mode, and a full pytest suite.
The contract: one schema, two paths
The most important architectural decision is that callers never know which path served a request. Every response, local or cloud, returns the same dataclass:
class InferencePath(str, Enum):
LOCAL = "local"
CLOUD = "cloud"
LOCAL_FALLBACK = "local_fallback" # cloud attempted, fell back to local
CLOUD_FALLBACK = "cloud_fallback" # local attempted, fell back to cloud
@dataclass
class AgentResponse:
answer: str
path: InferencePath
model: str
reason: str
confidence: float
latency_ms: float
correlation_id: str
prompt_tokens: Optional[int] = None
completion_tokens: Optional[int] = None
fallback: bool = False
fallback_reason: Optional[str] = None
metadata: dict = field(default_factory=dict)
This is what makes the design honest. The router can change, the cloud model can be swapped from gpt-4o to gpt-5.4, fallback policies can flip — and the calling code never breaks. The four InferencePath values give you full observability without leaking implementation details into the API surface.
Architecture in one diagram
┌─────────────┐ prompt ┌──────────────────────────┐
│ caller │ ──────────► │ HybridAgentService │
└─────────────┘ │ .ask(prompt) │
└────────────┬─────────────┘
│
┌────────────▼─────────────┐
│ RoutingPolicy │
│ 1. Heuristic gate │
│ 2. Local router LLM │
│ 3. Hard policy gates │
└─────┬─────────────┬──────┘
│ │
LOCAL ◄┘ └► CLOUD
│ │
┌──────────▼──┐ ┌──────▼───────┐
│ Foundry │ │ Microsoft │
│ Local SDK │ │ Foundry │
│ (phi-4-mini)│ │ (gpt-5.4) │
└─────────────┘ └──────────────┘
Best practice: the two-stage router pattern
Before walking through the implementation, it is worth stating the design pattern explicitly, because it is the part that generalises beyond this specific repo. The cleanest design for hybrid inference is a two-stage router.
- Stage 1 — local router. A small local model performs intent and complexity classification first. It does not answer the question; it decides where the question should go.
- Stage 2 — route the answer.
- If the prompt is simple, private, latency-sensitive, or clearly within local capability, route to a local task model on the device.
- If the prompt is complex, needs deeper reasoning, a larger context window, or a capability unavailable locally, escalate to a cloud frontier model in Microsoft Foundry.
Microsoft's current guidance for the cloud side is to use the Responses API and choose one of two control modes:
- Pass a specific deployment name (for example
gpt-5.4) when you want deterministic control over which model serves the request, which is the right choice for regulated workloads, repeatable evaluations, or cost ceilings. - Pass
model-routeras the deployment when you want Microsoft Foundry to automatically select the best available cloud model for each request. This is a sensible default for general-purpose agents where you would rather let the platform optimise the model choice as new ones are released.
The reference repo exposes both as environment variables so you can switch without code changes:
# .env.example
FOUNDRY_CLOUD_MODEL_DEPLOYMENT=gpt-5.4 # deterministic
FOUNDRY_CLOUD_ROUTER_DEPLOYMENT=model-router # auto-select
Best practice: pin the right SDK versions
Two SDKs do the heavy lifting and both have had recent breaking changes, so version discipline matters.
- Local development —
foundry-local-sdk. The current public guidance is to use the Foundry Local SDK packagefoundry-local-sdk, which provides model discovery, download, cache, load, unload, chat completions, embeddings, audio transcription, and an optional built-in web service. Use version 1.1.0, released on 5 May 2026. Earlier versions used an OpenAI-compatible client surface that has since been replaced by theFoundryLocalManager → load_model → get_chat_client → complete_chatchain shown above. Pin it explicitly:# requirements.txt foundry-local-sdk>=1.1.0 - Cloud orchestration and agents —
azure-ai-projects. For cloud-side orchestration, Microsoft's current Python guidance is to useazure-ai-projects, which the docs describe as part of the Microsoft Foundry SDK and as the entry point for agents, deployments, connections, datasets, evaluations, and an OpenAI-compatible client returned byget_openai_client(). The current PyPI listing shows azure-ai-projects 2.1.0. Pin it explicitly:# requirements.txt azure-ai-projects>=2.1.0 azure-identity>=1.17.0
If you find yourself reading old samples that import azure.ai.inference as the cloud entry point, or that initialise Foundry Local through a raw openai.OpenAI(base_url=...) client, you are looking at pre-2026 patterns. The current shape is what the reference repo uses: FoundryLocalManager.initialize(Configuration(...)) for the device and AIProjectClient(...).get_openai_client() for the cloud.
Stage 1: a deterministic privacy gate
Before any model touches a prompt, a deterministic heuristic classifier scans for sensitive patterns — passwords, API keys, SSN/NHS numbers, PII signals, explicit "do not share" flags. If the heuristic returns PrivacyClass.RESTRICTED, the prompt is forced local. The router LLM is not called. The cloud provider is not called. The decision is auditable from a single regex pass.
# app/routing/policy.py
def decide(self, prompt: str, correlation_id: str = "") -> RoutingDecision:
hint, privacy, complexity, h_reason = self._heuristic.classify(prompt)
# Hard gate: restricted content never leaves the device
if privacy == PrivacyClass.RESTRICTED:
return self._make_decision(
target=RouteTarget.LOCAL,
confidence=1.0,
reason=f"Policy hard-gate: {h_reason}",
privacy=privacy,
complexity=complexity,
deterministic=True,
correlation_id=correlation_id,
)
# Hard gate: very high complexity always goes to cloud
if complexity == ComplexityBand.VERY_HIGH:
return self._make_decision(
target=RouteTarget.CLOUD,
confidence=1.0,
reason="Policy hard-gate: very_high complexity requires frontier model",
...
)
This is the most important responsible-AI control in the whole system. If your privacy review depends on an LLM correctly classifying every prompt, you do not have a privacy control — you have a probability distribution. Deterministic gates first, model judgement second.
Stage 2: a local LLM as the router
For everything that passes the privacy gate, a small local model classifies whether the prompt needs frontier capability. This is the bit that surprises most engineers: you can do useful routing with a 4B parameter model running on a laptop CPU. The router does not need to answer the question. It only needs to classify it.
The reference implementation uses phi-4-mini via Foundry Local. Initialising it is two lines:
# app/providers/local_provider.py (excerpt)
from foundry_local import FoundryLocalManager
from foundry_local.models import Configuration
self._manager = FoundryLocalManager.initialize(
Configuration(app_name="hybrid-agent")
)
self._router_model = self._manager.load_model(self._config.local_router_alias)
self._chat_client = self._router_model.get_chat_client()
response = self._chat_client.complete_chat(
messages=[
{"role": "system", "content": ROUTER_SYSTEM_PROMPT},
{"role": "user", "content": prompt},
],
)
The router prompt asks for a strict JSON response: { "target": "local|cloud", "confidence": 0.0-1.0, "complexity": "low|medium|high|very_high", "reason": "..." }. The application parses it, applies the confidence threshold from config (default 0.6), and falls back to the heuristic decision if the router LLM is unsure or its JSON is malformed. The router never blocks the answer path — that is a deliberate reliability choice.
Cloud inference via Microsoft Foundry
When the policy returns RouteTarget.CLOUD, the request goes through AIProjectClient, which gives you an openai.OpenAI-compatible client wired to your Foundry project with DefaultAzureCredential. No API keys. No secrets in .env.
# app/providers/cloud_provider.py (excerpt)
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
self._project = AIProjectClient(
endpoint=self._config.foundry_project_endpoint,
credential=DefaultAzureCredential(),
)
self._openai_client = self._project.get_openai_client()
response = self._openai_client.chat.completions.create(
model=self._config.foundry_cloud_model_deployment, # e.g. "gpt-5.4"
messages=messages,
max_completion_tokens=max_tokens,
)
A subtle gotcha worth flagging: gpt-5 and o-series deployments reject the legacy max_tokens parameter and require max_completion_tokens. They also reject custom temperature values. The reference repo handles this by trying the new parameter first and falling back to the legacy one only when the API returns the specific unsupported parameter error. That keeps the same code working against older deployments without forking the provider.
Graceful degradation: the fallback paths
Hybrid systems fail in interesting ways. The cloud can be down. The local model can throw because the GPU ran out of memory. A reasoning model can return an empty completion. The service handles all of these by attempting the alternative path and labelling the response so observability stays honest:
- Cloud route fails → local fallback. The response carries
path=LOCAL_FALLBACK,fallback=true, and a populatedfallback_reason. The user gets an answer instead of an error. - Local route fails → cloud fallback, but only if privacy class is not RESTRICTED. A sensitive prompt that the local model could not handle never leaks to the cloud as a fallback. It returns a clear error instead. This is the second hard gate in the system.
- Both fail. A structured error response with a correlation ID, never a stack trace.
That last rule — fallback respects privacy class — is the kind of decision that is easy to skip and impossible to bolt on later. Encode it once in the service layer and your privacy reviewers will thank you.
What it looks like in practice
The diagnostics panel in the Gradio UI shows the routing decision live: path, model, confidence, latency, privacy class, complexity band, and the full JSON response. Five canonical scenarios shake out the entire decision tree:
"hello"→path=local, confidence=1.0, complexity=low. Heuristic only. No router LLM call. ~3 seconds end-to-end with phi-4-mini cached."explain transformer self-attention in depth with maths"→path=cloud, model=gpt-5.4, complexity=high. Router LLM classifies, hard gate confirms."my password is hunter2, suggest a stronger one"→path=local, privacy=restricted, deterministic=true. Privacy gate fires before any model sees it."summarise this 8 KB document"with cloud unavailable →path=cloud_fallback(local handles it, response is labelled).- Complex prompt with local model error →
path=local_fallback,fallback_reasonpopulated.
You can reproduce all five without any models installed by running python -m app.main --demo. The demo mode swaps the providers for deterministic stubs so you can validate the routing logic and the response schema in under a second on any machine.
Operational lessons learned
Some things the reference implementation only gets right because it got them wrong first:
- Pick a non-reasoning model for the router. Reasoning-tuned local models (Phi-4-reasoning, o-style) wrap their output in
<think>blocks and blow your JSON parser.phi-4-miniis faster and more reliable for classification. - Cache the local model. First load can take 30–60 seconds while Foundry Local downloads weights. Initialise the service once at process startup, not per request.
- Use correlation IDs everywhere. The service attaches one per request and the structured JSON logger emits it on every event. When you are debugging a fallback path across two model providers, this is the difference between five minutes and five hours.
- Run the privacy heuristic on every fallback path too. A naive implementation might route locally, fail, and then send the same sensitive prompt to the cloud as a "graceful" fallback. That is not graceful, it is a data leak.
- Keep configuration in
.envand out of code. Privacy mode, fallback toggles, confidence threshold, model aliases — all environment-driven. Theconfig.pymodule is the only place that reads them.
Responsible AI in a hybrid topology
Hybrid does not make responsible AI harder, but it does make it different. Three controls earn their keep:
- Data residency by default. The local path keeps prompts and answers on the device. For RESTRICTED content this is mandatory; for everything else it is a free latency and cost win.
- Auditability. Every routing decision is logged with the deterministic reason, the heuristic class, the router LLM output, the confidence, and the correlation ID. You can answer "why did this prompt go to the cloud?" months later.
- Keyless auth.
DefaultAzureCredentialmeans there is no API key to leak, rotate, or commit by accident. The repo's.gitignore,SECURITY.md, and pre-push checklist enforce this end-to-end.
Try it
Five minutes, no Azure account needed for the demo:
git clone https://github.com/leestott/fl-mixedmodel.git
cd fl-mixedmodel
python -m venv .venv
.venv\Scripts\activate # Windows
# source .venv/bin/activate # macOS / Linux
pip install -r requirements.txt
python -m app.main --demo # all five scenarios, no models required
To run with real models, install Foundry Local, copy .env.example to .env, set your FOUNDRY_PROJECT_ENDPOINT, then:
az login
python -m app.main --ui --port 7860
Where to go next
- Repository: github.com/leestott/fl-mixedmodel — full source, tests, specification, screenshots.
- Foundry Local SDK: pypi.org/project/foundry-local-sdk and the Foundry Local docs.
- Azure AI Projects SDK: pypi.org/project/azure-ai-projects and the Microsoft Foundry docs.
- Azure Identity: DefaultAzureCredential reference.
- Phi-4-mini: Phi-4-mini on Hugging Face.
Key takeaways
- The best-practice pattern is a two-stage router: local model classifies first, then either a local task model or a Microsoft Foundry cloud model answers.
- For cloud control, use the Responses API with either a named deployment (deterministic) or
model-router(auto-select). - Pin
foundry-local-sdk >= 1.1.0(5 May 2026) andazure-ai-projects >= 2.1.0. The 2026 SDK surfaces are not backwards-compatible with pre-2026 samples. - Hybrid inference is a routing problem, not a model problem. A small local model is enough to classify the request.
- Deterministic privacy gates beat probabilistic ones. Code the rules; let the LLM judge only what is left.
- Return the same response schema from every path. Label fallbacks honestly. Carry a correlation ID everywhere.
- Keep auth keyless with
DefaultAzureCredentialand your.envout of git. - Test the routing decisions, not just the model outputs. Demo mode and a strong
pytestsuite pay back every time you swap a model.
Hybrid AI is not a compromise between local and cloud. It is the supervisor pattern applied to inference — fast and private where you can be, frontier where you have to be, observable everywhere. The hard part is the contract, not the models.
Microsoft