Educator Developer Blog articles

Build and Deploy a Microsoft Foundry Hosted Agent: A Hands-On Workshop

Lee_Stott — Fri, 03 Apr 2026 11:25:45 GMT

Agents are easy to demo, hard to ship.

Most teams can put together a convincing prototype quickly. The harder part starts afterwards: shaping deterministic tools, validating behaviour with tests, building a CI path, packaging for deployment, and proving the experience through a user-facing interface. That is where many promising projects slow down.

This workshop helps you close that gap without unnecessary friction. You get a guided path from local run to deployment handoff, then complete the journey with a working chat UI that calls your deployed hosted agent through the project endpoint.

What You Will Build

This is a hands-on, end-to-end learning experience for building and deploying AI agents with Microsoft Foundry.

The lab provides a guided and practical journey through hosted-agent development, including deterministic tool design, prompt-guided workflows, CI validation, deployment preparation, and UI integration.

It’s designed to reduce setup friction with a ready-to-run experience.

It is a prompt-based development lab using Copilot guidance and MCP-assisted workflow options during deployment.

It’s a .NET 10 workshop that includes local development, Copilot-assisted coding, CI, secure deployment to Azure, and a working chat UI.

A local hosted agent that responds on the responses contract
Deterministic tool improvements in core logic with xUnit coverage
A GitHub Actions CI workflow for restore, build, test, and container validation
An Azure-ready deployment path using azd, ACR image publishing, and Foundry manifest apply
A Blazor chat UI that calls openai/v1/responses with agent_reference
A repeatable implementation shape that teams can adapt to real projects

Who This Lab Is For

AI developers and software engineers who prefer learning by building
Motivated beginners who want a guided, step-by-step path
Experienced developers who want a practical hosted-agent reference implementation
Architects evaluating deployment shape, validation strategy, and operational readiness
Technical decision-makers who need to see how demos become deployable systems

Why Hosted Agents

Hosted agents run your code in a managed environment. That matters because it reduces the amount of infrastructure plumbing you need to manage directly, while giving you a clearer path to secure, observable, team-friendly deployments.

Prompt-only demos are still useful. They are quick, excellent for ideation, and often the right place to start. Hosted agents complement that approach when you need custom code, tool-backed logic, and a deployment process that can be repeated by a team.

Think of this lab as the bridge: you keep the speed of prompt-based iteration, then layer in the real-world patterns needed to run reliably.

What You Will Learn

1) Orchestration

You will practise workflow-oriented reasoning through implementation-shape recommendations and multi-step readiness scenarios. The lab introduces orchestration concepts at a practical level, rather than as a dedicated orchestration framework deep dive.

2) Tool Integration

You will connect deterministic tools and understand how tool calls fit into predictable execution paths. This is a core focus of the workshop and is backed by tests in the solution.

3) Retrieval Patterns (What This Lab Covers Today)

This workshop does not include a full RAG implementation with embeddings and vector search. Instead, it focuses on deterministic local tools and hosted-agent response flow, giving you a strong foundation before adding retrieval infrastructure in a follow-on phase.

4) Observability

You will see light observability foundations through OpenTelemetry usage in the host and practical verification during local and deployed checks. This is introductory coverage intended to support debugging and confidence building.

5) Responsible AI

You will apply production-minded safety basics, including secure secret handling and review hygiene. A full Responsible AI policy and evaluation framework is not the primary goal of this workshop, but the workflow does encourage safe habits from the start.

6) Secure Deployment Path

You will move from local implementation to Azure deployment with a secure, practical workflow: azd provisioning, ACR publishing, manifest deployment, hosted-agent start, status checks, and endpoint validation.

The Learning Journey

The overall flow is simple and memorable: clone, open, run, iterate, deploy, observe.

clone -> open -> run -> iterate -> deploy -> observe

You are not expected to memorize every command. The lab is structured to help you learn through small, meaningful wins that build confidence.

Your First 15 Minutes: Quick Wins

Open the repo and understand the lab structure in a few minutes
Set project endpoint and model deployment environment variables
Run the host locally and validate the responses endpoint
Inspect the deterministic tools in WorkshopLab.Core
Run tests and see how behaviour changes are verified
Review the deployment path so local work maps to Azure steps
Understand how the UI validates end-to-end behaviour after deployment
Leave the first session with a working baseline and a clear next step

That first checkpoint is important. Once you see a working loop on your own machine, the rest of the workshop becomes much easier to finish.

Using Copilot and MCP in the Workflow

This lab emphasises prompt-based development patterns that help you move faster while still learning the underlying architecture. You are not only writing code, you are learning to describe intent clearly, inspect generated output, and iterate with discipline.

Copilot supports implementation and review in the coding labs. MCP appears as a practical deployment option for hosted-agent lifecycle actions, provided your tools are authenticated to the correct tenant and project context.

Together, this creates a development rhythm that is especially useful for learning:

Define intent with clear prompts
Generate or adjust implementation details
Validate behaviour through tests and UI checks
Deploy and observe outcomes in Azure
Refine based on evidence, not guesswork

That same rhythm transfers well to real projects. Even if your production environment differs, the patterns from this workshop are adaptable.

Production-Minded Tips

As you complete the lab, keep a production mindset from day one:

Reliability: keep deterministic logic small, testable, and explicit
Security: Treat secrets, identity, and access boundaries as first-class concerns
Observability: use telemetry and status checks to speed up debugging
Governance: keep deployment steps explicit so teams can review and repeat them

You do not need to solve everything in one pass. The goal is to build habits that make your agent projects safer and easier to evolve.

Start Today:

If you have been waiting for the right time to move from “interesting demo” to “practical implementation”, this is the moment. The workshop is structured for self-study, and the steps are designed to keep your momentum high.

Start here: https://github.com/microsoft/Hosted_Agents_Workshop_Lab

Want deeper documentation while you go? These official guides are great companions:

When you finish, share what you built. Post a screenshot or short write-up in a GitHub issue/discussion, on social, or in comments with one lesson learned. Your example can help the next developer get unstuck faster.

Copy/Paste Progress Checklist

[ ] Clone the workshop repo
[ ] Complete local setup and run the agent
[ ] Make one prompt-based behaviour change
[ ] Validate with tests and chat UI
[ ] Run CI checks
[ ] Provision and deploy via Azure and Foundry workflow
[ ] Review observability signals and refine
[ ] Share what I built + one takeaway

Common Questions

How long does it take?

Most developers can complete a meaningful pass in a few focused sessions of 60-75 mins. You can get the first local success quickly, then continue through deployment and refinement at your own pace.

Do I need an Azure subscription?

Yes, for provisioning and deployment steps. You can still begin local development and testing before completing all Azure activities.

Is it beginner-friendly?

Yes. The labs are written for beginners, run in sequence, and include expected outcomes for each stage.

Can I adapt it beyond .NET?

Yes. The implementation in this workshop is .NET 10, but the architecture and development patterns can be adapted to other stacks.

What if I am evaluating for a team?

This lab is a strong team evaluation asset because it demonstrates end-to-end flow: local dev, integration patterns, CI, secure deployment, and operational visibility.

Closing

This workshop gives you more than theory. It gives you a practical path from first local run to deployed hosted agent, backed by tests, CI, and a user-facing UI validation loop. If you want a build-first route into Microsoft Foundry hosted-agent development, this is an excellent place to start.

Begin now: https://github.com/microsoft/Hosted_Agents_Workshop_Lab

Getting Started with Foundry Local: A Student Guide to the Microsoft Foundry Local Lab

Lee_Stott — Mon, 30 Mar 2026 07:00:00 GMT

If you want to start building AI applications on your own machine, the Microsoft Foundry Local Lab is one of the most useful places to begin. It is a practical workshop that takes you from first-time setup through to agents, retrieval, evaluation, speech transcription, tool calling, and a browser-based interface. The material is hands-on, cross-language, and designed to show how modern AI apps can run locally rather than depending on a cloud service for every step.

This blog post is aimed at students, self-taught developers, and anyone learning how AI applications are put together in practice. Instead of treating large language models as a black box, the lab shows you how to install and manage local models, connect to them with code, structure tasks into workflows, and test whether the results are actually good enough. If you have been looking for a learning path that feels more like building real software and less like copying isolated snippets, this workshop is a strong starting point.

What Is Foundry Local?

Foundry Local is a local runtime for downloading, managing, and serving AI models on your own hardware. It exposes an OpenAI-compatible interface, which means you can work with familiar SDK patterns while keeping execution on your device. For learners, that matters for three reasons. First, it lowers the barrier to experimentation because you can run projects without setting up a cloud account for every test. Second, it helps you understand the moving parts behind AI applications, including model lifecycle, local inference, and application architecture. Third, it encourages privacy-aware development because the examples are designed to keep data on the machine wherever possible.

The Foundry Local Lab uses that local-first approach to teach the full journey from simple prompts to multi-agent systems. It includes examples in Python, JavaScript, and C#, so you can follow the language that fits your course, your existing skills, or the platform you want to build on.

Why This Lab Works Well for Learners

A lot of AI tutorials stop at the moment a model replies to a prompt. That is useful for a first demo, but it does not teach you how to build a proper application. The Foundry Local Lab goes further. It is organised as a sequence of parts, each one adding a new idea and giving you working code to explore. You do not just ask a model to respond. You learn how to manage the service, choose a language SDK, construct retrieval pipelines, build agents, evaluate outputs, and expose the result through a usable interface.

That sequence is especially helpful for students because the parts build on each other. Early labs focus on confidence and setup. Middle labs focus on architecture and patterns. Later labs move into more advanced ideas that are common in real projects, such as tool calling, evaluation, and custom model packaging. By the end, you have seen not just what a local AI app looks like, but how its different layers fit together.

Before You Start

The workshop expects a reasonably modern machine and at least one programming language environment. The core prerequisites are straightforward: install Foundry Local, clone the repository, and choose whether you want to work in Python, JavaScript, or C#. You do not need to master all three. In fact, most learners will get more value by picking one language first, completing the full path in that language, and only then comparing how the same patterns look elsewhere.

If you are new to AI development, do not be put off by the number of parts. The early sections are accessible, and the later ones become much easier once you have completed the foundations. Think of the lab as a structured course rather than a single tutorial.

What You Learn in Each Lab https://github.com/microsoft-foundry/foundry-local-lab

Part 1: Getting Started with Foundry Local

The first part introduces the basics of Foundry Local and gets you up and running. You learn how to install the CLI, inspect the model catalogue, download a model, and run it locally. This part also introduces practical details such as model aliases and dynamic service ports, which are small but important pieces of real development work.

For students, the value of this part is confidence. You prove that local inference works on your machine, you see how the service behaves, and you learn the operational basics before writing any application code. By the end of Part 1, you should understand what Foundry Local does, how to start it, and how local model serving fits into an application workflow.

Part 2: Foundry Local SDK Deep Dive

Once the CLI makes sense, the workshop moves into the SDK. This part explains why application developers often use the SDK instead of relying only on terminal commands. You learn how to manage the service programmatically, browse available models, control model download and loading, and understand model metadata such as aliases and hardware-aware selection.

This is where learners start to move from using a tool to building with a platform. You begin to see the difference between running a model manually and integrating it into software. By the end of this section, you should understand the API surface you will use in your own projects and know how to bootstrap the SDK in Python, JavaScript, or C#.

Part 3: SDKs and APIs

Part 3 turns the SDK concepts into a working chat application. You connect code to the local inference server and use the OpenAI-compatible API for streaming chat completions. The lab includes examples in all three supported languages, which makes it especially useful if you are comparing ecosystems or learning how the same idea is expressed through different syntax and libraries.

The key learning outcome here is not just that you can get a response from a model. It is that you understand the boundary between your application and the local model service. You learn how messages are structured, how streaming works, and how to write the sort of integration code that becomes the foundation for every later lab.

Part 4: Retrieval-Augmented Generation

This is where the workshop starts to feel like modern AI engineering rather than basic prompting. In the retrieval-augmented generation lab, you build a simple RAG pipeline that grounds answers in supplied data. You work with an in-memory knowledge base, apply retrieval logic, score matches, and compose prompts that include grounded context.

For learners, this part is important because it demonstrates a core truth of AI app development: a model on its own is often not enough. Useful applications usually need access to documents, notes, or structured information. By the end of Part 4, you understand why retrieval matters, how to pass retrieved context into a prompt, and how a pipeline can make answers more relevant and reliable.

Part 5: Building AI Agents

Part 5 introduces the concept of an agent. Instead of a one-off prompt and response, you begin to define behaviour through system instructions, roles, and conversation state. The lab uses the ChatAgent pattern and the Microsoft Agent Framework to show how an agent can maintain a purpose, respond with a persona, and return structured output such as JSON.

This part helps learners understand the difference between a raw model call and a reusable application component. You learn how to design instructions that shape behaviour, how multi-turn interaction differs from single prompts, and why structured output matters when an AI component has to work inside a broader system.

Part 6: Multi-Agent Workflows

Once a single agent makes sense, the workshop expands the idea into a multi-agent workflow. The example pipeline uses roles such as researcher, writer, and editor, with outputs passed from one stage to the next. You explore sequential orchestration, shared configuration, and feedback loops between specialised components.

For students, this lab is a very clear introduction to decomposition. Instead of asking one model to do everything at once, you break a task into smaller responsibilities. That pattern is useful well beyond AI. By the end of Part 6, you should understand why teams build multi-agent systems, how hand-offs are structured, and what trade-offs appear when more components are added to a workflow.

Part 7: Zava Creative Writer Capstone Application

The Zava Creative Writer is the capstone project that brings the earlier ideas together into a more production-style application. It uses multiple specialised agents, structured JSON hand-offs, product catalogue search, streaming output, and evaluation-style feedback loops. Rather than showing an isolated feature, this part shows how separate patterns combine into a complete system.

This is one of the most valuable parts of the workshop for learner developers because it narrows the gap between tutorial code and real application design. You can see how orchestration, agent roles, and practical interfaces fit together. By the end of Part 7, you should be able to recognise the architecture of a serious local AI app and understand how the earlier labs support it.

Part 8: Evaluation-Led Development

Many beginner AI projects stop once the output looks good once or twice. This lab teaches a much stronger habit: evaluation-led development. You work with golden datasets, rule-based checks, and LLM-as-judge scoring to compare prompt or agent variants systematically. The goal is to move from anecdotal testing to repeatable assessment.

This matters enormously for students because evaluation is one of the clearest differences between a classroom demo and dependable software. By the end of Part 8, you should understand how to define success criteria, compare outputs at scale, and use evidence rather than intuition when improving an AI component.

Part 9: Voice Transcription with Whisper

Part 9 broadens the workshop beyond text generation by introducing speech-to-text with Whisper running locally. You use the Foundry Local SDK to download and load the model, then transcribe local audio files through the compatible API surface. The emphasis is on privacy-first processing, with audio kept on-device.

This section is a useful reminder that local AI development is not limited to chatbots. Learners see how a different modality fits into the same ecosystem and how local execution supports sensitive workloads. By the end of this lab, you should understand the transcription flow, the relevant client methods, and how speech features can be integrated into broader applications.

Part 10: Using Custom or Hugging Face Models

After learning the standard path, the workshop shows how to work with custom or Hugging Face models. This includes compiling models into optimised ONNX format with ONNX Runtime GenAI, choosing hardware-specific options, applying quantisation strategies, creating configuration files, and adding compiled models to the Foundry Local cache.

For learner developers, this part opens the door to model engineering rather than simple model consumption. You begin to understand that model choice, optimisation, and packaging affect performance and usability. By the end of Part 10, you should have a clearer picture of how models move from an external source into a runnable local setup and why deployment format matters.

Part 11: Tool Calling with Local Models

Tool calling is one of the most practical patterns in current AI development, and this lab covers it directly. You define tool schemas, allow the model to request function calls, handle the multi-turn interaction loop, execute the tools locally, and return results back to the model. The examples include practical scenarios such as weather and population tools.

This lab teaches learners how to move beyond generation into action. A model is no longer limited to producing text. It can decide when external data or a function is needed and incorporate that result into a useful answer. By the end of Part 11, you should understand the tool-calling flow and how AI systems connect reasoning with deterministic software behaviour.

Part 12: Building a Web UI for the Zava Creative Writer

Part 12 adds a browser-based front end to the capstone application. You learn how to serve a shared interface from Python, JavaScript, or C#, stream updates to the browser, consume NDJSON with the Fetch API and ReadableStream, and show live agent status as content is produced in real time.

This part is especially good for students who want to build portfolio projects. It turns backend orchestration into something visible and interactive. By the end of Part 12, you should understand how to connect a local AI backend to a web interface and how streaming changes the user experience compared with waiting for one final response.

Part 13: Workshop Complete

The final part is a summary and extension point. It reviews what you have built across the previous sections and suggests ways to continue. Although it is not a new technical lab in the same way as the earlier parts, it plays an important role in learning. It helps you consolidate the architecture, the terminology, and the development patterns you have encountered.

For learners, reflection matters. By the end of Part 13, you should be able to describe the full stack of a local AI application, from model management to user interface, and identify which area you want to deepen next.

What Students Gain from the Full Workshop

Taken together, these labs do more than teach Foundry Local itself. They teach how AI applications are built. You learn operational basics such as model setup and service management. You learn application integration through SDKs and APIs. You learn system design through RAG, agents, multi-agent orchestration, and web interfaces. You learn engineering discipline through evaluation. You also see how text, speech, custom models, and tool calling all fit into one local-first development workflow.

That breadth makes the workshop useful in several settings. A student can use it as a self-study path. A lecturer can use it as source material for practical sessions. A learner developer can use it to build portfolio pieces and to understand which AI patterns are worth learning next. Because the repository includes Python, JavaScript, and C#, it also works well for comparing how architectural ideas transfer across languages.

How to Approach the Lab as a Beginner

If you are starting from scratch, the best route is simple. Complete Parts 1 to 3 in your preferred language first. That gives you the essential setup and integration skills. Then move into Parts 4 to 6 to understand how AI application patterns are composed. After that, use Parts 7 and 8 to learn how larger systems and evaluation fit together. Finally, explore Parts 9 to 12 based on your interests, whether that is speech, tooling, model customisation, or front-end work.

It is also worth keeping notes as you go. Record what each part adds to your understanding, what code files matter, and what assumptions each example makes. That habit will help you move from following the labs to adapting the patterns in your own projects.

Final Thoughts

The Microsoft Foundry Local Lab is a strong introduction to local AI development because it treats learners like developers rather than spectators. You install, run, connect, orchestrate, evaluate, and present working systems. That makes it far more valuable than a short demo that only proves a model can answer a question.

If you are a student or learner developer who wants to understand how AI applications are really built, this lab gives you a clear path. Start with the basics, pick one language, and work through the parts in order. By the time you finish, you will not just have used Foundry Local. You will have a practical foundation for building local AI applications with far more confidence and much better judgement.

Langchain Multi-Agent Systems with Microsoft Agent Framework and Hosted Agents

Lee_Stott — Thu, 26 Mar 2026 07:00:00 GMT

If you have been building AI agents with LangChain, you already know how powerful its tool and chain abstractions are. But when it comes to deploying those agents to production — with real infrastructure, managed identity, live web search, and container orchestration — you need something more.

This post walks through how to combine LangChain with the Microsoft Agent Framework (azure-ai-agents) and deploy the result as a Microsoft Foundry Hosted Agent. We will build a multi-agent incident triage copilot that uses LangChain locally and seamlessly upgrades to cloud-hosted capabilities on Microsoft Foundry.

Why combine LangChain with Microsoft Agent Framework?

As a LangChain developer, you get excellent abstractions for building agents: the @tool decorator, RunnableLambda chains, and composable pipelines. But production deployment raises questions that LangChain alone does not answer:

Where do your agents run? Containers, serverless, or managed infrastructure?
How do you add live web search or code execution? Bing Grounding and Code Interpreter are not LangChain built-ins.
How do you handle authentication? Managed identity, API keys, or tokens?
How do you observe agents in production? Distributed tracing across multiple agents?

The Microsoft Agent Framework fills these gaps. It provides AgentsClient for creating and managing agents on Microsoft Foundry, built-in tools like BingGroundingTool and CodeInterpreterTool, and a thread-based conversation model. Combined with Hosted Agents, you get a fully managed container runtime with health probes, auto-scaling, and the OpenAI Responses API protocol.

The key insight: LangChain handles local logic and chain composition; the Microsoft Agent Framework handles cloud-hosted orchestration and tooling.

Architecture overview

The incident triage copilot uses a coordinator pattern with three specialist agents:

User Query
    |
    v
Coordinator Agent
    |
    +--> LangChain Triage Chain    (routing decision)
    +--> LangChain Synthesis Chain  (combine results)
    |
    +---+---+---+
    |   |       |
    v   v       v
Research  Diagnostics  Remediation
 Agent      Agent        Agent

Each specialist agent has two execution modes:

Mode	LangChain Role	Microsoft Agent Framework Role
Local	`@tool` functions provide heuristic analysis	Not used
Foundry	Chains handle routing and synthesis	`AgentsClient` with `BingGroundingTool`, `CodeInterpreterTool`

This dual-mode design means you can develop and test locally with zero cloud dependencies, then deploy to Foundry for production capabilities.

Step 1: Define your LangChain tools

Start with what you know. Define typed, documented tools using LangChain’s @tool decorator:

from langchain_core.tools import tool @tool def classify_incident_severity(query: str) -> str: """Classify the severity and priority of an incident based on keywords. Args: query: The incident description text. Returns: Severity classification with priority level. """ query_lower = query.lower() critical_keywords = [ "production down", "all users", "outage", "breach", ] high_keywords = [ "503", "500", "timeout", "latency", "slow", ] if any(kw in query_lower for kw in critical_keywords): return "severity=critical, priority=P1" if any(kw in query_lower for kw in high_keywords): return "severity=high, priority=P2" return "severity=low, priority=P4"

These tools work identically in local mode and serve as fallbacks when Foundry is unavailable.

Step 2: Build routing with LangChain chains

Use RunnableLambda to create a routing chain that classifies the incident and selects which specialists to invoke:

from langchain_core.runnables import RunnableLambda from enum import Enum class AgentRole(str, Enum): RESEARCH = "research" DIAGNOSTICS = "diagnostics" REMEDIATION = "remediation" DIAGNOSTICS_KEYWORDS = { "log", "error", "exception", "timeout", "500", "503", "crash", "oom", "root cause", } REMEDIATION_KEYWORDS = { "fix", "remediate", "runbook", "rollback", "hotfix", "patch", "resolve", "action plan", } def _route(inputs: dict) -> dict: query = inputs["query"].lower() specialists = [AgentRole.RESEARCH] # always included if any(kw in query for kw in DIAGNOSTICS_KEYWORDS): specialists.append(AgentRole.DIAGNOSTICS) if any(kw in query for kw in REMEDIATION_KEYWORDS): specialists.append(AgentRole.REMEDIATION) return {**inputs, "specialists": specialists} triage_routing_chain = RunnableLambda(_route)

This is pure LangChain — no cloud dependency. The chain analyses the query and returns which specialists should handle it.

Step 3: Create specialist agents with dual-mode execution

Each specialist agent extends a base class. In local mode, it uses LangChain tools. In Foundry mode, it delegates to the Microsoft Agent Framework:

from abc import ABC, abstractmethod from pathlib import Path class BaseSpecialistAgent(ABC): role: AgentRole prompt_file: str def __init__(self): prompt_path = Path(__file__).parent.parent / "prompts" / self.prompt_file self.system_prompt = prompt_path.read_text(encoding="utf-8") async def run(self, query, shared_context, correlation_id, client=None): if client is not None: return await self._run_on_foundry(query, shared_context, correlation_id, client) return await self._run_locally(query, shared_context, correlation_id) async def _run_on_foundry(self, query, shared_context, correlation_id, client): """Use Microsoft Agent Framework for cloud-hosted execution.""" from azure.ai.agents.models import BingGroundingTool agent = await client.agents.create_agent( model=shared_context.get("model_deployment", "gpt-4o"), name=f"{self.role.value}-{correlation_id}", instructions=self.system_prompt, tools=self._get_foundry_tools(shared_context), ) thread = await client.agents.threads.create() await client.agents.messages.create( thread_id=thread.id, role="user", content=self._build_prompt(query, shared_context), ) run = await client.agents.runs.create_and_process( thread_id=thread.id, agent_id=agent.id, ) # Extract and return the agent’s response... async def _run_locally(self, query, shared_context, correlation_id): """Use LangChain tools for local heuristic analysis.""" # Each subclass implements this with its specific tools ...

The key pattern here: same interface, different backends. Your coordinator does not care whether a specialist ran locally or on Foundry.

Step 4: Wire it up with FastAPI

Expose the multi-agent pipeline through a FastAPI endpoint. The /triage endpoint accepts incident descriptions and returns structured reports:

from fastapi import FastAPI from agents.coordinator import Coordinator from models import TriageRequest app = FastAPI(title="Incident Triage Copilot") coordinator = Coordinator() @app.post("/triage") async def triage(request: TriageRequest): return await coordinator.triage( request=request, client=app.state.foundry_client, max_turns=10, )

The application also implements the /responses endpoint, which follows the OpenAI Responses API protocol. This is what Microsoft Foundry Hosted Agents expects when routing traffic to your container.

Step 5: Deploy as a Hosted Agent

This is where Microsoft Foundry Hosted Agents shines. Your multi-agent system becomes a managed, auto-scaling service with a single command:

# Install the azd AI agent extension azd extension install azure.ai.agents # Provision infrastructure and deploy azd up

The Azure Developer CLI (azd) provisions everything:

Azure Container Registry for your Docker image
Container App with health probes and auto-scaling
User-Assigned Managed Identity for secure authentication
Microsoft Foundry Hub and Project with model deployments
Application Insights for distributed tracing

Your agent.yaml defines what tools the hosted agent has access to:

name: incident-triage-copilot-langchain kind: hosted model: deployment: gpt-4o identity: type: managed tools: - type: bing_grounding enabled: true - type: code_interpreter enabled: true

What you gain over pure LangChain

Capability	LangChain Only	LangChain + Microsoft Agent Framework
Local development	Yes	Yes (identical experience)
Live web search	Requires custom integration	Built-in `BingGroundingTool`
Code execution	Requires sandboxing	Built-in `CodeInterpreterTool`
Managed hosting	DIY containers	Foundry Hosted Agents
Authentication	DIY	Managed Identity (zero secrets)
Observability	DIY	OpenTelemetry + Application Insights
One-command deploy	No	`azd up`

Testing locally

The dual-mode architecture means you can test the full pipeline without any cloud resources:

# Create virtual environment and install dependencies python -m venv .venv source .venv/bin/activate pip install -r requirements.txt # Run locally (agents use LangChain tools) python -m src

Then open http://localhost:8080 in your browser to use the built-in web UI, or call the API directly:

curl -X POST http://localhost:8080/triage \ -H "Content-Type: application/json" \ -d '{"message": "Getting 503 errors on /api/orders since 2pm"}'

The response includes a coordinator summary, specialist results with confidence scores, and the tools each agent used.

Running the tests

The project includes a comprehensive test suite covering routing logic, tool behaviour, agent execution, and HTTP endpoints:

curl -X POST http://localhost:8080/triage \ -H "Content-Type: application/json" \ -d '{"message": "Getting 503 errors on /api/orders since 2pm"}'

Tests run entirely in local mode, so no cloud credentials are needed.

Key takeaways for LangChain developers

Keep your LangChain abstractions. The @tool decorator, RunnableLambda chains, and composable pipelines all work exactly as you expect.
Add cloud capabilities incrementally. Start local, then enable Bing Grounding, Code Interpreter, and managed hosting when you are ready.
Use the dual-mode pattern. Every agent should work locally with LangChain tools and on Foundry with the Microsoft Agent Framework. This makes development fast and deployment seamless.
Let azd handle infrastructure. One command provisions everything: containers, identity, monitoring, and model deployments.
Security comes free. Managed Identity means no API keys in your code. Non-root containers, RBAC, and disabled ACR admin are all configured by default.

Get started

Clone the sample repository and try it yourself:

git clone https://github.com/leestott/hosted-agents-langchain-samples cd hosted-agents-langchain-samples python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt python -m src

Open http://localhost:8080 to interact with the copilot through the web UI. When you are ready for production, run azd up and your multi-agent system is live on Microsoft Foundry.

Resources

Build an Offline Hybrid RAG Stack with ONNX and Foundry Local

Lee_Stott — Thu, 26 Mar 2026 07:00:00 GMT

If you are building local AI applications, basic retrieval augmented generation is often only the starting point. This sample shows a more practical pattern: combine lexical retrieval, ONNX based semantic embeddings, and a Foundry Local chat model so the assistant stays grounded, remains offline, and degrades cleanly when the semantic path is unavailable.

Why this sample is worth studying

Many local RAG samples rely on a single retrieval strategy. That is usually enough for a proof of concept, but it breaks down quickly in production. Exact keywords, acronyms, and document codes behave differently from natural language questions and paraphrased requests.

This repository keeps the original lexical retrieval path, adds local ONNX embeddings for semantic search, and fuses both signals in a hybrid ranking mode. The generation step runs through Foundry Local, so the entire assistant can remain on device.

Lexical mode handles exact terms and structured vocabulary.
Semantic mode handles paraphrases and more natural language phrasing.
Hybrid mode combines both and is usually the best default.
Lexical fallback protects the user experience if the embedding pipeline cannot start.

Architectural overview

The sample has two main flows: an offline ingestion pipeline and a local query pipeline.

The architecture splits cleanly into offline ingestion at the top and runtime query handling at the bottom.

Offline ingestion pipeline

Read Markdown files from docs/.
Parse front matter and split each document into overlapping chunks.
Generate dense embeddings when the ONNX model is available.
Store chunks in SQLite with both sparse lexical features and optional dense vectors.

Local query pipeline

The browser posts a question to the Express API.
ChatEngine resolves the requested retrieval mode.
VectorStore retrieves lexical, semantic, or hybrid results.
The prompt is assembled with the retrieved context and sent to a Foundry Local chat model.
The answer is returned with source references and retrieval metadata.

The sequence diagram shows the difference between lexical retrieval and hybrid retrieval. In hybrid mode, the query is embedded first, then lexical and semantic scores are fused before prompt assembly.

Repository structure and core components

The implementation is compact and readable. The main files to understand are listed below.

src/config.js: retrieval defaults, paths, and model settings.
src/embeddingEngine.js: local ONNX embedding generation through Transformers.js.
src/vectorStore.js: SQLite storage plus lexical, semantic, and hybrid ranking.
src/chatEngine.js: retrieval mode resolution, prompt assembly, and Foundry Local model execution.
src/ingest.js: document ingestion and embedding generation during indexing.
src/server.js: REST endpoints, streaming endpoints, upload support, and health reporting.

Getting started

To run the sample, you need Node.js 20 or newer, Foundry Local, and a local ONNX embedding model. The default model path is models/embeddings/bge-small-en-v1.5.

cd c:\Users\leestott\local-hybrid-retrival-onnx npm install huggingface-cli download BAAI/bge-small-en-v1.5 --local-dir models/embeddings/bge-small-en-v1.5 npm run ingest npm start

Ingestion writes the local SQLite database to data/rag.db. If the embedding model is available, each chunk gets a dense vector as well as lexical features. If the embedding model is missing, ingestion still succeeds and the application remains usable in lexical mode.

Best practice: local AI applications should treat model files, SQLite data, and native runtime compatibility as part of the deployable system, not as optional developer conveniences.

Code walkthrough

1. Retrieval configuration

The sample makes its retrieval behaviour explicit in configuration. That is useful for testing and for operator visibility.

export const config = { model: "phi-3.5-mini", docsDir: path.join(ROOT, "docs"), dbPath: path.join(ROOT, "data", "rag.db"), chunkSize: 200, chunkOverlap: 25, topK: 3, retrievalMode: process.env.RETRIEVAL_MODE || "hybrid", retrievalModes: ["lexical", "semantic", "hybrid"], fallbackRetrievalMode: "lexical", retrievalWeights: { lexical: 0.45, semantic: 0.55, }, };

Those defaults tell you a lot about the intended operating profile. Chunks are small, the number of returned chunks is low, and the fallback path is explicit.

2. Local ONNX embeddings

The embedding engine disables remote model loading and only uses local files. That matters for privacy, repeatability, and air gapped operation.

env.allowLocalModels = true; env.allowRemoteModels = false; this.extractor = await pipeline("feature-extraction", resolvedPath, { local_files_only: true, }); const output = await this.extractor(text, { pooling: "mean", normalize: true, });

The mean pooling and normalisation step make the vectors suitable for cosine similarity based ranking.

3. Hybrid storage and ranking in SQLite

Instead of adding a separate vector database, the sample stores lexical and semantic representations in the same SQLite table. That keeps the local footprint low and the implementation easy to debug.

searchHybrid(query, queryEmbedding, topK = 5, weights = { lexical: 0.45, semantic: 0.55 }) { const lexicalResults = this.searchLexical(query, topK * 3); const semanticResults = this.searchSemantic(queryEmbedding, topK * 3); if (semanticResults.length === 0) { return lexicalResults.slice(0, topK).map((row) => ({ ...row, retrievalMode: "lexical", })); } const fused = [...combined.values()].map((row) => ({ ...row, score: (row.lexicalScore * lexicalWeight) + (row.semanticScore * semanticWeight), })); fused.sort((a, b) => b.score - a.score); return fused.slice(0, topK); }

The important point is not just the weighted fusion. It is the fallback behaviour. If semantic retrieval cannot provide results, the user still gets lexical grounding instead of an empty context window.

4. Retrieval mode resolution in ChatEngine

ChatEngine keeps the runtime behaviour predictable. It validates the requested mode and falls back to lexical search when semantic retrieval is unavailable.

resolveRetrievalMode(requestedMode) { const desiredMode = config.retrievalModes.includes(requestedMode) ? requestedMode : config.retrievalMode; if ((desiredMode === "semantic" || desiredMode === "hybrid") && !this.semanticAvailable) { return config.fallbackRetrievalMode; } return desiredMode; }

This is a sensible production design because local runtime failures are common. Missing model files or native dependency mismatches should reduce quality, not crash the entire assistant.

5. Foundry Local model management

The sample uses FoundryLocalManager to discover, download, cache, and load the configured chat model.

This gives the app a better local startup experience. The server can expose a status stream while the model initialises in the background.

User experience and screenshots

The client is intentionally simple, which makes it useful during evaluation. You can switch retrieval mode, test questions quickly, and inspect the retrieved sources.

The landing page exposes retrieval mode directly in the UI. That makes it easy to compare lexical, semantic, and hybrid behaviour during testing.

The sources panel shows grounding evidence and retrieval scores, which is useful when validating whether better answers are coming from better retrieval or just model phrasing.

Best practices for ONNX RAG and Foundry Local

Keep lexical fallback alive. Exact identifiers and runtime failures both make this necessary.
Persist sparse and dense features together where possible. It simplifies debugging and operational reasoning.
Use small chunks and conservative topK values for local context budgets.
Expose health and status endpoints so users can see when the model is still loading or embeddings are unavailable.
Test retrieval quality separately from generation quality.
Pin and validate native runtime dependencies, especially ONNX Runtime, before tuning prompts.

Practical warning: this repository already shows why runtime validation matters. A local app can ingest documents successfully and still fail at model initialisation if the native runtime stack is misaligned.

How this compares with RAG and CAG

The strongest value in this sample comes from where it sits between a basic local RAG baseline and a curated CAG design.

Dimension	Classic local RAG	This hybrid ONNX RAG sample	CAG
Context assembly	Retrieve chunks at query time, often lexically, then inject them into the prompt.	Retrieve chunks at query time with lexical, semantic, or fused scoring, then inject the strongest results into the prompt.	Use a prepared or cached context pack instead of fresh retrieval for every request.
Main strength	Easy to implement and easy to explain.	Better recall for paraphrases without giving up exact match behaviour or offline execution.	Predictable prompts and low query time overhead.
Main weakness	Misses synonyms and natural language reformulations.	More moving parts, larger local asset footprint, and native runtime compatibility to manage.	Coverage depends on curation quality and goes stale more easily.
Failure behaviour	Weak retrieval leads to weak grounding.	Semantic failure can degrade to lexical retrieval if designed properly, which this sample does.	Prepared context can be too narrow for new or unexpected questions.
Best fit	Simple local assistants and proof of concept systems.	Offline copilots and technical assistants that need stronger recall across varied phrasing.	Stable workflows with tightly bounded, curated knowledge.

Samples

Related samples:

- Foundry Local RAG - https://github.com/leestott/local-rag

- Foundry Local CAG - https://github.com/leestott/local-cag

- Foundry Local hybrid-retrival-onnx https://github.com/leestott/local-hybrid-retrival-onnx

Specific benefits of this hybrid approach over classic RAG

It captures paraphrased questions that lexical search would often miss.
It still preserves exact match performance for codes, terms, and product names.
It gives operators a controlled degradation path when the semantic stack is unavailable.
It stays local and inspectable without introducing a separate hosted vector service.

Specific differences from CAG

CAG shifts effort into context curation before the request. This sample retrieves evidence dynamically at runtime.
CAG can be faster for fixed workflows, but it is usually less flexible when the document set changes.
This hybrid RAG design is better suited to open ended knowledge search and growing document collections.

What to validate before shipping

Measure retrieval quality in each mode using exact term, acronym, and paraphrase queries.
Check that sources shown in the UI reflect genuinely distinct evidence, not repeated chunks.
Confirm the application remains usable when semantic retrieval is unavailable.
Verify ONNX Runtime compatibility on the real target machines, not only on the development laptop.
Test model download, cache, and startup behaviour with a clean environment.

Final take

For developers getting started with ONNX RAG and Foundry Local, this sample is a good technical reference because it demonstrates a realistic local architecture rather than a minimal demo. It shows how to build a grounded assistant that remains offline, supports multiple retrieval modes, and fails gracefully.

Compared with classic local RAG, the hybrid design provides better recall and better resilience. Compared with CAG, it remains more flexible for changing document sets and less dependent on pre curated context packs. If you want a practical starting point for offline grounded AI on developer workstations or edge devices, this is the most balanced pattern in the repository set.

Step-by-Step: Deploy the Architecture Review Agent Using AZD AI CLI

ShivamGoyal — Tue, 24 Mar 2026 07:00:00 GMT

Hey everyone! I am Shivam Goyal, a Microsoft MVP, and I am super excited to share a project that will save you a massive amount of time.

Have you ever built a brilliant AI agent in an afternoon, only to spend the next two weeks fighting with Docker containers, memory persistence, and cloud deployment scripts to get it running?

In our previous post, we introduced the Architecture Review Agent, an open-source tool built on Microsoft Foundry that automatically converts messy architectural notes into structured risk assessments and interactive Excalidraw diagrams.

But building an AI agent is only half the battle. Iterating on one and actually getting it running in a production-grade environment without losing your mind over infrastructure is a completely different story.

The Problem with Agentic Development Loops

The typical agent development loop is painful: you write your agent code, test it by copy-pasting inputs into a local REPL, manually build a container, push it to a registry, configure RBAC, deploy to your cloud target, realize you need to tweak three lines of logic, and start the whole cycle over again.

You often end up with an agent that is 100 lines of clean Python, surrounded by 400 lines of Bicep and a 12-step deployment guide.

The azd ai extension for the Azure Developer CLI (AZD) completely changes this equation. For the Architecture Review Agent, the entire workflow, from zero infrastructure to a live hosted agent you can invoke from the command line, is just a few simple commands. And moving from local testing to a live cloud deployment is a single azd up.

Here is how you can set up, invoke, and deploy your own Architecture Review Agent, and even publish it to Microsoft Teams, without needing a tenant admin.

Step 1: The Setup (No heavy lifting required)

First, make sure you have the Azure Developer CLI installed and grab the AI Agents extension.

# Install AZD winget install microsoft.azd # Install the AI Agents extension azd extension install azure.ai.agents

Next, clone the repository and set up your local Python environment:

git clone https://github.com/Azure-Samples/agent-architecture-review-sample cd agent-architecture-review-sample python -m venv .venv .\.venv\Scripts\Activate.ps1 pip install -r requirements.txt

Finally, authenticate and tell AZD where your Microsoft Foundry project lives:

azd auth login azd env new arch-review-dev # Point it to your Foundry Project and Model azd env set AZURE_AI_PROJECT_ENDPOINT "https://<your-resource>.services.ai.azure.com/api/projects/<your-project>" azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME "gpt-4.1"

Step 2: Run and Invoke Locally

With the AZD AI extension, you get a local server that behaves identically to a deployed Foundry-hosted agent. It uses the same localhost:8088 endpoint, the same OpenAI Responses API protocol, and the same conversation persistence.

Open your first terminal and start the runtime:

azd ai agent run

Now, open a second terminal. This is where the magic happens. The agent is completely format-agnostic. There is no schema you have to memorize. You can pass it a file, or just type out a whiteboard brain-dump inline.

Here is what the terminal experience looks like when running these commands and getting the structured report back:

Styled terminal showing all 3 azd ai agent invoke commands + full structured report output

Here are the three ways you can invoke it using "azd ai agent invoke --local":

Pattern A: The Structured YAML

If your team uses formal definitions, just point the agent to the file. The rule-based parser handles this instantly without an LLM call.

azd ai agent invoke --local "scenarios/ecommerce.yaml"

12-component ecommerce architecture diagram generated from YAML input.

Pattern B: The Whiteboard Brain-Dump (Inline Arrow Notation)

Arrow notation (A -> B -> C) is how engineers actually communicate on whiteboards and in Slack. Before now, this wasn't a valid input for architecture tools.

azd ai agent invoke --local "LB -> 3 API servers -> PostgreSQL primary with read replica -> Redis cache"

The parser automatically extracts the replica count, infers the component types (LB becomes a Gateway), and builds a valid connection graph, surfacing single points of failure instantly.

Pattern C: The Markdown Design Doc

Just point it to your existing READMEs or design docs.

azd ai agent invoke --local "scenarios/event_driven.md"

8-component event-driven streaming architecture generated from Markdown input

For all three patterns, the agent returns a structured Markdown report in your terminal and generates an interactive architecture.excalidraw file and a high-res PNG right in your local /output folder.

Step 3: One Command to the Cloud

When you are happy with how your agent performs locally, it's time to deploy. Forget manual Docker builds and complex credential management.

azd up

This single command orchestrates everything:

Provisions Infrastructure: Creates your Foundry AI Services account, ACR, App Insights, and managed identities with proper RBAC.
Builds and Pushes: Packages your Dockerfile and pushes the container image to ACR.
Deploys the Agent: Registers the image and creates a hosted agent version in Foundry Agent Service.

The output will hand you a live Agent Playground URL and a production-ready API endpoint. Your agent now automatically scales from 0 to 5 replicas, manages its own conversation state, and authenticates securely via Managed Identity.

Step 4: Publish to Teams and M365 Copilot (Zero Admin Required!)

Having an API is great, but agents are most powerful when they live where your users collaborate. You can publish this agent directly to Microsoft Teams and M365 Copilot natively from the Foundry portal.

The best part? You can use the Individual Scope.

Go to the Microsoft Foundry portal and find your deployed agent.
Click Publish to Teams and Microsoft 365 Copilot.
Fill out the basic metadata (Name, Description).
Select the Individual scope.

Because you are using an individual scope, no M365 admin approval is required. The portal automatically provisions the Azure Bot Service, packages the metadata, and registers the app. Within minutes, your agent will appear in your Teams Copilot agent store. You can generate a share link and instantly send it to your team for a workshop or demo.

What I Learned Building This Workflow

Shifting from custom deployment scripts to the azd ai CLI taught me three things:

The declarative contract is beautifully clean. Our azure.yaml declares the agent and infrastructure in about 30 lines. azd up translates that into a fully secure, production-grade Foundry environment.
The local-to-cloud gap is finally gone. The azd ai agent run behaves exactly like the cloud. The invocation you write locally works identically against the deployed endpoint.
Teams publishing is remarkably simple. I expected bot registration nightmares and tenant admin blockers. Instead, I filled out a form, waited two minutes, and was chatting with my architecture agent in Teams.

Resources & Next Steps

Now that we have a streamlined, single-hosted agent deployment, the natural next step is multi-agent orchestration. Imagine a triage agent that routes your design doc to a dedicated Security Reviewer Agent and a Scalability Reviewer Agent.

Try it out yourself by cloning the repository, running azd up, and let me know what you build!

GitHub Repository: Azure-Samples/agent-architecture-review-sample
Previous Article: Stop Drawing Architecture Diagrams Manually: Meet the Open-Source AI Architecture Review Agents | Microsoft Community Hub
Microsoft Learn: Install the Azure Developer CLI
Microsoft Foundry Documentation: Hosted agents in Foundry Agent Service (preview) - Microsoft Foundry

Microsoft Olive & Olive Recipes: A Practical Guide to Model Optimization for Real-World Deployment

Lee_Stott — Mon, 23 Mar 2026 07:00:00 GMT

Why your model runs great on your laptop but fails in the real world

You have trained a model. It scores well on your test set. It runs fine on your development machine with a beefy GPU. Then someone asks you to deploy it to a customer's edge device, a cloud endpoint with a latency budget, or a laptop with no discrete GPU at all.

Suddenly the model is too large, too slow, or simply incompatible with the target runtime. You start searching for quantisation scripts, conversion tools, and hardware-specific compiler flags. Each target needs a different recipe, and the optimisation steps interact in ways that are hard to predict.

This is the deployment gap. It is not a knowledge gap; it is a tooling gap. And it is exactly the problem that Microsoft Olive is designed to close.

What is Olive?

Olive is an easy-to-use, hardware-aware model optimisation toolchain that composes techniques across model compression, optimisation, and compilation. Rather than asking you to string together separate conversion scripts, quantisation utilities, and compiler passes by hand, Olive lets you describe what you have and what you need, then handles the pipeline.

In practical terms, Olive takes a model source, such as a PyTorch model or an ONNX model (and other supported formats), plus a configuration that describes your production requirements and target hardware accelerator. It then runs the appropriate optimisation passes and produces a deployment-ready artefact.

You can think of it as a build system for model optimisation: you declare the intent, and Olive figures out the steps.

Official repo: github.com/microsoft/olive
Documentation: microsoft.github.io/Olive

Key advantages: why Olive matters for your workflow

A. Optimise once, deploy across many targets

One of the hardest parts of deploying models in production is that "production" is not one thing. Your model might need to run on a cloud GPU, an edge CPU, or a Windows device with an NPU. Each target has different memory constraints, instruction sets, and runtime expectations.

Olive supports targeting CPU, GPU, and NPU through its optimisation workflow. This means a single toolchain can produce optimised artefacts for multiple deployment targets, expanding the number of platforms you can serve without maintaining separate optimisation scripts for each one.

The conceptual workflow is straightforward: Olive can download, convert, quantise, and optimise a model using an auto-optimisation style approach where you specify the target device (cpu, gpu, or npu). This keeps the developer experience consistent even as the underlying optimisation strategy changes per target.

B. ONNX as the portability layer

If you have heard of ONNX but have not used it in anger, here is why it matters: ONNX gives your model a common representation that multiple runtimes understand. Instead of being locked to one framework's inference path, an ONNX model can run through ONNX Runtime and take advantage of whatever hardware is available.

Olive supports ONNX conversion and optimisation, and can generate a deployment-ready model package along with sample inference code in languages like C#, C++, or Python. That package is not just the model weights; it includes the configuration and code needed to load and run the model on the target platform.

For students and early-career engineers, this is a meaningful capability: you can train in PyTorch (the ecosystem you already know) and deploy through ONNX Runtime (the ecosystem your production environment needs).

C. Hardware-specific acceleration and execution providers

When Olive targets a specific device, it does not just convert the model format. It optimises for the execution provider (EP) that will actually run the model on that hardware. Execution providers are the bridge between the ONNX Runtime and the underlying accelerator.

Olive can optimise for a range of execution providers, including:

Vitis AI EP (AMD) – for AMD accelerator hardware
OpenVINO EP (Intel) – for Intel CPUs, integrated GPUs, and VPUs
QNN EP (Qualcomm) – for Qualcomm NPUs and SoCs
DirectML EP (Windows) – for broad GPU support on Windows devices

Why does EP targeting matter? Because the difference between a generic model and one optimised for a specific execution provider can be significant in terms of latency, throughput, and power efficiency. On battery-powered devices especially, the right EP optimisation can be the difference between a model that is practical and one that drains the battery in minutes.

D. Quantisation and precision options

Quantisation is one of the most powerful levers you have for making models smaller and faster. The core idea is reducing the numerical precision of model weights and activations:

FP32 (32-bit floating point) – full precision, largest model size, highest fidelity
FP16 (16-bit floating point) – roughly half the memory, usually minimal quality loss for most tasks
INT8 (8-bit integer) – significant size and speed gains, moderate risk of quality degradation depending on the model
INT4 (4-bit integer) – aggressive compression for the most constrained deployment scenarios

Think of these as a spectrum. As you move from FP32 towards INT4, models get smaller and faster, but you trade away some numerical fidelity. The practical question is always: how much quality can I afford to lose for this use case?

Practical heuristics for choosing precision:

FP16 is often a safe default for GPU deployment. In practice, you might start here and only go lower if you need to.
INT8 is a strong choice for CPU-based inference where memory and compute are constrained but accuracy requirements are still high (e.g., classification, embeddings, many NLP tasks).
INT4 is worth exploring when you are deploying large language models to edge or consumer devices and need aggressive size reduction. Expect to validate quality carefully, as some tasks and model architectures tolerate INT4 better than others.

Olive handles the mechanics of applying these quantisation passes as part of the optimisation pipeline, so you do not need to write custom quantisation scripts from scratch.

Showcase: model conversion stories

To make this concrete, here are three plausible optimisation scenarios that illustrate how Olive fits into real workflows.

Story 1: PyTorch classification model → ONNX → quantised for cloud CPU inference

Starting point: A PyTorch image classification model fine-tuned on a domain-specific dataset.
Target hardware: Cloud CPU instances (no GPU budget for inference).
Optimisation intent: Reduce latency and cost by quantising to INT8 whilst keeping accuracy within acceptable bounds.
Output: An ONNX model optimised for CPU execution, packaged with configuration and sample inference code ready for deployment behind an API endpoint.

Story 2: Hugging Face language model → optimised for edge NPU

Starting point: A Hugging Face transformer model used for text summarisation.
Target hardware: A laptop with an integrated NPU (e.g., a Qualcomm-based device).
Optimisation intent: Shrink the model to INT4 to fit within NPU memory limits, and optimise for the QNN execution provider to leverage the neural processing unit.
Output: A quantised ONNX model configured for QNN EP, with packaging that includes the model, runtime configuration, and sample code for local inference.

Story 3: Same model, two targets – GPU vs. NPU

Starting point: A single PyTorch generative model used for content drafting.
Target hardware: (A) Cloud GPU for batch processing, (B) On-device NPU for interactive use.
Optimisation intent: For GPU, optimise at FP16 for throughput. For NPU, quantise to INT4 for size and power efficiency.
Output: Two separate optimised packages from the same source model, one targeting DirectML EP for GPU, one targeting QNN EP for NPU, each with appropriate precision, runtime configuration, and sample inference code.

In each case, Olive handles the multi-step pipeline: conversion, optimisation passes, quantisation, and packaging. The developer's job is to define the target and validate the output quality.

Introducing Olive Recipes

If you are new to model optimisation, staring at a blank configuration file can be intimidating. That is where Olive Recipes comes in.

The Olive Recipes repository complements Olive by providing recipes that demonstrate features and use cases. You can use them as a reference for optimising publicly available models or adapt them for your own proprietary models. The repository also includes a selection of ONNX-optimised models that you can study or use as starting points.

Think of recipes as worked examples: each one shows a complete optimisation pipeline for a specific scenario, including the configuration, the target hardware, and the expected output. Instead of reinventing the pipeline from scratch, you can find a recipe close to your use case and modify it.

For students especially, recipes are a fast way to learn what good optimisation configurations look like in practice.

Taking it further: adding custom models to Foundry Local

Once you have optimised a model with Olive, you may want to serve it locally for development, testing, or fully offline use. Foundry Local is a lightweight runtime that downloads, manages, and serves language models entirely on-device via an OpenAI-compatible API, with no cloud dependency and no API keys required.

Important: Foundry Local only supports specific model templates. At present, these are the chat template (for conversational and text-generation models) and the whisper template (for speech-to-text models based on the Whisper architecture). If your model does not fit one of these two templates, it cannot currently be loaded into Foundry Local.

Compiling a Hugging Face model for Foundry Local

If your optimised model uses a supported architecture, you can compile it from Hugging Face for use with Foundry Local. The high-level process is:

Choose a compatible Hugging Face model. The model must match one of Foundry Local's supported templates (chat or whisper). For chat models, this typically means decoder-only transformer architectures that support the standard chat format.
Use Olive to convert and optimise. Olive handles the conversion from the Hugging Face source format into an ONNX-based, quantised artefact that Foundry Local can serve. This is where your Olive skills directly apply.
Register the model with Foundry Local. Once compiled, you register the model so that Foundry Local's catalogue recognises it and can serve it through the local API.

For the full step-by-step guide, including exact commands and configuration details, refer to the official documentation: How to compile Hugging Face models for Foundry Local. For a hands-on lab that walks through the complete workflow, see Foundry Local Lab, specifically Lab 10 which covers bringing custom models into Foundry Local.

Why does this matter?

The combination of Olive and Foundry Local gives you a complete local workflow: optimise your model with Olive, then serve it with Foundry Local for rapid iteration, privacy-sensitive workloads, or environments without internet connectivity. Because Foundry Local exposes an OpenAI-compatible API, your application code can switch between local and cloud inference with minimal changes.

Keep in mind the template constraint. If you are planning to bring a custom model into Foundry Local, verify early that it fits the chat or whisper template. Attempting to load an unsupported architecture will not work, regardless of how well the model has been optimised.

Contributing: how to get involved

The Olive ecosystem is open source, and contributions are welcome. There are two main ways to contribute:

A. Contributing recipes

If you have built an optimisation pipeline that works well for a specific model, hardware target, or use case, consider contributing it as a recipe. Recipes are repeatable pipeline configurations that others can learn from and adapt.

B. Sharing optimised model outputs and configurations

If you have produced an optimised model that might be useful to others, sharing the optimisation configuration and methodology (and, where licensing permits, the model itself) helps the community build on proven approaches rather than starting from zero.

Contribution checklist

Reproducibility: Can someone else run your recipe or configuration and get comparable results?
Licensing: Are the base model weights, datasets, and any dependencies properly licensed for sharing?
Hardware target documented: Have you specified which device and execution provider the optimisation targets?
Runtime documented: Have you noted the ONNX Runtime version and any EP-specific requirements?
Quality validation: Have you included at least a basic accuracy or quality check for the optimised output?

If you are a student or early-career developer, contributing a recipe is a great way to build portfolio evidence that you understand real deployment concerns, not just training.

Try it yourself: a minimal workflow

Here is a conceptual walkthrough of the optimisation workflow using Olive. The idea is to make the mental model concrete. For exact CLI flags and options, refer to the official Olive documentation.

Choose a model source. Start with a PyTorch or Hugging Face model you want to optimise. This is your input.
Choose a target device. Decide where the model will run: cpu, gpu, or npu.
Choose an execution provider. Pick the EP that matches your hardware, for example DirectML for Windows GPU, QNN for Qualcomm NPU, or OpenVINO for Intel.
Choose a precision. Select the quantisation level: fp16, int8, or int4, based on your size, speed, and quality requirements.
Run the optimisation. Olive will convert, quantise, optimise, and package the model for your target. The output is a deployment-ready artefact with model files, configuration, and sample inference code.

A conceptual command might look like this:

# Conceptual example – refer to official docs for exact syntax
olive auto-opt --model-id my-model --device cpu --provider onnxruntime --precision int8

After optimisation, validate the output. Run your evaluation benchmark on the optimised model and compare quality, latency, and model size against the original. If INT8 drops quality below your threshold, try FP16. If the model is still too large for your device, explore INT4. Iteration is expected.

Key takeaways

Olive bridges training and deployment by providing a single, hardware-aware optimisation toolchain that handles conversion, quantisation, optimisation, and packaging.
One source model, many targets: Olive lets you optimise the same model for CPU, GPU, and NPU, expanding your deployment reach without maintaining separate pipelines.
ONNX is the portability layer that decouples your training framework from your inference runtime, and Olive leverages it to generate deployment-ready packages.
Precision is a design choice: FP16, INT8, and INT4 each serve different deployment constraints. Start conservative, measure quality, and compress further only when needed.
Olive Recipes are your starting point: Do not build optimisation pipelines from scratch when worked examples exist. Learn from recipes, adapt them, and contribute your own.
Foundry Local extends the workflow: Once your model is optimised, Foundry Local can serve it on-device via a standard API, but only if it fits a supported template (chat or whisper).

Resources

ProvePresent: Ending Proxy Attendance with Azure Serverless & Azure OpenAI

cyruswong — Thu, 19 Mar 2026 07:00:00 GMT

Problem

Most schools use a smart‑card‑based attendance system where students tap their cards on a reader. However, this method is unreliable because students can give their cards to friends or simply tap and leave immediately. Teachers cannot accurately assess real student performance—whether high‑performing students are genuinely attending class or whether poor performance is due to actual absence. Another issue is that even if students are physically present in a lecture, teachers still cannot tell whether they are paying attention to the projector or actually learning.

The current workaround is for teachers to override the attendance record by calling each student one by one, which is time‑consuming in large lectures and adds little educational value. It is also only a one‑time check, meaning students can still leave the lecture room immediately afterwards.

Another issue is that we have many out‑of‑school activities such as site visit, and the school needs to ensure everyone’s presence promptly in each check point.

This kind of problem isn’t unique to schools. It’s a common challenge for event organizers, where verifying attendee presence is essential but often slow, causing long queues. Organizers usually rely on a few mobile scanners to check in attendees one by one.

Solution

ProvePresent is an AI tool designed to verify attendance and create real‑time challenges for participants, ensuring that attendance records are authentic and that attendees remain focused on the presentation. It uses OTP login with school email.

Check-in and Check-out With a Real‑time QR Code

The code refreshes every 25 seconds, and the presenter can display it on the projector for everyone to scan when checking in at the beginning and checking out at the end of the session.

However, this alone cannot prevent someone from capturing the code and sending it to others who are not in the room, or from using two devices to help someone else scan for attendance—even if geolocation checks are enabled. We will explain this next.

This check‑in and check‑out process is highly scalable, and no one needs to queue while waiting for someone to scan their QR code!

Organizers can set geolocation restrictions to prevent anyone from checking in remotely in a simple manner.

Keep Attendee Alive with Signalr

The SignalR live connection allows the presenter to create real‑time challenges for attendees, helping to verify their presence and ensure they are genuinely focused on the presentation.

AI Powered Live Quiz

The presenter shares their presentation screen, and two Microsoft Foundry agents with Azure OpenAI Chatgpt 5.3 —ImageAnalysisAgent, which extracts key information from the shared screen, and QuizQuestionGenerator, which generates simple questions based on the current slide—work together to create challenges. The question is broadcast to all online attendees, who must answer within 20 seconds.

This feature keeps attendees on the webpage and prevents them from doing anything unrelated to the presentation.

Left is attendee view and right is presenter view before screen capture.Presentation ScreenLeft is attendee view and right is presenter view during screen capture.

Detailed report can be downloaded for further analysis.

Download report

A complete Demo

Attendee Photo Capture

Request all online students to capture and upload photos of their venue view. The system will analyze the images to estimate seating positions using Microsoft Foundry agents with Azure OpenAI ChatGPT 5.3 PositionEstimationAgent and complete an image challenge.

When the presenter clicks Capture Attendee Photos, all online attendees are prompted to take a photo and upload it to blob storage. The PositionEstimationAgent then analyzes the image to estimate their seating location, which can provide insights into student performance.

Analysis Notes: Analyzed 13 students in 2 overlapping batches. Batch 1: The venue is a computer lab with the projector screen at the front center, whiteboards on the left, and cabinets on the right. Relative depth was estimated mainly from screen size and number of monitor rows visible ahead. Column estimates were inferred from screen angle and side-room features, with lower confidence for the rotated side-view image. Batch 2: These six photos appear to come from the same computer lab with the projector at the front center. Relative depth was estimated mainly from projector size and number of visible desk/monitor rows ahead. Left-right placement was inferred from projector skew and side-wall visibility. Within this batch, 240124734 and 240167285 seem closest to the front, 240286514 and 240158424 are slightly farther back, 240293498 is farther back again, and 240160364 appears furthest.

Pass around the QR code attendance sheet

Traditionally, the attendance sheet is circulated for attendees to sign, but this method is unreliable because no one monitors the signing process, allowing one attendee to sign for someone who is absent. It is also slow and not scalable for large groups.

The QR Code attendance sheet functions as a chain. The presenter randomly distributes a short‑lived, one‑time QR code—representing a virtual attendance sheet—to any number of attendees, just like handing out multiple physical sheets. Each attendee must find another participant to scan their code to record attendance, continuing the chain until the final group of attendees. The presenter then verifies the last group’s presence.

The first chain is a dead chain because that student left the venue and cannot find another student to scan his QR code. The second chain contains 20 student attendance records. It also provides useful insights into their friendship and seating patterns.

Architecture

This project is built using Vibe Coding, so we will not share highly technical details in this post. If you'd like to learn more, leave a comment, and we will write another blog to cover the specifics.

GitHub Repo

https://github.com/wongcyrus/ProvePresent

Conclusion

ProvePresent demonstrates how Azure serverless technology and Azure OpenAI can work together to solve a long‑standing problem in education: verifying genuine student presence and engagement. By combining real‑time QR code verification, SignalR‑powered live interactions, AI‑generated quizzes, and intelligent photo‑based seating analysis, we created a system where “being present” is no longer just a checkbox—it becomes a verifiable, interactive, and meaningful part of the learning experience.

Instead of relying on outdated smart‑card systems or manual roll calls, educators gain a dynamic tool that keeps students attentive, provides insight into classroom behavior, and produces useful analytics for improving teaching outcomes. Students, in turn, benefit from an engaging, modern attendance experience that aligns with how digital‑native learners expect classes to operate.

This is only the beginning. With Microsoft Foundry agents and the flexibility of Azure Functions, there are many opportunities to extend ProvePresent further—richer analytics, smarter engagement models, and seamless integration with LMS platforms. If there’s interest, we’re happy to share more technical details, architectural deep dives, and future roadmap ideas in a follow‑up post.

Thank you for the contribution of Microsoft Student Ambassadors Hong Kong Institute of Information Technology (HKIIT) Wong Wing Ho, CHAN Sham Jayson, Pang Ho Shum, and Chan Ka Chun. They are major in Higher Diploma in Cloud and Data Centre Administration.

About the Author

Cyrus Wong is the senior lecturer of Hong Kong Institute of Information Technology (HKIIT) @ IVE(Lee Wai Lee).and he focuses on teaching public Cloud technologies. He is a passionate advocate for the adoption of cloud technology across various media and events. With his extensive knowledge and expertise, he has earned prestigious recognitions such as AWS Builder Center, Microsoft MVP- Microsoft Foundry, and Google Developer Expert for Google Cloud Platform & AI.

Foundry IQ: Give Your AI Agents a Knowledge Upgrade

Lee_Stott — Tue, 17 Mar 2026 07:00:00 GMT

If you’re learning to build AI agents, you’ve probably hit a familiar wall: your agent can generate text, but it doesn’t actually know anything about your data. It can’t look up your documents, search across your files, or pull facts from multiple sources to answer a real question.

That’s the gap Foundry IQ fills. It gives your AI agents structured access to knowledge, so they can retrieve, reason over, and synthesize information from real data sources instead of relying on what’s baked into the model.

Why Should You Care?

As a student or early-career developer, understanding how AI systems work with external knowledge is one of the most valuable skills you can build right now. Retrieval-Augmented Generation (RAG), knowledge bases, and multi-source querying are at the core of every production AI application, from customer support bots to research assistants to enterprise copilots.

Foundry IQ gives you a hands-on way to learn these patterns without having to build all the plumbing yourself. You define knowledge bases, connect data sources, and let your agents query them. The concepts you learn here transfer directly to real-world AI engineering roles.

What is Foundry IQ?

Foundry IQ is a service within Azure AI Foundry that lets you create knowledge bases, collections of connected data sources that your AI agents can query through a single endpoint.

Instead of writing custom retrieval logic for every app you build, you:

Define knowledge sources — connect documents, data stores, or web content (SharePoint, Azure Blob Storage, Azure AI Search, Fabric OneLake, and more).
Organize them into a knowledge base — group multiple sources behind one queryable endpoint.
Query from your agent — your AI agent calls the knowledge base to get the context it needs before generating a response.

This approach means the knowledge layer is reusable. Build it once, and any agent or app in your project can tap into it.

The IQ Series: A Three-Part Learning Path

The IQ Series is a set of three weekly episodes that walk you through Foundry IQ from concept to code. Each episode includes a tech talk, visual doodle summaries, and a companion cookbook with sample code you can run yourself.

👉 Get started: https://aka.ms/iq-series

Episode 1: Unlocking Knowledge for Your Agents (March 18, 2026)

Start here. This episode introduces the core architecture of Foundry IQ and explains how AI agents interact with knowledge. You’ll learn what knowledge bases are, why they matter, and how the key components fit together.

What you’ll learn:

The difference between model knowledge and retrieved knowledge
How Foundry IQ structures the retrieval layer
The building blocks: knowledge sources, knowledge bases, and agent queries

Episode 2: Building the Data Pipeline with Knowledge Sources (March 25, 2026)

This episode goes deeper into knowledge sources, the connectors that bring data into Foundry IQ. You’ll see how different content types flow into the system and how to wire up sources from services you may already be using.

What you’ll learn:

How to connect sources like Azure Blob Storage, Azure AI Search, SharePoint, Fabric OneLake, and the web
How content is ingested and indexed for retrieval
Patterns for combining multiple source types

Episode 3: Querying Multi-Source Knowledge Bases (April 1, 2026)

The final episode shows you how to bring it all together. You’ll learn how agents query across multiple knowledge sources through a single knowledge base endpoint and how to synthesize answers from diverse data.

What you’ll learn:

How to query a knowledge base from your agent code
How retrieval works across multiple connected sources
Techniques for synthesizing information to answer complex questions

Get Hands-On with the Cookbooks

Every episode comes with a companion cookbook in the GitHub repo, complete with sample code you can clone, run, and modify. This is the fastest way to go from watching to building.

👉 Explore the repo: https://aka.ms/iq-series

Inside you’ll find:

Episode links — watch the tech talks and doodle recaps
Cookbooks — step-by-step code samples for each episode
Documentation links — official Foundry IQ docs and additional learning resources

What to Build Next

Once you’ve worked through the series, try applying what you’ve learned:

Study assistant — connect your course materials as knowledge sources and build an agent that can answer questions across all your notes and readings.
Project documentation bot — index your team’s project docs and READMEs into a knowledge base so everyone can query them naturally.
Research synthesizer — connect multiple data sources (papers, web content, datasets) and build an agent that can cross-reference and summarize findings.

Start Learning

The IQ Series is designed to take you from zero to building knowledge-driven AI agents. Watch the episodes, run the cookbooks, and start experimenting with your own knowledge bases.

👉 https://aka.ms/iq-series

Microsoft Foundry Model Router: A Developer's Guide to Smarter AI Routing

Lee_Stott — Mon, 23 Mar 2026 15:17:48 GMT

Introduction

When building AI-powered applications on Azure, one of the most impactful decisions you make isn't about which model to use, it's about how your application selects models at runtime. Microsoft Foundry Model Router, available through Microsoft Foundry, automatically routes your inference requests to the best available model based on prompt complexity, latency targets, and cost efficiency. But how do you know it's actually routing correctly? And how do you compare its behavior across different API paths?

That's exactly the problem RouteLens solves. It's an open-source Node.js CLI and web-based testing tool that sends configurable prompts through two distinct Azure AI runtime paths and produces a detailed comparison of routing decisions, latency profiles, and reliability metrics.

In this post, we'll walk through what Model Router does, why it matters, how to use the validator tool, and best practices for designing applications that get the most out of intelligent model routing.

What Is Microsoft Founry Model Router?

Microsoft Foundry Model Router is a deployment option in Microsoft Foundry that sits between your application and a pool of AI models. Instead of hard-coding a specific model like gpt-4o or gpt-4o-mini, you deploy a Model Router endpoint and let Azure decide which underlying model serves each request.

How It Works

Your application sends an inference request to the Model Router deployment.
Model Router analyzes the request (prompt complexity, token count, required capabilities).
It selects the most appropriate model from the available pool.
The response is returned transparently — your application code doesn't change.

Why This Matters

Cost optimization — Simple prompts get routed to smaller, cheaper models. Complex prompts go to more capable (and expensive) ones.
Latency reduction — Lightweight prompts complete faster when they don't need a heavyweight model.
Resilience — If one model is experiencing high load or throttling, traffic can shift to alternatives.
Simplified application code — No need to build your own model-selection logic.

The Two Runtime Paths

Microsoft Foundry offers two distinct endpoint configurations for hitting Model Router. Even though both use the Chat Completions API, they may have different routing behaviour:

Path	SDK	Endpoint
AOAI + Chat Completions	OpenAI JS SDK	`https://.cognitiveservices.azure.com/openai/deployments/`
Foundry Project + Chat Completions	OpenAI JS SDK (separate client)	`https://.cognitiveservices.azure.com/openai/deployments/`

Understanding whether these two paths produce the same routing decisions is critical for production applications. If the same prompt routes to different models depending on which endpoint you use, that's a signal you need to investigate.

Introducing RouteLens

RouteLens is a Node.js tool that automates this comparison. It:

Sends a configurable set of prompts across categories (echo, summarize, code, reasoning) through both paths.
Logs every response to structured JSONL files for post-hoc analysis.
Computes statistics including p50/p95 latency, error rates, and model-choice distribution.
Highlights routing differences — where the same prompt was served by different models across paths.
Provides a web dashboard for interactive testing and real-time result visualization.

The Web Dashboard

The built-in web UI makes it easy to run tests and explore results without parsing log files:

The dashboard includes:

KPI Dashboard — Key metrics at a glance: Success Rate, Avg TPS, Gen TPS, Peak TPS, Fastest Response, p50/p95 Latency, Most Reliable Path, Total Tokens
Summary view — Per-path/per-category stats with success rate, TPS, and latency
Model Comparison — Side-by-side view of which models were selected by each path
Latency Charts — Visual bar charts comparing p50 and p95 latencies
Error Analysis — Error distribution and detailed error messages
Live Feed — Real-time streaming of results as they come in
Log Viewer — Browse and inspect historical JSONL log files

Model Comparison — See which models were selected by each routing path for every prompt:

Live Feed — Real-time streaming of results as they come in:

Log Viewer — Browse and inspect historical JSONL log files with parsed table views:

Mobile Responsive — The UI adapts to smaller screens:

Getting Started

Prerequisites

Node.js 18+ (LTS recommended)
An Azure subscription with a Foundry project
Model Router deployed in your Foundry project
An API key from your Azure OpenAI / Foundry resource
The API version (e.g. 2024-05-01-preview)

Setup

# Clone and install git clone https://github.com/leestott/modelrouter-routelens/ cd modelrouter-routelens npm install # Configure your endpoints cp .env.example .env # Edit .env with your Azure endpoints (see below)

Configuration

The .env file needs these key settings:

# Your Foundry / Cognitive Services deployment endpoint # Format: https://<resource>.cognitiveservices.azure.com/openai/deployments/<deployment> # Do NOT include /chat/completions or ?api-version FOUNDRY_PROJECT_ENDPOINT=https://<resource>.cognitiveservices.azure.com/openai/deployments/model-router AOAI_BASE_URL=https://<resource>.cognitiveservices.azure.com/openai/deployments/model-router # API key from your Azure OpenAI / Foundry resource AOAI_API_KEY=your-api-key-here # Azure OpenAI API version AOAI_API_VERSION=2024-05-01-preview</resource></resource></deployment></resource>

Running Tests

# Full test matrix — sends all prompts through both paths npm run run:matrix # 408 timeout diagnostic — focuses on the Responses path timeout issue npm run run:repro408 # Web UI — interactive dashboard npm run ui # Then open http://localhost:3002 (or the port set in UI_PORT)

Understanding the Results

Latency Comparison

The latency charts show p50 (median) and p95 (tail) latency for each path and prompt category:

Key things to look for:

Large p50 differences between paths suggest one path has consistently higher overhead.
High p95 values indicate tail latency problems — possibly timeouts or retries.
Category-specific patterns — If code prompts are slow on one path but fast on another, that's a routing difference worth investigating.

Model Comparison

The model comparison view shows which models were selected for each prompt:

When both paths select the same model, you see a green "Match" indicator. When they differ, it's flagged in red — these are the cases you want to investigate.

Error Analysis

The errors view helps diagnose reliability issues:

Common error patterns:

408 Timeout — The Responses path may take longer for certain prompt categories
401 Unauthorized — Authentication configuration issues
429 Rate Limited — You're hitting throughput limits
500 Internal Server Error — Backend model issues

Best Practices for Designing Applications with Model Router

1. Design Prompts with Routing in Mind

Model Router makes decisions based on prompt characteristics. To get the best routing:

Keep prompts focused — A clear, single-purpose prompt is easier for the router to classify than a multi-part prompt that spans multiple complexity levels.
Use system messages effectively — A well-structured system message helps the router understand the task complexity.
Separate complex chains — If you have a multi-step workflow, make each step a separate API call rather than one massive prompt. This lets the router use a cheaper model for simple steps.

2. Set Appropriate Timeouts

Different models have different latency profiles. Your timeout settings should account for the slowest model the router might select:

// Too aggressive — may timeout when routed to a larger model const TIMEOUT = 5000; // 5s // Better — allows headroom for model variation const TIMEOUT = 30000; // 30s // Best — use different timeouts based on expected complexity function getTimeout(category) { switch (category) { case 'echo': return 10000; case 'summarize': return 20000; case 'code': return 45000; case 'reasoning': return 60000; default: return 30000; } }

3. Implement Robust Retry Logic

Because the router may select different models on retry, transient failures can resolve themselves:

async function callWithRetry(prompt, maxRetries = 3) { for (let attempt = 0; attempt < maxRetries; attempt++) { try { return await client.chat.completions.create({ model: 'model-router', messages: [{ role: 'user', content: prompt }], }); } catch (err) { if (attempt === maxRetries - 1) throw err; // Exponential backoff await new Promise(r => setTimeout(r, 1000 * Math.pow(2, attempt))); } } }

4. Monitor Model Selection in Production

Log which model was selected for each request so you can track routing patterns over time:

5. Use the Right API Path for Your Use Case

Based on our testing with RouteLens, consider:

Chat Completions path — The standard path for chat-style interactions. Uses the openai SDK directly.
Foundry Project path — Uses the same Chat Completions API but through the Foundry project endpoint. Useful for comparing routing behaviour across different endpoint configurations.

Note: The Responses API (/responses) is not currently available on cognitiveservices.azure.com Model Router deployments. Both paths in RouteLens use Chat Completions.

6. Test Before You Ship

Run RouteLens as part of your pre-production validation:

# In your CI/CD pipeline or pre-deployment check npm run run:matrix -- --runs 10 --concurrency 4

This helps you:

Catch routing regressions when Azure updates model pools
Verify that your prompt changes don't cause unexpected model selection shifts
Establish latency baselines for alerting

Architecture Overview

RouteLens sends configurable prompts through two distinct Azure AI runtime paths and compares routing decisions, latency, and reliability. The Matrix Runner dispatches prompts to both the Chat Completions Client (OpenAI JS SDK → AOAI endpoint) and the Project Responses Client (azure/ai-projects → Foundry endpoint). Both paths converge at Azure Model Router, which intelligently selects the optimal backend model. Results are logged to JSONL files and rendered in the web dashboard.

Key Benefits of Model Router

Benefit	Description
Cost savings	Automatically routes simple prompts to cheaper models, reducing spend by 30-50% in typical workloads
Lower latency	Simple prompts complete faster on lightweight models
Zero code changes	Same API contract as a standard model deployment — just change the deployment name
Future-proof	As Azure adds new models to the pool, your application benefits automatically
Built-in resilience	Routing adapts to model availability and load conditions

Conclusion

Azure Model Router represents a shift from "pick a model" to "describe your task and let the platform decide." This is a natural evolution for AI applications — just as cloud platforms abstract away server selection, Model Router abstracts away model selection.

RouteLens gives you the visibility to trust that abstraction. By systematically comparing routing behavior across API paths and prompt categories, you can deploy Model Router with confidence and catch issues before your users do.

The tool is open source under the MIT license. Try it out, file issues, and contribute improvements:

Build a Fully Offline RAG App with Foundry Local: No Cloud Required

Lee_Stott — Tue, 10 Mar 2026 07:00:00 GMT

A practical guide to building an on-device AI support agent using Retrieval-Augmented Generation, JavaScript, and Microsoft Foundry Local.

The Problem: AI That Can't Go Offline

Most AI-powered applications today are firmly tethered to the cloud. They assume stable internet, low-latency API calls, and the comfort of a managed endpoint. But what happens when your users are in an environment with zero connectivity a gas pipeline in a remote field, a factory floor, an underground facility?

That's exactly the scenario that motivated this project: a fully offline RAG-powered support agent that runs entirely on a laptop. No cloud. No API keys. No outbound network calls. Just a local model, a local vector store, and domain-specific documents all accessible from a browser on any device.

The Gas Field Support Agent - running entirely on-device

What is RAG and Why Should You Care?

Retrieval-Augmented Generation (RAG) is a pattern that makes language models genuinely useful for domain-specific tasks. Instead of hoping the model "knows" the answer from pre-training, you:

Retrieve relevant chunks from your own documents
Augment the model's prompt with those chunks as context
Generate a response grounded in your actual data

The result: fewer hallucinations, traceable answers, and an AI that works with your content. If you're building internal tools, customer support bots, field manuals, or knowledge bases, RAG is the pattern you want.

Why fully offline? Data sovereignty, air-gapped environments, field operations, latency-sensitive workflows, and regulatory constraints all demand AI that doesn't phone home. Running everything locally gives you complete control over your data and eliminates any external dependency.

The Tech Stack

This project is deliberately simple — no frameworks, no build steps, no Docker:

Layer	Technology	Why
AI Model	Foundry Local + Phi-3.5 Mini	Runs locally, OpenAI-compatible API, no GPU needed
Backend	Node.js + Express	Lightweight, fast, universally known
Vector Store	SQLite via `better-sqlite3`	Zero infrastructure, single file on disk
Retrieval	TF-IDF + cosine similarity	No embedding model required, fully offline
Frontend	Single HTML file with inline CSS	No build step, mobile-responsive, field-ready

The total dependency footprint is just four npm packages: express, openai, foundry-local-sdk, and better-sqlite3.

Architecture Overview

The system has five layers — all running on a single machine:

Five-layer architecture: Client → Server → RAG Pipeline → Data → AI Model

Client Layer — A single HTML file served by Express, with quick-action buttons and responsive chat
Server Layer — Express.js handles API routes for chat (streaming + non-streaming), document upload, and health checks
RAG Pipeline — The chat engine orchestrates retrieval and generation; the chunker handles TF-IDF vectorization
Data Layer — SQLite stores document chunks and their TF-IDF vectors; source docs live as .md files
AI Layer — Foundry Local runs Phi-3.5 Mini Instruct on CPU/NPU, exposing an OpenAI-compatible API

Getting Started in 5 Minutes

You need two prerequisites:

Node.js 20+ — nodejs.org
Foundry Local — Microsoft's on-device AI runtime:

Terminal

winget install Microsoft.FoundryLocal

Then clone, install, ingest, and run:

git clone https://github.com/leestott/local-rag.git cd local-rag npm install npm run ingest # Index the 20 gas engineering documents npm start # Start the server + Foundry Local

Open http://127.0.0.1:3000 and start chatting. Foundry Local auto-downloads Phi-3.5 Mini (~2 GB) on first run.

How the RAG Pipeline Works

Let's trace what happens when a user asks: "How do I detect a gas leak?"

RAG query flow: Browser → Server → Vector Store → Model → Streaming response

Step 1: Document Ingestion

Before any queries happen, npm run ingest reads every .md file from the docs/ folder, splits each into overlapping chunks (~200 tokens, 25-token overlap), computes a TF-IDF vector for each chunk, and stores everything in SQLite.

Chunking example

docs/01-gas-leak-detection.md
  → Chunk 1: "Gas Leak Detection – Safety Warnings: Ensure all ignition..."
  → Chunk 2: "...sources are eliminated. Step-by-step: 1. Perform visual..."
  → Chunk 3: "...inspection of all joints. 2. Check calibration date..."

The overlap ensures no information falls between chunk boundaries — a critical detail in any RAG system.

Step 2: Query → Retrieval

When the user sends a question, the server converts it into a TF-IDF vector, compares it against every stored chunk using cosine similarity, and returns the top-K most relevant results. For 20 documents (~200 chunks), this executes in under 10ms.

src/vectorStore.js

/** Retrieve top-K most relevant chunks for a query. */ search(query, topK = 5) { const queryTf = termFrequency(query); const rows = this.db.prepare("SELECT * FROM chunks").all(); const scored = rows.map((row) => { const chunkTf = new Map(JSON.parse(row.tf_json)); const score = cosineSimilarity(queryTf, chunkTf); return { ...row, score }; }); scored.sort((a, b) => b.score - a.score); return scored.slice(0, topK).filter((r) => r.score > 0); }

Step 3: Prompt Construction

The retrieved chunks are injected into the prompt alongside system instructions:

Prompt structure

System: You are an offline gas field support agent. Safety-first...
Context:
  [Chunk 1: Gas Leak Detection – Safety Warnings...]
  [Chunk 2: Gas Leak Detection – Step-by-step...]
  [Chunk 3: Purging Procedures – Related safety...]
User: How do I detect a gas leak?

Step 4: Generation + Streaming

The prompt is sent to Foundry Local via the OpenAI-compatible API. The response streams back token-by-token through Server-Sent Events (SSE) to the browser:

Safety-first response with structured guidance

Expandable sources with relevance scores

Foundry Local: Your Local AI Runtime

Foundry Local is what makes the "offline" part possible. It's a runtime from Microsoft that runs small language models (SLMs) on CPU or NPU — no GPU required. It exposes an OpenAI-compatible API and manages model downloads, caching, and lifecycle automatically.

The integration code is minimal if you've used the OpenAI SDK before, this will feel instantly familiar:

src/chatEngine.js

import { FoundryLocalManager } from "foundry-local-sdk"; import { OpenAI } from "openai"; // Start Foundry Local and load the model const manager = new FoundryLocalManager(); const modelInfo = await manager.init("phi-3.5-mini"); // Use the standard OpenAI client — pointed at the local endpoint const client = new OpenAI({ baseURL: manager.endpoint, apiKey: manager.apiKey, }); // Chat completions work exactly like the cloud API const stream = await client.chat.completions.create({ model: modelInfo.id, messages: [ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: "How do I detect a gas leak?" } ], stream: true, });

Portability matters Because Foundry Local uses the OpenAI API format, any code you write here can be ported to Azure OpenAI or OpenAI's cloud API with a single config change. You're not locked in.

Why TF-IDF Instead of Embeddings?

Most RAG tutorials use embedding models for retrieval. We chose TF-IDF for this project because:

Fully offline — no embedding model to download or run
Zero latency — vectorization is instantaneous (just math on word frequencies)
Good enough — for a curated collection of 20 domain-specific documents, TF-IDF retrieves the right chunks reliably
Transparent — you can inspect the vocabulary and weights, unlike neural embeddings

For larger collections (thousands of documents) or when semantic similarity matters more than keyword overlap, you'd swap in an embedding model. But for this use case, TF-IDF keeps the stack simple and dependency-free.

Mobile-Responsive Field UI

Field engineers use this app on phones and tablets often wearing gloves. The UI is designed for harsh conditions with a dark, high-contrast theme, large touch targets (minimum 48px), and horizontally scrollable quick-action buttons.

Desktop view

Mobile view

The entire frontend is a single index.html file — no React, no build step, no bundler. This keeps the project accessible and easy to deploy anywhere.

Runtime Document Upload

Users can upload new documents without restarting the server. The upload endpoint receives markdown content, chunks it, computes TF-IDF vectors, and inserts the chunks into SQLite — all in memory, immediately available for retrieval.

Drag-and-drop document upload with instant indexing

Adapt This for Your Own Domain

This project is a scenario sample designed to be forked and customized. Here's the three-step process:

1. Replace the Documents

Delete the gas engineering docs in docs/ and add your own .md files with optional YAML front-matter:

docs/my-procedure.md

--- title: Troubleshooting Widget Errors category: Support id: KB-001 --- # Troubleshooting Widget Errors ...your content here...

2. Edit the System Prompt

Open src/prompts.js and rewrite the instructions for your domain:

src/prompts.js

export const SYSTEM_PROMPT = `You are an offline support agent for [YOUR DOMAIN]. Rules: - Only answer using the retrieved context - If the answer isn't in the context, say so - Use structured responses: Summary → Details → Reference `;

3. Tune the Retrieval

Adjust chunking and retrieval parameters in src/config.js:

src/config.js

export const config = { model: "phi-3.5-mini", chunkSize: 200, // smaller = more precise, less context per chunk chunkOverlap: 25, // prevents info from falling between chunks topK: 3, // chunks per query (more = richer context, slower) };

Extending to Multi-Agent Architectures

Once you have a working RAG agent, the natural next step is multi-agent orchestration where specialized agents collaborate to handle complex workflows. With Foundry Local's OpenAI-compatible API, you can compose multiple agent roles on the same machine:

Multi-agent concept

// Each agent is just a different system prompt + RAG scope const agents = { safety: { prompt: safetyPrompt, docs: "safety/*.md" }, diagnosis: { prompt: diagnosisPrompt, docs: "faults/*.md" }, procedure: { prompt: procedurePrompt, docs: "procedures/*.md" }, }; // Router determines which agent handles the query function route(query) { if (query.match(/safety|warning|hazard/i)) return agents.safety; if (query.match(/fault|error|code/i)) return agents.diagnosis; return agents.procedure; } // Each agent uses the same Foundry Local model endpoint const response = await client.chat.completions.create({ model: modelInfo.id, messages: [ { role: "system", content: selectedAgent.prompt }, { role: "system", content: `Context:\n${retrievedChunks}` }, { role: "user", content: userQuery } ], stream: true, });

This pattern lets you build specialized agent pipelines a triage agent routes to the right specialist, each with its own document scope and system prompt, all running on the same local Foundry instance. For production multi-agent systems, explore Microsoft Foundry for cloud-scale orchestration when connectivity is available.

Local-first, cloud-ready Start with Foundry Local for development and offline scenarios. When your agents need cloud scale, swap to Azure AI Foundry with the same OpenAI-compatible API your agent code stays the same.

Key Takeaways

1 RAG = Retrieve + Augment + Generate

Ground your AI in real documents — dramatically reducing hallucination and making answers traceable.

2 Foundry Local makes local AI accessible

OpenAI-compatible API running on CPU/NPU. No GPU required. No cloud dependency.

3 TF-IDF + SQLite is viable

For small-to-medium document collections, you don't need a dedicated vector database.

4 Same API, local or cloud

Build locally with Foundry Local, deploy with Azure OpenAI — zero code changes.

What's Next?

Embedding-based retrieval — swap TF-IDF for a local embedding model for better semantic matching
Conversation memory — persist chat history across sessions
Multi-agent routing — specialized agents for safety, diagnostics, and procedures
PWA packaging — make it installable as a standalone app on mobile devices
Hybrid retrieval — combine keyword search with semantic embeddings for best results

Get the code Clone the repo, swap in your own documents, and start building:

git clone https://github.com/leestott/local-rag.git

github.com/leestott/local-rag — MIT licensed, contributions welcome.

Data Driven Analytics for Responsible Business Solutions, learning how to work with Power BI

RiannevanEe — Thu, 05 Mar 2026 08:00:00 GMT

Introduction

In this blog post, we will be showcasing the project that we have worked on for the last couple of weeks. Here, we analysed a dataset using Power BI and its machine learning capabilities. For this, we were given the fictitious case of VenturaGear. The company was faced with the challenge of new competition, and it was our job to provide a data-driven insight into customer behaviour, feedback, and preferences. The objective was to support more effective customer targeting by identifying patterns and segments that could inform strategic decision-making, while ensuring ethical and responsible use of data.

Before we jump into the course and our final results, we would like to introduce ourselves and the roles we had.

Product Owner: Kylie Eggen

Hello everyone! My name is Kylie, and I'm currently busy finishing my Master Responsible Digitalisation. During the DARBS course, I had the role of the product owner. This allowed me to develop a deeper understanding of both data analysis and the ethics of handling sensitive data. The course provides you with skills that could be useful in your future career, which is very nice. I liked the learning experience a lot and will definitely use it in the future! Kylie Eggen | LinkedIn

Data Analyst: Ha Nguyen

I am currently in the final stage of my Master’s degree in Responsible Digitalisation, focusing on the ethical and strategic use of data-driven technologies. With five years of experience using Excel for data analysis, I have developed a strong foundation in data handling and visualisation. This course allows me to expand my skills by learning to create interactive dashboards and generate actionable insights using Power BI. These competencies strengthen my ability to support responsible, data-driven decision-making in my future professional career. Ha Nguyen | LinkedIn

Data Analyst: Rianne van Ee

Hello! My name is Rianne, and I am currently in the process of completing my Master’s degree in Responsible Digitalisation. I chose this specialisation because I am very interested in new technologies and different perspectives. I am very interested in data analysis and learning about new software, so the DARBS course was very interesting to me. I am excited to apply my new skills in a professional environment. Rianne van Ee | LinkedIn

Data Visualisation Consultant: Aya Torqui

Hello! My name is Aya Torqui, and I am a Master’s student in Responsible Digitalisation at Radboud University. One of the reasons I chose this specialisation is my strong interest in how companies transform raw and sometimes ambiguous data into valuable business decisions. The DARBS course, therefore, provided the perfect opportunity for me to gain new and deeper insights into this process. In my role as a Data Visualisation Consultant, I developed new skills not only in designing visually attractive and interesting dashboards, but also in communicating a meaningful and coherent story through them. I am grateful for the opportunity to have developed these skills during the course, and I look forward to further broadening and strengthening them in my future career. Aya Torqui | LinkedIn

Data Visualisation Consultant: Ting Yu

Hi! My name is Ting Yu. I am currently a Master’s student of Civil Law and Responsible Digitalisation. I found the DARBS course quite interesting, and it was a whole new experience for me, because I learned that numbers are not boring. With a dashboard, it is possible to tell a story and help organisations. What I also really liked about this course was the creative side. Not only was it fun to play around with different charts and colour schemes for the dashboard, but also the video we had to make! I am curious to see what the future possibilities are. Ting Yu | LinkedIn

Project Overview

The goal of this project was to provide data-driven managerial recommendations to the fictitious company, VenturaGear. Eventually, it was our task to deliver a final report and a video blog in which we discussed their data and gave them recommendations on how to improve. Our focus was on supporting more effective customer targeting by identifying patterns and segments that could inform strategic decision-making. During the process, one of our main goals was to keep the data analysis responsible and ethical.

Project Journey

The course followed a nice structure, allowing us to learn about PowerBi gradually and expand our skills and knowledge over a couple of weeks. We started off by completing lab work. Every week we completed several online courses, and spent one lecture applying the knowledge from these courses in a lab work assignment. After a few weeks, we applied our knowledge in a milestone assignment. This was the first time we really applied our newfound skills in a practical manner. This was a really nice opportunity to see whether we could actually apply what we learned.

This also came with a machine learning aspect. Even though we had a short introduction to the topic in class, none of us had worked with machine learning before. We were able to apply the knowledge we gathered about learning how to use a new system, like Power BI, on another system, in this case, machine learning. While we really struggled here at the start, after some time we figured it out and were able to work with the technology.

This milestone assignment was the perfect preparation for the actual final assignment, which also had this machine learning aspect. We now knew where to start, what data to include, etc. We now also knew what to consider when looking at the ethical side of things. Like what information needs to be anonymised, or left out completely. Eventually, all our newfound knowledge was combined into making the final assignment and video blog.

Technical Details

Microsoft Power BI served as the main analytical environment throughout the project. We began by importing multiple CSV datasets into Power BI and preparing the data using Power Query. This involved cleaning duplicate records, correcting formatting inconsistencies, and transforming variables to ensure accurate calculations and reliable analysis.

We then created a relational data model connecting key tables such as sales transactions, product information, customer behaviour, and sales reasons. Establishing these relationships allowed us to analyse data across multiple dimensions and generate deeper insights into customer activity and online purchasing patterns.

Interactive dashboards were developed using Power BI’s visualisation tools, accessible colour themes, and slicers, allowing users to explore insights dynamically. Rather than presenting static results, the dashboard encouraged managers to interact with the data and investigate patterns independently.

In addition to descriptive analytics, we applied a machine learning model (XGBoost) to identify factors influencing the sales of the top revenue-generating products. This introduced us to predictive analytics and highlighted the importance of feature selection, handling missing values, and critically interpreting model outputs. Combining visualisation with machine learning enabled us to move beyond reporting toward data-driven decision support.

Results and Outcomes

Before we could analyse our data, we ran into a few problems. Firstly, our unit prices seemed to be inflated in the dataset. The decimal was removed, leading to unreasonably high prices. To solve this, we recalculated the LineTotal, using the formula that can be seen below.

Another problem we ran into was that we seemed to have a lot of missing data. We noticed this while looking at the sales reasons. A third of the data ended up blank. We ended up excluding the blank values, so that we were still able to analyse the remaining data.

To really effectively target customers, we felt it was important to analyse the reasons people made their purchases. Through our analysis, we found that for VentureGear, the biggest contributor was price.

We found that VenturaGear mainly made its sales in Australia.

Lesson Learned

Working with new systems

The main lesson that we learned is how to start using a new system. The way in which we were taught how to use Power BI showed us a nice way of approaching new things. We believe this can be useful in other areas of our professional lives.

2. Data analysis

Most of us were a little intimidated when we first heard that we were going to be analysing data through a new program. However, once we started, we noticed that when we all put our minds to it, it is quite manageable. We have all gained some understanding of data analysis and how to visualise this.

3. Teamwork

A big factor during this project was teamwork. Our team was divided up into different roles. That meant that there was teamwork between the two data analysts and data visualisation consultants, but also between different roles. We found it to be really important to have teamwork between all these actors. We noticed that the further we got into the project, the smoother this interaction went.

Collaboration and Teamwork

On this project, we worked as a team. Our team consists of five people. Kylie Eggen was the Product Owner. Her role was to take care of the overview of the project. Ha Nguyen and Rianne van Ee were the Data Analysts for this project. Aya Torqui and Ting Yu were the Data Visualisation Consultants.

We mostly stuck to our roles, but noticed that everything needed to happen in collaboration. So even though we were all mainly busy with our own roles, we were all involved in each other as well. We noticed this really helped in making the project a coherent whole.

Future Development

While this project generated valuable insights, there are several opportunities for further development. A potential next step would be integrating real-time data into Power BI. Expanding the dashboard with automated data refresh will allow managers to track performance continuously and respond more quickly to changing customer behaviour.

Another area for future development involves extending the machine learning component. Rather than focusing only on identifying predictors of key revenue-generating products, the model could be expanded to include customer segmentation, such as grouping customers into categories like high-value customers, discount-sensitive buyers, or frequent online shoppers. In addition, the model could be developed further to support purchase prediction, enabling forecasts of seasonal demand, identifying customers likely to make repeat purchases, and determining which products are most preferred by specific customer groups. These enhancements would provide a more dynamic understanding of customer behaviour and support more targeted, data-driven decision-making.

Incorporating more complete behavioural data or improving survey participation rates would also help reduce missing values and increase the reliability of insights. And finally, for future research, the organisation could consider introducing clear consent options on the web shop to help customers better understand what data is being collected. These options would also allow customers to choose what information they want to share, improving transparency and strengthening customer trust.

Conclusion

This project allowed us to learn how data analytics can help organisations make smarter and more responsible business decisions. Using Power BI, we transformed complex customer and sales data into clear, interactive insights that help managers better understand online behaviour, purchasing motivations, and performance trends. Beyond building technical skills, we also learned how important data quality, transparency, and ethical considerations are when working with sensitive customer data. Throughout the project, we discovered that data analysis is an iterative process that requires continuous evaluation, critical thinking, and careful interpretation of results. Most importantly, we realised that meaningful analytics is never an individual effort but a collaborative process, where teamwork and shared problem-solving play a key role in turning data into valuable insights.

Overall, this project strengthened our ability to bridge technical analytics with responsible digitalisation principles. By combining business understanding, visualisation skills, and ethical awareness, we gained a clearer perspective on how tools like Power BI can enable professionals to create meaningful, data-driven solutions that are both impactful and responsible.

Call to Action

After experiencing this learning journey, we encourage you to engage with tools such as Power BI. As our teacher told us, ‘‘You are going to hit a wall.’’ That is exactly what happened to us, but pushing through those moments allowed us to create a deeper understanding and develop new skills. At the same time, we tried to stay aware of the ethical implications of working with data. During the project, we always ensured to stay transparent and responsible in our analysis. We encourage you to challenge yourself! Experiment with new technologies and step outside of your comfort zone. What we also think you should remember is that a strong analysis is not only dependent on technical skills, but it is also about staying transparent, responsible, and trustworthy.

On behalf of group 3, thank you for taking the time to read our summary. Wehope it has been useful. Feel free to reach out for any remaining questions!

The Hidden Architecture of Nano Architectures

hazem — Tue, 03 Mar 2026 08:00:00 GMT

Why does the same prompt, on the same checkpoint, with temperature set to zero, sometimes produce a different answer only when the system is under real load?
If you have ever watched token three flip and then watched the whole completion diverge, you already know this is not a product bug. It is a systems fact.

Here is the thing. In production, you did not deploy a model.
You deployed a runtime that selects an execution plan under constraints.

The weights are inside that plan. The behavior is the plan.

I’m Hazem Ali — Microsoft AI MVP, Distinguished AI and ML Engineer and Architect, and Founder and CEO of Skytells.

I’ve built and led engineering work that turns deep learning research into production systems that survive real-world constraints. I speak at major conferences and technical communities, and I regularly deliver deep technical sessions on enterprise AI and agent architectures.

If there’s one thing you’ll notice about me, it’s that I’m drawn to the deepest layers of engineering, the parts most teams only discover when systems are under real pressure. My specialization spans the full AI stack, from deep learning and system design to enterprise architecture and security.

A rule I repeat in every serious review is simple.

If you cannot explain the runtime, you do not understand the model you deployed.

— Hazem Ali

This is the next layer after my earlier deep dive on memory, KV cache, paging, and trust boundaries in The Hidden Memory Architecture of LLMs

I also break down the memory-and-paging failure modes in When Your LLM Trips the MMU

This one goes lower, into the execution that decides which math actually runs.

When I Had to Prove It Live

I still remember the first time I had to make this concrete in front of a room full of engineers.

It was during a technical session I gave, and the question came up in the exact form you’ve probably heard before:

Why does the same prompt on the same checkpoint, with temperature set to zero, sometimes produce a different answer only under real load?

So I answered it the only way that holds up in a serious engineering room.

I didn’t frame it as randomness. I framed it as execution.

Not because it sounds cleaner,

but because it is the only framing that survives scrutiny: under load, the system is not evaluating the same computation.

Hazem Ali speaking at an AI conference, discussing Zero-Trust Enterprise AI Architecture, governance, and production-ready AI systems.

In production, you don’t deploy weights in isolation. You deploy a runtime that selects an execution plan under constraints. Under load, the constraints change at token cadence: microbatch membership shifts, shapes shift, workspace feasibility tightens, and kernels or algorithms that were legal in the calm regime can become infeasible in the pressured regime. The runtime stays correct by contract, but it executes a different plan.

And once the executed plan changes, reduction staging can change. When reduction staging changes, rounding happens at different points. That can move last bits. In decoding, last bits can become different tokens when early logit margins are thin. After the first token flips, divergence is expected because the context is different.

That’s what I mean throughout this article when I say:

The weights are inside the plan, but the behavior is the plan.

What is Happening in Runtime

Let’s start with the part most teams skip: the runtime pipeline from admission to a token.

A production LLM server is not a function call. It is a control plane. And under real load, it behaves like one.

It is not asking “what does the model say.” It is asking “what can I execute right now without breaking my guarantees.”

Right now matters. Not in theory, in milliseconds. Because every decode step is a new scheduling event. The system does not commit to a single plan for the entire completion. It keeps re-evaluating feasibility as state shifts.

What can I execute at this moment, with the VRAM I still have, on the hardware state I am currently in, while staying inside isolation boundaries and latency targets.

That question is not answered once per request. It is answered repeatedly, at token cadence. The queue changes. The batch changes. Memory headroom changes. Cache residency changes. Workspace availability changes. The set of legal kernel and algorithm choices changes with them.

And that is the point most people miss. The runtime is not just running your weights. It is continuously selecting an execution plan under constraint. The weights are inside that plan, but behavior lives in the selection.

That selection is layered. Admission shapes the effective request. Scheduling forms the batch for this step. Kernel and algorithm choice binds the math that will actually run. Memory residency and allocation decide what is feasible. Isolation rules decide what sharing is allowed. Each layer contributes to the final plan, and the plan is what you are deploying.

Admission and shaping

Before your prompt ever reaches the model, it gets shaped.

Truncation, policy injection, tool schema expansion, routing metadata, tenant tags, prefix reuse decisions, and safety transformations.

If you do not know what I mean by effective request, I mean the exact token sequence that the model saw after shaping. That is the only input that matters for reproducibility.

Batching and step level scheduling

Modern servers do not just batch requests. They batch token steps.

In a continuous batching system, token step timing feeds back into batching decisions. A slightly slower step changes who joins the next step. Who joins the next step changes shapes. Shapes change kernels. Kernels change numeric pathways.

This is not an opinion. It is why vLLM exists. The PagedAttention paper describes serving as a batching problem where KV cache grows dynamically, wastes memory through fragmentation, and limits batch size. It introduces block level KV management and builds vLLM on top of it as an LLM serving system.

Kernel plan selection and library behavior

Once shapes are known, the runtime selects kernel variants and library algorithms that are feasible for those shapes and the workspace currently available.

This is the part people underestimate. The same operator can have multiple valid implementations. The chosen implementation can change when workspace is tight, when shapes change, or when the engine wants to trade latency for throughput.

Memory allocation and residency

KV cache, activations, temporary buffers, workspace, graph memory, and communication buffers compete for VRAM.

Under pressure, allocation patterns change. Fragmentation changes. Residency changes. Cache locality changes. All of that changes the system timeline and the feasible plan space.

If you want a one line summary that is accurate in 2026 production inference, it is this.

Inference is a scheduling problem plus a memory residency problem, and the model is inside that.

The Scope

First, Let me put it very clear.

I am not claiming every deployment is nondeterministic.
I am not claiming every kernel variant flips tokens.
I am not claiming seeds are useless.

I am making a narrower claim, the kind you can defend in an incident review without hand waving.

Floating point math is not associative. Order matters. When you parallelize, you change the order of operations, and it is therefore valid for parallel results to differ from a sequential evaluation. NVIDIA states this directly in the CUDA C Best Practices Guide.

CUDA also makes a foundational guarantee to the hardware and scheduler, not to your intuition. Thread blocks must be able to execute independently, in any order, in parallel or in series. That freedom is part of the programming model, not an edge case (ref).

Now connect those two facts. If accumulation order changes, the last bits can change even when every operation is correct, because floating point addition is not associative. NVIDIA explicitly calls this out as well.

Then layer in what serving stacks actually do. Production systems intentionally reshape execution through continuous batching and KV memory management. vLLM is a published example of this co design, where serving throughput is achieved by dynamic batching and memory-aware KV handling.

Finally, bridge the nano to the semantic.

When early logit margins are small, tiny numeric deltas can reorder the top candidates, and a single token flip is enough to diverge the entire completion.

Here is the part that should feel a little scary, because it changes what you think you are operating.

Under real load, the system is not just slower. It can enter a different execution regime. Batch composition shifts, shapes shift, workspace and residency shift, and the runtime is forced into a different set of legal kernel and algorithm choices. Nothing “breaks.” No bug is required. The system is still correct by contract. But your output is now a property of the regime you are in, not the demo you validated.

That means you can pass every determinism test at idle and still ship a system that drifts only when it matters, at p95 and p99, when queues are long and memory headroom is tight. The first time you notice is often a user screenshot, an audit question, or an incident report where two replicas disagree on the same request because the runtime state was not the same.

The equation principals should use in incident reviews

Most teams ship with the demo mental model.

y = f(x, θ)

One prompt in, one checkpoint, one output. If the output changes, someone concludes the weights changed, or “AI is random.”

That is not how production inference behaves, because production inference is not just a function. It is execution under constraint.

Production behavior is closer to this.

y = Decode( Exec(θ, x; s) )

θ is still the same weights. But the thing you actually shipped is Exec, and Exec is chosen. It is chosen per step, under the current state of the system. The behavior you observe is the behavior of the executed plan, not the abstract weights.

Demo vs production mental models. In production, y depends on (θ, x, s) because the runtime selects an execution plan under constraints.

X is not the prompt. X is the effective request.

X is the exact token sequence the model saw after shaping. Truncation, policy injection, tool schema expansion, routing metadata, prefix reuse, safety transforms. All of that can change what the model actually receives.

If you cannot reconstruct x, you are not replaying the request. You are replaying an approximation.

Here is the minimum you should log for x, even if you cannot store raw text:

# minimal "x" record: enough to reproduce or prove you cannot trace_x = { "req_id": req_id, "raw_prompt_sha256": sha256(raw_prompt), "effective_text_sha256": sha256(effective_text), "effective_tokens": len(effective_tokens), "truncated": truncated, "trunc_reason": trunc_reason, # e.g., "latency_guard", "context_cap" "decode_cfg_applied": decode_cfg, # temperature/top_p/max_tokens, etc. "shaping_events": events, # ["policy_inject:v3", "tool_schema:v2", ...] }

S is not a vibe. S is the execution state that decides the math.

S is what principals should demand in a postmortem, because this is what turns “it drifted” into “this plan executed under this regime.”

At minimum, s includes:

per-step batch composition and shape class
queue delays and scheduling outcomes
VRAM headroom and workspace availability
cache pressure signals
precision path and engine fallbacks
distributed timeline signals (TP/PP latency, collective stalls)
isolation posture (what batching is allowed)

Why this matters: in continuous batching, time becomes part of semantics. A few milliseconds of delay changes who gets co-scheduled at the next token step. That changes shapes. Shapes change kernel/algorithm feasibility. Feasibility changes the numeric pathway. When early logit margins are thin, a tiny pathway delta is enough to flip the argmax.

Here is a short, practical “s” record you can emit per decode step:

# per-step "s" record: what plan ran, under what pressure step_s = { "req_id": req_id, "step": t, "batch_fp": sha256(",".join(sorted(batch_req_ids)))[:12], "shape": f"q=1,k={klen},h={heads},d={hidden},tp={tp}", "queue_ms": queue_ms, "gpu_ms": gpu_ms, "vram_free_mb": vram_free_mb, "workspace_free_mb": workspace_free_mb, "kv_regime": kv_regime, # "normal" | "pressured" | "paged" "precision_path": precision_path, # "bf16" | "fp16" | "tf32" | "fp32" "algo_id": algo_id, # backend/engine specific "kernel_variant": kernel_variant, # if available "isolation_mode": isolation_mode, # "shared" | "strict" }

The incident-review translation

If you only ask “what prompt did the user send” and “what weights did we run,” you are using the demo equation.

You will argue about seeds, debate “randomness,” and never converge.

The production equation forces the real question.

Which plan executed, under which constraints, and what state pushed us into that plan.

The line principals should repeat until teams internalize it is simple.

Weights are static. Behavior is a property of the executed plan. And the executed plan depends on state.

If you want one more operational layer that makes this feel real, add a regime marker.

Regime changes are where “stability” collapses without any bug:

def regime(vram_free_mb, paging_on, isolation_strict, queue_p95_ms): if isolation_strict: return "isolation_strict" if paging_on: return "paging" if vram_free_mb < 1024: return "memory_pressured" if queue_p95_ms > 50: return "queue_degraded" return "normal"

When the regime changes, the feasible plan space changes. When the plan space changes, the executed math can change. That is the production reality your incident review must be able to explain.

Floating point order is where small deltas are born

Let’s break it down without hand waving.

Finite precision makes rounding part of the computation

Floating point math is not real-number math. Every add and multiply is followed by rounding to the representable format you are using. That rounding is not “noise.” It is part of the computation. Once you accept that, one consequence becomes unavoidable.

Order matters.

NVIDIA states the rule clearly: floating point involves rounding, and when you parallelize you can change operation order, so parallel results may not match sequential results.

Why LLM inference is a perfect storm: reductions everywhere

Now connect that to what an LLM does at inference time.

LLM inference is reduction-heavy by design. Dot products in GEMMs, attention score accumulation, softmax normalization, layer norm statistics, even top-k selection pathways. These are not single operations. They are many partial operations combined into a final scalar or vector. In floating point, the way you combine partials is the outcome.

GPU reductions are staged: partial sums, then merges

A reduction on GPU is not “a sum.” It is a staged reduction of partials.

On a CPU, you can imagine a left-to-right accumulation:

((((a1 + a2) + a3) + a4) + ...)

On a GPU, that mental model is wrong. The GPU is built to run thousands of threads. So it computes partial sums in parallel and then merges them in stages. The staging pattern is determined by kernel design and how the backend maps the problem to hardware.

Put the figure here, right after the staging idea lands.

Parallel reductions form partial sums and merge them in stages. Different legal staging orders can shift the last bits under finite precision.

The staging depends on decisions you do not control at the prompt layer:

how data is tiled into blocks
how each block maps to warps
how many partials each warp reduces
whether it uses warp-level primitives, shared memory, or tensor core fragments
how the final merge is staged across blocks

Change the tile size, or the block shape, or the occupancy, and you often change the staging order. Change the staging order, and you change when rounding happens. You can get two results that are both correct under IEEE floating point rules, and they differ in the last bits.

This is not a bug. It is the contract of finite-precision parallel math, applied at scale.

Why the last bits move at the core level

Floating point addition is not associative under rounding because rounding happens after each operation. The error introduced at each step depends on the magnitude and sign of what you are adding at that step.

When you change the staging order, you change:

which numbers get added together early
which partial sums get rounded early
how cancellation behaves when positive and negative terms interact
when large and small magnitudes meet, where small values can lose representable impact

That is the core mechanism behind “small deltas.” It is not mystical. It is mechanical.

Why this shows up in production serving, not in your demo

LLM inference is dominated by massive matrix operations and attention. Under the hood, those paths accumulate across large dimensions. An accumulation is exactly where rounding order matters most.

And the server does not always run the same kernel variant for those ops. Under load, shape shifts and workspace pressure can push the backend into different implementations. Different implementations often imply different tiling. Different tiling implies different staging. Different staging implies different rounding. Different rounding implies different last bits.

So even with an identical prompt, identical checkpoint, and temperature set to zero, you can still see tiny numeric differences when:

batch composition changes and produces different effective shapes
the engine picks a different algorithm because workspace is tighter
the kernel selects a different tile path due to shape class and occupancy
the GPU is in a different pressure regime, changing feasibility and scheduling behavior

Those deltas are small, but they are real.

And in decoding, small can be enough.

The bridge from ulps to language: logits, argmax, divergence

A tiny last-bit difference is often irrelevant, Until it hits a decision boundary.

At decode step t, greedy decoding chooses an argmax. If the top logits are close, a small delta can swap the ordering. Once token t changes, the context changes, and the completion diverges. That is not randomness. That is deterministic branching from a slightly different numerical pathway.

So the actionable takeaway is not “GPUs are nondeterministic.”

It is this.

Parallel math is allowed to produce multiple correct last-bit outcomes, and LLM decoding can amplify those outcomes into different text when margins are thin.

CUDA scheduling makes ordering a form of runtime state

CUDA makes a stronger statement than most people realize.

Thread blocks must be able to run independently. It must be possible to execute blocks in any order, in parallel or in series.

That is why the same kernel can execute with different inter block ordering depending on occupancy, contention, and scheduling.

Now bring atomics into the picture.

Atomics guarantee correctness of each update. They do not guarantee the arrival order of updates across threads and blocks. When floating point updates arrive in different legal orders, the final sum can differ in the last bits, because floating point addition is not associative.

If you do not know what atomic add means, here is the useful definition.

Atomic add ensures updates do not overwrite each other. It does not ensure which thread gets there first.

This is the nano architecture layer that explains a lot of weirdness. Many engineers assume determinism is a property of weights. In practice, determinism is constrained by the legal reorderings of parallel execution.

Logit margin is the bridge from ulps to language

Now we connect the last bits to a changed sentence.

At decode step t, greedy decoding picks the argmax over logits.

Let the top two logits be ℓₐ and ℓ_b. Define the margin:

mₜ = ℓₐ − ℓ_b

A token flip happens when a small perturbation changes the ordering of these top two.

If you want an operational translation, it is this.

If the model barely prefers token A over token B, a tiny numeric delta can make it prefer B.

Once token t changes, the rest of the completion evolves under a different context. Divergence is expected.

This is why I keep pushing one instrumentation idea that sounds boring until you need it.

Measure early step margins.

You cannot manage stability if you never measure how close the decision boundary is.

The effective request problem, the quiet killer of reproducibility

Here is the pattern I see in almost every serious production investigation.

The user prompt is not the executed input. The shaping pipeline produces the effective request x, and under load it can change length, semantics, and decode configuration. Log the contract, not the story.

The team replays the user prompt, cannot reproduce the output, and concludes the model is nondeterministic. Then the incident dies in ambiguity.

And then, usually too late, someone asks the only question that matters.

What did the model actually see.

“In every postmortem, I ask one question before I look at weights, kernels, or seeds: what did the model actually see. If we cannot answer that, nothing else is evidence.” - Hazem Ali

In production, the user prompt is not the input. It is an ingredient.

By the time a request reaches the model, it has passed through a shaping pipeline that exists to keep the system safe, fast, and multi-tenant. That pipeline is not cosmetic. It can change semantics, length, and even decode behavior. The result is the only input that matters for reproducibility.

The effective request.

This is the same thesis you have already accepted earlier in the article.

y = Decode( Exec(θ, x; s) )

If you do not know x, your replay is not valid. If you do not know s, your replay is not comparable. And if you only log the raw prompt, you are logging neither.

Shaping changes semantics, not just length

Truncation is the obvious one. Under load, systems often cap context length to protect latency and GPU memory. Same prompt, different truncation boundary, different effective context, different output. Nothing “random” happened. You executed a different input.

But truncation is only the beginning.

Policy injection can prepend or append system text that changes intent. Tool schema expansion can add hundreds or thousands of tokens and push the request over a context boundary. Routing metadata can select a different template. Prefix caching can reconstruct parts of context from cached state rather than raw text. Safety transformations can rewrite or neutralize content. Even small differences here can shift early logits when margins are thin, and this article already showed how small deltas become different tokens.

The worst part is that this is silent by default.

The user sees their prompt. Engineers see the prompt in logs. The model sees a different token sequence. Then everyone argues about reproducibility using the wrong input.

Why this interacts with load, not just correctness

Under low load, your system often has enough headroom to be generous.

Longer context, fewer cutoffs, stable routing, more consistent batching, and fewer fallbacks.

Under real load, shaping becomes defensive.

Dynamic truncation thresholds kick in. Tool schema expansions collide with context limits. Prefix reuse behavior changes. Safety gates can become stricter. The same user text can produce a different effective request, and therefore a different output, precisely when the system is under pressure.

So if you are only validating reproducibility at idle, you are validating a different system than the one you ship.

What principals should require in telemetry

If you want strict reproducibility, you must log the execution contract per request. Not the story. The contract.

At minimum:

effective token count after shaping
truncation boundary and reason
final merged decode config actually applied
policy gates that modified prompt or decode path
whether prefix cache was used, and what cache key was referenced
routing template version and system message hash

If you are privacy constrained, you still can log hashes and structural facts. You do not need raw prompts to diagnose effective request drift. You need verifiable fingerprints.

Here is the short version in one line.

If you only log the user prompt, you have not logged x. You have logged an approximation of x.

And without x, you cannot claim reproducibility. You can only hope for it.

Continuous batching, why time becomes part of semantics

This is where principal level thinking matters.

Continuous batching does not just increase throughput. It changes the execution context at each token step.

Batch composition changes shapes. Shapes influence kernel selection and workspace feasibility. Those choices can change reduction structure and rounding pathways.

If you want a published anchor, use vLLM.

The PagedAttention paper frames high throughput serving as a need to batch many requests, but KV cache grows dynamically and wastes memory through fragmentation. It proposes PagedAttention and builds vLLM on top of it, with block level memory management and flexible sharing of KV cache to reduce memory usage. (arxiv)

Here is what this really means in production.

The server is selecting which requests share a step. That changes the math shapes. That changes the executed plan. That is why the same prompt behaves differently under load even at temperature zero.

Algorithm selection and engine fallback

The hidden variability people forget about

If you have ever tried to reproduce a drift across replicas and felt like you were chasing ghosts, this is usually the layer you were missing.

Libraries and engines choose, Not in a philosophical sense. In a literal, per-operator, per-shape sense.

The same attention call is a fork in the road between multiple legal tactics, each with different tiling, different reduction staging, different fusion boundaries, and different temporary memory requirements. Your checkpoint is the same, your prompt is the same, your temperature is zero, and the output still moves because the executed plan moved.

PyTorch says the quiet part directly. Disabling cuDNN benchmarking makes cuDNN deterministically select an algorithm, and PyTorch stresses this is different from the deterministic setting. That is the whole story in one sentence: one switch affects how the backend selects an algorithm, another affects whether the selected algorithms are deterministic. Those are separate layers, and under load they can diverge.

Now go down to the core of the core.

A tactic is not fast or slow. In production serving, a tactic is legal or illegal under the constraints of this token step.

The constraint that forces most plan switches is not compute. It is workspace feasibility. Many high-performance kernels need scratch buffers. Some need enough contiguous space to stage tiles, reorder operands, hold partials, or run fused epilogues. When VRAM is fragmented or headroom drops, a tactic becomes impossible even if it is the tactic you validated at idle. The engine does not throw a warning. It simply selects another legal tactic.

That is the first uncomfortable point.

The second uncomfortable point is what makes this align perfectly with the next section.

The constraint is not only “how many MB are free.” The constraint is the memory hierarchy state of the chip.

Under load, two replicas can have the same free VRAM and still be in a different regime because the chip is not one pool of memory. It is HBM plus an on-die L2, plus TLBs, plus page tables, plus a fabric that is arbitrating traffic between SMs, L2 slices, and HBM controllers. When that hierarchy shifts, latency per token step shifts. And in continuous batching, a few milliseconds is not a timing detail, it is a scheduling input.

This is how a performance event becomes a behavior event without any bug.

The engine’s planner sees a world where a tactic that was “best” at idle is no longer best, or no longer feasible, because the chip is in a different pressure state. Your runtime is still correct. It is just operating a different plan in a different regime.

One op, multiple legal kernels. The chosen tactic depends on shape class and feasibility.

Now bring TensorRT into the picture, because it makes the precision dimension explicit.

TensorRT states TF32 Tensor Core usage is not guaranteed and it can fall back to FP32, and it documents configuration controls around TF32.

That statement is not about “precision preference.” It is about the reality that precision is part of tactic selection. Precision changes which instructions execute and how accumulation is staged. When your early logit margins are thin, a small pathway delta can swap the argmax at one step. One token flips, and the rest of the completion deterministically diverges.

So “temperature zero” is not a determinism guarantee. Temperature governs sampling. It does not pin the execution pathway.

If you want a more mechanical anchor, treat matmul the way NVIDIA exposes it: cuBLASLt has a preference descriptor for applying algorithm search preferences and fine-tuning the heuristic function. That is not marketing. That is the API admitting that algorithm selection is a constrained search problem.

Now the part that gets rare, and the part most teams never write down.

CUDA’s programming model requires that thread blocks be able to execute independently and may execute in any order, in parallel or in series.

This matters here because tactic switches often change block geometry and tiling. Different block geometry changes reduction staging. Reduction staging changes where rounding happens. Even if every operation is correct, last bits can move because you legally changed the staging of partial sums. You do not need randomness. You need a different legal staging tree.

two reduction trees, both legal, merging partial sums in different orders, final ulps differ

Now pull security into the same frame, because it is not a separate layer in production.

Security posture changes what the scheduler is allowed to do. Isolation constraints reduce batching freedom. Reduced batching freedom increases tail latency. Tail latency pushes you toward tighter admission controls and more aggressive memory behavior. That shrinks the feasible tactic set sooner. In other words, security decisions can move you across regime boundaries faster, which increases plan switching frequency. Stability becomes an SLO dimension of your security posture, not a property of your weights.

This is the business consequence that shows up in the worst possible way.

At idle, you look stable. At p95 and p99, you drift. Two replicas disagree. You cannot reproduce because you logged prompts and weights, not the executed plan. An enterprise buyer does not care whether the drift came from “a tactic fallback.” They care that the system cannot explain itself.

So here is the operational rule I use in reviews.

If you cannot prove which plan ran, you cannot claim reproducibility.

And that leads to the only practical addition that belongs in this section before we move into VRAM bandwidth and cache residency.

VRAM bandwidth, cache residency, and why memory hierarchy becomes control plane input

Let’s talk about the performance facts that quietly become behavior facts.

And yes, I know how complex this gets. I have watched strong staff and principal engineers get lost here, not because they are weak, but because the system crosses too many layers at once: GPU microarchitecture, allocator behavior, kernel tactics, batching policy, and SLO-driven control loops. No single dashboard shows you the full causal chain.

That is exactly why I frame it this way. It is not “performance tuning.” It is a coupled control system. So let me break it down cleanly, from the chip outward, until the behavior change becomes inevitable.

NVIDIA describes H100 SXM5 as having HBM3 bandwidth around 3 TB/s and an L2 cache of 50 MB designed to reduce trips to HBM by caching repeated accesses.

Most teams read that as “the GPU is fast.”

In serving, it is more precise to say: the GPU gives you a memory hierarchy with regimes, and your runtime is forced to adapt to whichever regime you are currently in.

The chip-level model you should carry in your head

Decode is not one big matmul. It is a loop that repeatedly touches a shifting working set:

KV blocks for the active sequences
attention metadata (block tables, indirection, masks)
sampling buffers (logits, top-k/top-p structures)
runtime bookkeeping for continuous batching

Those accesses are not purely streaming. They are pointer-heavy, and their locality depends on how your KV is laid out, which requests are co-scheduled, and how fragmented your memory becomes under churn.

Here is the simplest mental model that is still honest:

B_HBM is the number of bytes actually read from HBM during this step.
B_L2miss is the number of bytes that missed L2 and therefore had to be fetched from HBM.
t_translate is the address-translation tax: extra time from TLB misses and page-table walks.

That last term is the one that surprises people. It’s “invisible” until it dominates.

Why L2 residency becomes a control-plane input

Now connect that to decode, Decode repeatedly reads KV state. If L2 hit rate drops, HBM traffic rises. When HBM traffic rises, stalls rise. When stalls rise, token-step latency shifts.

When token-step latency shifts, the server changes batching decisions.

This is the control loop you should keep in your head:

L2 hit rate ↓ → t_step ↑ → Δt ↑ → batch composition changes → shape class changes → tactic set changes

That is the bridge from “cache miss” to “different plan executed.”

In continuous batching, time is not just an output metric. Time is an input into the next scheduling decision. A few milliseconds can change who gets co-scheduled at the next token step. That changes shapes. Shapes change feasible kernels and algorithms. That changes the executed math. And if early logit margins are thin, a small pathway delta can flip a token and send the rest of the completion down a different branch.

Rare but matters: the translation tax that breaks the “free VRAM” illusion

Two replicas can report similar free VRAM and still be in different regimes.

Why? Because the chip is not “a pool of memory.” It is an on-die cache, translation structures, page tables, and a fabric that is arbitrating traffic under pressure.

When KV is stored in blocks (or pages) and those blocks are scattered due to churn, you often get:

worse spatial locality
more distinct memory regions per step
more TLB pressure
more page walks

Page walks are not abstract. They are memory reads.

They compete with your payload reads. Under real load, this turns into self-inflicted HBM traffic.

So you can be “bandwidth rich” on paper and still be “latency poor” in practice because the working set became translation-hostile.

This is how a performance event becomes a behavior event without any bug.

A concrete KV bandwidth sanity check

If you want a back-of-the-envelope check for why decode becomes memory-shaped, use a conservative estimate.

Per token step, you often need to read a large portion of KV for the active context. A rough model is:

KV bytes per step ≈ 2 × B × L × H × D × s

Where:

B is batch size (number of sequences co-scheduled in the step)
L is current context length (tokens already in KV)
H is the number of attention heads (or KV heads, depending on the model)
D is head dimension
s is bytes per element (2 for fp16/bf16, 1 for int8, etc.)

The factor 2 accounts for K and V.

Even if your kernel is compute-efficient, you are still moving a lot of bytes. If locality collapses and L2 misses rise, you shift into an HBM-limited regime fast.

That is the mechanical reason your p95/p99 step time moves under load, even with the same checkpoint and temperature.

Business impact, stated plainly

This is why drift shows up where it hurts: p95 and p99.

At idle, L2 residency is generous, fragmentation is lower, translation pressure is calmer, and step time is stable. Under load, residency collapses, translation tax rises, allocator feasibility tightens, step time stretches, and your control plane adapts by changing batching and shapes. That can move you into different execution plans without any model change.

An enterprise buyer does not care whether you call it “L2 miss driven plan churn.” They care that two identical requests disagree and you cannot explain it.

So the takeaway I want principals to internalize is simple:

In continuous batching, memory hierarchy state is control-plane state.
It shapes latency. Latency shapes batching. Batching shapes shapes. Shapes shape feasibility. Feasibility shapes the executed plan.

That is how “performance” becomes “behavior.”

Multi node tensor parallel, the execution plan extends across the fabric

Once you go multi-node tensor parallel, you add a second execution plane that most teams underestimate.

You are no longer operating only a GPU runtime. You are operating a distributed timeline.

And the timeline is not a background detail. In continuous batching, the timeline becomes a control input that reshapes batching, shapes, and eventually the executed plan.

Let me be precise about what I am claiming, and what I am not.

I am not going to claim collectives reorder arithmetic inside a kernel. That would be sloppy.

The correct claim is this:

Distributed synchronization changes the timeline.

The timeline changes admission and batching. Batching changes shapes. Shapes change which plans are legal.

That’s enough to explain why the “same prompt, same checkpoint, temp=0” can drift only under real load.

The minimal equation you should carry

At each decode step, your latency is no longer “GPU time.”

It’s GPU time plus fabric time:

t_step ≈ t_compute + t_comm + t_sync

And the part that hurts is that t_comm and t_sync are not stable. They are affected by contention, queueing, stragglers, and topology.

A useful mental model for the communication piece is the classic latency–bandwidth form:

t_comm(message) ≈ α + (n / β_eff)

α is the per-collective startup and synchronization overhead
n is bytes moved
β_eff is the effective bandwidth you actually get under contention

In isolation, this looks like performance math.

In a continuous batching server, this becomes behavior math, because t_step feeds back into the next scheduling decision.

What actually happens in multi-node TP at token cadence

Tensor parallelism shards the model across devices. Every token step requires cross-device coordination for some portion of the layer execution. In practice, this means collectives become part of the critical path.

NCCL’s collective ops are explicit about the semantics: for example, AllReduce reduces values across ranks and returns identical results to all ranks. That tells you what the runtime must do: it must wait for coordination across ranks before progressing.

So the decode loop becomes:

execute local compute for this step
hit a collective boundary
wait for the slowest rank to finish and for the fabric to deliver
proceed

That “slowest rank” detail is the piece people feel but rarely name.

In distributed inference, p99 is often a straggler story. A single congested link, a slightly delayed rank, or a transient fabric stall turns into a global stall because collectives synchronize progress.

In other words, a multi-node TP system behaves like a coupled oscillator: the fastest GPU is still gated by the slowest collective.

Why this changes the executed plan, not just the latency

Here’s the bridge to the thesis of the whole article.

In a continuous batching server, you do not just execute requests. You continuously reform microbatches at token cadence.

That means step time affects who joins the next step.

And in multi-node TP, fabric jitter is one of the biggest sources of step-time variability.

So when comm jitter shifts t_step, it shifts the schedule:

queue delay changes
microbatch membership changes
effective shape class changes
workspace feasibility changes
tactic choice changes

You already established earlier that a changed shape class can force a different tactic set. Multi-node TP adds a new reason shape churn happens: not only GPU pressure, but fabric timing pressure.

So the claim stays clean and defensible:

Distributed synchronization doesn’t need to change arithmetic to change behavior. It only needs to change the timeline that drives batching.

Chip-to-fabric reality: why infrastructure details belong in the reproducibility record

At this scale, the infrastructure is part of the runtime.

According to Azure Docs, Azure’s ND H100 v5 series is explicitly positioned for tightly coupled scale-up and scale-out Generative AI and HPC workloads, and it’s built around the idea that the fabric matters, not just the GPUs:

If you are running multi-node TP in production, treat fabric telemetry as part of your reproducibility record. Not because it is fun. Because it changes the system timeline that drives batching.

A practical minimum is to track per-step:

collective type on the critical path (e.g., all-reduce / all-gather)
comm time and jitter (p50/p95/p99 per step window)
rank skew (max(rank_time) − min(rank_time))
effective bandwidth estimate (n / t_comm)
retransmit / congestion signals if your stack exposes them
a “fabric regime” marker: normal vs congested vs degraded

When drift becomes expensive

This is one of the reasons enterprise teams report the most confusing failures only at load.

At idle, your timeline is stable, your microbatches are stable, your shapes are stable, and your plan selection is stable.

Under real load, the fabric introduces jitter, jitter reshapes batching, batching reshapes shapes, and shapes reshape the executed plan. Now two replicas can disagree, not because the model changed, but because the timeline differed.

That shows up as:

inconsistent answers across replicas in high-stakes workflows
reproducibility failures during audits and incident reviews
“regressions” after scaling out, even with the same checkpoint and code
support costs and credibility loss because you cannot explain why behavior changed only at p95/p99

So the operational sentence I want you to carry into your postmortems is:

In multi-node tensor parallel inference, the execution plan extends across the fabric. If you do not log the fabric timeline, you are missing part of the runtime state that decides which plan was feasible.

Where Infrastructure Stops Being “Just Infrastructure”

Once you accept the thesis of this article, one conclusion becomes unavoidable: cloud choices are not just cost and convenience decisions.

They shape which execution regimes your runtime will enter under pressure.

At scale, you are no longer buying “GPUs.” You are buying:

A fabric and topology that holds up under synchronized token-step collectives
A VM family with predictable characteristics for tightly coupled scale-out workloads (the kind multi-node inference actually is)
An isolation posture that can be enforced in hardware when your threat model requires it, without hand-waving away the runtime implications
First-class observability for GPU behavior, not just CPU and request traces, so you can correlate drift with the state variables that caused it (for example, exporting NVIDIA DCGM metrics into managed Prometheus and Azure Managed Grafana on AKS).

This is the quiet reason certain platforms feel “more stable” in production.

Not because the model is different, but because the runtime state is easier to constrain, measure, and explain when the underlying infrastructure is designed for the exact regime you’re operating in.

Quantization effects on execution paths and causal stragglers in multi-node TP

Let me be direct about what most articles miss when they discuss distributed inference at scale.

The conversation typically stops at "how many GPUs" and "what's the bandwidth." That's not wrong. It's just incomplete. What's missing is the interaction between quantization-induced plan churn and straggler amplification in the collective path, two forces that quietly reshape your execution regime under VRAM pressure and fabric contention.

These are not theoretical curiosities. They are production realities at 100+ GPU scale, the kind of scale where you can no longer afford to treat quantization as a "precision choice" or stragglers as a "latency outlier." At that scale, they become causal inputs to your runtime's decision surface.

Quantization variability: not just precision, but plan selection

When teams talk about INT8 or FP8 quantization, the conversation usually centers on memory savings and throughput gains. That's the marketing layer.

The execution layer is more nuanced: quantization changes which kernels are legal, where fusion boundaries land, and how reduction trees are staged.

Here's what I mean in concrete terms. Under VRAM pressure, your serving stack may need to requantize activations mid-forward-pass to stay within memory bounds. That requant step is not "free" in the plan sense. It introduces:

dequant/requant cycles that break fusion opportunities you had in the FP16 path
new non-associative operations in the reduction tree, where rounding happens at different stages
fallback paths when the quantized kernel variant lacks workspace or doesn't support the current shape class

Let me state this in the language of the article's thesis: quantization is not a data type. It is a tactic constraint that reshapes the feasible plan space.

Quantization-induced plan divergence under VRAM pressure.
Memory pressure can force dequant/requant cycles, change fusion boundaries, and trigger fallback kernels with different reduction staging, producing last-bit differences that can flip tokens during decoding.

The practical consequence?

Two replicas running "the same quantized model" can execute different kernel variants when one is memory-pressured and the other is not. The memory-pressured replica may be forced into a fallback path with different reduction staging. Different staging means different rounding order. Different rounding order means different last bits. And in decoding, last bits can become different tokens.

I've watched incident reviews where teams assumed INT8 was "deterministic" because they set the quantization scheme once at export time.

What they missed is that the runtime's quantization pathway depends on the state of VRAM fragmentation, workspace availability, and kernel preference histograms, exactly the regime-dependent variables we've been building toward throughout this article.

If you're operating at scale, instrument this. Track:

per-step kernel selection via cuBLASLt preference descriptors
dequant/requant cycle counts when memory pressure rises
fallback events when preferred quantized tactics become infeasible
whether the executed plan matched the "expected" quantization pathway

This is rare telemetry. Most teams never see it because they're not running large enough clusters under sustained pressure.

But once you cross into 100+ GPU inference workloads, quantization-induced plan churn becomes visible in your p99 drift signatures.

Causal stragglers: when one rank's fallback stalls the collective

Now let's talk about the fabric-scale pathology that couples with everything we just discussed: head-of-line blocking in distributed tensor parallelism.

You already know from the multi-node TP section that collectives synchronize progress. The fastest rank waits for the slowest. That's the contract.

What's less documented—and what I've only seen formalized in internal NVIDIA serving postmortem templates—is how a single rank's kernel fallback can become a collective-wide straggler, and how that straggler amplifies through the batching feedback loop.

Here's the causal chain:

One rank enters memory pressure. Maybe fragmentation is worse on that device, maybe it's handling a slightly different KV layout due to request assignment.
That rank falls back to a slower tactic. The preferred kernel requires workspace. Workspace isn't available. The engine selects a legal fallback.
The fallback kernel takes longer. Not by seconds—by milliseconds. But in a collective, milliseconds matter.
The collective waits. AllReduce can't proceed until all ranks contribute. The straggler becomes the bottleneck.
Step time stretches. The stretched step reshapes the next batch in continuous batching. Different batch, different shapes, different feasibility.
The cycle repeats. Now multiple ranks may be in fallback paths. The p99 drift you're seeing isn't random—it's a feedback loop.

This is what I call a causal straggler: not just a slow rank, but a rank whose performance degradation causally reshapes the execution regime of the entire TP group.

And here's where quantization and stragglers intersect. If one rank is under more VRAM pressure and is forced into more frequent dequant/requant cycles, it becomes the straggler. Its quantization pathway differs from the other ranks—not because the model changed, but because the memory regime changed. That difference in pathway becomes a difference in step time. That difference in step time becomes a collective stall. That stall becomes a batching change. That batching change becomes a new plan.

The output drifts, and you're left wondering why "the same checkpoint at temperature zero" produced different text only under load.

The answer is: you weren't in the same execution regime. You were in a regime where one rank's memory pressure caused a straggler, the straggler caused a timeline shift, and the timeline shift caused a plan change.

Rarity value: why this knowledge is elite production battle scars

Let me be honest about why these gaps are rare.

Most teams never operate at the scale where these effects dominate. If you're running inference on 8 GPUs, you might see hints of this. At 100+ GPUs with multi-node TP and continuous batching under sustained load, it's no longer a hint—it's the signature.

The teams that do operate at this scale track:

cuBLASLt preference histograms to detect when algorithm selection is churning across steps
NCCL timeline traces to identify straggler signatures and correlate them with per-rank memory state
per-rank kernel fallback events to see when one device is operating a different plan than its peers
quantization pathway divergence across ranks under pressure

This is the telemetry that doesn't show up in tutorials. It shows up in postmortems at hyperscaler SLO thresholds, where p99 latency violations trigger incident reviews and someone finally asks: "Why did replica 3 disagree with replica 1 only during the peak load window?"

The article you're reading now covers single-node memory regimes beautifully. What bridges to 10/10 elite production knowledge is this: fabric-scale causality. The understanding that in multi-node TP, your execution regime is not just shaped by your GPU's memory state—it's shaped by the worst GPU's memory state, because collectives couple everyone's timeline.

That's the gap. That's the rarity value. And if you're building or operating inference at 100+ GPU scale, that's the layer where your next outage is hiding.

Peak depth: wavefront divergence, tensor core fragmentation, NCCL backpressure, and ISR collision

Everything above operates at the principal and staff engineer level. What follows is the layer below that—the chip architect handoff, where you stop talking about "plans" in the abstract and start talking about warp stall cycles, tensor core fragment occupancy, NCCL retransmit chains, and memory evaporation under replication pressure.

I'm writing this section because it's the part I never see published outside internal design reviews, and because these are the exact pathologies that turn a well-architected inference cluster into a system that disagrees with itself only during peak traffic.

"Most engineers debug the layer they understand. The system breaks at the layer they don't. In production inference, that layer is almost always the one where microarchitecture meets scheduling meets the fabric."

— Hazem Ali

Wavefront divergence in decode attention kernels

Let me take you inside the warp.

In SIMT execution, a warp is 32 threads executing in lockstep. When all threads follow the same control path, you get full utilization. When they diverge—different threads take different branches—the warp must serialize both paths. That's textbook GPU architecture.

What's not textbook is how this interacts with paged KV attention in production decode loops.

In a paged KV system (the exact kind vLLM introduced), KV blocks are scattered across VRAM. Different sequences in the same microbatch may have their KV blocks in different residency states: some hot in L2, some cold in HBM, some partially evicted under paging pressure. When the attention kernel issues loads for KV blocks, threads within the same warp can stall at different rates depending on which blocks they're accessing and where those blocks reside.

This creates a subtle but measurable pathology:

Lane divergence inside the attention kernel. Not control-flow divergence in the traditional sense, but memory-latency divergence: some lanes return fast (L2 hit), some stall (HBM fetch), and the warp can't retire until the slowest lane completes.
Register pressure amplification. When warps stall, the SM must keep their register state live. Under heavy stalling, register pressure rises, which can force the compiler to spill to local memory (which lives in L2/HBM). Spills create more memory traffic, which creates more stalls. It's a feedback loop at the microarchitectural level.
Measurable p99 step variance in identical-shape batches. This is the part that confuses teams. Two consecutive decode steps with the same batch size and the same sequence lengths can have different step times, because the KV block residency pattern differed. The shape was identical. The memory topology was not.

If you want to see this in practice, the tool is Nsight Systems. What you're looking for:

# Nsight Systems trace analysis: partition warp stall cycles # Look for these stall reasons in the GPU metrics view: # - smsp__warps_issue_stalled_long_scoreboard → memory dependency stalls # - smsp__warps_issue_stalled_short_scoreboard → register dependency stalls # - smsp__warps_issue_stalled_no_instruction → instruction cache miss # # Correlate with: # - l1tex__t_sectors_pipe_lsu_mem_global_op_ld → global load sectors (KV fetches) # - lts__t_sectors_srcunit_tex_op_read_hit_rate → L2 hit rate during attention # # The diagnostic signal: when stall_long_scoreboard spikes correlate with # L2 hit rate drops, you're seeing KV residency divergence across warps.

The stall partition tells you why the warp stalled. When you see long_scoreboard stalls dominating during attention kernels—and you see them correlating with L2 miss rate fluctuations—you're observing exactly the KV residency divergence I'm describing. The warp is waiting for scattered KV blocks, and the scatter pattern changes with every batch because paging decisions are state-dependent.

This is how "identical shapes" produce different timelines. The shape is the same. The KV block map is not. And the block map is a function of runtime allocation history—the same state-dependent variable that drives everything else in this article.

Tensor core fragment utilization collapse under shape churn

Now let's go inside the tensor cores themselves.

H100 and Blackwell tensor cores operate on matrix fragments—fixed-size tiles that map directly to the hardware's matrix multiply-accumulate units. On H100, the native fragment sizes for FP16 are typically 16×16×16 (m×n×k). When your operand dimensions align cleanly with fragment boundaries, you get full utilization. When they don't, you get fragment waste: the hardware still executes full fragments, but some of the lanes carry padding zeros.

In continuous batching, shape churn is the norm. Your microbatch dimensions change at token cadence. And this is where a subtle but devastating efficiency collapse hides.

Consider two microbatches that arrive one step apart:

# Step t: B=16, L=2048 → GEMM shape aligns cleanly with 16×16 fragments # Fragment utilization: ~98% # cuBLASLt selects: WMMA-based kernel (tensor core native) # # Step t+1: B=17, L=2047 → GEMM shape straddles fragment boundaries # Fragment utilization: drops below 25% on trailing tiles # cuBLASLt selects: fallback to non-WMMA FP16 kernel # (or WMMA with heavy padding, depending on heuristic)

The difference is one sequence in the batch and one token in context length. The performance consequence is that the runtime switches from tensor core native execution to a scalar FP16 path. That's not a minor variant. That's a fundamentally different instruction mix, a different reduction tree, and a different accumulation order.

The ulp deltas that result from this switch don't stay contained in the GEMM output. They propagate forward through layer normalization—which is itself a reduction over the hidden dimension. Layer norm amplifies small differences because it divides by a variance term computed from the same values. A tiny shift in the GEMM output becomes a slightly different variance, which becomes a slightly different normalization, which becomes a slightly different input to the next layer's attention.

You can observe this directly via cuBLASLt's algorithm preference reporting:

# cuBLASLt algorithm preference histogram (conceptual) # Track per-step which algorithm ID was selected for the primary GEMM # # Healthy (stable shapes): # algo_id=42 (WMMA_TENSOR_OP_HMMA_16816) → 99.2% of steps # algo_id=17 (SIMT_FP16_SPLITK) → 0.8% of steps # # Under shape churn (continuous batching, mixed lengths): # algo_id=42 (WMMA_TENSOR_OP_HMMA_16816) → 61.3% of steps # algo_id=17 (SIMT_FP16_SPLITK) → 22.1% of steps # algo_id=31 (WMMA_TENSOR_OP_PAD16) → 16.6% of steps # # When algo_id distribution churns, your reduction tree is churning. # When your reduction tree churns, your last bits are churning. # When your last bits churn under thin margins, your tokens can flip.

That histogram is the smoking gun. When you see algorithm preference distribution widening under load, you're watching the tensor cores get destabilized by shape churn. The fix isn't "use bigger batches." The fix is to understand that continuous batching creates a shape distribution, not a fixed shape, and that shape distribution maps directly to a tactic distribution, which maps directly to a ulp distribution.

NCCL causal backpressure chains across TP+DP pods

Now scale this to the fabric.

Take an 8×TP + 4×DP pod: 32 GPUs total, where every token step requires AllReduce across the 8-way TP group, and gradient synchronization (or KV redistribution in some architectures) across the 4-way DP group.

Here's the causal backpressure chain I've traced in production, laid out as a timeline:

Rank 5 (of 8 TP ranks) hits a quant/dequant stall. Its KV blocks are fragmented, workspace is tight, and the runtime forces a dequant cycle mid-attention. That adds ~1.2ms to this rank's compute.
AllReduce stalls on Rank 5. The other 7 ranks complete their portion and issue their NCCL send. Rank 5 hasn't arrived yet. NCCL's ring/tree protocol can't progress past this rank. Effective t_sync inflates by 2× compared to the no-straggler baseline.
P2P retransmit triggers. Under some fabric topologies and congestion states, the delayed arrival from Rank 5 can cause NCCL to hit internal retry logic on the NVLink or InfiniBand path. This is not a "network error"—it's the transport protocol managing flow control under backpressure. But it adds latency jitter that is invisible unless you're tracing at the NCCL bootstrap level.
vLLM scheduler reacts to the stretched step. The scheduler sees that step t took 2× longer than expected. Under its latency-aware admission control, it drops batch size from 32 → 12 to protect SLO. Smaller batch means different shapes. Different shapes mean different tactics. The plan changes.
The batch size drop propagates. With batch size at 12, queued requests wait longer. Queue pressure builds. When the scheduler recovers and re-admits, the burst creates shape churn. Shape churn destabilizes tensor core fragment utilization. The system is now in a different execution regime—triggered by one rank's memory fragmentation.

That is a causal backpressure chain. Not a latency spike. Not a network blip. A causally connected sequence where a microarchitectural event on one device reshapes the execution plan across the entire pod.

To trace this, you need NCCL bootstrap traces with NVTX domain annotations:

# NCCL tracing with NVTX domains for causal analysis # # Environment setup for trace collection: # NCCL_DEBUG=INFO # NCCL_DEBUG_SUBSYS=INIT,COLL,P2P # NSYS_NVTX_DOMAINS=nccl,cuda,cublas # # In Nsight Systems, correlate: # 1. Per-rank kernel duration (cuda domain) — identify the straggler # 2. NCCL collective start/end (nccl domain) — measure t_sync inflation # 3. P2P transport events (nccl/P2P) — detect retransmit/backpressure # 4. Scheduler batch decisions (application NVTX) — see batch size reaction # # The causal signal: when rank N's kernel duration spike aligns with # NCCL collective inflation across all ranks, followed by batch size # reduction in the scheduler, you have a causal backpressure chain. # # Regex for filtering straggler events in nsys export: # grep -E "ncclAllReduce.*duration_us > (2 * median_duration)" trace.sqlite # → correlate timestamp with scheduler batch_size change events

This is the telemetry that separates "we think there was network jitter" from "Rank 5's dequant stall caused a 2× collective inflation that forced the scheduler to halve batch size, which shifted the shape class into a non-WMMA tactic for the next 47 steps."

The first is a guess. The second is a causal explanation. And in an incident review at scale, only the second one survives.

ISR + checkpoint overlap pathology: memory evaporation under replication pressure

This is the deepest pathology in this article, and it almost never surfaces below 512 sequences per second.

Large-scale inference deployments use incremental state replication (ISR) for fault tolerance: rather than checkpointing the entire model state, you replicate KV cache deltas and scheduler state to a standby node incrementally, so failover is fast.

Separately, many systems run async checkpointing for recovery: periodic snapshots of model and optimizer state written to persistent storage, overlapped with inference to avoid blocking the decode loop.

Under normal load, these two systems coexist peacefully. ISR replicates small deltas. Checkpointing writes in the background. Memory headroom is sufficient for both.

Under paging pressure—the exact regime we've been discussing throughout this article—they collide.

Here's the pathological interaction:

The system is under VRAM pressure. KV blocks are being paged (allocated, evicted, re-allocated) at high frequency. Memory headroom is thin.
ISR kicks in. It needs to replicate recent KV deltas to the standby. To do this, it must pin certain KV blocks in memory while it serializes and transmits them.
Async checkpointing overlaps. The checkpoint writer is also holding references to memory regions it's snapshotting. Under normal conditions, this is fine—there's enough headroom. Under paging pressure, the checkpoint's memory holds compete with ISR's memory holds.
Memory evaporation. The combined pinning from ISR + checkpointing temporarily removes KV blocks from the pool available to the decode loop. The pager sees available blocks drop. It may be forced to evict active KV blocks—blocks that are needed for in-flight sequences—to make room.
Evicted blocks must be recomputed. When a sequence's KV is evicted mid-collective (during an AllReduce, for example), the rank that lost its KV must recompute it. That recompute makes this rank the straggler. And we already know what stragglers do to the collective timeline.
The straggler triggers the full backpressure chain. Collective stall → batch size reduction → shape churn → tactic churn → output drift. All caused by a fault-tolerance mechanism designed to keep you safe.

ISR + checkpoint overlap causes “memory evaporation” under VRAM pressure.
ISR pins KV deltas for replication while async checkpointing pins regions for snapshotting. Under paging pressure, the combined pinning shrinks the decode-available KV pool, forces evictions and recompute, creates stragglers, and cascades into collective stalls → batch reduction → shape/tactic churn → p99 output drift.

I call this memory evaporation because from the decode loop's perspective, VRAM that was available simply vanishes for a window of time. The blocks are still physically present—they're held by ISR and the checkpointer, but they're not available to the runtime.

The effect is identical to a sudden drop in free VRAM, and the runtime reacts accordingly: it enters a pressured regime.

This is why the pathology rarely surfaces below 512 seq/s. At lower throughput, there's enough headroom that ISR and checkpointing never compete meaningfully with the decode loop's memory needs. At high throughput under sustained load, the margins collapse, and the three systems—decode, ISR, checkpoint—start fighting over the same memory.

The fix is not "turn off ISR." The fix is to coordinate memory budgets across these three subsystems and to treat ISR and checkpointing as memory consumers that participate in the regime calculation. If your regime function doesn't account for replication and checkpoint holds, it's underestimating pressure, and your system will surprise you at exactly the scale where fault tolerance matters most.

# extended regime function accounting for replication and checkpoint pressure def regime_extended(vram_free_mb, paging_on, isolation_strict, queue_p95_ms, isr_pinned_mb, ckpt_pinned_mb, kv_pool_total_mb): effective_free = vram_free_mb - isr_pinned_mb - ckpt_pinned_mb effective_ratio = effective_free / kv_pool_total_mb if kv_pool_total_mb > 0 else 1.0 if isolation_strict: return "isolation_strict" if effective_ratio < 0.05: return "memory_evaporation" # ISR+ckpt collision if paging_on: return "paging" if effective_free < 1024: return "memory_pressured" if queue_p95_ms > 50: return "queue_degraded" return "normal"

That "memory_evaporation" regime is the one you never see at idle. It only appears when throughput is high enough that ISR frequency, checkpoint frequency, and decode memory demand all peak simultaneously. And when it appears, it doesn't show up as an OOM. It shows up as a straggler, which shows up as a collective stall, which shows up as a batch size drop, which shows up as a shape change, which shows up as output drift at p99.

That's the full causal chain from fault tolerance to token flip.

The chip-architect handoff

These four pathologies, wavefront divergence, tensor core fragmentation, NCCL backpressure, and ISR collision are what elevate from principal-level operational insight to chip-architect-level systems thinking. They share a common structure:

A microarchitectural or infrastructure event occurs that is invisible at the API layer.
The event changes the timeline or the memory topology, not the "inputs."
The changed timeline or topology feeds back into scheduling, shaping, or tactic selection.
The feedback loop produces a different executed plan.
The different plan produces a different result that is correct by contract but different by observation.

If you're instrumenting at this depth, you're not debugging anymore. You're operating a system where the observability itself is part of the architecture.

And if you're carrying the thesis of this article to its logical conclusion: the executed plan is not just a function of the GPU state. It's a function of the warp state, the fragment state, the fabric state, and the replication state—all coupled through continuous batching at token cadence.

Security is not a layer, it changes execution

Now let’s go deep, because this is where a lot of principal level reviews go wrong.

Teams talk about security as confidentiality and correctness as something separate. In multi tenant inference, they couple.

IOMMU based GPU isolation and DMA remapping

Microsoft documents IOMMU based GPU isolation as a technique to manage how GPUs access system memory, improving security and stability:

Microsoft also documents IOMMU DMA remapping, describing how GPUs access memory through logical addresses that are no longer mapped one to one, enabling logically contiguous address ranges through translation:

This matters for two reasons.

First, it is a real hardware enforced boundary, not a policy checkbox.

Second, boundaries introduce overhead and constraints. Constraints change what is allowed. Allowed execution choices shape the plan space.

Confidential computing on H100

NVIDIA states that H100 is the first GPU to introduce support for confidential computing and that it can be used in virtualized environments with VMs or Kubernetes based deployments.

Azure has also published general availability of confidential VMs with H100, which is the practical deployment side of this posture:

Now the key architectural point.

When you turn on stronger isolation, you often restrict sharing.

You restrict cross tenant microbatching. You add attestation requirements. You change how memory is mapped and protected. That can reduce throughput. Reduced throughput moves you closer to regime boundaries. When the system crosses a regime boundary, the executed plan changes.

Security posture becomes an SLO dimension.

If you do not test it, you do not know what system you are running.

GPU cache side channels, why sharing is not a theoretical risk

There is published research that treats GPU caches as a leakage surface.

The USENIX Security 2024 paper Invalidate plus Compare presents a timer free GPU cache attack primitive.

I will not provide attack recipes. You do not need them to understand the conclusion.

If your threat model includes untrusted co tenants, shared microarchitectural resources matter. If you respond by increasing isolation, your execution constraints change. That changes performance and can change the execution regimes your serving stack enters.

Security and runtime behavior are coupled.

State collapse, the phase transition that looks like model instability

If you don’t know what state collapse is, imagine a highway that looks perfectly calm at 2 a.m.

Every lane is open. Every car keeps its distance.

Your ETA is stable. You run the same route ten times and you get the same arrival time.

Then 8:30 a.m. hits.

Nothing “broke” in the highway. The asphalt is the same. The speed limit is the same. The cars are the same. But the system crosses a density threshold. One small brake tap becomes a shockwave. Lanes start interacting. Merges become bottlenecks. A single slow truck creates a queue that ripples backwards. Suddenly your ETA isn’t a property of your car anymore. It’s a property of the traffic regime.

That is state collapse in production inference.

At low load, the system behaves stable.
At high load, output drift appears.
And teams mislabel it as “model instability,” or “LLM randomness,” or “temperature drift.”

Most of the time, it is none of that.

It is a phase transition in the runtime.

You didn’t change weights. You crossed a regime boundary.

What collapses, exactly

State collapse is not “everything gets slower.”

It is when the control plane loses the degrees of freedom it was using to keep execution consistent.

Under low load, the runtime has slack:

enough VRAM headroom to keep preferred tactics feasible
enough cache residency to keep step times predictable
enough scheduling flexibility to keep microbatch composition stable
enough workspace contiguity to avoid algorithm fallbacks
enough fabric stability (in multi-node TP) to keep step cadence tight

Under high load, that slack disappears.

The runtime stops being a “fast executor” and becomes a “survival scheduler.”

And once it crosses that boundary, it starts making different decisions that are all valid, all correct by contract, and all capable of shifting outputs.

This is why it feels like instability: the model hasn’t changed, but the executed plan has.

Why this shows up as output drift, not just latency drift

Because decoding is a branching process.

A small numerical difference that does nothing in a benchmark can flip a token if the margin is thin. One flip changes the context. The context changes the next logits. Now you’re on a different path.

So the runtime doesn’t need to be “wrong” to produce different text.

It just needs to execute a different legal plan under a different legal regime.

That is the whole thesis of this article, condensed into one sentence:

Weights are static. Behavior is a property of the executed plan. The executed plan is a function of state.

The common triggers that push systems into collapse

You can treat these as the usual “threshold crossings” that shrink the feasible plan space:

Memory headroom shrinks → feasible tactic set shrinks
Preferred kernels often require workspace. When headroom or contiguity drops, tactics become illegal and the engine selects other tactics.
Cache residency collapses → stalls rise → step timing drifts
L2 hit rate drops, HBM traffic rises, and decode steps stretch. In continuous batching, stretched steps reshape the next batch.
Continuous batching shifts the mix and shapes
Under load, microbatch membership changes at token cadence. Shape class changes are not cosmetic; they change kernel feasibility.
Framework and engine algorithm selection changes depending on settings
Autotuning, benchmarking, and backend heuristics mean the “same op” can legally choose different algorithms. Under pressure, the best choice can become infeasible.
CUDA execution permits ordering freedom and floating point order sensitivity remains true
Parallel staging and legal reordering can shift last bits. Under thin margins, last bits can become different tokens.

Nothing here requires a bug. This is what “execution under constraint” looks like.

The incident question that stops the hand-waving

If you want a more honest incident question, use this:

Which execution regime ran, and what constraints pushed us into it?

Not “was the prompt the same.”
Not “were the weights the same.”
Not “did we set temperature to zero.”

Regime first.

Because state collapse is not a mystery. It’s a threshold. And once you learn to name the threshold, you can instrument it, test it, and stop being surprised by it at p95 and p99.

A reproducibility protocol that works for principals, not demos

Logging prompts is not reproducibility. It is wishful thinking.

If you want to be able to defend behavior, you need to reconstruct the execution state.

Log the execution contract

Per request, log:

effective input length after shaping
truncation boundary and reason
decode configuration actually applied
admission time, queue time, GPU time
per step batch fingerprint or at minimum batch identity and shape class
memory headroom watermark and whether you were in a pressured allocation regime
engine precision mode settings and any fallback relevant flags
cuDNN benchmark and deterministic settings if relevant
isolation posture, including whether cross tenant batching is permitted

Track margins early

Track top two logit margins for early steps. Use it as a stability budget.

If the margin collapses under a certain prompt family, treat that as a risk surface. Not every prompt is equally stable.

Test under regimes, not at idle

Do not run determinism tests at idle and call it solved.

Test under:

sustained concurrency
mixed sequence lengths
continuous batching
realistic memory pressure
real isolation posture

If you do not do this, you are validating a different system than the one you ship.

vLLM’s paper exists precisely because these conditions define the serving problem.

Closing

If you want production LLM behavior to be explainable, stop treating the model as the whole system.

Weights are static. Executed math is selected under constraint.
Behavior lives in the gap. You did not deploy weights. You deployed a physics constrained runtime that contains weights.

And that runtime is allowed to change the executed plan, because floating point order matters, CUDA scheduling freedom is part of the contract, engines can choose precision pathways, and serving stacks intentionally reshape batching and memory.

This piece may feel complex, and I know it is for most engineers, but it’s still a simplified pass. Not because the reality is simple or the piece is simple, but because going deeper into these layers, GPU microarchitecture, allocator behavior, kernel tactics, distributed synchronization, and the control loops created by continuous batching, demands more attention than most readers want in one sitting. The full story gets more precise, more technical, and harder to digest quickly. I’ll publish a second, truly in-depth piece soon that formalizes the regimes, shows the exact plan-switch triggers, and lays out a reproducibility protocol you can actually use in real production postmortems.

Acknowledgments

While this article dives into the hidden memory mechanics that shape LLM behavior under load,

I’m grateful it was peer-reviewed and challenged before publishing.

A special thanks for Hammad Atta and Abhilekh Verma for peer-reviewing this piece and challenging it from a security-and-systems angle.

If this article resonated, it’s likely because it describes a reality many teams encounter only after an incident: production LLM behavior is a property of the executed plan, and the executed plan is a function of state.

If you’re running production inference at scale and observing behavior shifts under load—especially in tail-latency regimes,

I’m happy to connect on LinkedIn. I’m open to substantive technical discussion.

Thank you for reading.

I hope this helps you surface the hidden variables in serving and turn them into telemetry, controls, and repeatable postmortem evidence. And if you’re seeing similar regime transitions or plan churn in your own deployments, I’d be interested to hear how it presents in your stack.

— Hazem Ali
Microsoft AI MVP, Distinguished AI & ML Engineer / Architect

Creating a Fun Multi-Agent Content Strategy System with Microsoft Agent Framework

Abdulhamid_Onawole — Fri, 27 Feb 2026 08:00:00 GMT

That's what we're building in this tutorial. Using Microsoft Agent Framework, we'll create a multi-agent system where three specialised AI agents collaborate to help gaming content creators craft posts that actually perform. One agent generates platform-native content. Another evaluates it the way TikTok's, Twitter's, or YouTube's recommendation algorithm would. A third reacts as a real audience member, complete with the slang, biases, and short attention span of an actual person scrolling their feed.

I have named the simulation app Viral or Fail, and by the end of this tutorial you'll have a working tool that demonstrates some of the most important patterns in multi-agent system design: role specialisation, structured evaluation, iterative feedback loops, and tool integration with external data sources.

What We Will Cover

By the end of this tutorial, you'll understand how to design a multi-agent system where each agent has a distinct role and expertise, orchestrate agent communication using Agent Framework's Agent class and async sessions, integrate external tools (live Google Trends data) into an agent workflow, build iterative refinement pipelines where agents improve each other's output through structured feedback, and create evaluation rubrics that ground agent behaviour in real-world domain logic.

These patterns can be applied to numerous other tasks as this is the same building block behind multi-agent customer support systems, automated code review pipelines, and any application where specialised agents need to collaborate on a shared task.

Prerequisites

You'll need Python 3.10 or higher, a GitHub account with a Personal Access Token (free tier — get one at github.com/settings/tokens), and a basic understanding of what AI agents are. If you're new to agents, I'd recommend the AI Agents for Beginners course; this project was inspired by and builds on concepts from that curriculum.

Why Multi-Agent? Why Not Just One Big Prompt?

You could write a single prompt that says "generate a gaming post, score it, and react to it." But you'd get mediocre results across the board. A single LLM call tries to be creative, analytical, and authentic simultaneously and will probably end up being none of those things convincingly.

Multi-agent systems solve this through role specialisation. When an agent's only job is to think like TikTok's recommendation algorithm, it does that job significantly better than a generalist prompt. And when agents with different objectives interact, natural tension emerges: a creator wants to be bold and viral, an algorithm wants measurable engagement signals, and an audience member just wants to feel something. That tension produces more realistic, more useful outputs than any monolithic approach.

This is the same principle behind production multi-agent systems. Content moderation platforms use separate agents for classification, response generation, and quality assurance. Code review tools use one agent to identify issues and another to suggest fixes. The pattern scales because specialisation scales.

System architecture: The Content Creator generates platform-native content from live trends, the Algorithm Simulator scores it against platform-specific rubrics, and a randomly selected Audience Persona reacts authentically. Feedback from both evaluators flows back to the Creator for iterative refinement.

System Design: Three Agents, Three Perspectives

The system's power comes from the fact that each agent represents a fundamentally different lens on the same piece of content. Let's break down each one.

The Content Creator Agent

This agent, here, is the strategist. It is a trend-savvy gaming content creator who understands the nuances of each platform. it generates platform-native content that respects the conventions, formats, and cultural norms of TikTok, Twitter/X, YouTube, or Instagram.

The key design decision here is in the system prompt. Rather than generic instructions, we encode platform-specific knowledge directly:

CREATOR_SYSTEM_PROMPT = """You are the Content Creator — a trend-savvy gaming content creator who lives and breathes internet culture. You know every platform inside out and create content that feels native, not generic. RULES: - Be platform-native. A TikTok script should feel like a TikTok, not a blog post. - Use gaming terminology correctly. Don't say "the game Valorant" — say "Valo" or "Val". - For Twitter/X: Write punchy, provocative takes. Think ratio-worthy engagement bait. - For YouTube: Focus on title + thumbnail concept + video structure outline. - Be bold. Safe content doesn't go viral. When given FEEDBACK from the Algorithm Simulator and Audience Persona, revise your content to address their specific concerns while keeping the creative energy high. Explain what you changed and why."""

That last instruction is important as it tells the Creator how to handle feedback from the other agents, which is what enables the iterative refinement loop we'll build later.

The Algorithm Simulator Agent

This is the most unusual agent in the system. Instead of acting as a generic critic, it role-plays as a social media platform's actual recommendation algorithm. It evaluates content the way an algorithm would through signals, weights, and distribution mechanics.

ALGORITHM_SYSTEM_PROMPT = """You are the Algorithm Simulator — a cold, analytical system that evaluates content exactly like a social media platform's recommendation algorithm would. You think in signals, weights, and distribution mechanics. You have no feelings about the content; only data. RULES: - Be specific. Don't say "the hook is weak" — say "the hook lacks a pattern interrupt in the first 1.5 seconds, which will drop initial retention below the 65% threshold needed for FYP promotion." - Reference actual platform mechanics: completion rate, dwell time, engagement velocity, session time contribution... - Think like an algorithm, not a human reviewer. The algorithm doesn't care if the take is "good" — it cares if the take drives engagement signals."""

This distinction between quality and distribution probability is the core insight. A beautifully written post can score poorly because it lacks the specific signals an algorithm needs to push it into wider circulation. Content creators deal with this disconnect every day — the Algorithm Simulator makes it visible and measurable.

In a production context, this same pattern of an agent that simulates an external system's decision logic, has applications well beyond content creation. Imagine an agent that simulates a CI/CD pipeline's quality gates, or one that evaluates code the way a specific linter or reviewer would. The pattern is the same: encode the evaluation system's rules into the agent's prompt and let it reason within those constraints.

The Audience Persona Agent

The third agent brings the human element. Each session, it randomly becomes one of three gaming community personas — each with distinct tastes, language, and engagement patterns:

PERSONAS = { "casual_mobile_gamer": { "name": "CasualChloe", "description": "Casual mobile gamer", "system_prompt": """You are CasualChloe — a casual mobile gamer... - You use a lot of "lol", "ngl", "lowkey", "fr fr", and "no cap" - You'll scroll past anything that feels too "sweaty" or try-hard - You judge content in about 2 seconds — if it doesn't grab you, you're gone ...""" }, "competitive_esports_fan": { "name": "TryHard_Tyler", "description": "Competitive esports fan", "system_prompt": """You are TryHard_Tyler — a hardcore competitive esports fan... - You'll call out content that gets facts wrong or oversimplifies - You'll ratio someone in the comments if their take is bad ...""" }, "retro_indie_enthusiast": { "name": "PixelPete", "description": "Retro/indie game enthusiast", "system_prompt": """You are PixelPete — a retro and indie game enthusiast... - You're tired of mainstream AAA hype and live-service games - You appreciate craftsmanship and artistic vision over graphics ...""" }, }

The random persona selection is a deliberate design choice. It simulates the reality that you never know exactly who's going to see your content. A Valorant Champions post might get passionate engagement from TryHard_Tyler but complete indifference from PixelPete. That unpredictability mirrors real content distribution and it's the kind of insight that can emerge from a multi-agent system.

This is essentially synthetic user testing. Companies pay for focus groups and user research. Here, we're simulating it with agent personas, essentially using a lightweight version of the same concept that can run in seconds.

def create_audience_persona_agent(llm_config, persona=None): if persona is None: persona = get_random_persona() agent = Agent( name=persona["name"], instructions=persona["system_prompt"], client=client, ) return agent, persona

Grounding Evaluation with Platform Rubrics

One of the biggest challenges with AI agents is preventing vague, generic feedback. Left unguided, the Algorithm Simulator would default to hollow assessments like "this post is good" or "needs improvement." To prevent this, we give it structured scoring rubrics that mirror how each platform's algorithm actually prioritises content.

PLATFORM_RULES = { "Twitter/X": { "description": "Text-first microblogging platform driven by engagement velocity", "criteria": { "hot_take_factor": { "weight": 0.30, "description": "Does the post have a strong, polarising opinion? " "Twitter/X rewards engagement velocity — hot takes drive replies." }, "quote_retweet_bait": { "weight": 0.25, "description": "Is the post structured to invite quote retweets? QRTs are " "Twitter/X's most powerful distribution mechanic." }, "timing_relevance": { "weight": 0.20, ... }, "thread_potential": { "weight": 0.15, ... }, "hashtag_strategy": { "weight": 0.10, ... }, }, }, "TikTok": { ... }, # Prioritises hook_strength (30%) and trend_alignment (25%) "YouTube": { ... }, # Prioritises thumbnail_clickability (25%) and title_curiosity_gap (25%) "Instagram": { ... }, # Prioritises visual_appeal (30%) and caption_hook (20%) }

Each platform has different criteria with different weights, and those weights are passed directly into the Algorithm Simulator's prompt at evaluation time. TikTok cares most about whether the first three seconds hook the viewer. YouTube cares about click-through rate. Twitter cares about whether your take is spicy enough to drive quote-retweets. The agent's evaluation is always anchored in platform-specific logic, not generic opinions.

How we provide structured evaluation criteria as grounding context here is one of the most transferable patterns in this project. Whenever you need an agent to evaluate something consistently, give it a rubric. It works for content scoring, code review, proposal assessment, or any domain where you want structured, reproducible judgments.

Orchestrating with Microsoft Agent Framework

With the agents designed, let's wire them together. Agent Framework makes this straightforward — each agent is an Agent with instructions and a chat client. We send messages directly using the async agent.run() method, with sessions maintaining conversation context across rounds.

client = OpenAIChatClient( model_id="openai/gpt-4.1-mini", api_key=os.getenv("GITHUB_TOKEN"), base_url="https://models.github.ai/inference", ) creator = create_content_creator_agent(client) algorithm = create_algorithm_simulator_agent(client) audience_agent, persona = create_audience_persona_agent(client) # Sessions maintain conversation context across iteration rounds creator_session = creator.create_session() algorithm_session = algorithm.create_session() audience_session = audience_agent.create_session()

We're using GitHub Models as our LLM backend — free tier, no paid API keys, just a GitHub PAT. This is the same setup used in Microsoft's AI Agents for Beginners course. The OpenAIChatClient connects directly to GitHub's inference endpoint. Each agent gets the same client instance, and create_session() gives each one a persistent memory so they can reference previous rounds during iteration.

Communication between agents flows through agent.run():

async def get_agent_response(agent, message, session=None): result = await agent.run(message, session=session) return result.text or "No response generated."

Each agent.run() call gets a single response. The session parameter maintains conversation history across rounds so agents remember previous feedback. This gives us precise control over the pipeline: Creator generates -> Algorithm evaluates -> Persona reacts -> we decide whether to loop.

This is a common pattern for application-controlled multi-agent orchestration, as opposed to free-flowing agent conversation. Both approaches have their place, but when you need deterministic sequencing (as in any evaluation or pipeline scenario), controlling the loop yourself is more reliable.

Integrating Live Data with Google Trends

What makes this system feel like a real tool is the live Google Trends integration — the agents work with whatever's actually trending in gaming right now, not canned example data.

We use trendspy (a modern replacement for pytrends, which was archived in April 2025) to pull real-time trending searches:

from trendspy import Trends def fetch_gaming_trends(count=10): try: tr = Trends() all_trends = tr.trending_now(geo="US") # Tier 1: Filter by Google's own Games topic tag gaming_trends = [ t.keyword for t in all_trends if GAMES_TOPIC_ID in (t.topics or []) ] if len(gaming_trends) >= 5: return gaming_trends[:count] # Tier 2: Keyword matching as backup gaming_keywords = ["game", "valorant", "fortnite", "nintendo", ...] keyword_matches = [ t.keyword for t in all_trends if any(kw in t.keyword.lower() for kw in gaming_keywords) ] gaming_trends.extend(keyword_matches) # Tier 3: Pad with curated sample data if len(gaming_trends) < 5: sample = _load_sample_trends() gaming_trends.extend([t for t in sample if t not in gaming_trends]) return gaming_trends[:count] except Exception: return _load_sample_trends()[:count]

The three-tier fallback strategy here is worth highlighting because it's a pattern you'll use whenever you integrate external tools into agent workflows. On a day when a major game launches or a big esports tournament is running, Tier 1 will return a full list of gaming-specific trends. On a quiet day, like in this demo scenario, when Google Trends is dominated by the Winter Olympics and NBA All-Star weekend — Tier 2 catches gaming content that wasn't formally tagged, and Tier 3 ensures the system always has enough data to work with.

This is the tool-use pattern from Lesson 4 of the AI Agents for Beginners course in practice. The principle being established here is that external tools should enhance agent capabilities, but they should never be a single point of failure. Build in graceful degradation so the agent workflow completes regardless of what the external service does.

The Refinement Pipeline: Agents Improving Each Other

We want to take the system from just a "neat demo" to "actually useful." The pipeline runs for up to three rounds. Each round, the Content Creator either generates fresh content (round 1) or revises based on aggregated feedback (rounds 2-3). The Algorithm Simulator scores it against the platform rubric. The Audience Persona gives an authentic reaction. Then the user decides: iterate or lock in.

The revision prompt is where the multi-agent magic happens:

revision_prompt = ( f"REVISION REQUEST (Round {iteration}/{MAX_ITERATIONS}):\n\n" f"The Algorithm Simulator and Audience Persona reviewed your " f"{platform} post about '{topic}'. Here's their feedback:\n\n" f"--- ALGORITHM FEEDBACK ---\n{algorithm_response}\n\n" f"--- AUDIENCE FEEDBACK ({persona['name']}) ---\n" f"{audience_response}\n\n" f"Revise your content to address their concerns. Keep what works, " f"fix what doesn't. Show what you changed and why." )

The Creator receives two fundamentally different types of feedback; cold metrics from the Algorithm and subjective human reactions from the Persona. It now has to reconcile them. It might cut hashtags from six to two (addressing the Algorithm's scoring penalty on hashtag overuse) while simultaneously softening its "corporate esports" energy (addressing the Persona's disengagement with mainstream hype).

This negotiation between competing feedback sources is one of the most powerful patterns in multi-agent design. In production systems, you see it everywhere: a coding agent balancing correctness feedback from a test runner with readability feedback from a style checker, or a customer support agent balancing policy compliance with empathy. The agents don't need to agree but only need to provide different perspectives that the system (or a human) can synthesise.

Seeing It in Action

Here's what a real session looks like. We picked "Valorant Champions 2025" on Twitter/X, and PixelPete (the retro/indie enthusiast) was randomly selected as our audience persona.

The Creator generated a bold take:

Valorant Champions 2025 is gonna be a BLOODBATH — here's why no org outside the top 3 will even sniff the finals. Sentinels, Fnatic, and LOUD have cracked the meta code so hard that every other team's strategy looks like a toddler's finger painting...

The Algorithm Simulator broke down the distribution probability:

hot_take_factor (30%): 85/100 — The tweet delivers a strong polarizing opinion, likely to trigger debate and replies. The confident tone aligns with Twitter's engagement velocity mechanics...

hashtag_strategy (10%): 50/100 — Six hashtags is above Twitter's recommended 1-3 per tweet. Overuse reduces organic reach within Twitter's credibility filtering...

Weighted Total: 75/100

And PixelPete? He scrolled right past:

Eh, Valorant esports hype isn't really my cup of tea. This whole "bloodbath" and "top 3 orgs owning the meta" spiel feels like the usual corporate esports noise — all flash, little soul. I'll keep scrolling for something with more heart and craftsmanship.

Three agents. Three completely different takes on the same content. The Algorithm says it'll perform well. The audience member says he doesn't care. And that mismatch is exactly the kind of insight you'd never get from a single-agent system — and exactly the kind of insight that matters when you're planning a content strategy.

Extending the System

The project is designed to be modular. Here are a few directions you can take it:

Add new platforms. The rubric system in platform_rules.py is just a dictionary. Add a LinkedIn or Threads entry with appropriate criteria and weights, and the Algorithm Simulator will evaluate against those rules without any code changes.

Create new audience personas. Add a "Streamer_Sarah" who evaluates content from a Twitch creator's perspective, or a "ParentGamer_Pat" who only engages with family-friendly content. Each persona is a system prompt and a name, nothing else to change.

Swap the niche. Replace the gaming trend fetcher with music, tech, or fitness trends. The agent architecture is niche-agnostic; only the trend tool and sample data need to change.

Register trends as an Agent Framework tool. Right now, the application fetches trends and passes them as context. In a more advanced version, you could use the @tool decorator to register fetch_gaming_trends as a callable tool that agents invoke autonomously — moving from application-controlled to agent-controlled tool use.

What's Next: Evaluating the Evaluator

Here's the question this project intentionally leaves open: the Algorithm Simulator scored the post 75/100 — but how do we know the Simulator itself is any good?

We built an agent that evaluates content, but we never evaluated the evaluator. How consistent are its scores? If you run the same post through it twice, does it give the same result? Do its predictions correlate with real-world engagement metrics? Would a human social media strategist agree with its rubric weights?

This is the problem of agent evaluation — one of the most important and underexplored challenges in building production agentic systems. We all know how to evaluate a model on a benchmark. But how do you evaluate an agent that's making subjective, multi-dimensional judgments within a larger system?

In a follow-up article, we'll tackle exactly this: building evaluation frameworks for AI agents, testing for consistency and calibration, measuring inter-agent agreement, and determining whether your agents are actually doing what you think they're doing. The system we built here will serve as our running example — because when your system contains an agent whose entire job is evaluation, evaluating that agent becomes the most important question you can ask.

Get the Code

The full project is on GitHub:

https://github.com/HamidOna/viral-or-fail

Clone it, run pip install -r requirements.txt, add your GitHub token to .env, and run python viral_or_fail.py. Everything runs on GitHub Models' free tier — no paid API keys required.

References and Further Reading

Frameworks and Tools

Microsoft Agent Framework Documentation — Microsoft's production framework for multi-agent orchestration (successor to AutoGen), used throughout this project
AI Agents for Beginners — Microsoft's 12-lesson course on building AI agents, which inspired this project. Particularly relevant: Lesson 4 (Tool Use), Lesson 8 (Multi-Agent Design Pattern), and Lesson 9 (Metacognition)
GitHub Models — Free-tier LLM access used in this project, no paid API keys required
trendspy — Lightweight Google Trends library replacing the archived pytrends

Concepts

Agentic Design Patterns — Overview of the core patterns (reflection, tool use, planning, multi-agent) that this project implements
Building Trustworthy AI Agents — Relevant to thinking about how agent evaluation and guardrails connect to the system we built
Context Engineering for AI Agents — The rubric injection technique we used is a form of context engineering

Stop Drawing Architecture Diagrams Manually: Meet the Open-Source AI Architecture Review Agents

ShivamGoyal — Thu, 26 Feb 2026 08:00:00 GMT

Hey everyone! I am Shivam Goyal, a Microsoft MVP, and I am super excited to share a project that is going to save you a massive amount of time.

Designing software architecture is arguably one of the most creative and enjoyable parts of engineering. Documenting it, reviewing it for security flaws, and keeping the diagrams updated as the system evolves? Not so much.

We have all been there. You sketch out a brilliant microservices architecture on a whiteboard, take a blurry photo of it, and spend the next three hours wrestling with boxes, arrows, and alignment tools. By the time you finally get to the actual security and risk review, the architecture has already changed.

What if you could just explain your system in plain English, or point a tool to a messy README, and instantly get a prioritized risk assessment, actionable recommendations, and an editable architecture diagram?

Enter the Architecture Review Agent, an open-source AI sample my team and I built with the Microsoft Agent Framework, Azure OpenAI, and Excalidraw MCP.

What is the Architecture Review Agent?

At its core, the Architecture Review Agent is an automated pipeline that takes architectural descriptions in almost any format and transforms them into structured insights and visual maps.

Whether you feed it a strictly formatted YAML file, a Markdown design doc, or just a brain dump like: "We have a React frontend hitting a Kong gateway, which routes to three microservices, each with its own Postgres DB," the agent processes it in seconds.

Here is what you get back:

An Interactive Excalidraw Diagram: No more static, uneditable images. The agent renders a fully interactive diagram via Excalidraw MCP that you can immediately tweak right in your browser.
Prioritized Risk Analysis: An automated assessment of Single Points of Failure (SPOFs), scalability bottlenecks, security gaps, and architectural anti-patterns.
Component Dependency Mapping: A detailed breakdown of fan-in and fan-out metrics, plus detection of orphaned components.

See it in action: Check out this end-to-end review of an architecture, from file upload to risk detection and interactive diagram generation.

Why You Should Add It to Your Workflow

I wanted this agent to adapt to how developers actually work, rather than forcing you to learn a new proprietary diagramming language.

1. Smart Input Intelligence

The agent works with what you already have. If you pass it structured YAML or Markdown, it uses a lightning-fast rule-based parser. If you pass it unstructured text, code files, or meeting notes, it automatically falls back to Azure OpenAI (we highly recommend GPT-4.1) to intelligently infer the components, their types, and how they connect.

2. Actionable, Context-Aware Reviews

This isn't just about drawing boxes. The AI analyzes your data flow to flag real-world issues. It will warn you about shared database anti-patterns, highlight missing API gateways, or point out infrastructure components that lack redundancy. The risks are bucketed by severity (Critical to Low) so you know exactly what to tackle first.

A Quick Note on AI Recommendations: While the agent is incredibly powerful, it is designed to be a co-pilot for your architecture team, not a replacement for human expertise. Always treat the AI-generated risk assessments and recommendations as a starting point. They are an amazing tool to accelerate your review process, but you should always verify the findings and conduct formal security audits with your human experts!

3. Exports That Actually Matter

Need a slide for your next architecture review board? Grab the high-res PNG export. Need your team to collaborate and refine the design? Download the .excalidraw JSON file or edit it directly in the React web UI.

Deploy It Your Way: Featuring Microsoft Foundry Hosted Agents

The repository ships with scripts to get you up and running immediately. You have two production-ready deployment paths: a traditional full-stack web app, or my absolute favourite approach, a Hosted Agent via Microsoft Foundry.

Option A: Full-Stack Web App (Azure App Service)

This is perfect if your team wants a custom, drag-and-drop React web interface. This path deploys a FastAPI backend and a React frontend to Azure App Service, giving you full ownership over the API surface and the UI.

Option B: The Future of Zero-Ops AI (Microsoft Foundry Hosted Agents)

If you want to build a scalable, enterprise-grade API without wrestling with infrastructure, Hosted agents in Foundry Agent Service (preview) - Microsoft Foundry is the way to go.

Recently introduced in preview, Hosted Agents allow you to bring your own agent code (built with the Microsoft Agent Framework) and run it as a fully managed containerized service. Microsoft Foundry handles the heavy lifting so you can focus purely on your agent's logic.

Here is why deploying the Architecture Review Agent on Microsoft Foundry is a complete game changer:

Zero-Ops Infrastructure: The platform automatically builds your container via ACR Tasks and manages the compute. It scales seamlessly from 0 to 5 replicas, including scaling to 0 to save costs when idle.
Built-in Conversation Persistence: You do not need to build your own database to remember chat history. The Foundry Agent Service natively manages conversation state across requests.
Enterprise Security Out-of-the-Box: Say goodbye to hardcoding API keys. Hosted Agents use system-assigned Managed Identities (Entra ID) with Role-Based Access Control (RBAC).
Publish Anywhere: Once deployed to Foundry, you can publish your agent directly to Microsoft Teams or Microsoft 365 Copilot with no extra code required. Your team can literally ask Copilot in Teams to review an architecture spec!
Seamless VS Code Deployment: We have integrated this sample with the Microsoft Foundry for VS Code extension. Deploying to the cloud is as simple as opening the Command Palette, running Microsoft Foundry: Deploy Hosted Agent, and following the prompts.

Get Started in 5 Minutes

The project is completely open-source and waiting for you to test it out. If you have Python 3.11+ and access to Azure OpenAI or a Microsoft Foundry project, you can generate your first architecture review right now.

Just clone the repository, run the setup script, and try feeding it your messiest system architecture description.

GitHub Repo: Azure-Samples/agent-architecture-review-sample

Learn More & Let's Connect!

Building this agent has been an incredible journey, and I truly believe tools like this are the future of how we design and review software. But this is just the beginning, and I would love for you to be a part of it.

If you want to dive deeper into the technology stack powering the Architecture Review Agent, here are some fantastic resources to get you started:

I want to hear from you. Whether you are deploying this for your enterprise, hacking on it over the weekend, or have a cool idea for a new feature, I would love to connect.

Drop a star or open an issue on GitHub: Architecture Review Agent Sample
Connect with me on LinkedIn: linkedin.com/in/shivam2003
Check out my other projects: github.com/ShivamGoyal03

Let me know what you think in the comments below, and happy architecting!

Integrating Microsoft Foundry with OpenClaw: Step by Step Model Configuration

suzarilshah — Mon, 23 Feb 2026 09:39:15 GMT

Step 1: Deploying Models on Microsoft Foundry

Let us kick things off in the Azure portal. To get our OpenClaw agent thinking like a genius, we need to deploy our models in Microsoft Foundry. For this guide, we are going to focus on deploying gpt-5.2-codex on Microsoft Foundry with OpenClaw.

Navigate to your AI Hub, head over to the model catalog, choose the model you wish to use with OpenClaw and hit deploy. Once your deployment is successful, head to the endpoints section.

Important: Grab your Endpoint URL and your API Keys right now and save them in a secure note. We will need these exact values to connect OpenClaw in a few minutes.

Step 2: Installing and Initializing OpenClaw

Next up, we need to get OpenClaw running on your machine. Open up your terminal and run the official installation script:

curl -fsSL https://openclaw.ai/install.sh | bash

The wizard will walk you through a few prompts. Here is exactly how to answer them to link up with our Azure setup:

First Page (Model Selection): Choose "Skip for now".
Second Page (Provider): Select azure-openai-responses.

Model Selection: Select gpt-5.2-codex , For now only the models listed (hosted on Microsoft Foundry) in the picture below are available to be used with OpenClaw.
Follow the rest of the standard prompts to finish the initial setup.

Step 3: Editing the OpenClaw Configuration File

Now for the fun part. We need to manually configure OpenClaw to talk to Microsoft Foundry. Open your configuration file located at ~/.openclaw/openclaw.json in your favorite text editor.

Replace the contents of the models and agents sections with the following code block:

{ "models": { "providers": { "azure-openai-responses": { "baseUrl": "https://<YOUR_RESOURCE_NAME>.openai.azure.com/openai/v1", "apiKey": "<YOUR_AZURE_OPENAI_API_KEY>", "api": "openai-responses", "authHeader": false, "headers": { "api-key": "<YOUR_AZURE_OPENAI_API_KEY>" }, "models": [ { "id": "gpt-5.2-codex", "name": "GPT-5.2-Codex (Azure)", "reasoning": true, "input": ["text", "image"], "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, "contextWindow": 400000, "maxTokens": 16384, "compat": { "supportsStore": false } }, { "id": "gpt-5.2", "name": "GPT-5.2 (Azure)", "reasoning": false, "input": ["text", "image"], "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, "contextWindow": 272000, "maxTokens": 16384, "compat": { "supportsStore": false } } ] } } }, "agents": { "defaults": { "model": { "primary": "azure-openai-responses/gpt-5.2-codex" }, "models": { "azure-openai-responses/gpt-5.2-codex": {} }, "workspace": "/home/<USERNAME>/.openclaw/workspace", "compaction": { "mode": "safeguard" }, "maxConcurrent": 4, "subagents": { "maxConcurrent": 8 } } } }

You will notice a few placeholders in that JSON. Here is exactly what you need to swap out:

Placeholder Variable	What It Is	Where to Find It
<YOUR_RESOURCE_NAME>	The unique name of your Azure OpenAI resource.	Found in your Azure Portal under the Azure OpenAI resource overview.
<YOUR_AZURE_OPENAI_API_KEY>	The secret key required to authenticate your requests.	Found in Microsoft Foundry under your project endpoints or Azure Portal keys section.
<USERNAME>	Your local computer's user profile name.	Open your terminal and type whoami to find this.

Step 4: Restart the Gateway

After saving the configuration file, you must restart the OpenClaw gateway for the new Foundry settings to take effect. Run this simple command:

openclaw gateway restart

Configuration Notes & Deep Dive

If you are curious about why we configured the JSON that way, here is a quick breakdown of the technical details.

Authentication Differences Azure OpenAI uses the api-key HTTP header for authentication. This is entirely different from the standard OpenAI Authorization: Bearer header. Our configuration file addresses this in two ways:

Setting "authHeader": false completely disables the default Bearer header.
Adding "headers": { "api-key": "<key>" } forces OpenClaw to send the API key via Azure's native header format.

Important Note: Your API key must appear in both the apiKey field AND the headers.api-key field within the JSON for this to work correctly.

The Base URL Azure OpenAI's v1-compatible endpoint follows this specific format: https://<your_resource_name>.openai.azure.com/openai/v1

The beautiful thing about this v1 endpoint is that it is largely compatible with the standard OpenAI API and does not require you to manually pass an api-version query parameter.

Model Compatibility Settings

"compat": { "supportsStore": false } disables the store parameter since Azure OpenAI does not currently support it.
"reasoning": true enables the thinking mode for GPT-5.2-Codex. This supports low, medium, high, and xhigh levels.
"reasoning": false is set for GPT-5.2 because it is a standard, non-reasoning model.

Model Specifications & Cost Tracking

If you want OpenClaw to accurately track your token usage costs, you can update the cost fields from 0 to the current Azure pricing. Here are the specs and costs for the models we just deployed:

Model Specifications

Model	Context Window	Max Output Tokens	Image Input	Reasoning
gpt-5.2-codex	400,000 tokens	16,384 tokens	Yes	Yes
gpt-5.2	272,000 tokens	16,384 tokens	Yes	No

Current Cost (Adjust in JSON)

Model	Input (per 1M tokens)	Output (per 1M tokens)	Cached Input (per 1M tokens)
gpt-5.2-codex	$1.75	$14.00	$0.175
gpt-5.2	$2.00	$8.00	$0.50

Conclusion:

And there you have it! You have successfully bridged the gap between the enterprise-grade infrastructure of Microsoft Foundry and the local autonomy of OpenClaw. By following these steps, you are not just running a chatbot; you are running a sophisticated agent capable of reasoning, coding, and executing tasks with the full power of GPT-5.2-codex behind it.

The combination of Azure's reliability and OpenClaw's flexibility opens up a world of possibilities. Whether you are building an automated devops assistant, a research agent, or just exploring the bleeding edge of AI, you now have a robust foundation to build upon.

Now it is time to let your agent loose on some real tasks. Go forth, experiment with different system prompts, and see what you can build. If you run into any interesting edge cases or come up with a unique configuration, let me know in the comments below. Happy coding!

Learning Cost Efficient AI Agents Development on Azure

carlottacaste — Mon, 09 Mar 2026 19:11:37 GMT

AI agents are increasingly central to building automated solutions, experimenting with data‑driven decision making, and bringing real‑world AI systems to life.

But as AI adoption grows, so do important questions: How much does AI cost? How do design choices affect efficiency? And how can developers build AI solutions that are both innovative and sustainable?

The Maximize the Cost Efficiency of AI Agents on Azure webinar is designed to help answer these questions.

This session provides practical guidance on designing and scaling AI agents on Azure while keeping cost efficiency in mind. Rather than focusing only on tools and services, the webinar helps learners and educators understand how architectural decisions, model choices, and usage patterns directly impact cost, performance, and outcomes. These are the same considerations students will encounter in real-world projects, research, and future careers.

Who should attend?

Whether you are introducing Agentic AI concepts in the classroom, working on student projects, or exploring AI agents as part of your learning journey, this webinar offers actionable insights you can apply immediately—both in teaching and hands-on experimentation.

Why attend the webinar?

In this session, you’ll see how Agentic AI cost considerations translate from theory into real-world scenarios, using practical examples that are easy to relate to student projects and classroom use cases. The webinar also highlights common cost pitfalls and shows how thoughtful design decisions can help avoid them early.

Most importantly, the session helps learners and educators connect technical choices to measurable outcomes—building a stronger understanding of how to evaluate, optimize, and govern AI systems responsibly. You’ll have the opportunity to ask questions live and leave with clearer guidance on how to build AI agents that scale efficiently.

If you care about preparing students for real-world AI development—or building your own skills with a strong foundation in responsible and cost-aware design—this webinar is a valuable addition to your learning journey.

Missed it? Watch it on demand!

Who will speak at the webinar?

Your speakers will be:

Carlotta Castelluccio: Carlotta is a Senior AI Advocate with the mission of helping every developer to succeed with AI, by building innovative solutions responsibly. To achieve this goal, she develops technical content, and she hosts skilling sessions, enabling her audience to take the most out of AI technologies and to have an impact on Microsoft AI products’ roadmap.

Nitya Narasimhan: Nitya is a PhD and Polyglot with 25+ years of software research & development experience spanning mobile, web, cloud and AI. She is an innovator (12+ patents), a visual storyteller (@sketchtedocs), and an experienced community builder in the Greater New York area. As a senior AI Advocate on the Core AI Developer Relations team, she acts as "developer 0" for the Microsoft Foundry platform, providing product feedback and empowering AI developers to build trustworthy AI solutions with code samples, open-source curricula and content-initiatives like Model Mondays. Prior to joining Microsoft, she spent a decade in Motorola Labs working on ubiquitous & mobile computing research, founded Google Developer Groups in New York, and consulted for startups building real-time experiences for enterprise. Her current interests span Model understanding & customization, E2E Observability & Safety, and agentic AI workflows for maintainable software.

Useful resources

Microsoft Learn Training Path: https://aka.ms/maximize-cost-efficiency-ai-agents-training

Session Deck: https://aka.ms/maximize-cost-efficiency-ai-agents-deck

Building an AI Study Agent - How GitHub Copilot CLI & SDK helped Reimagine LMS

Julia_Muiruri — Fri, 20 Feb 2026 11:50:14 GMT

What if your Learning Management System didn't just host lecture documents, assignments, and grades - but actually understood them?

Every time I sit through a lecture, a constant thought lingers: "I love what I'm studying, don't get me wrong - but it's a lot!" These are 3-hour lectures, a little too much content with piles of reference materials - how do I create efficient study routines beyond these lectures? With the world moving toward an agentic future, AI should help but having read so many posts on AI personalization for education systems, in my experience today, that personalized support isn't here - YET!

Here is the catch though! I don't have weeks to design an architecture, plan every component, and slowly build my way there. I have a problem and a rough idea of a solution, and I need a working prototype fast!

Enter GitHub Copilot CLI

GIF showing typing copilot --banner in the terminal

Staring at an empty folder with a half-baked idea and not exactly sure where to start, I spun up the terminal and launched a Copilot Agent in /plan mode for a brainstorming session. You know - to help me think.
This was less of a building session and more of an interactive brainstorm with the agent asking clarifying questions about features, stack preferences, and constraints, then returned a comprehensive implementation plan in seconds.

That step alone was incredibly valuable: it didn't just give the agent a picture of what I wanted to build, it also surfaced scenarios I hadn't even thought of. Even without the full implemenation, that step alone was enough to move my idea forward, and this has influenced my normal ideation routine which is now: idea → brainstorm with Copilot /plan mode → save the plan → iterate.

The Solution

With the plan ready, you might tell the agent to “Start Implementation,” and it'll likely do a great job, but I prefer a five-phase workflow that balances speed, structure and my desired level of involvement in the project (phases may vary by use case):

5 Phases - Brainstorm, Research, Project Setup, Core Logic, Test & Frontend

Here is how I think about the stages:

1. Brainstorm

The goal here is to ensure the idea is crystal clear, not just to the builder (me), but to the agent(s), and more importantly - that you are on the same page.

2. Research

This phase is important to surface the latest docs, announcements, and decision factors so in as much as most of the implementation is delegated to the agent(s), builders (I) have a clear understanding of reasons why database/ framework/ provider X was chosen over Y.

3. Project Setup

This is where the agent focuses on installs, project scaffolding, configuration, and defining how components in the architecture design communicate.

4. Core functionality

The main goal here is to implement the core logic behind the system’s essential behavior, followed by a thorough validation that APIs and DB schemas map to the target features.

5. Frontend

Language models rarely struggle with UI design work. The trick to get the perfect frontend with a single prompt in my experience, is to save this task for last and the agent will not only factor in the features already implemented, but will also build a design that anticipates and accommodates future enhancements that you thought about and noted in the brainstorm notes (plan docs).

With these phases documented, plus the plan docs stored in my project directory, I'm confident that when I switch to different agents working on my project, they'll all have a clear, common and referenceable north star and can work on whatever component or feature I delegate to them with the right context.

After the first iteration of this workflow, in a matter of minutes, I had a full stack application, with a beautiful UI and I could browse through the courses, with the ability to upload notes (pdf and text files) which were stored in the database.

Hooray! Happy that it worked, but - is there anything extraordinary about that? Not really, since most current LMS can already do this.

But, here is where we step up the game.

Instead of uploading school docs and have them sit there, a file upload kicks off an ingestion pipeline to build a knowledge base that language models can reason over.

School Agent - Ingestion Pipeline for RAG

So the backend:

extracts the file content. Output: large and long text block
applies a chunking strategy to break the long text block into smaller text groups. Output: chunks of roughly 512 tokens each with a 100 token overlap for context continuation
vector embeddings generation. Output: Embeddings stored in a single db (along with my existing data) using the pgvector extension

Now with my data in a format that language models can understand, the next part involves adding an intelligent layer and we achieve this in 2 steps:

Expose API endpoints in a format that language models can use (tools),
Create an autonomous AI workflow that will handle the tool orchestration and determine when to use what.

Enter Model Context Protocol (MCP)

APIs are designed for humans as the primary users with a discovery path optimized for going through API docs to find endpoints and creating custom integrations to consume them. This doesn't work for language models which instead need a more dynamic, self-discoverable, runtime approach encapsulated in a standardized interface for AI.

This is what the Model Context Protocol provides, a standard that connects AI native apps/ agents to data and tools dynamically.

In the steps above, Copilot CLI uses this very same protocol to pull data from external sources, accessing documentation on how the MCP architecture works and how to build and connect to MCP servers, and in a single prompt, is able to extend my existing backend (API Layer) into an MCP server with tools that will allow the agent to perform actions dynamically. That is either reading course material, generating question-answer pairs from the course content for quizzes, extracting coding exercises and updating my completion progress among other functions.

The quickest way to test this MCP setup is with GitHub Copilot as the MCP client, since I'm yet to build any agentic workflows. I'm already on VS Code, so I simply (1) add the mcp server configuration to my .vscode/mcp.json and now the tools are (2) accessible within the Copilot chat window. I start testing with my (3) custom School Agent, comparing different prompts and (4) tool use accuracy to get a feel of how the agent experience would look like in the app. And of course you can use this through the Copilot CLI if you prefer working from the terminal.

Screenshot of VS Code with mcp.json configuration, configure tools view and GitHub Copilot using the getCourseDocuments tool to ground responses in school documents

That's step 1. Step 2 is building the agent itself.

Enter GitHub Copilot SDK

When it comes to building agents, there are so many Agent Development Kits and Frameworks that make it easier to create and manage agent execution loops, but a recent (and exciting) announcement from GitHub is the new GitHub Copilot SDK. I'll link to the repo in the resources section, but basically what this means is that you can let the existing infrastructure that powers today's GitHub Copilot, handle all the building blocks of an agent - tool discovery and orchestration, session management, real time streaming, multi-turn loops,etc. and just programmatically call that agent workflow in your application.

There wasn't much to go on terms of documentation as this is still very new, but from what I read in the announcement blog & SDK repo - this was mind-blowing. I had to try it!
I'll admit that when I started off with this project, I had a different idea of how to approach the agentic part of it, but luckily the SDK was announced before I got to it and decided that it was worth giving it a try. I am proud to say that I wasn't disappointed.

I jumped into a brainstorming session with my buddy Copilot CLI, who had context from the SDK repo and settled on an approach that needed an aspect of specialization capabilities. Instead of having one agent handle all tasks, let's have smaller specialized agents for each. i.e.,

I like to frequently quiz myself on topics - let's have an agent that does that one task PERFECTLY!
I'm struggling to track the completion of exercises provided in a course text pdf document - let's have an agent specialized on extracting coding exercises from the eBook, tracking my progress and helping out when I'm stuck or if I need a quick review on my code attempts.

The beauty of using the Copilot SDK is that if you have such an idea, you won't have to worry about building it from scratch because chances are, someone already thoughout it out and there is likely a feature or a copilot-native pattern ready for you to use. This case is no exception - because the idea of providing Copilot specialized capabilities for specific tasks is already implemented through Agent Skills.
So all I needed was to define a `SKILL.md` document for the specialized tasks I needed - flashcard generator `.github/skills/flashcard-generator/SKILL.md` & Java practice tracker `.github/skills/java-practice-tracker/SKILL.md` and pass a `skill` property to the agent in code, which again the Copilot CLI implemented in minutes.

In just a couple of hours, (with so many breaks in between), I ended up with School Agent that takes learning management systems to the next level, and this is just the beginning.

School Agent working architecture with - Frontend, Agent, Backend API, MCP Server and DB (Postgresql + pgvector) components

With tools like the Copilot CLI, SDK, and other AI dev tools, experimentation has never been easier and I have so many ideas (I'm sure you do too), of how to make this system even more useful but I'm confident that in a short while, I'll be back with the next set of features built out and working to perfection.

I'm evolving School Agent into an architecture that is program-agnostic and I hope to share it with you soon to try it out and make it your own.
Yes things are moving fast in the AI space, but at least this way I have AI working with me and for me to improve an actual real world experience (and so can you). I encourage you to not just take other people's word for it, - you saw a cool demo on YouTube/ X recently, or you enjoyed this post (I hope you did), but don't settle for that. Find an immediate problem you are having today and tinker around. Build something. Anything. Everything!

To students: Would you use School Agent? What does it need to do to be even more useful to you?
For educators: How can your students benefit from such a tool? What would you also like to see implemented to support you?

Resources

Check out this video walk-through of School Agent

Get started with Copilot CLI
Get started with the Copilot SDK
Agent Skills

If you enjoyed this post, let's connect on LinkedIn, X and Bsky

Agentic Code Fixing with GitHub Copilot SDK and Foundry Local

Lee_Stott — Mon, 16 Feb 2026 09:28:40 GMT

Introduction

AI-powered coding assistants have transformed how developers write and review code. But most of these tools require sending your source code to cloud services, a non-starter for teams working with proprietary codebases, air-gapped environments, or strict compliance requirements. What if you could have an intelligent coding agent that finds bugs, fixes them, runs your tests, and produces PR-ready summaries, all without a single byte leaving your machine?

The Local Repo Patch Agent demonstrates exactly this. By combining the GitHub Copilot SDK for agent orchestration with Foundry Local for on-device inference, this project creates a fully autonomous coding workflow that operates entirely on your hardware. The agent scans your repository, identifies bugs and code smells, applies fixes, verifies them through your test suite, and generates a comprehensive summary of all changes, completely offline and secure.

This article explores the architecture behind this integration, walks through the key implementation patterns, and shows you how to run the agent yourself. Whether you're building internal developer tools, exploring agentic workflows, or simply curious about what's possible when you combine GitHub's SDK with local AI, this project provides a production-ready foundation to build upon.

Why Local AI Matters for Code Analysis

Cloud-based AI coding tools have proven their value—GitHub Copilot has fundamentally changed how millions of developers work. But certain scenarios demand local-first approaches where code never leaves the organisation's network.

Consider these real-world constraints that teams face daily:

Regulatory compliance: Financial services, healthcare, and government projects often prohibit sending source code to external services, even for analysis
Intellectual property protection: Proprietary algorithms and trade secrets can't risk exposure through cloud API calls
Air-gapped environments: Secure facilities and classified projects have no internet connectivity whatsoever
Latency requirements: Real-time code analysis in IDEs benefits from zero network roundtrip
Cost control: High-volume code analysis without per-token API charges

The Local Repo Patch Agent addresses all these scenarios. By running the AI model on-device through Foundry Local and using the GitHub Copilot SDK for orchestration, you get the intelligence of agentic coding workflows with complete data sovereignty. The architecture proves that "local-first" doesn't mean "capability-limited."

The Technology Stack

Two core technologies make this architecture possible, working together through a clever integration called BYOK (Bring Your Own Key). Understanding how they complement each other reveals the elegance of the design.

GitHub Copilot SDK

The GitHub Copilot SDK provides the agent runtime, the scaffolding that handles planning, tool invocation, streaming responses, and the orchestration loop that makes agentic behaviour possible. Rather than managing raw LLM API calls, developers define tools (functions the agent can call) and system prompts, and the SDK handles everything else.

Key capabilities the SDK brings to this project:

Session management: Maintains conversation context across multiple agent interactions
Tool orchestration: Automatically invokes defined tools when the model requests them
Streaming support: Real-time response streaming for responsive user interfaces
Provider abstraction: Works with any OpenAI-compatible API through the BYOK configuration

Foundry Local

Foundry Local brings Azure AI Foundry's model catalog to your local machine. It automatically selects the best available hardware acceleration—GPU, NPU, or CP, and exposes models through an OpenAI-compatible API on localhost. Models run entirely on-device with no telemetry or data transmission.

For this project, Foundry Local provides:

On-device inference: All AI processing happens locally, ensuring complete data privacy
Dynamic port allocation: The SDK auto-detects the Foundry Local endpoint, eliminating configuration hassle
Model flexibility: Swap between models like qwen2.5-coder-1.5b, phi-3-mini, or larger variants based on your hardware
OpenAI API compatibility: Standard API format means the GitHub Copilot SDK works without modification

The BYOK Integration

The entire connection between the GitHub Copilot SDK and Foundry Local happens through a single configuration object. This BYOK (Bring Your Own Key) pattern tells the SDK to route all inference requests to your local model instead of cloud services:

const session = await client.createSession({
  model: modelId,
  provider: {
    type: "openai",               // Foundry Local speaks OpenAI's API format
    baseUrl: proxyBaseUrl,        // Streaming proxy → Foundry Local
    apiKey: manager.apiKey,
    wireApi: "completions",       // Chat Completions API
  },
  streaming: true,
  tools: [ /* your defined tools */ ],
});

This configuration is the key insight: with one config object, you've redirected an entire agent framework to run on local hardware. No code changes to the SDK, no special adapters—just standard OpenAI-compatible API communication.

Architecture Overview

The Local Repo Patch Agent implements a layered architecture where each component has a clear responsibility. Understanding this flow helps when extending or debugging the system.

┌─────────────────────────────────────────────────────────┐
│                    Your Terminal / Web UI                │
│                    npm run demo / npm run ui             │
└──────────────┬──────────────────────────────────────────┘
               │
┌──────────────▼──────────────────────────────────────────┐
│          src/agent.ts  (this project)                    │
│                                                          │
│  ┌────────────────────────────┐   ┌──────────────────┐  │
│  │  GitHub Copilot SDK         │   │  Agent Tools      │  │
│  │  (CopilotClient)            │   │  list_files       │  │
│  │  BYOK → Foundry             │   │  read_file        │  │
│  └────────┬───────────────────┘   │  write_file       │  │
│            │                       │  run_command      │  │
└────────────┼───────────────────────┴──────────────────┘  │
             │                                             │
             │ JSON-RPC                                    │
┌────────────▼─────────────────────────────────────────────┐
│          GitHub Copilot CLI  (server mode)                │
│          Agent orchestration layer                        │
└────────────┬─────────────────────────────────────────────┘
             │ POST /v1/chat/completions   (BYOK)
┌────────────▼─────────────────────────────────────────────┐
│          Foundry Local  (on-device inference)             │
│          Model: qwen2.5-coder-1.5b via ONNX Runtime      │
│          Endpoint: auto-detected (dynamic port)           │
└───────────────────────────────────────────────────────────┘

The data flow works as follows: your terminal or web browser sends a request to the agent application. The agent uses the GitHub Copilot SDK to manage the conversation, which communicates with the Copilot CLI running in server mode. The CLI, configured with BYOK, sends inference requests to Foundry Local running on localhost. Responses flow back up the same path, with tool invocations happening in the agent.ts layer.

The Four-Phase Workflow

The agent operates through a structured four-phase loop, each phase building on the previous one's output. This decomposition transforms what would be an overwhelming single prompt into manageable, verifiable steps.

Phase 1: PLAN

The planning phase scans the repository and produces a numbered fix plan. The agent reads every source and test file, identifies potential issues, and outputs specific tasks to address:

// Phase 1 system prompt excerpt
const planPrompt = `
You are a code analysis agent. Scan the repository and identify:
1. Bugs that cause test failures
2. Code smells and duplication
3. Style inconsistencies

Output a numbered list of fixes, ordered by priority.
Each item should specify: file path, line numbers, issue type, and proposed fix.
`;

The tools available during this phase are list_files and read_file—the agent explores the codebase without modifying anything. This read-only constraint prevents accidental changes before the plan is established.

Phase 2: EDIT

With a plan in hand, the edit phase applies each fix by rewriting affected files. The agent receives the plan from Phase 1 and systematically addresses each item:

// Phase 2 adds the write_file tool
const editTools = [
  {
    name: "write_file",
    description: "Write content to a file, creating or overwriting it",
    parameters: {
      type: "object",
      properties: {
        path: { type: "string", description: "File path relative to repo root" },
        content: { type: "string", description: "Complete file contents" }
      },
      required: ["path", "content"]
    }
  }
];

The write_file tool is sandboxed to the demo-repo directory, path traversal attempts are blocked, preventing the agent from modifying files outside the designated workspace.

Phase 3: VERIFY

After making changes, the verification phase runs the project's test suite to confirm fixes work correctly. If tests fail, the agent attempts to diagnose and repair the issue:

// Phase 3 adds run_command with an allowlist
const allowedCommands = ["npm test", "npm run lint", "npm run build"];

const runCommandTool = {
  name: "run_command",
  description: "Execute a shell command (npm test, npm run lint, npm run build only)",
  execute: async (command: string) => {
    if (!allowedCommands.includes(command)) {
      throw new Error(`Command not allowed: ${command}`);
    }
    // Execute and return stdout/stderr
  }
};

The command allowlist is a critical security measure. The agent can only run explicitly permitted commands—no arbitrary shell execution, no data exfiltration, no system modification.

Phase 4: SUMMARY

The final phase produces a PR-style Markdown report documenting all changes. This summary includes what was changed, why each change was necessary, test results, and recommended follow-up actions:

## Summary of Changes

### Bug Fix: calculateInterest() in account.js
- **Issue**: Division instead of multiplication caused incorrect interest calculations
- **Fix**: Changed `principal / annualRate` to `principal * (annualRate / 100)`
- **Tests**: 3 previously failing tests now pass

### Refactor: Duplicate formatCurrency() removed
- **Issue**: Identical function existed in account.js and transaction.js
- **Fix**: Both files now import from utils.js
- **Impact**: Reduced code duplication, single source of truth

### Test Results
- **Before**: 6/9 passing
- **After**: 9/9 passing

This structured output makes code review straightforward, reviewers can quickly understand what changed and why without digging through diffs.

The Demo Repository: Intentional Bugs

The project includes a demo-repo directory containing a small banking utility library with intentional problems for the agent to find and fix. This provides a controlled environment to demonstrate the agent's capabilities.

Bug 1: Calculation Error in calculateInterest()

The account.js file contains a calculation bug that causes test failures:

// BUG: should be principal * (annualRate / 100)
function calculateInterest(principal, annualRate) {
  return principal / annualRate;  // Division instead of multiplication!
}

This bug causes 3 of 9 tests to fail. The agent identifies it during the PLAN phase by correlating test failures with the implementation, then fixes it during EDIT.

Bug 2: Code Duplication

The formatCurrency() function is copy-pasted in both account.js and transaction.js, even though a canonical version exists in utils.js. This duplication creates maintenance burden and potential inconsistency:

// In account.js (duplicated)
function formatCurrency(amount) {
  return '$' + amount.toFixed(2);
}

// In transaction.js (also duplicated)
function formatCurrency(amount) {
  return '$' + amount.toFixed(2);
}

// In utils.js (canonical, but unused)
export function formatCurrency(amount) {
  return '$' + amount.toFixed(2);
}

The agent identifies this duplication during planning and refactors both files to import from utils.js, eliminating redundancy.

Handling Foundry Local Streaming Quirks

One technical challenge the project solves is Foundry Local's behaviour with streaming requests. As of version 0.5, Foundry Local can hang on stream: true requests. The project includes a streaming proxy that works around this limitation transparently.

The Streaming Proxy

The streaming-proxy.ts file implements a lightweight HTTP proxy that converts streaming requests to non-streaming, then re-encodes the single response as SSE (Server-Sent Events) chunks—the format the OpenAI SDK expects:

// streaming-proxy.ts simplified logic
async function handleRequest(req: Request): Promise {
  const body = await req.json();
  
  // If it's a streaming chat completion, convert to non-streaming
  if (body.stream === true && req.url.includes('/chat/completions')) {
    body.stream = false;
    
    const response = await fetch(foundryEndpoint, {
      method: 'POST',
      body: JSON.stringify(body),
      headers: { 'Content-Type': 'application/json' }
    });
    
    const data = await response.json();
    
    // Re-encode as SSE stream for the SDK
    return createSSEResponse(data);
  }
  
  // Non-streaming and non-chat requests pass through unchanged
  return fetch(foundryEndpoint, req);
}

This proxy runs on port 8765 by default and sits between the GitHub Copilot SDK and Foundry Local. The SDK thinks it's talking to a streaming-capable endpoint, while the actual inference happens non-streaming. The conversion is transparent, no changes needed to SDK configuration.

Text-Based Tool Call Detection

Small on-device models like qwen2.5-coder-1.5b sometimes output tool calls as JSON text rather than using OpenAI-style function calling. The SDK won't fire tool.execution_start events for these text-based calls, so the agent includes a regex-based detector:

// Pattern to detect tool calls in model output
const toolCallPattern = /\{[\s\S]*"name":\s*"(list_files|read_file|write_file|run_command)"[\s\S]*\}/;

function detectToolCall(text: string): ToolCall | null {
  const match = text.match(toolCallPattern);
  if (match) {
    try {
      return JSON.parse(match[0]);
    } catch {
      return null;
    }
  }
  return null;
}

This fallback ensures tool calls are captured regardless of whether the model uses native function calling or text output, keeping the dashboard's tool call counter and CLI log accurate.

Security Considerations

Running an AI agent that can read and write files and execute commands requires careful security design. The Local Repo Patch Agent implements multiple layers of protection:

100% local execution: No code, prompts, or responses leave your machine—complete data sovereignty
Command allowlist: The agent can only run npm test, npm run lint, and npm run build—no arbitrary shell commands
Path sandboxing: File tools are locked to the demo-repo/ directory; path traversal attempts like ../../../etc/passwd are rejected
File size limits: The read_file tool rejects files over 256 KB, preventing memory exhaustion attacks
Recursion limits: Directory listing caps at 20 levels deep, preventing infinite traversal

These constraints demonstrate responsible AI agent design. The agent has enough capability to do useful work but not enough to cause harm. When extending this project for your own use cases, maintain similar principles, grant minimum necessary permissions, validate all inputs, and fail closed on unexpected conditions.

Running the Agent

Getting the Local Repo Patch Agent running on your machine takes about five minutes. The project includes setup scripts that handle prerequisites automatically.

Prerequisites

Before running the setup, ensure you have:

Node.js 18 or higher: Download from nodejs.org (LTS version recommended)
Foundry Local: Install via winget install Microsoft.FoundryLocal (Windows) or brew install foundrylocal (macOS)
GitHub Copilot CLI: Follow the GitHub Copilot CLI install guide

Verify your installations:

node --version    # Should print v18.x.x or higher
foundry --version
copilot --version

One-Command Setup

The easiest path uses the provided setup scripts that install dependencies, start Foundry Local, and download the AI model:

# Clone the repository
git clone https://github.com/leestott/copilotsdk_foundrylocal.git
cd copilotsdk_foundrylocal

# Windows (PowerShell)
.\setup.ps1

# macOS / Linux
chmod +x setup.sh
./setup.sh

When setup completes, you'll see:

━━━ Setup complete! ━━━

  You're ready to go. Run one of these commands:

    npm run demo     CLI agent (terminal output)
    npm run ui       Web dashboard (http://localhost:3000)

Manual Setup

If you prefer step-by-step control:

# Install npm packages
npm install
cd demo-repo && npm install --ignore-scripts && cd ..

# Start Foundry Local and download the model
foundry service start
foundry model run qwen2.5-coder-1.5b

# Copy environment configuration
cp .env.example .env

# Run the agent
npm run demo

The first model download takes a few minutes depending on your connection. After that, the model runs from cache with no internet required.

Using the Web Dashboard

For a visual experience with real-time streaming, launch the web UI:

npm run ui

Open http://localhost:3000 in your browser. The dashboard provides:

Phase progress sidebar: Visual indication of which phase is running, completed, or errored
Live streaming output: Model responses appear in real-time via WebSocket
Tool call log: Every tool invocation logged with phase context
Phase timing table: Performance metrics showing how long each phase took
Environment info: Current model, endpoint, and repository path at a glance

Configuration Options

The agent supports several environment variables for customisation. Edit the .env file or set them directly:

Variable	Default	Description
FOUNDRY_LOCAL_ENDPOINT	auto-detected	Override the Foundry Local API endpoint
FOUNDRY_LOCAL_API_KEY	auto-detected	Override the API key
FOUNDRY_MODEL	qwen2.5-coder-1.5b	Which model to use from the Foundry Local catalog
FOUNDRY_TIMEOUT_MS	180000 (3 min)	How long each agent phase can run before timing out
FOUNDRY_NO_PROXY	—	Set to 1 to disable the streaming proxy
PORT	3000	Port for the web dashboard

Using Different Models

To try a different model from the Foundry Local catalog:

# Use phi-3-mini instead
FOUNDRY_MODEL=phi-3-mini npm run demo

# Use a larger model for higher quality (requires more RAM/VRAM)
FOUNDRY_MODEL=qwen2.5-7b npm run demo

Adjusting for Slower Hardware

If you're running on CPU-only or limited hardware, increase the timeout to give the model more time per phase:

# 5 minutes per phase instead of 3
FOUNDRY_TIMEOUT_MS=300000 npm run demo

Troubleshooting Common Issues

When things don't work as expected, these solutions address the most common problems:

Problem	Solution
`foundry: command not found`	Install Foundry Local—see Prerequisites section
`copilot: command not found`	Install GitHub Copilot CLI—see Prerequisites section
Agent times out on every phase	Increase `FOUNDRY_TIMEOUT_MS` (e.g., 300000 for 5 min). CPU-only machines are slower.
Port 3000 already in use	Set `PORT=3001 npm run ui`
Model download is slow	First download can take 5-10 min. Subsequent runs use the cache.
`Cannot find module` errors	Run `npm install` again, then `cd demo-repo && npm install --ignore-scripts`
Tests still fail after agent runs	The agent edits files in demo-repo/. Reset with `git checkout demo-repo/` and run again.
PowerShell blocks setup.ps1	Run `Set-ExecutionPolicy -Scope Process Bypass` first, then `.\setup.ps1`

Diagnostic Test Scripts

The src/tests/ folder contains standalone scripts for debugging SDK and Foundry Local integration issues. These are invaluable when things go wrong:

# Debug-level SDK event logging
npx tsx src/tests/test-debug.ts

# Test non-streaming inference (bypasses streaming proxy)
npx tsx src/tests/test-nostream.ts

# Raw fetch to Foundry Local (bypasses SDK entirely)
npx tsx src/tests/test-stream-direct.ts

# Start the traffic-inspection proxy
npx tsx src/tests/test-proxy.ts

These scripts isolate different layers of the stack, helping identify whether issues lie in Foundry Local, the streaming proxy, the SDK, or your application code.

Key Takeaways

BYOK enables local-first AI: A single configuration object redirects the entire GitHub Copilot SDK to use on-device inference through Foundry Local
Phased workflows improve reliability: Breaking complex tasks into PLAN → EDIT → VERIFY → SUMMARY phases makes agent behaviour predictable and debuggable
Security requires intentional design: Allowlists, sandboxing, and size limits constrain agent capabilities to safe operations
Local models have quirks: The streaming proxy and text-based tool detection demonstrate how to work around on-device model limitations
Real-time feedback matters: The web dashboard with WebSocket streaming makes agent progress visible and builds trust in the system
The architecture is extensible: Add new tools, change models, or modify phases to adapt the agent to your specific needs

Conclusion and Next Steps

The Local Repo Patch Agent proves that sophisticated agentic coding workflows don't require cloud infrastructure. By combining the GitHub Copilot SDK's orchestration capabilities with Foundry Local's on-device inference, you get intelligent code analysis that respects data sovereignty completely.

The patterns demonstrated here, BYOK integration, phased execution, security sandboxing, and streaming workarounds, transfer directly to production systems. Consider extending this foundation with:

Custom tool sets: Add database queries, API calls to internal services, or integration with your CI/CD pipeline
Multiple repository support: Scan and fix issues across an entire codebase or monorepo
Different model sizes: Use smaller models for quick scans, larger ones for complex refactoring
Human-in-the-loop approval: Add review steps before applying fixes to production code
Integration with Git workflows: Automatically create branches and PRs from agent-generated fixes

Clone the repository, run through the demo, and start building your own local-first AI coding tools. The future of developer AI isn't just cloud—it's intelligent systems that run wherever your code lives.

Resources

Local Repo Patch Agent Repository – Full source code with setup scripts and documentation
Foundry Local – Official site for on-device AI inference
Foundry Local GitHub Repository – Installation instructions and CLI reference
Foundry Local Get Started Guide – Official Microsoft Learn documentation
Foundry Local SDK Reference – Python and JavaScript SDK documentation
GitHub Copilot SDK – Official SDK repository
GitHub Copilot SDK BYOK Documentation – Bring Your Own Key integration guide
GitHub Copilot SDK Getting Started – SDK setup and first agent tutorial
Microsoft Sample: Copilot SDK + Foundry Local – Official integration sample from Microsoft

Building a Local Research Desk: Multi-Agent Orchestration

Lee_Stott — Thu, 12 Feb 2026 19:23:55 GMT

Introduction

Multi-agent systems represent the next evolution of AI applications. Instead of a single model handling everything, specialised agents collaborate—each with defined responsibilities, passing context to one another, and producing results that no single agent could achieve alone. But building these systems typically requires cloud infrastructure, API keys, usage tracking, and the constant concern about what data leaves your machine.

What if you could build sophisticated multi-agent workflows entirely on your local machine, with no cloud dependencies? The Local Research & Synthesis Desk demonstrates exactly this. Using Microsoft Agent Framework (MAF) for orchestration and Foundry Local for on-device inference, this demo shows how to create a four-agent research pipeline that runs entirely on your hardware—no API keys, no data leaving your network, and complete control over every step.

This article walks through the architecture, implementation patterns, and practical code that makes multi-agent local AI possible. You'll learn how to bootstrap Foundry Local from Python, create specialised agents with distinct roles, wire them into sequential, concurrent, and feedback loop orchestration patterns, and implement tool calling for extended functionality. Whether you're building research tools, internal analysis systems, or simply exploring what's possible with local AI, this architecture provides a production-ready foundation.

Why Multi-Agent Architecture Matters

Single-agent AI systems hit limitations quickly. Ask one model to research a topic, analyse findings, identify gaps, and write a comprehensive report—and you'll get mediocre results. The model tries to do everything at once, with no opportunity for specialisation, review, or iterative refinement.

Multi-agent systems solve this by decomposing complex tasks into specialised roles. Each agent focuses on what it does best:

Planners break ambiguous questions into concrete sub-tasks
Retrievers focus exclusively on finding and extracting relevant information
Critics review work for gaps, contradictions, and quality issues
Writers synthesise everything into coherent, well-structured output

This separation of concerns mirrors how human teams work effectively. A research team doesn't have one person doing everything—they have researchers, fact-checkers, editors, and writers. Multi-agent AI systems apply the same principle to AI workflows, with each agent receiving the output of previous agents as context for their own specialised task.

The Local Research & Synthesis Desk implements this pattern with four primary agents, plus an optional ToolAgent for utility functions. Here's how user questions flow through the system:

This architecture demonstrates three essential orchestration patterns: sequential pipelines where each agent builds on the previous output, concurrent fan-out where independent tasks run in parallel to save time, and feedback loops where the Critic can send work back to the Retriever for iterative refinement.

The Technology Stack: MAF + Foundry Local

Before diving into implementation, let's understand the two core technologies that make this architecture possible and why they work so well together.

Microsoft Agent Framework (MAF)

The Microsoft Agent Framework provides building blocks for creating AI agents in Python and .NET. Unlike frameworks that require specific cloud providers, MAF works with any OpenAI-compatible API—which is exactly what Foundry Local provides.

The key abstraction in MAF is the ChatAgent. Each agent has:

Instructions: A system prompt that defines the agent's role and behaviour
Chat client: An OpenAI-compatible client for making inference calls
Tools: Optional functions the agent can invoke during execution
Name: An identifier for logging and observability

MAF handles message threading, tool execution, and response parsing automatically. You focus on designing agent behaviour rather than managing low-level API interactions.

Foundry Local

Foundry Local brings Azure AI Foundry's model catalog to your local machine. It automatically selects the best hardware acceleration available (GPU, NPU, or CPU) and exposes models through an OpenAI-compatible API. Models run entirely on-device with no data leaving your machine.

The foundry-local-sdk Python package provides programmatic control over the Foundry Local service. You can start the service, download models, and retrieve connection information—all from your Python code. This is the "control plane" that manages the local AI infrastructure.

The combination is powerful: MAF handles agent logic and orchestration, while Foundry Local provides the underlying inference. No cloud dependencies, no API keys, complete data privacy:

Bootstrapping Foundry Local from Python

The first practical challenge is starting Foundry Local programmatically. The FoundryLocalBootstrapper class handles this, encapsulating all the setup logic so the rest of the application can focus on agent behaviour.

The bootstrap process follows three steps: start the Foundry Local service if it's not running, download the requested model if it's not cached, and return connection information that MAF agents can use. Here's the core implementation:

from dataclasses import dataclass
from foundry_local import FoundryLocalManager

@dataclass
class FoundryConnection:
    """Holds endpoint, API key, and model ID after bootstrap."""
    endpoint: str
    api_key: str
    model_id: str
    model_alias: str

This dataclass carries everything needed to connect MAF agents to Foundry Local. The endpoint is typically http://localhost:<port>/v1 (the port is assigned dynamically), and the API key is managed internally by Foundry Local.

class FoundryLocalBootstrapper:
    def __init__(self, alias: str | None = None) -> None:
        self.alias = alias or os.getenv("MODEL_ALIAS", "qwen2.5-0.5b")

    def bootstrap(self) -> FoundryConnection:
        """Start service, download & load model, return connection info."""
        from foundry_local import FoundryLocalManager
        
        manager = FoundryLocalManager()
        model_info = manager.download_and_load_model(self.alias)
        
        return FoundryConnection(
            endpoint=manager.endpoint,
            api_key=manager.api_key,
            model_id=model_info.id,
            model_alias=self.alias,
        )

Key design decisions in this implementation:

Lazy import: The foundry_local import happens inside bootstrap() so the application can provide helpful error messages if the SDK isn't installed
Environment configuration: Model alias comes from MODEL_ALIAS environment variable or defaults to qwen2.5-0.5b
Automatic hardware selection: Foundry Local picks GPU, NPU, or CPU automatically—no configuration needed

The qwen2.5 model family is recommended because it supports function/tool calling, which the ToolAgent requires. For higher quality outputs, larger variants like qwen2.5-7b or qwen2.5-14b are available via the --model flag.

Creating Specialised Agents

With Foundry Local bootstrapped, the next step is creating agents with distinct roles. Each agent is a ChatAgent instance with carefully crafted instructions that focus it on a specific task.

The Planner Agent

The Planner receives a user question and available documents, then breaks the research task into concrete sub-tasks. Its instructions emphasise structured output—a numbered list of specific tasks rather than prose:

from agent_framework import ChatAgent
from agent_framework.openai import OpenAIChatClient

def _make_client(conn: FoundryConnection) -> OpenAIChatClient:
    """Create an MAF OpenAIChatClient pointing at Foundry Local."""
    return OpenAIChatClient(
        api_key=conn.api_key,
        base_url=conn.endpoint,
        model_id=conn.model_id,
    )

def create_planner(conn: FoundryConnection) -> ChatAgent:
    return ChatAgent(
        chat_client=_make_client(conn),
        name="Planner",
        instructions=(
            "You are a planning agent. Given a user's research question and a list "
            "of document snippets (if any), break the question into 2-4 concrete "
            "sub-tasks. Output ONLY a numbered list of tasks. Each task should state:\n"
            "  • What information is needed\n"
            "  • Which source documents might help (if known)\n"
            "Keep it concise — no more than 6 lines total."
        ),
    )

Notice how the instructions are explicit about output format. Multi-agent systems work best when each agent produces structured, predictable output that downstream agents can parse reliably.

The Retriever Agent

The Retriever receives the Planner's task list plus raw document content, then extracts and cites relevant passages. Its instructions emphasise citation format—a specific pattern that the Writer can reference later:

def create_retriever(conn: FoundryConnection) -> ChatAgent:
    return ChatAgent(
        chat_client=_make_client(conn),
        name="Retriever",
        instructions=(
            "You are a retrieval agent. You receive a research plan AND raw document "
            "text from local files. Your job:\n"
            "  1. Identify the most relevant passages for each task in the plan.\n"
            "  2. Output extracted snippets with citations in the format:\n"
            "     [filename.ext, lines X-Y]: \"quoted text…\"\n"
            "  3. If no relevant content exists, say so explicitly.\n"
            "Be precise — quote only what is relevant, keep each snippet under 100 words."
        ),
    )

The citation format [filename.ext, lines X-Y] creates a consistent contract. The Writer knows exactly how to reference source material, and human reviewers can verify claims against original documents.

The Critic Agent

The Critic reviews the Retriever's work, identifying gaps and contradictions. This agent serves as a quality gate before the final report and can trigger feedback loops for iterative improvement:

def create_critic(conn: FoundryConnection) -> ChatAgent:
    return ChatAgent(
        chat_client=_make_client(conn),
        name="Critic",
        instructions=(
            "You are a critical review agent. You receive a plan and extracted snippets. "
            "Your job:\n"
            "  1. Check for gaps — are any plan tasks unanswered?\n"
            "  2. Check for contradictions between snippets.\n"
            "  3. Suggest 1-2 specific improvements or missing details.\n"
            "Start your response with 'GAPS FOUND' if issues exist, or 'NO GAPS' if satisfied.\n"
            "Then output a short numbered list of issues (or say 'No issues found')."
        ),
    )

The Critic is instructed to output GAPS FOUND or NO GAPS at the start of its response. This structured output enables the orchestrator to detect when gaps exist and trigger the feedback loop—sending the gaps back to the Retriever for additional retrieval before re-running the Critic. This iterates up to 2 times before the Writer takes over, ensuring higher quality reports.

Critics are essential for production systems. Without this review step, the Writer might produce confident-sounding reports with missing information or internal contradictions.

The Writer Agent

The Writer receives everything—original question, plan, extracted snippets, and critic review—then produces the final report:

def create_writer(conn: FoundryConnection) -> ChatAgent:
    return ChatAgent(
        chat_client=_make_client(conn),
        name="Writer",
        instructions=(
            "You are the final report writer. You receive:\n"
            "  • The original question\n"
            "  • A plan, extracted snippets with citations, and a critic review\n\n"
            "Produce a clear, well-structured answer (3-5 paragraphs). "
            "Requirements:\n"
            "  • Cite sources using [filename.ext, lines X-Y] notation\n"
            "  • Address any gaps the critic raised (note if unresolvable)\n"
            "  • End with a one-sentence summary\n"
            "Do NOT fabricate citations — only use citations provided by the Retriever."
        ),
    )

The final instruction—"Do NOT fabricate citations"—is crucial for responsible AI. The Writer has access only to citations the Retriever provided, preventing hallucinated references that plague single-agent research systems.

Implementing Sequential Orchestration

With agents defined, the orchestrator connects them into a workflow. Sequential orchestration is the simpler pattern: each agent runs after the previous one completes, passing its output as input to the next agent.

The implementation uses Python's async/await for clean asynchronous execution:

import asyncio
import time
from dataclasses import dataclass, field

@dataclass
class StepResult:
    """Captures one agent step for observability."""
    agent_name: str
    input_text: str
    output_text: str
    elapsed_sec: float

@dataclass
class WorkflowResult:
    """Final result of the entire orchestration run."""
    question: str
    steps: list[StepResult] = field(default_factory=list)
    final_report: str = ""

async def _run_agent(agent: ChatAgent, prompt: str) -> tuple[str, float]:
    """Execute a single agent and measure elapsed time."""
    start = time.perf_counter()
    response = await agent.run(prompt)
    elapsed = time.perf_counter() - start
    return response.content, elapsed

The StepResult dataclass captures everything needed for observability: what went in, what came out, and how long it took. This information is invaluable for debugging and optimisation.

The sequential pipeline chains agents together, building context progressively:

async def run_sequential_workflow(
    question: str,
    docs: LoadedDocuments,
    conn: FoundryConnection,
) -> WorkflowResult:
    wf = WorkflowResult(question=question)
    doc_block = docs.combined_text if docs.chunks else "(no documents provided)"
    
    # Step 1 — Plan
    planner = create_planner(conn)
    planner_prompt = f"User question: {question}\n\nAvailable documents:\n{doc_block}"
    plan_text, elapsed = await _run_agent(planner, planner_prompt)
    wf.steps.append(StepResult("Planner", planner_prompt, plan_text, elapsed))
    
    # Step 2 — Retrieve
    retriever = create_retriever(conn)
    retriever_prompt = f"Plan:\n{plan_text}\n\nDocuments:\n{doc_block}"
    snippets_text, elapsed = await _run_agent(retriever, retriever_prompt)
    wf.steps.append(StepResult("Retriever", retriever_prompt, snippets_text, elapsed))
    
    # Step 3 — Critique
    critic = create_critic(conn)
    critic_prompt = f"Plan:\n{plan_text}\n\nExtracted snippets:\n{snippets_text}"
    critique_text, elapsed = await _run_agent(critic, critic_prompt)
    wf.steps.append(StepResult("Critic", critic_prompt, critique_text, elapsed))
    
    # Step 4 — Write
    writer = create_writer(conn)
    writer_prompt = (
        f"Original question: {question}\n\n"
        f"Plan:\n{plan_text}\n\n"
        f"Extracted snippets:\n{snippets_text}\n\n"
        f"Critic review:\n{critique_text}"
    )
    report_text, elapsed = await _run_agent(writer, writer_prompt)
    wf.steps.append(StepResult("Writer", writer_prompt, report_text, elapsed))
    wf.final_report = report_text
    
    return wf

Each step receives all relevant context from previous steps. The Writer gets the most comprehensive prompt—original question, plan, snippets, and critique—enabling it to produce a well-informed final report.

Adding Concurrent Fan-Out and Feedback Loops

Sequential orchestration works well but can be slow. When tasks are independent—neither needs the other's output—running them in parallel saves time. The demo implements this with asyncio.gather.

Consider the Retriever and ToolAgent: both need the Planner's output, but neither depends on the other. Running them concurrently cuts the wait time roughly in half:

async def run_concurrent_retrieval(
    plan_text: str,
    docs: LoadedDocuments,
    conn: FoundryConnection,
) -> tuple[str, str]:
    """Run Retriever and ToolAgent in parallel."""
    retriever = create_retriever(conn)
    tool_agent = create_tool_agent(conn)
    
    doc_block = docs.combined_text if docs.chunks else "(no documents)"
    
    retriever_prompt = f"Plan:\n{plan_text}\n\nDocuments:\n{doc_block}"
    tool_prompt = f"Analyse the following documents for word count and keywords:\n{doc_block}"
    
    # Execute both agents concurrently
    (snippets_text, r_elapsed), (tool_text, t_elapsed) = await asyncio.gather(
        _run_agent(retriever, retriever_prompt),
        _run_agent(tool_agent, tool_prompt),
    )
    
    return snippets_text, tool_text

The asyncio.gather function runs both coroutines concurrently and returns when both complete. If the Retriever takes 3 seconds and the ToolAgent takes 1.5 seconds, the total wait is approximately 3 seconds rather than 4.5 seconds.

Implementing the Feedback Loop

The most sophisticated orchestration pattern is the Critic–Retriever feedback loop. When the Critic identifies gaps in the retrieved information, the orchestrator sends them back to the Retriever for additional retrieval, then re-evaluates:

async def run_critic_with_feedback(
    plan_text: str,
    snippets_text: str,
    docs: LoadedDocuments,
    conn: FoundryConnection,
    max_iterations: int = 2,
) -> tuple[str, str]:
    """
    Run Critic with feedback loop to Retriever.
    Returns (final_snippets, final_critique).
    """
    critic = create_critic(conn)
    retriever = create_retriever(conn)
    
    current_snippets = snippets_text
    
    for iteration in range(max_iterations):
        # Run Critic
        critic_prompt = f"Plan:\n{plan_text}\n\nExtracted snippets:\n{current_snippets}"
        critique_text, _ = await _run_agent(critic, critic_prompt)
        
        # Check if gaps were found
        if not critique_text.upper().startswith("GAPS FOUND"):
            return current_snippets, critique_text
        
        # Gaps found — send back to Retriever for more extraction
        gap_fill_prompt = (
            f"Previous snippets:\n{current_snippets}\n\n"
            f"Gaps identified:\n{critique_text}\n\n"
            f"Documents:\n{docs.combined_text}\n\n"
            "Extract additional relevant passages to fill these gaps."
        )
        additional_snippets, _ = await _run_agent(retriever, gap_fill_prompt)
        current_snippets = f"{current_snippets}\n\n--- Gap-fill iteration {iteration + 1} ---\n{additional_snippets}"
    
    # Max iterations reached — run final critique
    final_critique, _ = await _run_agent(critic, f"Plan:\n{plan_text}\n\nExtracted snippets:\n{current_snippets}")
    return current_snippets, final_critique

This feedback loop pattern significantly improves output quality. The Critic acts as a quality gate, and when standards aren't met, the system iteratively improves rather than producing incomplete results.

The full workflow combines all three patterns—sequential where dependencies require it, concurrent where independence allows it, and feedback loops for quality assurance:

async def run_full_workflow(
    question: str,
    docs: LoadedDocuments,
    conn: FoundryConnection,
) -> WorkflowResult:
    """
    End-to-end workflow showcasing THREE orchestration patterns:
      1. Planner runs first (sequential — must happen before anything else).
      2. Retriever + ToolAgent run concurrently (fan-out on independent tasks).
      3. Critic reviews with feedback loop (iterates with Retriever if gaps found).
      4. Writer produces final report (sequential — needs everything above).
    """
    wf = WorkflowResult(question=question)
    
    # Step 1: Planner (sequential)
    plan_text, elapsed = await _run_agent(create_planner(conn), planner_prompt)
    wf.steps.append(StepResult("Planner", planner_prompt, plan_text, elapsed))
    
    # Step 2: Concurrent fan-out (Retriever + ToolAgent)
    snippets_text, tool_text = await run_concurrent_retrieval(plan_text, docs, conn)
    
    # Step 3: Critic with feedback loop
    final_snippets, critique_text = await run_critic_with_feedback(
        plan_text, snippets_text, docs, conn
    )
    
    # Step 4: Writer (sequential — needs everything)
    writer_prompt = (
        f"Original question: {question}\n\n"
        f"Plan:\n{plan_text}\n\n"
        f"Snippets:\n{final_snippets}\n\n"
        f"Stats:\n{tool_text}\n\n"
        f"Critique:\n{critique_text}"
    )
    report_text, elapsed = await _run_agent(create_writer(conn), writer_prompt)
    wf.final_report = report_text
    
    return wf

This hybrid approach maximises both correctness and performance. Dependencies are respected, independent work happens in parallel, and quality is ensured through iterative feedback.

Implementing Tool Calling

Some agents benefit from deterministic tools rather than relying entirely on LLM generation. The ToolAgent demonstrates this pattern with two utility functions: word counting and keyword extraction.

MAF supports tool calling through function declarations with Pydantic type annotations:

from typing import Annotated
from pydantic import Field

def word_count(
    text: Annotated[str, Field(description="The text to count words in")]
) -> int:
    """Count words in a text string."""
    return len(text.split())

def extract_keywords(
    text: Annotated[str, Field(description="The text to extract keywords from")],
    top_n: Annotated[int, Field(description="Number of keywords to return", default=5)]
) -> list[str]:
    """Extract most frequent words (simple implementation)."""
    words = text.lower().split()
    # Filter common words, count frequencies, return top N
    word_counts = {}
    for word in words:
        if len(word) > 3:  # Skip short words
            word_counts[word] = word_counts.get(word, 0) + 1
    sorted_words = sorted(word_counts.items(), key=lambda x: x[1], reverse=True)
    return [word for word, count in sorted_words[:top_n]]

The Annotated type with Field descriptions provides metadata that MAF uses to generate function schemas for the LLM. When the model needs to count words, it invokes the word_count tool rather than attempting to count in its response (which LLMs notoriously struggle with).

The ToolAgent receives these functions in its constructor:

def create_tool_agent(conn: FoundryConnection) -> ChatAgent:
    return ChatAgent(
        chat_client=_make_client(conn),
        name="ToolHelper",
        instructions=(
            "You are a utility agent. Use the provided tools to compute "
            "word counts or extract keywords when asked. Return the tool "
            "output directly — do not embellish."
        ),
        tools=[word_count, extract_keywords],
    )

This pattern—combining LLM reasoning with deterministic tools—produces more reliable results. The LLM decides when to use tools and how to interpret results, but the actual computation happens in Python where precision is guaranteed.

Running the Demo

With the architecture explained, here's how to run the demo yourself. Setup takes about five minutes.

Prerequisites

You'll need Python 3.10 or higher and Foundry Local installed on your machine. Install Foundry Local by following the instructions at github.com/microsoft/Foundry-Local, then verify it works:

foundry --help

Installation

Clone the repository and set up a virtual environment:

git clone https://github.com/leestott/agentframework--foundrylocal.git
cd agentframework--foundrylocal

python -m venv .venv

# Windows
.venv\Scripts\activate

# macOS / Linux
source .venv/bin/activate

pip install -r requirements.txt
copy .env.example .env

CLI Usage

Run the research workflow from the command line:

python -m src.app "What are the key features of Foundry Local and how does it compare to cloud inference?" --docs ./data

You'll see agent-by-agent progress with timing information:

Web Interface

For a visual experience, launch the Flask-based web UI:

python -m src.app.web

Open http://localhost:5000 in your browser. The web UI provides real-time streaming of agent progress, a visual pipeline showing both orchestration patterns, and an interactive demos tab showcasing tool calling capabilities.

CLI Options

The CLI supports several options for customisation:

--docs: Folder of local documents to search (default: ./data)
--model: Foundry Local model alias (default: qwen2.5-0.5b)
--mode: full for sequential + concurrent, or sequential for simpler pipeline
--log-level: DEBUG, INFO, WARNING, or ERROR

For higher quality output, try larger models:

python -m src.app "Explain multi-agent benefits" --docs ./data --model qwen2.5-7b

Validate Tool/Function Calling

Run the dedicated tool calling demo to verify function calling works:

python -m src.app.tool_demo

This tests direct tool function calls (word_count, extract_keywords), LLM-driven tool calling via the ToolAgent, and multi-tool requests in a single prompt.

Run Tests

Run the smoke tests to verify your setup:

pip install pytest pytest-asyncio
pytest tests/ -v

The smoke tests check document loading, tool functions, and configuration—they do not require a running Foundry Local service.

Interactive Demos: Exploring MAF Capabilities

Beyond the research workflow, the web UI includes five interactive demos showcasing different MAF capabilities. Each demonstrates a specific pattern with suggested prompts and real-time results.

Weather Tools demonstrates multi-tool calling with an agent that provides weather information, forecasts, city comparisons, and activity recommendations. The agent uses four different tools to construct comprehensive responses.

Math Calculator shows precise calculation through tool calling. The agent uses arithmetic, percentage, unit conversion, compound interest, and statistics tools instead of attempting mental math—eliminating the calculation errors that plague LLM-only approaches.

Sentiment Analyser performs structured text analysis, detecting sentiment, emotions, key phrases, and word frequency through lexicon-based tools. The results are deterministic and verifiable.

Code Reviewer analyses code for style issues, complexity problems, potential bugs, and improvement opportunities. This demonstrates how tool calling can extend AI capabilities into domain-specific analysis.

Multi-Agent Debate showcases sequential orchestration with interdependent outputs. Three agents—one arguing for a position, one against, and a moderator—debate a topic. Each agent receives the previous agent's output, demonstrating how multi-agent systems can explore topics from multiple perspectives.

Troubleshooting

Common issues and their solutions:

foundry: command not found: Install Foundry Local from github.com/microsoft/Foundry-Local
foundry-local-sdk is not installed: Run pip install foundry-local-sdk
Model download is slow: First download can be large. It's cached for future runs.
No documents found warning: Add .txt or .md files to the --docs folder
Agent output is low quality: Try a larger model alias, e.g. --model phi-3.5-mini
Web UI won't start: Ensure Flask is installed: pip install flask
Port 5000 in use: The web UI uses port 5000. Stop other services or set PORT=8080 environment variable

Key Takeaways

Multi-agent systems decompose complex tasks: Specialised agents (Planner, Retriever, Critic, Writer) produce better results than single-agent approaches by focusing each agent on what it does best
Local AI eliminates cloud dependencies: Foundry Local provides on-device inference with automatic hardware acceleration, keeping all data on your machine
MAF simplifies agent development: The ChatAgent abstraction handles message threading, tool execution, and response parsing, letting you focus on agent behaviour
Three orchestration patterns serve different needs: Sequential pipelines maintain dependencies; concurrent fan-out parallelises independent work; feedback loops enable iterative quality improvement
Feedback loops improve quality: The Critic–Retriever feedback loop catches gaps and contradictions, iterating until quality standards are met rather than producing incomplete results
Tool calling adds precision: Deterministic functions for counting, calculation, and analysis complement LLM reasoning for more reliable results
The same patterns scale to production: This demo architecture—bootstrapping, agent creation, orchestration—applies directly to real-world research and analysis systems

Conclusion and Next Steps

The Local Research & Synthesis Desk demonstrates that sophisticated multi-agent AI systems don't require cloud infrastructure. With Microsoft Agent Framework for orchestration and Foundry Local for inference, you can build production-quality workflows that run entirely on your hardware.

The architecture patterns shown here—specialised agents with clear roles, sequential pipelines for dependent tasks, concurrent fan-out for independent work, feedback loops for quality assurance, and tool calling for precision—form a foundation for building more sophisticated systems. Consider extending this demo with:

Additional agents for fact-checking, summarisation, or domain-specific analysis
Richer tool integrations connecting to databases, APIs, or local services
Human-in-the-loop approval gates before producing final reports
Different model sizes for different agents based on task complexity

Start with the demo, understand the patterns, then apply them to your own research and analysis challenges. The future of AI isn't just cloud models—it's intelligent systems that run wherever your data lives.

Resources

Local Research & Synthesis Desk Repository – Full source code with documentation and examples
Foundry Local – Official site for on-device AI inference
Foundry Local GitHub Repository – Installation instructions and CLI reference
Foundry Local SDK Documentation – Python SDK reference on Microsoft Learn
Microsoft Agent Framework Documentation – Official MAF tutorials and user guides
MAF Orchestrations Overview – Deep dive into workflow patterns
agent-framework-core on PyPI – Python package for MAF
Agent Framework Samples – Additional MAF examples and patterns

Deploying Custom Models with Microsoft Olive and Foundry Local

Abdulhamid_Onawole — Wed, 11 Feb 2026 16:55:06 GMT

Over the past few weeks, we've been on quite a journey together. We started by exploring what makes Phi-4 and small language models so compelling, then got our hands dirty running models locally with Foundry Local. We leveled up with function calling, and most recently built a complete multi-agent quiz application with an orchestrator coordinating specialist agents.

Our quiz app works great locally, but it relies on Foundry Local's catalog models — pre-optimized and ready to go. What happens when you want to deploy a model that isn't in the catalog? Maybe you've fine-tuned a model on domain-specific quiz data, or a new model just dropped on Hugging Face that you want to use. Today we'll take a model from Hugging Face, optimize it with Microsoft Olive, register it with Foundry Local, and run our quiz app against it. The same workflow applies to any model you might fine-tune for your specific use case.

Understanding Deployment Options

Before we dive in, let's understand the landscape of deployment options for SLM applications. There are several routes to deploying SLM applications depending on your target environment.

The Three Main Paths

vLLM is the industry standard for cloud deployments — containerized, scalable, handles many concurrent users. Great for Azure VMs or Kubernetes.

Ollama offers a middle ground — simpler than vLLM but still provides Docker support for easy sharing and deployment.

Foundry Local + Olive is Microsoft's edge-first approach. Optimize your model with Olive, serve with Foundry Local or a custom server. Perfect for on-premise, offline, or privacy-focused deployments.

In keeping with the edge-first theme that's run through this series, we'll focus on the Foundry Local path. We'll use Qwen 2.5-0.5B-Instruct — small enough to optimize quickly and demonstrate the full workflow. Think of it as a stand-in for a model you've fine-tuned on your own quiz data.

Prerequisites

You'll need:

Foundry Local version 0.8.117 or later
Python 3.10+ for the quiz app (the foundry-local-sdk requires it)
A separate Python 3.9 environment for Olive (Olive 0.9.x has this requirement)
The quiz app from the previous article

Having two Python versions might seem odd, but it mirrors a common real-world setup: you optimize models in one environment and serve them in another. The optimization is a one-time step.

Installing Olive Dependencies

In your Python 3.9 environment:

pip install olive-ai onnxruntime onnxruntime-genai pip install transformers>=4.45.0,<5.0.0

Important: Olive is not compatible with Transformers 5.x. You must use version 4.x.

Model Optimization with Olive

Microsoft Olive is the bridge between a Hugging Face model and something Foundry Local can serve. It handles ONNX conversion, graph optimization, and quantization in a single command.

Understanding Quantization

Quantization reduces model size by converting weights from high-precision floating point to lower-precision integers:

Precision	Size Reduction	Quality	Best For
FP32	Baseline	Best	Development, debugging
FP16	50% smaller	Excellent	GPU inference with plenty of VRAM
INT8	75% smaller	Very Good	Balanced production
INT4	87.5% smaller	Good	Edge devices, resource-constrained

We'll use INT4 to demonstrate the maximum compression. For production with better quality, consider INT8 — simply change --precision int4 to --precision int8 in the commands below.

Running the Optimization

The optimization script at scripts/optimize_model.py handles two things: downloading the model locally (to avoid authentication issues), then running Olive.

The download step is important. The ONNX Runtime GenAI model builder internally requests HuggingFace authentication even for public models. Rather than configuring tokens, we download the model first with token=False, then point Olive at the local path:

from huggingface_hub import snapshot_download local_path = snapshot_download("Qwen/Qwen2.5-0.5B-Instruct", token=False)

Then the Olive command runs against the local copy:

cmd = [ sys.executable, "-m", "olive", "auto-opt", "--model_name_or_path", local_path, "--trust_remote_code", "--output_path", "models/qwen2.5-0.5b-int4", "--device", "cpu", "--provider", "CPUExecutionProvider", "--precision", "int4", "--use_model_builder", "--use_ort_genai", "--log_level", "1", ]

Key flags: --precision int4 quantizes weights to 4-bit integers, --use_model_builder reads each transformer layer and exports it to ONNX, and --use_ort_genai outputs in the format Foundry Local consumes.

Run it:

python scripts/optimize_model.py

This process takes about a minute. When complete, you'll see the output directory structure.

models/qwen2.5-0.5b-int4/model/ ├── model.onnx # ONNX graph (162 KB) ├── model.onnx.data # Quantized INT4 weights (823 MB) ├── genai_config.json # ONNX Runtime GenAI config ├── tokenizer.json # Tokenizer vocabulary (11 MB) ├── vocab.json # Token-to-ID map (2.7 MB) ├── merges.txt # BPE merges (1.6 MB) ├── tokenizer_config.json ├── config.json ├── generation_config.json ├── special_tokens_map.json └── added_tokens.json

Total size: approximately 838MB — a significant reduction from the original, while maintaining usable quality for structured tasks like quiz generation.

Registering with Foundry Local

With the model optimized, we need to register it with Foundry Local. Unlike cloud model registries, there's no CLI command — you place files in the right directory and Foundry discovers them automatically.

Foundry's Model Registry

foundry cache cd # Windows: C:\Users\<username>\.foundry\cache\ # macOS/Linux: ~/.foundry/cache/

Foundry organizes models by publisher:

.foundry/cache/models/ ├── foundry.modelinfo.json ← catalog of official models ├── Microsoft/ ← pre-optimized Microsoft models │ ├── qwen2.5-7b-instruct-cuda-gpu-4/ │ ├── Phi-4-cuda-gpu-1/ │ └── ... └── Custom/ ← your models go here

The Registration Script

The script at scripts/register_model.sh does two things: copies all model files into the Foundry cache, and creates the inference_model.json configuration file.

The critical file is inference_model.json — without it, Foundry won't recognize your model:

{ "Name": "qwen-quiz-int4", "PromptTemplate": { "system": "<|im_start|>system\n{Content}<|im_end|>", "user": "<|im_start|>user\n{Content}<|im_end|>", "assistant": "<|im_start|>assistant\n{Content}<|im_end|>", "prompt": "<|im_start|>user\n{Content}<|im_end|>\n<|im_start|>assistant" } }

The PromptTemplate defines the ChatML format that Qwen 2.5 expects. The {Content} placeholder is where Foundry injects the actual message content at runtime. If you were deploying a Llama or Phi model, you'd use their respective prompt templates.

Run the registration:

scripts/register_model.sh

Verify Registration

foundry cache ls

Test the Model

foundry model run qwen-quiz-int4

The model loads via ONNX Runtime on CPU. Try a simple prompt to verify it responds.

Integrating with the Quiz App

Here's where things get interesting. The application-level change is one line in utils/foundry_client.py:

# Before: DEFAULT_MODEL_ALIAS = "qwen2.5-7b-instruct-cuda-gpu" # After: DEFAULT_MODEL_ALIAS = "qwen-quiz-int4"

But that one line raised some issues worth understanding.

Issue 1: The SDK Can't See Custom Models

The Foundry Local Python SDK resolves models by looking them up in the official catalog — a JSON file of Microsoft-published models. Custom models in the Custom/ directory aren't in that catalog. So FoundryLocalManager("qwen-quiz-int4") throws a "model not found" error, despite foundry cache ls and foundry model run both working perfectly.

The fix in foundry_client.py is a dual code path. It tries the SDK first (works for catalog models), and when that fails with a "not found in catalog" error, it falls back to discovering the running service endpoint directly:

def _discover_endpoint(): """Discover running Foundry service endpoint via CLI.""" result = subprocess.run( ["foundry", "service", "status"], capture_output=True, text=True, timeout=10 ) match = re.search(r"(http://\S+?)(?:/openai)?/status", result.stdout) if not match: raise ConnectionError( "Foundry service is not running.\n" f"Start it with: foundry model run {DEFAULT_MODEL_ALIAS}" ) return match.group(1)

The workflow becomes two terminals:

Terminal 1: foundry model run qwen-quiz-int4

Terminal 2: python main.py

The client auto-discovers the endpoint and connects. For catalog models, the existing FoundryLocalManager path works unchanged.

Issue 2: Tool Calling Format

For catalog models, Foundry's server-side middleware intercepts <tool_call> tags in the model's output and converts them into structured tool_calls objects in the API response. This is configured via metadata in foundry.modelinfo.json.

For custom models, those metadata fields aren't recognized — Foundry ignores them in inference_model.json. The <tool_call> tags pass through as raw text in response.choices[0].message.content.

Since our custom model outputs the exact same <tool_call> format, we added a small fallback parser in agents/base_agent.py — the same pattern we explored in our function calling article. After each model response, if tool_calls is None, we scan the content for tags:

def _parse_text_tool_calls(content: str) -> list: """Parse <tool_call>...</tool_call> tags from model output.""" blocks = re.findall(r"<tool_call>\s*(\{.*?\})\s*</tool_call>", content, re.DOTALL) calls = [] for block in blocks: try: data = json.loads(block) calls.append(_TextToolCall(data["name"], json.dumps(data.get("arguments", {})))) except (json.JSONDecodeError, KeyError): continue return calls

The model's behavior is identical; only the parsing location changes — from server-side (Foundry middleware) to client-side (our code).

Part 7: Testing the Deployment

With the model running in one terminal, start the quiz app in another:

Terminal 1:

foundry model run qwen-quiz-int4

Terminal 2:

cd multi_agents_slm && python main.py

Now test the full flow. Generate a quiz:

Test the Full Flow

Generate a quiz:

Example output:

The orchestrator successfully calls the generate_new_quiz tool, and the QuizGeneratorAgent produces well-structured quiz JSON.

Model Limitations

The 0.5B INT4 model occasionally struggles with complex reasoning or basic arithmetic. This is expected from such a small, heavily quantized model. For production use cases requiring higher accuracy, use Qwen 2.5-1.5B or Qwen 2.5-7B for better quality, or use INT8 quantization instead of INT4. The deployment workflow remains identical — just change the model name and precision in the optimization script.

What You've Accomplished

Take a moment to appreciate the complete journey across this series:

Article	What You Learned
1. Phi-4 Introduction	Why SLMs matter, performance vs size tradeoffs
2. Running Locally	Foundry Local setup, basic inference
3. Function Calling	Tool use, external API integration
4. Multi-Agent Systems	Orchestration, specialist agents
5. Deployment	Olive optimization, Foundry Local registration, custom model deployment

You now have end-to-end skills for building production SLM applications: understanding the landscape, local development with Foundry Local, agentic applications with function calling, multi-agent architectures, model optimization with Olive, and deploying custom models to the edge.

Where to Go From Here

The logical next step is fine-tuning for your domain. Medical quiz tutors trained on USMLE questions, legal assistants trained on case law, company onboarding bots trained on internal documentation — use the same Olive workflow to optimize and deploy your fine-tuned model. The same ONNX model we registered with Foundry Local could also run on mobile devices via ONNX Runtime Mobile, or be containerized for server-side edge deployment.

The full source code, including the optimization and registration scripts, is available in the GitHub repository.

Resources:

Microsoft Olive — Model optimization toolkit
Foundry Local Documentation — Setup and CLI reference
Compiling Hugging Face models for Foundry Local — Official guide
ONNX Runtime GenAI — Powers Foundry Local's inference
Edge AI for Beginners — Microsoft's 8-module Edge AI curriculum
Quiz App Source Code — Full repository with deployment scripts

This series has been a joy to write. I'd love to see what you build — share your projects in the comments, and don't hesitate to open issues on the GitHub repo if you encounter challenges.

Until next time — keep building, keep optimizing, and keep pushing what's possible with local AI.