Beyond One-Size-Fits-All: Decision Routing for GitHub Copilot in CLI & VS Code
As developer tools move into the age of agentic AI, it is easy to fall into a familiar habit: when a task matters, choose the strongest—and usually most expensive—model available. Real software work is more varied than that. Explaining a function, editing a README, adding a boundary test, debugging a cross-module race condition and reviewing authentication logic demand very different levels of reasoning, context, tool use and reliability. A fixed model for every turn either wastes latency and budget or leaves difficult tasks underpowered. That is the motivation behind decision-ghc-router: GitHub Copilot continues to execute, while a Decision Router selects a sufficiently reliable and economically sensible model at the start of each new user turn. (The design also draws on gargpratyush/jev-router, especially its per-turn routing, native CLI preservation and model pinning across tool continuations.) The goal is not to build another agent outside Copilot or replace the native developer experience with a custom chat UI. GitHub Copilot CLI and the VS Code GitHub Copilot extension should keep their existing authentication, model catalog, tools, permissions, sessions and streaming behavior. The router adds one short, explicit decision step—and then gets out of the way. 1. Start with the distinction: a Decision Model is not a Model Router Before discussing implementation, two frequently conflated concepts need to be separated. A Decision Model makes bounded, structured judgments A Decision Model is not prompted with “write an article.” It receives: a current state, expressed as text or JSON; one or more bounded questions; explicit options or grading criteria. It returns typed results and probabilities that software can act on immediately, rather than free-form prose. For example: noul: Does this tool call touch production data? Return a probability between 0 and 1. choice: Should this task use the Fast, Balanced or Strong tier? Return the selected option and a probability for every option. score: How risky is this code change? Return an ordered score and its probability distribution. A Decision Model is therefore a fast, calibratable judgment primitive. Its uses extend beyond model selection to classification, ranking, prioritization, validation, agent control, safety screening and AI judging. A Model Router turns judgment into execution A Model Router is a system pattern, not a single model. A production-ready router must typically: discover models actually available to the signed-in account; reject candidates that are unavailable, unpriced, context-insufficient or API-incompatible; build a minimal routing state; invoke a scorer—which might be rules, a benchmark table, an LLM classifier or a Decision Model; apply confidence, cost, latency, safety and organizational policy; select a concrete model and start native execution; pin that model across tool continuations; emit decisions, estimates and failures for audit and tuning. In short: The Decision Model judges what is most suitable. The Model Router makes that judgment usable, compliant, executable, observable and recoverable. A Decision Router does not replace the larger routing system. It gives that system a specialized, fast and threshold-friendly scoring engine. Decision Model versus Model Router Dimension Decision Model Model Router Nature Specialized model capability Application or platform architecture pattern Input State, bounded questions, options or grading criteria User task, account catalog, prices, context limits and policy Output Typed answer and probabilities Concrete model, execution path, policy outcome and telemetry Content generation Does not produce free-form content Usually dispatches a generative model Account entitlement Not responsible Responsible API compatibility Not responsible Responsible Failure handling Provides uncertainty Must define fallback or stop behavior Broader use Classification, scoring, routing and validation Multi-model orchestration, cost governance and runtime control 2. Microsoft-Decision-1: why it fits developer-tool routing Microsoft introduced Microsoft-Decision-1 in October 2026 as a specialized model for fast decision scoring, available through Microsoft Foundry. Unlike a generative LLM, it does not produce an essay or a written chain of reasoning. It makes a focused judgment about text or JSON and returns typed numerical decisions. Microsoft describes the model as a post-trained Qwen3.5-9B designed for single-pass scoring over fixed options. It reports the highest accuracy in a 36-benchmark comparison spanning nearly 150,000 questions kept blind from training, P50 latency roughly 35 times faster than GPT-6 Sol, and an average 1.3% decision-flip rate across eight perturbation types. At launch, Microsoft listed input at US$0.042 per million tokens with no output-token charge; pricing is dynamic, so a deployment should use the current Microsoft Foundry listing. Those are Microsoft-published benchmark results, not a substitute for evaluation on a team’s own development workload. Microsoft-Decision-1 exposes three question types that map naturally to an AI control plane: noul for binary gates such as “May this command run automatically?” choice for bounded routing such as “Which model tier should handle this turn?” score for ordered grading such as “How severe is the change risk?” Why not use a small LLM classifier? That is a valid architecture. Many model routers ask a lightweight LLM to emit easy, medium or hard. GitHub Agentic Workflows’ experimental model routing also classifies a task before ranking allowed model-and-effort combinations through precompiled routing tables. A dedicated Decision Model, however, offers four useful properties for developer tools: Bounded output. A router needs a label and probabilities, not persuasive prose. Predictable latency. Agent workflows amplify the cost of every sequential control-plane call. Policy-ready confidence. If the top probability is below 0.65, a local policy can avoid downgrading, keep the current model or request manual selection. Composable governance. One structured request can score model tier, operational risk and the need for human review. A conceptual choice request for Copilot routing The following is simplified pseudodata—not repository source—and omits deployment and authentication details: { "state": { "task": "Fix intermittent failures in the authentication integration tests", "estimatedContextTokens": 18000, "toolUseExpected": true, "workspaceScope": "multi-file", "latencyPreference": "balanced" }, "questions": { "model_tier": { "type": "choice", "instructions": "Choose the least expensive tier likely to complete the task reliably in one pass.", "criteria": { "fast": "Simple explanation, formatting, lookup, or small deterministic edit", "balanced": "Normal coding, tests, debugging, or limited multi-file work", "strong": "Ambiguous architecture, security, difficult debugging, or broad multi-file reasoning" } } } } The response identifies a tier and assigns a probability to every tier. The router then maps the abstract choice to a concrete model that is truly available to the current GitHub account. That separation matters: a Decision Model should not assume that a user owns or can access a particular model. The account catalog is the source of truth. 3. The decision-ghc-router scenario and reference architecture The architecture has two hosts but one consistent decision model. 3.1 GitHub Copilot CLI: put the router in native /model In the public predecessor implementation, a user-level extension adds a router entry to GitHub Copilot CLI’s native /model menu. The user can select the router or choose a concrete model directly: Select Decision Router: route every new user turn. Select a concrete model: bypass routing and return to native fixed-model behavior. Continue a tool call: keep the model chosen for that turn; do not classify again. Explicitly request a model: the explicit request should win. The CLI path uses a random-port 127.0.0.1 proxy to expose the router as a temporary provider while preserving Copilot authentication, sessions, tools, permissions and streaming. GitHub credentials should remain in memory and be used only for HTTPS requests to GitHub. The local proxy uses separate random credentials; GitHub credentials must not be written to logs or sent to the Decision API. This is the design principle I care about most: The router changes which model is selected, not how Copilot works. 3.2 VS Code and VS Code Insiders: enter the Chat Model Picker The editor integration is a separate extension. It registers decision-router in VS Code Chat’s model picker and uses the public Language Model API to discover Copilot models available to the account signed into that editor. The privacy boundary is intentional: The Decision Model receives an unambiguously extracted task or a routing-only summary entered by the user. If the task cannot be reliably isolated from editor context, the extension should not automatically upload attachments, source files or the full conversation. The selected Copilot model still receives the full conversation, tool definitions and workspace context. VS Code invokes native tools, while tool-call IDs maintain model consistency. The short version is: minimize data for decision; maximize context for execution. 3.3 The seven-step loop for each new user turn Receive the task through native Copilot CLI or VS Code Chat. Discover candidates from account models, prices, context capabilities and API compatibility. Build routing state with the minimum sufficient task summary and bounded choices. Call Microsoft-Decision-1 and receive choice probabilities. Apply local policy for confidence, cost, compatibility and manual overrides. Execute natively with the selected model, existing tools, permissions, session and streaming. Pin and trace the model for tool continuations while recording the decision and estimate. 3.4 Why not reroute every tool continuation? Imagine a model runs tests, reads the failure output and then edits code. If the router chooses a new model after every tool result: prompt-cache benefits may disappear, increasing total cost; the new model may interpret earlier tool calls differently; the debugging chain becomes harder to reproduce and audit. I therefore retain one of jev-router’s strongest ideas: make one decision at the start of a new user turn, then pin that model for the resulting tool chain. Main conversations and sub-agents should be pinned independently. 4. Main technologies and frameworks Based on mechanisms verifiable in the public predecessor and the Microsoft-Decision-1 evolution described here, the project separates into the following layers. Node.js and ESM modules The core router uses Node.js and .mjs modules to handle: CLI startup and argument forwarding; model catalog and price adaptation; the local HTTP proxy; configuration and environment variables; Decision API calls; policy, errors and telemetry. The public predecessor requires Node.js 22.16 or later and uses Microsoft’s npm package source for relevant dependencies. GitHub Copilot CLI extension interfaces The CLI integration relies on experimental extension interfaces to register the router as a native model entry. It explicitly enables the provider-endpoint interface and obtains the model catalog, runtime inference endpoint and authentication from the active session. Three rules are essential: Do not modify the user’s global default model. Do not automatically grant shell or other tool permissions. If initialization fails, report a clear error instead of pretending routing succeeded. Local loopback proxy The loopback proxy is the adapter that preserves the native CLI: listen only on 127.0.0.1; use a random port and random local credentials; map the router’s selection to a real Copilot execution model; preserve streaming, tool calls, sessions and permission prompts; never create a second hidden agent loop. VS Code Extension API The editor extension discovers current candidates through: vscode.lm.selectChatModels({ vendor: "copilot" }) and is packaged as a VSIX. The CLI and editor are independent hosts and may use different GitHub accounts with different entitlements. Candidate discovery must therefore be host-specific, session-specific and dynamic—not a hardcoded list of model IDs. Microsoft Foundry and the Decision API Microsoft-Decision-1 can be deployed in Microsoft Foundry through GlobalStandard or, in selected regions, DataZoneStandard. Authentication can use Microsoft Entra ID (recommended) or an API key. In this architecture, the Decision API only scores choices: Input: minimal task state, bounded candidates and explicit criteria. Output: selected option, option probabilities and usage. Not its responsibility: GitHub credentials, developer tools, full workspace access or code execution. That separation makes the decision plane and execution plane independently governable. Model catalog, pricing and capability adapters The public predecessor discovers models from the signed-in account instead of hardcoding them. Before scoring, a router should exclude: auto, hidden, disabled or unconfigured entries; models without usable price information; candidates with insufficient input context; models incompatible with the host’s wire API; models restricted by administrator policy. Cost estimation supports ranking and user feedback; it is not a final bill. Promotions, cache discounts, hidden context, retries and later tool rounds all affect the result. GitHub Billing remains authoritative. 5. What I borrow from jev-router—and what changes The value of gargpratyush/jev-router is not merely “use a small model for easy tasks.” It embeds routing into familiar CLIs without taking away their native behavior. The ideas I retain include: one routing decision per new user turn; a router sentinel in the native model picker; concrete-model selection pauses routing; abstract Fast, Balanced, Strong and Long tiers map to host-specific models; tool continuations keep the tier selected at the start of the turn; explicit model requests win; low confidence should not trigger an aggressive downgrade; large conversations must account for cache and migration cost; explanation data should show why a tier was selected. The decision-ghc-router evolution replaces the external JEV/System One scorer with Microsoft-Decision-1 and emphasizes five changes: Probability is a first-class signal. Policy can threshold the decision instead of accepting a label blindly. Bounded choices come from the real catalog. Decision-1 only chooses among already eligible candidates. Local policy owns the final answer. Organizational rules, compatibility and manual overrides remain deterministic. The Decision Model is reusable. A single structured call can also score operational risk and the need for human review. Local and cloud responsibilities are explicit. Cloud scoring is isolated from credentials and execution. 6. Representative use cases A. Fast explanation and small edits “Explain this regular expression,” “turn this note into a table,” or “add a null-value test to an existing function.” The router can often select the Fast tier. The objective is not the weakest model; it is the least expensive model likely to complete the task reliably in one pass. B. Normal development and debugging “Fix the failing test,” “refactor this module without changing its API,” or “use the stack trace to locate the bug.” These tasks require code reading, tool use and moderate reasoning, so they often land in Balanced. C. Security, architecture and broad multi-file reasoning “Review this OAuth callback for state confusion,” “design cross-region failover,” or “find a race condition across services.” The router should choose Strong even when it costs more. Routing optimization means “economical subject to reliable completion,” not “always cheaper.” D. Long context and large repositories If a candidate cannot hold the estimated input, it must be removed before scoring. For an already large conversation, the router should also avoid downgrading when lost cache work or context migration would cost more than the model-price saving. E. Organizational policy and risk gates The same Decision API request can include additional noul or score questions: Does the task involve production data? Is the proposed command destructive? What is the security risk level of the change? Should execution be handed to a human reviewer? At that point the router grows from a cost optimizer into a lightweight control plane for agents. 7. Policies I would keep for production Filter deterministically before scoring Entitlement, API compatibility, context limits and administrator restrictions are facts. The Decision Model should not be asked to guess them. Do not automatically downgrade on low confidence When the top probability falls below a calibrated threshold, the router can: keep the current model; cap automatic movement at Balanced; request manual selection; send high-risk tasks to Strong or human review. The threshold must be calibrated on labeled tasks from the target environment, not copied from an example. Explicit intent beats automatic routing If the user says “use the Strong model” or selects a concrete entry in the picker, the router should step aside immediately. Never substitute silently Missing credentials, Decision API timeouts, unknown pricing and insufficient context should produce an explicit state. Whether to fail open, fail closed or keep the current model must be an intentional host or organizational policy. Record decisions without logging every prompt by default Useful telemetry includes: anonymized session and turn IDs; candidates and exclusion reasons; selected tier, probabilities and policy overrides; estimated input/output and one-turn cost; completion, retry and manual-reselection outcomes. Full prompts, source code and tool results should be excluded by default or governed by explicit retention and data-handling policy. Evaluate the router on task success, not the cheapest request Better metrics include: first-pass completion rate; total time to completion; total cost including retries and tool rounds; manual model-change rate; low-confidence and misrouting rates; safety and policy violations. 8. Why this matters beyond one repository From a developer-advocacy perspective, the Decision Router is not simply a token-saving utility. It illustrates a clearer division of responsibilities in agent engineering: Generative models understand, reason, write code and call tools. Decision Models make fast, structured and probabilistic judgments. Routers and policy engines place those judgments inside real account, budget, compatibility and governance constraints. Developer-tool hosts retain identity, permission, context and execution control. Future AI developer tools will not be tied to one “default large model.” They will manage a portfolio of models with different speed, cost, context and capability profiles. The differentiator will no longer be merely which model a product connects to, but: Can it select the right model at the right moment, in an explainable and governable way, without breaking the developer experience? That is why I am adding a Decision Router to GitHub Copilot CLI and the VS Code GitHub Copilot extension. GitHub Copilot keeps executing. Microsoft-Decision-1 makes the fast judgment. The local router turns that judgment into a usable engineering decision. References decision-ghc-router jev-ghc-router gargpratyush/jev-router Microsoft-Decision-1: Our model for fast decision-making Deploy and use Microsoft-Decision-1 in Microsoft Foundry GitHub Agentic Workflows — Model Routing GitHub Copilot model billing and pricing
I NEED HUMAN HELP -- I am having issues with my Microsoft business email, I can't access Word
888 319 8920 I am having issues with my Microsoft business email, I can't access Word. I've tried calling multiple numbers and haven't been able to reach a human and the automated machine is no help. If someone would please give me a good number to call or an email address, I'd greatly appreciate it.Introducing bicep docs: generate documentation from your Bicep modules
TL;DR: The Bicep CLI has a new experimental command, bicep docs generate . Point it at a Bicep module and it writes a README.md next to it that describes the module's resources, parameters, types, outputs and usage examples, all read straight from your code. You can change the layout with your own templates, add details that aren't in the code, and document every module in a repository with one command. It's available now in Bicep CLI v0.47.16 and later. bicep docs generate ./main.bicep Why we built it Module documentation has a habit of drifting away from the code it describes. Someone renames a parameter, adds an output or changes an allowed value, and the README quietly goes out of date. Teams that publish a lot of modules often end up writing their own scripts to keep the two in step. bicep docs moves that job into the Bicep CLI. It compiles your module and uses the compiler's own understanding of it, so it picks up the details Bicep already knows: metadata , @description decorators, default values, allowed values, length limits, user-defined types, exported types, variables and functions, outputs, and the local modules it references. Run it again whenever the module changes, and the documentation will match the code. Get started You need Bicep CLI v0.47.16 or later. Check your version with bicep --version , and see Install Bicep tools if you need to upgrade. There's no feature flag to turn on. Here's a small storage account module: metadata name = 'Storage Account' metadata description = 'Deploys a storage account with secure defaults.' @description('Required. The name of the storage account. Must be globally unique.') @minLength(3) @maxLength(24) param name string @description('Optional. The Azure region to deploy the storage account to.') param location string = resourceGroup().location @description('Optional. The storage account SKU.') @allowed([ 'Standard_LRS' 'Standard_GRS' 'Standard_ZRS' ]) param skuName string = 'Standard_LRS' resource storageAccount 'Microsoft.Storage/storageAccounts@2025-06-01' = { name: name location: location kind: 'StorageV2' sku: { name: skuName } properties: { allowBlobPublicAccess: false minimumTlsVersion: 'TLS1_2' } } @description('The resource ID of the storage account.') output resourceId string = storageAccount.id Generate its documentation: bicep docs generate ./modules/storage-account/main.bicep Bicep writes README.md next to main.bicep . For this module, the built-in layout includes a list of sections, the resource types, a usage example (more on those below), the parameters with the details of each one, and the outputs. When a module has them, it also covers exported types, variables and functions, and referenced modules. Here's an excerpt: # Storage Account Deploys a storage account with secure defaults. ## Parameters | Name | Type | Required | Description | | :-- | :-- | :-- | :-- | | `location` | `string` | No | Optional. The Azure region to deploy the storage account to. | | `name` | `string` | Yes | Required. The name of the storage account. Must be globally unique. | | `skuName` | `string` | No | Optional. The storage account SKU. | ### `location` - Default value: `resourceGroup().location` ### `name` - Min length: 3 - Max length: 24 ### `skuName` - Default value: `'Standard_LRS'` - Allowed values: `Standard_GRS`, `Standard_LRS`, `Standard_ZRS` A couple of options you'll use often: --stdout prints the result to the terminal instead of writing a file, which is handy for a quick preview. --outfile writes to a file of your choice, and --outdir writes the documentation into another folder. Because the command is experimental, Bicep prints a warning each time you run it. Usage examples are included automatically Examples are often the most useful part of module documentation, so bicep docs looks for them next to your module. By default it picks up: .bicep files in an examples folder, and main.bicep files in its subfolders *.test.bicep files anywhere under a tests folder Files named dependencies*.bicep are skipped. Each example's heading comes from its metadata name . Without one, Bicep uses the name of the folder the example is in, or its file name if it's not in a subfolder. The description comes from metadata description , or else the // comments at the top of the file. For example, this file in modules/storage-account/examples becomes a "Zone-redundant storage account" example in the README: metadata name = 'Zone-redundant storage account' metadata description = 'Deploys a storage account that keeps copies of your data in three availability zones.' module storageAccount '../main.bicep' = { params: { name: 'stdocsdemo001' skuName: 'Standard_ZRS' } } If your examples live somewhere else, list their locations in documentation.examples.sources in bicepconfig.json . Your list replaces the default locations. Make it your own with templates The built-in layout is a good start, but you'll probably have your own conventions. bicep docs renders documentation with Scriban, a lightweight text templating language. Write a template, then point to it from bicepconfig.json : { "documentation": { "template": { "file": "templates/readme.scriban" } } } The template path is relative to the bicepconfig.json file. As with its other settings, Bicep uses the bicepconfig.json nearest to each module. This template writes a parameters table and an outputs table: # {{ module.name }} {{ module.description }} ## Parameters | Name | Type | Required | Description | | :-- | :-- | :-- | :-- | {{~ for parameter in module.parameters ~}} | `{{ parameter.name }}` | `{{ parameter.type }}` | {{ if parameter.required }}Yes{{ else }}No{{ end }} | {{ parameter.description }} | {{~ end ~}} ## Outputs | Name | Type | Description | | :-- | :-- | :-- | {{~ for output in module.outputs ~}} | `{{ output.name }}` | `{{ output.type }}` | {{ output.description }} | {{~ end ~}} The ~ inside the braces removes the spaces and line breaks next to a tag, so the loops don't add blank lines that would break the tables. Your template receives a module object with the module's name , description , targetScope , resourceTypes , parameters , exportedTypes , exportedVariables , exportedFunctions , outputs , references and usageExamples . The command documentation lists every field. A couple of tips: Split a big template into smaller files with Scriban's include , and set documentation.template.includeRoot to the folder that holds them. The built-in Markdown template is a Scriban template too, so it's a good place to start. Add details that aren't in your code Some information doesn't belong in Bicep, such as the team that owns a module, where to get support, or the module's version. Pass it in as custom values and use them in your template as custom.<key> : **Owner:** {{ custom.owner }} | **Version:** {{ custom.version }} | [Get support]({{ custom.supportUrl }}) Supply values one at a time with --custom-template-value , or load several from a JSON file with --custom-template-value-file-path : bicep docs generate ./modules/storage-account/main.bicep ` --custom-template-value owner="Platform Team" ` --custom-template-value-file-path ./docs-values.json Where docs-values.json contains: { "supportUrl": "https://contoso.example/support", "version": "1.0.0" } Both options can be repeated. If the same key is set more than once, the last one on the command line wins. Go beyond Markdown Scriban writes whatever text your template contains, so the output doesn't have to be Markdown. A template could produce JSON for a module catalogue, HTML for an internal portal, or input for another tool. This template produces JSON, and Scriban's object.to_json function takes care of quotes and escaping: { "name": {{ module.name | object.to_json }}, "description": {{ module.description | object.to_json }}, "parameters": [ {{~ for parameter in module.parameters ~}} { "name": {{ parameter.name | object.to_json }}, "type": {{ parameter.type | object.to_json }}, "required": {{ parameter.required | object.to_json }} }{{ if !for.last }},{{ end }} {{~ end ~}} ] } Set documentation.template.file to this template, then use --outfile to save the result with a .json file name: bicep docs generate ./modules/storage-account/main.bicep --outfile ./modules/storage-account/module.json To change the default file name for every module instead, set documentation.output.file in bicepconfig.json . Each module uses the one template set in its nearest bicepconfig.json . To get both Markdown and JSON from the same modules, make that template a small switch that includes the right template based on a custom value: {{~ if custom.format == "json" ~}} {{~ include "catalog.scriban" ~}} {{~ else ~}} {{~ include "readme.scriban" ~}} {{~ end ~}} Set documentation.template.includeRoot to the folder that holds the templates. Then run the command as normal for the README, and add --custom-template-value format=json --outfile ./modules/storage-account/module.json for the JSON. Document every module in one go Use --pattern to document every module that matches a wildcard: bicep docs generate --pattern './modules/**/main.bicep' Each module gets its own README.md next to its main.bicep . Add --outdir ./docs to put them in a separate folder instead, and Bicep recreates the folder structure beneath it. If a module has compile errors, Bicep reports them, carries on documenting the other modules, and returns exit code 1 at the end. To see this at scale, the demo repository generates Azure Verified Modules (AVM) style documentation for 20 App Service modules, including their child modules, with a single command. Its template works out each module's registry reference and main resource type from the module itself. The AVM team is standardising and consolidating its tooling, and bicep docs is part of that work. Keep documentation up to date in CI Because the documentation comes from the code, your pipeline can check that it's current. Regenerate it, then fail the build if anything changed: bicep docs generate --pattern './modules/**/main.bicep' if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } $changes = git status --porcelain if ($changes) { $changes throw 'Module documentation is out of date. Run bicep docs generate and commit the result.' } The generated documentation is only as good as your descriptions, so the new description linter rules in Bicep v0.47 make a good partner. They're off by default and flag missing or empty @description decorators. Turn on the ones you need in bicepconfig.json : { "analyzers": { "core": { "rules": { "use-description-params": { "level": "warning" }, "use-description-outputs": { "level": "warning" }, "use-description-types": { "level": "warning" }, "use-description-type-properties": { "level": "warning" } } } } } There's also use-description-vars for variables. Thanks to @johnlokerse for contributing these rules! Building tools on top of Bicep? The same generator is available through the Bicep CLI's JSON-RPC interface as bicep/generateDocs . It returns the rendered text and leaves writing files to you. For .NET, the Azure.Bicep.RpcClient package wraps it. It's experimental, and we'd love your feedback bicep docs is experimental. The command options, configuration, template model and built-in layout may change between releases without a breaking-change notice. One change is already on the way. In v0.47 you can also choose a template with the --template-file and --template-root options. The next release removes these options, so templates are set only with documentation.template.file and documentation.template.includeRoot in bicepconfig.json . The examples in this post already use bicepconfig.json , so they'll keep working. Try it on your own modules and tell us what works and what's missing by opening an issue in the Bicep repository. Learn more bicep docs command documentation Demo repository, with step-by-step demos from the built-in layout to AVM-style documentation Bicep v0.47.16 release notes Scriban language reference bicep docs is also featured in the September 2026 Bicep community call. Community call recordings are on the Azure Deployments & Governance YouTube channel, and you can sign up to join future calls.
Building a Fully Automated Azure Landing Zone Deployment Using Azure DevOps and Terraform
Discover how to build a fully automated Azure Landing Zone using Azure DevOps and Terraform. This article walks through real-world Git workflows, CI/CD automation, environment promotion strategies, and governance integration to create secure, scalable, and enterprise-ready Azure environments.From Multi-Model Chaos to a Governed AI Gateway: Cost Optimization on Azure
What is Multi-Model Chaos, and what cost and security challenges does it pose? Multi-model chaos describes the sprawl that emerges when an organization rapidly adopts many large language and foundation models—OpenAI, Anthropic, Meta Llama, Mistral, and a long tail of open-source and fine-tuned variants—across teams and applications without any unifying control plane. Instead of a single governed entry point, each team wires its own keys, endpoints, SDKs, and prompts directly to whichever provider it prefers, leaving the enterprise with a fragmented, duplicated, and largely invisible AI estate. On the cost side, this fragmentation makes spend almost impossible to predict or contain, identical workloads run against premium models when cheaper ones would suffice, token consumption goes unmeasured, redundant calls and missing caching inflate bills, and finance teams have no consolidated view to attribute usage back to a team, product, or customer. On the security and governance side, the risks compound: API keys are scattered across code and config files, sensitive or regulated data flows to external endpoints with no data-loss prevention or residency guarantees, prompt-injection and jailbreak attempts go unmonitored, and there is no centralized authentication, rate limiting, auditing, or content filtering. The net effect is an uncontrolled attack surface and a compliance blind spot—precisely the conditions that motivate consolidating model access behind a governed AI gateway. In short, multi-model chaos trades short-term speed for runaway costs and an unmanaged security risk, making a governed AI Gateway essential. What is a Governed AI Gateway, and how do they help reduce cost and improve security? A governed AI gateway is an enterprise control plane built on Azure API Management (APIM) that consolidates every model behind a single, governed endpoint. It unifies Azure OpenAI (the gpt-5.4 family) and Azure AI Foundry (open-source and partner models such as grok-4.3 and DeepSeek-V4-Pro), so consumers reach any of them through one consistent, policy-enforced entry point rather than a tangle of direct connections. Every backend is password-less, authenticated through managed identity, which eliminates scattered API keys. On top of this foundation, the gateway enforces per-consumer model permissions, token-based rate limits, and cost-based budget downgrade—automatically routing teams to more economical models as they approach their spend limits—all administered from a self-service Admin UI. One governed endpoint for every backend. Azure OpenAI and Azure AI Foundry (OSS and partner) models are bundled behind a single governance endpoint. Each backend is reachable only over a private endpoint with key authentication disabled, so APIM authenticates using its own managed identity—no model keys ever live on the gateway. Per-consumer governance, edited live in the Admin UI with no redeployment: Allowed models — a consumer can call only the models explicitly granted to it; anything else returns a 403. Rate limits — per-consumer TPM and token-quota tiers (small / medium / large), returning a 429 once exceeded. Cost budget — a daily USD spend limit; when it is exceeded, requests are automatically downgraded to a cheaper model along a configured ladder, including cross-backend downgrades (e.g. gpt → OSS or OSS → gpt). Self-service Admin UI (React + FastAPI, Entra ID login, gated to an admin group) to issue consumer keys, set model, limit, and budget policies, and review the usage dashboard and request logs. Built-in observability — per-call token metrics, broken down by consumer and model, stream to Application Insights, surfaced through the Admin UI's usage dashboard and a request / blocked & downgrade-event log. Flexible client authentication — an APIM subscription key by default, or an Entra ID JWT (client_auth_mode). How is it different from APIM AI Gateway? APIM already provides useful GenAI gateway primitives: token rate limiting, token-usage metrics, semantic caching, backend routing, endpoint import, authentication, authorization, and monitoring. The difference is that APIM enables the enforcement runtime and policy control point, but not the full operating model required to run a shared, multi-tenant AI platform across teams, models, and budgets. Inside the policy pipeline, APIM remains the load-bearing layer: llm-token-limit enforces per-consumer token-per-minute and quota limits, llm-emit-token-metric streams token usage into our metrics namespace, and standard APIM capabilities handle endpoint exposure, access control, and platform monitoring. The governed AI Gateway adds the governance layer APIM does not provide out of the box: Self-service onboarding — a platform team can issue or revoke consumer keys and manage access from the Admin UI, without raising a pull request or redeploying infrastructure. Per-consumer model entitlements — every consumer has an explicit allow-list of model deployments. The gateway calculates the effective allowed set per request and returns 403 when a caller asks for a model it is not entitled to use. Live configuration without redeployment — entitlements, rate tiers, token quotas, budgets, and downgrade levels live in the configuration store. A sync worker projects those values into APIM named values continuously, so operational changes can take effect without a terraform apply while the policy logic stays version-controlled in IaC. Managed-identity-only, private backends — key-based authentication is disabled on Azure OpenAI and Azure AI Foundry. APIM injects a managed identity token on every backend call, and the backends are reachable only over private endpoints. Cost-based downgrade across backends — when a consumer approaches its budget, the gateway can route to a cheaper model while preserving availability, including cross-backend downgrades between Azure OpenAI and Azure AI Foundry. APIM’s AI gateway is the enforcement runtime while the governed AI Gateway is the platform operating model around it. APIM handles the gateway primitives extremely well, while our governance layer adds identity, self-service administration, entitlement management, live configuration, cost controls, and cross-model routing so teams can safely consume multiple models without creating new cost, security, or compliance sprawl. Solution overview Figure 1 shows the end-to-end architecture of the governed AI gateway. Client applications never talk to the models directly; instead, every request passes through Azure API Management, which acts as the single governed entry point that authenticates callers, applies per-consumer policy, and routes traffic privately to the appropriate model backend. Around this gateway sit the supporting planes for administration, identity, and observability, giving the organization one consistent place to control access, contain cost, and monitor usage across both Azure OpenAI and Azure AI Foundry models. This solution is also completely serverless. Key components: Client / consumer applications — the apps and services that call for model inference, each identified by its own consumer key or Entra ID identity. Azure API Management (the gateway) — the single governance endpoint that handles authentication, allowed-model checks, rate limiting, and cost-based budget downgrade before any request reaches a model. Model backends — Azure OpenAI (the gpt-5.4 family) and Azure AI Foundry (OSS and partner models such as grok-4.3 and DeepSeek-V4-Pro), each reachable only over a private endpoint. Microsoft Entra ID — provides identity for both clients (optional JWT auth) and the gateway's own managed identity used to reach the backends without password credentials. Admin UI (React + FastAPI) — the self-service control plane for issuing consumer keys and setting model, rate-limit, and budget policies. Application Insights — collects per-call token metrics by consumer and model, powering the usage dashboard and request / blocked-event logs. 1: Solution architecture diagram Request flow Authenticate — a client calls the gateway with an APIM subscription key (or an Entra ID JWT) instead of any model key. Authorize the model — APIM checks whether the consumer is permitted to call the requested model; if not, it returns 403. Enforce limits — the gateway applies the consumer's TPM and token-quota tier, returning 429 when the limit is exceeded. Apply the cost budget — if the consumer's daily USD budget is exhausted, the request is automatically downgraded to a cheaper model along the configured ladder. Route to the backend — APIM forwards the request over a private endpoint, authenticating with its managed identity to Azure OpenAI or Azure AI Foundry. Return and record — the model response is returned to the client while per-call token metrics are emitted to Application Insights and surfaced in the Admin UI dashboard and logs. Implement the solution This section describes how to deploy the solution architecture. In this post, you’ll perform the following tasks: Create APIM Create Cosmos DB Create Microsoft foundry with Gpt-5.4, Gpt-5.4-mini, DeepSeek-V4-Pro and Grok-4.3 deployed Create the Admin UI on container apps Create a consumer with an APIM subscription key on the Admin UI Integrate APIM endpoint with Github Copilot chat and Copilot CLI Create a budget and rate limit in the Admin UI Simulate and validate auto downgrade feature Ensure that you have the following prerequisites deployed before moving to the next section An Azure subscription with model quota (Azure OpenAI and, optionally, Azure AI Foundry models). Tools: Git, Terraform ≥ 1.7, Azure CLI, and az login to the subscription. Container images are built remotely in Azure Container Registry, so Docker is not required. VScode and Copilot CLI Deploy the Azure AI Gateway Clone the repository from https://github.com/microsoft/apim-foundry-governance git clone https://github.com/microsoft/apim-foundry-governance git checkout english By default the solution deploys in koreacentral region. Export your custom variables if needed. export location=eastus2 export backend-rg=rg-aigw-tfstate-dev-eastus2 export storage-prefix=staigwtfstate export state-key=ai-gateway-eus2.tfstate Bootstrap the Terraform state backend (once per subscription) This creates an eastus2 resource group + storage account for remote state (Entra auth, public blob access blocked). ./scripts/bootstrap-backend.sh \ --location $location \ --backend-rg $backend-rg \ --storage-prefix $storage-prefix \ --state-key $state-key Set Terraform variables cp infra/terraform.tfvars.example infra/terraform.tfvars # Edit infra/terraform.tfvars: prefix, location, owner, cost_center, apim_publisher_*, budget_* Create the Gateway Core On the first apply, leave worker_image and admin_ui_image empty (default ""). The images don't exist yet, and the worker Job / Admin UI app are count-gated on these variables. cd infra terraform init # If you are moving an existing state from another backend, run `terraform init -migrate-state` instead. terraform apply Build and push the container images with app registrations After the registry is created, build the worker and Admin UI images remotely (no local Docker needed). acr=$(terraform output -raw registry_login_server) reg=$(terraform output -raw registry_name) az acr build --registry $reg --image config-sync-worker:latest ../app/config-sync-worker az acr build --registry $reg --image admin-ui:latest ../app/admin-ui The worker and Admin UI requires entra app registrations for a user to access the frontend. Create the admin security group, BFF API App registrations and SPA public-client app registrations. ./scripts/app-registration.sh Enable the worker and Admin UI From the output above, populate the image references and the three Entra variables from the prerequisites into infra/terraform.tfvars and apply again. worker_image = "<registry_login_server>/config-sync-worker:latest" admin_ui_image = "<registry_login_server>/admin-ui:latest" admin_ui_public = true # external FQDN (still Entra-gated). false = VNet-only admin_group_object_id = "<entra security group object id>" bff_api_audience = "api://<bff app id>" spa_client_id = "<spa app id>" entra_tenant_id = "<tenant id>" CosmosDB Seed configuration Cosmos is private with key auth disabled, so the initial config is seeded from a jumpbox inside the VNet. Default confguration of enable_jumpbox = true in infra/terraform.tfvars triggers Terraform to: provision the jumpbox VM, grant it’s managed identity the Cosmos DB Built-in Data Contributor role (scoped to the config container), and runs a VM run-command that seeds both documents automatically: Global config (id=global) — allowed models + token limits. Per-model pricing (id=pricing) — prompt/completion rates for cost-based budgeting. To seed manually instead (jumpbox connected via Bastion), the same scripts can be run directly: # Global allowed models + limits ./scripts/seed-cosmos-jumpbox.sh https://<cosmos-account>.documents.azure.com:443/ # Per-model pricing (for cost-based budgeting) ./scripts/seed-pricing-jumpbox.sh https://<cosmos-account>.documents.azure.com:443/ Access the AdminUI Update the SPA with your containerapps url spa_app_id="$(az ad app list --display-name "AI Gateway SPA" --query "[].appId" -o tsv)" # spa_client_id fqdn=$(terraform output -raw admin_ui_fqdn) # run from infra/ oid=$(az ad app show --id "$spa_app_id" --query id -o tsv) az rest --method PATCH \ --uri "https://graph.microsoft.com/v1.0/applications/$oid" \ --headers 'Content-Type=application/json' \ --body "{\"spa\":{\"redirectUris\":[\"https://$fqdn\"]}}" Browse to the admin_ui_fqdn, which is also the container apps fqdn. You will need to login via EntraID (Users will need to be added to the Entra group for them to login). Go ahead and register the consumer with a name and issue the API key. The API key is the APIM subscription key and will only be shown once on the UI, so copy and paste it somewhere safe. 2: AI Gateway Consumers and Keys Next, on the left hand tab, click on budgets. This will set the daily budget limit a user is allowed to consume in a day and is also where the model downgrade logic resides. For the purpose of demonstration, set a low budget of $1.8 and select the model priority that you want the downgrade to occur. In this case, gpt-5.4 will be used first, followed by gpt-5.4-mini, DeepSeek then Grok. 3: AI Gateway Budgets Lastly, on the land hand tab, select Rate limits. This sets the amount of tokens a user can consume in a day. It is a daily limit and resets after 24 hours. Select the large tier. 4: AI Gateway Rate Limits Browse to Dashboard, it shows you all the token information, request status codes and group them by consumer and model. You can also view the budget downgrade for a specific user. 5: AI Gateway Captions Integrate endpoint with github copilot chat in vscode In VScode, type “Ctrl + Shift + p” and select “Chat: Manage Language Model”. Select Add Models and choose Azure. 6: Add models toGithubCopilot Chat Follow through the prompts. It will create or edit a chatLanguageModels.json file. Your file should look like this. Take note that you will need to use the /vscode path. [ { "name": "Azure", "vendor": "azure", "models": [ { "id": "gpt-5.4", "name": "gpt-5.4 (APIM)", "url": "https://<REPLACE WITH YOUR APIM ENDPOINT>.azure-api.net/vscode/openai/deployments/gpt-5.4/chat/completions?api-version=2025-01-01-preview", "toolCalling": true, "vision": true, "maxInputTokens": 128000, "maxOutputTokens": 16000, "requestHeaders": { "Ocp-Apim-Subscription-Key": "<REPLACE WITH YOUR SUBSCRIPTION KEY" } } ] } ] Now select the gpt-5.4 (APIM) model and ask it a question. Integrate endpoint with copilot cli As copilot only accepts api-key headers, a separate api is used. Replace and export the following variables before using copilot cli. export COPILOT_PROVIDER_TYPE="azure" export COPILOT_PROVIDER_BASE_URL="<REPLACE WITH YOUR APIM ENDPOINT>" export COPILOT_PROVIDER_API_KEY="<REPLACE WITH YOUR SUBSCRIPTION KEY>" export COPILOT_MODEL="gpt-5.4" export COPILOT_PROVIDER_AZURE_API_VERSION="2025-01-01-preview" export COPILOT_PROVIDER_MODEL_ID="gpt-5.4" You should see a similar response. 7: Integration of APIM to copilot cli Simulate downgrade feature Continue to ask more questions to consume more tokens. Once it hits the 80% cost threshold, you should see that the tag has been switched to “Auto-switch level 1”, meaning it will downgrade to gpt-5.4-mini for future requests. 8: AI Gateway Downgrade Feature Validate by running this command in your terminal with your own endpoints and api-key. curl -sS -i -X POST "https://<REPLACE>.azure-api.net/openai/deployments/gpt-5.4/chat/completions?api-version=2025-01-01-preview" -H "api-key: <REPLACE WITH API KEY>" -H "Content-Type: application/json" -d '{"messages":[{"role":"user","content":"hi"}],"max_completion_tokens":8}' Inspect the headers, you should see that the downgrade level is 1 and the effective model is gpt-5.4-mini despite hitting the same endpoint of gpt-5.4. 9: Model downgrade Conclusion This post started with the problem of multi-model chaos: teams moving quickly with different models, endpoints, SDKs, keys, quotas, and cost profiles, but without a common control plane resulting in ineffective cost control and potential security leaks with model API keys. The governed AI Gateway addresses that by putting Azure OpenAI and Azure AI Foundry behind a single APIM-based entry point, where access, limits, routing, identity, telemetry, and budget behavior can be applied consistently for every consumer. We also walked through how the gateway is different from APIM’s native AI gateway capabilities. APIM provides the enforcement runtime and the GenAI policy primitives, such as token limits, token metrics, semantic caching, and backend routing. The governed AI Gateway builds the operating model around those primitives: self-service onboarding, per-consumer model entitlements, live configuration without redeployment, managed-identity-only private backends, per-call cost telemetry, and cost-based downgrade across model providers. From there, we integrated the APIM endpoint with Github Copilot Chat and Copilot CLI, and validated the downgrade behavior when spend thresholds were reached. The result is not just an AI proxy, but a reusable enterprise pattern for running AI access as a governed platform: developers keep a simple model endpoint experience, while the platform team keeps control over security, cost, observability, and operational change. Overall, this post helps organizations bring multi-model AI usage under one governed entry point, reducing sprawl across endpoints, keys, policies, and cost controls. It also gives platform teams centralized control over model access, rate limits, budgets, telemetry, and private backend access while preserving a simple endpoint experience for developers. References AI gateway capabilities in Azure API Management Policies in Azure API Management Azure API Management policy reference - llm-emit-token-metric Using GitHub Copilot CLI - GitHub Docs AI language models in VS CodeMicrosoft Leads a New Era of Software Supply Chain Transparency
Microsoft announces the general availability of Microsoft’s Signing Transparency (MST) – a first-of-its-kind capability that brings unprecedented visibility and trust to our software supply chain. With this release, Microsoft is leading the industry by recording the build of critical cloud services into a publicly readable and verifiable SCITT standard (Supply Chain Integrity, Transparency, and Trust) compliant ledger. This means every production software build for in scope services like Azure Attestation and Azure Managed HSM (Hardware Security Module), Azure confidential ledger, Microsoft Signing Transparency itself (and others over time) – is now logged in an immutable, tamper-evident record. Only builds that are in the MST ledger are deployed to production; this gives customers confidence that the supply chain for these critical services can be audited at anytime. Notably, the MST ledger is fully open source and built to align with the emerging IETF SCITT standard. By embracing SCITT’s principles and open protocols, Microsoft ensures that MST not only secures our own ecosystem but also contributes to a broader industry movement toward standardized supply chain transparency. The open-source MST ledger serves as a verifiable trust anchor that any organization or researcher can inspect, audit, or even integrate with their own tooling. MST itself meets the highest levels of transparency, backed by a tamper-proof confidential ledger, open-source, and independently verified. Specifically, we are making the foundation of our trust model transparent and accessible to everyone – reinforcing that trust must be earned through proof, not just promises. This launch marks a major milestone in our commitment to Zero Trust principles, extending “never trust, always verify” all the way into the build itself. Building on a public preview introduced late last year, MST’s general availability delivers verifiable transparency at the software level. It transforms traditional code signing with an additive trust layer that is accessible via an open verification model. Every new software update is accompanied by a publicly auditable proof of integrity, enabling security teams to proactively confirm that each update is authentic and unaltered. To help organizations get the most out of this capability, we are also introducing a free tool to explore the contents – Ledger Explorer – an offline tool that allows security teams to examine MST ledger entries, verify cryptographic proofs, and even validate the ledger’s integrity independently. This tool, combined with MST’s open design, ensures that every Microsoft customer – and the broader community – can hold us accountable in real time for the software we run on their behalf. Key Benefits of Microsoft’s Signing Transparency (MST) Verified Code Integrity – Every software release is cryptographically logged in MST’s ledgers. This makes each build tamper-evident and traceable. If an attacker attempts to inject malicious code or sign an unauthorized update, it will be evident through the well-defined validation step built into the SCITT standard. Organizations gain the assurance that code integrity can be independently confirmed at any time. Independent Verification & Zero Trust – MST enables customers and auditors to verify software authenticity on their own, without having to solely rely on vendor attestations. For each update, Microsoft provides a transparency “receipt” (proof of logging) that you can use to prove the update was officially published and unaltered. This fosters a “don’t just trust, verify” approach, empowering security teams to double-check everything running in their environment aligns with what Microsoft intended. Audit-Trail & Compliance – The transparency ledger creates a permanent, auditable timeline of code deployments. Every entry is a record of what was released and when, backed by cryptographic proofs. This simplifies compliance reporting and accelerates forensic analysis. In the event of an incident, you can quickly audit the ledger to see if any unexpected code was introduced. For highly regulated industries, MST offers concrete evidence of software integrity and policy compliance over time. Leadership & Open Standards – We are delivering real transparency now, encouraging a future where all critical software is released with verifiable integrity. MST’s open source implementation and SCITT-compliant design exemplify our commitment to openness and collaboration. We believe widespread adoption of these standards will strengthen supply chain security for everyone, making trust verification a universal practice. Next Steps Microsoft’s Signing Transparency is more than a new security feature and shapes the advances in trust technology. As threats grow more sophisticated, we must evolve the way we assure our customers about the software they depend on. With MST now generally available, we are leading by example: proving that it is possible to open up the traditionally opaque process of software deployment and turn it into a source of strength and trust, i.e. empowering each person with verifiable transparency. We invite the industry to join us on this journey and get started by reading the documentation and exploring Ledger Explorer today! Together, by embracing transparency and open standards, we can turn “trust but verify” from a slogan into an everyday reality for digital infrastructure.We Gave Ourselves 20 Minutes to Build an AI Agent for a Lumber Company. The Timer's Still on Screen.
Here's a confession: most "build with AI" webinars are 60 minutes of slides, 5 minutes of a polished demo someone rehearsed for a week, and a closing CTA. You leave inspired but not really sure what you saw. So we tried something different. We put a visible countdown timer on the screen and gave ourselves 20 minutes to do two things, live: Build an AI agent that solves a real business problem Deploy a working AI application to Azure No edits to hide the awkward parts. No "and here's one I prepared earlier." Just the timer, the screen, and a working app at the end. The on-demand recording is up now. Here's what's in it and why you should carve out 20 minutes for it this week. The setup: why lumber? 🏘️ We needed a real business problem, not a toy one. So for the demo, we role-play as the owner of Contoso Lumber — a regional lumber business with a very specific, very real headache: Should we sell our inventory now, or hold it longer? Sell too early, miss a better price. Hold too long, eat storage costs. Lumber prices fluctuate with global competition, macro shifts, even the weather. In the past, decisions like this came from morning meetings and gut instinct, or maybe the occasional ad-hoc spreadsheet that nobody could reuse a month later. It's the kind of decision that should have an analyst behind it — except most growing businesses can't afford to hire one full-time. So we build the AI agent that does. (Yes, lumber. We know. Stick with us — the boring industry is exactly the point. If it works here, it works for your business too.) What we actually build (in 20 minutes flat) The webinar walks through the entire flow, end to end: Part 1 — The agent. We open Microsoft Foundry at ai.azure.com, browse the model leaderboard (there are over 11,000 models to choose from — we compare a few on the cost-vs-quality chart), pick one, write a plain-English instruction for the agent, upload a CSV of historical lumber pricing, and ask it a real question: "If I cannot sell one of my products today unless I offer my clients a 35% discount, and knowing the historical pricing data, should I still sell it?" The agent runs a break-even analysis and comes back with a reasoned recommendation — hold for 3–6 months, here's the math on why, here's where storage costs start eating the upside. Then we add voice mode (now you can ask the agent for pricing recs from a coffee shop on your phone), and lock down guardrails to block jailbreaks, prompt injection, data leakage, and — because we're feeling fancy — profanity in responses. Part 2 — The app. With the agent done, we pivot to deploying a full AI chat application to Azure. From scratch. Using exactly five commands in Azure Cloud Shell: azd auth login git clone <repo> cd <folder> azd up azd down # (this one's for when you're done — kills everything to avoid surprise bills) That's it. The template handles the Container Apps setup, the architecture-aligned-to-Well-Architected-Framework stuff, all the boilerplate that usually eats half a sprint. By the end of the segment, there's a working AI chatbot running on a real Azure URL. We even pause the timer when we're just explaining things, so you know the 20-minute clock is honest about build time, not talk time. Why this format is more useful than another slide deck A few things this webinar shows that a written tutorial can't: The Foundry UI is super navigable. You watch someone do it. You see where the buttons are. You see what the leaderboard looks like when you're comparing GPT-5.3 Codex against Kimi K2.5 on a cost-to-quality chart. (Spoiler: Kimi wins this particular trio. Your mileage will vary depending on your workload.) The "no-stitching" claim is real. Models, data, agents, guardrails, deployment — all in one place. You don't need to leave Foundry to wire seven products together. The webinar makes that concrete by showing you the actual flow without cutting. Five commands really is five commands. This is the part people are most skeptical about until they see it. azd up does the work. The infrastructure provisioning, the container app, the AI service hookup — all of it. You can delete it just as fast. azd down tears everything back down. Useful when you're experimenting and don't want a $40 surprise on your Azure bill next month. What's on screen at the end By the 20-minute mark: A published AI agent named for the lumber business, with guardrails, voice mode enabled, ready to be called from Teams, Microsoft 365 Copilot, or any application via endpoint A separate AI chat application deployed to Azure Container Apps, with a live URL Logs, observability, the full Foundry control plane — all available out of the box And in the closing minutes, four very concrete next steps for what you do next if this sparked an idea for your own business — including Azure Accelerate (if you want Microsoft experts in the room with you), the partner network, and the Microsoft marketplace if you'd rather buy than build. Watch the recording The on-demand recording is available now. Block 20 minutes — that's literally all it takes — and ideally watch with your Azure portal open in another tab so you can follow along. If you're the kind of person who learns by doing, pause at the agent-building section and try it yourself in parallel. Foundry is free to explore; the agent we build in the webinar costs cents to run. → Watch the on-demand webinar A few things we'd love feedback on If you watch it, we'd genuinely love to know: Did the timer help or distract? (We thought it would feel gimmicky. It turned out to be the most-mentioned thing in early feedback.) What use case from your business would you want to see in the next one? We're picking the next demo problem from comments. Was the lumber thing weirdly compelling or were you just here for the Azure parts? Drop a comment, tag us, or grab a partner and try building your own version this week. The timer's reset. Your 20 minutes start whenever you press play. Want to go deeper than the webinar? Two companion reads: From Idea to Impact: How Growing Businesses Scale with Azure (five real customer stories with the full architectures) and AI Made Simple: 3 Practical Moves for Growing Businesses (the structured playbook for figuring out what to build first).Building an On-Device Voice Assistant with Microsoft Foundry Local
Why on-device voice still matters Most "voice AI" tutorials assume your audio leaves the machine. You ship a WAV to Whisper-API, your transcript to GPT-4, and a synthesized response back over the wire. That works — but it also means three round trips, three per-token bills, and three places your user's voice gets logged. The new wave of small, hardware-optimised models changes the trade-off. NVIDIA's Nemotron Speech Streaming En 0.6B is a 600M-parameter streaming ASR model published into the Microsoft Foundry Local catalog. Paired with a small chat model like qwen2.5-0.5b or phi-4-mini , you can run the entire capture → transcribe → reason → respond loop in-process on a developer laptop, with no API keys and no network egress. This post walks through how the fl-nemotron sample does it, the SDK pitfalls we hit on the way, and the design decisions that made the pipeline reliable. What we're building A browser-hosted assistant served by FastAPI at http://127.0.0.1:8000 . The page captures microphone audio, posts it to /api/transcribe , then streams the chat reply back over Server-Sent Events from /api/chat . All inference runs locally through two Foundry Local models loaded into the same process. The shape of the pipeline: Microphone (browser MediaRecorder) │ WebM/Opus blob ▼ Client-side WAV encoder (16 kHz, mono, PCM-16) │ multipart/form-data ▼ FastAPI /api/transcribe │ ▼ Nemotron Speech Streaming En 0.6B (Foundry Local audio client) │ transcript text ▼ Chat LLM e.g. qwen2.5-0.5b (Foundry Local chat client) │ streamed tokens ▼ FastAPI /api/chat → SSE → browser bubble The version that bit us: foundry-local-sdk >= 1.1.0 Before any code, the single most important fact about this project: The Nemotron Speech Streaming model only appears in the Foundry Local 1.1.x catalog. Older SDKs (0.5.x / 0.6.x) cannot resolve the alias nemotron-speech-streaming-en-0.6b and fail with model not found . The module name also changed in 1.1.0 — it is now foundry_local_sdk (with the underscore- sdk suffix), not foundry_local . The pip wheel for foundry-local-core is bundled, so there is no separate MSI / winget install to worry about. Pin it explicitly: pip install --upgrade "foundry-local-sdk>=1.1.0,<2" And verify before anything else: python -c "import importlib.metadata as m; print('sdk', m.version('foundry-local-sdk'))" # expect: sdk 1.1.0 Loading both models from one manager The 1.1.x SDK exposes a single FoundryLocalManager that owns the runtime. Each loaded model gives you back a per-model OpenAI-compatible client — get_chat_client() for text models and get_audio_client() for ASR. There is no need to bring your own openai Python package; the SDK ships its own thin client. The wrapper used in the repo ( src/foundry_client.py ) does this: from foundry_local_sdk import Configuration, FoundryLocalManager FoundryLocalManager.initialize(Configuration(app_name="fl-nemotron")) manager = FoundryLocalManager.instance chat_model = manager.load_model("qwen2.5-0.5b") stt_model = manager.load_model("nemotron-speech-streaming-en-0.6b") chat_client = chat_model.get_chat_client() audio_client = stt_model.get_audio_client() Both models are downloaded on first use into the Foundry Local cache and stay resident for the lifetime of the process. On a laptop with 16 GB RAM, the combined working set sits comfortably under 4 GB. The transcription surprise The first naive approach was the obvious one: with open(wav_path, "rb") as f: result = audio_client.transcribe(file=f, model="nemotron-speech-streaming-en-0.6b") That call fails on Nemotron. The bundled ONNX Runtime GenAI in foundry-local-core does not register the nemotron_speech multi-modal model type that the standard AudioClient.transcribe() path tries to instantiate. The error surfaces as a cryptic model-type registration failure deep inside the native runtime. The fix is to use the streaming session API instead — a different native entry point ( core_interop.start_audio_stream ) that the streaming model does support. The repo isolates this in src/_nemotron_live.py : def transcribe_wav_live(audio_client, wav_path, *, language="en"): with wave.open(str(wav_path), "rb") as w: sample_rate = w.getframerate() channels = w.getnchannels() sample_width = w.getsampwidth() pcm = w.readframes(w.getnframes()) session = audio_client.create_live_transcription_session() session.settings.sample_rate = sample_rate session.settings.channels = channels session.settings.bits_per_sample = sample_width * 8 session.settings.language = language session.start() # Feed PCM in ~100 ms chunks from a worker thread, then stop. bytes_per_sec = sample_rate * channels * sample_width chunk_bytes = max(bytes_per_sec // 10, 1024) def _pusher(): try: for offset in range(0, len(pcm), chunk_bytes): session.append(pcm[offset:offset + chunk_bytes]) finally: session.stop() threading.Thread(target=_pusher, daemon=True).start() parts = [] for resp in session.get_stream(): for cp in getattr(resp, "content", []) or []: text = getattr(cp, "text", "") or getattr(cp, "transcript", "") or "" if text: parts.append(text) return " ".join(p.strip() for p in parts if p.strip()).strip() Two things to notice: Push from a thread, read from the main coroutine. session.append() is a blocking write into the native stream and session.get_stream() is a blocking generator. Run one in a worker thread so the other can drain in parallel — otherwise you deadlock the session. Chunk to ~100 ms. Smaller chunks (e.g. 10 ms) spend more time crossing the FFI boundary than transcribing; larger chunks (e.g. 1 s) hold back partial results and hurt perceived latency. Always session.stop() . Without it the generator never terminates and the request hangs. The other transcription surprise: browsers don't send WAV Inside the browser, MediaRecorder defaults to audio/webm; codecs=opus . That's great for size but bad for our STT model, which expects a 16-bit mono PCM WAV at a known sample rate. Decoding WebM/Opus server-side would require ffmpeg as a runtime dependency — which is exactly the kind of friction this project exists to remove. The cleaner solution is to encode WAV on the client. AudioContext.decodeAudioData already understands WebM/Opus, so the page can decode the recording, resample to 16 kHz, mix to mono, and emit a PCM-16 WAV blob in 30 lines of JavaScript: // Inside src/static/index.html async function webmToWav(blob) { const ctx = new (window.AudioContext || window.webkitAudioContext)({ sampleRate: 16000 }); const buf = await ctx.decodeAudioData(await blob.arrayBuffer()); // Mix to mono const ch = buf.numberOfChannels; const mono = new Float32Array(buf.length); for (let c = 0; c < ch; c++) { const data = buf.getChannelData(c); for (let i = 0; i < data.length; i++) mono[i] += data[i] / ch; } return encodeWav(mono, 16000); } function encodeWav(samples, sampleRate) { const buffer = new ArrayBuffer(44 + samples.length * 2); const view = new DataView(buffer); // RIFF header writeStr(view, 0, "RIFF"); view.setUint32(4, 36 + samples.length * 2, true); writeStr(view, 8, "WAVE"); // fmt chunk writeStr(view, 12, "fmt "); view.setUint32(16, 16, true); // PCM chunk size view.setUint16(20, 1, true); // PCM format view.setUint16(22, 1, true); // mono view.setUint32(24, sampleRate, true); view.setUint32(28, sampleRate * 2, true); // byte rate view.setUint16(32, 2, true); // block align view.setUint16(34, 16, true); // bits per sample // data chunk writeStr(view, 36, "data"); view.setUint32(40, samples.length * 2, true); // PCM-16 samples let o = 44; for (let i = 0; i < samples.length; i++, o += 2) { const s = Math.max(-1, Math.min(1, samples[i])); view.setInt16(o, s < 0 ? s * 0x8000 : s * 0x7FFF, true); } return new Blob([view], { type: "audio/wav" }); } Now the server's /api/transcribe endpoint just writes the bytes to a temp file and hands them to transcribe_wav_live() — no audio decoding libraries on the Python side. Wiring it into FastAPI The server ( src/app.py ) is deliberately small. The notable detail is that the same process holds both Foundry Local model handles for its entire lifetime, so there is no warm-up cost per request: @app.post("/api/transcribe") async def transcribe(audio: UploadFile = File(...)): data = await audio.read() with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f: f.write(data); path = f.name text = _ai_client.transcribe(path) return {"text": text} @app.post("/api/chat") async def chat(req: ChatRequest): if req.stream: return StreamingResponse( _sse(_ai_client.stream_completion(req.messages)), media_type="text/event-stream", ) return {"text": _ai_client.chat_completion(req.messages)} Streaming uses Server-Sent Events because they are trivially supported in both fetch() and the FastAPI runtime, and they don't require a WebSocket upgrade through any proxy a developer might have in front of localhost . What it looks like The repo includes screenshots of the running UI: a welcome screen with both models loaded, a streamed haiku reply, an inline code block with copy-to-clipboard, and the recording state for the microphone. Performance, honestly This is a small-model, CPU-friendly stack. On an Arm64 Surface running the x64 SDK under emulation: First model load (cold cache): tens of seconds — downloads ~600 MB for Nemotron and ~400 MB for qwen2.5-0.5b . Subsequent loads (warm cache): a few seconds per model. End-to-end transcription of a 5-second utterance: well under a second after warm-up. First chat token from qwen2.5-0.5b : typically 200–500 ms; full short reply within 1–2 s. On x64 silicon with a recent CPU the numbers improve substantially, and the SDK will pick the best execution provider it finds (CPU / DirectML / CUDA) for each model. Trade-offs to know about Model quality. qwen2.5-0.5b is a 500M-parameter model. It is fast and small enough to ship on a laptop, but it is not GPT-4. Swap in phi-4-mini or mistral-nemo-12b-instruct if you have the RAM and want better reasoning — the wrapper accepts any chat alias in the Foundry Local catalog. STT is English-only here. The current Nemotron streaming model in the catalog is ...-en-0.6b . Multilingual variants are likely to follow. Browser microphone needs a real browser. Headless / automated browsers (Playwright, Puppeteer) deny getUserMedia by default. Open the page in Edge / Chrome / Firefox to grant the permission and capture audio for real. No agent framework yet. This sample is deliberately a single-turn loop over a chat client — there is no tool calling, planning, or multi-agent orchestration. Adding the Microsoft Agent Framework on top would be a natural next step for richer behaviour. Responsible AI considerations Running locally removes the cloud-egress class of privacy concerns, but it does not remove responsibility: Disclose recording. The browser prompts for mic permission; your UI should make it obvious when capture is active. The sample shows a red ⏹ button and a "Recording…" banner for that reason. Don't log raw audio. The sample writes audio to a per-request NamedTemporaryFile and deletes it after transcription. Treat the WAV as sensitive data even when it never leaves the device. Small models hallucinate. A 0.5B chat model is great for snappy local replies, but unsuitable for high-stakes answers. Pair it with retrieval, ground it on your own data, or escalate to a larger model when accuracy matters. Try it Clone github.com/leestott/fl-nemotron. ./setup.ps1 (or ./setup.sh ) to create a virtualenv and install the pinned SDK. python scripts/prefetch.py nemotron-speech-streaming-en-0.6b qwen2.5-0.5b to download both models. .venv\Scripts\uvicorn.exe app:app --app-dir src --port 8000 Open http://127.0.0.1:8000 in a real browser and click the 🎤 button. Where to go next Foundry Local documentation — official docs for the runtime, catalog, and SDK. microsoft/Foundry-Local — upstream samples and issue tracker. NVIDIA Nemotron model family — background on the speech and language models being published into the catalog. leestott/fl-nemotron — the full source for this post. Key takeaways Pin foundry-local-sdk >= 1.1.0 . Earlier SDKs cannot see the Nemotron Speech Streaming model. Use the LiveAudioTranscriptionSession API for Nemotron, not AudioClient.transcribe() . Encode WAV in the browser. It eliminates a heavy server-side ffmpeg dependency for a few lines of JS. Push audio chunks on a worker thread and drain the response generator on the main one to avoid deadlocks. A small Foundry Local chat model plus Nemotron STT gives you a credible local voice loop in a single Python process — no cloud, no keys, no data egress.
