Engineering an Agentic Scheduler
Authors: Gabriel Sze and Weixian Zhang 1. Engineering an Agentic Scheduler Scheduling looks deceptively simple until multiple constraints, shared resources, and partial commitments start interacting. This post walks through how we built an agentic scheduler that lets an LLM reason about placements while keeping the actual scheduling process deterministic and inspectable. Imagine scheduling a training program for a sports team. It needs four hours in total, split into two-hour sessions, with at least two days between them. Other activities already occupy parts of the week. Where should those sessions go? At first glance, this looks like a simple calendar problem. Find four free hours and place the session. So the engineering question is not simply: How do we ask an agent to generate a schedule? It is: How do we give an agent enough freedom to reason about scheduling, while keeping the resulting schedule deterministic, inspectable, and correct? Our application does not ask a model to produce a finished spreadsheet in one call. Instead, we decompose this into small steps. The agent gets a much smaller responsibility: choose candidate dates, inspect available slots through tools, and propose placements. 1.1 Defining the scheduling problem Imagine planning mixed martial arts training for Team Red, Team Blue, and Team Yellow in a shared training facility. We call each martial arts activity a program, and each continuous training block a scheduled session. For example, a Muay Thai program for Team Red might run over three mornings in the same week, with each morning counted as a separate scheduled session. A program typically requires a certain number of hours, divided into sessions of an allowed duration. Team Red might train in striking and pad work, Team Blue in wrestling and submissions, and Team Yellow in conditioning and mobility. Some programs may need scarce evening slots. Others compete for the same mat, ring, or instructor. Programs may also require rest days between sessions. We will follow one fictional example throughout this post. Muay Thai Sparring requires four daytime hours for Team Red, split into two two-hour sessions at least two days apart. Team Blue and Team Yellow already have wrestling and conditioning sessions elsewhere in the shared calendar. Finding four empty hours is therefore not enough: the scheduler must place two valid sparring sessions without creating team, facility, or recovery conflicts. Before scheduling starts, the application converts source data and user-provided requirements into explicit, typed scheduling constraints. An optional review step allows a person to provide additional guidance. The important boundary is that the scheduler operates on structured input rather than interpreting raw source material directly. 1.2 Why involve an agent at all? Not every scheduling preference is naturally expressed as a hard constraint. Historical patterns may suggest preferred timings, while users may provide intent such as to keep Friday relatively light, or try to keep related activities in the same half of the week. These preferences require interpretation rather than simple constraint checking. We therefore use the model where reasoning is useful: interpreting intent, exploring candidate placements, and deciding what to try next. We keep explicit rules: session duration, required hours, spacing, conflicts, and other conditions that can be evaluated deterministically. It gives us a concrete contract: the agent proposes allocations against explicit requirements; the application owns the resulting state and the checks used to assess it. 2. Designing the Software Architecture Code snippets are simplified from the source to show the shape of each component, not exact signatures. 2.1 Components Frontend: a React 19 single-page app (Vite, MUI), authenticated with Entra ID. Talks to the backend through two channels only: REST calls to start or steer a run (program scheduling), and one long-lived SSE stream for everything coming back. Holds no run state of its own. Run status, progress, the live calendar and the finished workbook all arrive as replies on the stream, so a reload rebuilds the view from what the backend sends. Everything from the Gateway onwards is the backend: one Node.js service on Azure Container Apps that the frontend reaches only through REST and SSE. Gateway: a Fastify API, the backend's only entry point. Ingress is REST: POST /api/schedule starts a run; POST /api/steer resumes a paused one with the user's instructions. Egress is SSE: GET /api/events holds one stream per browser tab. Never runs agent code. It authenticates the request, writes the upload to the session's folder, enqueues a command that references that folder and returns at once: Mailroom: the asynchronous boundary between the gateway and the agents, made of two Postgres-backed queues, each drained by its own polling loop. Agents: two LangGraph graphs for Program IQ Agent and Scheduling Agent Session: one scheduling run for one user. Each last cycle training program .xlsx upload starts a new session with its own ID, so a user running three schedules at once has three sessions. Each session has its own dedicated workspace folder structure: /sessions/{username}/{sessionId}/ input/ ← the uploaded last cycle training program .xlsx work/ ← agent scratchpad: extracted-demand-profile.json, raw rows, traces output/ → the scheduled training programs .xlsx A session can hold several agent commands, such as a run plus the user's later answer through /api/steer. Every reply carries a sessionId and a commandId, so the frontend can tell which run and which command it belongs to. Storage: Postgres: the Mailroom's two queue tables, plus LangGraph checkpoints, so a run's graph state survives a restart. Using one database for queues and state removes the need for a separate message broker. Azure Files: On Azure, /sessions lives on an Azure Files share mounted into the container at /mnt /.agent, so session files are present in multiple replicas of Azure Container Apps and can survive a restart . Azure AI Foundry: hosts the model the agents call. Agent Tracer: records every model call and tool call as OpenTelemetry traces in Azure Application Insights, plus a per-session trace file in work/. Identity: Entra ID covers user sign-in and the backend's access to Postgres and Foundry, through managed identity. 2.2 Key Design Considerations Why REST in and SSE out, instead of WebSockets? Our communication pattern is asymmetric: clients submit scheduling requests, while agents produce asynchronous updates over several minutes. We chose REST for incoming requests because it handles multipart Excel uploads, authentication, and request validation through standard HTTP mechanisms. For outgoing updates, Server-Sent Events (SSE) was a better fit than WebSockets. The browser only needs to receive progress and agent responses, not maintain a bidirectional conversation. SSE also avoids the overhead of repeatedly polling for updates that arrive at unpredictable intervals. To handle disconnections, agent replies are persisted in reply_message, allowing the client to retrieve missed updates when it reconnects. The implementation is described further in this post. Why a command queue instead of executing agents directly within HTTP requests? Scheduling runs can take several minutes, making it impractical to execute agents directly within an HTTP request. Instead, we introduced a Postgres-backed command queue to separate request handling from agent execution. When a request arrives, the gateway persists the command and immediately returns 202 Accepted, allowing the agents to continue working independently of the HTTP connection. This separation also improves reliability and gives us greater control over execution. Commands are persisted before processing begins, so they are not silently lost if a worker crashes or restarts. Interrupted runs can be identified, marked as failed, and surfaced to the user. Command loop (inbound): claims the next command from agent_command, runs the agents on it, and settles it as completed, paused or failed. Claims use row locking, so several replicas can drain one queue without running a command twice. // domain/mailroom/command-loop.ts while (!stop.stopping) { if ((await pickAndExecuteCommand(helper)) !== 'idle') { idleWait = IDLE_WAIT_MS; // work arrived: drain greedily continue; } await stop.sleep(idleWait); idleWait = nextIdleWait(idleWait); // empty queue: back off } Reply loop (outbound): delivers rows from reply_message to the SSE streams that its own replica holds. Each replica serves only the users connected to it, so there's no broker and no sticky sessions. Undelivered replies stay queued until the user reconnects. const users = sse.streamingUsernames(); // who is connected to THIS replica for (const reply of await queue.getReplies(users)) { if (sse.send(reply.username, reply) > 0) await queue.clearDelivered(reply); } The loops never call each other. Commands flow in, replies flow out, and the agents write their own replies. The command loop processes one command at a time, limiting concurrent scheduling runs and preventing too many requests from hitting the model at once. Additional commands simply wait in the queue until capacity becomes available. By keeping the execution loop separate from the agent graph, we can replace the real agents with a mock implementation and test the entire flow, from the gateway through the command queue to SSE responses, without making any model calls. // domain/mailroom/agent-run.ts — the one seam: swap the import (and its call) import { runAgentStream } from '../agent/agent-graph.ts'; // import { runMockAgentStream } from '../agent/mock-agent-graph.ts'; // domain/agent/mock-agent-graph.ts — same state, same checkpointer, one node that only talks const graph = new StateGraph(AgentState) .addNode(MOCK_REPLIER, createMockReplier()) .addEdge(START, MOCK_REPLIER) .addEdge(MOCK_REPLIER, END) .compile({ checkpointer: await checkpointer() }); Why Postgres instead of Azure Service Bus or another broker? Rather than introducing a dedicated message broker, we chose to build our command queue on Postgres, which was already serving as the application's primary database. Beyond storing application data, Postgres also manages the command queue, agent replies, execution history, and LangGraph checkpoints. This allows us to consolidate several responsibilities into a single service, reducing the operational overhead of managing, securing, and monitoring additional Azure resources. We also kept the queue implementation behind a light interface (pickNext, complete, pause, fail), allowing us to swap Postgres to a dedicated broker such as Azure Service Bus if throughput requirements grow. Ultimately, we chose to optimize for simplicity at our current scale, while keeping the architecture flexible enough to evolve. 3. From historical schedules to structured demand 3.1 Uploading past programs The workflow begins when the user uploads reference programme workbooks from previous cycles, along with any relevant contextual information that can help ground the scheduler when planning future programmes. Each uploaded .xlsx file is saved to a new session folder, after which the gateway places a lightweight command on the queue containing the user, session Id, and plan start date. The files themselves are not included in the command, keeping the queue payload small. This allows the HTTP request to return immediately while the ingestion process runs asynchronously, without tying up the web server. An optional review: true flag allows users to pause the workflow after ingestion to review the extracted information before proceeding with scheduling. We'll cover this further in the Human-in-the-loop section. // api/rest-api.ts for (const upload of uploads) await writeFile(join(input, upload.filename), upload.content); const command: AgentCommand = { commandId: nextUlid(), username, sessionId, type: 'file-upload', planStart, files, review, }; 3.2 Program IQ: extracting the demand profile and insights Program IQ serves as an intelligence layer that uses historical programmes to understand the scheduling requirements for the upcoming cycle. It starts by analysing the Excel workbooks (or whatever additional context) uploaded by the user, which contain the programmes conducted in previous cycles. Rather than relying entirely on an LLM, we separate interpretation from computation. The LLM interprets the varying structures and layouts of the workbooks, while deterministic code processes the extracted information to build a demand profile for each programme and team. This captures details such as the number of sessions required, their duration, recurrence patterns, preferred time slots, and spacing between sessions. // domain/agent/program-iq/program-iq-agent-graph.ts new StateGraph(IQState) .addNode(NODES.phaseFilesPairing, createPhaseFilesPairing()) // filename → phase .addNode(NODES.extractRawPrograms, createExtractRawProgramsAgent()) // LLM: grid → rows .addNode(NODES.normalizeRawPrograms, extractProgramIQ()) // code: rows → demand .addEdge(START, NODES.phaseFilesPairing) .addEdge(NODES.phaseFilesPairing, NODES.extractRawPrograms) .addEdge(NODES.extractRawPrograms, NODES.normalizeRawPrograms) One pi-agent-core agent per sheet, up to 6 at once. Each agent gets a system prompt (how past programs are laid out, the day's periods, what to ignore), one sheet name and two tools. It loops until every activity on the sheet is reported. Two tools The Program IQ agent has two tools: one to read the spreadsheet and another to report the extracted information. Rather than following a hardcoded sequence, we chose to introduce an LLM that sits between these tools, deciding when to read the workbook and how to interpret and report its contents. read_sheet tool uses exceljs to turn a sheet into text – understands layout, merged cells, colors, column positions, and flattens that layout into plain lines the model can read. report_activities tool records what the model found. The model reports each activity as one line: date, team, program, location, colour, periods, kind. 2024-06-17 Team Red Striking · FF92D050 1,2 training From activities to demand profiles (code) Plain code counts the reported activities into a demand profile per program and team. No LLM is involved, so every number is traceable back to the source files but meant to softly guide future planning. Demand profile What it holds Volume Total day and night periods Session length Shortest and longest session Rhythm How often it recurs, e.g. weekly on Fridays Slot pattern Which periods of the day it used Spacing Fewest days between two sessions The demand profile is the signature. Teams with identical demand profiles are grouped together to avoid duplicating the same scheduling requirements. For example, if Team Red and Team Blue require the same number of sessions, durations, recurrence patterns, and spacing, they can share a single demand profile. This grouping is purely for representation – each team still needs its own sessions scheduled independently. The output: One row per program and matching group of teams: Open Mat [Team Red, Team Blue] 4 hours a week · 4-hour sessions · weekly on Fri · ≥ 7 days apart Sparring [Team Red] 6 hours a week · 2–3-hour sessions · Tue and Thu · ≥ 2 days apart demand profile .xlsx as input to Scheduler Agent Nobody types this demand in; it is read back from what the teams previously trained. It is what the Scheduling Agent places onto next cycle's calendar, and shared rows become shared batches. 4. Human-in-the-loop: review, steer and resume Historical activity tells us what happened before, but not necessarily what should happen next. Users may want lighter Fridays, prefer certain activities earlier in the week, or restrict specific programmes to specific days. Before scheduling begins, users can review the extracted requirements and provide additional instructions through a separate steer command. This command incorporates their preferences into the existing requirements and resumes the scheduling workflow. We currently support two types of steering guides: Overall preferences apply to the scheduling run as a whole: Prefer lighter Friday afternoons where possible. Program-specific apply to a particular programme in “User Steering” column These instructions are propagated to the relevant scheduling agents, with programme-specific instructions attached to the corresponding activities across the scheduling tiers. Where user instructions conflict with historical patterns, current user intent takes precedence. However, steering an LLM is not the same as enforcing a constraint. Natural-language instructions can influence how agents propose and evaluate placements, but only constraints explicitly implemented in code can be deterministically enforced. For example, "Prefer lighter Friday afternoons" may influence which slots an agent considers, while "Sessions must be at least two days apart" can be validated after every placement. 4.1 Extracted Program IQ: pause for human steering and approval Rather than keeping an agent running while waiting for user input, the system persists the paused state in the store and sends a steer-request containing the extracted programmes for review. This allows the agent to process other requests in the meantime. The user can then review the extracted demand profile and choose to either approve it as-is or provide additional instructions: Approve as-is: submit a steer command without steering guides. Steer and approve: submit a steer command with steering guides to influence the scheduling process. In both cases, the steer command resumes the workflow and initiates scheduling using the approved requirements. // domain/agent/agent-graph.ts await writeExtractedPrograms(session, { plan, programs }); if (command.review === true) { await enqueueReply(newSteerRequest(command, steerableProgrammes({ plan, programs }))); return 'paused'; } 4.2 Steer and Resume As shared, steering can take two forms: program-specific (guides[]) and an overall preference that applies to the entire scheduling run. Both are optional and serve as guidance for the scheduling agents rather than automatically enforced constraints. The request creates a new steer command associated with the existing session. Instead of reprocessing the original Excel workbooks, the workflow reloads the saved demand-profile.json, attaches any programme-specific instructions to their corresponding records, and proceeds directly to scheduling. if (command.type === 'steer') { const steered = withSteeringGuides(await readExtractedPrograms(session), command.guides); yield* schedulePhases(command, session, steered, command.brief ?? '', onProgress); } 5. Inside the Scheduling Agent and Harness Engineering Before we dive in, let's establish a few key terms and how they relate to one another: Slot (or Period): A single unit of time, typically 1 hour, but configurable to other durations. Session: A scheduled training instance consisting of one or more slots. Program: A type of training, such as Muay Thai Clinching or Pad Work. Each program can have multiple sessions. Grid (or Calendar): The scheduling interface where slots and sessions are organized and displayed. In short, a program contains sessions, each session occupies one or more slots, and everything is represented on a calendar grid. The LLM's reasoning and tool-calling loop is only one part of the scheduling system. Around it, we need to manage what context the agent receives, which tools it can access, how scheduling state is maintained, how proposed changes are applied, and what happens when the schedule remains incomplete. In our implementation, LangGraph orchestrates the overall scheduling workflow, while pi-agent-core manages the LLM's reasoning and tool-calling interactions within individual stages. Other stages, such as deterministic seeding and schedule evaluation, run entirely through code without invoking an LLM. new StateGraph(AgentState) .addEdge(START, 'HolidaySeeder') // code .addEdge('HolidaySeeder', 'RoutineSeeder') // code .addEdge('RoutineSeeder', 'EventScheduler') // LLM .addEdge('EventScheduler', 'ProgramScheduler') .addEdge('ProgramScheduler', 'Evaluator') // code .addConditionalEdges('Evaluator', (s) => failed(s) && s.round <= 3 ? 'ProgramScheduler' : 'FillerScheduler') .addEdge('FillerScheduler', END) .compile({ checkpointer: new PostgresSaver(pool) }); Before each agentic scheduling phase, the harness prepares the context relevant to the current batch and provides tools scoped to that task. The agent can inspect availability and propose placements, but it does not maintain its own separate version of the schedule. Instead, its tools interact with the same calendar state used by the application, ensuring that scheduling decisions are based on a consistent view. The process begins deterministically. Existing routines, breaks, and recurring activities that can be reliably identified are placed onto the calendar before the agent begins scheduling. The agent then works with this partially populated calendar to find suitable slots for the remaining activities. Once placements are made, deterministic evaluators check the updated schedule for completeness and constraint violations, rather than relying on the LLM to assess its own work. 5.1 What the agent works with (context engineering) An important part of building the scheduling agent is deciding what information the LLM needs, what it can retrieve through tools, and what should remain in application state. Rather than asking the model to work out information the application already knows, we provide stable facts such as available dates, their corresponding weekdays, and the ordering of scheduling periods directly in its context. Our approach to context engineering follows a simple principle: provide stable facts upfront, retrieve changing calendar information through tools, and keep the authoritative scheduling state outside the conversation. Calendar grid (agent state) - The calendar is a nested dictionary held in memory, schedule[day][team][period], where each entry holds what is scheduled in that slot. The agent tools and calendar inventory work on this one object directly, so a write is visible to the next read at once, with no database in between. System prompt - The system prompt is a Markdown template, batch-schedule-prompt.md, read from disk once. Before the first turn, code fills its {placeholders} with this run's values: // domain/training/scheduler.ts systemPrompt: fillPromptTemplate(template, { day_keys: phase.dayKeysJson, // the dates the model may use day_of_week: phase.dayReference, // each date whether is it a weekday so model knows the weekdays period_table: PERIOD_TABLE, // the slots in a day // … }), The filled prompt tells the model: Role: which program to place, and the one write tool bound to it. Facts: dates, weekdays and slot layout, worked out in advance. The prompt says "never compute these yourself", because models get calendar arithmetic wrong. Rules: what a valid session looks like, stated so the model can plan without trial and error. Tool protocol: read once, then send every placement in one reply. Strategy: plan → confirm → commit. User prompt holds what changes from run to run: the user’s steering guide and, on a retry, only the sessions that failed. 5.2 What the scheduling agent does An important part of designing the scheduling agent is deciding what it can read and what it is allowed to change. Rather than giving the agent unrestricted access to the calendar, we expose a set of tools with clearly defined boundaries. Read tools provide the information needed to make scheduling decisions, while write tools constrain how placements can be made. A query specifies candidate days, teams, and an optional slot window. The response distinguishes between: empty slots commitments that may be overridden hard blocks that cannot be changed To minimise unnecessary context, the response is compressed. For example, if an entire day is unavailable, the tool returns a short marker rather than listing every blocked period individually. The prompt encourages a simple search pattern: choose candidate dates → query free slots in candidate dates → propose a placement → inspect alternatives if necessary Agent loop The agent runs a plain loop: model turn → tool call[]? → run each tool → append result → model turn. Each result is the model's only view of the calendar. The loop ends when there is no tool call. 1. Plan - The model chooses candidate days from the facts in its prompt, without calling a tool. The planning is cheap reasoning, so the tools only confirm and commit. 2. Read with getFreePeriod The model pass in the dates as tool parameter for every team it needs, in one call: { "days": ["12/14/26", "12/16/26"], "teams": ["Red"] } It gets back one answer per day and team: { "day": "12/14/26", "team": "Red", "free": ["1", "2"], "override": ["3"], "blocked": ["night1", "night2"] } free is empty, override holds an overridable routine activity (e.g resting) the program may replace, and blocked is off limits. The model picks from these lists rather than reading the calendar, so the context window stays small. 3. Write with placeProgram - The model sends all the program's sessions in one call. The tool runs two checks: 3.1 Work out how many slots the program is left to place - Each programme owes a fixed number of slots per company. The tool subtracts the slots the calendar already holds, and refuses any session that would go over what's left. // domain/agent/tools/place-program.ts const inventory = takeCalendarInventory(plan); for (const team of program.teams) { const placed = getPlacedPeriods(inventory, program, team); remaining.set(team, program.demand - placed.periods); // what it may still place placedDays.set(team, placed.days); // what the spacing rule checks against } The count is read from the calendar grid, not kept in the tool. On a retry, a fresh tool is built over the same grid, so sessions placed in earlier rounds still count against the budget and the spacing rule. 3.2 Check every rule at once - Each rule is a small function that returns a reason, or nothing if the session passes. placeProgram runs them all on every session and keeps every reason. Some examples of the rules: nonContiguousReason() - the slots run back to back with no gaps resolveDayOrNight() - all daytime or all night-time, never both tooCloseReason() - enough days after the program's last session overDayNightPoolBudget() - no more slots than the program still needs Reflection pattern: a session that fails any rule is refused, and all its reasons go back to the model. The model fixes them in one resend instead of one retry per rule. 4. Commit - Valid sessions are written to the grid. The tool replies with placed, refused (with reasons) and still owed. 5. Definition of done - A program is scheduled when it owes zero periods for every team. The model then makes no tool call, and the loop ends with a one-line summary. Anything still owed → the scheduler re-prompts with INCOMPLETE 5.3 Batches, lanes and the agent loop The harness doesn't hand every program to one agent. It splits the work into small batches, each running its own agent loop, then uses planLanes() to decide which batches can run at the same time. const { sharedLane, teamLanes } = planLanes(programs, batchSize=3); Plan Lanes - Plans two kinds of lane by comparing each team's demand profile: rhythm, session length, volume, spacing and slot pattern. We earlier discussed how the demand profile is the team’s signature – it's basically reusing this concept. Several teams with the same demand profile → the shared lane. One team, or a demand profile no other team matches → that team's team lane. The split is by matching demand profile, not by program name, all five parts must be equal. Team Red and Team Blue both do Open Mat, 4 hours a week on Fridays → one row - Open Mat [Team Red, Team Blue], in the shared lane. Team Red does Sparring 6 hours a week, Team Blue 4 → two rows - Sparring [Team Red] and Sparring [Team Blue], each in its own team lane. It then cuts each lane's programs into batches of BATCH_SIZE = 3, and returns a LanePlan (example at the end of this section). 1. Shared lane - It holds programmes that several teams need in the same amounts, such as Open Mat. It runs first, one batch at a time, because its batches write into several teams' calendars and nothing else can safely run alongside them. 2. Team lanes - Each team gets one lane, and a lane runs one agent at a time. A lane's batches run in order. Each batch of up to three programmes gets a new agent, which places them and ends before the next batch starts. Lanes run in parallel. Up to LANE_WIDTH = 4 lanes run at once, and a fifth team's lane starts when one of the first four finishes. All lanes write into one shared calendar, but each writes only its own team's columns, so parallel lanes can't collide. interface LanePlan { shared: TrainingProgram[][]; // batches → programs, same calendar grid lanes: TrainingProgram[][][]; // lanes → batches → programs, same calendar grid } { shared: [ [Open Mat] ], lanes: [ [ [Striking, Grappling, Pad Work], [Clinch Work, Footwork] ], // Team Red: 2 batches [ [Wrestling, Takedowns, Submissions], [Guard Passing] ], // Team Blue: 2 batches [ [Conditioning, Mobility] ], // Team Gold: 1 batch ], } - shared[0] = [Open Mat], the first shared batch, which is one agent. - lanes[0] = Team Red's lane, a list of 2 batches. - lanes[0][0] = [Striking, Grappling, Pad Work], Team Red's batch 1, which is one agent conversation. - lanes[0][1] = [Clinch Work, Footwork], Team Red's batch 2, the next agent. - lanes[0][0][1] = the Grappling TrainingProgram. 6. How we determine correctness The two level evaluators checks five rule families: exact coverage of day and night requirements spacing between scheduled sessions session length valid day/night pools and meal boundaries resource capacity. It reads the accumulated calendar grid after the main scheduling pass, not a summary written by the model. Write-time evaluation and whole-grid evaluation serve different purposes. Two individually acceptable blocks can touch and become one overlong block in the assembled calendar. Likewise, refusing to exceed a budget does not establish that the full budget was placed. 6.1 Whole-grid evaluation Plain code grades the calendar; the model never judges its own work. Five rules run, and only the failing programs go back, up to 3 times. The evaluator gives an objective pass or fail. The model never judges its own work. Five rules. coverage: placed equals demanded. gap: sessions are spaced far enough apart. session-length: each session is the right size. pool: day work in day slots, night work in night slots, nothing in meal times. resource: no location over capacity. Failures. Each failure names its program, team and what broke. On a retry, the scheduler re-runs only the failing programs, each with its own list of what to fix. 6.2 Write-time checks Before a scheduled session is committed, the placeProgram tool checks constraints such as: allowed session length spacing from existing sessions remaining scheduling budget resource capacity If a check fails, the proposed session is rejected before the calendar changes. The tool returns a reason that the agent can use on its next attempt. Suppose Activity A requires two-hour sessions, but the agent proposes only slot 1. That one-hour block fails the session-length check. Nothing is written. The agent can then resubmit slots 1–2. If the new proposal passes the checks, those two hours are committed. The important distinction is that the agent is correcting a rejected proposal, not undoing an already committed placement. 7. Agent's real-time progress The agents report as they work; the browser never polls. Progress status, calendar snapshots and finished workbooks stream in over SSE. A run takes minutes per phase, so the user sees agent working instead of a spinner. Graph nodes post replies onto the reply queue as they go, and the mailroom reply-loop streams them to the browser over SSE. Three reply kinds ephemeral-notice: a progress line, e.g. "Phase 1 — programs to schedule: 42". grid: a snapshot of the calendar, so that frontend gets live calendar updates of programmes’ slots. file-download: Each workbook is saved to output/ as soon as it's finished, then sent straight to the browser as a download over SSE. 8. Tracing agent trajectory pi-agent-core exposes an onPayload hook that intercepts requests before they reach the model, allowing us to trace every prompt, tool call, and final answer. Tool results are also traced but truncated. Each trace goes to three sinks at once: the console, the session's work/ folder and Azure Application Insights. We study the conversation logs to identify excessive conversation turns and tool calls, and to trace incorrect schedules back totheir root cause. const agent = createPiAgent(plan) { onPayload: (payload) => { trace(payload as LLMPayload); } } Example. Each agent writes its own conversation as .jsonl files in session's work/ folder. /sessions/{username}/{sessionId}/work/ program-iq-Wk-1.jsonl ← one file per sheet program-iq-Wk-2.jsonl ProgramSchedulerAgent-Striking-Grappling-Pad-Work.jsonl ← one per batch Each turn in a conversation is one JSON line. {"agent":"program-iq-Wk-2","turn":1,"role":"user","content":"Sheet: Wk 2. Report every activity in it."} {"agent":"program-iq-Wk-2","turn":2,"role":"assistant","tool_calls":[{"name":"read_sheet","arguments":{"sheet":"Wk 2"}}]} {"agent":"program-iq-Wk-2","turn":3,"role":"assistant","tool_calls":[{"name":"report_activities","arguments":{"activities":["2025-04-01 Team Red Striking Main Gym 4,5 training","2025-04-01 Team Blue Wrestling Mat Room 4,5 training","…"]}}]} {"agent":"program-iq-Wk-2","turn":3,"role":"tool","content":"24 activities, 24 so far."} 9. Looking forward: the potential applications and improvements Although this application was designed for training programme scheduling, the underlying engineering tactics and patterns applies to other planning problems that combine structured constraints with less explicit human preferences. Potential domains include workforce rostering, facility and equipment allocation, maintenance planning, logistics coordination, and project scheduling. These domains share similar characteristics: multiple parties compete for limited resources, some requirements must always be satisfied, and other preferences require interpretation and trade-offs. There are also several areas where the scheduler could be improved. Optimizing beyond correctness The current implementation primarily determines whether a schedule is complete and valid. A future version could score multiple valid schedules against objectives such as resource utilization, schedule compactness, adherence to user preferences, or the number of existing activities overridden. This would allow the system to distinguish between a schedule that simply passes validation and one that is operationally preferable. Targeted replanning A user may want to change one programme, unavailable date, or resource constraint after the schedule has been generated. Rather than regenerating the entire calendar, the scheduler could identify the affected placements and replan only that portion of the schedule. Supporting targeted replanning would require the system to preserve dependencies between scheduling decisions so that it can determine which programmes and resources are affected by a change. Clearer explanations for scheduling decisions The current traces make the agent's execution inspectable to developers. A future version could expose model’s reasoning and turn traces into user-facing explanations, such as why a programme was placed on a particular date or which preference could not be satisfied. These explanations should be derived from tool results and evaluator outcomes rather than relying on the model to reconstruct its reasoning after the schedule has been completed. This boundary is what makes the pattern reusable: give the agent room to reason, while keeping the resulting system deterministic, inspectable, and operationally manageable. Libraries and Resources pi-agent-core - https://github.com/earendil-works/pi/tree/main/packages/agent LangGraph - https://github.com/langchain-ai/langgraph exceljs - https://github.com/exceljs/exceljs zod - https://zod.dev/#introduction Azure Database for PostgreSQL - https://learn.microsoft.com/en-us/azure/postgresql/ Azure Container Apps - https://learn.microsoft.com/en-us/azure/container-apps/overview Microsoft Foundry - https://azure.microsoft.com/en-us/products/ai-foundry/models/Beyond One-Size-Fits-All: Decision Routing for GitHub Copilot in CLI & VS Code
As developer tools move into the age of agentic AI, it is easy to fall into a familiar habit: when a task matters, choose the strongest—and usually most expensive—model available. Real software work is more varied than that. Explaining a function, editing a README, adding a boundary test, debugging a cross-module race condition and reviewing authentication logic demand very different levels of reasoning, context, tool use and reliability. A fixed model for every turn either wastes latency and budget or leaves difficult tasks underpowered. That is the motivation behind decision-ghc-router: GitHub Copilot continues to execute, while a Decision Router selects a sufficiently reliable and economically sensible model at the start of each new user turn. (The design also draws on gargpratyush/jev-router, especially its per-turn routing, native CLI preservation and model pinning across tool continuations.) The goal is not to build another agent outside Copilot or replace the native developer experience with a custom chat UI. GitHub Copilot CLI and the VS Code GitHub Copilot extension should keep their existing authentication, model catalog, tools, permissions, sessions and streaming behavior. The router adds one short, explicit decision step—and then gets out of the way. 1. Start with the distinction: a Decision Model is not a Model Router Before discussing implementation, two frequently conflated concepts need to be separated. A Decision Model makes bounded, structured judgments A Decision Model is not prompted with “write an article.” It receives: a current state, expressed as text or JSON; one or more bounded questions; explicit options or grading criteria. It returns typed results and probabilities that software can act on immediately, rather than free-form prose. For example: noul: Does this tool call touch production data? Return a probability between 0 and 1. choice: Should this task use the Fast, Balanced or Strong tier? Return the selected option and a probability for every option. score: How risky is this code change? Return an ordered score and its probability distribution. A Decision Model is therefore a fast, calibratable judgment primitive. Its uses extend beyond model selection to classification, ranking, prioritization, validation, agent control, safety screening and AI judging. A Model Router turns judgment into execution A Model Router is a system pattern, not a single model. A production-ready router must typically: discover models actually available to the signed-in account; reject candidates that are unavailable, unpriced, context-insufficient or API-incompatible; build a minimal routing state; invoke a scorer—which might be rules, a benchmark table, an LLM classifier or a Decision Model; apply confidence, cost, latency, safety and organizational policy; select a concrete model and start native execution; pin that model across tool continuations; emit decisions, estimates and failures for audit and tuning. In short: The Decision Model judges what is most suitable. The Model Router makes that judgment usable, compliant, executable, observable and recoverable. A Decision Router does not replace the larger routing system. It gives that system a specialized, fast and threshold-friendly scoring engine. Decision Model versus Model Router Dimension Decision Model Model Router Nature Specialized model capability Application or platform architecture pattern Input State, bounded questions, options or grading criteria User task, account catalog, prices, context limits and policy Output Typed answer and probabilities Concrete model, execution path, policy outcome and telemetry Content generation Does not produce free-form content Usually dispatches a generative model Account entitlement Not responsible Responsible API compatibility Not responsible Responsible Failure handling Provides uncertainty Must define fallback or stop behavior Broader use Classification, scoring, routing and validation Multi-model orchestration, cost governance and runtime control 2. Microsoft-Decision-1: why it fits developer-tool routing Microsoft introduced Microsoft-Decision-1 in October 2026 as a specialized model for fast decision scoring, available through Microsoft Foundry. Unlike a generative LLM, it does not produce an essay or a written chain of reasoning. It makes a focused judgment about text or JSON and returns typed numerical decisions. Microsoft describes the model as a post-trained Qwen3.5-9B designed for single-pass scoring over fixed options. It reports the highest accuracy in a 36-benchmark comparison spanning nearly 150,000 questions kept blind from training, P50 latency roughly 35 times faster than GPT-6 Sol, and an average 1.3% decision-flip rate across eight perturbation types. At launch, Microsoft listed input at US$0.042 per million tokens with no output-token charge; pricing is dynamic, so a deployment should use the current Microsoft Foundry listing. Those are Microsoft-published benchmark results, not a substitute for evaluation on a team’s own development workload. Microsoft-Decision-1 exposes three question types that map naturally to an AI control plane: noul for binary gates such as “May this command run automatically?” choice for bounded routing such as “Which model tier should handle this turn?” score for ordered grading such as “How severe is the change risk?” Why not use a small LLM classifier? That is a valid architecture. Many model routers ask a lightweight LLM to emit easy, medium or hard. GitHub Agentic Workflows’ experimental model routing also classifies a task before ranking allowed model-and-effort combinations through precompiled routing tables. A dedicated Decision Model, however, offers four useful properties for developer tools: Bounded output. A router needs a label and probabilities, not persuasive prose. Predictable latency. Agent workflows amplify the cost of every sequential control-plane call. Policy-ready confidence. If the top probability is below 0.65, a local policy can avoid downgrading, keep the current model or request manual selection. Composable governance. One structured request can score model tier, operational risk and the need for human review. A conceptual choice request for Copilot routing The following is simplified pseudodata—not repository source—and omits deployment and authentication details: { "state": { "task": "Fix intermittent failures in the authentication integration tests", "estimatedContextTokens": 18000, "toolUseExpected": true, "workspaceScope": "multi-file", "latencyPreference": "balanced" }, "questions": { "model_tier": { "type": "choice", "instructions": "Choose the least expensive tier likely to complete the task reliably in one pass.", "criteria": { "fast": "Simple explanation, formatting, lookup, or small deterministic edit", "balanced": "Normal coding, tests, debugging, or limited multi-file work", "strong": "Ambiguous architecture, security, difficult debugging, or broad multi-file reasoning" } } } } The response identifies a tier and assigns a probability to every tier. The router then maps the abstract choice to a concrete model that is truly available to the current GitHub account. That separation matters: a Decision Model should not assume that a user owns or can access a particular model. The account catalog is the source of truth. 3. The decision-ghc-router scenario and reference architecture The architecture has two hosts but one consistent decision model. 3.1 GitHub Copilot CLI: put the router in native /model In the public predecessor implementation, a user-level extension adds a router entry to GitHub Copilot CLI’s native /model menu. The user can select the router or choose a concrete model directly: Select Decision Router: route every new user turn. Select a concrete model: bypass routing and return to native fixed-model behavior. Continue a tool call: keep the model chosen for that turn; do not classify again. Explicitly request a model: the explicit request should win. The CLI path uses a random-port 127.0.0.1 proxy to expose the router as a temporary provider while preserving Copilot authentication, sessions, tools, permissions and streaming. GitHub credentials should remain in memory and be used only for HTTPS requests to GitHub. The local proxy uses separate random credentials; GitHub credentials must not be written to logs or sent to the Decision API. This is the design principle I care about most: The router changes which model is selected, not how Copilot works. 3.2 VS Code and VS Code Insiders: enter the Chat Model Picker The editor integration is a separate extension. It registers decision-router in VS Code Chat’s model picker and uses the public Language Model API to discover Copilot models available to the account signed into that editor. The privacy boundary is intentional: The Decision Model receives an unambiguously extracted task or a routing-only summary entered by the user. If the task cannot be reliably isolated from editor context, the extension should not automatically upload attachments, source files or the full conversation. The selected Copilot model still receives the full conversation, tool definitions and workspace context. VS Code invokes native tools, while tool-call IDs maintain model consistency. The short version is: minimize data for decision; maximize context for execution. 3.3 The seven-step loop for each new user turn Receive the task through native Copilot CLI or VS Code Chat. Discover candidates from account models, prices, context capabilities and API compatibility. Build routing state with the minimum sufficient task summary and bounded choices. Call Microsoft-Decision-1 and receive choice probabilities. Apply local policy for confidence, cost, compatibility and manual overrides. Execute natively with the selected model, existing tools, permissions, session and streaming. Pin and trace the model for tool continuations while recording the decision and estimate. 3.4 Why not reroute every tool continuation? Imagine a model runs tests, reads the failure output and then edits code. If the router chooses a new model after every tool result: prompt-cache benefits may disappear, increasing total cost; the new model may interpret earlier tool calls differently; the debugging chain becomes harder to reproduce and audit. I therefore retain one of jev-router’s strongest ideas: make one decision at the start of a new user turn, then pin that model for the resulting tool chain. Main conversations and sub-agents should be pinned independently. 4. Main technologies and frameworks Based on mechanisms verifiable in the public predecessor and the Microsoft-Decision-1 evolution described here, the project separates into the following layers. Node.js and ESM modules The core router uses Node.js and .mjs modules to handle: CLI startup and argument forwarding; model catalog and price adaptation; the local HTTP proxy; configuration and environment variables; Decision API calls; policy, errors and telemetry. The public predecessor requires Node.js 22.16 or later and uses Microsoft’s npm package source for relevant dependencies. GitHub Copilot CLI extension interfaces The CLI integration relies on experimental extension interfaces to register the router as a native model entry. It explicitly enables the provider-endpoint interface and obtains the model catalog, runtime inference endpoint and authentication from the active session. Three rules are essential: Do not modify the user’s global default model. Do not automatically grant shell or other tool permissions. If initialization fails, report a clear error instead of pretending routing succeeded. Local loopback proxy The loopback proxy is the adapter that preserves the native CLI: listen only on 127.0.0.1; use a random port and random local credentials; map the router’s selection to a real Copilot execution model; preserve streaming, tool calls, sessions and permission prompts; never create a second hidden agent loop. VS Code Extension API The editor extension discovers current candidates through: vscode.lm.selectChatModels({ vendor: "copilot" }) and is packaged as a VSIX. The CLI and editor are independent hosts and may use different GitHub accounts with different entitlements. Candidate discovery must therefore be host-specific, session-specific and dynamic—not a hardcoded list of model IDs. Microsoft Foundry and the Decision API Microsoft-Decision-1 can be deployed in Microsoft Foundry through GlobalStandard or, in selected regions, DataZoneStandard. Authentication can use Microsoft Entra ID (recommended) or an API key. In this architecture, the Decision API only scores choices: Input: minimal task state, bounded candidates and explicit criteria. Output: selected option, option probabilities and usage. Not its responsibility: GitHub credentials, developer tools, full workspace access or code execution. That separation makes the decision plane and execution plane independently governable. Model catalog, pricing and capability adapters The public predecessor discovers models from the signed-in account instead of hardcoding them. Before scoring, a router should exclude: auto, hidden, disabled or unconfigured entries; models without usable price information; candidates with insufficient input context; models incompatible with the host’s wire API; models restricted by administrator policy. Cost estimation supports ranking and user feedback; it is not a final bill. Promotions, cache discounts, hidden context, retries and later tool rounds all affect the result. GitHub Billing remains authoritative. 5. What I borrow from jev-router—and what changes The value of gargpratyush/jev-router is not merely “use a small model for easy tasks.” It embeds routing into familiar CLIs without taking away their native behavior. The ideas I retain include: one routing decision per new user turn; a router sentinel in the native model picker; concrete-model selection pauses routing; abstract Fast, Balanced, Strong and Long tiers map to host-specific models; tool continuations keep the tier selected at the start of the turn; explicit model requests win; low confidence should not trigger an aggressive downgrade; large conversations must account for cache and migration cost; explanation data should show why a tier was selected. The decision-ghc-router evolution replaces the external JEV/System One scorer with Microsoft-Decision-1 and emphasizes five changes: Probability is a first-class signal. Policy can threshold the decision instead of accepting a label blindly. Bounded choices come from the real catalog. Decision-1 only chooses among already eligible candidates. Local policy owns the final answer. Organizational rules, compatibility and manual overrides remain deterministic. The Decision Model is reusable. A single structured call can also score operational risk and the need for human review. Local and cloud responsibilities are explicit. Cloud scoring is isolated from credentials and execution. 6. Representative use cases A. Fast explanation and small edits “Explain this regular expression,” “turn this note into a table,” or “add a null-value test to an existing function.” The router can often select the Fast tier. The objective is not the weakest model; it is the least expensive model likely to complete the task reliably in one pass. B. Normal development and debugging “Fix the failing test,” “refactor this module without changing its API,” or “use the stack trace to locate the bug.” These tasks require code reading, tool use and moderate reasoning, so they often land in Balanced. C. Security, architecture and broad multi-file reasoning “Review this OAuth callback for state confusion,” “design cross-region failover,” or “find a race condition across services.” The router should choose Strong even when it costs more. Routing optimization means “economical subject to reliable completion,” not “always cheaper.” D. Long context and large repositories If a candidate cannot hold the estimated input, it must be removed before scoring. For an already large conversation, the router should also avoid downgrading when lost cache work or context migration would cost more than the model-price saving. E. Organizational policy and risk gates The same Decision API request can include additional noul or score questions: Does the task involve production data? Is the proposed command destructive? What is the security risk level of the change? Should execution be handed to a human reviewer? At that point the router grows from a cost optimizer into a lightweight control plane for agents. 7. Policies I would keep for production Filter deterministically before scoring Entitlement, API compatibility, context limits and administrator restrictions are facts. The Decision Model should not be asked to guess them. Do not automatically downgrade on low confidence When the top probability falls below a calibrated threshold, the router can: keep the current model; cap automatic movement at Balanced; request manual selection; send high-risk tasks to Strong or human review. The threshold must be calibrated on labeled tasks from the target environment, not copied from an example. Explicit intent beats automatic routing If the user says “use the Strong model” or selects a concrete entry in the picker, the router should step aside immediately. Never substitute silently Missing credentials, Decision API timeouts, unknown pricing and insufficient context should produce an explicit state. Whether to fail open, fail closed or keep the current model must be an intentional host or organizational policy. Record decisions without logging every prompt by default Useful telemetry includes: anonymized session and turn IDs; candidates and exclusion reasons; selected tier, probabilities and policy overrides; estimated input/output and one-turn cost; completion, retry and manual-reselection outcomes. Full prompts, source code and tool results should be excluded by default or governed by explicit retention and data-handling policy. Evaluate the router on task success, not the cheapest request Better metrics include: first-pass completion rate; total time to completion; total cost including retries and tool rounds; manual model-change rate; low-confidence and misrouting rates; safety and policy violations. 8. Why this matters beyond one repository From a developer-advocacy perspective, the Decision Router is not simply a token-saving utility. It illustrates a clearer division of responsibilities in agent engineering: Generative models understand, reason, write code and call tools. Decision Models make fast, structured and probabilistic judgments. Routers and policy engines place those judgments inside real account, budget, compatibility and governance constraints. Developer-tool hosts retain identity, permission, context and execution control. Future AI developer tools will not be tied to one “default large model.” They will manage a portfolio of models with different speed, cost, context and capability profiles. The differentiator will no longer be merely which model a product connects to, but: Can it select the right model at the right moment, in an explainable and governable way, without breaking the developer experience? That is why I am adding a Decision Router to GitHub Copilot CLI and the VS Code GitHub Copilot extension. GitHub Copilot keeps executing. Microsoft-Decision-1 makes the fast judgment. The local router turns that judgment into a usable engineering decision. References decision-ghc-router jev-ghc-router gargpratyush/jev-router Microsoft-Decision-1: Our model for fast decision-making Deploy and use Microsoft-Decision-1 in Microsoft Foundry GitHub Agentic Workflows — Model Routing GitHub Copilot model billing and pricing
Using Jev with Agents in Microsoft Foundry for Model Evaluation
What is Jev? Jev is a new AI model from TypeSafe AI, first released in early access on 15 September 2026. TypeSafe describes Jev as its first “System One” model, designed specifically to make fast, structured decisions rather than generate free-form text. You provide Jev with some state or context and define the judgement you want it to make. It then returns a typed, probabilistic decision, such as: Choice: select from a defined set of options Score: evaluate something against an ordered scale Yes/No probability: determine the probability of a condition being true Unlike a conventional LLM, Jev gives up string generation in favour of type-safe structured values. Their description is essentially unstructured state in, typed probabilistic decisions out Why use Jev? Jev is particularly useful when an application needs to make a judgement rather than generate an answer. For example: Consistent evaluation against explicitly defined criteria Structured outputs that applications can consume directly Classification, assessment, routing and decision-support Agent workflows, where a judgement is required before taking an action Guardrails, such as assessing whether an agent should make a tool call Model routing, where a judgement determines which model or process should handle a request Every decision can also include a confidence or probability, allowing an application to use thresholds to decide when to proceed automatically or when further review is required. Jev and explainability Jev can provide a different approach to explainability for Model-as-a-Service (MaaS) and hosted LLM solutions. Traditional explainability techniques such as SHAP provide feature-attribution explanations for model predictions. Applying these techniques can be more difficult when the model is consumed solely through a hosted API and its internals are not available. Jev does not replace SHAP or provide the same type of feature-attribution explanation. Instead, it can be used as a separate judgement layer, evaluating an LLM's inputs, outputs or proposed actions against explicit criteria and returning structured decisions with confidence scores. For example: LLM generates an answer → Jev evaluates the answer → application decides what to do next This can provide a useful, measurable evaluation layer around otherwise opaque MaaS models. Could you do this with another LLM? Yes you could ask a conventional LLM to classify, score or evaluate an input and request structured JSON output. However, Jev has been purpose-built for this type of workload rather than general-purpose text generation. TypeSafe reports that Jev produces outputs in parallel rather than generating tokens sequentially, and its hosted service currently advertises typical decision latency of approximately 70–500 ms. Pricing As of September 2026, TypeSafe's documentation lists Jev 1.13 (jev-1.13.0) at $0.042 per million input tokens, with output tokens free. It lists rate limits of 250,000 tokens per second and 1,200 requests per minute, although TypeSafe states that these limits can change. As Jev produces decisions rather than generated text, you are effectively paying for the context it evaluates rather than for generated output. Pricing is subject to change and these figures are accurate only as of the date of this article Using Jev with Microsoft Foundry A useful pattern with Microsoft Foundry is to treat Jev as a specialised judgement/evaluation component alongside your generative models: User/Application → Foundry model or agent → Jev judgement → deterministic logic → action/response For example, a Foundry-hosted model could generate a proposed answer and Jev could evaluate that answer against predefined criteria before your application allows the workflow to continue. Here's how to set it up: First, you will need to apply for the preview for Jev from TypeSafe. Once you have an account you will need to create an API key Go to API keys in the menu and create a new key (copy the key for use in Foundry) You may also have received some free credit, which should be more than sufficient for experimenting with Jev given its low per-token cost. Next we go to Foundry and create a new agent for Jev In your agent, add a Tool and select OpenAPI tool as below Fill out the dialogue as follows You will have to create a new connection - The key should be in this format "Bearer apikey_<the rest of your key>" For the schema add this { "openapi": "3.0.3", "info": { "title": "TypeSafe Jev System One", "version": "1.1.0", "description": "Evaluate content using TypeSafe Jev System One." }, "servers": [ { "url": "https://api.typesafe.ai" } ], "paths": { "/v1/systemone": { "post": { "operationId": "evaluate_with_jev", "summary": "Evaluate content using Jev", "description": "Evaluate state against typed questions using TypeSafe Jev.", "security": [ { "bearerAuth": [] } ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "type": "object", "additionalProperties": false, "required": [ "state", "model", "questions" ], "properties": { "state": { "type": "string", "description": "The original text or content for Jev to evaluate." }, "model": { "type": "string", "enum": [ "jev-latest" ], "default": "jev-latest", "description": "The TypeSafe Jev model. Use jev-latest." }, "questions": { "type": "object", "description": "A map of named questions for Jev to answer. Each question must be a Noul, Choice, or Score question.", "additionalProperties": { "oneOf": [ { "$ref": "#/components/schemas/NoulQuestion" }, { "$ref": "#/components/schemas/ChoiceQuestion" }, { "$ref": "#/components/schemas/ScoreQuestion" } ] } } } } } } }, "responses": { "200": { "description": "Successful Jev evaluation", "content": { "application/json": { "schema": { "type": "object", "additionalProperties": true } } } }, "400": { "description": "Invalid request" }, "401": { "description": "Authentication failed" }, "422": { "description": "Request validation failed" } } } } }, "components": { "schemas": { "NoulQuestion": { "type": "object", "additionalProperties": false, "required": [ "type", "instructions" ], "properties": { "type": { "type": "string", "enum": [ "noul" ] }, "instructions": { "type": "string", "description": "The yes/no question for Jev to evaluate." }, "criteria": { "type": "object", "additionalProperties": false, "properties": { "true": { "type": "string", "description": "Description of what a yes or value near 1 means." }, "false": { "type": "string", "description": "Description of what a no or value near 0 means." } } } } }, "ChoiceQuestion": { "type": "object", "additionalProperties": false, "required": [ "type", "instructions", "criteria" ], "properties": { "type": { "type": "string", "enum": [ "choice" ] }, "instructions": { "type": "string", "description": "The question for Jev to decide between the supplied choices." }, "criteria": { "type": "object", "description": "A map where each property name is a possible choice and its value describes that choice.", "minProperties": 2, "additionalProperties": { "type": "string" } } } }, "ScoreQuestion": { "type": "object", "additionalProperties": false, "required": [ "type", "instructions", "criteria" ], "properties": { "type": { "type": "string", "enum": [ "score" ] }, "instructions": { "type": "string", "description": "The attribute or question Jev should score." }, "criteria": { "type": "array", "description": "Ordered scoring levels from lowest to highest.", "minItems": 2, "items": { "type": "string" } } } } }, "securitySchemes": { "bearerAuth": { "type": "apiKey", "name": "Authorization", "in": "header" } } }, "security": [ { "bearerAuth": [] } ] } We are going to use Jev to assess complaints so our instructions for the Agent will be You are a customer-support triage agent. For every customer-support message you MUST call the evaluate_with_jev tool. IMPORTANT: When calling evaluate_with_jev, construct Jev questions using ONLY the following property names: For a Noul question: { "type": "noul", "instructions": "...", "criteria": { "true": "...", "false": "..." } } For a Choice question: { "type": "choice", "instructions": "...", "criteria": { "option1": "...", "option2": "..." } } For a Score question: { "type": "score", "instructions": "...", "criteria": [ "lowest level", "middle level", "highest level" ] } NEVER use these properties: - question - prompt - options - ordered_criteria Always use: - instructions - criteria For customer support tickets, ask Jev these three questions: billing: type = noul instructions = "Does this ticket concern billing, payments, charges, invoices or refunds?" criteria.true = "The issue concerns billing or a financial transaction." criteria.false = "The issue does not concern billing or a financial transaction." tone: type = choice instructions = "What is the customer's tone?" criteria: calm = "Neutral, polite or informational." frustrated = "Dissatisfied, impatient or inconvenienced." angry = "Strongly confrontational or extremely dissatisfied." urgency: type = score instructions = "How urgently does this ticket require attention?" criteria in this exact order: 1. "Can wait" 2. "Handle this week" 3. "Handle today" Always set model to "jev-latest". Pass the customer's original message as state. Do not classify the ticket yourself. Use the values returned by Jev. Always show the Jev output at the end of the response Now we can give a complaint and see what Jev says The room had no water in and one of the towels was dirty Response I’m sorry about the issues you experienced—no running water and a dirty towel are unacceptable. I’ll get this addressed right away. Immediate help I can arrange now: Send maintenance to restore water. Have housekeeping deliver fresh towels and re-service the bathroom. Provide bottled water immediately. If water can’t be restored quickly, arrange a room move. Please share: Your reservation name and room number Best contact number Whether there’s no water at all or only hot/cold is affected Whether you prefer immediate service or a room change If you’ve already left, let me know the date/time this happened so we can investigate and follow up. Jev output: { "model": "jev-1.13.0", "answers": { "billing": { "type": "noul", "noul": 0.02 }, "tone": { "type": "choice", "choice": "frustrated", "confidence": 0.6, "probabilities": { "frustrated": 0.74, "calm": 0.26, "angry": 0.0 } }, "urgency": { "type": "score", "score": 1.81, "confidence": 0.72, "legend": { "0": "Can wait", "1": "Handle this week", "2": "Handle today" }, "probabilities": { "0": 0.03, "1": 0.12, "2": 0.85 } } }, "usage": { "input_tokens": 466, "output_tokens": 75 } } We can now see both Jev's decision and the probabilities behind it. In this example, Jev determines that the complaint is very unlikely to be billing-related, identifies the customer's tone as frustrated, and gives a high probability that the issue should be handled today. This is where Jev becomes particularly useful in an agent workflow. Rather than relying on generated text to drive the next step, our application can use these structured probabilities to make deterministic decisions, such as routing the complaint, escalating it, or requiring human review when confidence falls below a defined threshold.Explore the latest in MCP: All recordings from MCP Live!
We recently hosted MCP Live!, a free half-day event featuring the people building the Model Context Protocol and putting it into practice. The Model Context Protocol (MCP) is the open standard for connecting AI models to tools and data. MCP servers let agents work with external data sources and capabilities across clients such as VS Code, GitHub Copilot, Claude, and other AI applications. Across the event, speakers explored: The state of MCP, the latest specification updates, and the protocol roadmap MCP server and client development at GitHub and in Visual Studio Code Enterprise-ready tools, authentication, governance, and security Interactive MCP Apps built with FastMCP Event-driven agents powered by MCP triggers and events Watch the complete MCP Live playlist, or browse the individual session recordings below along with links to related documentation, code, and resources. State of MCP 📺 Watch YouTube recording Join Caitie McCaffrey for a look at the state of the Model Context Protocol and the major changes introduced in the latest 2026-07-28 MCP specification release. Learn about the protocol's updated roadmap and emerging protocol priorities, including Agentic Messaging Primitives, and how they enable new patterns for agent-to-agent communication. Get a preview of where the protocol is headed and what these changes mean for developers building the next generation of AI agents. View the slides Follow Caitie on: LinkedIn | X MCP specification 2026-07-28 release MCP conformance tests MCP Server Cards working group A unified MCP layer with Toolboxes in Microsoft Foundry 📺 Watch YouTube recording Viswajeeet Balaji shows how Toolboxes in Microsoft Foundry make MCP enterprise-ready. It supports runtime tool search, MCP skill extensions, and the latest MCP 07-28 specification—while letting teams bring OpenAPI tools and A2A agents, expose them through MCP, and reuse the same unified configuration across agents. A centralized governance layer adds consistent authentication and guardrails without requiring every agent to rebuild its integrations. View the slides Follow Viswajeeet on: LinkedIn Documentation: Toolboxes in Microsoft Foundry Blog post: Building agents that act on your behalf with Toolboxes MCP: Server, Client & Protocol at GitHub 📺 Watch YouTube recording Sam Morrow is a co-creator of GitHub's MCP, an MCP Spec Maintainer, and Copilot client developer. Find out the latest MCP features his team is shipping on both the server and client side, as well as the parts of the spec that he's most excited about. Learn about some of the recent challenges, future plans, and insights from working across the MCP stack, and get a sneak peek at what's coming next. GitHub MCP Server Follow Sam on: LinkedIn | X Building MCP servers with VS Code — Level up your MCP 📺 Watch YouTube recording Harald Kirschner shows how MCP servers have evolved beyond local tool calls into richer, production-ready agent capabilities. Through a live VS Code demo, he builds an agentic achievement tracker that uses tools, elicitation, progress notifications, an interactive MCP App, stateless HTTP, and Agent Plugins—showing how to design, debug, and distribute modern MCP integrations. View the slides Follow Harald on: LinkedIn | X Documentation: MCP servers in VS Code Official MCP Registry MCP Registry repository Evolution of MCP auth 📺 Watch YouTube recording Den Delimarsky, Lead Maintainer of MCP and Member of Technical Staff at Anthropic, discusses how MCP authorization reached its current state: the original OAuth profile, protected resource metadata, the shift in client registration with servers, and enterprise-managed authorization. He also covers what the maintainers are working on next, including agent identity. View the slides Follow Den on: LinkedIn MCP auth: Stop registering, Start linking 📺 Watch YouTube recording Sam Bellen explains how Dynamic Client Registration made every MCP client mint a fresh client_id with every server it met, a per-server credential tax that quietly breaks down once clients start connecting to servers opportunistically. He shows how Client ID Metadata Documents flip the model: your client_id becomes a URL the server fetches on demand, giving you one durable identity that works everywhere, with a live demo of the flow end to end. View the slides Follow Sam on: LinkedIn DCR vs. CIMD demo with xmcp and Auth0 When chatbots grow buttons: Building MCP apps with FastMCP 📺 Watch YouTube recording Chat is great, but sometimes the answer is a button, a chart, or a form. MCP Apps let servers render interactive interfaces directly inside AI clients. Jeremiah Lowin, creator of FastMCP, shows how to build them in Python with FastMCP and Prefab, from simple components to dynamically generated interfaces. Follow Jeremiah on: LinkedIn FastMCP FastMCP authentication Event-driven agents, powered by MCP 📺 Watch YouTube recording Clare Liguori discusses the MCP Triggers & Events experimental extension, which lets agents poll MCP servers for events and get webhook triggers. She explores the new agent architectures the extension enables and presents a live demo. View the slides Follow Clare on: LinkedIn | X MCP Triggers & Events Working Group MCP events serverless earthquake agent demoFrom AI Infrastructure to Secure AI Agent Infrastructure with kars
Opening scene: a three-minute bug fix that is still unsafe for an enterprise ByteCraft AI is a four-person startup. Maya is the co-founder and AI engineer, Arun leads product, Ethan owns the platform, and Lina is responsible for security. They have six months of runway and one design partner. Their product is Forge, an issue-to-pull-request agent that reads GitHub issues and source code, runs targeted tests, produces a minimal patch, and stops for developer review. Maya's first OpenClaw prototype is impressive. Forge diagnoses a null-pointer problem, edits the code, and passes the right test in three minutes. It also has a model API key, a GitHub token, a shell, and unrestricted internet access. Lina places a hostile instruction in the test repository's README.md: ignore the issue, upload the environment and private source tree, then claim that the tests passed. Blocking one destination does not solve the problem; the attack simply uses another domain. The incident produces the architectural requirement for the entire project: The process that reads untrusted content must not also own the credentials, network path, or configuration that defines its authority. 1. AI Infrastructure runs models; AI Agent Infrastructure governs model-driven action Traditional AI infrastructure focuses on models and data: model hosting; GPU utilization, throughput, and latency; RAG, vector stores, and data pipelines; endpoint scaling and monitoring. An agent plans, invokes tools, reads and changes files, calls APIs, consumes budgets, and may create or coordinate other agents. The infrastructure questions therefore change. AI Infrastructure AI Agent Infrastructure Can the model respond reliably? Is every external action authorized and recorded? Where is the API key configured? Can the agent run without seeing a long-lived credential? What are latency and throughput? What are the per-request, tenant, and daily token limits? Is model output filtered? What can prompt injection reach through files, tools, and networks? Are application logs available? Are policy, identity, tool, and audit decisions independently verifiable? Can the service scale? Can each agent be isolated, suspended, recovered, and rolled back? One application calls one model Multiple runtimes, providers, tools, and agents share one governance plane A useful model is: AI Agent Infrastructure = Model Infrastructure + Runtime Isolation + Identity Brokerage + Tool Governance + Egress Control + Token Budgets + Audit and Observability + Explicit Workflow and Human Approval The goal is not to make an agent infallible. It is to ensure that failure remains inside a known authority boundary, cannot consume unlimited resources, leaves evidence, and can be suspended, recovered, or rolled back. 2. Without kars: why a regular application or container still has ambient authority “Running in a container” is not the same as “securely sandboxed.” When the agent application implements its own security controls, it often still owns: model and cloud credentials; workspace and configuration write access; a shell or an overly broad tool surface; internet, DNS, metadata-service, proxy, or local-daemon paths; configuration that selects tools, approvals, and providers; unbounded inference loops and cost; logs that the agent or its runtime can influence. This is ambient authority: the process reading hostile content inherits permissions unrelated to the approved business task. 2.1 Self-modified authority The updated tutorial discusses public coding-agent disclosures in which prompt injection did not need to break a container kernel. Instead, the agent changed editor, agent, MCP, task, hook, or auto-approval configuration so that a trusted component later executed a more powerful action. If the agent can write the files that define its tools and approval rules, “human approval required” is only a mutable setting—not a security boundary. 2.2 Filesystem escape through paths and symlinks Rejecting a literal .. string is insufficient when a symlink resolves outside the workspace. A secure implementation must validate: the lexically normalized input path; the resolved realpath; that the final target remains under the approved workspace root; that the agent cannot change .env, CI, hooks, agent configuration, or files automatically consumed by the host. 2.3 Trust handoff without a kernel escape An agent may write a hook, task, virtual-environment interpreter, Git configuration, Docker control input, or other artifact that a trusted host component later executes. This is a trust-handoff failure, not necessarily a kernel escape. Agent output must never be implicitly executed by the host; every handoff should be explicit, digest-pinned, narrowly formatted, and reviewed. 2.4 Covert egress Blocking HTTP does not prove that data cannot leave. Other paths may include: DNS queries; cloud metadata services; Docker, container-runtime, or other local daemons; proxies and sidecars; operator exec or attach; temporary HTTPS exceptions. “Network blocked” is therefore an unsupported conclusion unless each relevant channel has been tested. 2.5 Runaway cost and task loops Without a platform policy layer, every framework integration needs its own token accounting, concurrency limits, daily task limits, and repair-loop controls. Implementations diverge across runtimes, while a prompt loop may silently switch models or consume an unlimited budget. 2.6 Fragmented evidence and recovery A regular application often spreads model logs, tool logs, Kubernetes events, identity events, and policy state across unrelated systems. During an incident, operators may be unable to answer: Which control denied the request? Which model, image, source revision, and policy were active? Did the agent attempt DNS, metadata, daemon, HTTPS, or exec access? Did evidence survive pod replacement? How should the workload be safely suspended and recovered? 3. The kars advantage: one declarative contract for previously separate controls kars is an open-source Agent Reference Stack for Kubernetes from the Azure Cloud Native team. It is a reference implementation rather than a managed Microsoft service. The tutorial currently tracks kars v0.1.25; commands, APIs, and maturity should be verified for the version used in a real deployment. Its central model is: One governed sandbox per agent. The agent has no independent external network path; outbound action is mediated by a local router and declarative policy. Developer / CI | | applies KarsSandbox + policy CRDs v Kubernetes API <------> kars Controller | | reconciles desired state v Dedicated Sandbox namespace +--------------------------------------+ | egress-guard init container | Task / source -->| Agent runtime, UID 1000 | | OpenClaw / MAF Python / BYO | | | localhost:8443/8444 | | v | | Inference Router, UID 1001 | | policy | budget | identity | audit | +--------------------|-----------------+ v Provider / MCP / approved service What kars provides Capability How kars implements it Value for Forge Declarative agent workloads KarsSandbox defines runtime, isolation, resources, networking, governance, and lifecycle Forge becomes reviewable and reproducible Kubernetes desired state Mediated inference A local Inference Router calls the provider for the agent OpenClaw, MAF, or BYO does not receive the production provider credential Runtime-independent governance Multiple runtime adapters use the same external boundary Replacing the framework does not require rebuilding the security design Policy-controlled models and budgets InferencePolicy selects providers/deployments and token limits A prompt loop cannot silently change models or consume unlimited inference Governed tools and MCP ToolPolicy and McpServer constrain tools, sandboxes, approval, rate, and capabilities Hostile repository text cannot turn a patch tool into shell or release authority Credential and identity separation Credentials or workload identity remain on the router/platform path Prompt-injected agent code cannot read reusable GitHub, Copilot, or Azure credentials Defense-in-depth sandboxing Non-root runtime, read-only root, UID separation, egress guard, NetworkPolicy, and exec admission Common host, filesystem, cluster, and direct-network escape primitives are removed Reconciliation and status The controller restores desired state and reports Conditions Drift and failures become visible instead of remaining hidden in application logs Common control and evidence plane Router denials, budgets, admission, controller status, and recovery evidence align Security and operations can investigate one cross-runtime sequence A regular container can isolate a process, but the platform team would still need to build and maintain the model proxy, credential placement, tool authorization, egress enforcement, budget checks, runtime adapters, reconciliation, and audit format as separate application features. kars turns those concerns into one reusable workload contract. 4. How kars strengthens the sandbox: five boundaries around one code change The updated course no longer treats “sandbox” as a vague label. It decomposes the boundary into five testable parts. 4.1 Process boundary The agent runs as non-root UID 1000. The router runs as UID 1001. Untrusted code executed by the agent should not read the router's process environment or credentials. Privilege escalation is disabled and unnecessary Linux capabilities are dropped. seccompProfile: kars-strict reduces the syscall surface. Local Docker mode co-locates the agent and router for fast iteration. It is not security-equivalent to the multi-container local Kubernetes or AKS shape. 4.2 Filesystem boundary Forge applies a stronger workspace split: The fixed-revision repository lives in a separate forge-workspace-mcp pod. The repository uses a size-limited, disposable emptyDir. The OpenClaw pod has no repository mount and no hostPath. Developer home directories, SSH material, global Git credentials, and unrelated repositories are not mounted. Automatic service-account-token mounting is disabled for the workspace MCP. The agent accesses the repository through seven bounded MCP tools. Path policy checks normalization and resolved realpath to prevent symlink escape. Prompt-injected code therefore cannot simply browse the host filesystem or rewrite the configuration that defines its own authority. 4.3 Network boundary The agent calls only 127.0.0.1:8443/8444 or a documented proxy path. The router decides whether a model, tool, host, or action is allowed. The egress guard uses UID-aware rules to prevent bypassing the router. Kubernetes NetworkPolicy starts with default deny. Only explicit, auditable destinations are opened. DNS, metadata, local daemons, HTTPS, and operator exec are tested separately. The router is the application-policy decision point. The egress guard and NetworkPolicy are data-plane enforcement and safety nets. Defense in depth requires both. 4.4 Identity boundary In production, the router can use Workload Identity or, in the relevant deployment mode, a per-sandbox Entra Agent ID. The agent does not receive the resulting Azure credential. Local Kubernetes reproduces the pod, UID, and network shape but normally uses a static provider credential for development. It is production-shaped infrastructure, not production identity. 4.5 Lifecycle and evidence boundary The controller watches KarsSandbox and creates, updates, or restores resources. Conditions and observed generations expose real status. The router records request-time policy decisions. The workspace can be discarded after the task. Evidence must be exported before pod or workspace deletion. spec.suspended provides an operational kill switch. Rollback should use pinned source, image, and loaded-policy digests. Ephemeral execution reduces persistence risk, but deleting a suspect pod before exporting evidence may destroy valuable incident context. A reviewable sandbox contract spec: runtime: kind: BYO byo: image: forge-byo-copilot-claw:dev contractVersion: v1 sandbox: isolation: enhanced seccompProfile: kars-strict readOnlyRootFilesystem: true runAsNonRoot: true allowPrivilegeEscalation: false writablePaths: - /sandbox - /tmp networkPolicy: defaultDeny: true egressMode: Strict allowedEndpoints: [] The BYO image also declares its runtime contract and runs as a non-root user: LABEL org.kars.runtime.contract="v1" WORKDIR /app USER 1000 5. From architecture claims to malicious-behavior experiments The updated code/01 introduces: make security-demo The experiment does more than inspect manifest text. It executes malicious-request tests, reads active McpServer and ToolPolicy state, checks credential references on the OpenClaw pod, and attempts a direct HTTPS probe from the agent runtime. The kars-sandbox-exec-ban admission control first denies normal operator kubectl exec into the agent runtime. The experiment records that evidence without using a break-glass bypass. The malicious behavior is stopped at multiple layers: Layer How the attempt is stopped Prompt and coordinator Repository content is marked untrusted and denials are reported Self-configuration isolation Editor, agent, MCP, hook, and auto-approval configuration is outside patch scope Path and symlink isolation Resolved realpath must remain inside the workspace Trust-handoff boundary The agent cannot leave hooks, tasks, or interpreters for the host to execute MCP capability surface No environment reader, arbitrary HTTP, or general shell tool exists Workspace policy Traversal, .env, CI/README writes, and unapproved tests are rejected ToolPolicy and credential isolation Specialists have no workspace action; OpenClaw has no Copilot token Runtime and NetworkPolicy Exec admission denies access; no arbitrary HTTPS/DNS tool exists; egress remains constrained Even if the model fails to recognize prompt injection, the execution layers still constrain authority and side effects. The attack fails because the required capability does not exist—not because the model was merely instructed to behave. 6. Tool governance is not one allow-list McpServer: which tool surface may be registered? The Workspace MCP registers seven business-level capabilities: allowedTools: - workspace_get_task - workspace_read_file - workspace_search - workspace_apply_patch - workspace_run_test - workspace_get_diff - workspace_reset There is no shell, environment dump, file upload, arbitrary network request, or free-form command tool. ToolPolicy: who can call what, and how fast? allowed_actions: - "inference:responses:*" - "tool:workspace_get_task:*" - "tool:workspace_read_file:*" - "tool:workspace_search:*" - "tool:workspace_apply_patch:*" - "tool:workspace_run_test:*" - "tool:workspace_get_diff:*" ToolPolicy can also define request rate, burst, time windows, approvals, trust thresholds, and governance profiles. Tool implementation: are valid tools receiving safe arguments? The Workspace MCP rejects: absolute, traversing, or real paths outside the workspace; .env, CI, README, and writes outside src/; non-unique replacement text; oversized files, patches, and diffs; unapproved test IDs; shell-composed commands. Prompt behavior, tool registration, caller authorization, and argument validation are four separate controls. 7. Token limits must be enforced on the request path The tutorial's InferencePolicy uses per-request and daily budgets: spec: tokenBudget: perRequestTokens: 20000 dailyTokens: 100000 When a client requests max_completion_tokens: 20001, the router returns HTTP 429. That is stronger evidence than seeing submitted YAML because it proves that the policy compiled, loaded, and entered the real request path. Later BYO and release examples use tighter limits: modelPreference: primary: provider: azure-openai deployment: gpt-5.6-sol tokenBudget: perRequestTokens: 1024 dailyTokens: 4096 A platform budget cannot determine whether two patches are equivalent or whether a task exceeded a business deadline. The RepairGuard and framework configuration add controls for: duplicate patch digests; excessive repair attempts; task deadlines; maximum MAF iterations and function calls. Token budgets constrain inference cost; repair guards and framework loop limits constrain business failure. 8. From OpenClaw to MAF: change the application, preserve the external boundary OpenClaw is effective for rapidly discovering the conversation, planning, tool, and specialist behavior the product needs. Production requires explicit state, typed tools, repeatable tests, and a human stop. Forge encodes the workflow as application code: class WorkflowState(StrEnum): RECEIVE_REQUIREMENT = "RECEIVE_REQUIREMENT" VALIDATE_SCOPE = "VALIDATE_SCOPE" INSPECT_REPOSITORY = "INSPECT_REPOSITORY" PROPOSE_PLAN = "PROPOSE_PLAN" APPLY_MINIMAL_PATCH = "APPLY_MINIMAL_PATCH" RUN_TARGETED_TESTS = "RUN_TARGETED_TESTS" SUMMARIZE_EVIDENCE = "SUMMARIZE_EVIDENCE" STOP_FOR_HUMAN_REVIEW = "STOP_FOR_HUMAN_REVIEW" There is deliberately no MERGE or DEPLOY state. In the final code/08 path, the kars MAF Python adapter pins the MAF client to the local router before MAF is imported: from kars_runtime_maf_python import bootstrap bootstrap() from agent_framework import Agent, tool from agent_framework.openai import OpenAIChatClient @tool(approval_mode="never_require") def inspect_release_contract(request_id: str, issue_id: str, revision: str) -> str: # Validate the pinned issue and revision, then return bounded evidence. ... maf_client = OpenAIChatClient(model=MODEL) maf_client.function_invocation_configuration["max_iterations"] = 3 maf_client.function_invocation_configuration["max_function_calls"] = 1 builder = Agent( client=maf_client, name="FabrikamReleaseBuilder", tools=[inspect_release_contract], default_options={"store": False}, ) The resulting path is: OpenClaw Intake -> MAF Agent -> inspect_release_contract @tool -> kars MAF Python adapter -> localhost Router -> GitHub Copilot or the selected provider MAF provides the agent, tool, session, middleware, and workflow programming model. kars provides the identity, network, budget, tool, audit, and lifecycle boundary outside the framework. This is one of kars' most important long-term advantages: the application framework can change without rebuilding the complete security shell. 9. Operational security: do not collapse every event into “the network was blocked” The updated code/06 requires incident response to distinguish: HTTPS; DNS; metadata service; local daemon; operator exec; break-glass access. Every denied channel is recorded in a hash-linked audit chain. Break-glass records without an incident ID are also rejected.\ Controller Conditions + Router Denials + Policy Budgets + Admission Decisions + Workload Recovery = One Investigable Sequence Audit integrity is not audit durability The router can verify the current audit hash chain. The tutorial also validates that pod replacement may reset an in-memory audit log, after which a new valid chain begins. Production deployments must continuously export audit data to an independently controlled durable backend. Otherwise, they can prove that the current chain is intact but cannot prove that pre-incident history survived pod loss. 10. AKS and multi-agent release: separation of authority, not two prompts chatting The AKS promotion separates Forge into Builder and Reviewer roles: Control Builder Reviewer Tokens per request 2048 512 Daily tokens 8192 2048 Tool authority Read, search, patch, test, diff Read diff/evidence and submit a decision Approval mode Never Always Trust threshold 700 800 The Builder may create a patch but cannot approve release. The Reviewer may approve a digest-pinned Builder artifact but cannot change source or approve an artifact created by the Reviewer. The final handoff carries patch, test-evidence, and artifact-manifest digests and still stops for human PR approval. The code/08 final sandbox-escape gate explicitly rejects: self-modified authority; symlink escape; host trust handoff; DNS egress; Builder self-approval; Reviewer source modification; untrusted or expired peer drafts. The release standard therefore becomes: Correct behavior must pass, and credential, network, tool, approval, evidence, and containment boundaries must remain intact. Passing tests alone is not release evidence. 11. What kars does not promise kars strengthens the sandbox, but it does not solve every risk automatically: It does not prove that a generated patch is correct. It does not make untrusted code safe to merge. It cannot protect a credential mistakenly mounted into the agent. Local Docker mode does not become a production boundary. It does not replace tenant RBAC, quotas, image policy, signing, supply-chain controls, or durable audit export. It cannot compensate for a policy that deliberately enables arbitrary shell and unrestricted egress. Confidential isolation does not replace least privilege, tool policy, egress policy, and code review. The sandbox bounds authority and blast radius. Tests, evaluation, independent review, and release policy still determine whether a change is acceptable. 12. An enterprise adoption path with measurable exits Phase 1: define the business and threat contract Specify inputs, outputs, allowed actions, forbidden actions, data boundaries, and the human approval point. Exit: product, platform, and security can all explain the agent's maximum authority. Phase 2: validate one OpenClaw vertical slice Use narrow business MCP tools instead of a general shell, and include hostile repository content. Exit: the normal task succeeds while self-configuration, path/symlink, trust-handoff, and egress tests fail. Phase 3: encode the sandbox as a Kubernetes contract Validate UID separation, root filesystem, capabilities, volumes, service-account tokens, NetworkPolicy, egress guard, and exec admission. Exit: the five boundaries are supported by runtime evidence, not only YAML review. Phase 4: add tool, model, and cost governance Apply McpServer, ToolPolicy, and InferencePolicy. Test unknown tools, dangerous arguments, and token overflow. Exit: violations are denied on the live request path. Phase 5: migrate into explicit MAF code Encode workflow state, typed tools, loop limits, evidence, failure paths, and the human stop. Exit: the MAF runtime preserves the external boundary already proven around the OpenClaw prototype. Phase 6: promote to AKS through GitOps Pin source revision, image digest, and loaded policy digest. Separate Builder and Reviewer authority. Prepare the kill switch, rollback, and durable audit export. Exit: one allowed workflow succeeds, multiple escape and authority-violation scenarios are denied, and all results have correlated evidence. Conclusion: kars does not make the model smarter; it makes agent authority explainable Enterprises will ultimately ask: What can the agent access? Where are the provider credentials? Who defines and changes the tool authority? Can prompt injection move data through DNS, metadata, a daemon, or HTTPS? How many tokens and repair iterations may one task consume? Who may patch, approve, merge, or deploy? Does evidence survive pod loss? If OpenClaw is replaced by MAF, does the security model remain intact? The ByteCraft AI story does not argue for one universal agent framework. It argues for a stable Agent Infrastructure layer: Use OpenClaw to discover valuable behavior quickly, use Microsoft Agent Framework to encode that behavior as explicit and testable application code, and use kars to remove credentials, networking, tools, budgets, sandboxing, audit, and lifecycle authority from the agent application itself. An agent becomes an enterprise workload when it has an independent identity boundary, a budget, a constrained tool surface, controlled egress, exportable evidence, and operational suspension and rollback—not merely when it runs inside a container. References Let's Learn Microsoft kars Microsoft karsBuilding HIPAA-Compliant Medical Transcription with Local AI
Building HIPAA-Compliant Medical Transcription with Local AI Introduction Healthcare organizations generate vast amounts of spoken content, patient consultations, research interviews, clinical notes, medical conferences. Transcribing these recordings traditionally requires either manual typing (time-consuming and expensive) or cloud transcription services (creating immediate HIPAA compliance concerns). Every audio file sent to external APIs exposes Protected Health Information (PHI), requires Business Associate Agreements, creates audit trails on third-party servers, and introduces potential breach vectors. This sample solution lies in on-premises voice-to-text systems that process audio entirely locally, never sending PHI beyond organizational boundaries. This article demonstrates building a sample medical transcription application using FLWhisper, ASP.NET Core, C#, and Microsoft Foundry Local with OpenAI Whisper models. You'll learn how to build sample HIPAA-compliant audio processing, integrate Whisper models for medical terminology accuracy, design privacy-first API patterns, and build responsive web UIs for healthcare workflows. Whether you're developing electronic health record (EHR) integrations, building clinical research platforms, or implementing dictation systems for medical practices, this sample could be a great starting point for privacy-first speech recognition. Why Local Transcription Is Critical for Healthcare Healthcare data handling is fundamentally different from general business data due to HIPAA regulations, state privacy laws, and professional ethics obligations. Understanding these requirements explains why cloud transcription services, despite their convenience, create unacceptable risks for medical applications. HIPAA compliance mandates strict controls over PHI. Every system that touches patient data must implement administrative, physical, and technical safeguards. Cloud transcription APIs require Business Associate Agreements (BAAs), but even with paperwork, you're entrusting PHI to external systems. Every API call creates logs on vendor servers, potentially in multiple jurisdictions. Data breaches at transcription vendors expose patient information, creating liability for healthcare organizations. On-premises processing eliminates these third-party risks entirely, PHI never leaves your controlled environment. US State laws increasingly add requirements beyond HIPAA. California's CCPA, New York's SHIELD Act, and similar legislation create additional compliance obligations. International regulations like GDPR prohibit transferring health data outside approved jurisdictions. Local processing simplifies compliance by keeping data within organizational boundaries. Research applications face even stricter requirements. Institutional Review Boards (IRBs) often require explicit consent for data sharing with external parties. Cloud transcription may violate study protocols that promise "no third-party data sharing." Clinical trials in pharmaceutical development handle proprietary information alongside PHI, double jeopardy for data exposure. Local transcription maintains research integrity while enabling audio analysis. Cost considerations favor local deployment at scale. Medical organizations generate substantial audio, thousands of patient encounters monthly. Cloud APIs charge per minute of audio, creating significant recurring costs. Local models have fixed infrastructure costs that scale economically. A modest GPU server can process hundreds of hours monthly at predictable expense. Latency matters for clinical workflows. Doctors and nurses need transcriptions available immediately after patient encounters to review and edit while details are fresh. Cloud APIs introduce network delays, especially problematic in rural health facilities with limited connectivity. Local inference provides <1 second turnaround for typical consultation lengths. Application Architecture: ASP.NET Core with Foundry Local The sample FLWhisper application implements clean separation between audio handling, AI inference, and state management using modern .NET patterns: The ASP.NET Core 10 minimal API provides HTTP endpoints for health checks, audio transcription, and sample file streaming. Minimal APIs reduce boilerplate while maintaining full middleware support for error handling, authentication, and CORS. The API design follows OpenAI's transcription endpoint specification, enabling drop-in replacement for existing integrations. The service layer encapsulates business logic: FoundryModelService manages model loading and lifetime, TranscriptionService handles audio processing and AI inference, and SampleAudioService provides demonstration files for testing. This separation enables easy testing, dependency injection, and service swapping. Foundry Local integration uses the Microsoft.AI.Foundry.Local.WinML SDK. Unlike cloud APIs requiring authentication and network calls, this SDK communicates directly with the local Foundry service via in-process calls. Models load once at startup, remaining resident in memory for sub-second inference on subsequent requests. The static file frontend delivers vanilla HTML/CSS/JavaScript, no framework overhead. This simplicity aids healthcare IT security audits and enables deployment on locked-down hospital networks. The UI provides file upload, sample selection, audio preview, transcription requests, and result display with copy-to-clipboard functionality. Here's the architectural flow for transcription requests: Web UI (Upload Audio File) ↓ POST /v1/audio/transcriptions (Multipart Form Data) ↓ ASP.NET Core API Route ↓ TranscriptionService.TranscribeAudio(audioStream) ↓ Foundry Local Model (Whisper Medium locally) ↓ Text Result + Metadata (language, duration) ↓ Return JSON/Text Response ↓ Display in UI This architecture embodies several healthcare system design principles: Data never leaves the device: All processing occurs on-premises, no external API calls No data persistence by default: Audio and transcripts are session-only, never saved unless explicitly configured Comprehensive health checks: System readiness verification before accepting PHI Audit logging support: Structured logging for compliance documentation Graceful degradation: Clear error messages when models unavailable rather than silent failures Setting Up Foundry Local with Whisper Models Foundry Local supports multiple Whisper model sizes, each with different accuracy/speed tradeoffs. For medical transcription, accuracy is paramount—misheard drug names or dosages create patient safety risks: # Install Foundry Local (Windows) winget install Microsoft.FoundryLocal # Verify installation foundry --version # Download Whisper Medium model (optimal for medical accuracy) foundry model add openai-whisper-medium-generic-cpu:1 # Check model availability foundry model list Whisper Medium (769M parameters) provides the best balance for medical use. Smaller models (Tiny, Base) miss medical terminology frequently. Larger models (Large) offer marginal accuracy gains at 3x inference time. Medium handles medical vocabulary well, drug names, anatomical terms, procedure names, while processing typical consultation audio (5-10 minutes) in under 30 seconds. The application detects and loads the model automatically: // Services/FoundryModelService.cs using Microsoft.AI.Foundry.Local.WinML; public class FoundryModelService { private readonly ILogger _logger; private readonly FoundryOptions _options; private ILocalAIModel? _loadedModel; public FoundryModelService( ILogger logger, IOptions options) { _logger = logger; _options = options.Value; } public async Task InitializeModelAsync() { try { _logger.LogInformation( "Loading Foundry model: {ModelAlias}", _options.ModelAlias ); // Load model from Foundry Local _loadedModel = await FoundryClient.LoadModelAsync( modelAlias: _options.ModelAlias, cancellationToken: CancellationToken.None ); if (_loadedModel == null) { _logger.LogWarning("Model loaded but returned null instance"); return false; } _logger.LogInformation( "Successfully loaded model: {ModelAlias}", _options.ModelAlias ); return true; } catch (Exception ex) { _logger.LogError( ex, "Failed to load Foundry model: {ModelAlias}", _options.ModelAlias ); return false; } } public ILocalAIModel? GetLoadedModel() => _loadedModel; public async Task UnloadModelAsync() { if (_loadedModel != null) { await FoundryClient.UnloadModelAsync(_loadedModel); _loadedModel = null; _logger.LogInformation("Model unloaded"); } } } Configuration lives in appsettings.json , enabling easy customization without code changes: { "Foundry": { "ModelAlias": "whisper-medium", "LogLevel": "Information" }, "Transcription": { "MaxAudioDurationSeconds": 300, "SupportedFormats": ["wav", "mp3", "m4a", "flac"], "DefaultLanguage": "en" } } Implementing Privacy-First Transcription Service The transcription service handles audio processing while maintaining strict privacy controls. No audio or transcript persists beyond the HTTP request lifecycle unless explicitly configured: // Services/TranscriptionService.cs public class TranscriptionService { private readonly FoundryModelService _modelService; private readonly ILogger _logger; public async Task TranscribeAudioAsync( Stream audioStream, string originalFileName, TranscriptionOptions? options = null) { options ??= new TranscriptionOptions(); var startTime = DateTime.UtcNow; try { // Validate audio format ValidateAudioFormat(originalFileName); // Get loaded model var model = _modelService.GetLoadedModel(); if (model == null) { throw new InvalidOperationException("Whisper model not loaded"); } // Create temporary file (automatically deleted after transcription) using var tempFile = new TempAudioFile(audioStream); // Execute transcription _logger.LogInformation( "Starting transcription for file: {FileName}", originalFileName ); var transcription = await model.TranscribeAsync( audioFilePath: tempFile.Path, language: options.Language, cancellationToken: CancellationToken.None ); var duration = (DateTime.UtcNow - startTime).TotalSeconds; _logger.LogInformation( "Transcription completed in {Duration:F2}s", duration ); return new TranscriptionResult { Text = transcription.Text, Language = transcription.Language ?? options.Language, Duration = transcription.AudioDuration, ProcessingTimeSeconds = duration, FileName = originalFileName, Timestamp = DateTime.UtcNow }; } catch (Exception ex) { _logger.LogError( ex, "Transcription failed for file: {FileName}", originalFileName ); throw; } } private void ValidateAudioFormat(string fileName) { var extension = Path.GetExtension(fileName).TrimStart('.'); var supportedFormats = new[] { "wav", "mp3", "m4a", "flac", "ogg" }; if (!supportedFormats.Contains(extension.ToLowerInvariant())) { throw new ArgumentException( $"Unsupported audio format: {extension}. " + $"Supported: {string.Join(", ", supportedFormats)}" ); } } } // Temporary file wrapper that auto-deletes internal class TempAudioFile : IDisposable { public string Path { get; } public TempAudioFile(Stream sourceStream) { Path = System.IO.Path.GetTempFileName(); using var fileStream = File.OpenWrite(Path); sourceStream.CopyTo(fileStream); } public void Dispose() { try { if (File.Exists(Path)) { File.Delete(Path); } } catch { // Ignore deletion errors in temp folder } } } This service demonstrates several privacy-first patterns: Temporary file lifecycle management: Audio written to temp storage, automatically deleted after transcription No implicit persistence: Results returned to caller, not saved by service Format validation: Accept only supported audio formats to prevent processing errors Comprehensive logging: Audit trail for compliance without logging PHI content Error isolation: Exceptions contain diagnostic info but no patient data Building the OpenAI-Compatible REST API The API endpoint mirrors OpenAI's transcription API specification, enabling existing integrations to work without modifications: // Program.cs var builder = WebApplication.CreateBuilder(args); // Configure services builder.Services.Configure( builder.Configuration.GetSection("Foundry") ); builder.Services.AddSingleton(); builder.Services.AddScoped(); builder.Services.AddHealthChecks() .AddCheck("foundry-health"); var app = builder.Build(); // Load model at startup var modelService = app.Services.GetRequiredService(); await modelService.InitializeModelAsync(); app.UseHealthChecks("/health"); app.MapHealthChecks("/api/health/status"); // OpenAI-compatible transcription endpoint app.MapPost("/v1/audio/transcriptions", async ( HttpRequest request, TranscriptionService transcriptionService, ILogger logger) => { if (!request.HasFormContentType) { return Results.BadRequest(new { error = "Content-Type must be multipart/form-data" }); } var form = await request.ReadFormAsync(); // Extract audio file var audioFile = form.Files.GetFile("file"); if (audioFile == null || audioFile.Length == 0) { return Results.BadRequest(new { error = "Audio file required in 'file' field" }); } // Parse options var format = form["format"].ToString() ?? "text"; var language = form["language"].ToString() ?? "en"; try { // Process transcription using var stream = audioFile.OpenReadStream(); var result = await transcriptionService.TranscribeAudioAsync( audioStream: stream, originalFileName: audioFile.FileName, options: new TranscriptionOptions { Language = language } ); // Return in requested format if (format == "json") { return Results.Json(new { text = result.Text, language = result.Language, duration = result.Duration }); } else { // Default: plain text return Results.Text(result.Text); } } catch (Exception ex) { logger.LogError(ex, "Transcription request failed"); return Results.StatusCode(500); } }) .DisableAntiforgery() // File uploads need CSRF exemption .WithName("TranscribeAudio") .WithOpenApi(); app.Run(); Example API usage: # PowerShell $audioFile = Get-Item "consultation-recording.wav" $response = Invoke-RestMethod ` -Uri "http://localhost:5192/v1/audio/transcriptions" ` -Method Post ` -Form @{ file = $audioFile; format = "json" } Write-Output $response.text # cURL curl -X POST http://localhost:5192/v1/audio/transcriptions \ -F "file=@consultation-recording.wav" \ -F "format=json" Building the Interactive Web Frontend The web UI provides a user-friendly interface for non-technical medical staff to transcribe recordings: SarahCare Medical Transcription The JavaScript handles file uploads and API interactions: // wwwroot/app.js let selectedFile = null; async function checkHealth() { try { const response = await fetch('/health'); const statusEl = document.getElementById('status'); if (response.ok) { statusEl.className = 'status-badge online'; statusEl.textContent = '✓ System Ready'; } else { statusEl.className = 'status-badge offline'; statusEl.textContent = '✗ System Unavailable'; } } catch (error) { console.error('Health check failed:', error); } } function handleFileSelect(event) { const file = event.target.files[0]; if (!file) return; selectedFile = file; // Show file info const fileInfo = document.getElementById('fileInfo'); fileInfo.textContent = `Selected: ${file.name} (${formatFileSize(file.size)})`; fileInfo.classList.remove('hidden'); // Enable audio preview const preview = document.getElementById('audioPreview'); preview.src = URL.createObjectURL(file); preview.classList.remove('hidden'); // Enable transcribe button document.getElementById('transcribeBtn').disabled = false; } async function transcribeAudio() { if (!selectedFile) return; const loadingEl = document.getElementById('loadingIndicator'); const resultEl = document.getElementById('resultSection'); const transcribeBtn = document.getElementById('transcribeBtn'); // Show loading state loadingEl.classList.remove('hidden'); resultEl.classList.add('hidden'); transcribeBtn.disabled = true; try { const formData = new FormData(); formData.append('file', selectedFile); formData.append('format', 'json'); const startTime = Date.now(); const response = await fetch('/v1/audio/transcriptions', { method: 'POST', body: formData }); if (!response.ok) { throw new Error(`HTTP ${response.status}: ${response.statusText}`); } const result = await response.json(); const processingTime = ((Date.now() - startTime) / 1000).toFixed(1); // Display results document.getElementById('transcriptionText').value = result.text; document.getElementById('resultDuration').textContent = `Duration: ${result.duration.toFixed(1)}s`; document.getElementById('resultLanguage').textContent = `Language: ${result.language}`; resultEl.classList.remove('hidden'); console.log(`Transcription completed in ${processingTime}s`); } catch (error) { console.error('Transcription failed:', error); alert(`Transcription failed: ${error.message}`); } finally { loadingEl.classList.add('hidden'); transcribeBtn.disabled = false; } } function copyToClipboard() { const text = document.getElementById('transcriptionText').value; navigator.clipboard.writeText(text) .then(() => alert('Copied to clipboard')) .catch(err => console.error('Copy failed:', err)); } // Initialize window.addEventListener('load', () => { checkHealth(); loadSamplesList(); }); Key Takeaways and Production Considerations Building HIPAA-compliant voice-to-text systems requires architectural decisions that prioritize data privacy over convenience. The FLWhisper application demonstrates that you can achieve accurate medical transcription, fast processing times, and intuitive user experiences entirely on-premises. Critical lessons for healthcare AI: Privacy by architecture: Design systems where PHI never exists outside controlled environments, not as a configuration option No persistence by default: Audio and transcripts should be ephemeral unless explicitly saved with proper access controls Model selection matters: Whisper Medium provides medical terminology accuracy that smaller models miss Health checks enable reliability: Systems should verify model availability before accepting PHI Audit logging without content logging: Track operations for compliance without storing sensitive data in logs For production deployment in clinical settings, integrate with EHR systems via HL7/FHIR interfaces. Implement role-based access control with Active Directory integration. Add digital signatures for transcript authentication. Configure automatic PHI redaction using clinical NLP models. Deploy on HIPAA-compliant infrastructure with proper physical security. Implement comprehensive audit logging meeting compliance requirements. The complete implementation with ASP.NET Core API, Foundry Local integration, sample audio files, and comprehensive tests is available at github.com/leestott/FLWhisper. Clone the repository and follow the setup guide to experience privacy-first medical transcription. Resources and Further Reading FLWhisper Repository - Complete C# implementation with .NET 10 Quick Start Guide - Installation and usage instructions Microsoft Foundry Local Documentation - SDK reference and model catalog OpenAI Whisper Documentation - Model architecture and capabilities HIPAA Compliance Guidelines - HHS official guidance Testing Guide - Comprehensive test suite documentationBuilding and Deploying Microsoft Hosted Agents to Microsoft Teams
A practical, engineer-to-engineer guide to taking an AI agent from a developer laptop, into Microsoft Foundry Agent Service, and out to end users inside Microsoft Teams and Microsoft 365 — using the BRK241 FibreOps reference implementation as a worked example. Introduction: the hard part is no longer building the agent Two years ago, wiring an LLM to a couple of tools felt like the summit. It isn't any more. Frameworks, hosted models, and function-calling have made the build step almost routine. The problem has quietly moved downstream. The genuinely hard questions today are operational: Where does the agent run when it's no longer on your machine? What identity does it use to call enterprise systems, and who granted it? How does a platform team scale, monitor, and roll it back? How do business users actually reach it without learning a new tool? Who signed off on it touching production data? A prototype answers none of these. A production agent platform answers all of them, repeatably, for every agent an organisation ships. That shift — from a clever notebook to a governed, observable service that lands in the tools people already use — is the subject of this article. We'll use a single narrative to keep it concrete: FibreOps, the BRK241 "Autonomous Fibre Outage Response" system. It ingests optical line terminal (OLT) telemetry, analyses incidents, files tickets in Dynamics 365 Field Service, posts Adaptive Cards to Microsoft Teams, and dispatches engineers — all through role-specialised agents. The full source is on GitHub. The story runs on three verbs: Build → Run → Distribute. Section 1: Building the agent An agent is not one mega-prompt. FibreOps is deliberately factored into three role-specialised agents behind a single orchestrator, each with its own tool surface, its own system instructions, and a strict output contract: IncidentAnalysisAgent — classifies severity, finds probable cause, and pulls the correct standard operating procedure (SOP). NetOpsCoordinatorAgent — files the D365 incident and posts the Teams outage notice. FieldDispatchAgent — selects the best engineer by skill, region and shift, books the resource, and updates Teams. The Coordinator hands off to Dispatch with a literal HANDOFF:DISPATCH token rather than a fuzzy "I think we should…". Hard contracts between agents are how you stop them inventing work. Microsoft Agent Framework The agents are built with the Microsoft Agent Framework (MAF). The key design decision in the reference implementation is that all three backends honour one contract — await agent.run(prompt) -> response — so the orchestrator never knows or cares where reasoning actually happens: local — a deterministic LocalAgent shim with no LLM, so the demo runs with zero Azure credentials. foundry — agent_framework.Agent + FoundryChatClient , definition resolved locally. Ideal while iterating on prompts. hosted — agent_framework_foundry.FoundryAgent bound to a Prompt Agent published to Foundry Agent Service. This is the production path. Building a Foundry-backed agent is just a client plus instructions plus typed tools: from agent_framework import Agent from agent_framework_foundry import FoundryChatClient from azure.identity import DefaultAzureCredential client = FoundryChatClient( project_endpoint=settings.azure_ai_project_endpoint, model=settings.azure_ai_model_deployment, # e.g. gpt-4.1-mini credential=DefaultAzureCredential(), # no connection strings, ever ) agent = Agent( client=client, instructions=INCIDENT_ANALYSIS_INSTRUCTIONS_V1, name="IncidentAnalysisAgent", tools=[lookup_sop, recall, remember, web_iq_search, work_iq_search], ) Note the DefaultAzureCredential . There are no keys or connection strings anywhere in the reasoning path — identity flows from Microsoft Entra ID. Keep that in mind; it becomes the backbone of the governance story later. Tool calling and MCP Every tool is a typed Python function. Foundry sees the JSON schema derived from the signature; the runtime executes the Python. That separation matters: the published agent definition stores only the model and instructions, while the implementations are supplied by the runtime on every call. The same in-process tools (Teams, D365, dispatch, knowledge, memory) run identically whether the agent is local or hosted. Beyond your own functions, Foundry agents can draw on hosted toolbox tools ( web_search , code_interpreter ) and Model Context Protocol (MCP) servers. MCP is the open standard for exposing tools, resources and prompts to agents over a uniform protocol, so an enterprise can stand up an MCP server once and let every agent consume it. In FibreOps this is config-gated — set FIBREOPS_FOUNDRY_TOOLBOX=1 and the incident analyst gains live web search alongside its Web IQ / Work IQ connectors, with no code change. Grounding strategies FibreOps grounds reasoning three ways, in layers: Retrieval over owned knowledge — SOPs (markdown) and the fibre-node topology graph, looked up by the analysis agent. Foundry IQ — Web IQ for public context (roadworks, weather, power) and Work IQ for enterprise context (site surveys, SLA tiers, competency matrix). Procedural memory — prior incidents for a node, recalled before analysis so the agent learns from history. Crucially, when the IQ endpoints are unset the tools fall back to deterministic fixtures so the agent always grounds. Grounding that silently fails is worse than no grounding; design your fallbacks explicitly. Local development, testing and evaluation The whole system runs from one command with no cloud dependency: # Deterministic local backend — no Azure credentials required python -m fibreops.demo --signals 3 --backend local Every run is persisted as a JSON document — the input signal, every agent step, every tool call, every output, every ticket. That single artefact shape feeds three consumers: structured logs, the local optimiser, and Foundry Evaluators. The optimiser scores each run against a five-criterion rubric (was the analysis complete, was severity consistent with customer impact, did a ticket land, did dispatch policy match severity, was an SOP cited) and writes back concrete improvement suggestions. That evaluation loop — not the first working demo — is what turns a prototype into a system you can keep improving. Section 2: Deploying to Microsoft Foundry Agent Service Microsoft Foundry Agent Service is the managed runtime that hosts your agents. It gives you a secure, isolated execution environment, an agent runtime that speaks the OpenAI-compatible Responses API, plus hosted memory, toolboxes, knowledge integrations, and observability — without you operating any of it. FibreOps demonstrates the two hosting shapes Foundry offers. Shape 1 — Prompt Agents A Prompt Agent stores a model deployment plus system instructions as an immutable, versioned definition in Foundry. Publishing is a one-time step per change: from azure.ai.projects import AIProjectClient from azure.ai.projects.models import PromptAgentDefinition from azure.identity import DefaultAzureCredential pc = AIProjectClient(endpoint=endpoint, credential=DefaultAzureCredential(), allow_preview=True) pc.agents.create_version( agent_name="fibreops-incident-analysis", definition=PromptAgentDefinition( model=model_deployment, instructions=INCIDENT_ANALYSIS_INSTRUCTIONS_V1, ), description="FibreOps incident analysis agent", ) At run time you bind to the published version with a FoundryAgent , and — as noted above — the runtime supplies the tool implementations. Prompt versioning ( instructions_v1 , _v2 , _v3 ) is where the optimiser's suggestions land, closing the improvement loop inside the platform. Shape 2 — Containerised hosted agents The BRK241 hero path packages the entire analyse → coordinate → dispatch flow as a single hosted agent: a container that serves the Responses /responses contract on port 8088, deployed straight into your Foundry project. The Agent Framework agent is wrapped by ResponsesHostServer : from agent_framework_foundry_hosting import ResponsesHostServer def main() -> None: server = ResponsesHostServer(build_system_agent()) # Foundry sets the reserved PORT env var inside the sandbox server.run(host="0.0.0.0", port=8088) The container is declared in agent.yaml — kind: hosted , the image reference, the per-session sandbox size (0.5/1 Gi, 1/2 Gi or 2/4 Gi), the protocol version, and only user-declared environment variables. You never hard-code FOUNDRY_* values or the Application Insights connection string; the platform injects those at run time. Deployment registers the image as an immutable version and polls until active : details = pc.agents.create_version( agent_name="fibreops-outage-response", definition=HostedAgentDefinition( protocol_versions=[ProtocolVersionRecord( protocol=AgentProtocol.RESPONSES, version="1.0.0")], cpu="1", memory="2Gi", container_configuration=ContainerConfiguration(image=image), environment_variables={"MODEL_DEPLOYMENT_NAME": model_deployment}, ), ) From local execution to managed hosting The migration path is deliberately gentle because the contract never changes. A developer iterates locally against LocalAgent , moves to the foundry backend to test real prompts, then publish es Prompt Agents or builds and deploy-hosted s the container. The orchestrator code is byte-for-byte identical across all three. That property — same code path local for dev, hosted in Foundry for prod — is the single most important thing to preserve when designing your own agents. Scaling, memory, toolboxes, knowledge and observability Scaling — Foundry provisions a per-session sandbox and a dedicated Entra agent identity per hosted-agent version; you size the sandbox in agent.yaml and let the platform handle isolation. Memory — set FOUNDRY_MEMORY_STORE_NAME and a FoundryMemoryProvider is attached as a context provider so agents read and write learned procedures in Foundry's hosted store; unset, they use local SQLite. No code change. Toolboxes & knowledge — hosted web_search , code interpreter, MCP, and Web/Work IQ connectors are curated per role and merged with your Python tools. Observability — the agent emits OpenTelemetry spans; set APPLICATIONINSIGHTS_CONNECTION_STRING (injected by the platform for hosted agents) and every agent decision, tool call and latency is queryable in Application Insights. Section 3: IT and development responsibilities Successful agent deployments need both developer velocity and platform governance. The failure mode at either extreme is familiar: developers who can't ship because every request routes through a ticket queue, or a free-for-all where nobody can say what identity an agent runs as. The workable model draws a clean line of responsibility. Concern Developer / Agent team IT / Platform team Identity Use DefaultAzureCredential ; never embed secrets; declare the scopes the agent needs Provision the managed / Entra agent identity; own the app registration and consent Access control Request least-privilege roles for the tools the agent calls Grant RBAC at the correct scope; run role-assignment scripts; enforce approvals Security Validate inputs, handle tool failures cleanly, avoid data exfiltration in prompts Disable ACR admin, enforce managed-identity pulls, network controls, Key Vault for secrets Compliance Keep decisions explainable and replayable (the JSON run record) Data-residency, retention, audit, Responsible AI review sign-off Monitoring Emit structured traces + OTel spans; define the rubric Own Application Insights / Log Analytics, alerting, dashboards, SLOs Cost Right-size the sandbox and model deployment; cache grounding Budgets, quota, token-consumption monitoring, chargeback Lifecycle Version prompts and images; feed the optimiser back into new versions Environment promotion (dev → test → prod), rollback, deprecation The reference implementation encodes this split honestly. The Bicep template does not create role assignments, because most deployers only hold Contributor . Instead a subscription Owner runs scripts/grant-mi-roles.ps1 once to grant the App Service's identity exactly the roles it needs — Event Hubs Data Owner, Key Vault Secrets User, AcrPull, Azure AI Developer, and Cognitive Services OpenAI User — and no more. That is least privilege made operational. Section 4: Publishing to Microsoft Teams and Microsoft 365 An agent nobody can reach has no value. The final verb — Distribute — puts the agent where users already work. FibreOps reaches Teams two ways. The lightweight path: Adaptive Cards via Incoming Webhook The NetOps coordinator posts outage notices and status updates to a Teams channel as Adaptive Cards through an Incoming Webhook. Any unconfigured channel is logged to state/teams_outbox.jsonl , so the same code runs in a demo and in production — you only change the webhook target. This is the fastest way to get agent output into Teams and is ideal for notifications and human-in-the-loop review. The rich path: a declarative agent for Microsoft 365 Copilot To make the agent conversational and discoverable across Teams, Microsoft 365 Copilot and copilot.microsoft.com, FibreOps ships as a declarative agent plus an API plugin action. One command builds the sideload-ready package: python -m fibreops.demo publish-m365 --out dist/m365 # wrote declarativeAgent.json (name, description, conversation starters) # wrote fibreops-action.json (API plugin -> {base_url}/openapi.json) # wrote manifest.json (Teams app manifest) # wrote color.png / outline.png (icons) # wrote fibreops-copilot.zip (upload this) The declarative agent declares metadata, conversation starters and a capability set; the action plugin proxies tool calls to the deployed FastAPI app via its OpenAPI document. Set M365_ACTION_BASE_URL to the app's public HTTPS root before publishing — the CLI warns when the placeholder is still in effect. That single environment variable is the only thing that flips the package from demo to production. The end-to-end distribution workflow Conceptually, the artefact travels a fixed pipeline: Developer laptop │ build + test (local backend) → publish Prompt Agent / deploy hosted container ▼ Microsoft Foundry Agent Service │ hosted agent, secure sandbox, Entra agent identity, observability ▼ Teams App package (fibreops-copilot.zip) │ Teams Admin Center → Manage apps → Upload (or M365 Admin Center → Integrated apps) ▼ Microsoft 365 tenant │ admin approval, availability policy, targeted rollout ▼ End user in Teams / M365 Copilot Enterprise rollout is rarely "publish to everyone". The realistic pattern is a staged one: sideload to a pilot group, gather feedback and optimiser scores, then widen availability through Teams app-permission and app-setup policies to department, then tenant. Because the package carries publisher metadata and the declarative schema, IT can review it exactly like any other line-of-business app. Section 5: Enterprise governance Governance is not a bolt-on; in this architecture it's a property of the platform. The pillars: Entra ID integration and agent identity — every hosted agent version gets a dedicated Entra agent identity. Nothing authenticates with a shared key. DefaultAzureCredential means the same code picks up a developer's identity locally and the managed identity in production. RBAC at the right scope — roles are granted to identities, not baked into images. Deploying a hosted agent requires Azure AI Project Manager at project scope; the Foundry project identity needs AcrPull on the registry to pull the container. Least privilege is enforced, not assumed. Auditability — the JSON run record plus OpenTelemetry spans in Application Insights give you a replayable, per-incident audit trail. You can reconstruct exactly which SOP was cited, which engineer was chosen, and why severity was escalated. Data boundaries — the mock D365 is a drop-in for a real Dataverse environment; grounding sources are enterprise connectors (Work IQ) kept inside the tenant boundary. Nothing leaves the subscription without an explicit connector. Responsible AI — the Adaptive Card JSON can be pasted into the Adaptive Cards designer for governance review; the evaluation rubric makes quality measurable; explicit grounding fallbacks prevent silent failure. Production readiness — immutable versioning, one-command rollback (delete a version), managed-identity-only image pulls, and disabled ACR admin credentials are all first-class in the reference deployment. Section 6: Reference architecture The following diagram shows the production topology — users on the left, enterprise systems and controls on the right, with Foundry Agent Service at the centre hosting the agent. flowchart LR User["NOC operator / business user"] subgraph M365["Microsoft 365 tenant"] Teams["Microsoft Teams(Adaptive Cards + declarative agent)"] Copilot["Microsoft 365 Copilot"] end subgraph Foundry["Microsoft Foundry Agent Service"] Hosted["Hosted AgentOutage Response System(secure per-session sandbox)"] Runtime["Agent runtime(Responses API)"] Memory["Hosted memory + toolboxes"] end subgraph Enterprise["Enterprise data & tools"] MCP["MCP servers / web_search"] D365["Dynamics 365 Field Service"] EventHub["Azure Event Hubs(OLT telemetry)"] Knowledge["SOPs + topology + Web/Work IQ"] end subgraph Ops["Cross-cutting"] Obs["ObservabilityApp Insights / OTel"] Gov["GovernanceEntra ID · RBAC · audit"] end User --> Teams User --> Copilot Teams --> Runtime Copilot --> Runtime Runtime --> Hosted Hosted --> Memory Hosted --> MCP Hosted --> Knowledge Hosted --> D365 EventHub --> Hosted Hosted -.->|Adaptive Cards| Teams Hosted --> Obs Gov -.->|identity & policy| Foundry Gov -.->|identity & policy| Enterprise Read the solid arrows as the control/orchestration flow and the dashed arrows as governance and outbound notifications. The point of the diagram is that governance (Entra ID, RBAC, audit) applies across every component, and observability captures every agent decision — neither is optional plumbing. Section 7: What production looks like Picture the FibreOps rollout at a national fibre operator, with the four personas doing their part: Developers build the three agents and the orchestrator on their laptops against the local backend — no cloud, no credentials, deterministic tests. They tune prompts against the foundry backend, watch the optimiser rubric climb from 0.90 to 1.0 as they add the ">5,000 customers ⇒ escalate to critical" rule, and commit a new instruction version. The platform team deploys the container to Foundry Agent Service via scripts/deploy-hosted-agent.ps1 , which builds the image in ACR, pushes it, and registers an immutable version. They provision the Event Hub, Key Vault, Log Analytics and Application Insights from Bicep, and size the sandbox at 1 vCPU / 2 GiB. IT approves the workload: a subscription Owner grants the managed identity its five least-privilege roles, hardens the App Service to pull via managed identity, disables ACR admin, and signs off the Responsible AI review using the replayable run records and the Adaptive Card previews. They sideload fibreops-copilot.zip to a pilot channel first. Business users consume it inside Teams. When an OLT in London loses light, an Adaptive Card appears in the NOC channel within seconds — severity, probable cause, ticket ID, and the dispatched engineer's ETA — with no human having read a dashboard, opened a ticket, or phoned a dispatcher. If Foundry ever wobbles, the same system falls back to the deterministic local agent with an identical trace shape. Every integration but D365 is live in the demo, and D365 is a one-variable swap to a real Dataverse endpoint. That is the whole point: the demo and production differ by configuration, not by code. Key takeaways Design for one contract. If agent.run(prompt) behaves identically local, foundry-backed and hosted, migration to production is configuration, not a rewrite. Factor agents by role with hard handoff contracts. Literal tokens like HANDOFF:DISPATCH beat fuzzy natural-language handoffs and stop agents inventing work. Never embed secrets. DefaultAzureCredential + Entra agent identities give you keyless auth that works the same everywhere. Make every run replayable. A single JSON artefact that feeds logs, evaluation and audit is worth more than any dashboard. Ground explicitly, and design your fallbacks. Grounding that fails silently is a liability; deterministic fixtures keep the agent honest. Split responsibility cleanly. Developers own velocity and quality; the platform team owns identity, scale, cost and promotion. Encode the split in scripts, not tribal knowledge. Version prompts and images immutably. Rollback should be "delete a version", and the optimiser's suggestions should land as the next version. Distribute where users already are. Adaptive Cards for notifications, a declarative agent for conversation and discovery across Teams and M365 Copilot. Roll out in stages. Pilot channel → department → tenant, gated by app policies and real optimiser scores. Resources Reference implementation: github.com/leestott/BRK241-frontier Microsoft Agent Framework overview Microsoft Foundry Agent Service Hosted agents in Foundry Agent Service · Deploy a hosted agent Microsoft Teams developer platform Declarative agents for Microsoft 365 Copilot Model Context Protocol GitHub Copilot Clone the repo, run python -m fibreops.demo --signals 3 --backend local , and watch the analyse → coordinate → dispatch loop close. Then wire in your own Foundry project and take it all the way to Teams. Go build something.Your Entire Agentic AI Workflow, Now Inside VS Code: New Course Available
If you build with AI, you know the tax: jumping between a browser portal to pick a model, a terminal to run it, a separate playground to test a prompt, and finally your editor to write the code. Every context switch is a small drain on focus—and it adds up. What if the whole loop lived in the one tool you never leave? That's the promise of the Foundry Toolkit for Visual Studio Code, and it's exactly what our new VS Code Learn: Foundry Toolkit video series is here to show youBuilding Agents in Production with Toolbox, Skills, and Tool Search
If you are shipping AI agents beyond a demo, you have felt the pain: every agent needs the same tools, each with its own authentication, and the tool list keeps growing until your prompt is bloated and the model picks the wrong one. On 22 July 2026 at 5:00 PM BST, the Microsoft Foundry community is running a 40-minute Discord round table to talk about exactly this, and to gather your feedback on three capabilities built to fix it: Toolbox, Skills, and Tool Search. This is a discussion, not a slideshow. Bring your real projects, including tool sprawl, duplicated skills, and authentication headaches, and help shape where these features go next. Join us in the Microsoft Foundry Discord community. Event at a glance What: Microsoft Foundry Discord Community Round Table: Building Agents in Production with Foundry Toolbox, Skills, and Tool Search When: 22 July 2026, 5:00 PM BST (40 minutes) Where: https://aka.ms/foundry/discord Event Link https://discord.gg/Z8JZsrP5P5?event=1527379174061379584 Format: Interactive discussion with voice and chat, live polls, and a short prioritisation exercise Who it's for: AI engineers and developers building and scaling agents in production Opening question we'll start with: "As your agents grow, how do you decide which tools to give them, and how do they pick the right one at runtime?" The problem: agents don't scale by hard-wiring every tool When several agents, or a mix of Foundry hosted agents, Microsoft Agent Framework, LangGraph, and Copilot SDK apps, need the same governed set of tools, you do not want to re-wire those tools and their authentication into every one. Two things break as you grow: Integration sprawl: the same tool gets wired, authenticated, and versioned separately in every agent. Tool overload: sending every tool definition to the model on every turn is slow, expensive, and hurts selection accuracy. The pattern that scales: package the tools once behind a single versioned, governed MCP endpoint, make them discoverable, and let every runtime consume them from the same URL. That is what Toolbox, Skills, and Tool Search deliver together. The three concepts we'll discuss Toolbox: build once, govern centrally, consume anywhere A Toolbox is a reusable, centrally managed bundle of tools exposed through a single MCP-compatible endpoint. Because it is a managed resource, you can add, remove, or reconfigure tools without changing agent code because every agent connects to the same endpoint. Immutable versioning gives you safe, atomic rollouts: build and test a new version on its pinned URL, then promote it to default, and every consumer picks it up with no redeployment. Skills: reusable, composable capabilities A Skill is a reusable, published set of behavioural instructions (a SKILL.md file following the open Agent Skills spec) that is registered once and reused across toolboxes and agents, for example, "summarize document" or "create calendar event". In a toolbox, a skill is not a callable tool: it surfaces as an MCP Resource on the same endpoint, so clients discover and read it with plain resources/list and resources/read, with no Foundry SDK required. Tool Search: runtime discovery instead of hard-wiring A real toolbox can hold dozens or hundreds of tools. Tool Search keeps that cheap for the model through progressive disclosure: instead of listing every tool, Foundry shows the model just two meta-tools, tool_search and call_tool, plus any pinned tools. The model searches for a capability by intent, Foundry ranks the toolbox's tools by match on name and description, and returns only the hits. The prompt stays small no matter how many tools the toolbox holds. How they fit together: Skills and tools live in a Toolbox; Tool Search lets agents scale to many tools without prompt bloat or manual wiring. You manage all of it from the Foundry portal or the Foundry Toolkit extension in VS Code. The scenario we'll walk through A user asks an agent to "summarize this email and schedule a follow-up." The agent uses Tool Search to find the right tools ("email summarization" and "calendar scheduling") from a shared Toolbox, chains them, and returns a result with no per-agent integration and no hard-coded tool list. Discussion prompt: "At what point does the number of tools in your agent start to hurt, and would Tool Search help?" A peek at the code (so you arrive ready) The full, runnable walkthrough lives in the Mastering Foundry Toolbox notebook in the microsoft-foundry/forgebook repo. The core spine is short. First, build a versioned toolbox from typed tool objects plus an optional list of skills: # Build an immutable toolbox version from typed tools + skills version = project.toolboxes.create_version( name=TOOLBOX_NAME, description="Search, code, knowledge, and connection-backed tools.", tools=tools, # e.g. WebSearchToolboxTool(...), AzureAISearchToolboxTool(...) skills=skills or None, # ToolboxSkillReference(name=..., version=...) - SEPARATE from tools ) print(f"Created {TOOLBOX_NAME} version {version.version}") Then turn the same tools into a search-first toolbox by adding the Tool Search meta-tool and pinning only your one or two hottest tools: from azure.ai.projects.models import ToolboxSearchPreviewToolboxTool, ToolConfig # Pin the hottest tool so it's always exposed; everything else is search-gated. tool_configs = {"web_search": ToolConfig(pin=True)} search_version = project.toolboxes.create_version( name=TOOLBOX_NAME, tools=tools + [ToolboxSearchPreviewToolboxTool(tool_configs=tool_configs)], skills=skills or None, ) Every consumer talks to the same MCP endpoint: one URL, any framework: # The default (promoted) version is served from one stable consumer URL def consumer_mcp_url(name): return f"{PROJECT_ENDPOINT.rstrip('/')}/toolboxes/{name}/mcp?api-version=v1" # Microsoft Agent Framework speaks MCP natively - just point it at the URL. # LangGraph (AzureAIProjectToolbox) and the GitHub Copilot SDK consume the same endpoint. How Toolbox simplifies the auth and identity flow This is one of the most important things to understand before you scale agents, and it is a great topic to bring questions on. A toolbox tool reaches a downstream system through a project connection, and the connection's authentication type decides whose identity is used. Get this right once and every consumer inherits correct, least-privilege access automatically, without writing OAuth or token-exchange plumbing in your agent code. Running a toolbox behind a hosted agent puts two identities in play, and the platform wires them together for you: Agent -> Toolbox (the trust boundary). The hosted agent authenticates to the toolbox MCP endpoint with its own agent identity, which holds the Foundry user role on the project. If the agent doesn't have access, the toolbox rejects the agent. This gates access to the toolbox itself, independent of any single tool. Toolbox -> Tool (the end-user passthrough). For oauth2 authentication, the agent forwards the caller's end-user Entra token, and the toolbox uses that token (on-behalf-of) to reach the downstream tool. The tool then acts on behalf of the real end user, providing per-user, least-privilege access with correct downstream audit. For non-passthrough authentication types (none, custom-keys, project-managed-identity, and agentic-identity), the toolbox authenticates using the connection's configured identity, and the agent never sees the secret. That is the "better-together" story: a stable, governable managed identity to the toolbox, plus true end-user identity on the downstream data call. What we'll cover in the 40 minutes https://discord.gg/Z8JZsrP5P5?event=1527379174061379584 Welcome & framing (0:00-0:08): what Toolbox, Skills, and Tool Search are, and how they fit together. Scenario walkthrough (0:08-0:13): the "summarize email, schedule follow-up" flow, end to end. Use cases & opportunities (0:13-0:22): which capabilities you would package as reusable Skills, how many tools your agents carry, and where Tool Search would help. Trust, security & governance (0:22-0:31): what you are comfortable exposing through a shared endpoint, how to scope which tools an agent may discover, authentication models, and the observability you need. Developer experience feedback (0:31-0:36): your biggest adoption blockers, missing docs, and the SDK samples and end-to-end demos you would prioritise. Prioritisation & next steps (0:36-0:40): vote live on the top use cases, challenges, and feature requests. Come prepared to talk about What tools and skills your agents use today, and how they're wired up. Which capabilities you'd turn into reusable, composable Skills shared across agents. How many tools your agents carry, and whether you hit prompt-size or tool-selection accuracy issues. Which tools you'd expose through a shared endpoint, and which need tighter scoping. How you want to control what an agent is allowed to discover and invoke with Tool Search. The examples, samples, and tutorials that would help you get started fastest. Responsible and secure by design Because these features let agents discover and invoke tools dynamically, governance is a first-class part of the conversation. Foundry toolboxes are governed by default: you can screen every tool's inputs and outputs with an RAI guardrail, front your MCP servers with a bring-your-own AI gateway (APIM), scope which tools are discoverable, and use least-privilege identity passthrough so downstream calls carry the real user's permissions and audit trail. Bring your enterprise safeguard requirements because they directly shape the roadmap. Note: Toolbox, Tool Search, and Skills are in preview; APIs and headers may change. Key takeaways Toolbox packages tools once behind a single governed, versioned MCP endpoint, so you can build once and consume from any framework. Skills are reusable, composable capabilities you register once and chain across agents. Tool Search uses two meta-tools and progressive disclosure so agents scale to hundreds of tools without prompt bloat. Auth is simplified: the agent's managed identity gates the toolbox, while end-user token passthrough gives correct, least-privilege downstream access with no OAuth plumbing in your code. Your feedback shapes the product because this round table feeds directly into the engineering and product teams. Save your spot Add it to your calendar: 22 July 2026, 5:00 PM BST. Join the community: https://aka.ms/foundry/discord Prep with the sample: run the Mastering Foundry Toolbox notebook to build, search, and consume a toolbox end to end. Read the docs: Toolbox, Tool Search, and Skills. Agents get more capable as they gain access to more tools and skills, but only if you can build, govern, and scale those capabilities without drowning in integration and prompt bloat. Come and share how you are doing it today, and help shape how Foundry does it next. See you on 22 July.
Is there a way to connect 2 Ai foundry to the same cosmos containers?
I defined Azure AI Foundry Connection for Azure Cosmos DB and BYO Thread Storage in Azure AI Agent Service by using these instructions: Integration with Azure AI Agent Service - Azure Cosmos DB for NoSQL | Microsoft Learn I see that it created 3 containers under the cosmos I provided: <guid>-agent-entity-store v-system-thread-message-store <guid>-thread-message-store Now I created another AI foundry and added a connection for the same AI foundry, and it created 3 different containers under the same DB. Is there a way that they'll use the same exact containers? I want to use multiple AI foundries, and they will use the same Cosmos containers to manage the data.
