Enterprise Deployment Architecture for Agentic AI: ChatGPT-Style UI on IIS over Agents & MCP Servers
Research Disclaimer: This article is a technical research paper intended solely for educational, research, and illustrative purposes. It reflects the author’s personal technical perspectives. All concepts, architectural designs, diagrams, and implementation approaches are provided “AS IS” for technical exploration and review, not as production-ready development guidance. Organizations considering implementing these approaches should consult their internal software development standards, enterprise architecture teams, corporate security policies, and governance requirements. Any production implementation must undergo appropriate end-to-end architectural, security, privacy, compliance, and responsible AI or ethics reviews, along with necessary testing and organizational approvals before deployment. Abstract. Large language model agents that call tools through the Model Context Protocol (MCP) are usually shown from a console or a developer IDE. Enterprise users expect something different. They want a familiar chat interface, single sign-on and steady behavior when many people use the system at once. This paper gives architectural guidance for a ChatGPT-style web interface, hosted in Microsoft Internet Information Services (IIS) with Windows Authentication, that fronts an MCP-based agent for 25 to 30 concurrent users. The work is broken into nine steps: the engine boundary, state isolation, a streaming API, the chat UI, a Windows service host, IIS reverse-proxy configuration, security, testing and deployment. The paper explains why a long-running Python service behind IIS beats launching Python for every request, and why no Kerberos delegation (the double hop) is needed. It also describes a small transport-neutral tool-client layer, so the same agent can call a local STDIO MCP server and several remote MCP servers over HTTPS without any change to the user interface. The reference scenario is ChefAI, a fictitious restaurant-operations assistant that needs real-time answers, invoice generation, and log and crash-dump analysis for technical troubleshooting. Its STDIO server generates invoices. Three HTTP servers handle document retrieval through an Agentic RAG system, log analysis and dump analysis. The agent logic itself is treated as a black box. The paper closes with a threat model, a capacity argument and an evaluation methodology. Keywords: Model Context Protocol, agent user interface, IIS, Windows Authentication, reverse proxy, Server-Sent Events, agentic RAG, concurrency control, tool routing 1. Introduction Tool-using language model agents have become practical because protocols such as the Model Context Protocol (MCP) standardize how an agent discovers and calls external capabilities [1]. A typical agent loop sends a user request and the list of available tools to a model, receives a tool choice, runs the tool, and feeds the result back until the model can answer. The loop is easy to demonstrate in a terminal. Putting it in front of a team is harder, because the interface, identity, concurrency, streaming and deployment all become real concerns. This paper addresses that gap for a common enterprise setting: Windows infrastructure, an intranet, Active Directory identities and IIS as the standard web host. It is written as a reference architecture. It describes the components, what each one is responsible for, the trust boundaries between them and the order in which to build them. A fictitious reference application, ChefAI (Section 1.2), keeps the discussion concrete. End users reach the application through a browser chat window with their Windows identity, while the agent behind the scenes talks to one local MCP server over STDIO and to several remote MCP servers over HTTPS. 1.1 Contributions A nine-step design for a ChatGPT-style web interface that fronts an MCP agent and is hosted on IIS with Windows Authentication (Sections 4 and 5). A concurrency design sized for 25 to 30 simultaneous users, with explicit queues, per-user limits, timeouts and cleanup (Sections 5.2, 5.5 and 10). A transport-neutral tool-client layer made of an interface, a server registry and a router, which keeps the UI independent of how tools are reached (Section 6). A reference fleet of four MCP servers with different capabilities and data sensitivities, including an Agentic RAG document server, used to reason about routing and security (Section 7). An analysis of identity flow showing that no Kerberos delegation is needed, a threat model and an evaluation methodology (Sections 8 to 10). 1.2 The ChefAI reference application What ChefAI is. ChefAI is a fictitious restaurant-operations application used throughout this paper as the reference scenario. It is an AI assistant for a restaurant business. Staff need real-time answers during service, invoices generated from purchasing data, and help with technical troubleshooting when systems such as point-of-sale terminals, kitchen displays or online-ordering services misbehave. Staff describe what they need in plain language in a browser. The assistant decides which capability to use and answers with figures, documents, diagnostics or a generated report. How it is structured. The application is split into an agent and four MCP servers. Agent.py is the agent. It receives a request, consults a language model and calls tools exposed by MCP servers. The first server, server.py , is a local STDIO server and acts as the invoice generator. It reads purchasing data and produces HTML invoices and reports. The other three servers are remote HTTP(S) servers ( server1.py , server2.py and server3.py ), not STDIO, and the agent calls them to get information and to support technical issues. They are a document processor that answers questions by retrieving data and documents from an Agentic RAG system, a log analyzer and a dump analyzer. Table 0 summarizes the scenario. Capability Example request from staff Served by Transport Real-time spend answers “What did we spend on tomatoes last year?” Invoice server ( server.py ) STDIO, local Invoice generation “Generate the year-to-date invoice.” Invoice server ( server.py ) STDIO, local Knowledge and documents “What is our return policy for dairy deliveries?” “How do we descale the combi oven?” Document processor with Agentic RAG ( server1.py ) HTTPS, remote Log analysis “Why did the point-of-sale terminals time out on Friday evening?” Log analyzer ( server2.py ) HTTPS, remote Dump analysis “The kitchen display crashed. What does this dump show?” Dump analyzer ( server3.py ) HTTPS, remote Table 0. The ChefAI reference scenario: capabilities, the server that provides each, and its transport. The scenario deliberately mixes a business need (spend, invoices, policies) with an operational need (logs, dumps and runbooks). One interface therefore has to route between servers with very different data and risk profiles. The restaurant details, hostnames and example questions are illustrative and do not describe a real business. 1.3 Scope The focus is the user-interface tier and its hosting. The agent internals are out of scope except for the configuration the UI depends on, namely how the agent is invoked and how it reaches tools. The tool names, capacity figures and timeouts in this paper are illustrative design values. They show how to reason about a deployment and should be checked against the real workload (Section 10). 2. Background 2.1 The Model Context Protocol and its transports MCP defines how a client discovers a server’s tools, resources and prompts and calls them using JSON- RPC messages [1]. Two standard transports matter here. STDIO launches the server as a child process and exchanges messages over standard input and output. It is simple and local and needs no network port, but it ties the server’s lifetime to the client process. Streamable HTTP exposes the server at a URL. That allows servers on other machines, shared by several clients and protected by TLS and HTTP authentication. The official SDKs let the same tool code be exposed over either transport, so a server can move from one to the other without rewriting its tools. 2.2 Server-Sent Events for progress Agent requests can last tens of seconds because they chain model calls and tool calls. Server-Sent Events (SSE) let the server push a one-way stream of text events over an ordinary HTTP response [2]. This suits a chat interface that wants to show “queued”, “analyzing logs” and “building invoice” before the final answer. SSE also passes through reverse proxies more easily than WebSockets, as long as buffering is turned off. 2.3 IIS, ARR and Windows Authentication IIS can authenticate users with the Negotiate provider, which uses Kerberos where available and falls back to NTLM [3]. With the URL Rewrite and Application Request Routing (ARR) extensions, IIS can act as a reverse proxy. It forwards selected paths to a back-end service and can copy server variables such as LOGON_USER into request headers [4]. 2.4 The double-hop problem A “double hop” happens when a server has to use the caller’s Windows identity to reach a second server on the caller’s behalf. That requires Kerberos delegation. Section 8 shows why this design avoids it. 3. Requirements and Constraints Category Requirement Interface ChatGPT-style chat in a browser, real-time progress, a stop button, report cards with preview, and no CDN so it works offline on an intranet. Identity Windows Authentication in IIS. The application knows the user’s name and never handles user passwords. Concurrency 25 to 30 simultaneous users without cross-user data leakage, orphaned processes or unbounded resource use. Tools A local STDIO MCP server and several remote HTTPS MCP servers. The UI must not change when servers are added or moved. Security LLM keys never in files, logs or the browser. Loopback-only back end, output escaping, restrictive report serving and allow-listed remote servers. Operations Runs as a Windows service with restart on failure, structured logs and idempotent deployment scripts. Constraints System Python without virtual environments, standard-library HTTP for model calls, IIS as the only public listener and no Kerberos delegation. 4. System Architecture Figure 1 shows the end-to-end architecture. Users reach IIS over HTTPS and authenticate with Windows credentials. IIS serves the static single-page interface and reverse-proxies /api/* over loopback to a Python web service. The service owns the agent engine, which plans with a language model and calls tools through a router. The router reaches the local invoice server over STDIO and the remote servers over HTTPS. Figure 1. End-to-end architecture. Solid boxes and arrows are local components and the local STDIO server. Dashed boxes and arrows are remote MCP servers reached over HTTPS. Adding a server only touches the service and the tool layer. The browser and IIS tiers stay the same. 4.1 Request lifecycle Figure 2 follows a single prompt. The browser posts to /api/chat . IIS authenticates the user, overwrites the identity header and proxies the call. The service queues and admits the request, then streams status events while the engine consults the model and calls tools. The loop repeats, up to a fixed number of tool calls, until the model reports completion or, for report requests, an HTML file exists. The final answer and report links go out as the last events. Figure 2. Request lifecycle for one prompt. Messages 3 and 12 are Server-Sent Events returned through IIS on the original HTTP response. The browser never talks to the engine, the model or any MCP server directly. 4.2 Why a web service sits between the browser and the agent, and why the browser cannot call MCP servers directly The browser cannot call the agent directly, for reasons of feasibility and of fitness. On feasibility, clientside JavaScript can only issue HTTP requests and cannot start or import a Python program. IIS, as a static-file and ASP.NET host, has no in-process Python runtime. An agent is also a script and not a network endpoint, so something has to listen for requests and invoke it. On fitness, an agent request is long-lived, stateful and costly. It chains model and tool calls for tens of seconds, spawns MCP child processes that must be cleaned up on timeout or disconnect, keeps per-conversation state and has to stream progress. It therefore needs a persistent process that can enforce admission control (per-user gates, a bounded queue, semaphores and timeouts), isolate sessions, emit Server-Sent Events and hold the model key and service credentials in a service account, not in the client or the IIS application pool. Launching Python for every request would add seconds of start-up cost and make streaming, cancellation and cleanup fragile. Letting IIS supervise the process, for example through FastCGI or HttpPlatformHandler, changes who restarts it but not the need for it. The browser also cannot call the MCP servers directly. A STDIO server has no network listener at all. A Streamable HTTP server does carry MCP’s JSON-RPC over HTTP, so transport is not the real obstacle. The real obstacle is that an MCP server exposes typed tools and does not understand natural language, so some component must decide which tool to call with which arguments and iterate on the results. That component is the modeldriven agent loop, and it needs the LLM key. Direct access would also place server tokens or client certificates in every browser and expose the sensitive log and dump servers to every desktop. It would bypass central rate limiting, audit, allow-listing and redaction, remove the agent’s defence against prompt injection in tool results, and force the interface to carry discovery, routing and namespacing logic for every server, so adding a server would change the UI. Putting the service in between reduces the browser’s contract to a single HTTP endpoint, /api/chat , and keeps MCP, model credentials and trust decisions on the server side. 5. User-Interface-Centered Design in Nine Steps The design is organized as nine steps (Figure 3). Steps 1 and 2 define the boundary to the agent, Steps 3 and 4 deliver the API and the interface, Steps 5 and 6 host it, Step 7 secures it, Step 8 tests it and Step 9 documents and deploys it. Figure 3. The nine-step roadmap. Amber is the agent boundary, green is the API and interface, grey is hosting, red is security, and purple is verification and delivery. 5.1 Step 1: Define an engine boundary the UI can call An agent is often first written as a console program that mixes the planning loop with prompts and printing. The first step separates them. The agent exposes one asynchronous function, handle_prompt(prompt, history, progress_callback) , that returns {answer, html_files, tool_trace, error} and reports progress through a callback instead of printing. It never reads from or writes to a console. A single prompt box replaces any menu of modes, and the model decides from the live tool list whether the request needs a generated file or only an answer. Recent conversation turns, six by default, are passed as context so that follow-ups such as “now compare it to 2026” work. Only aggregates and provenance go back to the model, and raw rows stay on the server. This boundary is the only contract between the interface tier and the agent. 5.2 Step 2: Remove shared mutable state Agents and tool servers often keep module-level state, such as loaded data and cached query results. With 25 to 30 concurrent users, that state becomes a correctness and privacy hazard. The design therefore does three things. It loads and parses reference data once into an immutable shared dataset, reloaded only when the source file’s size or modification time changes and guarded by a lock. It keeps query results in a per-session store, so each query identifier belongs to exactly one session. And it gives each request an isolated tool connection, or a bounded pool of server processes with per-request session identifiers. Shared outputs are written atomically (a temporary file, then a rename) so a reader never sees a half-written file. 5.3 Step 3: Provide a streaming API The service exposes four endpoints (Table 1). The chat endpoint returns a Server-Sent Event stream. Conversation history is kept per user and conversation with a time-to-live and a cap, and no other user can read it. Endpoint Behavior POST /api/chat Body {prompt, conversation id} . The prompt is validated and capped at 2000 characters. Streams status , tool , result , done and error events (Appendix A). GET /api/reports/{name} Serves generated HTML from one folder only. Filenames are allow-listed and the resolved path must stay inside the folder. Sends a restrictive Content-Security-Policy and nosniff . Reports are scoped per user, plus shared built-in reports. GET /api/health Status, provider, model, data-loaded flag, queue depth and running count. Never keys or paths outside the project. GET /api/me Authenticated display name taken from the trusted header. Table 1. API surface of the web service. 5.4 Step 4: Build the ChatGPT-style interface The interface is plain HTML, CSS and vanilla JavaScript. There is no framework and no CDN, so it works on an isolated intranet and adds no supply-chain exposure. Figure 4 shows the layout. The sidebar lists this browser’s recent conversations from localStorage . The main pane shows user and assistant bubbles, a live status line fed by the event stream, and report cards that open in a new tab or preview in a sandboxed iframe with scripts disabled. The composer grows with its content, sends on Enter, inserts a newline on Shift+Enter, shows a character counter and is disabled while that user has a request running. A Stop button aborts the stream, which in turn cancels the server-side run. Example prompts fill an empty chat. Figure 4. Wireframe of the chat interface. (1) Conversation sidebar, (2) message bubbles, (3) live status line from the event stream, (4) report card with Open and Preview, (5) composer with Send and Stop, (6) data-egress notice. All model and server text is inserted as escaped text. Minimal formatting (line breaks, bold, simple tables) is added by the interface itself and never by interpreting model output as HTML. Errors are visible and actionable. Network loss, 401 and 403, 409 (a request is already running), 429 (system busy, with a retry hint) and timeouts each show a message and a Retry button. Figure 5. Typical event sequence for an invoice request. The browser adds each event to the status line as it arrives and renders the final answer and report cards when the closing done event comes in. 5.5 Step 5: Host the service The Python application runs under a Windows service wrapper (for example NSSM or WinSW), bound to 127.0.0.1 on a fixed port, with automatic restart and a dedicated low-privilege account. It reads the provider name, the model override, the data path and the limits from environment variables, and it reads the LLM key from the service environment only. The account gets read access to the project and data, and write access only to the report output and log folders. Capacity is protected by explicit limits (Figure 6). There is a per-user gate (one running request and a rate limit), a bounded queue that reports “queued, position N” through the stream and returns HTTP 429 when full, a semaphore on concurrent agent runs (default 8) and a separate semaphore on concurrent model calls (default 6). Every request has a wall-clock timeout (default 120 s). On timeout, cancellation or browser disconnect, the request’s child processes are terminated and its slots released, so no orphaned processes remain. Blocking work such as file and network I/O runs off the event loop, so one slow call cannot stall other users. Figure 6. Admission control and cleanup. A request has to pass the per-user gate and the queue before it gets an agent slot, and model calls are throttled separately. 5.6 Step 6: Configure IIS IIS is configured as a thin front end. The site’s application pool uses “No Managed Code” because IIS only serves static files and proxies. URL Rewrite forwards /api/* to the loopback service and, in the same rule, sets an X-Forwarded-User header from the authenticated LOGON_USER value, overwriting anything the client sent. The ARR proxy is enabled with a timeout longer than the engine’s request timeout, and response buffering and dynamic compression are turned off so the event stream is delivered live. Request size is capped, risky file extensions ( .py , .csv , .env , .log , .md ) are denied and an HTTPS binding is required. Windows Authentication is enabled and Anonymous is disabled. Those sections are locked at server level by default, so they are set through appcmd at the application-host scope. Appendix C lists the key settings. 5.7 Step 7: Secure the system Section 8 covers the security controls in detail. In short, identity comes only from IIS and the service accepts connections from loopback only. The model key is read from the service environment and scrubbed from every error and log. All text is escaped in the browser. Reports are served with a restrictive policy from one folder. Tools are read-only against the data and write only to report folders. Users are told that aggregated figures are sent to the external model provider. 5.8 Step 8: Test Tests use a mocked model, so no paid calls are made and results are repeatable (Section 10.1). They cover the API contract, path traversal, cross-user isolation, malformed model output and missing keys. They also run a simulated 30-user mixed workload, check cleanup on timeout and disconnect, scan responses and logs to confirm the key never appears, and exercise the tool router with fake clients. 5.9 Step 9: Document and deploy Two idempotent PowerShell scripts install the service and configure IIS. The README documents installation, service commands, key rotation and troubleshooting. Idempotence matters because configuration drift is a frequent cause of failure in reverse-proxy deployments, and a script that can be safely rerun repairs drift instead of adding to it. 6. The Agent Boundary and a Transport-Neutral Tool Layer The agent is treated as a black box that takes a prompt and history and returns an answer, files and a trace. Only one part of its configuration matters to the interface design, and that is how it reaches tools. A deployment may reach a tool server as a local STDIO child process or as a remote HTTPS endpoint, and may move a server from one to the other. Neither choice should change the browser, IIS or the API contract. Three small abstractions achieve this (Figure 7). ToolClient interface. Three asynchronous operations: list_tools() , which returns name, description and input schema, call_tool(name, arguments) , which returns a result object, and close() . An STDIO client, an in-process fallback and a Streamable HTTP client all implement it. Server registry. A list of entries, each with a name, transport, command or URL, the name of the environment variable holding its credential, a timeout and an enabled flag. A deployment can start with a single entry, such as the local invoice server over STDIO, and grow by loading more entries from a configuration file. Tool router. It combines the tools of every enabled client into one live list for the model and sends each call to the client that owns the tool. Once more than one server is registered, it namespaces tool names ( server__tool ) so collisions cannot happen. A slow or unreachable server is skipped with a status event and does not fail the whole request. Application-specific behavior, such as the data summary sent to the model, query identifiers and the rule that report requests must produce an HTML file, lives in an adapter and not in the generic loop. That way, servers that know nothing about invoices can still be used. Figure 7. The transport-neutral tool layer. A remote server is added with a registry entry. One HTTP client implementation serves all remote servers, and no code above the router changes. 6.1 Registry example The following shows the shape of the registry for the four ChefAI servers. Credentials are referenced by environment-variable name, and the secrets themselves never appear in the file. Here docs , logs and dumps correspond to server1.py , server2.py and server3.py . With several servers registered, the model sees namespaced tools such as invoices__generate_invoice or logs__search_logs . Each extra server makes the tool list sent with every model call longer, so a practical deployment should filter or pre-route the list by request intent. Section 11 returns to this. 7. The ChefAI MCP Server Fleet The ChefAI scenario (Section 1.2) uses four servers with deliberately different capabilities (Table 2). Tool names are illustrative, and an implementation may name and group tools differently. Server Transport Capability and example tools Data sensitivity Operational notes Invoice generator server.py STDIO, local Queries purchasing data by period, item and vendor, and generates HTML invoices and reports with totals computed in code. Tools: query expenses ,generate invoice ,generate report , chef summary . Business- confidential totals and vendor names. One child process per request. Read- only on data, writes only to report folders. Document processor (Agentic RAG) server1.py HTTPS, remote Answers questions by retrieving data and documents (supplier contracts, return policies, menus, allergen sheets, equipment manuals, runbooks) from an Agentic RAG system and returning an answer with citations. Tools: search documents , get document, answer with citations . Mixed. Some documents are confidential (contracts, pricing), so it needs document-level access control. Read-only. Always returns source references. Has a retrieval budget and timeout, and shows index freshness. Log analyzer server2.py HTTPS, remote Searches and summarizes application and device logs from point-of-sale, kitchen-display and ordering systems: error counts by time bucket, top messages, correlation by request id. Tools: search logs ,summarize errors ,time histogram . May contain user identifiers, IP addresses or secrets, so it needs redaction. Read-only. Limits result size and redacts on the server side. Dump analyzer server3.py HTTPS, remote Triages crash or memory dumps from restaurant systems:exception summary, faulting module, thread stacks. Tools: list dumps , analyze dump , stack summary. Dumps can contain memory contents,including credentials. The most sensitive server. Long-running, so it needs a larger timeout. Strict access control. Returns summaries only, never raw dumps, to the model. Table 2. The four MCP servers in the ChefAI scenario. 7.1 Agentic RAG in the document processor Retrieval-augmented generation (RAG) grounds a model’s answer in retrieved documents instead of the model’s own memory. An agentic RAG system goes further. It plans the retrieval, may rewrite the query, searches several collections, reads the most promising documents, checks whether the evidence is enough and iterates before answering. In ChefAI this behavior lives inside the document server, behind a small tool surface. The main agent sees one tool, such as answer_with_citations , and does not need to know how the retrieval works. This has three consequences for the design. The document server must return citations so the interface can show sources. Its latency is usually higher than a simple lookup and is bounded by its own timeout and retrieval budget. And retrieved text is untrusted input that can carry injected instructions (threat T2 in Section 8). 7.2 Cross-server tasks The value of a multi-server fleet is that the servers can work together. Take a request such as “The kitchen display froze during Friday dinner service. What happened and what should we do?” It could need the log analyzer for an error histogram around the incident, the dump analyzer for the crash cause and the document processor for the vendor’s runbook and escalation policy. A request such as “Generate the year-to-date invoice and check our dairy return policy” combines the invoice generator with the document processor. The router makes these tools look like one catalogue. Per-server timeouts and failure isolation mean that if the dump server is down, the other tools still answer. Combining servers also raises a data-flow question, because results from one server become inputs to another through the model. Every result passes through the engine, so the engine is the right place to apply size limits, redaction rules and per-server allow-lists. 8. Windows Authentication, Trust Boundaries and Threats 8.1 Identity flow and the absence of a double hop The browser authenticates to IIS using Negotiate (Kerberos or NTLM). IIS knows the user’s account, copies it into X-Forwarded-User and forwards the request over loopback. From that point the application uses only the name, for scoping history, report access, rate limits and logs. The user’s Windows token is never impersonated or forwarded. Resources are reached with service-level credentials: the data file with the service account, the model with an API key, and remote MCP servers with a bearer token or client certificate (Figure 8). No downstream system needs to act as the user, so Kerberos constrained delegation is unnecessary. Figure 8. Identity and credential flow. The user identity is used only for scoping, and every downstream resource is reached with a service-level credential. When delegation would be needed. Delegation, or token forwarding, becomes necessary only if a downstream system must authorize the end user itself. Examples are a per-user permissioned file share, a SQL Server using Windows authentication as the user, or a remote MCP server (such as the document processor) that must apply the caller’s own document entitlements. Those cases would need Kerberos constrained delegation or an OAuthstyle token exchange, and they are outside this design. 8.2 Trust boundaries Figure 9 marks four zones and the threats that cross them. The key rule is that the Python service trusts the identity header only because it listens on loopback and IIS overwrites the header on every proxied request. Figure 9. Trust zones and threat markers T1 to T6 (Table 3). Arrows show the direction of requests. ID Threat Mitigation in this design T1 Spoofed identity header or direct access to the service port IIS overwrites the header on every proxied request. The service binds to 127.0.0.1 and rejects non-loopback callers, and the port is not opened in the firewall. T2 Prompt injection through tool results, retrieved documents or logs (for example text that says “ignore your instructions”) Tool results and retrieved text are treated as untrusted data. Tools are read- only. Tool calls are allow-listed and capped, arguments are validated against schemas, and results are size-limited and redacted before they reach the model. T3 Pointing the agent at arbitrary URLs (server-side request forgery) Remote servers come only from the registry. HTTPS is required, hosts are allow-listed and the model cannot supply URLs. T4 Sensitive data leaving the network through the model provider Only aggregates, provenance and short cited excerpts are sent, never raw rows. Log and dump summaries are redacted. The UI shows a notice and the provider is chosen by policy. T5 Cross-user leakage (history, query results, reports, documents) Per-session stores keyed by user and conversation. Query identifiers are owned by one session, reports are scoped, and the retrieval system applies document-level access control. Tests try cross-access on purpose. T6 Path traversal or exposure of secrets in files served by IIS or the service Report filenames are allow-listed and the resolved path is checked. IIS denies .py , .csv , .env , .log and .md . Keys live only in the service environment and are scrubbed from errors and logs. Table 3. Threat model. The model key is stored in the service’s environment, which administrators can read from the registry. Sites that need stronger protection should use a machine-level secret store read at start-up. 9. IIS Hosting and Deployment Figure 10 shows the physical deployment. Only port 443 is exposed to clients. The Python service listens on loopback and needs no inbound firewall rule. Outbound traffic is limited to the model provider and the approved remote MCP hosts. Figure 10. Deployment topology and firewall posture. 9.1 Provisioning sequence Install IIS with static content, request filtering and Windows Authentication. Install URL Rewrite and ARR. Enable the ARR proxy, set its timeout above the engine timeout, set the response buffer limit to zero and allow the HTTP_X_FORWARDED_USER server variable. Create the application pool (No Managed Code) and the site pointing at the UI folder, then add the HTTPS binding with an internal CA or test certificate. Disable Anonymous and enable Windows Authentication at the application-host scope. Add the rewrite rule, request limits, security headers and file-type deny list to the site configuration. Create the service account, grant the narrowest file permissions, install the service wrapper, set the environment variables and start the service. Verify the result: the service is running, the health endpoint answers on loopback, sign-in from a client is silent, a streamed answer arrives, a report card appears, access to data and source files is blocked, and two simultaneous users get independent results. 9.2 Common failure modes Symptom Typical cause 502.3 on /api Service stopped, wrong port, or ARR proxy not enabled. 500.50 from URL Rewrite The identity server variable is not in the allowed list. Repeated sign-in prompts Authentication sections not set at application-host scope, browser zone settings, or a missing service principal name for the site name. Progress arrives only at the end Response buffering or dynamic compression on the proxied route. Timeouts near two minutes Proxy timeout shorter than the engine timeout. Leftover Python processes Cleanup path not triggered on timeout or disconnect. 10. Evaluation Methodology and Capacity Reasoning 10.1 Evaluation plan An implementation of this architecture should be evaluated against the criteria in Table 4 before it is released to users. A mocked model keeps functional tests repeatable and free of provider cost. Latency and quality are then measured with the real provider. Question Method Metric or pass criterion Correctness of the API Mocked model, test client and deterministic scenarios. All contract and error-path tests pass, and totals equal direct data queries. Isolation 30 simulated users with mixed queries and reports, plus deliberate cross-access attempts. Zero cross-user data, zero unhandled exceptions and zero orphaned processes. Concurrency limits Ramp from 1 to 30 users, saturate agent slots and fill the queue. Running agents never exceed the configured cap, overflow returns 429, and the system recovers after the burst. Real-time responsiveness Mocked model with a fixed delay, then the real provider, with a browser trace through IIS. Median and 95th-percentile end-to-end time, time to first event, first event before the first tool completes, and no buffering. Cleanup Timeout and abrupt disconnect tests. Child processes terminated, slots released and no handle growth after repeated runs. Secret hygiene Automated scan of responses, logs and error text. The key never appears. Router Two to four fake clients, one slow and one failing. Correct routing and namespacing, and the request still succeeds when one client fails. Retrieval quality (document server) A question set with known source documents. Share of answers with correct citations, and refusal when evidence is missing. Table 4. Recommended evaluation plan. 10.2 A capacity argument Little’s law relates the average number of requests in service N to the arrival rate λ and the average time in the system W: N = λ × W [7]. Suppose, as an assumption, that each of 30 users submits one request per minute on average (λ = 0.5 requests per second). If the mean service time is 15 s, then on average N = 7.5 requests are in service at once. That fits the default of eight agent slots with little queueing. If the mean time is 30 s, then N = 15, the slots saturate and queueing delay grows, so either the cap or the speed of each run has to change. The model call is the biggest part of a run, and an Agentic RAG lookup adds its own multi-step latency. Provider latency and the number of tool calls per request therefore set the capacity you need. These figures illustrate the method, and each deployment should substitute its own measured arrival rate and service time. Memory per STDIO child process also grows with N, so it must be measured before choosing between one child per request and a pool. 11. Limitations and Future Work Reference architecture. ChefAI is a fictitious scenario and the figures in this paper are design values, not benchmark results. Capacity, latency and memory must be measured for each real deployment. Single-process assumption. Per-user limits, history and the shared data cache live in memory. Scaling to several worker processes needs a shared store, such as a database or cache, or sticky routing. Tool-list growth. More servers mean larger prompts and more mistakes in tool selection. Pre- routing by intent, per-user tool entitlements and lazy tool loading are natural extensions. Per-user authorization downstream. The design uses service credentials. If remote servers, notably the document processor, must enforce each user’s own rights, token exchange or delegation will be required. Model-provider dependence. Answers depend on an external provider’s availability, cost and data policy. An on-premises model would change the data-egress analysis. Human factors. The paper does not prescribe a usability study. Candidate measures include task completion, perceived trust in numbers and citations, and how clear the status line is. Secret storage. A service environment variable is fine for a pilot. A managed secret store is better for production. 12. Conclusion Putting an MCP agent behind a familiar chat interface is mostly an engineering problem at the edges. The questions are how the request enters, who the user is, how long-running work is streamed, how concurrent users are kept apart and how new tool servers are added without disturbing the rest. The architecture in this paper keeps IIS thin, with static content, TLS, Windows Authentication and a reverse proxy. It puts the agent behind a loopback Python service with explicit admission control, and it separates the interface from the way tools are reached by using a small transport-neutral tool layer. User identity is used only for scoping and every downstream resource is reached with a service credential, so no Kerberos delegation is needed. For the fictitious ChefAI restaurant, this gives staff real-time answers, invoices, document-grounded guidance and log and dump diagnostics from one chat window. The recommended path is to build the nine steps in order, validate the result with the evaluation plan, and add remote MCP servers one at a time through registry entries. References Model Context Protocol. “Model Context Protocol specification.” modelcontextprotocol.io (accessed October 2026). WHATWG. “HTML Living Standard: Server-sent events.” html.spec.whatwg.org (accessed October 2026). Jaganathan, K., Zhu, L., and Brezak, J. “SPNEGO-based Kerberos and NTLM HTTP Authentication in Microsoft Windows,” RFC 4559, IETF, June 2006. Microsoft. “URL Rewrite” and “Application Request Routing” documentation for IIS, learn.microsoft.com and iis.net (accessed October 2026). Fielding, R., Nottingham, M., and Reschke, J. “HTTP Semantics,” RFC 9110, IETF, 2022. OWASP Foundation. “OWASP Top 10 for Large Language Model Applications.” owasp.org (accessed October 2026). Little, J. D. C. “A Proof for the Queuing Formula: L = λW.” Operations Research 9(3), 1961. Appendix A. Event stream schema Each SSE message has an event name and a JSON data field. The schema is the only contract between the browser and the engine. Appendix B. Environment variables Variable Purpose LLM_PROVIDER, LLM_MODEL Model provider and optional model override. LLM_API_KEY Provider key. Service environment only, never logged or returned. CHEFAI_CSV_PATH Location of the purchasing data file used by the invoice server. CHEFAI_REQUIRE_USER 1 rejects requests without the identity header (production). 0 uses a single anonymous user (development). CHEFAI_MAX_AGENTS Concurrent agent runs (default 8). CHEFAI_REQUEST_TIMEOUT Per-request wall-clock limit in seconds (default 120). MCP_*_TOKEN Per-server credentials for remote MCP servers, referenced by name in the registry. Appendix C. Key IIS settings These settings are indicative. Module versions and server policies vary, so validate them in a non-production environment before rolling out.Introducing Azure Container Apps Sandboxes: Secure Infrastructure for Agentic Workloads
Today we are announcing the public preview of Azure Container Apps Sandboxes - a new first-class resource type that gives you fast, secure, ephemeral compute environments with built-in suspend and resume. This is the underlying infrastructure on which products like Cloud sandboxes in GitHub Copilot, Foundry Hosted Agents, and Azure Container Apps Express are built, you now have the opportunity to build your solutions leveraging this infrastructure. Azure Container Apps Sandboxes unlocks two massive opportunities. For platform developers and ISVs, sandboxes give you the same isolated compute fabric that powers many Microsoft products. You get the building blocks to create your own multi-tenant platform on proven, enterprise-scale infrastructure. For AI agents, sandboxes become a self-configurable tool that lets agents extend their own capabilities on the fly. An agent can spin up a fresh sandbox in milliseconds and use it to execute untrusted code, compile source, test HTTP requests against a live app, launch a browser session, or tackle whatever needs a quick and scalable infrastructure. On one side it empowers humans to build platforms, on the other it empowers agents to build their own capabilities. Both get enterprise-grade isolation, instant startup, and snapshot-based persistence out of the box. We'll walk through the resource model, sandbox lifecycle, the features that set Sandboxes apart - like snapshots, lifecycle policies, network egress controls, volumes, and managed identities - and show you how to get started with the portal and CLI. What Are Container Apps Sandboxes? Container Apps Sandboxes are secure, isolated compute environments that start in sub-second time, scale to thousands, and cost nothing when idle. Each sandbox runs in its own hardware-isolated microVM boundary - fully separated from the host, the platform, and every other sandbox. You bring your own Open Container Initiative (OCI) image, and Sandboxes handle the rest: provisioning from prewarmed pools, strong multi-tenant isolation, and snapshot-based suspend/resume that preserves full memory and disk state across sessions. There are many ways Sandboxes can help you build your next project - here are a few: Your own build & test systems - wire a Sandbox into your CI/CD flow to run builds while your laptop stays cool. Agents that can run anything safely - an agent spawns a sandbox, executes work inside it, and returns the output with no agent host privileges required. Agent swarms - decompose a research question, spawn N sandbox workers in parallel (each pinned to its own image and egress policy), and synthesize the result. Early access customers are already unlocking significant benefits by leveraging Azure Container Apps Sandboxes. "With Azure Container Apps sandboxes, SitecoreAI can safely enable agents to take real action. The combination of multi-tenant isolation, rapid scale-out, and full automation allows Sitecore to run long-lived, autonomous agents that securely execute code, manage workflows, and interact with enterprise systems within secure, governed environments. With this foundation, we can build agents that do real work: assembling content, personalizing experiences, and optimizing campaigns in production. Agents that operate continuously, learn from results, and improve over time, so our customers get better outcomes without giving up control." - Mo Cherif, VP of AI and Innovation, Sitecore "We got early access to Azure Container Apps Sandboxes, and got the first prototype integrated with Atlas AI in hours, and it's already shaping a new Atlas AI capability that we plan to launch in preview in Q3. It gives every Atlas AI agent a safe, sandboxed workspace (file system, terminal, code execution) on a customer's live data in Cognite Data Fusion. The value: Industrial process, reliability, and production engineers spend days and weeks on questions like "which wells are underperforming and why?" These questions are tractable but expensive, so they are asked rarely and decisions are made on gut feel. With this, an agent pulls the data, runs the analysis, cross-references maintenance and inspection records, and returns a cited draft in minutes. Sandboxes make it practical: Aligned feature set, per-customer isolation, pause/resume across multi-day investigations, scale-to-zero economics." - Kelvin Sundli, Product manager, Atlas AI, Cognite Resource Model: Sandbox Groups and Sandboxes The top-level ARM resource is Microsoft.App/SandboxGroups. A Sandbox Group is the management boundary for a collection of sandboxes that share configuration - think of it like a Container Apps Environment, but purpose-built for sandboxes. When you create a Sandbox Group, you specify: Subscription, Resource Group, and Region Sandbox defaults (optional): default CPU, memory, disk, max sandbox count, and default idle timeout Networking: optionally deploy into a custom VNet with a dedicated subnet for private networking Identity: System or user assigned Entra identity. Individual sandboxes are created within a Sandbox Group. Each sandbox has its own source (disk image or snapshot), resource tier, lifecycle policy, network egress policy, environment variables, ports, volumes, and connections. Sandbox Lifecycle Sandboxes have a well-defined lifecycle with the following states: State Description Creating Provisioning the sandbox from a disk image or snapshot Running Actively executing - backed by a live microVM Idle System-suspended after inactivity; can auto-resume on the next request Suspended Full state (memory + disk) preserved as a snapshot; no compute costs Resuming Restoring from a suspended or idle state - sub-second for most workloads Stopped User-initiated stop; can be resumed Stopping Graceful shutdown in progress Deleting Teardown in progress The key insight here is the distinction between Idle and Suspended. When a sandbox goes idle (e.g., no traffic for a configured timeout), the system can automatically suspend it and capture a snapshot. When a new request arrives, the sandbox resumes transparently. This gives you scale-to-zero economics with stateful compute - something that wasn't possible before without significant custom engineering. Disk Images: Bring Your Own Container Sandboxes boot from Disk Images - Open Container Initiative (OCI) images converted into an optimized root filesystem format. You point to any OCI image (public or private registry), and the platform builds a bootable disk image from it. You can start with public, pre-built images maintained by the platform (for example, Ubuntu base images), or bring your own private images. For private registries, you can authenticate with username/token or use a user-assigned managed identity for Azure Container Registry (ACR) – integrated with Azure as you expect. Snapshots: Full-State Persistence Snapshots capture the complete state of a running sandbox - memory, disk, and all running processes. When you resume a sandbox from a snapshot, every process, open file handle, and in-memory data structure is restored exactly as it was. A snapshot captures the full state of a running sandbox: memory pages, disk, processes. Two ways to make one - automatically on suspend, or manually on demand. Three things they're great for: Checkpointing mid-task so a long-running agent can resume exactly where it left off Cloning an environment that's already warm - dependencies installed, caches populated, services running Shipping a "ready-to-go" state that resumes in sub-second instead of cold-booting Snapshots are free during the preview, after which they will be stored as Azure Blob Storage at standard rates. Each snapshot records the source sandbox, resource allocation (CPU, memory, disk), and container metadata - so what you get back is exactly what you snapshotted. Resource Tiers Every sandbox is assigned to a resource tier that determines its CPU, memory, and disk allocation: Tier CPU Memory Disk XS 0.25 vCPU 0.5 GiB 5 GiB S 0.5 vCPU 1 GiB 10 GiB M (default) 1vCPU 2 GiB 20 GiB L 2 vCPU 4 GiB 40 GiB XL 4 vCPU 8 GiB 80 GiB When creating a sandbox from a snapshot, the resource tier is inherited from the snapshot and cannot be changed - this ensures the restored environment has the exact resources it was running with when the snapshot was taken. Lifecycle Policies: Auto-Suspend and Auto-Delete Every sandbox can be configured with lifecycle policies that automate state transitions and cleanup: Auto-Suspend Idle timeout: How long a sandbox can sit idle before being suspended (configurable: 1m, 2m, 5m, 10m, 30m, 60m) Suspend mode: Disk + Memory (default): Full snapshot including memory state - resume picks up exactly where you left off, with all processes and in-memory data intact. Disk: Only the disk is preserved; the VM restarts fresh on resume. Useful when you only need file persistence, not process continuity. Auto-Delete Automatically delete sandboxes after a configurable number of days of inactivity Prevents accumulation of abandoned sandboxes that consume snapshot storage These lifecycle policies are what make Sandboxes economically viable at scale. A platform serving thousands of tenants can configure aggressive idle timeouts (say, 60 seconds) with Memory suspend mode, and each tenant's sandbox disappears from the billing meter almost immediately - but resumes in sub-second time the moment they return. Network Egress Policy For scenarios involving untrusted code - AI agents executing LLM-generated scripts, multi-tenant SaaS with user-submitted workloads - controlling outbound network access is critical. Sandboxes provide a per-sandbox Network Egress Policy: Default action: Allow or Deny all outbound traffic Host rules: Domain-pattern rules (e.g., *.github.com → Allow) to permit specific destinations Custom CIDR rules: Network-level rules for IP ranges (e.g., 10.0.0.0/8 → Deny) Skip egress proxy: Option to bypass the egress proxy entirely when custom VNet routing handles policy enforcement This means you can run a sandbox in a deny-by-default posture and allowlist only the specific endpoints it needs (your API server, a package registry, etc.) - without setting up NSGs or firewall appliances. Managed Volumes: Persistent and Shared Storage Sandboxes support two types of mountable volumes, both managed by Microsoft: Volume Type Backed By Best For Managed Azure Blob Azure Blob Storage Shared data across sandboxes, file uploads/downloads, persistent artifacts Managed Data Disk Azure Disk Storage High-performance storage for databases, build caches, large working sets - only available to one sandbox at a time Blob volumes come with a built-in file explorer in the portal - you can browse, upload, download, create folders, and drag-and-drop files directly. Data Disk volumes provide dedicated block storage with configurable sizes. Secrets and Identity Secrets Sandbox Groups support key-value secrets scoped to the group. Secrets can be created, edited, and referenced by sandboxes within the group. These secrets can be used in egress policies to modify requests with transform or header-injection rules, without exposing the secrets to code running inside the sandbox. Managed Identity Sandbox Groups support both system-assigned and user-assigned managed identities, with full RBAC role assignment management. This means your sandboxes can authenticate to Azure services (Key Vault, Storage, Cosmos DB, etc.) without managing credentials - the same identity model you use everywhere else in Azure. MCP Connectors and Triggers ACA Sandboxes now supports managed connectors through the Model Context Protocol (MCP), giving sandboxes access to external APIs - including Microsoft 365, Salesforce, ServiceNow, GitHub, and 1,400+ other systems - without managing credentials directly. Attach a Connector Gateway to your sandbox group, and every sandbox in the group can call external APIs through a standardized MCP interface at runtime. Pair connectors with triggers to build event-driven automation: route an Outlook email to a sandbox that triages it with an AI agent, or react to a SharePoint file upload by extracting and processing the document all without writing glue code. Triggers can fire a shell command inside a sandbox or invoke an HTTP endpoint the sandbox exposes, so your automation shapes fit naturally around your workload. The integration is built on the new Connector Namespace service (az connector-namespace), the same runtime behind Logic Apps and Power Platform connectors, now available as a programmable layer for sandboxes. See the end-to-end samples for runnable azd up-deployable examples covering email triage and document automation scenarios. The Portal Experience Azure Container Apps Sandboxes are only available in the new Azure Container Apps portal that provides a rich, IDE-like experience for working with sandboxes. Creating a Sandbox The portal offers multiple creation paths: Standard Sandbox - full configuration control over source, resources, lifecycle, networking, and volumes GitHub Copilot Sandbox - preset, Copilot CLI ready to go, GitHub credentials can be wired through the Access Token before the sandbox is created Claude Sandbox - Claude CLI pre-installed, ready for agentic coding inside the sandbox Using Coding Agents (Copilot CLI / Claude Code) If you live inside Copilot CLI or Claude Code, you don't need to learn a new CLI. Install the azure-sandbox skill once and your agent picks up the right skills: # GitHub Copilot CLI # Add as a plugin marketplace /plugin marketplace add microsoft/azure-container-apps # Install all skills /plugin install sandboxes@Azure-Container-Apps # Claude Code claude plugin add microsoft/azure-container-apps The skill runs prerequisite checks silently (az --version, az account show, node --version, aca --version), prompts only if something's missing, and maps natural-language asks to the right aca commands. Bundled runbooks cover Copilot CLI BYOK (bring your own Azure OpenAI key), the deploy-a-web-app walkthrough, and shell setup. Sandbox Detail Page Once your sandbox is running, the detail page gives you immediate access to the sandbox terminal and additional details, such as - Network Audit - real-time egress traffic log showing allowed and denied requests Monitor - live CPU, memory, disk, and network utilization charts Connectors - attached connections with an "Add" action Volumes - mounted volumes with an "Add" action Log Stream - streaming container logs Processes - running process list inside the sandbox Files - file explorer to browse the sandbox filesystem The toolbar actions let you manage the state of the sandbox - Resume or Stop. In the Ellipsis menu (⁝) you can find additional settings to manage network Egress Policy and ingress (Add port), take a Snapshot of the sandbox, Commit (save disk state as a new disk image), set Lifecycle Policy or permanently Delete the sandbox. Finally, you can see additional Details in a side panel. Getting Started with the CLI and Python SDK All sandbox and sandbox-group operations go through the aca CLI. There are no az containerapp sandbox commands, - az is only used for az login, az account show, and resource-group management. Install (CLI) # Mac, Linux curl -fsSL https://aka.ms/aca-cli-install | sh # Windows irm https://aka.ms/aca-cli-install-ps | iex Run aca --help to get started. Install (Python SDK) pip install azure-containerapps-sandbox For more details, quick start and examples on ACA CLI and Python SDK, please go to https://sandboxes.azure.com Evolution from Dynamic Sessions If you've used Azure Container Apps Dynamic Sessions, Sandboxes are the next evolution of that capability. Everything Sessions can do, Sandboxes can do - and significantly more: Capability Dynamic Sessions Sandboxes Sub-second startup ✓ ✓ Strong isolation ✓ ✓ Custom container images ✓ ✓ Custom VNet integration ✓ (Partial) ✓ Suspend/resume with Memory and Disk snapshots - ✓ Lifecycle policies (auto-suspend, auto-delete) - ✓ Network egress policy (per-sandbox) - ✓ Persistent managed volumes (Blob, Data Disk) - ✓ Managed identity (system + user-assigned) - ✓ Secrets management - ✓ Configurable resource tiers - ✓ Direct access to sandbox in Portal experience - ✓ We will continue to support Dynamic Sessions, but all new investment goes into Sandboxes. If you're building new workloads on isolated ephemeral compute, start with Sandboxes. How It All Fits Together ACA Sandboxes is a platform primitive. It's the foundation on which multiple Microsoft products are already built - including ACA Express, Cloud sandboxes in GitHub Copilot, and Foundry Hosted Agents. When you build on Sandboxes, you're building on the same infrastructure that powers Microsoft's own portfolio. This is the evolution of what we shared with Project Legion in 2024. Legion described the internal infrastructure; Sandboxes exposes it as a customer-facing primitive that you can use directly. What's Next • Deeper Azure integrations - first-class connectivity with Azure networking, identity, storage, and AI services • Enhanced SDK and CLI - richer programmatic experiences for managing sandboxes at scale • More Microsoft services built on Sandboxes - this is just the beginning Get Started Today • Portal: https://sandboxes.azure.com/ • Documentation: Azure Container Apps Sandboxes • Pricing: Azure Container Apps Pricing (per-second vCPU/memory billing, scale-to-zero, snapshots at Blob Storage rates) We'd love to hear your feedback. You can ask questions, or file issues on the Azure Container Apps GitHub (prefix with [Sandbox] for Sandboxes-specific issues).Meet the Hosted Skills Canvas: Build, Run, and Debug in GitHub Copilot
Build, run, and debug event-driven AI apps right inside GitHub Copilot. The new Azure Functions Hosted Skills canvas brings instructions, triggers, and live results into one workspace—so you can spend less time switching tools and more time building.
Bulletproof agents with the durable task extension for Microsoft Agent Framework
Today, we're thrilled to announce the public preview of the durable task extension for Microsoft Agent Framework. This extension transforms how you build production-ready, resilient and scalable AI agents by bringing the proven durable execution (survives crashes and restarts) and distributed execution (runs across multiple instances) capabilities of Azure Durable Functions directly into the Microsoft Agent Framework. Now you can deploy stateful, resilient AI agents to Azure that automatically handle session management, failure recovery, and scaling, freeing you to focus entirely on your agent logic. Whether you're building customer service agents that maintain context across multi-day conversations, content pipelines with human-in-the-loop approval workflows, or fully automated multi-agent systems coordinating specialized AI models, the durable task extension gives you production-grade reliability, scalability and coordination with serverless simplicity. Key features of the durable task extension include: Serverless Hosting: Deploy agents on Azure Functions with auto-scaling from thousands of instances to zero, while retaining full control in a serverless architecture. Automatic Session Management: Agents maintain persistent sessions with full conversation context that survives process crashes, restarts, and distributed execution across instances Deterministic Multi-Agent Orchestrations: Coordinate specialized durable agents with predictable, repeatable, code-driven execution patterns Human-in-the-Loop with Serverless Cost Savings: Pause for human input without consuming compute resources or incurring costs Built-in Observability with Durable Task Scheduler: Deep visibility into agent operations and orchestrations through the Durable Task Scheduler UI dashboard Click here to create and run a durable agent # Python endpoint = os.getenv("AZURE_OPENAI_ENDPOINT") deployment_name = os.getenv("AZURE_OPENAI_DEPLOYMENT_NAME", "gpt-4o-mini") # Create an AI agent following the standard Microsoft Agent Framework pattern agent = AzureOpenAIChatClient( endpoint=endpoint, deployment_name=deployment_name, credential=AzureCliCredential() ).create_agent( instructions="""You are a professional content writer who creates engaging, well-structured documents for any given topic. When given a topic, you will: 1. Research the topic using the web search tool 2. Generate an outline for the document 3. Write a compelling document with proper formatting 4. Include relevant examples and citations""", name="DocumentPublisher", tools=[ AIFunctionFactory.Create(search_web), AIFunctionFactory.Create(generate_outline) ] ) # Configure the function app to host the agent with durable session management app = AgentFunctionApp(agents=[agent]) app.run() // C# var endpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT"); var deploymentName = Environment.GetEnvironmentVariable("AZURE_OPENAI_DEPLOYMENT") ?? "gpt-4o-mini"; // Create an AI agent following the standard Microsoft Agent Framework pattern AIAgent agent = new AzureOpenAIClient(new Uri(endpoint), new DefaultAzureCredential()) .GetChatClient(deploymentName) .CreateAIAgent( instructions: """You are a professional content writer who creates engaging, well-structured documents for any given topic. When given a topic, you will: 1. Research the topic using the web search tool 2. Generate an outline for the document 3. Write a compelling document with proper formatting 4. Include relevant examples and citations""", name: "DocumentPublisher", tools: [ AIFunctionFactory.Create(SearchWeb), AIFunctionFactory.Create(GenerateOutline) ]); // Configure the function app to host the agent with durable thread management // This automatically creates HTTP endpoints and manages state persistence using IHost app = FunctionsApplication .CreateBuilder(args) .ConfigureFunctionsWebApplication() .ConfigureDurableAgents(options => options.AddAIAgent(agent) ) .Build(); app.Run(); Why the durable task extension? As AI agents evolve from simple chatbots to sophisticated systems handling complex, long-running tasks, new challenges emerge: Conversations span multiple days and weeks, requiring persistent state across process restarts, crashes, and disruptions. Tool calls might take longer than typical timeouts allow, needing automatic checkpointing and recovery. High-volume workloads require elastic scaling across distributed instances to handle thousands of concurrent agent conversations. Multiple specialized agents need coordination with predictable, repeatable execution for reliable business processes. Agents sometimes must wait for human approval before proceeding, ideally without consuming resources. The Durable Extension addresses these challenges by extending Microsoft Agent Framework with capabilities from Azure Durable Functions, enabling you to build AI agents that survive failures, scale elastically, and execute predictably through durable and distributed execution. The extension is built on four foundational value pillars, which we refer to as the 4D’s: Durability Every agent state change (messages, tool calls, decisions) is durably checkpointed automatically. Agents survive and automatically resume from infrastructure updates, crashes, and can be unloaded from memory during long waiting periods without losing context. This is essential for agents that orchestrate long-running operations or wait for external events. Distributed Agent execution is accessible across all instances, enabling elastic scaling and automatic failover. Healthy nodes seamlessly take over work from failed instances, ensuring continuous operation. This distributed execution model allows thousands of stateful agents to scale up and run in parallel. Deterministic Agent orchestrations execute predictably using imperative logic written as ordinary code. Define the execution path, enabling automated testing, verifiable guardrails, and business-critical workflows that stakeholders can trust. This complements agent-directed workflows by providing explicit control flow when needed. Debuggability Use familiar development tools (IDEs, debuggers, breakpoints, stack traces, and unit tests) and programming languages to develop and debug. Your agent and agent orchestrations are expressed as code, making them easily testable, debuggable, and maintainable. Features in action Serverless hosting Deploy agents to Azure Functions (with expansion to other Azure computes soon) with automatic scaling to thousands of instances or down to zero when not in use. Pay only for the compute resources you consume. This code-first deployment approach gives you full control over the compute environment while maintaining the benefits of a serverless architecture. # Python endpoint = os.getenv("AZURE_OPENAI_ENDPOINT") deployment_name = os.getenv("AZURE_OPENAI_DEPLOYMENT_NAME", "gpt-4o-mini") # Create an AI agent following the standard Microsoft Agent Framework pattern agent = AzureOpenAIChatClient( endpoint=endpoint, deployment_name=deployment_name, credential=AzureCliCredential() ).create_agent( instructions="""You are a professional content writer who creates engaging, well-structured documents for any given topic. When given a topic, you will: 1. Research the topic using the web search tool 2. Generate an outline for the document 3. Write a compelling document with proper formatting 4. Include relevant examples and citations""", name="DocumentPublisher", tools=[ AIFunctionFactory.Create(search_web), AIFunctionFactory.Create(generate_outline) ] ) # Configure the function app to host the agent with durable session management app = AgentFunctionApp(agents=[agent]) app.run() Automatic session management Agent sessions are automatically checkpointed in durable storage that you configure in your function app, enabling durable and distributed execution across multiple instances. Any instance can resume an agent's execution after interruptions or process failures, ensuring continuous operation. Under the hood, agents are implemented as durable entities. These are stateful objects that maintain their state across executions. This architecture enables each agent session to function as a reliable, long-lived entity with preserved conversation history and context. Example scenario: A customer service agent handling a complex support case over multiple days and weeks. The conversation history, context, and progress are preserved even if the agent is redeployed or moves to a different instance. # First interaction - start a new thread to create a document curl -X POST https://your-function-app.azurewebsites.net/api/agents/DocumentPublisher/threads \ -H "Content-Type: application/json" \ -d '{"message": "Create a document about the benefits of Azure Functions"}' # Response includes thread ID and initial document outline/draft # {"threadId": "doc789", "response": "I'll create a comprehensive document about Azure Functions benefits. Let me search for the latest information... [Document Draft] # Benefits of Azure Functions\n\n## Introduction\nAzure Functions is a serverless compute service that enables you to run event-driven code without managing infrastructure...\n\n## Cost Efficiency\n- Pay only for execution time\n- No charges for idle resources\n- Automatic scaling reduces over-provisioning...\n\n## Developer Productivity\n- Multiple language support (C#, Python, JavaScript, Java)\n- Integrated development tools and CI/CD...\n\n## Scalability\n- Automatic scaling based on demand\n- Handle millions of requests seamlessly...\n\nReferences: [Azure documentation, serverless computing studies]"} # Second interaction - continue the same thread to refine the document curl -X POST https://your-function-app.azurewebsites.net/api/agents/DocumentPublisher/threads/doc789 \ -H "Content-Type: application/json" \ -d '{"message": "Can you add a section about integration with other Azure services?"}' # Agent remembers the Azure Functions document context and adds the requested section # {"threadId": "doc789", "response": "I've added a comprehensive integration section to your Azure Functions document:\n\n## Integration with Azure Services\n\n### Azure Storage\nSeamless triggers and bindings for Blob Storage, Queue Storage, and Table Storage enable event-driven architectures...\n\n### Azure Event Grid & Event Hubs\nProcess real-time event streams and implement publish-subscribe patterns at scale...\n\n### Azure Cosmos DB\nBuilt-in bindings for document database operations with automatic change feed processing...\n\n### Azure Service Bus\nReliable message processing with enterprise messaging features...\n\n### Azure AI Services\nEasily integrate OpenAI, Cognitive Services, and AI Search for intelligent applications...\n\nThe document now includes this section after the Scalability section. Would you like me to add use cases or deployment best practices?"} Deterministic multi-agent orchestrations Coordinate multiple specialized durable agents using imperative code where you define the control flow. This differs from agent-directed workflows where the agent decides the next steps. Deterministic Orchestrations provide predictable, repeatable execution patterns with automatic checkpointing and recovery. Example scenario: An email processing system that uses a spam detection agent, then conditionally routes to different specialized agents based on the classification. The orchestration automatically recovers if any step fails and completed agent calls are not re-executed. # Python app.orchestration_trigger(context_name="context") def document_publishing_orchestration(context: DurableOrchestrationContext): """Deterministic orchestration coordinating multiple specialized agents.""" doc_request = context.get_input() # Get specialized agents from the orchestration context research_agent = context.get_agent("ResearchAgent") writer_agent = context.get_agent("DocumentPublisherAgent") # Step 1: Research the topic using web search research_result = yield research_agent.run( messages=f"Research the following topic and gather key information: {doc_request.topic}", response_schema=ResearchResult ) # Step 2: Generate outline based on research findings outline = yield context.call_activity("generate_outline", { "topic": doc_request.topic, "research_data": research_result.findings }) # Step 3: Write the document with the research and outline document = yield writer_agent.run( messages=f"""Create a comprehensive document about {doc_request.topic}. Research findings: {research_result.findings} Outline: {outline} Write a well-structured, engaging document with proper formatting and citations.""", response_schema=DocumentResponse ) # Step 4: Save and publish the generated document return yield context.call_activity("publish_document", { "title": doc_request.topic, "content": document.text, "citations": document.citations }) Human-in-the-loop Orchestrations and agents can pause for human input, approval, or review without consuming compute resources. Durable execution enables orchestrations to wait for days or even weeks while waiting for human responses, even if the app crashes or restarts. When combined with serverless hosting, all compute resources are spun down during the wait period, eliminating compute costs until the human provides their input. Example scenario: A content publishing agent that generates drafts, sends them to human reviewers, and waits days for approval without running (or paying for) compute resources during the review period. When the human response arrives, the orchestration automatically resumes with full conversation context and execution state intact. # Python app.orchestration_trigger(context_name="context") def content_approval_workflow(context: DurableOrchestrationContext): """Human-in-the-loop workflow with zero-cost waiting.""" topic = context.get_input() # Step 1: Generate content using an agent content_agent = context.get_agent("ContentGenerationAgent") draft_content = yield content_agent.run(f"Write an article about {topic}") # Step 2: Send for human review yield context.call_activity("notify_reviewer", draft_content) # Step 3: Wait for approval - no compute resources consumed while waiting approval_event = context.wait_for_external_event("ApprovalDecision") timeout_task = context.create_timer(context.current_utc_datetime + timedelta(hours=24)) winner = yield context.task_any([approval_event, timeout_task]) if winner == approval_event: timeout_task.cancel() approved = approval_event.result if approved: result = yield context.call_activity("publish_content", draft_content) return result else: return "Content rejected" else: # Timeout - escalate for review result = yield context.call_activity("escalate_for_review", draft_content) return result Built-in agent observability Configure your Function App with the Durable Task Scheduler as the durable backend (what persists agents and orchestration state). The Durable Task Scheduler is the recommended durable backend for your durable agents, offering the best throughput performance, fully managed infrastructure, and built-in observability through a UI dashboard. The Durable Task Scheduler dashboard provides deep visibility into your agent operations: Conversation history: View complete conversation threads for each agent session, including all messages, tool calls, and conversation context at any point in time Multi-agent visualization: See the execution flow when calling multiple specialized agents with visual representation of agent handoffs, parallel executions, and conditional branching Performance metrics: Monitor agent response times, token usage, and orchestration duration Execution history: Access detailed execution logs with full replay capability for debugging Demo Video Language support The Durable Extension supports: C# (.NET 8.0+) with Azure Functions Python (3.10+) with Azure Functions Support for additional computes coming soon. Get started today Click here to create and run a durable agent Learn more Overview documentation C# Samples Python SamplesMicrosoft Foundry Hosted Agents and MCP in Practice: Building Fibey Field Ops
An agent can produce a convincing answer while the system around it is still difficult to deploy, authorize, debug, and recover. For AI engineers and developers, that is often the real gap between a promising prototype and an application people can depend on. Fibey Field Ops makes that gap concrete. It is a synthetic fiber-operations assistant built with Microsoft Foundry Hosted Agents, Model Context Protocol (MCP), and Azure Container Apps. This walkthrough follows one field-service task through the implementation, then examines the deployment and operational decisions behind it. Introduction: the application is more than the model Imagine a technician preparing a fiber work order. Before leaving the depot, they need the job details, available parts, relevant procedures, and network status. Those facts belong to different systems. A useful assistant must retrieve them, combine them, and explain what is missing without inventing an answer. The challenge is not simply selecting a capable model. It is establishing reliable contracts between the model, its tools, the hosting platform, and the application. Fibey demonstrates those contracts with one hosted agent, five instruction skills, eleven operational tools, and five supporting Container Apps. There is an important qualification: Fibey is a protected, synthetic-data demonstration, not a production-ready field-service system. Its gateway mappings and work orders remain in memory, and it does not enforce per-user session ownership or approval of operational writes. Those limitations are useful teaching material rather than details to hide. The Fibey repository contains the implementation, infrastructure, documentation, and presentation deck. Repository access depends on its permissions. 1. Separate reasoning, instructions, and tool execution MCP is a standard interface for discovering and invoking tools. In Fibey, the agent connects to one Microsoft Foundry Toolbox MCP endpoint. The toolbox exposes capabilities backed by an inventory MCP server, a work-orders OpenAPI service, and a Search knowledge base. That common interface does not make the underlying systems identical. OpenAPI still describes an HTTP API, inventory still implements MCP, and knowledge retrieval still depends on indexed documents. The toolbox centralizes the agent-facing integration and connection configuration while preserving those implementation choices. Four terms describe different responsibilities: Concept Responsibility in Fibey Agent Classifies the request, loads instructions, selects tools, and constructs the response Skill An instruction document for a task such as inventory lookup or field briefing Tool An operation with an advertised input schema and a result Toolbox The curated MCP surface and references to downstream connections Fibey's five skills cover inventory lookup, work-order management, knowledge retrieval, work-order preparation, and field briefings. The last two coordinate several tools; they do not create additional agents. The configured toolbox exposes eleven operational tools directly: six inventory/status operations, four work-order operations, and one knowledge-retrieval operation. The agent uses their actual names and schemas. Discovery wrappers such as tool_search and call_tool are relevant only when a toolbox exposes them; they are not mandatory steps before every call. This distinction matters when debugging. A missing wrapper is not necessarily a broken integration. A skill mentioning a capability is also not proof that the runtime can invoke it. The current tool schema is the executable contract. 2. Follow the hosted Azure architecture Microsoft Foundry hosts the agent container and exposes its endpoint. Azure Container Apps (ACA) hosts the application services around it. The deployment source of truth is azure.yaml , which declares the GPT-5.4-mini model deployment and the hosted agent's Responses 2.0.0 protocol. The design keeps the browser-facing application separate from agent execution and backend integration. That makes it easier to inspect each boundary, but it does not make the whole deployment private or remove the need for application authorization. Treat GPT-5.4-mini as the sample's configured baseline, not a claim that it is optimal for every workload. When evaluating another model, measure tool-selection accuracy, schema compliance, grounded answers, latency, and cost per completed task rather than choosing from a fluent demo response alone. The engineering architecture expands this presentation view with resource ownership, identities, configuration, and telemetry paths. The application request path The browser signs in through Microsoft Entra ID at the UI's ACA authentication boundary. An explicit allowlist restricts access to the intended user. Nginx serves the React application and proxies chat requests to the internal FastAPI gateway. The gateway invokes the Foundry-hosted agent using its managed identity. Server-sent events (SSE), a streaming HTTP format, carry answer text, tool activity, citations, failures, and completion back to the UI. Gateway and dashboard ingress are internal to the ACA environment. Inventory and work orders have external ingress so the toolbox can reach them, but their operational endpoints require separate API keys. External accessibility is not anonymous access, and internal ingress is not a complete network-isolation strategy. The knowledge and status paths The knowledge pipeline starts with eight Markdown documents in the repository. A setup script uploads them to a private Blob Storage container, configures a Search data source and indexer, verifies ingestion, and creates the knowledge source and knowledge base used by Foundry IQ. This is not an embedding pipeline assembled implicitly by the chat application. The current configuration uses minimal reasoning and extractive retrieval, with defaults of three output documents and 6,000 output tokens. Its knowledge-base configuration and MCP endpoint use preview APIs, which need lifecycle and support review before production adoption. Network status takes a different path. Inventory's get_network_status tool reads the configured internal HTML dashboard. It is a narrow HTTP fetch, not browser automation, arbitrary website navigation, or a real operational clearance. Azure Container Registry supplies container images. ACA logs go to Log Analytics. Hosted code enables OpenTelemetry, the standard instrumentation framework for traces and metrics, but the repository does not provision Application Insights. The project's linked trace destination must be configured and verified separately. 3. Trace a work-order briefing through the implementation The most useful demonstration starts with one concrete request: prepare a technician for a job. The field-briefing skill provides the instructions for combining work-order data, inventory, procedures, and status without pretending that one backend contains everything. Use a request such as the following in a prepared synthetic environment. The exact wording and tool order may vary; the important evidence is which operations succeeded and which facts support the answer. Brief me on WO-007, including stock, relevant procedures, safety, and network status. The intended flow is: Load the field-briefing instructions and retrieve WO-007. Identify required parts and use a batch stock check when several parts need checking. Combine procedure and safety questions into one focused knowledge retrieval. Read the configured synthetic status dashboard through inventory MCP. Produce a briefing grounded in successful results, with source references and explicit gaps. Batching is a useful engineering choice, not a claim of a measured performance improvement. One batch stock call avoids unnecessary repeated requests. Combining related retrieval questions can also reduce duplicate context and tool traffic. This screenshot was captured from an authenticated deployment. The displayed briefing identifies an unavailable connector kit and available test equipment. It illustrates a specific synthetic response, not current stock or a benchmark. Use the supported hosted integration The relevant implementation is the hosted entrypoint. It uses FoundryToolbox from agent_framework_foundry_hosting , a FoundryChatClient , a skills provider, and ResponsesHostServer.run_async() . The hosting helper does more than attach a bearer token. It authenticates MCP requests and forwards the hosted runtime's per-request call ID. Replacing it with a generic transport can lose context that the platform expects. The entrypoint also closes credentials and clients when execution exits. Hosted skill discovery prefers published skills when available and retains bundled instructions as a fallback in auto mode. Explicit mcp mode fails if published skills cannot be loaded; file mode uses the bundled documents. This makes the fallback intentional rather than silently running without the task instructions. Distinguish history from compute affinity The gateway maintains two hosted mappings. previous_response_id links conversation history, while agent_session_id preserves affinity to the hosted compute session. Losing one is not equivalent to losing the other. Both mappings are held in gateway memory, which is why it remains at one replica. A UUID (universally unique identifier) is a conversation handle, not proof of ownership. Reset clears the local mappings but does not reset work orders or delete the old remote compute session. There is also a privacy distinction between storage and telemetry. Gateway requests use stored Responses history, while the agent's model-call options use store: false . Sensitive tracing being disabled does not mean all conversation persistence is disabled. 4. Run locally without confusing development and cloud boundaries Local development is useful for inspecting the gateway and agent without rebuilding the hosted image. It still calls a real Foundry project, model, and toolbox. It is not an offline simulation, and tool writes can affect the configured synthetic backend. Use Python 3.12+, uv, Node.js 24 LTS, and an authorized Azure developer identity. The commands below run from the repository root in PowerShell. Keep the existing dependency locks and copy .env.example only when creating a new local configuration. uv sync --frozen Copy-Item .env.example .env Set FOUNDRY_PROJECT_ENDPOINT , FOUNDRY_MODEL , and TOOLBOX_MCP_URL in the ignored .env . Use your environment's actual values. Backend API keys belong in Foundry connections, not in the browser or prompts. Start the gateway: az login uv run --frozen uvicorn fibey.gateway.api_server:app --host 127.0.0.1 --port 8080 In another terminal, start the UI: Set-Location ui npm ci npm run dev Open http://localhost:5173 . Vite forwards /api to the gateway on port 8080. These development servers do not reproduce the cloud Entra boundary, and a cloud toolbox cannot reach your workstation's localhost . Inspect the API progressively With the local gateway running, start with a health request in a separate PowerShell terminal: $base = "http://127.0.0.1:8080" Invoke-RestMethod "$base/api/health" Next, create a UUID conversation and submit one synthetic request: $session = [guid]::NewGuid().ToString() $body = @{ message = "Show me WO-007."; session_id = $session } | ConvertTo-Json -Compress $response = Invoke-WebRequest "$base/api/chat" -Method Post -ContentType "application/json" -Body $body $response.Headers["X-Session-Id"] $response.Content This prints the completed SSE body rather than animating the stream. Reuse the same session for a follow-up by extracting a small helper: function Invoke-FibeyTurn { param([string] $Message, [string] $SessionId) $payload = @{ message = $Message; session_id = $SessionId } | ConvertTo-Json -Compress $result = Invoke-WebRequest "$base/api/chat" -Method Post ` -ContentType "application/json" -Body $payload return $result.Content } Invoke-FibeyTurn -Message "What parts does that work order need?" -SessionId $session The helper uses $base and $session from the preceding examples. It makes continuity explicit without hiding the API contract. The browser remains the better place to watch incremental text and activity. See local development for reset behavior and supporting-service details. 5. Deploy artifacts and infrastructure together Fibey's initial deployment is intentionally staged. A container image, its target port, its health probes, its registry association, and its runtime permissions must agree. Successfully building an image does not establish that agreement. The first supporting-infrastructure pass creates placeholder apps on port 80. Once all five real images have been published, a second pass applies the images with their application ports and probes. Plain azd deploy of a supporting service is not the initial port-switch mechanism for this sample. The deployment guide gives the complete sequence and prerequisites: Select the intended environment and provision the Foundry layer. Configure project aliases, the Entra application, the allowed user, and separate API keys. Provision placeholder supporting apps and verify scoped access and registry identity associations. Publish all five supporting images and confirm every image setting is populated. Provision supporting infrastructure again to apply the matching images, ports, and probes. Run knowledge setup, then toolbox setup. Deploy fibey-agent and perform protected-path acceptance checks. For example, this publication step comes after placeholders and access checks, not at the start of an unconfigured environment: azd publish status-dashboard azd publish inventory-mcp azd publish work-orders-api azd publish gateway azd publish ui Only after all five SERVICE_*_IMAGE_NAME settings exist should the next azd provision infra apply them. Keep those settings: an empty value selects a placeholder again. This is where DevOps becomes tangible. Review source, Bicep, locks, schemas, skills, and configuration together; record accepted image references, model configuration, toolbox version, and hosted-agent version. GitOps adds controlled reconciliation of reviewed desired state. The repository supplies the ingredients, not an existing CI/CD pipeline or GitOps controller. Toolbox promotion deserves the same discipline. The unversioned consumer endpoint follows the published default version. A version-specific developer endpoint can test a candidate before promotion. Updating a shared connection or default can affect consumers without rebuilding their images, so Git history alone is not a rollback mechanism. 6. Treat governance, observability, and scale as separate concerns Role-based access control (RBAC) grants an identity permission at a resource scope. The control plane creates and configures Azure resources; the data plane performs application work such as invoking agents, querying Search, or reading blobs. The identities used across those operations are not interchangeable. In particular, a successful UI login is not automatic end-user identity passthrough to every tool, and resource provisioning permissions do not establish all runtime permissions. Boundary Identity or credential Browser to UI Entra user, application registration, and user allowlist Gateway to Foundry Gateway managed identity with project-scoped invocation access Hosted agent to model and toolbox Foundry-provided runtime agent identity Toolbox to inventory and work orders Separate API keys stored in project connections Toolbox to Search knowledge base Foundry project managed identity Search indexer to documents Search managed identity with Blob reader access Image delivery adds another boundary: each supporting app needs both AcrPull and a registry association selecting its identity. The Foundry project identity used for infrastructure operations is also distinct from the hosted agent's runtime identity. Approval must exist outside the prompt Fibey's synthetic work-order writes run without an enforced human approval round trip. That is appropriate to understand in a disposable demo and inappropriate to conceal when discussing production. The hosted toolbox documentation is explicit: approval metadata alone does not block tools/call . The runtime must pause, collect a decision, and resume or reject the exact proposed operation. A prompt saying "ask first" is not equivalent to that control. Real writes need server-side authorization, approval tied to the arguments, idempotency, and a durable audit record. If a write's response is uncertain, read its state before retrying. Treat tool results as untrusted data rather than instructions. Debug the boundary that failed The activity sidebar makes tool use visible, but it is not a durable audit log or access to the model's hidden reasoning. Combine it with timestamps, request/session identifiers, ACA logs, and configured hosted traces. Keep sensitive message tracing off unless a controlled investigation explicitly requires it. Verify that a synthetic trace reaches the intended sink; enabled instrumentation alone does not prove ingestion. Symptom First checks UI returns 502 Gateway revision, target port, fixed Nginx upstream, SNI, and certificate trust Agent or tool returns 401/403 Identity, token audience, role scope, and downstream connection credential Knowledge retrieval fails Indexer completion, project Search role, API version, and advertised input schema Briefing receives 429 Model quota, concurrency, retrieval budgets, and repeated tool calls Stream ends early Terminal response event, timeout, malformed SSE, and transport failure The gateway must surface failed or truncated streams rather than treating partial text as a completed action. A final stream terminator after an error is not application success. Scale only after identifying state and capacity limits Foundry manages hosted runtime capabilities, but it does not externalize Fibey's gateway mappings or in-memory work orders. Adding replicas before redesigning that state would undermine continuity and consistent updates. Model throughput, hosted compute, downstream API capacity, Search, and ACA are separate constraints. Measure latency, errors, tokens, tool counts, and recovery under concurrent load. Registry storage/builds, model inference, compute, Search, Blob Storage, and telemetry all have costs; using a managed platform does not remove the need for budgets. 7. Evaluate the result before adding more agents The practical result is an inspectable workflow that combines heterogeneous systems into a grounded response. We can demonstrate inventory lookup, a synthetic work order, cited knowledge, a fixed status fetch, and a combined briefing through one agent-facing toolbox. That is functional evidence, not a production certification or benchmark. This post does not establish latency percentiles, cost per successful task, throughput, or availability under load. Those measurements should be acceptance criteria for an adaptation, not numbers inferred from a screenshot. Multi-agent coordination is a possible next design, not deployed Fibey behavior. A coordinator could delegate read-only inventory and procedure tasks while a restricted specialist handles approved writes. Each handoff would need typed inputs/results, correlation IDs, deadlines, cancellation, and budgets. Sharing a toolbox alone does not implement that coordination. Specialists also introduce additional failure paths, identity decisions, and state. Start with them only when task complexity, ownership, or isolation requirements justify the cost. The current five skills already provide modular instructions without creating five independently operated agents. 8. Summary: reuse the engineering boundaries Fibey's reusable pattern is the separation of model reasoning, task instructions, tool contracts, hosted execution, and operational controls. MCP gives the agent a common tool interface; Foundry supplies managed hosting and integration capabilities. The application team remains responsible for the guarantees around its data and actions. Start with one workflow and verify each boundary. Then add durable state, per-user authorization, enforced approvals, secret rotation, appropriate networking, evaluation, and recovery before introducing real operational data or more autonomous behavior. For a practical starting point, use the repository, follow the deployment guide, and reproduce the session walkthrough with synthetic data. The presentation deck provides the same architecture and demo sequence for a team discussion. References The repository describes what this sample implements. Microsoft documentation describes the surrounding platform capabilities, responsibilities, and supported integration contracts. Read both before adapting the solution, especially where preview APIs, identity behavior, or approval enforcement affect your requirements. Fibey Field Ops repository Fibey engineering architecture Fibey deployment guide Fibey toolbox integration Microsoft Foundry hosted agents Use a toolbox with a hosted agent Official Python hosted-agent toolbox sample
Azure function require private git package
At the moment we are deploying our python application to a server-less azure function app. For this we use the kudu config-zip deployment. az functionapp deployment source config-zip -g "xxxx" -n "xxxx" --src "xxxx.zip" --build-remote We also want a remote build, because this will install the correct version of the packages. Because some packages have different versions voor different python versions (e.g. 3.8 vs 3.10) and different environment (windows vs linux). The remote build will make sure the correct packages are installed, cause the build (azure's default oryx) will run in the same environment. Recently we moved some of our code to another package. This package is shared by multiple other applications. To install it, we add it to our requirements.txt: git+ssh://Email address removed/xxxxx/xxxxx.git@f4e2bf2e3dxxxxxxxxx This works perfect on our local machines. But not once we deploy to azure. Unfortunately there are no logs. Well the logs shows "oryx build...." and that's it. There is no way to access the build logs. Anyway, we know the cause of the issue: the build doesn't have access to the repository. We do have a ssh key, which can be used to access the git repo. But we have no clue how to pass it to the orxy builder. We tried to make a work around with the "PRE_BUILD_COMMAND" environment variable, but since there are no logs, we cannot determine what is failing during the build. So we cannot install private python packages with azure serverless functions. We see two ways to solve this issue, but for neither we have a clue how to do it: Make the orxy builder use the ssh key Do a local build and push it to the azure function Did some tried this before or can give someone some pointers how to get started on this?Building Autonomous Agents with Microsoft Agent Framework and GitHub Copilot SDK Part 2/5
This is the second post in our series on the Microsoft agent platform. Here we dive deep into building autonomous agents, the development experience, the Microsoft Agent Framework, tool design patterns, and how the GitHub Copilot SDK brings conversational AI to your agent system. All examples reference the FibreOps repository, an autonomous fibre outage response system demonstrated at Microsoft Build BRK241. The Microsoft Agent Framework The Microsoft Agent Framework (now GA) provides a unified programming model for building agents. It supports multiple backends through a single .run() contract: Hosted — FoundryAgent connected to a Prompt Agent published to Microsoft Foundry Agent Service. Foundry — Agent + FoundryChatClient with the definition resolved locally (ideal for prompt iteration). Local — Deterministic LocalAgent for offline development and testing. This design means your orchestration code never changes regardless of where the agent runs. The factory pattern in FibreOps selects the backend at startup: # src/fibreops/agents/factory.py — simplified from agent_framework_foundry import FoundryAgent from agent_framework import Agent, FoundryChatClient def build_agent(role: str, backend: str, config: Config): if backend == "hosted": return FoundryAgent(agent_id=config.foundry_agents[role]) elif backend == "foundry": return Agent( instructions=get_instructions(role), chat_client=FoundryChatClient(endpoint=config.endpoint), tools=get_tools(role), ) else: return LocalAgent(role=role) Set FIBREOPS_AGENT_BACKEND to override the backend, or leave it as auto for intelligent detection. Designing Role-Specialised Agents FibreOps demonstrates a key pattern: role specialisation. Rather than one monolithic agent, the system uses three focused agents, each with a clear responsibility boundary: Agent Role Tools Available IncidentAnalysisAgent Classify severity, find root cause, retrieve SOP Knowledge (SOPs + topology), Web IQ, Work IQ NetOpsCoordinatorAgent File D365 incident, post Teams notice Ticketing, Teams, Memory FieldDispatchAgent Select engineer, book resource, update team Dispatch, Teams, Voice Why Role Specialisation? Focused system prompts — Each agent has a tightly scoped instruction set, reducing hallucination and improving reliability. Independent evaluation — You can score each agent separately against role-specific criteria. Parallel development — Teams can iterate on agents independently. Selective upgrade — Swap one agent's model or implementation without touching others. Tool Design: Typed Python Functions Tools in the Microsoft Agent Framework are typed Python functions that the runtime supplies to the hosted agent definition. FibreOps demonstrates several tool categories: Knowledge Tools # src/fibreops/tools/knowledge.py — simplified def sop_lookup(node_id: str, signal_type: str) -> dict: """Retrieve the Standard Operating Procedure for a given signal type. Args: node_id: The fibre node identifier (e.g., FN-LDN-001) signal_type: The type of signal (loss_of_light, high_ber, signal_degradation) Returns: SOP with steps, escalation path, and estimated resolution time. """ # Load from local markdown SOPs or Foundry IQ ... def web_iq_search(query: str, *, limit: int = 5) -> list[dict]: """Search public web for context relevant to the incident. Grounding against roadworks, weather, power outages, splice guidance. Falls back to deterministic fixtures when endpoint is unset. """ ... def work_iq_search(query: str, *, limit: int = 5) -> list[dict]: """Search enterprise knowledge for context relevant to the incident. Site surveys, SLA tiers, competency matrix, MTTR trends. """ ... Integration Tools # src/fibreops/tools/teams.py — simplified def post_outage_notice( incident_id: str, node_id: str, severity: str, summary: str, engineer: str | None = None, ) -> dict: """Post an Adaptive Card outage notice to the configured Teams channel. If TEAMS_WEBHOOK_URL is not set, appends to state/teams_outbox.jsonl for offline review. """ card = build_adaptive_card(incident_id, node_id, severity, summary, engineer) if config.teams_webhook_url: requests.post(config.teams_webhook_url, json=card) else: append_to_outbox(card) return {"status": "posted", "incident_id": incident_id} Design Principles for Agent Tools Typed parameters with docstrings — The runtime uses type hints and docstrings to generate the tool schema for the LLM. Graceful degradation — Every tool works offline by falling back to local fixtures or file-based state. Idempotent where possible — Tools that create resources return existing records if called with the same parameters. Observable — Every tool invocation emits an OpenTelemetry span for tracing and debugging. The Orchestrator Pattern The orchestrator drives signals through the agent pipeline. It is deliberately simple — a linear flow with error handling: # src/fibreops/orchestrator.py — simplified async def handle_signal(signal: TelemetrySignal) -> RunResult: """Process a telemetry signal through the agent pipeline.""" # Stage 1: Incident Analysis analysis = await incident_agent.run( f"Analyse this signal: {signal.model_dump_json()}" ) # Stage 2: NetOps Coordination coordination = await netops_agent.run( f"Coordinate response for: {analysis.summary}" ) # Stage 3: Field Dispatch dispatch = await dispatch_agent.run( f"Dispatch engineer for incident: {coordination.incident_id}" ) return RunResult( signal=signal, analysis=analysis, coordination=coordination, dispatch=dispatch, ) The orchestrator honours the same contract regardless of backend — hosted , foundry , or local — because all backends implement await agent.run(prompt) . GitHub Copilot SDK Integration (GA) The GitHub Copilot SDK enables conversational interaction with your agent system. FibreOps implements FibreOpsCopilotClient with the same interface as github/copilot-sdk : # src/fibreops/sdk/__init__.py — simplified from fibreops.sdk.client import FibreOpsCopilotClient client = FibreOpsCopilotClient() session = client.create_session() # Query agent status response = session.send_and_wait("status") print(response.text) # Human-readable summary print(response.data) # Structured JSON # Inject a telemetry signal via conversation response = session.send_and_wait(json.dumps({ "signal_id": "sig-demo", "node_id": "FN-LDN-001", "signal_type": "loss_of_light", "severity": "critical" })) The adapter routes prompts by shape: JSON signal-shaped dicts — Forwarded to the orchestrator for processing. Free-form text — Answered by a deterministic responder ( help , status , nodes , engineers , optimiser , dispatch ). Drive it from the terminal: python -m fibreops.demo chat "help" python -m fibreops.demo chat "status" python -m fibreops.demo chat '{"signal_id":"sig-demo","node_id":"FN-LDN-001","signal_type":"loss_of_light","severity":"critical"}' Or hit the embedded HTTP endpoint when the NOC console is running: Invoke-RestMethod -Method Post http://127.0.0.1:8800/sdk/chat -Body '{"prompt":"status"}' -ContentType application/json Development Workflow with Foundry Toolkit for VS Code The Foundry Toolkit for VS Code provides an integrated development experience: Author prompts — Edit system instructions with live preview and token counting. Test locally — Run against the foundry backend with FoundryChatClient pointing at your development model. Iterate fast — The foundry backend resolves definitions locally, so prompt changes take effect immediately without republishing. Publish when ready — python -m fibreops.demo publish creates hosted Prompt Agents in Foundry. Multi-Model Support The Microsoft Agent Framework supports multiple models. FibreOps defaults to gpt-4.1-mini (the model available in most demo Foundry accounts), but any chat-completions deployment works: # .env AZURE_AI_MODEL_DEPLOYMENT=gpt-4.1-mini # or gpt-4o-mini, gpt-4o, gpt-4.1 The framework also supports Claude Code connectors and Magentic-One for multi-agent collaboration scenarios. Testing Strategy FibreOps demonstrates a layered testing approach: Unit tests — Test tools in isolation with mocked dependencies. Local backend tests — Run the full pipeline with LocalAgent for deterministic assertions. Integration tests — Run against real Foundry agents with pytest -q . Rubric evaluation — The optimizer scores every run against defined criteria. # Run the test suite .\.venv\Scripts\python.exe -m pytest -q Key Takeaways The Microsoft Agent Framework provides a unified .run() contract across hosted, foundry, and local backends. Role specialisation keeps agents focused, testable, and independently evolvable. Tools are typed Python functions with docstrings — the runtime generates schemas automatically. The GitHub Copilot SDK (GA) enables conversational interaction with any agent system. Graceful degradation means the entire system works offline for development. The factory pattern lets you switch backends without changing orchestration code. Next Steps Clone the FibreOps repository and run python -m fibreops.demo --signals 3 Microsoft Agent Framework documentation Next in this series: Running Hosted Agents in Microsoft Foundry Agent Service
Vector search finds candidates. Reranking decides what your RAG app reads
You ask a retrieval-augmented generation (RAG) application a question. Vector search returns ten passages that are clearly related to the topic. The passage that actually contains the answer, however, is ranked seventh, while the language model receives only the first five. Retrieval did not completely fail. It found the evidence, but ordered it below less useful context. Reranking addresses that gap between a passage that is semantically similar and a passage that is relevant to the user's specific question. This article demonstrates that pattern in four Azure services using the Stanford Question Answering Dataset (SQuAD). The goal is not to declare a winning service or publish a quality benchmark. It is to show where retrieval, rank fusion, and model-based reranking run in each architecture, and to illustrate how the position of a known source passage can change. What this demonstration establishes The examples show rank movement for three selected questions. They do not establish that one reranker or service is universally more accurate. A production decision requires a larger, representative query set and aggregate relevance, latency, and cost measurements. Get the full Python implementation: pauldj54/azure-vector-reranking-squad Retrieval and reranking are different stages A production search pipeline commonly uses two stages: Retrieve for recall. Fast retrieval narrows a large corpus to a bounded candidate set. It can use vector search, keyword search, or both. Rerank for precision. A more expensive model evaluates only those candidates against the original query and produces the final order. Reciprocal Rank Fusion (RRF) belongs between those two ideas. RRF is a model-free rank aggregation method that merges independent result lists, usually vector and keyword results. For a document d, a typical score is: RRF(d) = ∑ r ∈ R 1 k + rank r (d) Here, R is the set of ranked lists and k is commonly 60. RRF works with positions rather than raw scores, so it can combine signals such as cosine distance and BM25 without pretending their score scales are comparable. This gives a clearer three-part vocabulary: Stage Purpose Typical mechanism Retrieve Find broad candidate set Vector search, BM25, filters Fuse Combine independent rankings RRF Rerank Reassess query-document relevance Semantic ranker or cross-encoder RRF often improves hybrid retrieval when exact names, dates, identifiers, or terms matter. A learned reranker can then read the query and each candidate together, capturing interactions that separately generated embeddings can miss. The learned stage costs more, so it should operate on tens of candidates rather than the whole corpus. The following image describes the general process: Why use SQuAD for this demonstration? SQuAD 1.1 contains crowd-written questions over more than 500 Wikipedia articles. Its packaged splits contain 87,599 training rows and 10,570 validation rows. Each row includes a question, a context passage, and one or more answer spans inside that passage. That source-context mapping gives this demonstration a useful label: the context associated with a question is treated as its gold passage. We can then inspect whether each search stage moves that passage up or down. This is convenient, but it is not a perfect passage-ranking benchmark. SQuAD was designed for extractive question answering, and another passage in the corpus might also answer a question. The gold context is therefore a reproducible reference, not proof that every other passage is irrelevant. The results shown here use the 2,067 unique contexts in the SQuAD validation split and 1,536-dimensional embeddings. The repository default should be set to the same corpus size before treating the screenshots or rank transitions as directly reproducible. Three illustrative questions Question Expected answer Gold context According to game stats, which Super Bowl 50 quarterback had his worst year since his first NFL season? Peyton Manning 12, Super Bowl 50 What else did Tesla do for work at this time? Various electrical repair jobs 165, Nikola Tesla Who acts as laborer, paymaster, and design team for a renovation project? The property owner 1306, Construction Each notebook selects a seeded demonstration question when it runs. The three saved examples were collected across separate runs; the current notebooks do not execute all three questions in one pass. A benchmark harness should iterate over a fixed question list and save all stage results in one structured output. Capability boundaries at a glance Service Retrieval and Fusion Learned Reranking Boundary to Keep in Mind Azure AI Search Native keyword and vector retrieval with native RRF Built-in semantic ranker Semantic ranking only reorders the retrieved top 50 Azure SQL Database Exact vector retrieval in the current notebook External Cohere model invoked through native REST procedure SQL issues the HTTPS request; Foundry performs inference PostgreSQL Flexible Server pgvector plus hand-written SQL RRF over full-text search Optional external Cohere call from Python Retrieval primitives are native; this RRF query and Cohere path are application code Azure Cosmos DB for NoSQL Native vector search and native hybrid RRF SDK-integrated Semantic Reranker, currently preview Reranking is a separate inference call over at most 50 supplied documents Azure AI Search: native hybrid retrieval and semantic ranking How it works: Azure AI Search provides the most integrated pipeline in this demonstration. A hybrid query runs keyword and vector retrieval, combines the lists with RRF, and passes up to the top 50 results to the built-in semantic ranker. The semantic ranker assigns @search.rerankerScore values from 0 to 4 and can return extractive captions and answers. The semantic configuration identifies the fields that carry the meaning of each document: semantic_search = SemanticSearch( configurations=[ SemanticConfiguration( name=SEMANTIC_CONFIG, prioritized_fields=SemanticPrioritizedFields( title_field=SemanticField(field_name="title"), content_fields=[SemanticField(field_name="content")], ), ) ] ) This tells the semantic ranker which text fields to evaluate. The query then enables semantic ranking after hybrid retrieval: results = search_client.search( search_text=question, vector_queries=[vector_query], query_type="semantic", semantic_configuration_name=SEMANTIC_CONFIG, top=10, ) The important constraint is candidate recall. Semantic ranking does not search the corpus again. If the correct passage is absent from the hybrid top 50, the semantic stage cannot recover it. See 01_azure_ai_search_reranking.ipynb for the complete setup and query path. Test results for Azure AI Search These examples show that semantic reranking improves relevance selectively, not universally. It strongly helps the construction query, moving the correct passage from rank 4 to rank 1, but slightly degrades the Super Bowl and Tesla queries by one position. This reinforces that semantic ranking should be evaluated across a representative query set using aggregate metrics such as MRR or NDCG, rather than judged from a single result. Azure SQL Database: vector retrieval plus external Cohere reranking How it works. The Azure SQL notebook retrieves 20 candidates with exact cosine distance and sends their text to Cohere Rerank v4.0 Fast through sys.sp_invoke_external_rest_endpoint. The vector column and query vector must have the same dimensions. This repository uses 1,536-dimensional embeddings: SELECT TOP (@ candidate_count) context_id, title, content, 1 - VECTOR_DISTANCE( 'cosine', CAST(@ query_vector AS VECTOR(1536)), embedding ) AS similarity FROM dbo.documents ORDER BY similarity DESC; For reranking, we selected Cohere Rerank v4.0 Fast (Cohere-rerank-v4.0-fast), a fast version of Cohere’s fourth-generation relevance-ranking model. The model is deployed in Microsoft Foundry, where its Azure Direct inference endpoint is available in the deployment details within the Foundry portal. Azure SQL can call REST APIs directly using sp_invoke_external_rest_endpoint. Because Azure SQL allowlists Azure AI’s *.cognitiveservices.azure.com domain, we translate the equivalent Foundry endpoint from *.services.ai.azure.com while preserving the Cohere reranking route. from urllib.parse import urlsplit, urlunsplit def sql_compatible_endpoint(endpoint: str) -> str: """Convert an Azure Direct endpoint to Azure SQL's allowed hostname.""" parts = urlsplit(endpoint) if parts.hostname.endswith(".services.ai.azure.com"): resource = parts.hostname.removesuffix(".services.ai.azure.com") hostname = f"{resource}.cognitiveservices.azure.com" elif parts.hostname.endswith(".cognitiveservices.azure.com"): hostname = parts.hostname else: raise ValueError("Expected an Azure AI Services endpoint.") return urlunsplit( (parts.scheme, hostname, parts.path, parts.query, "") ) Then I defined a re-rank with cohere function, starting by loading the endpoint and setting the authentication: def rerank_with_cohere( cursor, question: str, candidates: list[dict], top_n: int = 10, ) -> list[dict]: """ Rerank candidate documents by calling Cohere through Azure SQL. Each candidate must contain a 'content' field. """ if not candidates: return [] sql_endpoint = sql_compatible_endpoint( os.environ["COHERE_RERANK_ENDPOINT"] ) model = os.environ["COHERE_RERANK_MODEL"] access_token = credential.get_token( "https://cognitiveservices.azure.com/.default" ).token headers = json.dumps({"Authorization": f"Bearer {access_token}"}) payload = json.dumps( { "model": model, "query": question, "documents": [row["content"] for row in candidates], "top_n": min(k, len(candidates)), }, ensure_ascii=False, ) cursor.execute( """ DECLARE @url NVARCHAR(4000) = CAST(? AS NVARCHAR(4000)); DECLARE @headers NVARCHAR(4000) = CAST(? AS NVARCHAR(4000)); DECLARE Payload NVARCHAR(MAX) = CAST(? AS NVARCHAR(MAX)); DECLARE Response NVARCHAR(MAX); DECLARE @status INT; EXEC @status = sys.sp_invoke_external_rest_endpoint @url = @url, @method = 'POST', @headers = @headers, Payload = Payload, @timeout = 60, @retry_count = 2, Response = Response OUTPUT; SELECT @status, Response; """, sql_endpoint, headers, payload, ) status, response_text = cursor.fetchone() if status != 0: raise RuntimeError(f"Reranker endpoint returned HTTP status {status}.") response = json.loads(response_text)["result"] You can see the complete implementation in the 02_azure_sql_reranking.ipynb notebook. Test results for Azure SQL Db Across the three sample questions, Cohere reranking consistently moved the correct SQuAD passage closer to the top: from rank 5 to 1 for the Super Bowl question, 3 to 2 for the Tesla question, and 8 to 1 for the construction question. These examples show how vector search provides a strong candidate set, while reranking applies deeper query-document relevance scoring to improve the final ordering. The results are illustrative rather than a complete quality benchmark, so broader evaluation across many queries is still recommended. Azure Database for PostgreSQL flexible server: pgvector, SQL RRF, and an optional model How it works: PostgreSQL makes the pipeline components explicit. The notebook uses pgvector for vector similarity, PostgreSQL full-text search for keyword retrieval, and SQL to implement RRF. Vector retrieval uses cosine distance: SELECT context_id, title, content, 1 - (embedding <= > % (query_vector) s:: vector) AS similarity FROM squad_docs ORDER BY embedding <= > % (query_vector) s:: vector LIMIT % (candidate_count) s; The hybrid query independently ranks vector and keyword hits, then combines positions rather than raw scores: SELECT d.context_id, COALESCE(1.0 / (60 + v.rank), 0) + COALESCE(1.0 / (60 + k.rank), 0) AS rrf_score FROM squad_docs AS d LEFT JOIN vector_hits AS v USING (context_id) LEFT JOIN keyword_hits AS k USING (context_id) WHERE v.context_id IS NOT NULL OR k.context_id IS NOT NULL ORDER BY rrf_score DESC; This is not a built-in PostgreSQL RRF operator. It is transparent, hand-written SQL over native retrieval primitives, which makes weighting and debugging flexible but leaves implementation and tuning with the application team. The notebook's optional learned stage sends the vector candidates from Python to a Foundry deployment of Cohere Rerank v4.0 Fast. This path was chosen because the tested Flexible Server azure_ai extension version expected the older serverless reranking endpoint contract. Microsoft documentation still describes azure_ai.rank() as a preview function whose default model is Cohere Rerank v3.5, even though that model retired on May 14, 2026. Treat this as a version-specific compatibility issue and verify current extension behavior before selecting an architecture. Azure HorizonDB is a different product path. Its AI Model Management feature can provision Cohere Rerank v4.0 Fast as default-reranker, but that management feature is currently a limited preview. It should not be described as a generally available Flexible Server capability. See 03_azure_postgres_reranking.ipynb for the full SQL and optional external model path. Test results for Azure SQL for PostgreSQL Flexible Server The tests show that PostgreSQL vector search provides a useful candidate set, SQL RRF can substantially improve results when keyword evidence is strong, and the Cohere semantic reranker is the most consistent overall: it moved the correct passage to rank 1 in two tests and from rank 3 to rank 2 in the Tesla test. RRF produced the biggest gain for the construction question, moving the correct passage from outside the vector top five to rank 1, but did not improve every query. The scores across stages are not directly comparable because cosine similarity, RRF score, and Cohere relevance use different scales. Azure Cosmos DB for NoSQL: hybrid search with built-in RRF How it works: Azure Cosmos DB for NoSQL supports native hybrid ranking with VectorDistance, FullTextScore, and RRF inside ORDER BY RANK: SELECT TOP K C.context_id, c.title, c.text FROM c ORDER BY RANK RRF( VectorDistance(c.vector, @query_vector), FullTextScore(c.text, @term1, @term2, @term3) ) The notebook extracts distinct terms from the question before building the full-text part of the query. That token selection is application logic and can materially affect the hybrid ranking, so production evaluation should test analyzers, languages, term extraction, and optional RRF weights. Cosmos DB Semantic Reranker is an SDK-integrated preview feature. The application first runs a query, serializes the resulting documents, and submits those documents with the user's context string: result = container.semantic_rerank( context=question, documents=documents, options={ "return_documents": False, "top_k": min(k, len(documents)), "sort": True, "document_type": "json", "target_paths": "title,text", }, ) The service accepts at most 50 documents per rerank call and returns relevance scores from 0 to 1, plus inference latency and token usage. It uses the Microsoft semantic ranking model also used by Azure AI Search. The reranking call requires Microsoft Entra authentication, the appropriate Semantic Reranker role, and an account-linked inference endpoint. The 04_azure_cosmosdb_reranking.ipynb in the shared repo contains and end-to-end implementation. Test results for Azure Cosmos Db The results show that vector search provides a strong baseline, while hybrid RRF and semantic reranking improve different queries in different ways. Hybrid RRF helps when exact keywords matter, moving the construction answer into the top results, while the semantic reranker delivers the strongest overall ordering, promoting the correct construction passage from hybrid rank 3 to rank 1 and improving the Super Bowl answer from rank 5 to rank 2. However, it does not always place the gold passage first, as seen in the Tesla example, confirming that reranking improves relevance but is query-dependent and should be evaluated across a larger test set. What the examples do and do not show The four services expose different ownership boundaries: • Azure AI Search owns hybrid fusion and learned semantic ranking inside the search service. • Azure SQL owns vector retrieval and outbound REST invocation in this example, while Foundry owns model inference. • PostgreSQL supplies vector and full-text primitives; the application owns the RRF SQL and optional Cohere call. • Cosmos DB provides native hybrid RRF and integrates a separate preview inference call through its SDK. Across three selected questions, the known source passage often moved substantially. That supports the practical value of testing a second-stage ranker. It does not prove that semantic reranking always improves top-1 accuracy, that RRF is universally beneficial, or that scores from different stages can be compared directly. Cosine similarity, RRF score, Azure AI Search reranker score, Cohere relevance, and Cosmos DB semantic relevance all have different definitions and scales. Compare rank positions and task-level metrics, not raw values across systems. Turn the demonstration into an evaluation For a production RAG system, convert the notebook pattern into a repeatable evaluation harness: Build a representative labeled query set from real user tasks. Freeze corpus, chunking, embedding model, dimensions, and candidate counts for each run. Record ranks after retrieval, fusion, and learned reranking. Measure Recall@k or Hit@k to verify that retrieval finds relevant evidence. Measure Mean Reciprocal Rank (MRR) when the position of the first relevant result matters. Use NDCG when judgments include multiple passages or graded relevance. Record latency percentiles, inference usage, request cost, and failure rates. Evaluate the generated answer separately for correctness, citation support, and refusal behavior. Also test the operational cases that a three-question demonstration cannot cover: empty keyword results, missing gold passages, long documents, multilingual text, filters, partial outages, token expiration, throttling, model retirement, and low-confidence scores. Practical guidance Retrieve broadly enough that the correct evidence can reach the learned stage. Use RRF when vector and keyword retrieval provide complementary signals. Rerank a bounded candidate set, commonly 20 to 50 passages, and measure the latency cost. Keep citations and source identifiers through every rank transformation. Version the corpus, embedding model, dimensions, query set, and reranker deployment. Do not hard-code assumptions about model endpoints or lifecycle dates. Verify current service documentation and the deployed extension or SDK version. Add thresholds or fallback behavior only after calibrating scores on your own data. Judge the full RAG chain. Better passage order is valuable only when it improves grounded answers for users. Vector search is built to find plausible candidates quickly. Rank fusion can reconcile retrieval signals, and a learned reranker can decide which candidates best address the question. The right architecture depends on where your data lives, which service boundaries you want to operate, and what your evaluation says about quality, latency, and cost. Resources Companion repository Azure AI Search semantic ranker Azure SQL VECTOR_DISTANCE Azure SQL sp_invoke_external_rest_endpoint Azure Database for PostgreSQL AI functions Microsoft Foundry model retirement schedule Azure Cosmos DB hybrid search Azure Cosmos DB Semantic Reranker SQuAD dataset card Dataset attribution Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP 2016. SQuAD 1.1 is distributed under CC BY-SA 4.0.
