🚀 Foundry Toolkit for VS Code — August 2026 Update
This is the August round-up for the Foundry Toolkit for VS Code. Four releases shipped this month: 1.6.7, 1.6.8, 1.6.9, and 1.6.10. August was about turning agent development into a workflow you can follow end to end — start from the right path, connect reusable tools and other agents, run with real user isolation, and inspect exactly where the time and tokens went. Have feedback or hit a bug? File an issue on GitHub — the roadmap moves on what you tell us. Highlights Prompt Agent toolboxes — attach a centrally managed toolbox, inspect its tools and skills, manage versions and approval policies, and configure nested tools without leaving Agent Builder. 1.6.10 Agent-to-Agent connections (preview) — connect an Agent2Agent (A2A)-compatible agent from a configured connection, the Foundry account catalog, or a custom HTTPS endpoint. 1.6.10 Agent Inspector Overview — read a latency waterfall and an ordered timeline of model, reasoning, and tool activity for all runs or one selected run. 1.6.9 User-scoped Hosted Agent sessions — set a user identity so Responses conversations and session files stay isolated per user. 1.6.8 A clearer Create Agent start — choose Microsoft Agent Framework, Copilot SDK, LangGraph, Copilot-assisted coding, Agent Builder, or the full sample catalog from one redesigned page. 1.6.9 🤖 Create Agents — start on the right path, then stay in context Starting an agent shouldn't begin with choosing the wrong abstraction. The redesigned Create Agent page gives you direct routes to Microsoft Agent Framework, Copilot SDK, and LangGraph samples, Copilot-assisted coding, Agent Builder, and the complete sample catalog. You decide whether you want code, a guided build, or a prompt agent first — not after scaffolding the wrong project. 1.6.9 Hosted Agent setup is also less brittle. You can choose Skip for now during model setup even when existing deployments fail to load, then wire the model connection later. Administrator-connected Foundry models now appear alongside regular deployments in playgrounds and Hosted Agent creation, so the models your organization already configured are available where you build. 1.6.8 1.6.9 Once an agent is running, identity matters. The Hosted Agent Playground can now set a user identity for Responses conversations, keeping conversation state and session files isolated for each user instead of blending everyone into one test session. And when somebody sends you a Microsoft Foundry portal link, deep links can open that named Hosted Agent's Details or Optimization page directly in VS Code — not the portal home, not a search screen. 1.6.8 1.6.10 🔧 Toolboxes and A2A — connect capabilities once, reuse them An agent with five tools can become five separate configurations, five approval stories, and five places to make the same update. Toolbox changes that shape: it packages centrally managed tools behind one Model Context Protocol (MCP)-compatible endpoint, with shared versioning and policy controls. In August, Prompt Agents gained toolbox workflows inside Agent Builder. Open Add tools to browse toolboxes, or use Add to Prompt Agent from the Toolbox resource list. The attached toolbox appears as a collapsible card where you can inspect tools and skills, switch versions, configure approval policies and nested tools, replace or remove the toolbox, or opt out. You manage the collection — not a loose pile of one-off connections. 1.6.10 Agent-to-agent composition arrives in the same flow. Agent-to-Agent connections (preview) let you add an A2A-compatible agent from an existing connection, the Foundry account catalog, or a custom HTTPS endpoint. Attach it directly to a Prompt Agent or put it inside a toolbox for reuse across agents and runtimes. Your pipeline can now be agent → toolbox → specialist agent — with the connection managed as a real resource instead of buried in prompt text. 1.6.10 🔍 Agent Inspector — see the run, not just the answer A final answer can look right while the run behind it is slow, expensive, or calling the wrong tool. Agent Inspector now gives you the sequence and the evidence. The new default Overview tab shows every run or one selected run through two synchronized views: a latency waterfall and an ordered timeline of model, reasoning, and tool activity. Response footers add the model, duration, total tokens, and timestamp; hover over the token total to split input from output. Raw reasoning and reasoning summaries appear in separate collapsible sections when the agent provides them. 1.6.9 Tool inspection goes deeper in 1.6.10. Calls are grouped by response run, with status, call ID, arguments, and results, and each Responses event can show when it reached Agent Inspector. The Overview waterfall and timeline now scroll independently, while long streaming responses and Details views update more smoothly. You can move from "the tool failed" to the exact call and payload without reconstructing the run from chat bubbles. 1.6.10 The conversation itself is easier to drive: press Up or Down to recall and edit earlier requests without losing your unsent draft, or choose Clear Chat to reset the conversation plus Events and Details state. Pending MCP approvals and OAuth consent requests stay pinned above the input, with bulk actions and expandable details, until every decision is resolved. 1.6.7 1.6.9 🎯 Models and resources — faster to open, steadier when you return Resource pages should remember your work, not reset it. Models and Tools now load the selected tab first and show core rows before fetching the extra details. When you return to Agents, Models, Tools, Knowledge, or Evaluations, the toolkit preserves rows, search, filters, and pagination while refreshing the active view in the background. A manual refresh still gets the latest service state when you ask for it. 1.6.7 The sidebar does less work too. Collapsed My Resources sections load only when you open them, while Search and Recent Agents remain available. Evaluations, Routines, Tools, Skills, and Toolboxes now share consistent loading feedback, and a direct link to Tools or Skills opens the requested tab without loading Toolboxes first. The result isn't a new destination — it's less waiting on the way there. 1.6.7 1.6.8 Model deployment guidance got one sharp fix as well: quota errors now open the token quota page for your current Foundry project, so the recovery path lands on the project that actually needs capacity. 1.6.10 💻 Activity protocol agents — debugging that matches the agent Activity Protocol agents target Microsoft 365 channels, so local debugging should speak the same language. Newly scaffolded Python projects now open Microsoft 365 Agents Playground inside VS Code for local debugging. You stay in the editor and test the activity-shaped conversation before deployment instead of forcing it through an incompatible playground. 1.6.7 Copilot-assisted creation also follows the current Hosted Agent path: current Foundry project and model setup, a managed Python environment, workspace-root debugging, and the latest local run and deployment flow. When you reuse the selected Foundry project, Copilot no longer asks you to choose its Azure location again. 1.6.10 🪲 Fixes and polish Agent Inspector — streamed response and reasoning text stays complete; response text, reasoning, tool calls, and permission decisions keep their original order; replacement turns reject obsolete stream events; and unmatched tool calls or results no longer appear in Details. 1.6.9 1.6.10 Approvals and consent — human-in-the-loop pauses no longer duplicate tool or approval cards, Clear Chat remains available while a turn waits, and continuation responses retain pending approvals until every request is resolved. 1.6.8 1.6.9 Activity Protocol deployment — Azure Bot settings are validated before submission, compatible Bots are reused, identity and application ID conflicts get recovery guidance, and successful deployments no longer open an unsupported Agent Playground. 1.6.7 1.6.8 Agent Builder and MCP OAuth — reopening a Foundry Prompt Agent preserves its selected version and tool configuration, while authorization callbacks complete only the matching connection request. 1.6.8 Accessibility — screen readers announce Model Catalog actions, collapsible Agent Builder and Model Preference controls, and project and model fields with their labels and state; prompt placeholders also meet minimum contrast requirements. 1.6.10 ⚠️ Breaking change and migration GitHub Models has been removed from the Model Catalog, playground, model comparison, Agent Builder, and evaluations following the service's retirement. If a saved workflow or evaluation references GitHub Models, open it and select another available model before running it again. 1.6.7 🚀 Get it and tell us what to build next August connected the whole agent loop: choose the right starting point, reuse governed tools, compose agents through A2A, isolate real users, and inspect the run down to timing, tokens, arguments, and results. Install or update from the Visual Studio Code Marketplace. Read the docs — Foundry Toolkit for Visual Studio Code and the Microsoft Foundry documentation. Explore samples in the Microsoft Foundry samples repository. Browse the full changelog in WHATS_NEW.md. File issues and feature requests at github.com/microsoft/foundry-toolkit/issues. Join the Microsoft Foundry community on Discord. Try a toolbox with your next Prompt Agent, open the run in Agent Inspector, and tell us where the workflow still slows you down. Happy building. 🚀
Zonal redundancy in API management Standard v2
APIs are the backbone of modern applications, powering everything from mobile experiences and microservices to AI-driven applications and business-critical integrations. As customers continue to modernize their platforms on Azure, they increasingly expect their API infrastructure to remain available even in the face of datacenter-level disruptions. With zone redundancy in Standard v2, Azure API Management now enables customers to increase resilience against Availability Zone failures while continuing to benefit from the simplicity, performance, and cost efficiency of the v2 platform. Why Zone Redundancy Matters Azure Availability Zones are physically separate locations within an Azure region, each with independent power, cooling, and networking infrastructure. By distributing API Management resources across multiple zones, organizations can reduce the impact of a single datacenter failure and improve service continuity for their APIs. Until now, customers who required built-in zone-level resiliency often needed to evaluate higher-end deployment options. With this enhancement, Standard v2 customers can now deploy API gateways across Availability Zones and benefit from improved reliability while maintaining the streamlined operational model of the v2 platform. What’s New Zone Redundancy for Standard v2 extends the platform's resiliency by distributing service capacity across multiple Availability Zones within a supported Azure region. Key benefits include: Higher Availability: API traffic continues to flow even if a single Availability Zone experiences an outage. Built-in Resiliency: Redundancy is provided at the platform layer, reducing the need for customers to design and manage complex intra-region failover solutions. Production-Ready Reliability: Customers can confidently run critical API workloads on Standard v2 with stronger availability guarantees. Operational Simplicity: The service automatically manages capacity distribution, health monitoring, and recovery behavior across zones. Cost-Effective Resilience: Customers gain zone-level protection without requiring an enterprise-tier deployment model. Built on the Modern v2 Platform The v2 platform was designed from the ground up to provide a faster, more reliable, and more scalable API Management experience. Standard v2 already delivers capabilities such as rapid deployment, simplified networking, workspace support, and flexible scaling. Zone Redundancy further strengthens the platform by expanding its reliability story for production workloads. This announcement builds on our broader investment in making Azure API Management more accessible to a wider range of organizations, from digital-native startups to large enterprises modernizing their application estates. Ideal Scenarios Zone Redundancy in Standard v2 is particularly valuable for customers who: Run business-critical APIs that must remain available during datacenter incidents. Consolidate multiple application workloads behind a single API gateway. Expose APIs consumed by mobile, partner, and customer-facing applications. Support AI applications and agent-based architectures that depend on highly available API endpoints. For organizations adopting modern cloud and AI native architectures, this capability helps ensure that API infrastructure remains aligned with broader application resiliency strategies. A Foundation for Reliable AI and API Platforms As AI-powered applications continue to proliferate, APIs increasingly become the critical connection layer between models, agents, business systems, and data platforms. Downtime at the API layer can have a direct impact on application availability, customer experience, and business operations. By bringing zone redundancy to Standard v2, we are making it easier for organizations to build highly resilient API platforms that can serve as the foundation for next-generation AI and digital transformation initiatives. Getting Started Zone Redundancy for Standard v2 can be enabled in supported Azure regions, allowing customers to deploy API Management with built-in protection against Availability Zone failures. We recommend reviewing your application's overall resiliency architecture, including backend redundancy, traffic management, and disaster recovery requirements, to maximize the benefits of zone-resilient API infrastructure. Enable Zone Redundancy in the Azure Portal Getting started with Zone Redundancy in Azure API Management Standard v2 is straightforward and can be configured during service creation. Create a New Standard v2 Instance with Zone Redundancy Sign in to the Azure portal. Select Create a Resource and search for Azure API Management. Choose Standard v2 as the service tier. Select a region that supports Availability Zones. In the Availability Zones section, enable Zone Redundancy. Review and create the service. After deployment, Azure API Management automatically distributes service capacity across multiple Availability Zones within the selected region, helping maintain API availability during a zone-level outage. Looking Ahead This release represents another step in our ongoing investment in the Azure API Management v2 platform. We remain committed to delivering the reliability, scalability, security, and developer experiences that organizations expect from a modern API management service. We are excited to see what our customers build with a more resilient Standard v2 platform and look forward to your feedback as you continue modernizing and scaling your API ecosystems on Azure. Learn more by visiting the Azure API Management documentation and exploring the latest reliability guidance for API Management deployments.Bringing all your Integration workloads to Logic Apps Standard
We recently announced the end of life of BizTalk Server and provided a path forward for our customers. As part of that commitment, we’re investing in tooling and guidance that reduces migration complexity and helps teams modernize confidently to Azure Logic Apps Standard. Because enterprise integration programs are rarely “lift and shift,” we’re pairing automation with best practices, reference architectures, and field-proven guidance to support you from assessment through cutover. In our December 2025 announcement, we outlined a long-term direction for enterprise integration: Azure Logic Apps is the successor to BizTalk Server. Customers can modernize at a pace that balances continuity with innovation—while moving to a cloud platform designed for scale, hybrid operations, DevOps, and AI-assisted automation. The strategy centers on three principles: a predictable BizTalk lifecycle runway, preservation of existing investments, and a practical, guided migration path to Logic Apps. What makes this strategy credible is not just the vision—but the concrete tooling and guidance that back it up. Announcing the Logic Apps Migration Agent: An Open-source project to provide an AI End-to-End Modernization Experience Today we’re announcing the Logic Apps Migration Agent—an open-source Microsoft project that delivers an AI-assisted, end-to-end modernization experience with a structured, stage-gated workflow. Built by the product group and shaped by direct field feedback, the agent operationalizes how migrations should be executed: discover what you have, plan what you’ll modernize, convert incrementally, and validate continuously. The result is a repeatable approach that helps customers (and partners) migrate from BizTalk and other integration platforms to Azure Logic Apps with greater speed and confidence—without compromising governance or correctness. The agent reinforces the modernization strategy through: Discovery → Planning → Conversion: Aligns to Microsoft modernization guidance so teams understand scope, dependencies, and gaps before committing to conversion. Human-in-the-loop checkpoints: Uses AI to accelerate analysis and baseline conversions while enforcing review and approval steps for mission-critical correctness and governance. VS Code + GitHub Copilot integration: Brings migrations into a code-first workflow—enabling developer-centric refactoring, DevOps practices, and consistent implementation patterns for Logic Apps. Incremental, flow-group migration: Modernize one logical unit at a time to reduce risk, support phased cutovers, and avoid big-bang rewrites. Bring-your-own black-box testing: Import existing files, test cases, and specifications to validate behavior and reduce custom test harness work. In short, the Migration Agent turns high-level modernization guidance into a repeatable, auditable process teams can trust. This alignment is critical for customers running mission‑critical integrations. It replaces uncertainty with a clear path: modernize incrementally, reuse what works, validate every step, and emerge on Azure Logic Apps with a platform ready for the next decade of integration and AI-driven automation. What you should focus on? Target architecture decisions: the agent will propose integration patterns, but will not choose partitioning strategy, reliability approach, or network topology—you will decide what “great” looks like. Semantic equivalence: The Agent will generate baseline artifacts, but domain-specific mapping, transformation nuances, error handling semantics, and edge cases still require human validation. Connector and parity gaps must be addressed: if a source platform capability has no 1:1 equivalent, the migration may require redesign (custom code, Local Functions, API Management, Service Bus patterns, or alternative connectors). Performance, security, and operations hardening remain essential: identity, secrets, policies, monitoring, cost controls, and SRE practices are not “one-click.” Cutover planning is outside the scope of automation: data/backlog reconciliation, dual-run strategies, and rollback plans remain project workstreams. More mission critical features for Logic Apps Standard and Hybrid We are weeks away from shipping the following features, aimed at any customers in the Enterprise Application Integration space: HL7 In-App operations in general availability. MLLP Receive/Send In-App connector in Public Preview. Rules Engine In-App operation for XML facts in Public Preview. MSMQ In-App connector in Public Preview. Oracle DB In-App connector in Public Preview. Flat File generation In-App operations in Public Preview. Support for local container registry (Hybrid deployment model). Integration accounts support (Hybrid On premises). NMS In-App connector in Public Preview. Improvements to our EDI capabilities. BizTalk Mapper to Data Mapper Migration path What about other integration platforms? Yes—the Logic Apps Migration Agent is designed to be customizable so you can migrate from any integration platform to Logic Apps (not just BizTalk). The open architecture lets you plug in new discovery, analysis, and conversion skills for the source product you’re modernizing, while keeping the same stage-gated workflow and human-in-the-loop checkpoints. We provide guidance and examples to help you extend the agent for other platforms than BizTalk —so you can tailor mappings, transformation rules, and validation to your customer’s standards and target patterns in Logic Apps. Benefits Faster time to value with a guided process: A structured discovery→planning→conversion workflow reduces uncertainty and helps teams move from assessment to execution with clear checkpoints. Higher confidence migrations: Human-in-the-loop validation, artifacts generation, and black-box testing support mission‑critical correctness and governance. Customizable for your source platform and standards: Extend the agent with product-specific discovery and conversion steps, tailor mappings and transformation rules, and align outputs with your target Logic Apps patterns and engineering conventions. Open-source transparency and control: Review how the tool works end-to-end, validate what it produces, and adopt changes at your pace without waiting for a closed release cycle. Community-driven innovation: Benefit from contributions across Microsoft, partners, and customers—new adapters, mapping packs, and best practices can be shared and reused. Lower total migration cost: Automating repeatable tasks reduces manual effort while preserving the ability to invest partner expertise where it matters most (architecture, governance, reliability, and operations). Reusable accelerators for partners: Partners can create differentiated offerings by packaging templates, validation suites, CI/CD pipelines, and domain-specific patterns on top of the agent. For companies providing professional services: this agent is meant to augment your delivery—not replace it. By automating repeatable groundwork (inventory, baseline conversion, and validation scaffolding), it frees your teams to focus on the higher‑value work customers rely on you for: defining target architecture, refining mappings and patterns, hardening security and governance, implementing CI/CD, performance tuning, and driving cutover and operating model changes. Because the project is open source and extensible, partners can also package reusable accelerators (templates, connectors, mapping packs, test harnesses) and build differentiated migration offerings on top of the same trusted process. Review our public documentation here: https://learn.microsoft.com/en-us/azure/logic-apps/migration/migration-agent-overview Recommendations: Consider the following recommendations when using the Agent: Structure your projects along with all dependencies in directories. Include bindings, MSIs. configuration files, source code, schemas, maps, pipelines, even documentation. Review each stage thoroughly. Suggest changes to tailor each stage to include all dependencies, and your architecture and requirements preference. While you can run the agent in any workstation with VSCode, if you want to test your Logic Apps solution, make sure your VSCode workstation has access to a test environment if you want the agent to test against any target system or dependencies. Make sure you increase the Maximum number of requests for the Copilot Chat as follows (we recommend changing the value from 60 to 1000) Check the following video for a demonstration on how the Agent works and let us know if you have any questions in the comments.Give your Copilot agents real tools, without hand-wiring MCP
You’re running agents in the GitHub Copilot app - maybe three at once, each on its own branch, each working a task while you steer. It’s a good way to work, right up until an agent needs to reach outside your repo. It needs to read a SharePoint library, file a Salesforce record, or check an Outlook calendar, and suddenly it’s blind. An agent is only as capable as the tools you hand it. The usual way to hand it one of those tools is to wire up a Model Context Protocol (MCP) server by hand: find the endpoint, paste it into config, add an auth header, keep the token alive, and hope you got the casing right. Then your teammate does the same thing on their machine, and the next person after that. The MCP Connectors canvas removes the hand-wiring. It’s a plugin for the GitHub Copilot app: it lists the hosted MCP servers already published in your Azure Connector Namespace, and you connect one to Copilot by selecting it - no URL, no header, no local proxy. Where the servers come from Connector Namespace is the managed Azure service on the other end. It provides a list of curated MCP Servers that you can use to quickly create and manage MCPs connections. It also does the work you’d rather not: it stores and rotates the credentials, applies retry and throttling policies, and scales the server. Your machine never holds a raw secret related to the end systems that MCP touches, and access is governed by access policies, allowing fine grained control on who can use the MCP servers. So, when you connect a server from the canvas, you’re not standing up infrastructure. The server already exists, already authenticated, already managed. You’re pointing Copilot at it. Prerequisites GitHub Copilot app with canvas extension support. An Azure subscription with permission to cerate and view the namespace and to create connections and hosted managed MCP server configurations. Connector Namespace is in preview, and availability varies by region. Install From the GitHub Copilot app, add the following prompt. Install connector-namespaces v1.2.0 for my user account, reload it, verify it is running, then open the MCP Connectors canvas. Open the Connector Namespace canvas in a new session Once you install if you need to open the Connector Namespace in a new session, just request it in a prompt: Open the MCP Connectors canvas. Connect an MCP server Open the MCP Connectors canvas. Select Sign in to Azure, then choose a subscription and Connector Namespace. Browse or search the MCP servers grouped under Microsoft and Partners. Select Connect and complete the connector’s separate authentication or consent flow. Confirm the server appears under My MCPs. Restart GitHub Copilot so a new session loads the added tools. ℹ️Note That last step matters more than it look. Copilot loads an agent’s tools when a session starts, so a server you add mid-session shows up in the next one, not the one you’re in. Manage connected servers My MCPs shows the servers connected to Copilot. Sandbox opens a connected server in the Connector Namespace playground, so you can try a tool call yourself before you let an agent lean on it. Disconnect removes the MCP registration from your MCP configuration. The MCP is still available in Connector namespace to be added again later. Also if you are sharing the connector namespace with other developers, this only remove the your local configuration. Connect adds configured MCP connectors that are not registered locally yet to your local MCP configuration. Delete from namespace… removes the MCP registration from your MCP configuration and deletes the MCP connection from your connector namespace. If you are sharing the connector namespace with other developers this will affect any developer using this MCP connector – be careful when using this. Switch namespace switches the active subscription or namespace - useful when you keep dev and production connections apart. How it works and security The extension registers each server directly in Copilot’s user-scoped MCP configuration. There’s no local MCP proxy sitting in the tool path between the agent and the server. The canvas itself runs on loopback. Your Azure tokens stay in process memory, the selected namespace coordinates may be cached locally, and gateway credentials are sent only to the server’s configured HTTPS endpoints. The connector’s own secrets never reach your machine. That’s the namespace’s job, which is also why an admin can rotate or revoke a connection in one place instead of chasing down every developer’s config. Where can I use my connected MCP servers You manage the servers from the canvas, but they don’t only work there. The canvas writes to Copilot’s user-scoped MCP configuration at ~/.copilot/mcp-config.json, and the GitHub Copilot app is built on GitHub Copilot CLI. Connect a server once and it’s available to both. Agent sessions in the app. Any session on your machine picks the server up, whatever repository or branch it’s working on, because the config is user-scoped rather than per-project. Copilot CLI. The same server shows up in the terminal. Run copilot mcp list to confirm it, or /mcp show SERVER-NAME in an interactive session to see the tools it exposes. Two things to know: Tools load when a session starts, so connect first and start the session second. A project-level .mcp.json or .github/mcp.json takes precedence over your user config when the names collide, which is what you want if a repository pins its own version of a server. VS Code is the exception. Copilot Chat there reads .vscode/mcp.json, a separate file the CLI doesn’t use, so a server you connect in the canvas won’t appear in VS Code until you configure it there too. Where this all sits today The GitHub Copilot app went generally available in June 2026, and canvases shipped with it, so neither is a preview feature. The MCP Connectors canvas is a published extension you install at personal scope. You get the version we ship and tested, not one an agent generates on the fly from a prompt. Two caveats. GitHub doesn't publish a compatibility guarantee for the canvas extension format, so an app update may need a matching update from us. Re-run the install commands above to pick it up if you see any issues. On the Azure side, Connector Namespace is still in preview. Check the documentation for more details on this service outside the GitHub Copilot App support. If you're on Copilot Business or Enterprise, check with your admin that the GitHub Copilot app policy is enabled. It's on by default, and it's separate from the Copilot CLI policy. Try it Install the plugin, point it at a namespace, and connect one server your agents keep reaching for: the SharePoint library, the CRM, whatever it is. Then watch an agent actually use it in the next session. Tell me where the canvas saved you a config headache and where it got in your way. That’s the feedback we act on before general availability.AI Gateway tier of API Management now in public preview
Today, we are introducing the AI Gateway tier of Azure API Management, now in public preview. It gives platform teams a purpose-built experience built specifically for AI workloads - publishing and governing models and MCP servers. Controls are configured through policy cards rather than XML and expressions, and the portal experience and control plane are structured around models, MCP servers, and tools rather than APIs. (For brevity, we refer to the AI Gateway tier as AI Gateway throughout the rest of this article.) AI Gateway is built on Azure API Management, bringing proven operational capabilities to AI workloads. The resource runs in your subscription, uses your Entra tenant, and sends telemetry to destinations you control. The operating model will be familiar to existing API Management customers, but the interface is built around AI workloads. The AI Gateway tier is intended for teams that want this focused experience; other API Management tiers remain the right choice when organizations also need general-purpose API management or capabilities not included in the AI Gateway experience. A practical model for platform teams The AI Gateway gives platform teams a shared place to manage models, MCP servers, policies, and observability destinations, with access controlled through Azure RBAC. For example, a central platform group can connect a set of approved models and tools and publish them for application teams. The application teams can test those assets in the test console and build against them without routing every change through the central group. The platform group still owns the shared guardrails and can see how the assets are being used. After an asset is published, developers can create a named runtime key and begin calling the gateway immediately. Bring the models and tools you already use Most organizations don't standardize on a single model provider. Different models are selected based on quality, latency, cost, geography, or specialized capabilities. The preview supports models from Microsoft Foundry including OpenAI, Anthropic, Mistral, and other Foundry hosted models, as well as models hosted in AWS Bedrock, Google Vertex AI, OpenAI, and Anthropic. A guided wizard simplifies importing models from Microsoft Foundry. Other providers can be added by configuring a connection, with backend authentication configured as part of that connection. All published models are available under the same stable endpoint. Applications continue to use supported API formats such as OpenAI Chat Completions and Responses or Anthropic Messages directly or via SDKs. The AI Gateway extends governance beyond models to the MCP servers and tools agents use to interact with enterprise systems. You can expose an existing MCP server over SSE or Streamable HTTP, turn all or selected operations from a REST API into an MCP server by uploading its OpenAPI specification, or use more than 1,400 connector-backed tools from the Power Platform and Logic Apps library. You can also federate multiple MCP servers behind a single server, so an agent connects once and sees the tools across those servers. Backend authentication supports an API key, OAuth client credentials, managed identity, or mTLS. Governance that's built in Organizations need consistent governance across models and MCP servers without requiring every application team to implement those capabilities independently. The AI Gateway portal presents governance policies through an intuitive card-based experience rather than requiring policy XML. The same policies are expressed as JSON properties, making them easy to manage as infrastructure as code and to audit and enforce across a fleet with Azure Policy. In the public preview, those cards cover request and token rate limits, token quotas, Azure AI Content Safety, and fallback to a secondary model. Policies are applied per asset, making it clear which controls protect each model or MCP server. OpenTelemetry-based token metrics The AI Gateway emits token-usage metrics through OpenTelemetry, with attributes following GenAI and cloud semantic conventions. Metrics can be sent to Application Insights, Datadog, Splunk, Grafana Cloud, or another OTLP endpoint. The portal provides a monitoring view over Application Insights data. Better together: Microsoft Foundry and AI Gateway With AI Gateway, teams can extend the same governance controls, for example token rate limits and quotas, across models hosted in Microsoft Foundry and models hosted elsewhere. Foundry and non-Foundry models are published through gateway-managed endpoints, giving applications and agents a consistent way to access governed models regardless of where they are hosted. Foundry-hosted agents can consume curated sets of tools from Foundry toolboxes, with access to the underlying MCP servers and APIs governed through AI Gateway. Together, Microsoft Foundry and AI Gateway cover the enterprise application lifecycle: Foundry for building and running AI applications, and AI Gateway for publishing, governing, and observing models, tools, and MCP servers across your AI estate. The new AI Gateway tier will soon be available through the gateway experience in Microsoft Foundry portal. We are working toward a seamless, integrated AI Gateway experience within Foundry portal and will share more about that work separately. Available today in public preview The AI Gateway tier is available today at no cost in public preview in East US 2 and Sweden Central. Pricing will be shared separately. To provision a resource, add a model or MCP server, and make a first call click this to go to the AI Gateway tier portal and try it. If you prefer to start from code, use a sample to deploy all the required resources for a Foundry-hosted agent configured to access its model and tools through AI Gateway. We look forward to your feedback as we continue to rapidly evolve AI Gateway.Power Azure SRE Agent with the tools it needs
What is the Azure SRE Agent Azure SRE Agent is an AI-powered service designed to reduce operational toil. Teams can use it to: Investigate incidents and identify probable causes. Automate health checks, compliance reviews, and other scheduled work. Answer questions such as “What changed before this service became degraded?” Propose remediations while allowing teams to require human approval. An effective investigation rarely depends on one source of information. An alert might originate in Azure Monitor, while deployment history lives in source control, telemetry stored in another observability platform, and incident records in a service-management tool. Without access to those systems, you must retrieve and transfer the information manually, adding context switching and slowing diagnosis. MCP servers can give SRE Agent tools to query telemetry, inspect deployments, retrieve database records, look up incidents, etc. SRE Agent provides native connection for some servers such as GitHub, Datadog, New Relic, and Splunk. Connector Namespace makes it easier to host additional remote MCP servers you want the agent to use. Removing remote MCP server hosting burden Connecting SRE Agent to an existing remote endpoint is straightforward. Hosting that endpoint yourself is not. You must deploy the server, provide secure HTTPS infrastructure, configure authentication, manage downstream credentials, scale the runtime, monitor its health, recover failed instances, and maintain it over time. These responsibilities are necessary, but the value is in the server’s tools not in operating another service. Azure Connector Namespace is a fully managed service for hosting connectors and MCP servers. You select the server you need and let the namespace handles the operational and maintenance tasks. The offering is currently in preview. See documentation for supported regions and other preview considerations. You’ll find a wide variety of servers in the Connector Namespace’s catalog. Some examples of useful servers for the SRE agent include: Database servers such as Azure SQL and Azure Cosmos DB Source control and CI/CD servers like GitLab Incident management servers like Jira and PagerDuty A note on what's currently in development: We're building “bring-your-own” server support, allowing you to supply your own server image while the namespace handles hosting and operations. Please keep an eye out for the blog post about this! Deploy server and connect it to SRE Agent The following example deploys the SQL MCP server in Connector Namespace and connects it to Azure SRE Agent. 1. Server deployment Prerequisite: Install the Azure Developer CLI (azd). Clone the sql-server-samples repo: git clone https://github.com/microsoft/sql-server-samples.git Navigate to the azure-sql-mcp sample cd sql-server-samples/samples/applications/azure-sql-mcp From the azure-sql-mcp folder, run the following to log into your Azure subscription and then deploy the server and related resources: azd auth login azd up The last command will prompt for the following before deployment: Prompt Suggested value Explanation Enter a unique environment name mcp-dev This name added as prefix to Azure resources created Select an Azure Subscription Pick your subscription Resources will deploy under this subscription Enter value for connectorNamespaceIdentityType UserAssigned User assigned identity is recommended as it’s not tied to resource lifecycle Enter value for deployerLoginName Enter your Azure subscription login email To give your identity access to the MCP server Enter value for the location Pick a supported region Supported regions: West Central US, Central US, East Asia, North Europe Once deployment finishes, copy the MCP endpoint for use later. It looks similar to: https://<app-name>.<region>.logic.azure.com/api/connectorGateways/123abc456defg7890/mcpServerConfigs/sql-mcp/mcp (Optional) Test deployed server in Visual Studio Code GitHub Copilot: Open command palette > search MCP: Add server > pick HTTP > enter MCP endpoint and server name > pick Local Workspace. Inside .vscode/mcp.json, click Start above server name, then allow authentication with Microsoft in the popup and log into Azure subscription account. 2. Configure MCP connector in SRE Agent Connector Namespace does not create the connection in SRE Agent. Add the server endpoint through SRE Agent’s existing MCP connection experience. Open the Azure SRE Agent portal On the left menu, go to Builder > Connectors, and select + Add connector Under Choose a connector, select the MCP tab, choose MCP server, and select Next Configure the connector: Field Value Name A descriptive name for the server Connection type Streamable-HTTP URI The hosted server endpoint from Connector Namespace Authentication method Managed identity (Selecting managed identity automatically creates an identity for the connector.) Azure AD token scope https://apihub.azure.com/.default Select Next. Before testing the connection, grant the managed identity access to the MCP server. 3. Authorize the managed identity Open the Azure portal, search for the managed identity by name. In the identity’s Overview page, click JSON View (top right) and copy the tenantId and principalId. The principal ID is also called the object ID. Open Connector Namespace portal and search for the deployed namespace. Inside the namespace, navigate to the MCP Connectors tab on the left, then select the SQL MCP server. Inside the MCP server, click Access Policies, then select Add Access Policy. Enter the tenant ID and principal ID, then select Create. 4. Test and finish the connection Return to Azure SRE Agent portal and select Test connection. After the test succeeds, select the server tools the agent should use. Select Add connector. Establishing the connection can take a minute. Select Refresh at the top of the connectors page until its status changes to Connected. The agent can now use the selected server tools in chat threads. The azd deployment from previous created and seeded a SQL database with sample blog post data, so you can ask something like: What are the top blog posts? For more details, see MCP connectors and tools in Azure SRE Agent. Focus on the server, not its infrastructure MCP servers can give Azure SRE Agent access to the additional systems it needs to investigate incidents and perform operational work effectively. However, operating every remote server yourself introduces infrastructure, security, and maintenance responsibilities that distract from that goal. Connector Namespace removes much of that friction. Your primary question becomes “Which MCP server do I want to host?” rather than “How will I deploy, secure, scale, monitor, and maintain it?” Once deployed, the hosted endpoint can be added to Azure SRE Agent through its existing MCP connection experience. That gives teams a straightforward path to extending the agent with more operational tools, without turning MCP server hosting into another platform they must build and run. Try Connector Namespace with Azure SRE Agent and share your feedback! Resources Azure SRE Agent Overview Set up an MCP connector in Azure SRE Agent Connector Namespace Overview Hosted MCP servers in Connector NamespaceDistributing Agents to Microsoft Teams and Microsoft 365 Copilot Part 4/5
This is the fourth post in our series on the Microsoft agent platform. We cover the Distribute in M365 pillar — publishing your agents to Microsoft Teams and Microsoft 365 Copilot so they reach users where they already work. All examples reference the FibreOps repository, demonstrated at Microsoft Build BRK241. The Distribution Story Building a great agent is only half the challenge. The other half is getting it into the hands of users without asking them to learn a new tool, visit a new URL, or change their workflow. Microsoft 365 Copilot and Microsoft Teams are where enterprise users already spend their day, making them the natural distribution surface for agents. With the GA release, publishing an agent to Teams and M365 Copilot is a single command. No separate app registration portal, no manual manifest assembly, no multi-step approval workflow for development and testing. Publishing to Microsoft 365 Copilot (GA) FibreOps ships as a declarative agent + action plugin ready for sideload. A single CLI command produces the complete package: python -m fibreops.demo publish-m365 --out dist/m365 # Output: # ✓ wrote dist/m365/declarativeAgent.json # ✓ wrote dist/m365/fibreops-action.json # ✓ wrote dist/m365/manifest.json # ✓ wrote dist/m365/color.png (192x192) # ✓ wrote dist/m365/outline.png ( 32x32) # ✓ wrote dist/m365/fibreops-copilot.zip What Gets Generated File Purpose declarativeAgent.json Defines the agent's persona, capabilities, and conversation starters for M365 Copilot fibreops-action.json Action plugin that proxies tool calls to the deployed FastAPI backend via OpenAPI manifest.json Teams app manifest with publisher metadata, permissions, and capabilities color.png / outline.png App icons for Teams and M365 surfaces fibreops-copilot.zip Ready-to-upload package for Teams Admin Center Configuration Set the base URL to your deployed FastAPI app before publishing — the action plugin uses this to resolve the OpenAPI runtime: # Set the public HTTPS hostname of the deployed FastAPI app $env:M365_ACTION_BASE_URL = "https://fibreops-demo.azurewebsites.net" # Optional: customise publisher metadata $env:M365_PUBLISHER_NAME = "Contoso Network Operations" $env:M365_PUBLISHER_WEBSITE = "https://contoso.com/noc" # Generate the package python -m fibreops.demo publish-m365 --out dist/m365 Environment Variable Purpose M365_ACTION_BASE_URL Public HTTPS root for the FastAPI /openapi.json (e.g., Container Apps FQDN) M365_APP_ID Override the generated Teams app GUID (default: deterministic per repo) M365_PUBLISHER_NAME Publisher name shown in M365 Admin Center M365_PUBLISHER_WEBSITE Publisher website link Uploading the Package Upload the generated fibreops-copilot.zip through either path: Teams Admin Center → Manage apps → Upload new app M365 Admin Center → Integrated apps → Upload custom apps Once uploaded, the declarative agent: Inherits the publisher metadata you configured Advertises conversation starters from the FibreOps deck (e.g., "What is the current outage status?", "Dispatch an engineer to FN-LDN-001") Proxies tool calls to the deployed FastAPI app via the action plugin Appears in Microsoft 365 Copilot as a specialised agent users can invoke How Declarative Agents Work A declarative agent in Microsoft 365 Copilot is defined by metadata rather than code running in the M365 surface. The intelligence lives in your backend — Copilot handles the conversational UX, tool orchestration schema, and user authentication. The flow: User invokes the agent in Microsoft 365 Copilot or Teams Copilot renders conversation starters and accepts natural language input When the agent needs to act, Copilot calls the action plugin (your OpenAPI endpoint) Your FastAPI backend processes the request using the full agent pipeline Results return to the user in the Copilot/Teams UX This architecture means your agent logic stays in one place — the backend. The M365 surface is purely a distribution and interaction layer. Action Plugins and OpenAPI The action plugin ( fibreops-action.json ) references your FastAPI app's /openapi.json endpoint. FibreOps exposes a JSON API that the action plugin can call: /api/runs — List and query agent runs /api/optimiser — Get optimizer scores and suggestions /sdk/chat — Natural language interaction with the agent system /healthz — Liveness probe Because FastAPI auto-generates OpenAPI schemas from your typed Python endpoints, the action plugin gets accurate parameter descriptions, response schemas, and error codes without any manual specification work. Publishing as Autopilots (Public Preview) Autopilots take distribution one step further — agents that operate autonomously without requiring a user to initiate each interaction. An Autopilot can: React to events (e.g., a critical telemetry signal) without human initiation Take actions within defined guardrails Notify users only when human intervention is needed Operate continuously across Microsoft 365 surfaces For FibreOps, an Autopilot would monitor the Event Hub stream continuously and only surface to the NOC team when an incident exceeds automated resolution capability — a fully autonomous operations agent. Teams Adaptive Cards FibreOps posts rich Adaptive Card notifications to Microsoft Teams throughout the agent pipeline. This is separate from the declarative agent — it is a push notification channel for real-time operational awareness. # The NetOps agent posts an outage notice via Incoming Webhook def post_outage_notice(incident_id, node_id, severity, summary, engineer=None): card = { "type": "AdaptiveCard", "body": [ {"type": "TextBlock", "text": f"🚨 Outage: {node_id}", "weight": "Bolder", "size": "Large"}, {"type": "FactSet", "facts": [ {"title": "Severity", "value": severity.upper()}, {"title": "Incident", "value": incident_id}, {"title": "Summary", "value": summary}, ]}, ], "actions": [ {"type": "Action.OpenUrl", "title": "View in NOC Console", "url": f"{base_url}/runs/{incident_id}"} ] } # POST to Teams webhook or append to outbox for offline mode ... If TEAMS_WEBHOOK_URL is not configured, cards are appended to state/teams_outbox.jsonl for review in the NOC console's Teams panel. End-to-End: From Code to Copilot Here is the complete flow from development to distribution: Build — Develop agents with Microsoft Agent Framework, test locally with python -m fibreops.demo --backend local Publish agents — python -m fibreops.demo publish creates hosted Prompt Agents in Foundry Deploy infrastructure — azd up provisions App Service, ACR, Event Hub, Key Vault, and Application Insights Deploy hosted agent — azd env set FIBREOPS_DEPLOY_HOSTED true && azd up Generate M365 package — python -m fibreops.demo publish-m365 --out dist/m365 Upload to Teams — Upload fibreops-copilot.zip via Teams Admin Center Users interact — The agent is now available in Microsoft 365 Copilot and Teams Security Considerations Managed Identity — The deployed app uses system-assigned managed identity for all Azure service access. No secrets in code. Least privilege — Each role grant is scoped to the minimum required (Event Hubs Data Owner, Key Vault Secrets User, AcrPull, Azure AI Developer). Authentication — The M365 Copilot surface handles user authentication; your backend receives authenticated requests. Guardrails — Autopilots operate within defined boundaries; human-in-the-loop escalation is built into the Routine and agent decision logic. Key Takeaways Publishing to Teams and M365 Copilot is GA — a single command generates the complete package. Declarative agents separate distribution (M365) from intelligence (your backend). Action plugins leverage your existing FastAPI OpenAPI schema — no manual specification needed. Autopilots (Public Preview) enable fully autonomous operation within guardrails. Adaptive Cards provide real-time push notifications alongside the conversational agent surface. The same backend serves the NOC console, the Copilot SDK, and the M365 declarative agent. Next Steps Explore the FibreOps repository — try python -m fibreops.demo publish-m365 Microsoft 365 Copilot extensibility documentation Next in this series: Voice Live and Observability for Production Agent SystemsBuilding HIPAA-Compliant Medical Transcription with Local AI
Building HIPAA-Compliant Medical Transcription with Local AI Introduction Healthcare organizations generate vast amounts of spoken content, patient consultations, research interviews, clinical notes, medical conferences. Transcribing these recordings traditionally requires either manual typing (time-consuming and expensive) or cloud transcription services (creating immediate HIPAA compliance concerns). Every audio file sent to external APIs exposes Protected Health Information (PHI), requires Business Associate Agreements, creates audit trails on third-party servers, and introduces potential breach vectors. This sample solution lies in on-premises voice-to-text systems that process audio entirely locally, never sending PHI beyond organizational boundaries. This article demonstrates building a sample medical transcription application using FLWhisper, ASP.NET Core, C#, and Microsoft Foundry Local with OpenAI Whisper models. You'll learn how to build sample HIPAA-compliant audio processing, integrate Whisper models for medical terminology accuracy, design privacy-first API patterns, and build responsive web UIs for healthcare workflows. Whether you're developing electronic health record (EHR) integrations, building clinical research platforms, or implementing dictation systems for medical practices, this sample could be a great starting point for privacy-first speech recognition. Why Local Transcription Is Critical for Healthcare Healthcare data handling is fundamentally different from general business data due to HIPAA regulations, state privacy laws, and professional ethics obligations. Understanding these requirements explains why cloud transcription services, despite their convenience, create unacceptable risks for medical applications. HIPAA compliance mandates strict controls over PHI. Every system that touches patient data must implement administrative, physical, and technical safeguards. Cloud transcription APIs require Business Associate Agreements (BAAs), but even with paperwork, you're entrusting PHI to external systems. Every API call creates logs on vendor servers, potentially in multiple jurisdictions. Data breaches at transcription vendors expose patient information, creating liability for healthcare organizations. On-premises processing eliminates these third-party risks entirely, PHI never leaves your controlled environment. US State laws increasingly add requirements beyond HIPAA. California's CCPA, New York's SHIELD Act, and similar legislation create additional compliance obligations. International regulations like GDPR prohibit transferring health data outside approved jurisdictions. Local processing simplifies compliance by keeping data within organizational boundaries. Research applications face even stricter requirements. Institutional Review Boards (IRBs) often require explicit consent for data sharing with external parties. Cloud transcription may violate study protocols that promise "no third-party data sharing." Clinical trials in pharmaceutical development handle proprietary information alongside PHI, double jeopardy for data exposure. Local transcription maintains research integrity while enabling audio analysis. Cost considerations favor local deployment at scale. Medical organizations generate substantial audio, thousands of patient encounters monthly. Cloud APIs charge per minute of audio, creating significant recurring costs. Local models have fixed infrastructure costs that scale economically. A modest GPU server can process hundreds of hours monthly at predictable expense. Latency matters for clinical workflows. Doctors and nurses need transcriptions available immediately after patient encounters to review and edit while details are fresh. Cloud APIs introduce network delays, especially problematic in rural health facilities with limited connectivity. Local inference provides <1 second turnaround for typical consultation lengths. Application Architecture: ASP.NET Core with Foundry Local The sample FLWhisper application implements clean separation between audio handling, AI inference, and state management using modern .NET patterns: The ASP.NET Core 10 minimal API provides HTTP endpoints for health checks, audio transcription, and sample file streaming. Minimal APIs reduce boilerplate while maintaining full middleware support for error handling, authentication, and CORS. The API design follows OpenAI's transcription endpoint specification, enabling drop-in replacement for existing integrations. The service layer encapsulates business logic: FoundryModelService manages model loading and lifetime, TranscriptionService handles audio processing and AI inference, and SampleAudioService provides demonstration files for testing. This separation enables easy testing, dependency injection, and service swapping. Foundry Local integration uses the Microsoft.AI.Foundry.Local.WinML SDK. Unlike cloud APIs requiring authentication and network calls, this SDK communicates directly with the local Foundry service via in-process calls. Models load once at startup, remaining resident in memory for sub-second inference on subsequent requests. The static file frontend delivers vanilla HTML/CSS/JavaScript, no framework overhead. This simplicity aids healthcare IT security audits and enables deployment on locked-down hospital networks. The UI provides file upload, sample selection, audio preview, transcription requests, and result display with copy-to-clipboard functionality. Here's the architectural flow for transcription requests: Web UI (Upload Audio File) ↓ POST /v1/audio/transcriptions (Multipart Form Data) ↓ ASP.NET Core API Route ↓ TranscriptionService.TranscribeAudio(audioStream) ↓ Foundry Local Model (Whisper Medium locally) ↓ Text Result + Metadata (language, duration) ↓ Return JSON/Text Response ↓ Display in UI This architecture embodies several healthcare system design principles: Data never leaves the device: All processing occurs on-premises, no external API calls No data persistence by default: Audio and transcripts are session-only, never saved unless explicitly configured Comprehensive health checks: System readiness verification before accepting PHI Audit logging support: Structured logging for compliance documentation Graceful degradation: Clear error messages when models unavailable rather than silent failures Setting Up Foundry Local with Whisper Models Foundry Local supports multiple Whisper model sizes, each with different accuracy/speed tradeoffs. For medical transcription, accuracy is paramount—misheard drug names or dosages create patient safety risks: # Install Foundry Local (Windows) winget install Microsoft.FoundryLocal # Verify installation foundry --version # Download Whisper Medium model (optimal for medical accuracy) foundry model add openai-whisper-medium-generic-cpu:1 # Check model availability foundry model list Whisper Medium (769M parameters) provides the best balance for medical use. Smaller models (Tiny, Base) miss medical terminology frequently. Larger models (Large) offer marginal accuracy gains at 3x inference time. Medium handles medical vocabulary well, drug names, anatomical terms, procedure names, while processing typical consultation audio (5-10 minutes) in under 30 seconds. The application detects and loads the model automatically: // Services/FoundryModelService.cs using Microsoft.AI.Foundry.Local.WinML; public class FoundryModelService { private readonly ILogger _logger; private readonly FoundryOptions _options; private ILocalAIModel? _loadedModel; public FoundryModelService( ILogger logger, IOptions options) { _logger = logger; _options = options.Value; } public async Task InitializeModelAsync() { try { _logger.LogInformation( "Loading Foundry model: {ModelAlias}", _options.ModelAlias ); // Load model from Foundry Local _loadedModel = await FoundryClient.LoadModelAsync( modelAlias: _options.ModelAlias, cancellationToken: CancellationToken.None ); if (_loadedModel == null) { _logger.LogWarning("Model loaded but returned null instance"); return false; } _logger.LogInformation( "Successfully loaded model: {ModelAlias}", _options.ModelAlias ); return true; } catch (Exception ex) { _logger.LogError( ex, "Failed to load Foundry model: {ModelAlias}", _options.ModelAlias ); return false; } } public ILocalAIModel? GetLoadedModel() => _loadedModel; public async Task UnloadModelAsync() { if (_loadedModel != null) { await FoundryClient.UnloadModelAsync(_loadedModel); _loadedModel = null; _logger.LogInformation("Model unloaded"); } } } Configuration lives in appsettings.json , enabling easy customization without code changes: { "Foundry": { "ModelAlias": "whisper-medium", "LogLevel": "Information" }, "Transcription": { "MaxAudioDurationSeconds": 300, "SupportedFormats": ["wav", "mp3", "m4a", "flac"], "DefaultLanguage": "en" } } Implementing Privacy-First Transcription Service The transcription service handles audio processing while maintaining strict privacy controls. No audio or transcript persists beyond the HTTP request lifecycle unless explicitly configured: // Services/TranscriptionService.cs public class TranscriptionService { private readonly FoundryModelService _modelService; private readonly ILogger _logger; public async Task TranscribeAudioAsync( Stream audioStream, string originalFileName, TranscriptionOptions? options = null) { options ??= new TranscriptionOptions(); var startTime = DateTime.UtcNow; try { // Validate audio format ValidateAudioFormat(originalFileName); // Get loaded model var model = _modelService.GetLoadedModel(); if (model == null) { throw new InvalidOperationException("Whisper model not loaded"); } // Create temporary file (automatically deleted after transcription) using var tempFile = new TempAudioFile(audioStream); // Execute transcription _logger.LogInformation( "Starting transcription for file: {FileName}", originalFileName ); var transcription = await model.TranscribeAsync( audioFilePath: tempFile.Path, language: options.Language, cancellationToken: CancellationToken.None ); var duration = (DateTime.UtcNow - startTime).TotalSeconds; _logger.LogInformation( "Transcription completed in {Duration:F2}s", duration ); return new TranscriptionResult { Text = transcription.Text, Language = transcription.Language ?? options.Language, Duration = transcription.AudioDuration, ProcessingTimeSeconds = duration, FileName = originalFileName, Timestamp = DateTime.UtcNow }; } catch (Exception ex) { _logger.LogError( ex, "Transcription failed for file: {FileName}", originalFileName ); throw; } } private void ValidateAudioFormat(string fileName) { var extension = Path.GetExtension(fileName).TrimStart('.'); var supportedFormats = new[] { "wav", "mp3", "m4a", "flac", "ogg" }; if (!supportedFormats.Contains(extension.ToLowerInvariant())) { throw new ArgumentException( $"Unsupported audio format: {extension}. " + $"Supported: {string.Join(", ", supportedFormats)}" ); } } } // Temporary file wrapper that auto-deletes internal class TempAudioFile : IDisposable { public string Path { get; } public TempAudioFile(Stream sourceStream) { Path = System.IO.Path.GetTempFileName(); using var fileStream = File.OpenWrite(Path); sourceStream.CopyTo(fileStream); } public void Dispose() { try { if (File.Exists(Path)) { File.Delete(Path); } } catch { // Ignore deletion errors in temp folder } } } This service demonstrates several privacy-first patterns: Temporary file lifecycle management: Audio written to temp storage, automatically deleted after transcription No implicit persistence: Results returned to caller, not saved by service Format validation: Accept only supported audio formats to prevent processing errors Comprehensive logging: Audit trail for compliance without logging PHI content Error isolation: Exceptions contain diagnostic info but no patient data Building the OpenAI-Compatible REST API The API endpoint mirrors OpenAI's transcription API specification, enabling existing integrations to work without modifications: // Program.cs var builder = WebApplication.CreateBuilder(args); // Configure services builder.Services.Configure( builder.Configuration.GetSection("Foundry") ); builder.Services.AddSingleton(); builder.Services.AddScoped(); builder.Services.AddHealthChecks() .AddCheck("foundry-health"); var app = builder.Build(); // Load model at startup var modelService = app.Services.GetRequiredService(); await modelService.InitializeModelAsync(); app.UseHealthChecks("/health"); app.MapHealthChecks("/api/health/status"); // OpenAI-compatible transcription endpoint app.MapPost("/v1/audio/transcriptions", async ( HttpRequest request, TranscriptionService transcriptionService, ILogger logger) => { if (!request.HasFormContentType) { return Results.BadRequest(new { error = "Content-Type must be multipart/form-data" }); } var form = await request.ReadFormAsync(); // Extract audio file var audioFile = form.Files.GetFile("file"); if (audioFile == null || audioFile.Length == 0) { return Results.BadRequest(new { error = "Audio file required in 'file' field" }); } // Parse options var format = form["format"].ToString() ?? "text"; var language = form["language"].ToString() ?? "en"; try { // Process transcription using var stream = audioFile.OpenReadStream(); var result = await transcriptionService.TranscribeAudioAsync( audioStream: stream, originalFileName: audioFile.FileName, options: new TranscriptionOptions { Language = language } ); // Return in requested format if (format == "json") { return Results.Json(new { text = result.Text, language = result.Language, duration = result.Duration }); } else { // Default: plain text return Results.Text(result.Text); } } catch (Exception ex) { logger.LogError(ex, "Transcription request failed"); return Results.StatusCode(500); } }) .DisableAntiforgery() // File uploads need CSRF exemption .WithName("TranscribeAudio") .WithOpenApi(); app.Run(); Example API usage: # PowerShell $audioFile = Get-Item "consultation-recording.wav" $response = Invoke-RestMethod ` -Uri "http://localhost:5192/v1/audio/transcriptions" ` -Method Post ` -Form @{ file = $audioFile; format = "json" } Write-Output $response.text # cURL curl -X POST http://localhost:5192/v1/audio/transcriptions \ -F "file=@consultation-recording.wav" \ -F "format=json" Building the Interactive Web Frontend The web UI provides a user-friendly interface for non-technical medical staff to transcribe recordings: SarahCare Medical Transcription The JavaScript handles file uploads and API interactions: // wwwroot/app.js let selectedFile = null; async function checkHealth() { try { const response = await fetch('/health'); const statusEl = document.getElementById('status'); if (response.ok) { statusEl.className = 'status-badge online'; statusEl.textContent = '✓ System Ready'; } else { statusEl.className = 'status-badge offline'; statusEl.textContent = '✗ System Unavailable'; } } catch (error) { console.error('Health check failed:', error); } } function handleFileSelect(event) { const file = event.target.files[0]; if (!file) return; selectedFile = file; // Show file info const fileInfo = document.getElementById('fileInfo'); fileInfo.textContent = `Selected: ${file.name} (${formatFileSize(file.size)})`; fileInfo.classList.remove('hidden'); // Enable audio preview const preview = document.getElementById('audioPreview'); preview.src = URL.createObjectURL(file); preview.classList.remove('hidden'); // Enable transcribe button document.getElementById('transcribeBtn').disabled = false; } async function transcribeAudio() { if (!selectedFile) return; const loadingEl = document.getElementById('loadingIndicator'); const resultEl = document.getElementById('resultSection'); const transcribeBtn = document.getElementById('transcribeBtn'); // Show loading state loadingEl.classList.remove('hidden'); resultEl.classList.add('hidden'); transcribeBtn.disabled = true; try { const formData = new FormData(); formData.append('file', selectedFile); formData.append('format', 'json'); const startTime = Date.now(); const response = await fetch('/v1/audio/transcriptions', { method: 'POST', body: formData }); if (!response.ok) { throw new Error(`HTTP ${response.status}: ${response.statusText}`); } const result = await response.json(); const processingTime = ((Date.now() - startTime) / 1000).toFixed(1); // Display results document.getElementById('transcriptionText').value = result.text; document.getElementById('resultDuration').textContent = `Duration: ${result.duration.toFixed(1)}s`; document.getElementById('resultLanguage').textContent = `Language: ${result.language}`; resultEl.classList.remove('hidden'); console.log(`Transcription completed in ${processingTime}s`); } catch (error) { console.error('Transcription failed:', error); alert(`Transcription failed: ${error.message}`); } finally { loadingEl.classList.add('hidden'); transcribeBtn.disabled = false; } } function copyToClipboard() { const text = document.getElementById('transcriptionText').value; navigator.clipboard.writeText(text) .then(() => alert('Copied to clipboard')) .catch(err => console.error('Copy failed:', err)); } // Initialize window.addEventListener('load', () => { checkHealth(); loadSamplesList(); }); Key Takeaways and Production Considerations Building HIPAA-compliant voice-to-text systems requires architectural decisions that prioritize data privacy over convenience. The FLWhisper application demonstrates that you can achieve accurate medical transcription, fast processing times, and intuitive user experiences entirely on-premises. Critical lessons for healthcare AI: Privacy by architecture: Design systems where PHI never exists outside controlled environments, not as a configuration option No persistence by default: Audio and transcripts should be ephemeral unless explicitly saved with proper access controls Model selection matters: Whisper Medium provides medical terminology accuracy that smaller models miss Health checks enable reliability: Systems should verify model availability before accepting PHI Audit logging without content logging: Track operations for compliance without storing sensitive data in logs For production deployment in clinical settings, integrate with EHR systems via HL7/FHIR interfaces. Implement role-based access control with Active Directory integration. Add digital signatures for transcript authentication. Configure automatic PHI redaction using clinical NLP models. Deploy on HIPAA-compliant infrastructure with proper physical security. Implement comprehensive audit logging meeting compliance requirements. The complete implementation with ASP.NET Core API, Foundry Local integration, sample audio files, and comprehensive tests is available at github.com/leestott/FLWhisper. Clone the repository and follow the setup guide to experience privacy-first medical transcription. Resources and Further Reading FLWhisper Repository - Complete C# implementation with .NET 10 Quick Start Guide - Installation and usage instructions Microsoft Foundry Local Documentation - SDK reference and model catalog OpenAI Whisper Documentation - Model architecture and capabilities HIPAA Compliance Guidelines - HHS official guidance Testing Guide - Comprehensive test suite documentationBuilding Autonomous Agents with Microsoft Agent Framework and GitHub Copilot SDK Part 2/5
This is the second post in our series on the Microsoft agent platform. Here we dive deep into building autonomous agents, the development experience, the Microsoft Agent Framework, tool design patterns, and how the GitHub Copilot SDK brings conversational AI to your agent system. All examples reference the FibreOps repository, an autonomous fibre outage response system demonstrated at Microsoft Build BRK241. The Microsoft Agent Framework The Microsoft Agent Framework (now GA) provides a unified programming model for building agents. It supports multiple backends through a single .run() contract: Hosted — FoundryAgent connected to a Prompt Agent published to Microsoft Foundry Agent Service. Foundry — Agent + FoundryChatClient with the definition resolved locally (ideal for prompt iteration). Local — Deterministic LocalAgent for offline development and testing. This design means your orchestration code never changes regardless of where the agent runs. The factory pattern in FibreOps selects the backend at startup: # src/fibreops/agents/factory.py — simplified from agent_framework_foundry import FoundryAgent from agent_framework import Agent, FoundryChatClient def build_agent(role: str, backend: str, config: Config): if backend == "hosted": return FoundryAgent(agent_id=config.foundry_agents[role]) elif backend == "foundry": return Agent( instructions=get_instructions(role), chat_client=FoundryChatClient(endpoint=config.endpoint), tools=get_tools(role), ) else: return LocalAgent(role=role) Set FIBREOPS_AGENT_BACKEND to override the backend, or leave it as auto for intelligent detection. Designing Role-Specialised Agents FibreOps demonstrates a key pattern: role specialisation. Rather than one monolithic agent, the system uses three focused agents, each with a clear responsibility boundary: Agent Role Tools Available IncidentAnalysisAgent Classify severity, find root cause, retrieve SOP Knowledge (SOPs + topology), Web IQ, Work IQ NetOpsCoordinatorAgent File D365 incident, post Teams notice Ticketing, Teams, Memory FieldDispatchAgent Select engineer, book resource, update team Dispatch, Teams, Voice Why Role Specialisation? Focused system prompts — Each agent has a tightly scoped instruction set, reducing hallucination and improving reliability. Independent evaluation — You can score each agent separately against role-specific criteria. Parallel development — Teams can iterate on agents independently. Selective upgrade — Swap one agent's model or implementation without touching others. Tool Design: Typed Python Functions Tools in the Microsoft Agent Framework are typed Python functions that the runtime supplies to the hosted agent definition. FibreOps demonstrates several tool categories: Knowledge Tools # src/fibreops/tools/knowledge.py — simplified def sop_lookup(node_id: str, signal_type: str) -> dict: """Retrieve the Standard Operating Procedure for a given signal type. Args: node_id: The fibre node identifier (e.g., FN-LDN-001) signal_type: The type of signal (loss_of_light, high_ber, signal_degradation) Returns: SOP with steps, escalation path, and estimated resolution time. """ # Load from local markdown SOPs or Foundry IQ ... def web_iq_search(query: str, *, limit: int = 5) -> list[dict]: """Search public web for context relevant to the incident. Grounding against roadworks, weather, power outages, splice guidance. Falls back to deterministic fixtures when endpoint is unset. """ ... def work_iq_search(query: str, *, limit: int = 5) -> list[dict]: """Search enterprise knowledge for context relevant to the incident. Site surveys, SLA tiers, competency matrix, MTTR trends. """ ... Integration Tools # src/fibreops/tools/teams.py — simplified def post_outage_notice( incident_id: str, node_id: str, severity: str, summary: str, engineer: str | None = None, ) -> dict: """Post an Adaptive Card outage notice to the configured Teams channel. If TEAMS_WEBHOOK_URL is not set, appends to state/teams_outbox.jsonl for offline review. """ card = build_adaptive_card(incident_id, node_id, severity, summary, engineer) if config.teams_webhook_url: requests.post(config.teams_webhook_url, json=card) else: append_to_outbox(card) return {"status": "posted", "incident_id": incident_id} Design Principles for Agent Tools Typed parameters with docstrings — The runtime uses type hints and docstrings to generate the tool schema for the LLM. Graceful degradation — Every tool works offline by falling back to local fixtures or file-based state. Idempotent where possible — Tools that create resources return existing records if called with the same parameters. Observable — Every tool invocation emits an OpenTelemetry span for tracing and debugging. The Orchestrator Pattern The orchestrator drives signals through the agent pipeline. It is deliberately simple — a linear flow with error handling: # src/fibreops/orchestrator.py — simplified async def handle_signal(signal: TelemetrySignal) -> RunResult: """Process a telemetry signal through the agent pipeline.""" # Stage 1: Incident Analysis analysis = await incident_agent.run( f"Analyse this signal: {signal.model_dump_json()}" ) # Stage 2: NetOps Coordination coordination = await netops_agent.run( f"Coordinate response for: {analysis.summary}" ) # Stage 3: Field Dispatch dispatch = await dispatch_agent.run( f"Dispatch engineer for incident: {coordination.incident_id}" ) return RunResult( signal=signal, analysis=analysis, coordination=coordination, dispatch=dispatch, ) The orchestrator honours the same contract regardless of backend — hosted , foundry , or local — because all backends implement await agent.run(prompt) . GitHub Copilot SDK Integration (GA) The GitHub Copilot SDK enables conversational interaction with your agent system. FibreOps implements FibreOpsCopilotClient with the same interface as github/copilot-sdk : # src/fibreops/sdk/__init__.py — simplified from fibreops.sdk.client import FibreOpsCopilotClient client = FibreOpsCopilotClient() session = client.create_session() # Query agent status response = session.send_and_wait("status") print(response.text) # Human-readable summary print(response.data) # Structured JSON # Inject a telemetry signal via conversation response = session.send_and_wait(json.dumps({ "signal_id": "sig-demo", "node_id": "FN-LDN-001", "signal_type": "loss_of_light", "severity": "critical" })) The adapter routes prompts by shape: JSON signal-shaped dicts — Forwarded to the orchestrator for processing. Free-form text — Answered by a deterministic responder ( help , status , nodes , engineers , optimiser , dispatch ). Drive it from the terminal: python -m fibreops.demo chat "help" python -m fibreops.demo chat "status" python -m fibreops.demo chat '{"signal_id":"sig-demo","node_id":"FN-LDN-001","signal_type":"loss_of_light","severity":"critical"}' Or hit the embedded HTTP endpoint when the NOC console is running: Invoke-RestMethod -Method Post http://127.0.0.1:8800/sdk/chat -Body '{"prompt":"status"}' -ContentType application/json Development Workflow with Foundry Toolkit for VS Code The Foundry Toolkit for VS Code provides an integrated development experience: Author prompts — Edit system instructions with live preview and token counting. Test locally — Run against the foundry backend with FoundryChatClient pointing at your development model. Iterate fast — The foundry backend resolves definitions locally, so prompt changes take effect immediately without republishing. Publish when ready — python -m fibreops.demo publish creates hosted Prompt Agents in Foundry. Multi-Model Support The Microsoft Agent Framework supports multiple models. FibreOps defaults to gpt-4.1-mini (the model available in most demo Foundry accounts), but any chat-completions deployment works: # .env AZURE_AI_MODEL_DEPLOYMENT=gpt-4.1-mini # or gpt-4o-mini, gpt-4o, gpt-4.1 The framework also supports Claude Code connectors and Magentic-One for multi-agent collaboration scenarios. Testing Strategy FibreOps demonstrates a layered testing approach: Unit tests — Test tools in isolation with mocked dependencies. Local backend tests — Run the full pipeline with LocalAgent for deterministic assertions. Integration tests — Run against real Foundry agents with pytest -q . Rubric evaluation — The optimizer scores every run against defined criteria. # Run the test suite .\.venv\Scripts\python.exe -m pytest -q Key Takeaways The Microsoft Agent Framework provides a unified .run() contract across hosted, foundry, and local backends. Role specialisation keeps agents focused, testable, and independently evolvable. Tools are typed Python functions with docstrings — the runtime generates schemas automatically. The GitHub Copilot SDK (GA) enables conversational interaction with any agent system. Graceful degradation means the entire system works offline for development. The factory pattern lets you switch backends without changing orchestration code. Next Steps Clone the FibreOps repository and run python -m fibreops.demo --signals 3 Microsoft Agent Framework documentation Next in this series: Running Hosted Agents in Microsoft Foundry Agent Service
Vector search finds candidates. Reranking decides what your RAG app reads
You ask a retrieval-augmented generation (RAG) application a question. Vector search returns ten passages that are clearly related to the topic. The passage that actually contains the answer, however, is ranked seventh, while the language model receives only the first five. Retrieval did not completely fail. It found the evidence, but ordered it below less useful context. Reranking addresses that gap between a passage that is semantically similar and a passage that is relevant to the user's specific question. This article demonstrates that pattern in four Azure services using the Stanford Question Answering Dataset (SQuAD). The goal is not to declare a winning service or publish a quality benchmark. It is to show where retrieval, rank fusion, and model-based reranking run in each architecture, and to illustrate how the position of a known source passage can change. What this demonstration establishes The examples show rank movement for three selected questions. They do not establish that one reranker or service is universally more accurate. A production decision requires a larger, representative query set and aggregate relevance, latency, and cost measurements. Get the full Python implementation: pauldj54/azure-vector-reranking-squad Retrieval and reranking are different stages A production search pipeline commonly uses two stages: Retrieve for recall. Fast retrieval narrows a large corpus to a bounded candidate set. It can use vector search, keyword search, or both. Rerank for precision. A more expensive model evaluates only those candidates against the original query and produces the final order. Reciprocal Rank Fusion (RRF) belongs between those two ideas. RRF is a model-free rank aggregation method that merges independent result lists, usually vector and keyword results. For a document d, a typical score is: RRF(d) = ∑ r ∈ R 1 k + rank r (d) Here, R is the set of ranked lists and k is commonly 60. RRF works with positions rather than raw scores, so it can combine signals such as cosine distance and BM25 without pretending their score scales are comparable. This gives a clearer three-part vocabulary: Stage Purpose Typical mechanism Retrieve Find broad candidate set Vector search, BM25, filters Fuse Combine independent rankings RRF Rerank Reassess query-document relevance Semantic ranker or cross-encoder RRF often improves hybrid retrieval when exact names, dates, identifiers, or terms matter. A learned reranker can then read the query and each candidate together, capturing interactions that separately generated embeddings can miss. The learned stage costs more, so it should operate on tens of candidates rather than the whole corpus. The following image describes the general process: Why use SQuAD for this demonstration? SQuAD 1.1 contains crowd-written questions over more than 500 Wikipedia articles. Its packaged splits contain 87,599 training rows and 10,570 validation rows. Each row includes a question, a context passage, and one or more answer spans inside that passage. That source-context mapping gives this demonstration a useful label: the context associated with a question is treated as its gold passage. We can then inspect whether each search stage moves that passage up or down. This is convenient, but it is not a perfect passage-ranking benchmark. SQuAD was designed for extractive question answering, and another passage in the corpus might also answer a question. The gold context is therefore a reproducible reference, not proof that every other passage is irrelevant. The results shown here use the 2,067 unique contexts in the SQuAD validation split and 1,536-dimensional embeddings. The repository default should be set to the same corpus size before treating the screenshots or rank transitions as directly reproducible. Three illustrative questions Question Expected answer Gold context According to game stats, which Super Bowl 50 quarterback had his worst year since his first NFL season? Peyton Manning 12, Super Bowl 50 What else did Tesla do for work at this time? Various electrical repair jobs 165, Nikola Tesla Who acts as laborer, paymaster, and design team for a renovation project? The property owner 1306, Construction Each notebook selects a seeded demonstration question when it runs. The three saved examples were collected across separate runs; the current notebooks do not execute all three questions in one pass. A benchmark harness should iterate over a fixed question list and save all stage results in one structured output. Capability boundaries at a glance Service Retrieval and Fusion Learned Reranking Boundary to Keep in Mind Azure AI Search Native keyword and vector retrieval with native RRF Built-in semantic ranker Semantic ranking only reorders the retrieved top 50 Azure SQL Database Exact vector retrieval in the current notebook External Cohere model invoked through native REST procedure SQL issues the HTTPS request; Foundry performs inference PostgreSQL Flexible Server pgvector plus hand-written SQL RRF over full-text search Optional external Cohere call from Python Retrieval primitives are native; this RRF query and Cohere path are application code Azure Cosmos DB for NoSQL Native vector search and native hybrid RRF SDK-integrated Semantic Reranker, currently preview Reranking is a separate inference call over at most 50 supplied documents Azure AI Search: native hybrid retrieval and semantic ranking How it works: Azure AI Search provides the most integrated pipeline in this demonstration. A hybrid query runs keyword and vector retrieval, combines the lists with RRF, and passes up to the top 50 results to the built-in semantic ranker. The semantic ranker assigns @search.rerankerScore values from 0 to 4 and can return extractive captions and answers. The semantic configuration identifies the fields that carry the meaning of each document: semantic_search = SemanticSearch( configurations=[ SemanticConfiguration( name=SEMANTIC_CONFIG, prioritized_fields=SemanticPrioritizedFields( title_field=SemanticField(field_name="title"), content_fields=[SemanticField(field_name="content")], ), ) ] ) This tells the semantic ranker which text fields to evaluate. The query then enables semantic ranking after hybrid retrieval: results = search_client.search( search_text=question, vector_queries=[vector_query], query_type="semantic", semantic_configuration_name=SEMANTIC_CONFIG, top=10, ) The important constraint is candidate recall. Semantic ranking does not search the corpus again. If the correct passage is absent from the hybrid top 50, the semantic stage cannot recover it. See 01_azure_ai_search_reranking.ipynb for the complete setup and query path. Test results for Azure AI Search These examples show that semantic reranking improves relevance selectively, not universally. It strongly helps the construction query, moving the correct passage from rank 4 to rank 1, but slightly degrades the Super Bowl and Tesla queries by one position. This reinforces that semantic ranking should be evaluated across a representative query set using aggregate metrics such as MRR or NDCG, rather than judged from a single result. Azure SQL Database: vector retrieval plus external Cohere reranking How it works. The Azure SQL notebook retrieves 20 candidates with exact cosine distance and sends their text to Cohere Rerank v4.0 Fast through sys.sp_invoke_external_rest_endpoint. The vector column and query vector must have the same dimensions. This repository uses 1,536-dimensional embeddings: SELECT TOP (@ candidate_count) context_id, title, content, 1 - VECTOR_DISTANCE( 'cosine', CAST(@ query_vector AS VECTOR(1536)), embedding ) AS similarity FROM dbo.documents ORDER BY similarity DESC; For reranking, we selected Cohere Rerank v4.0 Fast (Cohere-rerank-v4.0-fast), a fast version of Cohere’s fourth-generation relevance-ranking model. The model is deployed in Microsoft Foundry, where its Azure Direct inference endpoint is available in the deployment details within the Foundry portal. Azure SQL can call REST APIs directly using sp_invoke_external_rest_endpoint. Because Azure SQL allowlists Azure AI’s *.cognitiveservices.azure.com domain, we translate the equivalent Foundry endpoint from *.services.ai.azure.com while preserving the Cohere reranking route. from urllib.parse import urlsplit, urlunsplit def sql_compatible_endpoint(endpoint: str) -> str: """Convert an Azure Direct endpoint to Azure SQL's allowed hostname.""" parts = urlsplit(endpoint) if parts.hostname.endswith(".services.ai.azure.com"): resource = parts.hostname.removesuffix(".services.ai.azure.com") hostname = f"{resource}.cognitiveservices.azure.com" elif parts.hostname.endswith(".cognitiveservices.azure.com"): hostname = parts.hostname else: raise ValueError("Expected an Azure AI Services endpoint.") return urlunsplit( (parts.scheme, hostname, parts.path, parts.query, "") ) Then I defined a re-rank with cohere function, starting by loading the endpoint and setting the authentication: def rerank_with_cohere( cursor, question: str, candidates: list[dict], top_n: int = 10, ) -> list[dict]: """ Rerank candidate documents by calling Cohere through Azure SQL. Each candidate must contain a 'content' field. """ if not candidates: return [] sql_endpoint = sql_compatible_endpoint( os.environ["COHERE_RERANK_ENDPOINT"] ) model = os.environ["COHERE_RERANK_MODEL"] access_token = credential.get_token( "https://cognitiveservices.azure.com/.default" ).token headers = json.dumps({"Authorization": f"Bearer {access_token}"}) payload = json.dumps( { "model": model, "query": question, "documents": [row["content"] for row in candidates], "top_n": min(k, len(candidates)), }, ensure_ascii=False, ) cursor.execute( """ DECLARE @url NVARCHAR(4000) = CAST(? AS NVARCHAR(4000)); DECLARE @headers NVARCHAR(4000) = CAST(? AS NVARCHAR(4000)); DECLARE Payload NVARCHAR(MAX) = CAST(? AS NVARCHAR(MAX)); DECLARE Response NVARCHAR(MAX); DECLARE @status INT; EXEC @status = sys.sp_invoke_external_rest_endpoint @url = @url, @method = 'POST', @headers = @headers, Payload = Payload, @timeout = 60, @retry_count = 2, Response = Response OUTPUT; SELECT @status, Response; """, sql_endpoint, headers, payload, ) status, response_text = cursor.fetchone() if status != 0: raise RuntimeError(f"Reranker endpoint returned HTTP status {status}.") response = json.loads(response_text)["result"] You can see the complete implementation in the 02_azure_sql_reranking.ipynb notebook. Test results for Azure SQL Db Across the three sample questions, Cohere reranking consistently moved the correct SQuAD passage closer to the top: from rank 5 to 1 for the Super Bowl question, 3 to 2 for the Tesla question, and 8 to 1 for the construction question. These examples show how vector search provides a strong candidate set, while reranking applies deeper query-document relevance scoring to improve the final ordering. The results are illustrative rather than a complete quality benchmark, so broader evaluation across many queries is still recommended. Azure Database for PostgreSQL flexible server: pgvector, SQL RRF, and an optional model How it works: PostgreSQL makes the pipeline components explicit. The notebook uses pgvector for vector similarity, PostgreSQL full-text search for keyword retrieval, and SQL to implement RRF. Vector retrieval uses cosine distance: SELECT context_id, title, content, 1 - (embedding <= > % (query_vector) s:: vector) AS similarity FROM squad_docs ORDER BY embedding <= > % (query_vector) s:: vector LIMIT % (candidate_count) s; The hybrid query independently ranks vector and keyword hits, then combines positions rather than raw scores: SELECT d.context_id, COALESCE(1.0 / (60 + v.rank), 0) + COALESCE(1.0 / (60 + k.rank), 0) AS rrf_score FROM squad_docs AS d LEFT JOIN vector_hits AS v USING (context_id) LEFT JOIN keyword_hits AS k USING (context_id) WHERE v.context_id IS NOT NULL OR k.context_id IS NOT NULL ORDER BY rrf_score DESC; This is not a built-in PostgreSQL RRF operator. It is transparent, hand-written SQL over native retrieval primitives, which makes weighting and debugging flexible but leaves implementation and tuning with the application team. The notebook's optional learned stage sends the vector candidates from Python to a Foundry deployment of Cohere Rerank v4.0 Fast. This path was chosen because the tested Flexible Server azure_ai extension version expected the older serverless reranking endpoint contract. Microsoft documentation still describes azure_ai.rank() as a preview function whose default model is Cohere Rerank v3.5, even though that model retired on May 14, 2026. Treat this as a version-specific compatibility issue and verify current extension behavior before selecting an architecture. Azure HorizonDB is a different product path. Its AI Model Management feature can provision Cohere Rerank v4.0 Fast as default-reranker, but that management feature is currently a limited preview. It should not be described as a generally available Flexible Server capability. See 03_azure_postgres_reranking.ipynb for the full SQL and optional external model path. Test results for Azure SQL for PostgreSQL Flexible Server The tests show that PostgreSQL vector search provides a useful candidate set, SQL RRF can substantially improve results when keyword evidence is strong, and the Cohere semantic reranker is the most consistent overall: it moved the correct passage to rank 1 in two tests and from rank 3 to rank 2 in the Tesla test. RRF produced the biggest gain for the construction question, moving the correct passage from outside the vector top five to rank 1, but did not improve every query. The scores across stages are not directly comparable because cosine similarity, RRF score, and Cohere relevance use different scales. Azure Cosmos DB for NoSQL: hybrid search with built-in RRF How it works: Azure Cosmos DB for NoSQL supports native hybrid ranking with VectorDistance, FullTextScore, and RRF inside ORDER BY RANK: SELECT TOP K C.context_id, c.title, c.text FROM c ORDER BY RANK RRF( VectorDistance(c.vector, @query_vector), FullTextScore(c.text, @term1, @term2, @term3) ) The notebook extracts distinct terms from the question before building the full-text part of the query. That token selection is application logic and can materially affect the hybrid ranking, so production evaluation should test analyzers, languages, term extraction, and optional RRF weights. Cosmos DB Semantic Reranker is an SDK-integrated preview feature. The application first runs a query, serializes the resulting documents, and submits those documents with the user's context string: result = container.semantic_rerank( context=question, documents=documents, options={ "return_documents": False, "top_k": min(k, len(documents)), "sort": True, "document_type": "json", "target_paths": "title,text", }, ) The service accepts at most 50 documents per rerank call and returns relevance scores from 0 to 1, plus inference latency and token usage. It uses the Microsoft semantic ranking model also used by Azure AI Search. The reranking call requires Microsoft Entra authentication, the appropriate Semantic Reranker role, and an account-linked inference endpoint. The 04_azure_cosmosdb_reranking.ipynb in the shared repo contains and end-to-end implementation. Test results for Azure Cosmos Db The results show that vector search provides a strong baseline, while hybrid RRF and semantic reranking improve different queries in different ways. Hybrid RRF helps when exact keywords matter, moving the construction answer into the top results, while the semantic reranker delivers the strongest overall ordering, promoting the correct construction passage from hybrid rank 3 to rank 1 and improving the Super Bowl answer from rank 5 to rank 2. However, it does not always place the gold passage first, as seen in the Tesla example, confirming that reranking improves relevance but is query-dependent and should be evaluated across a larger test set. What the examples do and do not show The four services expose different ownership boundaries: • Azure AI Search owns hybrid fusion and learned semantic ranking inside the search service. • Azure SQL owns vector retrieval and outbound REST invocation in this example, while Foundry owns model inference. • PostgreSQL supplies vector and full-text primitives; the application owns the RRF SQL and optional Cohere call. • Cosmos DB provides native hybrid RRF and integrates a separate preview inference call through its SDK. Across three selected questions, the known source passage often moved substantially. That supports the practical value of testing a second-stage ranker. It does not prove that semantic reranking always improves top-1 accuracy, that RRF is universally beneficial, or that scores from different stages can be compared directly. Cosine similarity, RRF score, Azure AI Search reranker score, Cohere relevance, and Cosmos DB semantic relevance all have different definitions and scales. Compare rank positions and task-level metrics, not raw values across systems. Turn the demonstration into an evaluation For a production RAG system, convert the notebook pattern into a repeatable evaluation harness: Build a representative labeled query set from real user tasks. Freeze corpus, chunking, embedding model, dimensions, and candidate counts for each run. Record ranks after retrieval, fusion, and learned reranking. Measure Recall@k or Hit@k to verify that retrieval finds relevant evidence. Measure Mean Reciprocal Rank (MRR) when the position of the first relevant result matters. Use NDCG when judgments include multiple passages or graded relevance. Record latency percentiles, inference usage, request cost, and failure rates. Evaluate the generated answer separately for correctness, citation support, and refusal behavior. Also test the operational cases that a three-question demonstration cannot cover: empty keyword results, missing gold passages, long documents, multilingual text, filters, partial outages, token expiration, throttling, model retirement, and low-confidence scores. Practical guidance Retrieve broadly enough that the correct evidence can reach the learned stage. Use RRF when vector and keyword retrieval provide complementary signals. Rerank a bounded candidate set, commonly 20 to 50 passages, and measure the latency cost. Keep citations and source identifiers through every rank transformation. Version the corpus, embedding model, dimensions, query set, and reranker deployment. Do not hard-code assumptions about model endpoints or lifecycle dates. Verify current service documentation and the deployed extension or SDK version. Add thresholds or fallback behavior only after calibrating scores on your own data. Judge the full RAG chain. Better passage order is valuable only when it improves grounded answers for users. Vector search is built to find plausible candidates quickly. Rank fusion can reconcile retrieval signals, and a learned reranker can decide which candidates best address the question. The right architecture depends on where your data lives, which service boundaries you want to operate, and what your evaluation says about quality, latency, and cost. Resources Companion repository Azure AI Search semantic ranker Azure SQL VECTOR_DISTANCE Azure SQL sp_invoke_external_rest_endpoint Azure Database for PostgreSQL AI functions Microsoft Foundry model retirement schedule Azure Cosmos DB hybrid search Azure Cosmos DB Semantic Reranker SQuAD dataset card Dataset attribution Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP 2016. SQuAD 1.1 is distributed under CC BY-SA 4.0.
