Beyond text: Returning images and interactive apps from MCP servers
The Model Context Protocol (MCP) is becoming a richer foundation for agent experiences. Though most servers return plain text from their tool calls, MCP servers can also return binary results and provide interactive apps in clients that support those features, like VS Code. In this post, I'll use both capabilities to build an MCP server that searches a collection of nature photos with natural language, lets the model inspect the matching images, and presents selected results in an interactive gallery. The same approach can be adapted to product catalogs, digital asset managers, photo archives, and other multimedia libraries. Searching the image library Let's start with the search experience from a user's perspective, then dive into the code behind it. After connecting VS Code to the deployed MCP server, I can ask a question in GitHub Copilot about the images: Find landscape photos that show dramatic terrain and water. Show me the strongest options for a nature gallery. The GitHub Copilot agent realizes that it can use the image search MCP tool to answer that question. Here's what it looks like in the chat interface: The tool results include rendered thumbnails. I can click a thumbnail to inspect it directly in VS Code, much like a file in the workspace, while the Copilot agent can review both the image binary data and their textual descriptions. Behind the scenes, the agent called the image_search tool with these arguments: { "query": "dramatic natural landscapes with mountains and water", "max_results": 5 } The tool call returned a mix of binary files and structured data: a thumbnail for each matching image, plus JSON containing its filename, display name, and generated description. The thumbnails let a multimodal model inspect the actual pixels, while the structured content gives the agent compact metadata it can reference in later tool calls. { "results": [ { "filename": "Picture1.jpg", "display_name": "Picture1.jpg", "description": "A clear mountain lake surrounded by pine forest and steep rocky peaks." }, ...] } Returning images from MCP tools Now let's look at the code powering that tool call. I built the server with FastMCP, a popular Python framework for writing MCP servers. I declare each tool by decorating a function with mcp.tool() and annotating its arguments with types and helpful descriptions. FastMCP converts the function signature into a JSON Schema that helps GitHub Copilot decide when and how to call image_search : @mcp.tool(annotations={"readOnlyHint": True}) async def image_search( query: Annotated[ str, "Text description of images to find (e.g., 'sunlit mountain lake')" ], max_results: Annotated[int, "Maximum number of images to return (1-20)"] = 5) -> ToolResult: """ Search for images matching a natural language query. Returns the image data and descriptions. """ Inside the function, I use Azure AI Search to perform hybrid retrieval, combining the text query with its vector embedding. The target index contains multimodal image embeddings and LLM-generated descriptions. Then I retrieve the image from Azure Blob Storage and resize it to a thumbnail. The tool returns both the binary image data for the thumbnails and structured metadata with image details. results = await search_client.search(search_text=query, top=max_results, vector_queries=[VectorizableTextQuery(k_nearest_neighbors=max_results, fields="embedding", text=query)], select=["metadata_storage_path", "verbalized_image"]) blob_service_client = get_blob_service_client() files: list[File] = [] image_results: list[dict[str, str]] = [] async for result in results: url = result["metadata_storage_path"] description = result.get("verbalized_image") container_name, blob_name = get_blob_reference_from_url(url) blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name) stream = await blob_client.download_blob() image_bytes = await stream.readall() image_format = get_image_format(url) display_name = os.path.basename(blob_name) file_basename = Path(display_name).stem thumbnail_bytes = resize_image_bytes(image_bytes, image_format) files.append(File(data=thumbnail_bytes, format=image_format, name=file_basename)) image_results.append({"filename": blob_name, "display_name": display_name, "description": description}) return ToolResult( content=files, structured_content={ "query": query, "results": image_results, }, ) Displaying selected images Finding the right images is only the first half of the experience. Once the agent has review the thumbnails and their generated descriptions, it needs a better way to present its favorite selected images to the user. That is where MCP apps come in. An MCP app renders an interactive webpage inside a sandboxed iframe in the MCP client. For this server, the app is a small, JavaScript-powered carousel for browsing the selected images. GitHub Copilot calls the display_image_files tool when it wants to render the carousel app: Returning apps from MCP tools Let's check out the code that powers that MCP carousel app. An app is associated with a tool, so I once again decorate a Python function with mcp.tool() . This time, I pass an AppConfig that points to the image viewer's HTML resource. @mcp.tool( app=AppConfig(resource_uri=IMAGE_VIEW_URI), annotations={"readOnlyHint": True}, ) async def display_image_files( filenames: Annotated[list[str], "List of image filenames to retrieve and display in a carousel."], descriptions: Annotated[list[str], "Image descriptions, in the same order as filenames."] ) -> ToolResult: """Fetch images by filename and render in carousel with filenames, descriptions, and file details.""" Inside the function, I fetch the selected images from Azure Blob Storage by filename, then return both the binary image data and structured content describing each image—its filename, generated description, MIME type, dimensions, format, and size. blob_service_client = get_blob_service_client() image_blocks: list[types.ImageContent] = [] image_results: list[dict[str, str | int]] = [] for image_index, filename in enumerate(filenames): blob_client = blob_service_client.get_blob_client(container=IMAGE_CONTAINER_NAME, blob=filename) stream = await blob_client.download_blob() image_bytes = await stream.readall() mime_type = get_image_mime_type(filename) with Image.open(io.BytesIO(image_bytes)) as image: width, height = image.size image_format = image.format image_blocks.append(types.ImageContent( type="image", data=base64.b64encode(image_bytes).decode("utf-8"), mimeType=mime_type)) image_results.append( { "filename": filename, "description": descriptions[image_index], "mimeType": mime_type, "width": width, "height": height, "format": image_format, "sizeBytes": len(image_bytes), } ) return ToolResult( content=image_blocks, structured_content={ "images": image_results, }, ) Next, I define the resource that serves the image viewer HTML page. I decorate a Python function with @mcp.resource , assign it a ui:// URL that is unique to the MCP server, and use its Content Security Policy (CSP) to declare which external domains the app may load resources from: @mcp.resource(IMAGE_VIEW_URI, app=AppConfig(csp=ResourceCSP(resource_domains=["https://unpkg.com"]))) def image_view() -> str: """Render images returned by display_image_files as an MCP App.""" return load_image_viewer_html() The final piece is the HTML that renders inside the app's iframe. This small page imports ext-apps, a JavaScript package that manages bidirectional communication with the MCP client. The JavaScript creates an App instance, defines the ontoolresult callback, and connects the app. That callback receives images from the tool result and renders them in the carousel. MCP apps can also send messages back to the host, although this read-only viewer does not need to. <!DOCTYPE html> <html lang="en"> <body> <div id="carousel"> <button id="prev" type="button" aria-label="Previous">‹</button> <div id="frame"></div> <button id="next" type="button" aria-label="Next">›</button> <span id="counter" aria-live="polite"></span> </div> <script type="module"> import { App } from "https://unpkg.com/@modelcontextprotocol/ext-apps@0.4.0/app-with-deps"; const app = new App({ name: "Image Viewer", version: "1.0.0", }); let images = []; let index = 0; const frame = document.getElementById("frame"); const prevBtn = document.getElementById("prev"); const nextBtn = document.getElementById("next"); const counter = document.getElementById("counter"); function show(i) { index = i; const img = images[index]; frame.innerHTML = ""; const el = document.createElement("img"); el.src = `data:${img.mimeType || "image/jpeg"};base64,${img.data}`; el.alt = "Blob image"; frame.appendChild(el); prevBtn.disabled = index === 0; nextBtn.disabled = index === images.length - 1; counter.textContent = images.length > 1 ? `${index + 1} / ${images.length}` : ""; } prevBtn.addEventListener("click", () => { if (index > 0) { show(index - 1); } }); nextBtn.addEventListener("click", () => { if (index < images.length - 1) { show(index + 1); } }); app.ontoolresult = ({ content }) => { images = (content || []).filter((block) => block.type === "image"); if (images.length > 0) { show(0); } }; await app.connect(); </script> </body> </html> Try it yourself! The full MCP server code is available in Azure-Samples/image-search-aisearch, along with a minimal image search website and an Azure AI Search indexing pipeline. The indexer uses an Azure OpenAI model to describe each image and Azure AI Vision to create multimodal embeddings. The repository includes a sample nature dataset, but you can replace it with any image collection. Here are more ways you could extend it it: Support more media types: add transcript search and a video or audio player app, while keeping the same search-then-display tool pattern. Enrich the metadata: index dates, locations, creators, accessibility text, or domain-specific tags alongside generated descriptions and embeddings. Optimize token consumption: images require many tokens, so returning too many thumbnails can quickly consume the model's context window. Experiment with smaller previews, higher compression, metadata-only search results, or a two-stage retrieval flow. Add authentication: many media libraries contain private or licensed assets. You can add key-based authentication or OAuth with the FastMCP auth providers, as I described in the MCP auth livestream. Once search results can carry both structured metadata and real media, an agent can do more than locate files: it can compare, curate, and present them in the same conversation. I hope you'll try the sample with a multimedia collection of your own!Building AI Agents from Zero to Production
Building AI Agents from Zero to Production Most agent demos stop at "it answered my question." Production doesn't. The gap between a notebook that calls an LLM and a governed, observable, multi-agent system your organisation can actually depend on is where real engineering happens, evaluation, deployment, data sovereignty, tool governance, and cross-team interoperability. Microsoft's open-source course Building AI Agents from Zero to Production walks that entire arc in seven lessons, using one realistic use case and the Microsoft Agent Framework (MAF) plus Microsoft Foundry. This post is a developer-focused tour of what it teaches, the architecture decisions behind each stage, and the code patterns that matter when you move from prototype to production. Who this is for AI engineers building their first or first production, agent system. Backend and full-stack developers integrating agents into real applications and CI/CD. Cloud architects who need data sovereignty, private networking, and governance around agent workloads. Technical leads deciding how to standardise tools and orchestration across multiple teams. The samples are Python 3.12+, served through Microsoft Foundry using GPT-5 series models (for example gpt-5.1 ). Lesson 4 adds a TypeScript/React frontend. You will want an Azure subscription and the Azure CLI. The AI Agent Development Lifecycle The course is organised around a lifecycle rather than a feature list. Each lesson is a stage, and each stage assumes the previous one is solved: # Stage The production question it answers 1 Agent Design What should each agent do, and how do they hand off? 2 Agent Development How do I build and run them with the Agent Framework? 3 Agent Evaluations How do I know they actually work — and keep working? 4 Agent Deployment How do I ship one as a hosted service with a UI and CI gate? 5 Production Hosted Agents How do I meet enterprise data, network, and governance needs? 6 Microsoft Toolbox How do I govern tools once, and reuse them across teams? 7 Multi-Agent & A2A How do agents from different teams interoperate safely? The thread running through all seven is a single scenario: a Developer Onboarding agent system that helps a new hire find the right teammates, get a sensible first task, and pull learning resources and code snippets. It is deliberately mundane, which is exactly why it exposes the production concerns that flashy demos hide. Lesson 1 — Agent Design: three components, one graph The course defines an agent by three parts: an LLM for reasoning, tools to act, and memory to retain context. The design work is context engineering — making sure the right information reaches the model at the right moment, no more and no less. Rather than one monolithic assistant, the onboarding system is split into specialists coordinated by a triage agent using handoff orchestration: Agent Job Tool Employee Search Answer org and people questions Foundry file search over an employee-directory vector store Task Recommendation Suggest 1–3 GitHub issues for the new dev GitHub MCP Server (reads recent commits + open issues) Code Assistant Provide resources and runnable snippets Microsoft Learn MCP + Code Interpreter Architecturally this is a directed graph: User → Triage → [Employee, Learning, Coding] . Splitting responsibilities early pays off later, each agent gets a tightly scoped prompt (less hallucination), can be evaluated independently, and can be upgraded without touching its peers. Lesson 2 — Development: standalone agents with MAF Here the design becomes code. Each specialist is a small, independently runnable service built with the Microsoft Agent Framework, authenticated to Foundry with your Azure CLI login. Setup is deliberately boring: az login az account set --subscription "<your-subscription-id>" cp .env.example .env # Fill FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL (e.g. gpt-5.1) # Create the employee-directory vector store once; note the printed VECTOR_STORE_ID python lesson-2-agent-development/setup_vector_store.py # Start an agent — serves on http://localhost:8090 python lesson-2-agent-development/employee-search-agent.py The FoundryChatClient auto-reads any FOUNDRY_ -prefixed environment variables and uses AzureCliCredential , so there are no keys in code. The lesson ships six samples, each on its own port, so you can chat with them individually in the local DevUI before wiring them together: Sample Tool Port employee-search-agent.py Foundry file search / vector store 8090 task-recommendation-agent.py GitHub MCP Server 8095 azure-learning-agent.py Microsoft Learn MCP 8092 coding-agent.py Code Interpreter 8093 learning-recommendation-agent.py Learn MCP + reasoning 8091 agent-orchestration.py Multi-agent handoff 8094 Why this matters: keeping each agent as its own process with its own port is a testability decision, not an accident. You can smoke-test one specialist in isolation, then compose them in agent-orchestration.py . Lesson 3 — Evaluation: you can't unit-test a probability distribution This is the lesson that separates a demo from a product. Agents are non-deterministic, so traditional assertions don't fit. The course uses three complementary layers: Observability / tracing — always on, via OpenTelemetry to Application Insights. Smoke tests — fast, run on every deploy. Evaluations — deeper, model-based scoring run on-demand or nightly. Turning on tracing is a single call: from agent_framework.foundry import FoundryChatClient client = FoundryChatClient() client.configure_azure_monitor() # export traces + metrics to Application Insights For quality it uses Foundry's built-in "LLM-as-a-judge" evaluators against real persisted responses (identified by response_id ), not freshly regenerated ones: Evaluator evaluator_name Measures Relevance builtin.relevance Does the response address the request? Groundedness builtin.groundedness Is it supported by retrieved data (no hallucination)? Tool-call accuracy builtin.tool_call_accuracy Were the right tools called with the right arguments? Tool-output utilization builtin.tool_output_utilization Did the agent actually use tool results? The judge model is set independently via AZURE_AI_MODEL_DEPLOYMENT_NAME , so you can evaluate a cheap production model with a stronger one. The run prints a report_url that deep-links into the Foundry portal. Lesson 4 — Deployment: a hosted agent, a UI, and a CI gate Now the agent becomes a managed service. It is deployed as a Foundry Hosted Agent a Microsoft-managed execution environment and fronted by an OpenAI ChatKit React UI talking to a FastAPI backend: ChatKit React (3000) → FastAPI backend (8001) → Foundry Hosted Agent → tools Building the agent is declarative attach tools, name it, serve it: agent = client.as_agent( name="DevOnboardingAgent", instructions="...", tools=[file_search_tool, learn_mcp_tool], ) # served with: from_agent_framework(agent).run() The recommended deploy path is the Azure Developer CLI: cd hosted-agent azd auth login azd agent deploy The genuinely production-minded part is the smoke test as a post-deploy CI gate. Six cases cover reachability, each scenario, off-topic prompt adherence, and multi-turn threading (verifying state via previous_response_id ). The GitHub Action runs them against the freshly deployed agent: export FOUNDRY_TOKEN=$(az account get-access-token \ --resource https://ai.azure.com/ --query accessToken -o tsv) python runner.py \ --project-endpoint "https://<account>.services.ai.azure.com/api/projects/<project>" \ --agent-name dev-onboarding \ --tests-file tests/smoke-tests.json Pitfall to remember: the token audience must be https://ai.azure.com/ . A cognitiveservices.azure.com token is rejected by the Responses API — a mistake that costs many engineers an afternoon. Lesson 5 — Production: separating where an agent runs from where its data lives The pivotal concept for enterprise readiness is the distinction between a Hosted Agent (compute, scaling, identity) and a Capability Host (where conversation history, files, and embeddings actually reside): Concern Hosted Agent Capability Host Compute / scaling / identity ✅ Provided — Conversation history Microsoft-managed default Redirect to your Azure Cosmos DB File uploads Microsoft-managed default Redirect to your Azure Storage Vector embeddings Microsoft-managed default Redirect to your Azure AI Search Required to run the agent? ✅ Yes ❌ Optional Required for data sovereignty? ❌ Not sufficient ✅ Yes "Basic" setup uses Microsoft-managed storage and is perfect for getting started. "Standard" setup redirects each data plane to your own Azure resources through a project-level capability host, this is how you keep customer data in your tenant, inside your network boundary: PUT .../accounts/{account}/projects/{project}/capabilityHosts/{name}?api-version=2025-06-01 { "properties": { "capabilityHostKind": "Agents", "threadStorageConnections": ["my-cosmosdb-connection"], "vectorStoreConnections": ["my-ai-search-connection"], "storageConnections": ["my-storage-connection"] } } Operational constraints worth internalising before you provision: there is one capability host per scope (a second attempt returns 409 Conflict ), configuration is immutable (delete and recreate to change it), deletion is destructive, and the account-level host must exist before the project-level one. Lesson 6 — Toolbox: govern tools once, reuse everywhere Left unchecked, every team re-implements the same tools, scatters credentials, and loses governance visibility. The Microsoft Foundry Toolbox solves this by exposing a curated, versioned set of tools behind a single MCP-compatible endpoint, with credentials held in Foundry connections rather than agent code. You build a toolbox version once: from azure.ai.projects.models import MCPTool, ToolboxSearchPreviewTool, WebSearchTool toolbox_version = project.toolboxes.create_toolbox_version( name="agent-tools", description="Web search + an MCP server + tool search", tools=[ WebSearchTool(), MCPTool( server_label="myserver", server_url="https://your-mcp-server.example.com", require_approval="never", project_connection_id="my-key-auth-connection", # credentials live in Foundry ), ToolboxSearchPreviewTool(), ], ) And every agent consumes it through one endpoint, no per-team tool code: from agent_framework import MCPStreamableHTTPTool mcp_tool = MCPStreamableHTTPTool( name="toolbox", url=TOOLBOX_ENDPOINT, # {project_endpoint}/toolboxes/{name}/mcp?api-version=v1 http_client=http_client, load_prompts=False, ) agent = chat_client.as_agent(name="my-toolbox-agent", instructions="...", tools=[mcp_tool]) Versioning is blue/green: create a new version, test it on its version-specific endpoint, then promote it to default and every consumer picks it up with zero code changes. A Guardrail (RAI) policy can be applied at the toolbox layer, independent of model-level content filters. Note the toolbox management APIs are currently preview; the portal or VS Code Foundry Toolkit are practical alternatives for creation today. Lesson 7 — Multi-Agent & A2A: agents as networked peers The final lesson contrasts two ways agents collaborate: Handoff / Workflow — in-process, same codebase, fastest, tightest coupling. Agent-to-Agent (A2A) — cross-process over an open protocol, so agents from different teams, orgs, or frameworks interoperate. A2A gives each agent a discoverable Agent Card at /.well-known/agent-card.json and a task lifecycle (submitted → working → completed/failed). The elegant part: A2AExecutor wraps an existing MAF agent with no changes to that agent's code. from agent_framework.a2a import A2AExecutor from a2a.server.apps import A2AStarletteApplication from a2a.server.tasks import InMemoryTaskStore agent_card = AgentCard( name="Coding Assistant", url="http://localhost:9000/", version="1.0.0", capabilities=AgentCapabilities(streaming=True), skills=[AgentSkill(id="generate-code", name="Generate code", tags=["code"])], ) request_handler = DefaultRequestHandler( agent_executor=A2AExecutor(agent), # wraps your existing MAF agent unchanged task_store=InMemoryTaskStore(), ) app = A2AStarletteApplication(agent_card=agent_card, http_handler=request_handler).build() Consuming a remote agent then looks exactly like calling a local one: from agent_framework.a2a import A2AAgent remote_agent = A2AAgent(name="remote-coding-assistant", url="http://localhost:9000") result = await remote_agent.run("Write a Python function that reverses a string.") Because an A2AAgent can be a participant inside a HandoffBuilder workflow, you can mix in-process routing with remote services in the same orchestration. For enterprise use, A2AAgent accepts an auth_interceptor for bearer tokens, and the Agent Card carries security_schemes . Responsible and secure by design Production readiness in this course is not just uptime, it is governance: Identity over keys — AzureCliCredential and managed identity throughout; no secrets in code. Least privilege — CI runners get a scoped Azure AI User role assignment on the specific project. Data sovereignty — capability hosts keep conversation history, files, and embeddings in your own Cosmos DB, Storage, and AI Search. Tool approval and guardrails — MCP approval_mode and toolbox-level RAI policy gate what agents can do. Grounded evaluation — groundedness and tool-utilization scoring catch hallucination and unused-tool behaviour before users do. Cost hygiene — the lessons create real Azure resources; delete the resource group when done: az group delete --name <rg> --yes --no-wait . Key takeaways Design as a graph of specialists. Handoff orchestration with tightly scoped agents beats one monolith on reliability and testability. One .run() contract, many backends. The Agent Framework keeps orchestration code stable from local dev to hosted production. Evaluate continuously. Tracing + smoke tests + model-based evaluators are three layers, not alternatives. Separate compute from data. Hosted Agents run the agent; Capability Hosts give you sovereignty — you need both for enterprise. Govern tools centrally. A versioned toolbox behind one MCP endpoint kills tool sprawl and credential duplication. Open protocols for interop. A2A lets agents cross team, org, and framework boundaries without rewrites. Get started Clone the repo (skip the 50+ translations for a faster download) and work through the lessons in order: git clone --filter=blob:none --sparse https://github.com/microsoft/Building-AI-Agents-From-Zero-To-Production.git cd Building-AI-Agents-From-Zero-To-Production git sparse-checkout set --no-cone '/*' '!translations' '!translated_images' References Building AI Agents from Zero to Production — course repo Microsoft Agent Framework Microsoft Foundry documentation Agent-to-Agent (A2A) protocol specification a2a-python SDK AI Agents for Beginners MCP for Beginners Microsoft Foundry Discord
My Journey with Azure SRE Agent
Introduction A customer came to me with a problem that many organisations have. They control their infrastructure through Infrastructure as Code, but there are often scenarios where an admin needs to go in and make a change - even though they would ideally not want this to happen. The use an Entra feature Privileged Identity Management (PIM). Users statically don't have contributor access to Azure resources, but PIM allows them to elevate their access for a period of time. As part of PIM, the admin needs to give a reason for the elevation. Wouldn't it be good if an agent of some sort could look at this reason, then look at what the user actually did and make an assessment on whether what they did aligned with the reason given? Then alert if not. I initially built Python agents to handle this, but as with many "build vs. buy" decisions, I eventually discovered that Azure SRE Agent (in preview at the time of writing) could do what I needed – and more. This blog chronicles my journey from initial scepticism to building a fully autonomous PIM elevation audit agent. Along the way, I learned valuable lessons about what SRE Agent is designed for, how to work with its tooling model, and the difference between interactive exploration and production automation. The Starting Point: Python Agents and the Buy vs. Build Decision Before discovering SRE Agent, I had functional Python scripts that queried Azure Audit Logs and Activity Logs to correlate PIM activations with actual Azure operations. They worked, but they required maintenance, error handling, scheduling infrastructure, and ongoing attention. When I heard about Azure SRE Agent's capabilities as an autonomous monitoring platform, I decided to investigate. The decision: If there's a choice between buy versus build, buy should win – especially when the "buy" option is a managed Azure service with built-in security, monitoring, and integration capabilities. First Impressions: The Interactive Front End One of the first features that caught my attention was SRE Agent's chat interface. Unlike my static Python scripts, I could have conversational interactions with the agent, refining queries and exploring my Azure environment in natural language. This was powerful for discovery and prototyping. Initial Success (and Failure) When I first asked SRE Agent to analyse PIM elevation patterns, the results were... disappointing. The agent couldn't initially answer my PIM elevation questions effectively. However, this is where the interactive experience shone: through. With coaching in an interactive session, I could: - Explain what PIM activation events look like in Azure Audit Logs - Show the agent how to correlate `CorrelationId` between activation requests and justifications - Demonstrate how to build time windows from activation start to deactivation/expiration - Guide it through matching Azure Activity operations against justification keywords After several rounds of refinement, the agent eventually got excellent results. The interactive session wasn't just a chatbot – it was a learning tool that helped me shape the agent's behaviour. The Subagent Puzzle: Interactive vs. Headless What I really needed was an autonomous agent that could run on a schedule. As I got better results from the interactive sessions, Subagents is the tool in SRE Agent for this. I naturally wanted to convert the interactive session into a subagent that could run autonomously. This is where I hit my first conceptual stumbling block. The Aha Moment: Understanding SRE Agent's Purpose I was initially confused about how to structure a subagent. Should it replicate the interactive conversation flow? How do I capture all that back-and-forth in a static configuration? After discussions with the engineering, I learned a critical lesson: The interactive experience is fantastic for exploration, prototyping, and troubleshooting – but it's not what you should be aiming for in production automation. This reframed my entire approach. Instead of trying to replicate the conversational flow, I needed to distil my learnings from those sessions into the instructions for a subagent. Struggling with Subagent Format Even with this clarity, I struggled with the format of a subagent definition. The YAML structure, the `system_prompt` verbosity, the tool declarations – it felt overwhelming to translate my interactive sessions into a configuration file. The Game-Changer: Let the Agent Write Itself Then came the game-changing advice from engineering: This was brilliant in its simplicity. I had already what I wanted the agent to do in the interactive chat session. It was a simple as "generate a subagent from this conversation". I must admit, I did have to ask it to generate an email with the report, but the bulk of the effort in generating the YAML subagent file was done by the agent. What would have taken me hours of trial and error was done in minutes. Tool Configuration: The Missing Pieces With a subagent definition in hand, I deployed it and... nothing worked. This began the most educational part of my journey: understanding how tools work in Azure SRE Agent. Challenge #1: Accessing Log Analytics My subagent kept failing to query Log Analytics. I initially thought this was a role assignment issue – did the agent's managed identity have Log Analytics Reader permissions? I spent time checking RBAC, verifying workspace access, and reviewing Entra ID permissions. The real issue? I needed to add `QueryLogAnalyticsByWorkspaceId` as a tool in my subagent configuration! tools: - QueryLogAnalyticsByWorkspaceId The Azure SRE Agent UI supports selecting this tool during configuration, but I had missed it. More importantly, I needed to mention the Log Analytics workspace ID in my subagent's `system_prompt` so the agent knew which workspace to target: system_prompt: > ... Query the workspace: XXXXXX-d119-4550-86c0-YYYYYYYYYYY... Lesson learned: Tools aren't automatically available – you must explicitly declare them. The agent uses this to understand what capabilities it has and to configure the appropriate authentication and access patterns. Challenge #2: Sending Email Notifications The next hurdle was sending email reports. My PIM audit was working beautifully, but the results were only visible in logs. I needed email notifications. Initially, there didn't seem to be a built-in email tool I could choose from the portal. I attempted to write a custom Python tool that sent emails via Microsoft Graph API. This seemed logical – I'd done this in my previous Python agents. Problem: Corporate email policies blocked my application from sending emails via Graph. This was a security feature, not a bug, but it meant my custom tool approach was dead in the water. Discovering the Outlook Connector Then I noticed the Outlook connector in the SRE Agent configuration portal. This was a managed connector specifically for sending emails with pre-configured authentication. I set it up, configured it (noting the connector ID: `connector-abf2`), and waited for emails. Still nothing. The Manual YAML Edit Trawling through other sample subagent configurations, I discovered a tool called SendOutlookEmail. This tool wasn't available in the portal's dropdown menu, but it existed in the platform. I needed to **manually add this to my subagent YAML file**: tools: - QueryLogAnalyticsByWorkspaceId - SendOutlookEmail After this change and redeploying the subagent, emails started flowing perfectly. Lesson learned: The portal UI is evolving (remember, this is preview), and not all tools are exposed visually yet. Don't be afraid to hand-edit the YAML when you know a capability exists. The documentation and sample repositories are your friends. Making It Fully Autonomous: Scheduled Triggers With a working subagent that could query logs, analyse alignment, and send emails, I had one final step: scheduling it. I created a scheduled task trigger in Azure SRE Agent configured to run every 24 hours (UTC). This trigger invokes my PIM elevation subagent, which executes its entire workflow autonomously and emails stakeholders with any findings. The subagent configuration includes this execution schedule guidance: system_prompt: > Execution schedule: Run every 24h (UTC). Now, every morning, our security team receives a PIM elevation alignment report without any manual intervention. The Result: A Production PIM Elevation Agent My final solution is an **autonomous agent** that: Runs on a 24-hour schedule Queries Azure Audit Logs for PIM activations Extracts user justifications from the log Builds precise activation time windows Queries Azure Activity logs during that time window Classifies alignment: Aligned, Partial, or NotAligned Generates JSON and plaintext reports Emails stakeholders with flagged non-aligned activity No Python scripts. No custom authentication handling. No infrastructure to maintain. You can see the full subagent configuration in my GitHub repository: PIM Elevation Agent Reflections: SRE Agent's Power and Rough Edges Azure SRE Agent is powerful. The ability to define complex audit workflows in declarative YAML, leverage natural language prompts for behaviour specification, and integrate with Azure services through managed tools is genuinely impressive. It also integrates with incident response services - both being able to generate incidents and to trigger flows from incidents. All as a first-class Azure Platform as a Service (PaaS). However, it's important to remember that this is a preview service (as of February 2026). There are rough edges: - Tool discoverability: Not all tools are visible in the portal UI - Documentation gaps: Some capabilities require digging through samples - Learning curve: Understanding the interactive-vs-headless paradigm takes time - Debugging: Error messages aren't always clear about what's misconfigured These are typical preview-stage challenges, and I expect they'll improve as the service matures. The core platform is solid, and the engineering team is responsive to feedback. Key Takeaways If you're considering Azure SRE Agent, here are my lessons learned: Use interactive sessions for discovery – They're excellent for prototyping and learning Think headless/autonomous for production – Autonomous agents should be declarative, not conversational Let the agent write itself – Ask the interactive session to generate subagent configs Explicitly declare tools – They're not automatic; you must add them to your config Include context in prompts – Workspace IDs, connector IDs, schedules – be specific Don't fear manual YAML edits – The portal is evolving, hand-editing is ok Check samples and docs*– Other configurations show patterns and tools not yet in UI, so check the YAML of these Embrace "buy over build" – Managed services reduce long-term maintenance burden Resources: - SRE Agent Documentation - my PIM Elevation subagent sample - Kusto (KQL) Query Reference *This blog post represents my personal experience and opinions. Azure SRE Agent capabilities and UI may have changed since the time of writing.*
MCP Server Authorization with Azure API Management: From Simple to Advanced
Why put API Management in front of your MCP servers The Model Context Protocol (MCP) has quickly become the standard way for AI agents, such as GitHub Copilot in VS Code, to reach external tools and data. As soon as an MCP server does anything meaningful, the same questions that govern any API resurface: who is allowed to call it, what are they allowed to do, and how do you enforce that consistently across many servers without rewriting each one. Azure API Management (APIM) answers those questions for MCP. It sits between the MCP client and the tool backend and applies the controls you already trust for REST APIs: identity validation, OAuth, rate limiting, IP filtering, and observability. Crucially, APIM speaks the MCP authorization specification, which is built on OAuth 2.1 and Protected Resource Metadata (PRM, RFC 9728). That means APIM can do more than block bad requests. It can actively drive an interactive sign-in from the IDE, so the user logs in with their own identity and the agent acts on their behalf. This article walks through a progression of authorization scenarios, each one building on the last: The simple case: validate a token and block everything else. Triggering an interactive sign-in from VS Code for an MCP server that APIM hosts from your own APIs. Going beyond "is this a tenant user" to "does this user have the right attribute" with Entra app roles. Fronting an existing external MCP server and letting it drive its own OAuth flow (GitHub as the example). Governing which tools of an existing MCP server an agent is actually allowed to invoke. APIM MCP capabilities and the basic authorization options API Management exposes MCP servers in two distinct ways, and the authorization story differs slightly for each. Expose a REST API as an MCP server. APIM takes an API it already manages and projects selected operations as MCP tools. You own the operations, so you choose exactly which ones become tools at configuration time. This is the right mode when the capability you want to expose is an API you control. Expose an existing MCP server (passthrough). APIM fronts a remote MCP-compatible server (LangChain, an Azure Function, GitHub's remote MCP server, your own container) and relays the MCP protocol to it. APIM governs access, but the upstream server still owns its tool catalog. On top of either mode, you have a spectrum of authorization options: Subscription keys for simple, machine-to-machine access where a shared secret in a header is acceptable. Token validation with Microsoft Entra ID, where APIM acts as the protected resource and verifies a bearer token on every call. Interactive OAuth 2.1 sign-in, where APIM advertises Protected Resource Metadata so an MCP client can discover the authorization server, log the user in, and retry with a user token. Authorization passthrough, where an external MCP server presents its own authorization challenge and APIM relays it faithfully so the client authenticates directly against the upstream's identity provider. The rest of the article works through these options in increasing order of capability. The example setup The walkthroughs in the first three scenarios all use the same backend so you can reproduce them without standing up anything of your own: the publicly available Star Wars API at Star Wars API. It is a simple, read-friendly REST API (characters, films, planets, starships, and so on) imported into API Management as a normal API and then projected as an MCP server. The reason this single API is enough to illustrate the whole progression is that, in API Management, one underlying API can back several independent MCP servers, each exposing a different slice of its operations. For example, you can create: A read-only MCP server that exposes only the GET operations, for agents that should be able to query data but never change it. A write-capable MCP server that exposes the POST, PUT, or DELETE operations, for trusted automation that is allowed to mutate state. Same backend API, two MCP servers, two different tool surfaces. Each of these servers is an independent resource in APIM, so each one can carry its own authorization. Both can require an authenticated user (Scenarios 1 and 2), and you can go further by protecting only the sensitive one: gate the write-capable server behind an Entra app role so that, even among authenticated users, only those who carry a specific claim can reach the mutating tools. That app-role mechanism is the subject of Scenario 3, and it composes naturally with the multi-server split described here. Registering the MCP API in Microsoft Entra ID Before any of the policies below can validate a token, you need an application registration in Microsoft Entra ID that represents the MCP API. This registration is what defines the audience and scope that tokens are issued for, and it is the source of the mcp-audience, mcp-scope, and (indirectly) mcp-client-id values that the policies reference. Create it once and reuse it across all the MCP servers in this article. In the Azure portal, open Microsoft Entra ID, then App registrations, then New registration. Name it (for example, star-wars-mcp-api), choose single-tenant, and register. Record the Application (client) ID and the Directory (tenant) ID. Open Expose an API and add an Application ID URI. Accept the default api://<app-id>. This URI is your token audience. Still under Expose an API, add a delegated scope named MCP.Access, set its consent display name and description, set the state to Enabled, and save. Authorize the client that will request the scope. Under Expose an API, select Add a client application and enter the client ID of the MCP client. For VS Code, this is the built-in Microsoft authentication client aebc6443-996d-45c2-90f0-388ff96faa56. Check the MCP.Access scope and save. These steps produce the four constants the validation policy needs: Named value Comes from Example entra-tenant-id The Directory (tenant) ID from step 1 11111111-1111-1111-1111-111111111111 mcp-audience The Application ID URI from step 2 api://22222222-2222-2222-2222-222222222222 mcp-scope The scope name from step 3 MCP.Access mcp-client-id The client ID of the calling app from step 4 aebc6443-996d-45c2-90f0-388ff96faa56 [!NOTE] mcp-client-id is the identity of the application calling the MCP server, not the MCP API itself. For VS Code it is the built-in Microsoft authentication client, and its value lands in the token's appid claim, which is why the validation policy lists it under client-application-ids. If your tenant blocks the first-party VS Code client, register your own public client application and use its client ID instead. [!TIP] For the privileged-access feature in Scenario 3, you will also declare an app role on this same registration. You do not need it yet, but it is convenient to know that all identity configuration for these servers lives on this one app registration. With that backend and structure in mind, the scenarios below build up the authorization model one capability at a time. Scenario 1: The simple case, validate the token and block unauthorized access The most basic protection is to require a valid Entra ID token on every MCP request and reject anything that fails validation. No interactive flow, no roles, just a gate. APIM does this with the validate-azure-ad-token policy. The policy checks the issuing tenant, the audience (your MCP API), the calling client application, and the required scope. Anything that does not satisfy all four is rejected with a 401. <policies> <inbound> <base /> <validate-azure-ad-token tenant-id="{{entra-tenant-id}}" header-name="Authorization" failed-validation-httpcode="401" failed-validation-error-message="Unauthorized. Access token is missing or invalid."> <client-application-ids> <application-id>{{mcp-client-id}}</application-id> </client-application-ids> <audiences> <audience>{{mcp-audience}}</audience> </audiences> <required-claims> <claim name="scp" match="any"> <value>{{mcp-scope}}</value> </claim> </required-claims> </validate-azure-ad-token> </inbound> <backend> <base /> </backend> <outbound> <base /> </outbound> <on-error> <base /> </on-error> </policies> The values in double braces are APIM named values: centralized constants, defined once and shared by every MCP server. They map directly to the four values produced by the Entra app registration in the example setup (entra-tenant-id, mcp-audience, mcp-scope, and mcp-client-id). Storing them as named values keeps the policy free of hardcoded identifiers and lets every server reuse the same configuration. This gets you a server that nobody can call without a properly minted token. What it does not do is help a fresh client obtain that token in the first place. That is the next scenario. Scenario 2: Driving an interactive sign-in from VS Code for an APIM-hosted MCP server When you expose one of your own APIs as an MCP server, you usually want a developer to open VS Code, connect to the server, and be prompted to sign in with their Microsoft account. No pre-shared key, no manual token handling. APIM achieves this by behaving as a well-mannered OAuth 2.1 protected resource. Using the Star Wars MCP server from the example setup, each selected operation becomes a tool the agent can call, so an agent can answer "which films featured the character named Leia" by calling the underlying API through APIM. How the sign-in flow works The protocol choreography is what turns a plain 401 into an interactive login: Two ingredients make this work: a 401 challenge that points to a metadata document, and the metadata document itself. The challenge: a 401 that points the client to its metadata Instead of a bare 401, APIM returns a WWW-Authenticate header carrying the URL of the server's Protected Resource Metadata. This is what tells the client "you need a token, and here is where to learn how to get one." Keeping this logic in a shared policy fragment means every MCP server reuses it. Notice the mcpResourceMetadataUrl reference in the fragment below. It is not hardcoded; it is a context variable that each MCP server sets in its own server-level policy before including this fragment (you will see that wiring in the per-server policy later in this scenario). The fragment simply reads whatever value the calling server provided. This indirection is what keeps the fragment pluggable: the same shared challenge-and-validate logic serves every MCP server, while each server supplies its own PRM URL. In most deployments the PRM endpoint is a single, dynamic one (built in the next section) that derives the resource from the request path, so the variable just carries that server's path. But because the URL is configurable per server rather than baked into the fragment, you retain flexibility for the cases that need it. <fragment> <!-- No token: challenge with the per-server PRM URL set by the caller --> <choose> <when condition="@(!context.Request.Headers.ContainsKey("Authorization"))"> <return-response> <set-status code="401" reason="Unauthorized" /> <set-header name="WWW-Authenticate" exists-action="override"> <value>@("Bearer resource_metadata=\"" + (string)context.Variables.GetValueOrDefault("mcpResourceMetadataUrl", "") + "\"")</value> </set-header> </return-response> </when> </choose> <!-- Token present: validate against shared named values --> <validate-azure-ad-token tenant-id="{{entra-tenant-id}}" header-name="Authorization" failed-validation-httpcode="401" failed-validation-error-message="Unauthorized. Access token is missing or invalid."> <client-application-ids> <application-id>{{mcp-client-id}}</application-id> </client-application-ids> <audiences> <audience>{{mcp-audience}}</audience> </audiences> <required-claims> <claim name="scp" match="any"> <value>{{mcp-scope}}</value> </claim> </required-claims> </validate-azure-ad-token> </fragment> Creating the /.well-known PRM endpoint in APIM with a policy This is the part that often surprises people: APIM itself serves the metadata document. There is no separate identity service to stand up. You publish one small anonymous API at the service root that answers GET /.well-known/oauth-protected-resource/*, derives the resource value from the requested path, and returns a JSON document pointing at Microsoft Entra ID as the authorization server. Create a blank HTTP API named well-known with an empty API URL suffix so it resolves at the service root, add a GET operation with the template /.well-known/oauth-protected-resource/*, clear the subscription requirement so it is reachable anonymously, and apply this policy: <policies> <inbound> <base /> <!-- Build the resource URL from the requested PRM sub-path --> <set-variable name="resourceUrl" value="@{ var prefix = "/.well-known/oauth-protected-resource"; var path = context.Request.OriginalUrl.Path; var resourcePath = path.Length > prefix.Length ? path.Substring(prefix.Length) : ""; return "https://" + context.Request.OriginalUrl.Host + resourcePath; }" /> <return-response> <set-status code="200" reason="OK" /> <set-header name="Content-Type" exists-action="override"> <value>application/json</value> </set-header> <set-body>@{ return new JObject( new JProperty("resource", (string)context.Variables["resourceUrl"]), new JProperty("authorization_servers", new JArray( "https://login.microsoftonline.com/{{entra-tenant-id}}/v2.0")), new JProperty("scopes_supported", new JArray("{{mcp-prm-scope}}")), new JProperty("bearer_methods_supported", new JArray("header")) ).ToString(); }</set-body> </return-response> </inbound> <backend> <base /> </backend> <outbound> <base /> </outbound> <on-error> <base /> </on-error> </policies> The {{mcp-prm-scope}} named value populates the scopes_supported array of the metadata document. It tells the client which delegated scope to request when it goes to the authorization server, so it must be the fully qualified scope value: the token audience (the Application ID URI from the app registration) followed by the scope name. With the example values that is api://22222222-2222-2222-2222-222222222222/MCP.Access. In other words, it is the combination of the mcp-audience and mcp-scope values defined in the example setup. Named value Value to set Example mcp-prm-scope <mcp-audience>/<mcp-scope> api://22222222-2222-2222-2222-222222222222/MCP.Access [!NOTE] Keep mcp-prm-scope in sync with the scope the validation fragment requires. The PRM document advertises this scope so the client requests it, and validate-azure-ad-token then checks for it in the scp claim. A mismatch means the client obtains a token without the scope APIM expects, and validation fails. Because the policy builds the resource value from the request path, this single endpoint serves metadata for every MCP server you ever add. The Star Wars server, a future inventory server, and anything else all share it. Wiring it onto the MCP server Each MCP server only needs to declare its own metadata URL and include the shared fragment: <policies> <inbound> <base /> <set-variable name="mcpResourceMetadataUrl" value="https://apim-contoso-mcp.azure-api.net/.well-known/oauth-protected-resource/star-wars-mcp/mcp" /> <include-fragment fragment-id="mcp-entra-auth" /> </inbound> <backend> <base /> </backend> <outbound> <base /> </outbound> <on-error> <base /> <include-fragment fragment-id="mcp-auth-challenge-onerror" /> </on-error> </policies> On the VS Code side, the configuration is deliberately plain. With no subscription-key header present, the client falls straight into the OAuth flow: { "servers": { "star-wars-mcp": { "url": "https://apim-contoso-mcp.azure-api.net/star-wars-mcp/mcp", "type": "http" } } } Restart the server in VS Code, and it detects the 401, reads the metadata, opens a browser sign-in, requests consent on first use, and then loads the tools using the user's token. [!CAUTION] Do not read the response body with context.Response.Body inside MCP server policies. It forces response buffering and breaks the MCP streaming transport. If global diagnostic logging is enabled, set the Frontend Response payload bytes to log to 0 at the All APIs scope. Scenario 3: Beyond tenant membership, authorize on a user attribute with app roles Validating a token confirms the caller is a signed-in user in your tenant with the right scope. That is often not enough. Some MCP servers expose sensitive tools that only a subset of users should reach. You want to express "this user is not only part of the tenant, but has a specific attribute that permits this server." Microsoft Entra app roles are the optimal mechanism for this. You declare a role on the MCP API app registration, assign it to specific users or to a security group, and Entra ID emits a roles claim in the access token whenever your API is the audience. APIM then authorizes on that claim. App roles beat the groups claim here because they avoid the group overage problem, they are scoped to the application, and they travel with the app. Declaring and assigning the role On the MCP API app registration, under App roles, create a role: Setting Value Display name Privileged Access Allowed member types Users/Groups Value Privileged.Access Description Access to privileged MCP servers Then, on the matching enterprise application, under Users and groups, assign the users (or, better, a security group) to the Privileged Access role. The Value field is the exact string that lands in the token roles claim, so it cannot contain spaces. [!TIP] Keep User assignment required set to No on the enterprise application. Unassigned users still obtain a valid token with the MCP.Access scope and keep access to the non-privileged servers. They simply do not carry the roles claim, so the privileged servers reject them. Enforcing the claim in the per-server policy The shared mcp-entra-auth fragment is used by every server, so the role requirement must not live there. Place the check in the privileged server's own policy, right after the fragment include. The token is already validated at that point, so this step is pure authorization. Because the caller is authenticated but not authorized, return 403, not 401, and do not emit a challenge: re-authenticating will not grant a role the user does not have. <policies> <inbound> <base /> <set-variable name="mcpResourceMetadataUrl" value="https://apim-contoso-mcp.azure-api.net/.well-known/oauth-protected-resource/star-wars-mcp/mcp" /> <include-fragment fragment-id="mcp-entra-auth" /> <!-- Privileged guardrail: require the Privileged.Access app role --> <choose> <when condition="@(!context.Request.Headers.GetValueOrDefault("Authorization","").Replace("Bearer ","").AsJwt().Claims.GetValueOrDefault("roles", new string[0]).Contains("Privileged.Access"))"> <return-response> <set-status code="403" reason="Forbidden" /> <set-header name="Content-Type" exists-action="override"> <value>application/json</value> </set-header> <set-body>{"error":"forbidden","message":"You lack the Privileged.Access role required for this MCP server."}</set-body> </return-response> </when> </choose> </inbound> <backend> <base /> </backend> <outbound> <base /> </outbound> <on-error> <base /> <include-fragment fragment-id="mcp-auth-challenge-onerror" /> </on-error> </policies> One operational detail worth calling out: app-role assignments only appear in newly issued tokens. A user who is granted the role after they signed in must obtain a fresh token. In VS Code, run MCP: Reset Cached Tokens (or sign out of the Microsoft account from the Accounts menu), then restart the server and sign in again. You can confirm the result by pasting the access token into https://jwt.ms and checking for "roles": ["Privileged.Access"]. Scenario 4: Fronting an existing external MCP server that drives its own sign-in So far APIM has been the authorization resource. But many valuable MCP servers already exist and run their own identity. GitHub publishes a remote MCP server with dozens of tools, and it authenticates users against GitHub's own OAuth authorization server. You do not want to re-implement that. You want APIM to govern access (rate limits, IP rules, logging, a single managed endpoint) while letting the upstream own the login. This is the "expose an existing MCP server" passthrough mode. When you register GitHub's remote MCP server behind APIM, the gateway relays the upstream's own authorization challenge. The client never authenticates against Entra here. It authenticates directly against GitHub. The flow, confirmed by probing the gateway: A call to the APIM endpoint with no token returns GitHub's own 401 with a WWW-Authenticate header, relayed through APIM. The Protected Resource Metadata that GitHub serves advertises authorization_servers: ["https://github.com/login/oauth"], so the client knows to log in at GitHub. The PRM resource reflects the APIM host, because GitHub builds it from the forwarded Host header. The client trusts the APIM endpoint while still logging in at GitHub. VS Code completes the GitHub sign-in and the full tool catalog loads. In the proof of concept this surfaced all 47 GitHub tools through the single APIM endpoint. The client configuration is again just a URL pointing at APIM: { "servers": { "github-via-apim": { "url": "https://apim-contoso-mcp.azure-api.net/github-mcp/mcp", "type": "http" } } } The key insight is that APIM transparently relays the backend's authentication challenge. GitHub remains the authorization server, GitHub tolerates being fronted by APIM, and you get a governed, centrally managed entry point without owning the identity flow. [!NOTE] Passthrough only relays what the upstream advertises. If the backend's PRM resource value and the actual MCP transport endpoint differ by a path segment, some clients fall back to deriving the metadata location from the server URL and can miss it. When you onboard a custom self-authenticating server, verify that the resource it advertises matches the exact URL the client connects to. Scenario 5: Restricting which tools of an existing MCP server an agent may call Passthrough raises a governance question that token validation alone cannot answer. A developer may legitimately have permission to merge a pull request through GitHub, but you may not want their AI agent to perform that action autonomously. You want to allow the read and discovery tools while blocking the destructive write tools, at the gateway, regardless of what the client tries. What is and is not possible for an external server It is important to be precise here, because the capability differs from the REST-as-MCP mode: For a REST-API-exposed-as-MCP server, you pick which operations become tools at creation time. That is native tool selection and the cleanest possible filter. For an existing/external MCP server, APIM does not enumerate the upstream's tools. The portal Tools blade explicitly states that tools are not visible for external MCP servers, and there is no allow-list property for them. APIM also cannot safely rewrite the tools/list response, because reading the response body breaks the streaming transport and the list may arrive as text/event-stream. What APIM can do reliably, and server-agnostically, is block the invocation. Every tool call arrives as a JSON-RPC tools/call request in the request body, which APIM can inspect safely. The deny-listed tools remain visible in the catalog, but any attempt to invoke one is intercepted at the gateway and returned a JSON-RPC error before it ever reaches the upstream. The reusable deny-list fragment The block is driven by a per-server named value (a comma-separated list of tool names), so the same fragment governs every external server. Only the named value changes. <!-- Fragment: mcp-tool-filter (include after the auth fragment) --> <fragment> <choose> <when condition="@(context.Request.Body != null)"> <set-variable name="mcpMethod" value="@{ try { var body = context.Request.Body.As<JObject>(preserveContent: true); return (string)body?["method"] ?? string.Empty; } catch { return string.Empty; } }" /> <choose> <when condition="@(((string)context.Variables["mcpMethod"]).Equals("tools/call", StringComparison.OrdinalIgnoreCase))"> <set-variable name="mcpToolName" value="@{ var body = context.Request.Body.As<JObject>(preserveContent: true); return (string)body?["params"]?["name"] ?? string.Empty; }" /> <!-- mcpBlockedTools is a comma-separated deny-list set by the per-server policy before this include --> <set-variable name="mcpBlocked" value="@{ var tool = ((string)context.Variables["mcpToolName"]).Trim().ToLowerInvariant(); var deny = ((string)context.Variables.GetValueOrDefault("mcpBlockedTools", "")).ToLowerInvariant().Split(',').Select(t => t.Trim()); return deny.Contains(tool); }" /> <choose> <when condition="@((bool)context.Variables["mcpBlocked"])"> <return-response> <set-status code="200" reason="OK" /> <set-header name="Content-Type" exists-action="override"> <value>application/json</value> </set-header> <set-body>@{ var id = "null"; try { var body = context.Request.Body.As<JObject>(preserveContent: true); id = body?["id"]?.ToString(Newtonsoft.Json.Formatting.None) ?? "null"; } catch {} return "{\"jsonrpc\":\"2.0\",\"id\":" + id + ",\"error\":{\"code\":-32602,\"message\":\"Unknown tool: " + ((string)context.Variables["mcpToolName"]) + "\"}}"; }</set-body> </return-response> </when> </choose> </when> </choose> </when> </choose> </fragment> The deny-list itself lives in a named value, one per server: APIM named value. Comma-separated, case-insensitive. mcp-blocked-tools-github = merge_pull_request,create_repository,delete_repository,push_files,create_or_update_file,issue_write,label_write # <policies> <inbound> <base /> <set-variable name="mcpResourceMetadataUrl" value="https://apim-contoso-mcp.azure-api.net/.well-known/oauth-protected-resource/github-mcp/mcp" /> <include-fragment fragment-id="mcp-entra-auth" /> <set-variable name="mcpBlockedTools" value="{{mcp-blocked-tools-github}}" /> <include-fragment fragment-id="mcp-tool-filter" /> </inbound> <backend> <base /> </backend> <outbound> <base /> </outbound> <on-error> <base /> <include-fragment fragment-id="mcp-auth-challenge-onerror" /> </on-error> </policies> Generic per-server pattern: mcp-blocked-tools-<server> = <comma,separated,tool,names> Wiring it onto the GitHub passthrough server <policies> <inbound> <base /> <set-variable name="mcpResourceMetadataUrl" value="https://apim-contoso-mcp.azure-api.net/.well-known/oauth-protected-resource/github-mcp/mcp" /> <include-fragment fragment-id="mcp-entra-auth" /> <set-variable name="mcpBlockedTools" value="{{mcp-blocked-tools-github}}" /> <include-fragment fragment-id="mcp-tool-filter" /> </inbound> <backend> <base /> </backend> <outbound> <base /> </outbound> <on-error> <base /> <include-fragment fragment-id="mcp-auth-challenge-onerror" /> </on-error> </policies> Now when the agent tries to merge a pull request, the gateway returns a clean -32602 Unknown tool error and the upstream is never touched. Read and discovery tools continue to work. The tool still appears in the client's catalog. Adding governance for another external server is just one more named value plus the same fragment include. No new policy logic. Key takeaways API Management turns MCP servers into governed resources, applying the same identity, traffic, and observability controls you already use for APIs. Start simple with validate-azure-ad-token to gate access, then graduate to a full interactive sign-in by serving Protected Resource Metadata from a single APIM policy. You can publish multiple MCP servers from one underlying API, for example a read-only server and a read-write server, by selecting different operations. App roles let you authorize on a user attribute, not just tenant membership, and the check belongs in the per-server policy so shared logic stays clean. For existing external servers, APIM relays the upstream's own OAuth flow, so a server like GitHub keeps owning its identity while you keep central governance. When an external server's full tool surface is too broad, APIM can block specific tool invocations at the gateway with a reusable, named-value-driven policy, so a user's agent cannot perform actions the user could perform manually. References About MCP servers in Azure API Management Secure access to MCP servers in API Management Expose REST API in API Management as an MCP server Expose and govern an existing MCP server validate-azure-ad-token policy reference Policy fragments in API Management RFC 9728: OAuth 2.0 Protected Resource Metadata MCP authorization specification Star Wars API (example backend) MCP for Beginners
Microsoft Leads a New Era of Software Supply Chain Transparency
Microsoft announces the general availability of Microsoft’s Signing Transparency (MST) – a first-of-its-kind capability that brings unprecedented visibility and trust to our software supply chain. With this release, Microsoft is leading the industry by recording the build of critical cloud services into a publicly readable and verifiable SCITT standard (Supply Chain Integrity, Transparency, and Trust) compliant ledger. This means every production software build for in scope services like Azure Attestation and Azure Managed HSM (Hardware Security Module), Azure confidential ledger, Microsoft Signing Transparency itself (and others over time) – is now logged in an immutable, tamper-evident record. Only builds that are in the MST ledger are deployed to production; this gives customers confidence that the supply chain for these critical services can be audited at anytime. Notably, the MST ledger is fully open source and built to align with the emerging IETF SCITT standard. By embracing SCITT’s principles and open protocols, Microsoft ensures that MST not only secures our own ecosystem but also contributes to a broader industry movement toward standardized supply chain transparency. The open-source MST ledger serves as a verifiable trust anchor that any organization or researcher can inspect, audit, or even integrate with their own tooling. MST itself meets the highest levels of transparency, backed by a tamper-proof confidential ledger, open-source, and independently verified. Specifically, we are making the foundation of our trust model transparent and accessible to everyone – reinforcing that trust must be earned through proof, not just promises. This launch marks a major milestone in our commitment to Zero Trust principles, extending “never trust, always verify” all the way into the build itself. Building on a public preview introduced late last year, MST’s general availability delivers verifiable transparency at the software level. It transforms traditional code signing with an additive trust layer that is accessible via an open verification model. Every new software update is accompanied by a publicly auditable proof of integrity, enabling security teams to proactively confirm that each update is authentic and unaltered. To help organizations get the most out of this capability, we are also introducing a free tool to explore the contents – Ledger Explorer – an offline tool that allows security teams to examine MST ledger entries, verify cryptographic proofs, and even validate the ledger’s integrity independently. This tool, combined with MST’s open design, ensures that every Microsoft customer – and the broader community – can hold us accountable in real time for the software we run on their behalf. Key Benefits of Microsoft’s Signing Transparency (MST) Verified Code Integrity – Every software release is cryptographically logged in MST’s ledgers. This makes each build tamper-evident and traceable. If an attacker attempts to inject malicious code or sign an unauthorized update, it will be evident through the well-defined validation step built into the SCITT standard. Organizations gain the assurance that code integrity can be independently confirmed at any time. Independent Verification & Zero Trust – MST enables customers and auditors to verify software authenticity on their own, without having to solely rely on vendor attestations. For each update, Microsoft provides a transparency “receipt” (proof of logging) that you can use to prove the update was officially published and unaltered. This fosters a “don’t just trust, verify” approach, empowering security teams to double-check everything running in their environment aligns with what Microsoft intended. Audit-Trail & Compliance – The transparency ledger creates a permanent, auditable timeline of code deployments. Every entry is a record of what was released and when, backed by cryptographic proofs. This simplifies compliance reporting and accelerates forensic analysis. In the event of an incident, you can quickly audit the ledger to see if any unexpected code was introduced. For highly regulated industries, MST offers concrete evidence of software integrity and policy compliance over time. Leadership & Open Standards – We are delivering real transparency now, encouraging a future where all critical software is released with verifiable integrity. MST’s open source implementation and SCITT-compliant design exemplify our commitment to openness and collaboration. We believe widespread adoption of these standards will strengthen supply chain security for everyone, making trust verification a universal practice. Next Steps Microsoft’s Signing Transparency is more than a new security feature and shapes the advances in trust technology. As threats grow more sophisticated, we must evolve the way we assure our customers about the software they depend on. With MST now generally available, we are leading by example: proving that it is possible to open up the traditionally opaque process of software deployment and turn it into a source of strength and trust, i.e. empowering each person with verifiable transparency. We invite the industry to join us on this journey and get started by reading the documentation and exploring Ledger Explorer today! Together, by embracing transparency and open standards, we can turn “trust but verify” from a slogan into an everyday reality for digital infrastructure.Enterprise-ready Claude Desktop with Entra ID, APIM, and Microsoft Foundry (No Backend Required)
How I put corporate sign-in in front of Claude Desktop without writing a single line of backend code. TL;DR — In this post, I show how to securely enable Claude Desktop in enterprise environments using Microsoft Entra ID, Azure API Management, and Microsoft Foundry — without deploying a custom backend. This approach removes API keys from endpoints, enforces per-user identity, and aligns fully with Zero Trust principles. Who this is for: Enterprise architects evaluating secure AI client patterns Developers enabling Claude Desktop in regulated environments Platform teams standardizing identity and governance for LLM access Why this post exists: Microsoft Learn's Configure Claude Desktop with Foundry Models only shows the API-key path — a shared key pasted into every user's Claude Desktop config. That's fine for a quick demo, but it's a non-starter for most enterprises (no per-user identity, no MFA / Conditional Access, hard to revoke, hard to audit). This post fills that gap: same Foundry backend, but with Microsoft Entra ID SSO in front via Azure API Management, so each user signs in with their corporate identity and zero secrets land on the laptop. The problem For many teams experimenting with Claude Desktop, the blocker isn't capability — it's enterprise readiness. How do you enforce identity, eliminate shared secrets, and apply governance without standing up a custom backend service to sit in front of the model? If your team wants to use Claude Desktop with your own Anthropic deployment running on Microsoft Foundry, but with a few non-negotiable requirements: No shared API keys floating around on developer laptops. Per-user identity — every request must be attributable to a real person. MFA and Conditional Access must apply, the same way they do for every other internal app. Central rate-limiting and logging — a centralized control plane for governance. Claude Desktop 1.5+ supports a "Gateway SSO" mode where it can sign each user in with OpenID Connect and forward their token to a custom LLM gateway. Azure API Management (APIM) is a perfect fit for that gateway role: it validates the user's Entra ID token, then re-authenticates itself to Foundry behind the scenes. APIM acts as a centralized policy enforcement layer, enabling identity validation, traffic governance, and secure re-authentication to backend AI services without custom code. The end-to-end flow looks like this: %%{init: {'flowchart': {'nodeSpacing': 60, 'rankSpacing': 80, 'useMaxWidth': true}, 'themeVariables': {'fontSize':'16px'}} }%% flowchart TB User([Corporate user]) Claude["Claude Desktop"] Entra["Microsoft Entra ID<br/>(OIDC + MFA + Conditional Access)"] APIM["Azure API Management<br/>validate-jwt → rewrite headers<br/>(policy gateway)"] Foundry["Microsoft Foundry<br/>Claude deployment"] User -- "1. Sign in (browser PKCE)" --> Entra Entra -- "2. ID token" --> Claude Claude -- "3. POST /v1/messages<br/>Authorization: Bearer ID token" --> APIM APIM -- "4. OIDC discovery / JWKS" --> Entra APIM -- "5. x-api-key (or Managed Identity)" --> Foundry Foundry -- "6. Response" --> APIM APIM -- "7. Response" --> Claude classDef azure fill:#0a4d8c,stroke:#0a3a6b,color:#ffffff; classDef client fill:#f3f3f3,stroke:#888,color:#222; class Entra,APIM,Foundry azure; class Claude,User client; Or in plain text: Claude Desktop │ Authorization: Bearer <Entra ID token from the user's browser sign-in> ▼ Azure API Management (<your-apim>) │ ① validate-jwt → verifies user's Entra ID token │ ② re-auths to Foundry with an API key from a Named value │ Authorization stripped, x-api-key injected ▼ Microsoft Foundry /anthropic/v1/messages │ runs Claude (<your-deployment>) ▼ Response back to the user There are no API keys on user devices. Foundry's key lives only inside APIM. And every request carries the user's oid claim, so I can build dashboards and per-user quotas later. What you need before starting An Azure subscription with a Microsoft Foundry (AI Services) account and a Claude deployment. (Throughout this post I'll just call it Foundry.) An API Management instance, any tier. Permission to register applications in Entra ID for your tenant. Claude Desktop 1.5.0 or later. Azure CLI installed locally. Throughout this post I'll use placeholders for resource names: <apim-name> — your API Management service name <resource-group> — the resource group that holds it <foundry-account> — your Foundry account name <deployment-name> — the name of the Claude model deployment on Foundry Step 1 — Register an Entra ID app for Claude Desktop This is the OIDC client Claude Desktop signs users into. Claude Desktop requires a single-tenant, public PKCE client (no client secret) with a loopback redirect URI, configured under the Mobile and desktop applications platform in Entra ID — the only platform that allows any loopback port. I scripted it so the setup is one command and idempotent: # scripts/register-claude-entra-app.ps1 [CmdletBinding()] param( [string] $TenantId = '<your-tenant-id>', [string] $SubscriptionId = '<your-subscription-id>', [string] $ResourceGroup = '<resource-group>', [string] $ApimName = '<apim-name>', [string] $AppDisplayName = 'Claude Cowork gateway', [string] $RedirectUri = 'http://127.0.0.1/callback' ) az account set --subscription $SubscriptionId | Out-Null # 1. Create (or reuse) the app registration $appId = az ad app list --display-name $AppDisplayName --query "[0].appId" -o tsv if (-not $appId) { $appId = az ad app create --display-name $AppDisplayName ` --sign-in-audience AzureADMyOrg --query appId -o tsv } # 2. Configure as public PKCE client with the Mobile/Desktop redirect URI $objectId = az ad app show --id $appId --query id -o tsv $patch = @{ publicClient = @{ redirectUris = @($RedirectUri) } isFallbackPublicClient = $true } | ConvertTo-Json -Depth 5 -Compress az rest --method PATCH ` --uri "https://graph.microsoft.com/v1.0/applications/$objectId" ` --headers "Content-Type=application/json" --body $patch | Out-Null # 3. Ensure a service principal exists $sp = az ad sp list --filter "appId eq '$appId'" --query "[0].id" -o tsv if (-not $sp) { az ad sp create --id $appId | Out-Null } # 4. Push two Named values into APIM for the validate-jwt policy az apim nv create -g $ResourceGroup --service-name $ApimName ` --named-value-id entra-tenant-id --display-name entra-tenant-id ` --value $TenantId --secret false az apim nv create -g $ResourceGroup --service-name $ApimName ` --named-value-id entra-client-id --display-name entra-client-id ` --value $appId --secret false "Client ID: $appId" Run it once. The output prints the client ID you'll need in Claude Desktop later, and it leaves two Named values in APIM ( entra-tenant-id , entra-client-id ) that the gateway policy will reference. ⚠️ Common pitfall: if the redirect URI ends up under the Web platform instead of Mobile and desktop applications, Entra will demand a client secret on token exchange — Claude won't send one and you'll get Token exchange failed (HTTP 401) . The app type can't be changed after creation, so create a new app if that happens. Step 2 — Create the API in APIM In the portal under APIM → APIs → + Add API → HTTP: Field Value Display name Anthropic API Name anthropicapi Web service URL https://<foundry-account>.services.ai.azure.com/anthropic API URL suffix claude Subscription required Off (Entra ID is our only credential) Add two operations under it: Method URL Display name POST /v1/messages Create message GET /v1/models List models The /v1/models operation isn't strictly needed (Foundry's Anthropic surface doesn't implement it), but having it registered means you can decide later whether to stub it out or proxy it. Step 3 — Add an API key for Foundry as a Named value APIM → Named values → + Add: Name: foundry-key Type: Secret Value: paste a key from the Foundry account's Keys and Endpoint blade. This is the only place the key ever lives. Clients never see it. Alternative — keyless with Entra ID (managed identity): If you prefer not to manage a Foundry key at all, enable the APIM instance's system-assigned managed identity (APIM → Identity → System assigned → On), then grant that identity the Foundry User role on the Foundry account (role ID 53ca6127-db72-4b80-b1b0-d745d6d5456d — previously named Azure AI User; Microsoft renamed it but the ID and permissions are unchanged). In Step 4, replace the set-header that injects x-api-key with: <authentication-managed-identity resource="https://cognitiveservices.azure.com" output-token-variable-name="foundry-token" /> <set-header name="Authorization" exists-action="override"> <value>@("Bearer " + (string)context.Variables["foundry-token"])</value> </set-header> Then you can skip the foundry-key Named value entirely. Don't use the legacy Cognitive Services User role — per the Foundry RBAC doc, roles starting with Cognitive Services don't apply to Foundry scenarios. Step 4 — Write the gateway policy This is the core enforcement layer in the architecture. Open APIs → anthropicapi → All operations → Inbound processing → </> and paste: <policies> <inbound> <base /> <!-- USER → APIM: verify Entra ID token from Claude Desktop --> <validate-jwt header-name="Authorization" failed-validation-httpcode="401" failed-validation-error-message="Unauthorized" require-scheme="Bearer"> <openid-config url="https://login.microsoftonline.com/{{entra-tenant-id}}/v2.0/.well-known/openid-configuration" /> <audiences> <audience>{{entra-client-id}}</audience> </audiences> <issuers> <issuer>https://login.microsoftonline.com/{{entra-tenant-id}}/v2.0</issuer> </issuers> </validate-jwt> <!-- APIM → Foundry --> <set-backend-service base-url="https://<foundry-account>.services.ai.azure.com/anthropic" /> <set-header name="x-api-key" exists-action="override"> <value>{{foundry-key}}</value> </set-header> <set-query-parameter name="api-version" exists-action="skip"> <value>2024-05-01-preview</value> </set-query-parameter> </inbound> <backend><base /></backend> <outbound><base /></outbound> <on-error><base /></on-error> </policies> Two things to notice: validate-jwt uses the OIDC discovery URL — JWKS keys are fetched and cached automatically. It rejects any token whose aud claim is not the client ID of our Entra app, which is exactly what we want. The Authorization header from the user is not forwarded — once validate-jwt succeeds, the request is re-authenticated to Foundry with x-api-key . No user token ever leaves APIM. APIM becomes the security boundary — user identity is validated at the edge, and downstream services never see or rely on user tokens. Step 5 — Configure Claude Desktop Open Claude Desktop → Configure third-party inference and fill it in like this: Field Value Connection Gateway Credential kind Interactive sign-in Gateway base URL https://<apim-name>.azure-api.net/claude Client ID (the appId your script printed) Issuer URL https://login.microsoftonline.com/<tenant-id>/v2.0 Authorization URL / Token URL leave empty Bearer token ID token (default) Scopes leave default ( openid profile email offline_access ) Redirect port leave empty (ephemeral) Model discovery Off Model list → Model ID <deployment-name> (your Foundry deployment name) ℹ️ Why Model discovery is Off — Claude Desktop's discovery uses GET /v1/models , and the Foundry /anthropic surface doesn't implement that endpoint, so it 404s. Listing the model manually skips the call entirely. If you want to leave Model discovery On, stub /v1/models in APIM. Add a GET /v1/models operation to your API and give it this inbound policy that returns an Anthropic-shaped response without ever hitting the backend: <policies> <inbound> <base /> <return-response> <set-status code="200" reason="OK" /> <set-header name="Content-Type" exists-action="override"> <value>application/json</value> </set-header> <set-body>@{ return new JObject( new JProperty("data", new JArray( new JObject( new JProperty("id", "<deployment-name>"), new JProperty("type", "model"), new JProperty("display_name", "Claude on Foundry"), new JProperty("created_at", "2026-01-01T00:00:00Z") ) )), new JProperty("has_more", false), new JProperty("first_id", "<deployment-name>"), new JProperty("last_id", "<deployment-name>") ).ToString(); }</set-body> </return-response> </inbound> <backend><base /></backend> <outbound><base /></outbound> <on-error><base /></on-error> </policies> Add one entry per deployment you want to expose. The benefit of stubbing rather than turning discovery off is that adding new models becomes a policy edit — no need to re-export and redeploy Claude Desktop config to every user. Click Apply Changes then Sign in to your organization. Your browser opens to the normal Entra sign-in page; once approved you're returned to the app, and a quick connection test runs. The success indicator is a small green banner: ✅ Inference — 1-token completion in 1449 ms · via identity provider For broader rollout, hit the Export button at the top of the configuration window — it produces a .mobileconfig (macOS) or .reg (Windows) you can push via Intune / Jamf to every user's machine. Step 6 — Verify both hops In APIM → APIs → anthropicapi → Test → POST /v1/messages I sent: Headers: anthropic-version: 2023-06-01 Body: { "model": "<deployment-name>", "max_tokens": 64, "messages": [{"role":"user","content":"hi"}] } Click Send → Trace, and look at two places: Inbound → validate-jwt: should say succeeded and show the decoded claims (your oid , email , etc.). Backend → Request: outbound URL is https://<foundry-account>.services.ai.azure.com/anthropic/v1/messages?api-version=2024-05-01-preview , with x-api-key: **** present and Authorization absent. Backend → Response: 200, with a Claude message JSON body. That confirms both halves of the chain. Bumps I hit along the way A few common issues encountered during setup — sharing so you can skip them: Symptom Cause Fix Claude shows "Your provider's model list hasn't loaded yet" and /v1/models returns 404 Foundry's Anthropic surface doesn't implement that endpoint Turn Model discovery OFF in Claude Desktop and add the deployment name manually Claude shows "Authentication failed" even though sign-in worked The APIM API still had Subscription required = ON, blocking the call before validate-jwt ran with 401: Access denied due to missing subscription key Uncheck Subscription required on the API Portal Test panel shows "Cannot read properties of undefined (reading 'statusCode')" The test console doesn't attach an Entra token, so validate-jwt 401s and the panel's JavaScript crashes Comment out <validate-jwt> temporarily for portal testing, or test via curl with a real token OIDC discovery failed (HTTP 404) in Claude Desktop Pasted the metadata URL into Issuer URL Issuer must end at /v2.0 , not at /.well-known/openid-configuration Token exchange failed (HTTP 401) App registered under Web platform instead of Mobile and desktop applications Create a new app with the right platform — it can't be changed Where this leaves us This pattern is small in moving parts but has outsized architectural impact: Zero secrets on endpoints. Eliminates API-key sprawl across laptops, MDM profiles, and shared vaults. The Foundry key lives only inside APIM — or disappears entirely when you switch APIM to managed identity. Identity, not credentials. Every Claude Desktop user authenticates against Entra ID in their browser, the same as Office or Teams. MFA, Conditional Access, and Entra ID Protection apply automatically — no parallel auth story to maintain. Per-user observability built in. APIM logs carry the user's Entra oid , email , and group claims. That unlocks per-user dashboards, cost allocation, and abuse detection without any client-side instrumentation. Aligned with Zero Trust. Strong identity at the edge, no implicit trust between hops, single policy chokepoint for inspection and rate-limiting, and full revocability through a single Enterprise Application. Optional but trivial keyless path. Flip APIM to system-assigned managed identity + <authentication-managed-identity resource="https://cognitiveservices.azure.com" /> and one Foundry User role assignment (role ID 53ca6127-db72-4b80-b1b0-d745d6d5456d , formerly Azure AI User) on the Foundry account. See the Foundry RBAC doc — don't use any Cognitive Services * roles for Foundry. What I'd add next llm-token-limit and llm-emit-token-metric policies for per-user quotas and cost visibility. App Insights wiring on the API, with a workbook that pivots on the oid claim. Assignment required = Yes on the Entra Enterprise Application + a security group, so only approved users can sign in. Intune deployment of the exported .reg / .mobileconfig so the gateway URL and client ID land on devices automatically. But that's all incremental. The hard part — getting Claude Desktop, Entra ID, APIM, and Foundry to agree on who's allowed to talk to whom — is done. Total elapsed: about an afternoon, most of it spent learning where each portal hides its switches. Useful links Gateway single sign-on with your identity provider — Claude.ai Documentation Configure Claude Desktop with Foundry Models — Microsoft Learn Role-based access control for Microsoft Foundry — Microsoft LearnAzure Function App — Queue-Based Architecture for Long-Running Sync Jobs
The Problem: HTTP Triggers and Long-Running Jobs Don't Mix Here's a situation you've probably run into: you have a job that needs to loop over dozens of Azure resources, call APIs, and do real work. You wrap it in an HTTP-triggered Azure Function so it can be called on demand. It works great and after a few minutes, the caller gets a 504 Gateway Timeout. The 230-second limit is enforced by Azure Front Door / the platform load balancer. It cannot be overridden by app settings or host configuration. Any HTTP trigger that runs longer than ~3.5 minutes will timeout for the caller. In our case, the job iterates over 30+ Azure subscriptions — for each one it switches context, lists resources, and triggers image imports. Total runtime: anywhere from 2 to 10 minutes depending on how many ACRs need updating. Way over the limit. The Solution: Decouple Request from Execution via a Queue The fix is clean once you see it: the HTTP trigger shouldn't do the work — it should just accept the work and hand it off. That's what a queue is for. The flow splits into two independent phases: Request phase — The HTTP trigger validates the caller (JWT + app role check), packages the job parameters into a queue message, and returns 202 Accepted. This takes under 3 seconds. Execution phase — A Queue Trigger picks up the message and runs the actual sync. No HTTP connection involved, so there's no timeout. On a Dedicated (P-series) plan, execution time is unlimited. Approach What the caller gets Result HTTP trigger → run sync inline Waits for the full job to complete 504 TIMEOUT after 230 seconds HTTP trigger → Queue → Queue Trigger 202 Accepted immediately NO TIMEOUT job runs as long as needed 🤸♀️There's an added bonus - Reliability in Azure Queue Storage: Azure Storage Queues give you automatic retry out of the box. If the job crashes halfway through, the message becomes visible again after a visibility timeout and the Queue Trigger picks it up for a retry — up to 5 attempts before the message is moved to the poison queue. No retry logic to write 🤸♀️. Locking Down the Endpoint Since the HTTP trigger is the public entry point, it needs solid auth. We layer two things: ⭐Use EasyAuth for the "is this a real Entra ID token?" check, and a custom App Role for the "is this person allowed to trigger syncs?" check. These are independent concerns and should stay that way. Layer What it does How EasyAuth (Entra ID) Rejects requests without a valid Entra ID Bearer token — before your code even runs Configured at the Function App level via the Authentication blade App Role check Validates that the token contains the SyncJob.Execute role — only assigned users/SPs can trigger the job Decoded in the function code from the JWT roles claim Managed Identity Authenticates the Function App to Azure APIs (no credentials in code) Connect-AzAccount -Identity — identity assigned via RBAC One gotcha worth knowing: when using v2 tokens (which is the default with modern App Registrations), the aud claim in the token is the raw App ID GUID — not the api:// prefixed URI. You need to explicitly add both forms to your allowedAudiences in EasyAuth, otherwise valid tokens get rejected. APP_ID="<your-app-id>" TENANT_ID="<your-tenant-id>" FUNCTION_APP_URL="https://<your-function-app>.azurewebsites.net" # Interactive login (device code flow — works from any terminal) az login --tenant "${TENANT_ID}" \ --scope "api://${APP_ID}/.default" \ --use-device-code TOKEN=$(az account get-access-token \ --scope "api://${APP_ID}/.default" \ --query accessToken -o tsv) # Trigger the sync — returns 202 immediately curl -s -X POST "${FUNCTION_APP_URL}/api/SyncContainerRegistryHttpTrigger" \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" Passing Parameters Through the Queue One nice property of this pattern: the queue message is just JSON, so you can pass whatever parameters the job needs. In our case, we pass a subscriptionFilter wildcard so callers can target a subset of subscriptions without touching any code. The parameter travels the full chain: HTTP body → queue message → Queue Trigger → PowerShell script parameter. Here's how each step handles it. Step 1 — HTTP Trigger reads the body and enqueues the message using the Push-OutputBinding output binding. Azure Functions wires the binding to the queue automatically — no SDK call needed: param($Request, $TriggerMetadata) # ... decode the JWT, check role assignment $queuePayload = @{ triggeredBy = $decoded.Payload.upn ?? $decoded.Payload.oid triggeredAt = (Get-Date -Format 'o') subscriptionFilter = if ($body.subscriptionFilter) { $body.subscriptionFilter } else { "*" } } | ConvertTo-Json -Compress Push-OutputBinding -Name QueueMessage -Value $queuePayload Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{ StatusCode = [System.Net.HttpStatusCode]::Accepted Body = @{ message = "Sync job queued. Check Azure Monitor logs for execution status." } }) ⭐Push-OutputBinding is how Azure Functions PowerShell workers write to output bindings (queues, blobs, HTTP responses…). The binding name QueueMessage maps to the queue defined in function.json — the runtime handles serialisation and delivery. Step 2 — Queue Trigger passes the filter to the script as a named parameter: param($QueueItem, $TriggerMetadata) Write-Host "Triggered SyncContainerRegistry via Storage Queue. Payload: $QueueItem" $subscriptionFilter = if ($QueueItem.subscriptionFilter) { $QueueItem.subscriptionFilter } else { "*" } $SubscriptionFilter = $subscriptionFilter . "$PSScriptRoot/../SyncContainerRegistry/run.ps1" Step 3 — Long running job with the filter as parameter: param($Timer) if (-not $SubscriptionFilter) { $SubscriptionFilter = "*" } $subscriptions = Get-AzSubscription | Where-Object { $_.Name -like $SubscriptionFilter } foreach ($subscription in $subscriptions) { Set-AzContext -SubscriptionId $subscription.Id | Out-Null # ... do the work } Targeting a subset of subscriptions # Sync all subscriptions (default — omit the body) curl -s -X POST "${FUNCTION_APP_URL}/api/SyncContainerRegistryHttpTrigger" \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" # Sync only subscriptions matching a pattern curl -s -X POST "${FUNCTION_APP_URL}/api/SyncContainerRegistryHttpTrigger" \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ -d '{"subscriptionFilter": "*project-alpha*"}' ⭐PowerShell's -like operator uses * as a wildcard anywhere in the string. The pattern *project-alpha* matches sub-mycompany-project-alpha-prd, sub-mycompany-project-alpha-dev, etc. A pattern without a leading * only matches from the start of the string — keep this in mind when naming subscriptions. Pushing a Message Directly via PowerShell You can also push a message straight to the queue without going through the HTTP trigger — useful for testing, scripting, or bypassing the auth layer in a controlled environment. Connect-AzAccount # or -Identity for a Managed Identity context $storageAccount = "<your-storage-account>" $queueName = "sync-job-queue" # Build the payload — same shape the HTTP trigger produces $payload = @{ triggeredBy = $env:USERNAME triggeredAt = (Get-Date -Format 'o') subscriptionFilter = "*project-alpha*" # or "*" for all } | ConvertTo-Json -Compress # Get a queue client via the connected account (no key needed) $ctx = New-AzStorageContext -StorageAccountName $storageAccount -UseConnectedAccount $queue = Get-AzStorageQueue -Name $queueName -Context $ctx $queue.QueueClient.SendMessage($payload) ⭐ -UseConnectedAccount authenticates via the current Connect-AzAccount session — no storage key required, as long as your identity has the Storage Queue Data Message Sender role on the storage account. The Queue Message The HTTP trigger packages the caller identity and filter into a simple JSON payload before enqueuing. The Queue Trigger reads it back as a deserialised PowerShell object — no manual JSON parsing needed. { "triggeredBy": "user@company.com", "triggeredAt": "2026-06-01T11:03:55.570+02:00", "subscriptionFilter": "*project-alpha*" } Design Decisions at a Glance Decision Choice Why Async execution Azure Storage Queue HTTP trigger has a hard 230s timeout. The sync job takes 2–10 minutes. The queue decouples acceptance from execution — and gives us retry for free. Authentication EasyAuth + App Role No credentials in code. Access is controlled via Entra ID app roles — revocable per user without touching infrastructure. Azure identity Managed Identity No secrets to rotate or store. The Function App authenticates to Azure APIs using its platform-assigned identity. Job parameter Wildcard filter via queue payload Lets callers target any subscription subset without code changes. The filter travels through the queue — the Queue Trigger just passes it along. Hosting plan Dedicated (P-series) Consumption plan caps function execution at 10 minutes. A Dedicated plan has no execution time limit — essential when the job can run longer. See you in the Cloud JamesdldAgents That Test Agents: A Cloud-Native Skill-Eval Harness on Foundry Hosted Agents
Skills are an agent's must-have. So test them. A skill is the lightest way to give an agent durable, reusable behavior: a SKILL.md file you author once, store centrally in Foundry's versioned Skills API, and inject into a Hosted Agent's context — no code change, no redeploy. That's why skills have quietly become standard equipment for production agents. But the moment a skill carries real behavior, a hard question follows: how do you know it still works? When you edit a skill you can't feel whether you improved it or just changed it. It might stop triggering, skip a required section, or quietly produce a worse result on one model than another. The cure is the same discipline we use for any prompt — evaluate it: run the agent, capture what happened, and grade it against a small set of checks. This is exactly what azure_skill_eval does for one concrete skill: edu-video-script, which writes an education short-video script for a given knowledge point (the sample's smoke test asks it to script the "P vs NP problem"). And it does the whole thing cloud-native, on Foundry Hosted Agents. The scenario: one skill, two models, four hosted agents The skill under test is edu-video-script. The clever part of the harness is that it doesn't just check one run — it puts the skill on a stand and stresses it from three sides, using four Foundry Hosted Agents wired together by the Agent Framework FoundryAgent: Hosted agent Role skill-eval-business-agent-gpt System under test (SUT), running edu-video-script on gpt-5.5 skill-eval-business-agent-deepseek The same skill, running on DeepSeek-V4-Pro skill-eval-attacker-agent Multi-turn adversarial prompt generator skill-eval-judge-agent LLM-as-judge that returns a rubric score as JSON Two business agents run the same skill on different models, so every case becomes an apples-to-apples comparison: which model executes this skill better? The attacker and judge are the graders. What we measure (define "done" first) Good evals start from a checkable definition of done — outcome, process, style, efficiency. For an education-video script that means: Did it produce a valid script (outcome)? Did it actually follow the edu-video-script template (process/style)? Does it hold up when a user pushes on it across turns (robustness)? The harness answers these with three grading layers. 1. Deterministic checks first (validator.py) The cheapest, most explainable signal: does the output match the script template the skill is supposed to produce? validator.py runs fixed, deterministic template checks — no model needed. These catch the obvious regressions instantly and never cost a token. 2. The LLM judge (skill-eval-judge-agent) Template checks answer "did it do the basics?" but not "is the script any good?" — pacing, clarity, whether it teaches the concept. For that, a dedicated judge hosted agent grades the result and returns structured JSON so scores compare cleanly across runs and models: { "overall_pass": true, "score": 100, "checks": [] } Structured output is the point: stable fields (overall_pass, score, checks) diff cleanly between GPT and DeepSeek, and between today's skill version and last week's. 3. The multi-turn attacker (test_agent.py + skill-eval-attacker-agent) A skill that looks great on a clean prompt can still fall apart when a user pushes on it. The attacker agent generates adversarial prompts for a knowledge point using a chosen strategy — for example extreme length — and keeps the pressure on across multiple turns (max_turns, default 3). This is where you find out whether edu-video-script stays on-template under stress, not just on the happy path. # the attacker takes a knowledge point + a strategy, emits one user prompt azd ai agent invoke skill-eval-attacker-agent \ "Topic: P vs. NP problem Recommended attack strategy: Extreme length Please output the unique user prompt text." The eval loop, end to end runner.py is a ghcsdk-style pipeline that runs cases × models, with each side toggleable: pick all models / GPT only / DeepSeek only, run a single case (e.g. edge-03), and switch adversarial mode, single-turn vs multi-turn, and judge grading on or off. The same switches are query parameters on POST /api/run: model, only_case, use_attack, single_turn, use_judge, max_turns. The test set lives in shared/test_cases.py — 10 built-in edge cases (edge-01 … edge-10) exported to evals/evals.json. You don't need a giant benchmark; a small, sharp set catches regressions, and you grow it whenever a real failure shows up: python -m evals.export_evals # regenerate evals/evals.json from shared/test_cases.py Every SUT call goes through runtime.py, which follows the official Agent Framework hosted-agent sample: it opens a fresh hosted session per turn, invokes via Responses, and tears the session down afterward. # shared/runtime.py — the documented Foundry hosted-agent pattern project = AIProjectClient(endpoint=FOUNDRY_PROJECT_ENDPOINT, credential=cred, allow_preview=True) agent = FoundryAgent(project_client=project, name=agent_name, # e.g. skill-eval-business-agent-gpt allow_preview=True) session = project.beta.agents.create_session(agent_name=agent_name) # ... send the (possibly adversarial) prompt, collect the Responses output ... So a single case flows: runner → business agent (skill runs) → validator → judge, optionally with the attacker driving multiple turns first. Cloud-native by design — and why that matters for eval This is the part that makes the harness production-grade rather than a laptop script. The hard parts of an eval harness — provisioning agents, recording every run, scaling trials, governing access — are handled by Azure, not by you. Foundry Hosted Agents are the runtime. The SUT, attacker, and judge all run as managed hosted agents in your Foundry project. You bring the skill and the cases; Foundry hosts the agents, models, and sessions. The business agents deploy with host: azure.ai.agent and docker.remoteBuild: true, so azd deploy builds the containers in Azure Container Registry — local Docker doesn't even need to be running. The UI is serverless. A FastAPI app on Azure Container Apps lets you upload evals.json, watch progress live, and browse the dashboard — scale-to-zero when no one's running evals. Every run is durable. Results land in Azure Blob Storage (skill-eval-runs), one yymmdd-XXXXXX/ folder per run, with a newest-first runs.json index. Nothing lives only in a terminal scrollback. Access is identity-based. In the cloud, a user-assigned Managed Identity carries exactly two roles — Storage Blob Data Contributor + Azure AI User; locally it's AzureCliCredential. No keys in env files. It's reproducible infra. azd up runs infra/main.bicep to stand up Storage, the container, Log Analytics, the Container Apps environment, the identity, and the role assignments in one shot. The payoff: the scores you read came from the same hosted runtime you actually ship to — not a local approximation — and the run that produced them is sitting in Blob, comparable against every run before it. Run it Local (no deploy): conda activate agentdev cd Skill_eval/azure_skill_eval pip install -r requirements.txt cp .env.example .env # FOUNDRY_PROJECT_ENDPOINT + AZURE_STORAGE_* uvicorn webapp.app:app --reload --port 8000 Open http://localhost:8000, upload evals/evals.json, pick your models and modes, and click Run. Cloud (azd): azd auth login azd env new skill-eval-dev azd env set FOUNDRY_PROJECT_ENDPOINT https://<project>.services.ai.azure.com/api/projects/<project> azd env set MODEL_GPT gpt-5.5 azd env set MODEL_DEEPSEEK DeepSeek-V4-Pro azd up Provision the skill once, deploy the four hosted agents, then smoke-test them: python -m hosted_agent.provision_skills # upload edu-video-script to Foundry Skills azd deploy skill-eval-business-agent-gpt azd deploy skill-eval-business-agent-deepseek azd deploy skill-eval-attacker-agent azd deploy skill-eval-judge-agent azd ai agent invoke skill-eval-business-agent-gpt "Here is a script for an educational short video on the P vs. NP problem." Read the results Each run is self-contained on Blob: summary.json gives you the headline — pass rate and judge averages — and the per-{case}__{model}.json files let you open any single result and see exactly what the skill produced and why it passed or failed. The dashboard streams these straight from Blob via /api/runs/{run_id}/files/{filename}. Because GPT and DeepSeek ran the same cases, the comparison is right there in one folder. Takeaways A skill you can't evaluate is a skill you can't trust. edu-video-script is treated like code — versioned in Foundry, run, and graded. Stack your graders cheap-to-expensive. Deterministic template checks first (validator.py), then an LLM judge for quality, then a multi-turn attacker for robustness. Make the judge return structured JSON. overall_pass / score / checks compare cleanly across models and skill versions. Compare models on the same skill. Running GPT-5.5 and DeepSeek-V4-Pro side by side turns "which model?" from a guess into a measured answer. Let the platform carry the harness. Foundry Hosted Agents are the runtime; Azure Container Apps, Blob Storage, Managed Identity, and azd/Bicep make the whole loop reproducible and durable. Write the skill. Then build the harness that proves it. On Foundry, that second step is mostly configuration — and the result is a skill you can actually trust in production. Conclusion Skills moved agent behavior out of code and into versioned Markdown — a huge win for reuse, but only if you can prove a skill still works after every edit. azure_skill_eval answers that for edu-video-script by treating evaluation as a first-class, repeatable step rather than a gut check. The shape is simple and worth copying for any skill of your own: Pin down "done" as checkable criteria, then encode a small set of sharp cases (here, 10 edge cases). Grade in layers, cheap to expensive — deterministic template checks, then a structured LLM-judge rubric, then a multi-turn adversarial pass. Run the same cases across models (GPT-5.5 vs DeepSeek-V4-Pro) so model choice becomes a measurement, not a guess. Let the cloud carry it — Foundry Hosted Agents as the runtime, FastAPI on Azure Container Apps for the UI, Blob Storage for durable runs, Managed Identity for access, and azd/Bicep so the whole thing is reproducible. The result is a feedback loop where every skill change is confirmed, every regression is visible, and every score traces back to the same hosted runtime you ship to. That's the difference between building skills and being able to trust them — and on Foundry, the gap between the two is mostly configuration. Sample Code : https://github.com/kinfey/Multi-AI-Agents-Cloud-Native/tree/main/code/Skill_eval
Troubleshooting ML Model Loading, GPU Issues, and Memory Pressure in Azure Container Apps
Introduction Deploying an AI application to Azure Container Apps is fundamentally different from deploying a web API. When you containerize a Django REST API, the application starts in a few seconds, the memory footprint is predictable, and the CPU usage scales linearly with requests. When you containerize a PyTorch model server, a LangChain agent, or an ONNX inference service, you are dealing with a completely different category of problem. Large language models, computer vision models, and embedding pipelines can take minutes to load, consume gigabytes of memory before serving a single request, and produce bizarre errors when they encounter resource limits that look nothing like a standard out-of-memory exception. Add to that the challenge of running GPU workloads (or simulating them on CPU) in a containerized environment, and you have a troubleshooting landscape that catches even experienced ML engineers off guard. This part of the series covers the real-world scenarios you will encounter when running AI workloads on Azure Container Apps, with specific focus on what goes wrong during deployment and startup — and how to fix it methodically. Scenario 1: The Model Takes 3+ Minutes to Load and the Container Gets Killed Before It Starts What You See You deploy your model inference service. In the logs you can see it is downloading or loading the model from disk: INFO: Loading model from /app/models/model.bin INFO: Loading tokenizer... INFO: Loading weights layer 0/48... INFO: Loading weights layer 12/48... And then, abruptly, the container disappears and a new one starts. The liveness or startup probe has timed out and Container Apps has killed the container before the model finished loading. You end up in an endless restart loop where the model never fully loads. Why This Happens The default probe configuration does not account for long model loading times. The liveness probe begins firing almost immediately after the container starts. If your model takes 3 minutes to load and your liveness probe allows only 30 seconds of failure before killing the container, the model never gets a chance to finish loading. Container Apps is doing exactly what it was configured to do — it just was not configured with your workload in mind. There is a second, related problem: if your model file is downloaded at container startup (from Azure Blob Storage, Hugging Face Hub, or a mounted file share), the download time is added on top of the load time, making the window even wider. Step-by-Step Fix Step 1 — Separate your startup probe from your liveness probe. A startup probe fires repeatedly until it succeeds, and while it is in progress, the liveness probe is suppressed. This gives your model the time it needs to load without the risk of being killed. Set a generous `failureThreshold` and `periodSeconds`: probes: - type: Startup httpGet: path: /health/startup port: 8080 initialDelaySeconds: 10 periodSeconds: 15 failureThreshold: 40 # 40 * 15s = 600 seconds (10 minutes) for model to load - type: Liveness httpGet: path: /health/live port: 8080 periodSeconds: 30 failureThreshold: 3 - type: Readiness httpGet: path: /health/ready port: 8080 periodSeconds: 10 failureThreshold: 3 Step 2 — Implement a proper three-tier health endpoint in your model server. Each probe endpoint should return the appropriate status based on what it knows about the model: # FastAPI model server with staged health endpoints from fastapi import FastAPI, HTTPException from contextlib import asynccontextmanager import asyncio model = None model_loaded = False @asynccontextmanager async def lifespan(app: FastAPI): global model, model_loaded # Model loads in the background so the HTTP server starts immediately asyncio.create_task(load_model_async()) yield # Cleanup on shutdown model = None app = FastAPI(lifespan=lifespan) async def load_model_async(): global model, model_loaded import logging logger = logging.getLogger(__name__) logger.info("Starting model load...") try: # Import heavy libraries only when needed import torch from transformers import AutoModelForCausalLM, AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("/app/models/my-model") model_obj = AutoModelForCausalLM.from_pretrained( "/app/models/my-model", torch_dtype=torch.float16, device_map="auto" ) model = {"tokenizer": tokenizer, "model": model_obj} model_loaded = True logger.info("Model loaded successfully") except Exception as e: logger.error(f"Model loading failed: {e}", exc_info=True) raise @app.get("/health/startup") def startup_probe(): # Returns 200 immediately — the container is alive but model may still be loading return {"status": "starting"} @app.get("/health/live") def liveness_probe(): # Returns 200 as long as the process has not entered a broken state return {"status": "alive"} @app.get("/health/ready") def readiness_probe(): # Returns 200 ONLY when the model is fully loaded if not model_loaded: raise HTTPException(status_code=503, detail="Model not yet loaded") return {"status": "ready"} Step 3 — Pre-download model weights into the container image at build time. Downloading models at container startup is a significant reliability and performance risk. If the Hugging Face Hub or your storage account is temporarily unreachable, the container cannot start. Instead, bake the model weights directly into the image or use a separate initialization Container App Job to pre-populate a persistent volume: # Option A: Bake the model into the image (simple, but creates a large image) FROM python:3.11-slim WORKDIR /app RUN pip install transformers torch --index-url https://download.pytorch.org/whl/cpu # Download model at build time RUN python -c "from transformers import AutoTokenizer, AutoModel; AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2', cache_dir='/app/models'); AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2', cache_dir='/app/models')" COPY . . CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"] # Option B: Pre-populate an Azure Files volume using a Container App Job az containerapp job create --name model-downloader-job --resource-group my-rg --environment my-aca-env --trigger-type Manual --replica-timeout 600 --image python:3.11-slim --command "bash" --args "-c" "pip install huggingface_hub && huggingface-cli download my-org/my-model --local-dir /mnt/models" --volume-mounts "model-storage:/mnt/models" --volumes "model-storage:azureFile:my-fileshare" Scenario 2: The Model Server Runs Out of Memory and Gets OOM-Killed What You See Your model server starts successfully in development with 8 GB of RAM. In Azure Container Apps with a 4.0 vCPU / 8.0 Gi configuration (the maximum without GPU support), it crashes intermittently. The container restarts with no error in the application logs. When you check the system logs, you see the container exit code is `137`. Exit code 137 indicates the process was killed by the kernel OOM killer. Or in Log Analytics: ContainerAppSystemLogs_CL | where ContainerAppName_s == "my-model-server" | where Log_s contains "OOMKilled" or Log_s contains "137" | project TimeGenerated, Log_s Why This Happens Exit code 137 means the container was sent SIGKILL (`128 + 9 = 137`) because it exceeded its memory limit. Container Apps enforces memory limits strictly. When the process tries to allocate more memory than the container is allowed, the Linux kernel's OOM (Out Of Memory) killer terminates the process. With ML models, memory usage is not constant. A model might use 4 GB at rest but spike to 7 GB during inference when the input batch is large, when attention maps are being computed, or when the model is warming up its KV cache. If your container limit is 8 Gi and the model uses 7 Gi at rest plus 2 Gi during inference, you will hit OOM under load even if the container "usually" has enough memory. Step-by-Step Fix Step 1 — Quantize your model to reduce its memory footprint. Full-precision (FP32) models use twice the memory of half-precision (FP16) models, and INT8 quantized models use half the memory of FP16. For inference workloads where a small accuracy trade-off is acceptable, quantization is the single most impactful optimization you can make: import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig # 4-bit quantization — reduces a 7B parameter model from ~14GB to ~4GB quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True ) model = AutoModelForCausalLM.from_pretrained( "my-model-path", quantization_config=quantization_config, device_map="auto" ) Step 2 — Implement request batching to flatten memory spikes. Processing one request at a time with a large model creates memory spikes every time a new request arrives. Batching requests together means the model stays in a steady memory state rather than spiking per request: from asyncio import Queue, wait_for import asyncio class BatchedInferenceService: def __init__(self, model, tokenizer, max_batch_size=4, timeout=0.05): self.model = model self.tokenizer = tokenizer self.max_batch_size = max_batch_size self.timeout = timeout self.queue = Queue() async def infer(self, text: str) -> str: future = asyncio.get_event_loop().create_future() await self.queue.put((text, future)) return await future async def process_batches(self): while True: batch = [] futures = [] # Wait for the first item text, future = await self.queue.get() batch.append(text) futures.append(future) # Try to collect more items within the timeout window try: while len(batch) < self.max_batch_size: text, future = await wait_for(self.queue.get(), timeout=self.timeout) batch.append(text) futures.append(future) except asyncio.TimeoutError: pass # Process the whole batch at once try: results = self._run_inference_batch(batch) for future, result in zip(futures, results): future.set_result(result) except Exception as e: for future in futures: future.set_exception(e) def _run_inference_batch(self, texts): with torch.no_grad(): inputs = self.tokenizer(texts, return_tensors="pt", padding=True, truncation=True) outputs = self.model(**inputs) return outputs.logits.tolist() Step 3 — Set memory limits explicitly in your Container App to match what you expect at peak. Do not let the container use all available node memory and get killed unexpectedly. Set explicit limits that match your model's peak usage: # For a model that uses 5.5 GB at peak, allocate 6 GB az containerapp update --name my-model-server --resource-group my-rg --cpu 2.0 --memory 4.0Gi Step 4 — Add memory monitoring to your health endpoint. Your readiness probe should check available memory and return 503 if memory pressure is too high, which will cause the load balancer to route traffic to other healthy replicas: import psutil @app.get("/health/ready") def readiness_probe(): if not model_loaded: raise HTTPException(status_code=503, detail="Model not yet loaded") memory = psutil.virtual_memory() if memory.percent > 90: raise HTTPException( status_code=503, detail=f"Memory pressure too high: {memory.percent:.1f}% used" ) return { "status": "ready", "memory_percent": memory.percent, "available_gb": memory.available / (1024**3) } Scenario 3: GPU Workloads Fail to Initialize What You See You deploy a container that uses PyTorch with CUDA support. The container starts, but you see errors like: CUDA error: no kernel image is available for execution on the device torch.cuda.is_available() returned False RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB Or the model silently falls back to CPU without telling you, and your inference times are 50x slower than expected. Why This Happens Azure Container Apps supports GPU workloads through a dedicated GPU consumption plan and specialized GPU-enabled environments. If you deploy a CUDA-enabled container to a standard Container Apps environment (which uses CPU-only nodes), `torch.cuda.is_available()` returns `False` and PyTorch either errors out or silently falls back to CPU depending on how your code handles it. Even when you are in a GPU-enabled environment, CUDA version mismatches between the CUDA toolkit installed in your container image and the CUDA drivers on the host node will produce the "no kernel image" error. Step-by-Step Fix Step 1 — Detect whether GPU is available and log it explicitly at startup. Never assume — always log the device your model is using: import torch import logging logger = logging.getLogger(__name__) def initialize_device(): if torch.cuda.is_available(): device = torch.device("cuda") logger.info(f"Using GPU: {torch.cuda.get_device_name(0)}") logger.info(f"CUDA version: {torch.version.cuda}") logger.info(f"Available GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB") else: device = torch.device("cpu") logger.warning("GPU not available, falling back to CPU. Inference will be slower.") return device device = initialize_device() model = model.to(device) Step 2 — Match your CUDA toolkit version to the host driver. The CUDA toolkit version in your image must be less than or equal to the CUDA driver version on the host. Use the official PyTorch images which bundle compatible CUDA versions: # Use PyTorch's official image with a specific CUDA version # Check Azure Container Apps GPU documentation for supported CUDA versions FROM pytorch/pytorch:2.1.0-cuda11.8-cudnn8-runtime WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"] Step 3 — Create your Container Apps environment with GPU support enabled. GPU support requires a dedicated GPU workload profile in your Container Apps environment: # Create an environment with GPU workload profil az containerapp env create --name my-gpu-env --resource-group my-rg --location eastus --workload-profile-type "NC24-A100" --workload-profile-name "gpu-profile" # Deploy your app to the GPU profile az containerapp create --name my-model-server --resource-group my-rg --environment my-gpu-env --image myregistry.azurecr.io/my-model-server:cuda11.8 --cpu 4.0 --memory 16.0Gi --workload-profile-name "gpu-profile" --min-replicas 1 Step 4 — Handle the CPU fallback gracefully so you know when it happens. If you want your application to work on both GPU and CPU environments, implement a clean fallback that makes the degraded state visible in monitoring: import os import torch class ModelConfig: def __init__(self): self.force_cpu = os.environ.get("FORCE_CPU", "false").lower() == "true" self.device = self._select_device() def _select_device(self): if self.force_cpu: return torch.device("cpu") if torch.cuda.is_available(): return torch.device("cuda") # GPU was expected but not found — emit a warning metric import warnings warnings.warn( "CUDA requested but not available. Running on CPU. " "Inference latency will be significantly higher.", RuntimeWarning ) # Optionally: push a custom metric to Azure Monitor here return torch.device("cpu") @property def is_gpu(self): return self.device.type == "cuda" Scenario 4: LangChain / AI Agent Timeouts at Startup What You See You deploy a LangChain-based agent or a RAG (Retrieval-Augmented Generation) pipeline to Container Apps. During startup, the application connects to Azure OpenAI, loads an embedding model, and populates an in-memory vector store. But the readiness probe times out while the vector store is being populated from a large document set. Why This Happens LangChain applications often do expensive work at startup — embedding thousands of documents, pre-populating vector indexes, or loading conversation history from a database. This work happens synchronously in many LangChain components, blocking the main thread and preventing the HTTP server from responding to health probes. Step-by-Step Fix Step 1 — Move initialization work to a background task that runs after the HTTP server is up from fastapi import FastAPI from contextlib import asynccontextmanager import asyncio import logging logger = logging.getLogger(__name__) vector_store = None initialization_complete = False initialization_error = None @asynccontextmanager async def lifespan(app: FastAPI): global initialization_complete, initialization_error # Start the heavy initialization in the background task = asyncio.create_task(initialize_vector_store()) yield # Server starts and begins serving health probe requests # Cleanup task.cancel() app = FastAPI(lifespan=lifespan) async def initialize_vector_store(): global vector_store, initialization_complete, initialization_error try: logger.info("Starting vector store initialization...") # Run CPU-bound work in a thread pool so we don't block the event loop loop = asyncio.get_event_loop() vector_store = await loop.run_in_executor(None, _build_vector_store) initialization_complete = True logger.info("Vector store initialization complete") except Exception as e: initialization_error = str(e) logger.error(f"Vector store initialization failed: {e}", exc_info=True) def _build_vector_store(): from langchain_community.vectorstores import FAISS from langchain_openai import AzureOpenAIEmbeddings from langchain_community.document_loaders import DirectoryLoader loader = DirectoryLoader("/app/documents") documents = loader.load() embeddings = AzureOpenAIEmbeddings( azure_deployment=os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"], azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], api_key=os.environ["AZURE_OPENAI_API_KEY"] ) return FAISS.from_documents(documents, embeddings) @app.get("/health/ready") def readiness(): if initialization_error: raise HTTPException(status_code=500, detail=f"Initialization failed: {initialization_error}") if not initialization_complete: raise HTTPException(status_code=503, detail="Initializing vector store, please wait...") return {"status": "ready"} Step 2 — Use Azure AI Search instead of an in-memory vector store for large document sets. In-memory vector stores like FAISS are fine for development but become a liability in production Container Apps because the index is lost every time the container restarts, and rebuilding it adds minutes to your startup time. Azure AI Search persists the index and provides near-instant load times: from langchain_community.vectorstores.azuresearch import AzureSearch from langchain_openai import AzureOpenAIEmbeddings embeddings = AzureOpenAIEmbeddings( azure_deployment=os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"], azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], api_key=os.environ["AZURE_OPENAI_API_KEY"] ) # The index already exists in Azure AI Search — no rebuild needed on startup vector_store = AzureSearch( azure_search_endpoint=os.environ["AZURE_SEARCH_ENDPOINT"], azure_search_key=os.environ["AZURE_SEARCH_KEY"], index_name="my-document-index", embedding_function=embeddings.embed_query ) # This is nearly instantaneous — just connects to the existing index initialization_complete = True Debugging AI Workload Logs in Log Analytics When something goes wrong with your AI workload, these Log Analytics queries will help you quickly identify the pattern: // Find all OOM kills in the last 24 hours ContainerAppSystemLogs_CL | where TimeGenerated > ago(24h) | where Log_s contains "OOMKilled" or ExitCode_d == 137 | project TimeGenerated, ContainerAppName_s, Log_s, ExitCode_d | order by TimeGenerated desc // Track model loading time across restarts ContainerAppConsoleLogs_CL | where ContainerAppName_s == "my-model-server" | where Log_s contains "Model loaded" or Log_s contains "Starting model load" | project TimeGenerated, Log_s, ContainerName_s | order by TimeGenerated asc // Find slow inference requests (if you are logging inference latency) ContainerAppConsoleLogs_CL | where ContainerAppName_s == "my-model-server" | where Log_s contains "inference_latency_ms" | extend latency = toint(extract("inference_latency_ms=([0-9]+)", 1, Log_s)) | where latency > 5000 // Requests taking more than 5 seconds | project TimeGenerated, latency, Log_s | order by latency desc Summary: AI Workload Startup Checklist When your AI workload fails to start or behaves unexpectedly, work through this checklist: - Is the model loading time covered by an appropriate startup probe `failureThreshold`? - Are model weights baked into the image or pre-loaded onto a mounted volume, rather than downloaded at runtime? - Is the container's memory limit large enough for peak inference load (model size + activation memory)? - Have you verified whether the workload is running on GPU or CPU and logged that explicitly at startup? - Does the CUDA version in your image match or precede the driver version on the host? - For LangChain or RAG workloads, does initialization happen in the background so health probes can respond? - Are you using a persistent vector store (Azure AI Search, Azure Cosmos DB) instead of rebuilding in-memory indexes on every restart? References and Sample Resources Use these links alongside the scenarios above to go deeper on configuration details and production patterns. Azure Container Apps docs (core) Health probes in Azure Container Apps Workload profiles overview (Consumption, Dedicated, Memory-optimized, GPU) Serverless GPU overview and supported regions Azure Container Apps Jobs (for preloading models or maintenance tasks) Monitoring and logging options in Azure Container Apps AI-specific docs AI integration with Azure Container Apps LangChain tutorial with Container Apps dynamic sessions Azure AI Search vector search overview Azure OpenAI with Python quickstart Official sample repositories Chat app with Azure OpenAI on Container Apps AI MCP server sample on Container Apps .NET MCP agent app (client/server pattern on Container Apps) TypeScript MCP container sample Python code interpreter dynamic session sample ML runtime and model-loading references PyTorch CUDA notes and best practices Hugging Face Transformers loading models from local/Hub paths bitsandbytes quantization guide What's Next In the final part of this series, we step back from reactive troubleshooting and look at how to build a proactive observability and automation layer so that you catch these problems before your users do — or better yet, have them resolve themselves automatically. Part of the series: Troubleshooting Azure Container Apps in Production Next: Part 4 — Eyes Open, Hands Free: Automating Observability, Alerts, and Self-Healing Diagnostics for Azure Container AppsWe Gave Ourselves 20 Minutes to Build an AI Agent for a Lumber Company. The Timer's Still on Screen.
Here's a confession: most "build with AI" webinars are 60 minutes of slides, 5 minutes of a polished demo someone rehearsed for a week, and a closing CTA. You leave inspired but not really sure what you saw. So we tried something different. We put a visible countdown timer on the screen and gave ourselves 20 minutes to do two things, live: Build an AI agent that solves a real business problem Deploy a working AI application to Azure No edits to hide the awkward parts. No "and here's one I prepared earlier." Just the timer, the screen, and a working app at the end. The on-demand recording is up now. Here's what's in it and why you should carve out 20 minutes for it this week. The setup: why lumber? 🏘️ We needed a real business problem, not a toy one. So for the demo, we role-play as the owner of Contoso Lumber — a regional lumber business with a very specific, very real headache: Should we sell our inventory now, or hold it longer? Sell too early, miss a better price. Hold too long, eat storage costs. Lumber prices fluctuate with global competition, macro shifts, even the weather. In the past, decisions like this came from morning meetings and gut instinct, or maybe the occasional ad-hoc spreadsheet that nobody could reuse a month later. It's the kind of decision that should have an analyst behind it — except most growing businesses can't afford to hire one full-time. So we build the AI agent that does. (Yes, lumber. We know. Stick with us — the boring industry is exactly the point. If it works here, it works for your business too.) What we actually build (in 20 minutes flat) The webinar walks through the entire flow, end to end: Part 1 — The agent. We open Microsoft Foundry at ai.azure.com, browse the model leaderboard (there are over 11,000 models to choose from — we compare a few on the cost-vs-quality chart), pick one, write a plain-English instruction for the agent, upload a CSV of historical lumber pricing, and ask it a real question: "If I cannot sell one of my products today unless I offer my clients a 35% discount, and knowing the historical pricing data, should I still sell it?" The agent runs a break-even analysis and comes back with a reasoned recommendation — hold for 3–6 months, here's the math on why, here's where storage costs start eating the upside. Then we add voice mode (now you can ask the agent for pricing recs from a coffee shop on your phone), and lock down guardrails to block jailbreaks, prompt injection, data leakage, and — because we're feeling fancy — profanity in responses. Part 2 — The app. With the agent done, we pivot to deploying a full AI chat application to Azure. From scratch. Using exactly five commands in Azure Cloud Shell: azd auth login git clone <repo> cd <folder> azd up azd down # (this one's for when you're done — kills everything to avoid surprise bills) That's it. The template handles the Container Apps setup, the architecture-aligned-to-Well-Architected-Framework stuff, all the boilerplate that usually eats half a sprint. By the end of the segment, there's a working AI chatbot running on a real Azure URL. We even pause the timer when we're just explaining things, so you know the 20-minute clock is honest about build time, not talk time. Why this format is more useful than another slide deck A few things this webinar shows that a written tutorial can't: The Foundry UI is super navigable. You watch someone do it. You see where the buttons are. You see what the leaderboard looks like when you're comparing GPT-5.3 Codex against Kimi K2.5 on a cost-to-quality chart. (Spoiler: Kimi wins this particular trio. Your mileage will vary depending on your workload.) The "no-stitching" claim is real. Models, data, agents, guardrails, deployment — all in one place. You don't need to leave Foundry to wire seven products together. The webinar makes that concrete by showing you the actual flow without cutting. Five commands really is five commands. This is the part people are most skeptical about until they see it. azd up does the work. The infrastructure provisioning, the container app, the AI service hookup — all of it. You can delete it just as fast. azd down tears everything back down. Useful when you're experimenting and don't want a $40 surprise on your Azure bill next month. What's on screen at the end By the 20-minute mark: A published AI agent named for the lumber business, with guardrails, voice mode enabled, ready to be called from Teams, Microsoft 365 Copilot, or any application via endpoint A separate AI chat application deployed to Azure Container Apps, with a live URL Logs, observability, the full Foundry control plane — all available out of the box And in the closing minutes, four very concrete next steps for what you do next if this sparked an idea for your own business — including Azure Accelerate (if you want Microsoft experts in the room with you), the partner network, and the Microsoft marketplace if you'd rather buy than build. Watch the recording The on-demand recording is available now. Block 20 minutes — that's literally all it takes — and ideally watch with your Azure portal open in another tab so you can follow along. If you're the kind of person who learns by doing, pause at the agent-building section and try it yourself in parallel. Foundry is free to explore; the agent we build in the webinar costs cents to run. → Watch the on-demand webinar A few things we'd love feedback on If you watch it, we'd genuinely love to know: Did the timer help or distract? (We thought it would feel gimmicky. It turned out to be the most-mentioned thing in early feedback.) What use case from your business would you want to see in the next one? We're picking the next demo problem from comments. Was the lumber thing weirdly compelling or were you just here for the Azure parts? Drop a comment, tag us, or grab a partner and try building your own version this week. The timer's reset. Your 20 minutes start whenever you press play. Want to go deeper than the webinar? Two companion reads: From Idea to Impact: How Growing Businesses Scale with Azure (five real customer stories with the full architectures) and AI Made Simple: 3 Practical Moves for Growing Businesses (the structured playbook for figuring out what to build first).
