Evaluating Agentic AI in Microsoft Foundry: Beyond Final-Answer Quality
As AI systems evolve from single-turn assistants into tool-using agents, evaluation has to evolve with them. For a traditional LLM application, teams often focus on whether the final response is relevant, coherent, or grounded. For an agent, that is only part of the problem. An agent may produce a plausible answer while still selecting the wrong tool, sending incorrect tool parameters, ignoring tool output, violating user constraints, making unnecessary calls, or failing to complete the task. This is why Microsoft Foundry separates agent evaluation into multiple layers and supports both system-level and process-level evaluation. Microsoft describes agent evaluators as a way to systematically assess quality, safety, and performance across agent workflows—not only the final response. https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/evaluate-agent Start with the evaluation target Foundry allows evaluations against different targets: Agent — run the selected agent against test inputs and evaluate newly generated behavior Model — evaluate a model directly Dataset — score outputs that already exist in a database This distinction matters because the evaluation workflow changes depending on the target. When the target is an Agent, Foundry generates a fresh response for each input and evaluates that result. When the target is a Dataset, Foundry evaluates the responses already present in the dataset. https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app For agentic systems, I find it useful to evaluate in two scopes: Individual turns for detailed debugging of tool use and response behavior. Full conversations for multi-turn task completion, conversational coherence, and user satisfaction. Microsoft currently recommends starting with full conversations and simulated data for controlled testing, then using real conversations in production. System evaluation: did the agent actually accomplish the task? The first layer is the end-to-end outcome. Important evaluators include: Task Completion Did the agent actually complete the user’s task? For example: “Find black running shoes under $120 in size 9 in Dallas.” An agent may correctly search for products but stop before checking inventory. The response may look reasonable, but the task is still incomplete. Task Adherence Did the agent follow its instructions, policies, and explicit user constraints? If the customer says the budget is $120 and the agent recommends a $129 product without clearly identifying the budget violation, adherence should suffer. Intent Resolution Did the agent correctly understand and address what the user wanted? A tool call can be technically valid but semantically wrong. Searching generic sneakers when the user explicitly requested running shoes is a good example. Customer Satisfaction This looks beyond technical correctness and asks whether the overall interaction is likely to satisfy the user. These metrics answer a simple but important question: Did the system actually solve the user’s problem? Microsoft groups these kinds of measurements under system evaluation for agent workflows. Process evaluation: did the agent take the right path? This is where agent evaluation becomes much more interesting. Foundry includes dedicated process evaluators for tool-using agents. Tool Selection This checks whether the agent chose the appropriate tool. Suppose an agent has: search_products check_inventory get_promotions web search If the user asks: “Is DailyRun X available in size 9 in Dallas?” The right enterprise action is to use the inventory tool. If the agent uses web search instead, the answer may still sound plausible, but the process is wrong. Tool Input Accuracy This checks whether the tool was called with the correct inputs. For example: check_inventory( product_id="DEMO-SHOE-002", size="9", location="Dallas-TX" ) The agent may choose the right tool but pass the wrong size or location. That means Tool Selection can pass while Tool Input Accuracy fails. Tool Call Success This evaluates whether the invocation itself succeeded. For example: the API returned successfully there was no timeout there was no execution failure But a successful tool call does not mean it was the correct call. An API may return HTTP 200 for the wrong SKU or wrong location. So: Tool Call Success measures execution reliability, not semantic correctness. Tool Output Utilization This checks whether the agent correctly used what the tool returned. Imagine the inventory tool returns: { "available": false } but the agent replies: “Yes, size 9 is available.” The tool worked. The input may even have been correct. The failure is in how the agent used the result. Tool Call Accuracy This gives a broader signal about whether tool invocation behavior was correct overall. Together, these evaluators answer: Did the agent execute the right workflow, not just produce a convincing answer? Response quality still matters Even when the process is correct, the final response can still be poor. Foundry supports quality evaluators such as: Relevance Groundedness Completeness Coherence Fluency Groundedness Groundedness asks whether claims are supported by the available context or evidence. For enterprise applications, this is especially important because answers should be based on authoritative sources rather than unsupported model memory. Completeness Did the agent answer all important parts of the request? If the user asks: “Is this available and is there a member discount?” an answer that checks inventory but ignores the promotion question is incomplete. Coherence and Fluency These evaluate communication quality. Microsoft defines coherence around logical and orderly presentation of ideas, while fluency focuses on readability, grammar, vocabulary, and clarity. https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/general-purpose-evaluators Built-in evaluators are not enough for every enterprise agent A retail agent, finance agent, claims agent, or operations agent will always have business-specific requirements that generic evaluators cannot fully capture. Microsoft’s current guidance recommends using a rubric evaluator as a primary measure when teams need to express application-specific criteria such as policy enforcement, tool-use accuracy, or communication standards. Built-in evaluators can then be layered on top for broader coverage. A retail-agent rubric might include: never claim availability without inventory verification never invent discounts never silently violate a stated budget prefer enterprise catalog tools over public search avoid duplicate or unnecessary tool calls clearly identify compromises when no exact match exists This is where evaluation starts to look much more like business acceptance testing. https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rubric-evaluators The dataset becomes a regression suite Foundry evaluation datasets are reusable test collections. They are useful when comparing: Prompt V1 vs V2 one model vs another tool-definition changes orchestration changes release candidate vs production version When Foundry evaluates an agent against a dataset, it generates a new response for each input and scores it. When the target is the live agent, Foundry ignores existing responses in the dataset. This changes prompt engineering from: “This prompt looks better.” to: “This version performs better on the same regression suite.” That is a much more defensible engineering practice. Golden datasets, synthetic data, and production traces all serve different purposes A mature evaluation strategy uses multiple data sources. Golden datasets: Best for critical known scenarios and deterministic regression testing. Synthetic data: Useful for expanding coverage and generating edge cases before enough production traffic exists. Full-conversation simulation: Useful for testing multi-turn user journeys and end-to-end task behavior. Production traces: Useful for evaluating what real users actually experienced. Foundry supports evaluation directly from Application Insights traces and can evaluate deployed interactions without replaying the original request. https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/cloud-evaluation-deployed-interactions Production evaluation should operate on traces Production failures are often different from development failures. Real users introduce: unexpected phrasing missing information contradictory constraints unusual tool sequences edge cases that synthetic test data may not cover Foundry can evaluate traces already captured in Application Insights. The trace-evaluation workflow supports selecting traces by trace ID or by agent filter, and also supports intelligent sampling to select a representative subset rather than evaluating every interaction. That creates a strong operational loop: Run → Trace → Evaluate → Identify failures → Add to regression suite → Fix → Re-evaluate Microsoft Foundry’s evaluation stack becomes especially useful as teams can evaluate not only what an agent said, but also how it behaved operationally, which tools it used, whether it completed the task, and how that behavior changes over time.Adding a Fallback Model to Hermes with Microsoft Foundry
So the plan was simple. Leave the Bedrock configuration untouched, then wire Microsoft Foundry in behind it as a fallback, so Hermes always has somewhere else to go when the primary provider is not responding. A few other reasons pushed me towards Foundry in particular: Redundancy that does not need me. If Bedrock is throttled or out of quota, I want Hermes to fail over on its own rather than waiting for me to notice. A catalogue I already pay for. Foundry puts the latest GPT models next to open-weight and partner models in one place, so I can pick a model that suits the task instead of settling for whatever a single provider happens to offer. Enterprise controls out of the box. Region pinning, private networking, content filters and per-deployment quota all sit in the same portal, which makes the setup far easier to defend to a security reviewer. Learning the mechanics before I need them. Working out how Hermes handles a provider chain is much nicer on a quiet Tuesday than during a live outage. Here is the short version, if you are deciding whether to read on. Time: about thirty minutes if nothing goes wrong. Cost: pay-as-you-go tokens only, and none at all while the fallback sits idle. Result: an assistant that keeps answering when your primary provider stops. Before you start, you will need three things: a machine with Hermes already installed and a working primary provider configured, an Azure subscription with access to Microsoft Foundry in a region you can actually deploy into, and enough quota in that region to create a deployment. One thing that made this easy to justify: Foundry deployments bill per token on the standard pay-as-you-go tier. A fallback provider that never gets invoked costs nothing beyond the requests it actually serves, so the insurance is close to free until the day you need it. Chat surface (CLI, messaging) → Hermes Gateway → Primary Amazon Bedrock → Fallback Microsoft Foundry Figure 1: Where the fallback sits. Every request goes through the Hermes gateway to the primary provider; only when that provider is unavailable does the chain continue to Microsoft Foundry. Part 1: Deploying a Model on Microsoft Foundry The first half of this job happens entirely inside the Microsoft Foundry portal and has nothing to do with Hermes yet. All you are really doing here is making sure your Azure subscription can serve a model, and that you hold an endpoint and key Hermes can authenticate with later. 1. Deploy model in Foundry → 2. Copy endpoint + key → 3. hermes fallback add → 4. Authenticate → 5. Select models, test Figure 2: The whole setup in five moves. The first two happen in the Microsoft Foundry portal (orange); the rest happen on the Hermes machine (blue). Go to Microsoft Foundry > Build > Models > Deploy > Deploy a base model. You can deploy a fine-tuned model instead if you already have one, which works just as well with Hermes. Check the region shown at the top of the portal before you commit, because both model availability and deployment quota differ from one region to the next. Then deploy the model you have selected: In this case I deployed gpt-5.6-sol, which is the model Hermes will fall back to. The choice was deliberate rather than exciting. My primary model on Bedrock is a general-purpose chat model, and a fallback is only useful if the answers it gives feel like a continuation of the same conversation rather than a different assistant wearing the same name. The gpt-5.6-sol deployment matches that behaviour closely, it was available in the region I wanted to pin, and the quota I was granted comfortably covers a day of normal use. If a fallback surprises you the first time it fires, it is the wrong fallback. Once the deployment finishes, open it and take note of two values: the target endpoint URI and the API key. Copy both somewhere safe now, because you will be pasting them into Hermes in the next part. If your organisation rotates keys on a schedule, use a key with the longest life you are allowed, since a fallback secured with a credential that expires quietly stops being a fallback. What to copy Where it lives in the portal Where Hermes asks for it Target endpoint URI Deployment > Endpoint > Target URI "Endpoint" prompt in hermes fallback add API key Deployment > Endpoint > Key "API key" prompt, or choose Entra ID instead Deployment name Deployment > Details > Name Shown in the model list Hermes returns Region Top of the portal, next to the resource Must match the region you deployed into Figure 3: Everything Hermes will ask for, and where to find each value before you leave the portal. Part 2: Adding Foundry to Hermes as a Fallback With the Foundry side sorted, everything from here happens in the Hermes CLI. One thing worth knowing before you start: this is the fallback command, not the primary model command, so your existing Bedrock configuration is left completely alone. Nothing in this section can break what is already working, which makes it a good one to try on a live setup. Run the Hermes fallback command: When Hermes asks which provider to add, choose Azure Foundry. The picker still carries the old name; it is the same service that the portal now calls Microsoft Foundry. Paste the target endpoint URI from your deployment, then authenticate with the API key you copied earlier. Hermes also offers Microsoft Entra ID at this prompt, which is the better option if your organisation would rather not have a static key sitting on the machine. If authentication fails here, check the endpoint before you start suspecting the key. In my experience the endpoint is wrong far more often than the credential is, usually because the deployment name at the end of the URI does not match the deployment you actually created. Once authentication succeeds, Hermes lists the deployments your Foundry resource exposes and asks which ones you want to use. You can select more than one, and the order is not cosmetic: Hermes walks down the chain from top to bottom whenever the provider above is unavailable. Treat that list as a priority order, not a shopping basket. What happens to the primary What Hermes does What you see in the chat Responds normally Routes every request to the primary and never touches the chain Nothing. The fallback stays idle Throttled or out of quota Retries the next provider down the chain on the same request A reply, served by the fallback model Endpoint unreachable Keeps failing over on each new request until the primary recovers Slightly different tone and latency, but a working assistant Every provider fails Returns the error rather than hanging An error worth chasing with hermes status Figure 4: The chain in practice. The fallback only earns its keep in the middle two rows, which is exactly why it is easy to forget you configured it. Part 3: Promoting Foundry to the Primary Model At this stage Foundry is sitting in the back seat as a backup. I wanted to reverse the arrangement and make Foundry the primary while Bedrock slides down into the fallback slot, partly because I preferred keeping day-to-day traffic inside my Azure subscription, and partly because I wanted proof the chain works in both directions. Before promotion After promotion Primary: Amazon Bedrock → Primary: Microsoft Foundry Fallback: Microsoft Foundry → Fallback: Amazon Bedrock Figure 5: The promotion, in effect. Nothing is added or removed; the two providers simply trade places in the chain. There is no dedicated "promote" command in Hermes, so the manual route is a short sequence of steps rather than a single instruction: Select the fallback provider/model as the new primary: hermes model Remove the now-duplicate model from the fallback chain: hermes fallback remove Optionally add the old primary model as a fallback: hermes fallback add Restart the messaging gateway: hermes gateway restart Verify the result: hermes status / hermes fallback list That sequence works, and it is good to know what is happening underneath. But since I already had a working provider configured, I would rather just ask Hermes to rearrange itself. This is the part I genuinely enjoy about the tool: the configuration is something you can talk to, not only something you type commands at. Prompt: Okay now please make the model I configured on Microsoft Foundry into the main model, and make the Bedrock one the fallback model! Hermes rewrites the provider chain on its own and confirms the swap once it is done, which is a great deal less error-prone than running the five commands by hand. Part 4: Testing the Switch Configuration you have not tested is just an assumption with extra steps, so the next thing is to confirm Hermes really is talking to Foundry. Type /model when running Hermes to bring up the model picker. You will be prompted to select a provider first. Pick the Microsoft Foundry entry, then choose the specific deployment from the list underneath it. The active model should switch straight away. Send it a plain "Hello" to check that the deployment actually responds, rather than just looking correct in the menu. Appearing in a dropdown and serving a request are two very different things. A second test is worth the thirty seconds it costs: run hermes status to confirm which provider is live, then hermes fallback list to confirm the chain is ordered the way you intended. The picker tells you what you selected; those two commands tell you what Hermes will actually do at three in the morning. Part 5: The Obstacle, and What It Actually Taught Me Every walkthrough has the part the author quietly leaves out. Here is mine: the wrinkle was not the model, it was capacity. My first attempt deployed into the region closest to me out of habit, and the portal turned it down because there was no capacity left for that model at the tier I asked for. The model was clearly listed in the catalogue; being listed and being deployable in your region, on your subscription, at your quota, are three separate questions. Redeploying in a different region fixed it in a couple of minutes, but it meant the endpoint URI changed, which in turn meant the value I had already pasted into Hermes was stale. Re-running hermes fallback add against the new endpoint sorted it out. The lesson is cheap enough to hand over for free: check quota and regional capacity for your specific subscription before you design a walkthrough, a demo or a production fallback around one deployment. In the Foundry portal, Management then Quota shows exactly what you have been granted per region and per model family, which is the only list that matters. Symptom Likely cause Fix Deployment rejected in the portal No capacity for that model at the tier you asked for, in that region Deploy in another region, or drop to a smaller tier Hermes rejects the credential Endpoint URI does not match the deployment you created Re-copy the target URI from the deployment, not the resource Provider authenticates but lists nothing Key belongs to a different Foundry resource Check you are in the right resource, then re-run hermes fallback add Fallback never fires Chain ordered the wrong way round hermes fallback list, then reorder Figure 6: The four things that went wrong, or nearly did, and what fixed each one. There is a silver lining worth stating plainly. Because the fallback chain was already in place, a deployment that refused to come up did not take the assistant down with it. That is precisely the scenario this whole exercise was meant to cover, and it turned up on day one without me having to simulate it. Command Cheat Sheet Everything used in this walkthrough, collected in one place: hermes fallback add: attach a provider to the fallback chain hermes fallback remove: drop a provider from the chain hermes fallback list: show the chain in priority order hermes model: set the primary model hermes gateway restart: restart the messaging gateway after a change hermes status: confirm which provider is currently live /model: switch models from inside a running session Conclusion Adding Microsoft Foundry as a fallback behind my existing Bedrock setup took an afternoon, and most of that was spent recovering from a regional capacity limit I should have checked first. The work itself is small: deploy a model, copy the endpoint and key, run hermes fallback add, authenticate, pick your deployments. The payoff is that Hermes no longer depends on one provider staying healthy. Three things are worth carrying away from this: Check quota, not just the catalogue. The Foundry catalogue shows what Microsoft offers. It does not show what your subscription and region can actually deploy today. Confirm that first, before you build anything on top of a specific deployment. Order your fallback chain deliberately. Hermes works down the list from top to bottom, so the sequence you choose during setup is the failover policy you are going to live with. Put the model you actually trust at the top. Treat the endpoint as part of the credential. Redeploying in a new region changes the endpoint URI, and a fallback pointed at an endpoint that no longer exists is not a fallback. Re-run the setup whenever the deployment moves. The switch from Bedrock primary to Foundry primary also proved the chain runs in both directions, which is the real point. Provider redundancy is only useful if you have watched it work. Next on my list is deliberately breaking the primary provider to confirm the failover triggers on its own, without me typing a single command. If you run this against a different model, region or provider pairing, I would genuinely like to know how it went, particularly if your quota experience was better than mine. Drop it in the comments.
Microsoft Foundry External MCP Server Traffic Routing via Corporate Firewall
The customer would like to confirm whether traffic from an Azure AI Foundry agent to an external MCP server can be routed through a corporate firewall and whether this scenario is officially supported. To validate this scenario, I configured the following in my lab: Deployed an Azure AI Foundry resource using the Standard Agent Service with network injection. Created a dedicated subnet for the Foundry Agent Service and delegated it to Microsoft.App/environments. Associated a route table with the Foundry Agent subnet to route outbound traffic through Azure Firewall. Configured the required application and network rules on Azure Firewall. The Foundry agent is able to successfully retrieve data from the external MCP server. However, no corresponding traffic is visible in the Azure Firewall logs. Could you please confirm whether outbound traffic from the Foundry agent to an external MCP server can be routed through Azure Firewall or a corporate firewall? Also, is this routing scenario officially supported? Appreciate your support!Building an Event-Driven AI HelpDesk on Azure (with Zero API Keys)
Building AI agents is one thing, but deploying them securely at enterprise scale is another challenge. If you’re still relying on hardcoded API keys to connect your AI services, it’s time to move on. Join a live, in-depth demo of HelpDesk Copilot—an open-source, event-driven AI service desk built entirely on Azure with zero API keys. This session will cover the foundry-ticketing architecture, showing how to orchestrate autonomous AI agents using cloud-native patterns, combining Microsoft Foundry Agents’ conversational intelligence with Dapr’s event-routing, all running on Azure Container Apps. You’ll also learn about the “Zero API Key” security model using Managed Identities and Azure RBAC, and pick up best practices for serverless, scale-to-zero cloud deployments. Perfect for Cloud Architects, AI/DevOps Engineers, and Backend Developers aiming to build secure, scalable, event-driven AI apps on the Microsoft stack.The Next Generation of Agents with Azure and Microsoft Foundry
Every company has a help desk, and every help desk answers the same twenty questions over and over: my VPN keeps dropping, I lost my MFA device, I need access to the Finance share. Sound familiar? That is exactly what makes it the perfect proving ground for an AI agent — not another chat demo that just talks, but a system that answers from real documentation, knows when it is not allowed to answer, and hands off to humans through a real channel. So that is what we are building today: HelpDesk Copilot, a Contoso IT service desk where a Microsoft Foundry agent triages employee questions, answers them grounded on an IT knowledge base with citations, and — when policy demands a human — creates a ticket that flows asynchronously through Dapr and Azure Service Bus into Table Storage and an Adaptive Card in a Microsoft Teams channel. The whole thing runs on Azure Container Apps, is provisioned entirely with Terraform, and — my favorite part — contains zero API keys. Every service-to-service call, from pulling container images to invoking the Foundry agent, uses Microsoft Entra ID and managed identities. The Foundry account has local key authentication disabled outright. Here is what we will cover: The architecture: three ACA apps, one Foundry Prompt Agent, and an event-driven ticket pipeline Why I chose one agent instead of a multi-agent orchestra — and why that was the honest choice Grounded, streaming answers with Foundry File Search and citations The escalation path: Dapr pub/sub, a Service Bus topic, deterministic ticket IDs, and idempotency The identity model: five managed identities, zero connection strings Terraform notes: the Foundry provider landscape is not what you expect Observability with OpenTelemetry and Foundry's cloud evaluation API Grab a coffee — let's build! The Architecture Three container apps live inside one ACA environment, and each one has a deliberately different network posture: Frontend — React 18 + Vite served by nginx, with external ingress. This is the only public URL: the chat UI and a live ticket panel. API — FastAPI with a Dapr sidecar, internal ingress only. It runs the agent conversation loop, streams Server-Sent Events, executes tools, and publishes ticket events. Ticket worker — FastAPI with a Dapr sidecar and no ingress at all. It exists only to consume Service Bus messages via Dapr, and KEDA wakes it from zero replicas based on subscription backlog. The frontend's nginx proxies browser calls to the API over the ACA environment's internal DNS — the API is never exposed to the internet. Around the environment sit Microsoft Foundry (a Prompt Agent plus a File Search vector store), a Service Bus topic, Table Storage as the ticket read model, Key Vault holding exactly one secret, Azure Container Registry, and Application Insights on a Log Analytics workspace. One Agent, Not an Orchestra My original design called for an orchestrator agent routing to specialist agents. Reality intervened: the Connected Agents pattern I planned to use is deprecated, and the workflow orchestration alternatives are still in preview. I could have demoware'd my way around that — instead, I redesigned around one Foundry Prompt Agent with three capabilities: File Search over the Contoso IT knowledge base, for grounded answers with citations A local create_ticket function tool, for escalation A local get_ticket_status function tool, for lookup Its instructions enforce the policy: search first, cite your source, and only create a ticket when no procedure covers the problem — or when the procedure explicitly requires human intervention (Finance-share access, a lost device, all MFA methods gone). Here is the takeaway I want you to keep: a single well-instructed agent with sharp tools beats a fragile multi-agent mesh for this problem size. Multi-agent is a topology, not a virtue. When the platform's orchestration story stabilizes, this design has an obvious seam to split along — until then, one agent is simpler to reason about, cheaper to run, and easier to evaluate. Similar honesty applies to retrieval: with ten markdown documents, Azure AI Search would be architectural cosplay. Foundry's built-in File Search vector store is the right-sized tool. When the corpus grows into thousands of documents needing hybrid or semantic ranking, that is the upgrade path. The Knowledge Path: Streaming Grounded Answers An employee asks: "My VPN keeps dropping every hour." The flow: The frontend POSTs to /chat with the message and an optional conversation_id The API creates (or continues) a Foundry conversation and requests a streamed response The agent runs File Search over the IT docs and gets relevant chunks back Text deltas and file citation annotations stream back through the API as Server-Sent Events The employee watches the answer type itself out, with the source document cited beneath it Citations are not decoration. In an IT support context, "the answer came from the official VPN procedure" is the difference between a trustworthy assistant and a liability. The frontend de-duplicates cited filenames per answer and shows them inline. The heart of the API is the tool-call loop. When the agent requests a local function, the API executes it and feeds the result back into the same Foundry conversation, so the agent composes the final employee-facing message. The agent stays the author of the conversation; the API stays the executor of side effects. Here is the loop, from agent_service.py: while True: stream = openai.responses.create( input=pending_input, conversation=conversation_id, stream=True, extra_body={"agent_reference": agent_reference}, ) function_outputs = [] for chunk in stream: if chunk.type == "response.output_text.delta": yield {"event": "delta", "data": {"text": chunk.delta}} elif chunk.type == "response.output_item.done": item = chunk.item if item.type == "function_call": yield {"event": "tool_call", "data": {"name": item.name}} output = self._execute_tool(item.name, item.arguments, conversation_id) function_outputs.append({ "type": "function_call_output", "call_id": item.call_id, "output": json.dumps(output), }) if function_outputs: pending_input = function_outputs continue # submit tool outputs and let the agent finish its answer break The Escalation Path: Events, Not Awaits Now the interesting request: "I need Finance-share access for an audit." The knowledge base says restricted Finance access always requires a ticket. The agent emits create_ticket, and this is where the architecture earns its keep: The API validates the tool input and computes a deterministic ticket ID It publishes a ticket.created event via its Dapr sidecar to the Service Bus topic ticket-events — and immediately streams the confirmation with the ticket ID back to the employee Dapr delivers the event to the worker through the ticket-worker subscription The worker upserts the ticket into Table Storage, reads the optional Teams webhook URL from Key Vault, and posts the payload to a Power Automate HTTP flow The IT team gets an Adaptive Card in their Teams channel. A human is now in the loop — a real one. Chat acknowledgement never waits for persistence or Teams delivery. Three deliberate consequences follow. Idempotency end-to-end. The ticket ID is derived, not generated: def compute_ticket_id(conversation_id: str, subject: str) -> str: """Deterministic ticket ID from (conversation, subject) so a repeated create_ticket tool call for the same issue in the same conversation collapses to the same ID instead of creating a duplicate. """ key = f"{conversation_id}:{subject.strip().lower()}" return hashlib.sha256(key.encode("utf-8")).hexdigest()[:16] If the model retries the tool call, or Service Bus redelivers the event (the subscription allows up to 10 deliveries), the worker upserts the same row instead of minting duplicate tickets. Idempotency is designed in at the ID level, not bolted on with dedup logic afterwards. Eventual consistency, explained honestly. The ticket row may not exist for a few seconds while KEDA wakes the worker. Both the agent's status tool and the ticket endpoints treat "not visible yet" as a normal state and say so, and the UI polls every five seconds. Distributed systems do not hide their nature here — they narrate it. A topic, not a queue. Today there is one subscription, so operationally it behaves like a work queue. But "a ticket was created" is an event, and tomorrow an ITSM connector, an audit log, or an analytics pipeline can each get their own subscription without the API changing a single line. Publishers describe facts; subscribers decide what facts mean. This is the payload that travels unchanged from tool call, through Dapr and Service Bus, into the worker, Table Storage, and the Teams flow: { "type": "ticket.created", "ticket_id": "9a549ad5d5f723d4", "conversation_id": "conversation-id", "subject": "Request for access to finance shared drive", "description": "I need access to the finance shared drive for an audit.", "category": "shared-drive-access", "urgency": "high", "requester_email": "email address removed for privacy reasons", "status": "New", "created_at": "2026-07-17T16:30:28.396538+00:00", "updated_at": "2026-07-17T16:30:28.396538+00:00" } And the failure mode is designed too: if the Teams webhook is unset or down, the worker logs a warning and keeps the persisted ticket. Persistence happens first and returns success or retry to Dapr based only on the table write — so a Teams outage cannot cause repeated ticket writes. Zero Keys: The Identity Model This is the part I am proudest of. Every hop authenticates with Entra ID via DefaultAzureCredential — in ACA, AZURE_CLIENT_ID selects each app's user-assigned managed identity; locally, the same code rides on az login. API identity — AcrPull, Foundry agent access, Storage Table Data Reader, Key Vault Secrets User, Service Bus Sender. It invokes the agent, reads tickets, publishes events. Worker identity — AcrPull, Storage Table Data Contributor, Key Vault Secrets User, Service Bus Receiver. The sole writer of tickets. Frontend identity — AcrPull. It pulls its image, nothing more. Shared Dapr identity — Service Bus Data Owner, scoped to authenticating the Dapr component and the KEDA scaler. Notice the reader/writer split: the API physically cannot modify a ticket, and the worker is the only writer. Least privilege is not a slide bullet here; it is enforced by RBAC per identity. The single unavoidable secret — the Power Automate webhook URL, which is bearer-style by nature — lives in Key Vault, and nowhere else. Terraform Notes: The Provider Landscape Is Not What You Expect Terraform is the source of truth for all Azure resources, split into four modules: platform, foundry, observability, and aca. Two lessons here were worth the price of admission. The obvious-looking resources are the wrong ones.When I started, I assumed I would needazapi for the Foundry pieces. The real surprise was different: azurerm 4.x does ship azurerm_ai_foundry and azurerm_ai_foundry_project — but those provision the classic, hub-based Foundry model, not the GA project-based Foundry Agent Service. The current model is provisioned directly on a Cognitive Services account, fully covered by azurerm, no azapi required: resource "azurerm_cognitive_account" "this" { name = "${var.prefix}-${var.environment}-foundry-${var.random_suffix}" resource_group_name = var.resource_group_name location = var.location kind = "AIServices" sku_name = "S0" # Required for the account to work as a Foundry resource # (agents, projects) rather than plain Cognitive Services. custom_subdomain_name = "${var.prefix}-${var.environment}-foundry-${var.random_suffix}" project_management_enabled = true # Enforces "no API keys anywhere" at the account level: only # Entra ID auth is accepted, key-based auth is rejected outright. local_auth_enabled = false identity { type = "SystemAssigned" } } There was one genuine gotcha, though: the built-in Foundry Agent Consumer role grants enough to call the Responses API against an existing thread, but conversations.create() — which the API calls on every new chat — needs the agents/write data action too. I confirmed that live, with a 403 to show for it. The broader Foundry User role would work, but it also grants key-listing and the whole Cognitive Services surface. The fix is a small custom role definition granting exactly the three data actions the chat runtime exercises: interact, agents read, agents write. Least privilege, again. 2. Terraform provisions infrastructure — it does not configure agents. The vector store, document upload, and agent version are deliberately not Terraform resources. A seed_knowledge.py bootstrap runs after terraform apply, authenticated as a principal Terraform granted the Foundry Project Manager role. Agent instructions and knowledge content change on an application cadence, not an infrastructure cadence — mixing the two lifecycles is how you end up re-uploading your knowledge base because you resized a container app. Dapr's entity management is disabled for the same reason: Terraform explicitly owns the topic and subscription. Observability and Evaluation The API initializes Azure Monitor OpenTelemetry, and every Foundry invocation gets a custom agent.invoke span carrying the agent name, conversation ID, selected tools, and input/output token counts when the response exposes them. Platform logs and metrics from all three apps flow to the same Log Analytics workspace. Ask "what did that conversation cost and which tool did it pick?" and App Insights answers. There is also an evaluation script that pushes ten fixed questions through Foundry's cloud evaluation API, scoring intent resolution, coherence, and task adherence. File-search questions evaluate end-to-end; locally executed function tools need a response-capture approach — a limitation worth knowing before you promise your boss automated agent QA. If this section feels short, good — agent observability and governance in production deserves its own post, and it is getting one. Consider this the trailer. What This Deliberately Is Not Honesty section. HelpDesk Copilot is production-shaped, not production-finished: No browser authentication yet. Conversation IDs partition the ticket panel but are not an authorization boundary. A real rollout adds Entra ID sign-in and server-side authorization before exposing ticket data. Tickets stay New. The human handoff is the real Teams notification, not a simulated ITSM lifecycle. Wiring status updates back from an ITSM tool is exactly what the topic's future subscriptions are for. Global ticket lookup scans partitions. Fine at demo volume; a high-volume system adds an index. I would rather ship a clear boundary than a hidden one. Try It The full source — Terraform modules, all three services, the knowledge base, seed and evaluation scripts — is on GitHub: passadis/foundry-ticketing Quickstart: terraform apply, run seed_knowledge.py, build and push the three images, and ask the public URL why your VPN keeps dropping. With scale-to-zero on all three apps and a small model deployment, idle cost is close to nothing — the architecture only spends money when someone needs help. Conclusion The AI part of this solution is maybe twenty percent of the code. The rest is the unglamorous engineering that makes an agent deployable: identity, ingress boundaries, idempotent events, honest eventual consistency, IaC lifecycle boundaries, and telemetry. That ratio is the real lesson — and it is exactly why Azure Container Apps plus Microsoft Foundry is such a productive pairing: the platform absorbs the undifferentiated heavy lifting so the interesting decisions remain yours. Next up in this series: taking the agent.invoke spans further — tracing, token economics, drift, and governance for agents in production with Foundry's control plane and Azure Monitor. Until then — happy building! 🚀Microsoft Agent Framework Multi-Agent Workflow Architecture for Automated Kubernetes Assessments
Why this system generates tests (design rationale) This project does not generate tests just to "check code." It generates tests because, in a Kubernetes learning game, the test suite is the grading contract. The design goals are: Scale content creation: instructors should not hand-author every task and checker. Keep grading objective: student success is measured against Kubernetes API state, not subjective review. Avoid fragile tasks: generated tasks must survive empty/wrong cluster states without crashing. Make failures repairable: when checks break, the system should patch and re-validate automatically. So the pipeline generates a full task package (setup, answer, check, cleanup) where tests define exactly what "correct" means. Lifecycle: from concept to production-ready grader Reader mental model: this is a content compiler with validation stages, not a single chat completion. Phase 1: Pedagogical intent -> structured concept The Idea Agent converts a topic into a constrained concept object (objective, progression, task IDs, difficulty variants). Memory rules block duplicates and previously failed concepts. Phase 2: Concept -> executable grading package The Generator Agent turns that concept into files: student instructions (instruction.md) learning material (concept.md) parameter source (session.json) setup and answer manifests (setup.template.yaml, answer.template.yaml) deterministic pytest flow (test_01 ... test_06) At this point, output is still untrusted draft content. Phase 3: Structural correctness gate Deterministic validation checks file presence, syntax, JSON/YAML shape, and template correctness. This catches basic integrity issues before cluster execution. Phase 4: Behavioral correctness gate (real cluster) Pytest executes against Kubernetes and verifies runtime behavior using real kubectl-derived state. This proves that generated checks actually evaluate cluster resources as intended. Phase 5: Anti-false-positive gate (skip-answer mode) The same suite runs with SKIP_ANSWER_TESTS=True to verify grader integrity: answer deployment is skipped test_05_check.py must fail If it still passes, the grader is invalid (it would accept wrong student submissions). Phase 6: Self-healing repair loop On any failure, deterministic error logs are fed into the Fixer Agent, which patches only broken files. The workflow then re-enters validation + test gates. Phase 7: Finalization If all gates pass -> task is kept as production-ready content. If retries are exhausted -> task is moved to unsuccessful/ with FAILURE_REPORT.txt for human triage. This lifecycle explains the core architecture decision: LLMs generate candidate graders, deterministic execution certifies them. Concrete generated sample (what the pipeline actually produces) Below is a representative generated task for topic: ConfigMap Environment Variable Injection. Generated directory layout tests/game01/050_configmap_env_injection/ ├── __init__.py ├── instruction.md ├── concept.md ├── session.json ├── setup.template.yaml ├── answer.template.yaml ├── test_01_setup.py ├── test_02_ready.py ├── test_03_answer.py ├── test_05_check.py └── test_06_cleanup.py session.json (runtime variables) { "namespace": "{{random_name()}}{{random_number(100,999)}}{{student_id()}}", "configmap_name": "{{random_name()}}", "deployment_name": "{{random_name()}}", "container_name": "app", "env_key": "APP_MODE", "env_value": "production" } Why this exists: task values are randomized per student/session, so tests verify behavior by variable contract instead of hardcoded names. setup.template.yaml (baseline state only) apiVersion: v1 kind: Namespace metadata: name: {{ namespace }} Why this exists: setup should create prerequisites only. It must not accidentally include the final answer. answer.template.yaml (expected correct solution) apiVersion: v1 kind: ConfigMap metadata: name: {{ configmap_name }} namespace: {{ namespace }} data: {{ env_key }}: "{{ env_value }}" --- apiVersion: apps/v1 kind: Deployment metadata: name: {{ deployment_name }} namespace: {{ namespace }} spec: replicas: 1 selector: matchLabels: app: env-demo template: metadata: labels: app: env-demo spec: containers: - name: {{ container_name }} image: nginx:latest env: - name: {{ env_key }} valueFrom: configMapKeyRef: name: {{ configmap_name }} key: {{ env_key }} Why this exists: defines canonical "correct cluster state" that graders must detect. test_02_ready.py (wait for setup resources) import json import time from tests.helper.kubectrl_helper import build_kube_config, run_kubectl_command class TestReady: def test_001_namespace_active(self, json_input): kube_config = build_kube_config( json_input["cert_file"], json_input["key_file"], json_input["host"] ) time.sleep(2) result = run_kubectl_command( kube_config, f"kubectl get namespace {json_input['namespace']} -o json", ) data = json.loads(result) assert data.get("status", {}).get("phase") == "Active" Why this matters: validates setup-stage readiness only. It should not check answer resources yet. test_05_check.py (student grading contract) import json from tests.helper.kubectrl_helper import build_kube_config, run_kubectl_command class TestCheck: def test_001_configmap_key_exists(self, json_input): kube_config = build_kube_config( json_input["cert_file"], json_input["key_file"], json_input["host"] ) result = run_kubectl_command( kube_config, f"kubectl get configmap {json_input['configmap_name']} -n {json_input['namespace']} -o json", ) data = json.loads(result) assert data["data"][json_input["env_key"]] == json_input["env_value"] def test_002_deployment_uses_configmap_env(self, json_input): kube_config = build_kube_config( json_input["cert_file"], json_input["key_file"], json_input["host"] ) result = run_kubectl_command( kube_config, f"kubectl get deployment {json_input['deployment_name']} -n {json_input['namespace']} -o json", ) data = json.loads(result) env = data["spec"]["template"]["spec"]["containers"][0].get("env", []) matched = [ e for e in env if e.get("name") == json_input["env_key"] and e.get("valueFrom", {}).get("configMapKeyRef", {}).get("name") == json_input["configmap_name"] and e.get("valueFrom", {}).get("configMapKeyRef", {}).get("key") == json_input["env_key"] ] assert matched, "Deployment container must consume env var from ConfigMap key" Why this matters: this is the real grading logic. If a student deploys wrong resource wiring, this test fails with explicit reason. Why skip-answer validation is essential for this sample When SKIP_ANSWER_TESTS=True, answer deployment is skipped. In this mode: test_03_answer.py should be skipped test_05_check.py must fail (ConfigMap/Deployment wiring is absent) If test_05_check.py still passes, the grader is broken (false positive), and the workflow routes to Fixer. Rule Builder Workflow Flowchart Multi-graph architecture views 1) Control-plane graph (orchestration DAG) 2) Runtime sequence (who calls what) 3) State machine (task lifecycle) 4) Prompt lifecycle graph (how prompts evolve) 5) MCP + Kubernetes execution boundary graph Core Agent Framework Primitives Used This repository is a practical example of Agent Framework as a graph orchestrator, not just an agent wrapper. WorkflowBuilder builds a typed DAG with explicit edges. @executor functions implement deterministic nodes (validation, pytest, decisions, routing prep). AgentExecutor wraps LLM agents so they behave as graph nodes. WorkflowContext shared state carries typed data and retry metadata between nodes. add_multi_selection_edge_group(...) + selector functions enforce conditional routing. MCPStdioTool connects filesystem MCP tools into agents for controlled file I/O. Production graph construction (from workflow/builder.py) looks like this: workflow = ( WorkflowBuilder(start_executor=initialize_retry) .add_edge(initialize_retry, generator_executor) .add_edge(generator_executor, parse_generated_task) .add_edge(parse_generated_task, run_validation) .add_edge(run_validation, run_pytest) .add_edge(run_pytest, make_decision) .add_multi_selection_edge_group( make_decision, [keep_task, remove_task], selection_func=select_action, ) .add_edge(keep_task, run_pytest_skip_answer) .add_multi_selection_edge_group( run_pytest_skip_answer, [check_loop, complete_workflow], selection_func=select_skip_answer_action, ) .add_multi_selection_edge_group( check_loop, [fix_task, complete_workflow], selection_func=select_loop_action, ) .add_edge(fix_task, fixer_executor) .add_edge(fixer_executor, parse_generated_task) .build() ) This is the architectural heart of the system: agents and deterministic executors are first-class nodes in the same graph. Detailed Node-by-Node Mechanics 1. Idea Agent (🧠): Concept Synthesis with Memory Constraints The Idea Agent (agents/k8s_task_idea_agent.py) generates a structured concept with three difficulty variations (BEGINNER/INTERMEDIATE/ADVANCED). It is memory-aware: task_ideas_memory.json tracks successful concepts. task_ideas_failure_memory.json tracks concepts that failed downstream. Memory constraints are injected using AgentMiddleware (system-level prompt injection) to avoid duplicate or previously failed concepts. For Responses-only models, the agent switches to a tool-call contract (save_k8s_task_concept) instead of structured response formatting. 2. Generator Agent (⚙️): MCP-Backed File Authoring The Generator Agent receives a strict prompt and writes task files through MCP filesystem tools. Key framework details: Built through chat_client.as_agent(...). MCP tool attached via tools=mcp_tool. Function-call execution observability added with LoggingFunctionMiddleware. Uses absolute-path-only policy in instructions to prevent path drift. 3. Deterministic Validation + Test (✅): Non-LLM Gates After generation, the graph moves through deterministic executors: run_validation calls pure Python checks (k8s_task_validator). run_pytest executes pytest --import-mode=importlib --rootdir=. .... Raw pytest output is persisted in workflow state for later fixing. This is critical: no LLM is asked whether code is correct. 4. Skip-Answer Test (🧪): Grader Correctness Gate Even if standard tests pass, the workflow enforces a second tier: SKIP_ANSWER_TESTS=True pytest --import-mode=importlib --rootdir=. Implementation detail: the executor writes JUnit XML, parses it, and asserts that: test_03_answer.py is skipped test_05_check.py fails as expected If test_05_check.py does not fail, the task is treated as invalid and sent back to retry/fix. 5. Fixer Agent + Retry Loop (🔧): Bounded Self-Healing On failure, fix_task builds a targeted prompt containing: failure reasons from deterministic nodes full captured pytest output explicit rule to patch only broken files in place The Fixer Agent runs through AgentExecutor, writes patches via MCP, and the graph loops back to parse_generated_task. Retries are stateful (retry_count, max_retries) and hard-bounded. On exhaustion, complete_workflow moves the task to unsuccessful/<game>/ and writes FAILURE_REPORT.txt. Agent Prompt Design (The Part That Makes It Work) If you want to understand why this pipeline works, you need to inspect prompts as operational contracts, not generic instructions. Real Idea Agent Prompt (from code) IDEA_AGENT_INSTRUCTIONS = ( "You are a Kubernetes task idea generator that creates detailed task concepts with three difficulty variations. " "Read official K8s documentation and propose comprehensive learning concepts for a Kubernetes game. " "\n\nYour task:\n" "1. Choose ONE Kubernetes concept not yet covered (check context for existing concepts)\n" "2. Generate exactly 3 variations: BEGINNER, INTERMEDIATE, and ADVANCED\n" "3. Use 3-digit task IDs (001-999) in format: XXX_concept_name_level (e.g., 041_secrets_basic)\n" "4. Each variation should build on the previous one with increasing complexity\n" "5. Include practical, hands-on scenarios covering: Workloads, Services, Storage, Configuration, Security, Scheduling, Policies\n" "\nProvide the concept, tags, description, and 3 variations with task_id, difficulty, title, objective, key_skills, and estimated_time." ) Responses-only models use a stricter tool-call version: IDEA_AGENT_INSTRUCTIONS_TOOL_CALL = ( IDEA_AGENT_INSTRUCTIONS + "\n\n" "**CRITICAL**: You MUST call the save_k8s_task_concept tool to save your generated concept.\n" "...\n" "Always call save_k8s_task_concept with your generated concept." ) Idea Agent Prompt Contract The Idea Agent prompt enforces: one concept per run exactly three difficulty variations strict task ID format (XXX_concept_name_level) practical skill progression Core pattern: You are a Kubernetes task idea generator... 1. Choose ONE Kubernetes concept not yet covered 2. Generate exactly 3 variations: BEGINNER, INTERMEDIATE, ADVANCED 3. Use 3-digit task IDs in format XXX_concept_name_level ... It is strengthened by runtime memory injection: previously generated concepts are blocked previously failed concepts are blocked For Responses-only models, the contract becomes tool-driven: CRITICAL: You MUST call the save_k8s_task_concept tool... This reduces ambiguity in output structure and makes downstream parsing deterministic. Real Generator Agent Prompt (from code) def _get_generator_instructions(): return ( "You are a Kubernetes task generator with filesystem tools.\n" f"The MCP filesystem is rooted at: {PATHS.tests_root.parent}\n" f"You MUST use ABSOLUTE paths for ALL file operations.\n" f"Task directory: {PATHS.game_root}/XXX_task_name/\n" "...\n" "CRITICAL: test_02_ready.py checks resources from setup.template.yaml, NOT answer.template.yaml.\n" "...\n" "MUST use polling loops (60s timeout, 15s interval)\n" "MUST use try/except and safe .get() JSON access\n" ) The generator prompt is long on purpose: it encodes path correctness, file schema, YAML/Jinja structure, and testing strategy in a single deterministic contract. Generator Agent Prompt Contract The Generator prompt is intentionally long and prescriptive because it defines filesystem safety and grading correctness requirements. Key constraints encoded in the prompt: Absolute path writes only (prevents writing to wrong workspace paths) No directory creation (directory is pre-created by executor) Required file set (instruction.md, concept.md, session.json, templates, tests) test-flow invariants: test_01_setup.py deploys setup test_02_ready.py checks setup resources only test_03_answer.py deploys answer test_05_check.py validates final solution robust test coding style: polling loops, try/except, .get()-based JSON parsing, explicit debug output Example contract fragment: CRITICAL PATH RULES: ✅ CORRECT: /abs/path/tests/gameXX/050_task/file.py ❌ WRONG: tests/gameXX/050_task/file.py (relative) CRITICAL: test_02_ready.py checks resources from setup.template.yaml, NOT answer.template.yaml. This is why generation quality is high before the Fixer loop even starts. Real Runtime Retry Prompt Builder (from code) def _build_retry_generation_prompt(combined: CombinedValidationResult) -> str: task_id = combined.test.task_id failure_reasons = _build_failure_reasons(combined) return ( f"Generate a complete Kubernetes learning task with ID '{task_id}' about '{combined.target_topic}'. " f"This is retry attempt {combined.retry_count + 1} of {combined.max_retries}. " f"\n\n⚠️ PREVIOUS ATTEMPT FAILED:" f"\n{chr(10).join([f' - {reason}' for reason in failure_reasons])}" f"\n\nIMPORTANT: You MUST use the exact task ID '{task_id}' - do not generate a new ID." f"\n\n✅ Directory already exists: {PATHS.game_root}/{task_id}/" f"\nWrite all files directly into this directory. Do NOT call create_directory." "..." ) This means retries are not generic retries; they are failure-conditioned retries with precise constraints. Fixer Agent Prompt Contract The Fixer prompt is a repair protocol, not a regeneration prompt. It includes: exact failure reasons from deterministic validators raw pytest output instruction to read current task files first strict directive to patch only broken files Core behavior constraints: DO NOT rewrite all files. Make TARGETED FIXES to ONLY the broken files. Use ABSOLUTE paths for all file operations. This keeps retries cheap, preserves working artifacts, and improves convergence speed. Real Runtime Fix Prompt Builder (from code) def _build_fix_prompt(combined: CombinedValidationResult, raw_test_output: str) -> str: task_id = combined.test.task_id failure_reasons = _build_failure_reasons(combined) prompt = ( f"Fix the failed Kubernetes task '{task_id}' located in '{PATHS.game_root}/{task_id}/'." f"\n\nThis is fix attempt {combined.retry_count + 1} of {combined.max_retries}." f"\n\n⚠️ TASK FAILED WITH THESE ERRORS:" f"\n{chr(10).join([f' - {reason}' for reason in failure_reasons])}" ) if raw_test_output: prompt += f"\n\n📋 FULL TEST OUTPUT:\n```\n{raw_test_output}\n```" prompt += ( f"\n\n🔍 YOUR TASK:" f"\n1. READ all files from '{PATHS.game_root}/{task_id}/'" f"\n6. Make TARGETED FIXES to ONLY the broken files" f"\n7. WRITE ONLY the fixed files back" f"\n\n⚠️ CRITICAL: DO NOT rewrite all files! Only fix the broken ones!" ) return prompt How Prompt Output Enters the Agent Framework Graph The prompt builders above are used by deterministic executors and sent to agent nodes through AgentExecutorRequest: await ctx.send_message( AgentExecutorRequest( messages=[Message(role="user", contents=[fix_prompt])], should_respond=True ) ) So prompt generation and graph routing are tightly coupled: each route transition emits a specific prompt payload into the next LLM node. Runtime-Constructed Prompts in Executors The most important prompts are built dynamically in workflow executors: _build_retry_generation_prompt(...) _build_fix_prompt(...) These functions inject live context: retry_count / max_retries concept + objective metadata validation/test failure reasons full captured test logs So each retry is context-rich and specific, not another blind generation attempt. Prompt + Middleware + Deterministic Gates = Reliability In this repository, reliability does not come from prompt text alone. It comes from three layers working together: Prompt contracts constrain agent behavior. Middleware injects memory and logs tool invocations. Deterministic executors enforce objective pass/fail gates. That combination is why the workflow remains auditable and predictable even when LLM outputs vary. Agent Framework Execution Model in This Repo Strongly-Typed Message Passing workflow/models.py defines transport models used between nodes: ValidationResult and TestResult (Pydantic) CombinedValidationResult (dataclass with should_keep and should_retry) InitialWorkflowState (seed payload for each run) This keeps node contracts explicit and simplifies selector logic. Fail-Fast Shared State Management Executors use ctx.get_state(...) with a sentinel (_MISSING) and raise explicit exceptions if required state is absent. This prevents hidden fallback behavior and catches graph/data wiring errors early. Conditional Routing with Selectors Selectors (workflow/selectors.py) encode graph decisions: select_action → keep vs remove select_skip_answer_action → complete vs loop select_loop_action → fix vs complete This separates decision policy from executor implementation. Streaming Workflow Runtime workflow.run(initial_state, stream=True) emits output events incrementally. The runner (workflow/runner.py) consumes these events to detect successful completions and update concept memory accordingly. Agent Construction and API Selection Strategy The repository uses Azure CLI auth (AzureCliCredential) and dynamically selects API mode by deployment name (agents/config.py): Chat Completions path: OpenAIChatCompletionClient Responses-only model path: OpenAIChatClient or custom ResponsesAgent Why this matters: some codex-class deployments are Responses-only, so the architecture supports both without changing workflow logic. How MCP Actually Controls Kubernetes (Important Distinction) In this repo, MCP is used for filesystem control; Kubernetes control is done through kubectl tools. 1) MCP server role: controlled file I/O The workflow starts MCP stdio servers (official filesystem server) and mounts them into agents: docs_mcp_tool = MCPStdioTool( name="filesystem_docs", command="npx", args=["-y", "@modelcontextprotocol/server-filesystem", str(PATHS.k8s_docs_root)], load_prompts=False, ) tests_mcp_tool = MCPStdioTool( name="filesystem_tests", command="npx", args=["-y", "@modelcontextprotocol/server-filesystem", str(PATHS.tests_root.parent)], load_prompts=False, ) Those MCP tools are passed into Generator/Fixer agents, which then call MCP file functions (read/write/list) inside allowed roots only. 2) Kubernetes cluster control role: kubectl execution tool Cluster actions are not performed by MCP filesystem server; they are performed by a dedicated function tool: def run_kubectl_command(command: str) -> str: kubeconfig_path = os.environ.get("KUBECONFIG", "/home/developer/.kube/config") cmd_list = ["kubectl"] + command.split() result = subprocess.run( cmd_list, capture_output=True, text=True, check=True, env={**os.environ, "KUBECONFIG": kubeconfig_path}, ) return result.stdout And the Kubernetes agent forces tool usage: agent = responses_client.as_agent( name="KubernetesAgent", instructions="...You MUST use the run_kubectl_command tool...", tools=[run_kubectl_command], default_options={"tool_choice": "required"}, ) So the control plane is: MCP filesystem → manipulate generated task files. kubectl tool → query/mutate real cluster state. deterministic pytest/validator executors → accept or reject results. 3) End-to-end command flow in practice When generated tests run, they execute real kubectl get ... -o json checks in test code, and the deterministic runner captures raw output: pytest_command = f"pytest --import-mode=importlib --rootdir=. {task_with_val.task_directory}/" result = run_pytest_command(pytest_command) ctx.set_state(f"raw_output_{task_with_val.task_id}", raw_output) This means Kubernetes state verification is always grounded in live command output, not model speculation. Should MCP run Kubernetes tests? Short answer: not in this design. Current architecture keeps test execution deterministic and local: pytest is run by run_pytest_command(...) (pure Python subprocess runner) test results are parsed and stored in workflow state retry/fix routing uses those deterministic outputs This is intentional. If test execution were delegated to an LLM-facing MCP command tool, you would lose strict control over execution semantics and error handling. Recommended pattern: Use MCP for file/document access and controlled editing. Use deterministic executors for pytest and validation. Use LLM agents only for generation and repair. If you still want MCP-driven test execution, add a separate locked-down command MCP server (only whitelisted pytest/kubectl commands), but keep pass/fail decision logic in deterministic executors. How tests are run in this workflow (with code) The workflow executes tests in deterministic executors, not inside LLM agents. 1) Workflow node calls pytest runner run_pytest executor builds the command and calls the pure Python runner: @executor(id="run_pytest") async def run_pytest(task_with_val: TaskWithValidation, ctx: WorkflowContext[TestResult]) -> None: from agents.pytest_runner import run_pytest_command pytest_command = f"pytest --import-mode=importlib --rootdir=. {task_with_val.task_directory}/" result = run_pytest_command(pytest_command) raw_output = result["details"][0] if result.get("details") else "" ctx.set_state(f"raw_output_{task_with_val.task_id}", raw_output) ... 2) Deterministic subprocess execution The runner normalizes command flags and executes pytest via subprocess: def run_pytest_command(command: str) -> dict[str, Any]: normalized_command = _normalize_pytest_command(command) # adds -s if needed cmd_list = shlex.split(normalized_command) result = subprocess.run( cmd_list, capture_output=True, text=True, check=False, cwd=str(PATHS.pytest_rootdir), ) combined_output = result.stdout + "\n" + result.stderr _save_test_output(normalized_command, combined_output, skip_answer) ... Exit codes are interpreted deterministically: 0 → pass 5 → no tests collected (fail) others → fail with exit code reason 3) Skip-answer validation tier After normal pass, the workflow runs pytest again with SKIP_ANSWER_TESTS=True and parses JUnit XML: os.environ["SKIP_ANSWER_TESTS"] = "True" pytest_command = f"pytest --import-mode=importlib --rootdir=. --junitxml={junit_path} {task_dir}/" result = run_pytest_command(pytest_command) test_05_failed, test_03_skipped = _parse_skip_answer_junit(junit_path) The parser checks per-testcase outcomes: if "test_05_check.py" in context and has_failure_or_error: test_05_failed = True if "test_03_answer.py" in context and has_skipped: test_03_skipped = True 4) How failures trigger fix loop If pytest fails (or skip-answer logic fails), failure reasons and raw output are pushed into state, then the Fixer Agent receives a generated fix prompt containing that output: ctx.set_state(f"failure_reasons_{task_id}", reasons) ctx.set_state(f"raw_output_{task_id}", raw_output) fix_prompt = _build_fix_prompt(combined, raw_test_output) await ctx.send_message( AgentExecutorRequest( messages=[Message(role="user", contents=[fix_prompt])], should_respond=True ) ) That is the key loop: deterministic test output drives LLM repair, then deterministic tests re-run. ResponsesAgent Internals (Advanced Agent Framework Pattern) The custom ResponsesAgent (agents/responses_agent.py) demonstrates a lower-level integration pattern: Connect MCP tool lazily. Call Responses API. Parse ResponseFunctionToolCall items. Execute tools (MCP + custom callables). Feed function_call_output back to model. Repeat until final text response. It also runs a middleware chain around tool invocations, preserving observability and consistency with standard agent paths. Why This Architecture Is Robust This design works because Agent Framework is used as a deterministic orchestration layer around probabilistic generation: LLM creativity is constrained by typed state and strict prompts. deterministic executors act as objective quality gates. retries are targeted, bounded, and auditable. failures produce durable forensic artifacts (FAILURE_REPORT.txt + test logs). For Kubernetes education pipelines, this yields high throughput without sacrificing grader reliability. GitHub Repo - https://github.com/wongcyrus/k8s-game-rule-builder About the Author Cyrus Wong is the senior lecturer of Hong Kong Institute of Information Technology (HKIIT) @ IVE(Lee Wai Lee).and he focuses on teaching public Cloud technologies. He is a passionate advocate for the adoption of cloud technology across various media and events. With his extensive knowledge and expertise, he has earned prestigious recognitions such as AWS AI Hero, Microsoft MVP- Microsoft Foundry, and Google Developer Expert - Cloud(AI).
Is there a way to connect 2 Ai foundry to the same cosmos containers?
I defined Azure AI Foundry Connection for Azure Cosmos DB and BYO Thread Storage in Azure AI Agent Service by using these instructions: Integration with Azure AI Agent Service - Azure Cosmos DB for NoSQL | Microsoft Learn I see that it created 3 containers under the cosmos I provided: <guid>-agent-entity-store v-system-thread-message-store <guid>-thread-message-store Now I created another AI foundry and added a connection for the same AI foundry, and it created 3 different containers under the same DB. Is there a way that they'll use the same exact containers? I want to use multiple AI foundries, and they will use the same Cosmos containers to manage the data.
Missing equivalent for Python MemorySearchTool and AgentMemorySettings in C# SDK
Hi Team, I am currently working with the Azure AI Foundry Agent Service (preview). I’ve been reviewing the documentation for managed long-term memory, specifically the "Automatic User Memory" features demonstrated in the Python SDK here: https://learn.microsoft.com/en-us/azure/ai-foundry/agents/how-to/memory-usage?view=foundry&tabs=python. In Python, it is very straightforward to attach a MemorySearchTool to an agent and use AgentMemorySettings(scope="user_123") during a run. This allows the service to automatically extract, consolidate, and retrieve memories without manual intervention. However, in the https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/ai/Azure.AI.Projects#memory-store-operations, I only see the low-level MemoryStoreClient which appears to require manual CRUD operations on memory items. My Questions: Is there an equivalent high-level AgentMemorySearchTool or similar abstraction in the current C# NuGet package (Azure.AI.Projects) that handles automatic extraction and retrieval? If not currently available, is this feature on the immediate roadmap for the .NET SDK? Are there any samples showing how to achieve "automatic" memory (where the Agent extracts facts itself) using the C# SDK without having to build a custom orchestration layer or call REST APIs directly? Any guidance on the timeline for feature parity between the Python and .NET SDKs regarding Agent Memory would be greatly appreciated. SDK Version: Azure.AI.Projects 1.2.0-beta.5Unable to delete Foundry Agent identity Entra app in Azure
I'm trying to delete an Entra app in Azure created by Foundry Agent identity blueprint as its currently unused and is causing EntraID hygiene alerts. However getting an error mentioning that delete is not supported. Is there any other way to delete an unused Entra app for an agent identity blueprint? Error detail: Agent Blueprints are not supported on the API version used in this request.Building ShadowQuest: A Multi-Agent RPG
Artificial Intelligence is rapidly evolving beyond traditional chatbots. Today, developers are building intelligent systems where multiple AI agents collaborate, retrieve knowledge, and solve problems together. Microsoft's Agents League Hackathon provided the perfect opportunity to explore this new approach through the Reasoning Agents challenge. For this challenge, I built ShadowQuest, a fantasy role-playing game (RPG) powered by Microsoft Foundry, Foundry IQ, Azure AI Search, GPT-4.1, and GitHub Copilot. The project demonstrates how specialized AI agents can work together while using Retrieval-Augmented Generation (RAG) to deliver accurate and context-aware responses. About the Challenge Microsoft Agents League is a global developer challenge designed to encourage developers to build intelligent AI applications using Microsoft's latest AI technologies. Participants could choose from three tracks: Creative Apps, Reasoning Agents, and Enterprise Agents. I selected the Reasoning Agents track because I wanted to explore how multiple AI agents could collaborate instead of relying on a single large language model. Another important requirement for this year's challenge was integrating at least one Microsoft Intelligence Layer. For ShadowQuest, I chose Foundry IQ as the project's intelligence layer. The Idea Behind ShadowQuest Fantasy RPGs are built around storytelling, exploration, and collaboration between different characters. Every character usually has a unique role, whether it's a warrior protecting the team, a mage interpreting magical knowledge, or a rogue discovering hidden paths. I wanted to recreate this experience using AI. Instead of building one AI assistant responsible for everything, I designed a system where multiple specialized agents collaborate to create a richer and more immersive adventure. ShadowQuest is set in a fantasy world filled with magical artifacts, forgotten kingdoms, mysterious locations, and story-driven quests. Players can ask questions about the world, explore different locations, and learn about the game's lore through conversations with AI agents. Building the Multi-Agent Architecture The architecture follows a simple but scalable design. At the center of the system is the Game Master Agent, which acts as the orchestrator. Every player interaction starts with the Game Master. It receives the player's request, determines what information is needed, retrieves additional knowledge when required, and generates the final response. Supporting the Game Master are three specialized agents: Warrior Agent – Focuses on combat strategy and tactical decisions. Mage Agent – Provides magical knowledge, world lore, and information about ancient artifacts. Rogue Agent – Specializes in exploration, investigation, and discovering hidden information. Each agent has a clearly defined responsibility, making the system easier to understand, maintain, and extend in the future. Using Foundry IQ as the Knowledge Layer One of the most important parts of the project was integrating Foundry IQ. Instead of storing every piece of game information inside prompts, I created a dedicated knowledge base containing information about characters, magical artifacts, locations, quests, and the history of the ShadowQuest world. This approach separates knowledge from reasoning. Whenever a player asks a question, the Game Master Agent first retrieves relevant information from the knowledge base before generating a response. This ensures that answers remain consistent with the game's world while reducing hallucinations. Foundry IQ became the central source of truth for the entire project, making it easy to manage and expand the game world without constantly modifying prompts. Azure AI Search and Retrieval-Augmented Generation To enable intelligent retrieval, I connected Foundry IQ with Azure AI Search. The RPG documents were indexed, and vector embeddings were generated using Microsoft's embedding models. This enables semantic search, allowing the system to understand the meaning behind a player's question instead of relying only on keyword matching. For example, if a player asks about a magical relic without mentioning its exact name, Azure AI Search can still retrieve the correct information based on semantic similarity. The complete workflow looks like this: The player submits a question. The Game Master Agent receives the request. Foundry IQ queries Azure AI Search. Relevant documents are retrieved. GPT-4.1 generates a grounded response using the retrieved context. This Retrieval-Augmented Generation (RAG) approach significantly improves the quality and reliability of responses. Accelerating Development with GitHub Copilot GitHub Copilot played an important role throughout the development process. It helped generate Python classes, improve documentation, create helper functions, and speed up repetitive coding tasks. During the live demonstration, I also showed how Copilot could quickly generate a new Healer Agent, demonstrating how AI-assisted development makes it easier to extend a multi-agent application while maintaining a consistent architecture. Rather than replacing the developer, Copilot acted as an intelligent coding assistant, allowing me to focus more on architecture and design decisions. Demonstrating ShadowQuest During the Microsoft Agents League Reasoning Agents Battle, I demonstrated the Game Master Agent by asking questions about the ShadowQuest world, magical artifacts, and game lore. One of the most interesting parts of the demonstration was observing the retrieval process. Before generating a response, the Game Master Agent called the knowledge retrieval function through Foundry IQ. This confirmed that the system was retrieving relevant information from the indexed knowledge base rather than relying only on GPT-4.1's internal knowledge. This demonstrated how RAG can create more grounded, reliable, and context-aware AI experiences. Lessons Learned Building ShadowQuest taught me that designing multi-agent systems is as much about architecture as it is about AI models. Clearly defining responsibilities for each agent made the application easier to maintain and opened the door for future expansion. I also learned how valuable Retrieval-Augmented Generation can be for applications that depend on structured knowledge. Separating reasoning from knowledge allows AI systems to remain accurate while making it easier to update information over time. Finally, participating in the Microsoft Agents League was an incredible opportunity to experiment with Microsoft's latest AI technologies, learn from other developers, and share ideas with a global community passionate about agentic AI. Looking Ahead ShadowQuest is only the beginning. In future iterations, I plan to expand the project by introducing additional agents such as a Merchant Agent and Healer Agent, implementing persistent player memory, adding dynamic quest generation, improving combat mechanics, and enabling deeper collaboration between agents. These improvements will make the game world more immersive while continuing to explore the possibilities of agent-based AI systems. Conclusion ShadowQuest demonstrates how Microsoft Foundry, Foundry IQ, Azure AI Search, GPT-4.1, and GitHub Copilot can be combined to build intelligent multi-agent applications. More importantly, the project reinforced an important idea: the future of AI is not a single assistant performing every task, but a team of specialized agents collaborating with shared knowledge to solve increasingly complex problems. Participating in the Microsoft Agents League was an inspiring experience that allowed me to explore the next generation of AI development while building a project that combines storytelling, reasoning, and knowledge retrieval. I look forward to continuing this journey and discovering new ways to build intelligent applications using Microsoft's growing AI ecosystem.
