Evaluating Agentic AI in Microsoft Foundry: Beyond Final-Answer Quality
As AI systems evolve from single-turn assistants into tool-using agents, evaluation has to evolve with them. For a traditional LLM application, teams often focus on whether the final response is relevant, coherent, or grounded. For an agent, that is only part of the problem. An agent may produce a plausible answer while still selecting the wrong tool, sending incorrect tool parameters, ignoring tool output, violating user constraints, making unnecessary calls, or failing to complete the task. This is why Microsoft Foundry separates agent evaluation into multiple layers and supports both system-level and process-level evaluation. Microsoft describes agent evaluators as a way to systematically assess quality, safety, and performance across agent workflows—not only the final response. https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/evaluate-agent Start with the evaluation target Foundry allows evaluations against different targets: Agent — run the selected agent against test inputs and evaluate newly generated behavior Model — evaluate a model directly Dataset — score outputs that already exist in a database This distinction matters because the evaluation workflow changes depending on the target. When the target is an Agent, Foundry generates a fresh response for each input and evaluates that result. When the target is a Dataset, Foundry evaluates the responses already present in the dataset. https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app For agentic systems, I find it useful to evaluate in two scopes: Individual turns for detailed debugging of tool use and response behavior. Full conversations for multi-turn task completion, conversational coherence, and user satisfaction. Microsoft currently recommends starting with full conversations and simulated data for controlled testing, then using real conversations in production. System evaluation: did the agent actually accomplish the task? The first layer is the end-to-end outcome. Important evaluators include: Task Completion Did the agent actually complete the user’s task? For example: “Find black running shoes under $120 in size 9 in Dallas.” An agent may correctly search for products but stop before checking inventory. The response may look reasonable, but the task is still incomplete. Task Adherence Did the agent follow its instructions, policies, and explicit user constraints? If the customer says the budget is $120 and the agent recommends a $129 product without clearly identifying the budget violation, adherence should suffer. Intent Resolution Did the agent correctly understand and address what the user wanted? A tool call can be technically valid but semantically wrong. Searching generic sneakers when the user explicitly requested running shoes is a good example. Customer Satisfaction This looks beyond technical correctness and asks whether the overall interaction is likely to satisfy the user. These metrics answer a simple but important question: Did the system actually solve the user’s problem? Microsoft groups these kinds of measurements under system evaluation for agent workflows. Process evaluation: did the agent take the right path? This is where agent evaluation becomes much more interesting. Foundry includes dedicated process evaluators for tool-using agents. Tool Selection This checks whether the agent chose the appropriate tool. Suppose an agent has: search_products check_inventory get_promotions web search If the user asks: “Is DailyRun X available in size 9 in Dallas?” The right enterprise action is to use the inventory tool. If the agent uses web search instead, the answer may still sound plausible, but the process is wrong. Tool Input Accuracy This checks whether the tool was called with the correct inputs. For example: check_inventory( product_id="DEMO-SHOE-002", size="9", location="Dallas-TX" ) The agent may choose the right tool but pass the wrong size or location. That means Tool Selection can pass while Tool Input Accuracy fails. Tool Call Success This evaluates whether the invocation itself succeeded. For example: the API returned successfully there was no timeout there was no execution failure But a successful tool call does not mean it was the correct call. An API may return HTTP 200 for the wrong SKU or wrong location. So: Tool Call Success measures execution reliability, not semantic correctness. Tool Output Utilization This checks whether the agent correctly used what the tool returned. Imagine the inventory tool returns: { "available": false } but the agent replies: “Yes, size 9 is available.” The tool worked. The input may even have been correct. The failure is in how the agent used the result. Tool Call Accuracy This gives a broader signal about whether tool invocation behavior was correct overall. Together, these evaluators answer: Did the agent execute the right workflow, not just produce a convincing answer? Response quality still matters Even when the process is correct, the final response can still be poor. Foundry supports quality evaluators such as: Relevance Groundedness Completeness Coherence Fluency Groundedness Groundedness asks whether claims are supported by the available context or evidence. For enterprise applications, this is especially important because answers should be based on authoritative sources rather than unsupported model memory. Completeness Did the agent answer all important parts of the request? If the user asks: “Is this available and is there a member discount?” an answer that checks inventory but ignores the promotion question is incomplete. Coherence and Fluency These evaluate communication quality. Microsoft defines coherence around logical and orderly presentation of ideas, while fluency focuses on readability, grammar, vocabulary, and clarity. https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/general-purpose-evaluators Built-in evaluators are not enough for every enterprise agent A retail agent, finance agent, claims agent, or operations agent will always have business-specific requirements that generic evaluators cannot fully capture. Microsoft’s current guidance recommends using a rubric evaluator as a primary measure when teams need to express application-specific criteria such as policy enforcement, tool-use accuracy, or communication standards. Built-in evaluators can then be layered on top for broader coverage. A retail-agent rubric might include: never claim availability without inventory verification never invent discounts never silently violate a stated budget prefer enterprise catalog tools over public search avoid duplicate or unnecessary tool calls clearly identify compromises when no exact match exists This is where evaluation starts to look much more like business acceptance testing. https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rubric-evaluators The dataset becomes a regression suite Foundry evaluation datasets are reusable test collections. They are useful when comparing: Prompt V1 vs V2 one model vs another tool-definition changes orchestration changes release candidate vs production version When Foundry evaluates an agent against a dataset, it generates a new response for each input and scores it. When the target is the live agent, Foundry ignores existing responses in the dataset. This changes prompt engineering from: “This prompt looks better.” to: “This version performs better on the same regression suite.” That is a much more defensible engineering practice. Golden datasets, synthetic data, and production traces all serve different purposes A mature evaluation strategy uses multiple data sources. Golden datasets: Best for critical known scenarios and deterministic regression testing. Synthetic data: Useful for expanding coverage and generating edge cases before enough production traffic exists. Full-conversation simulation: Useful for testing multi-turn user journeys and end-to-end task behavior. Production traces: Useful for evaluating what real users actually experienced. Foundry supports evaluation directly from Application Insights traces and can evaluate deployed interactions without replaying the original request. https://learn.microsoft.com/en-us/azure/foundry/observability/how-to/cloud-evaluation-deployed-interactions Production evaluation should operate on traces Production failures are often different from development failures. Real users introduce: unexpected phrasing missing information contradictory constraints unusual tool sequences edge cases that synthetic test data may not cover Foundry can evaluate traces already captured in Application Insights. The trace-evaluation workflow supports selecting traces by trace ID or by agent filter, and also supports intelligent sampling to select a representative subset rather than evaluating every interaction. That creates a strong operational loop: Run → Trace → Evaluate → Identify failures → Add to regression suite → Fix → Re-evaluate Microsoft Foundry’s evaluation stack becomes especially useful as teams can evaluate not only what an agent said, but also how it behaved operationally, which tools it used, whether it completed the task, and how that behavior changes over time.Beyond Tokens: Rethinking AI Economics with Microsoft Foundry
Beyond Tokens: Rethinking AI Economics with Microsoft Foundry From the cost of intelligence to the value of outcomes Enterprise AI has an accounting problem. Executives expect agentic AI to return roughly 171% on investment, according to one widely cited survey. Yet McKinsey finds only about 39% of organizations can attribute any earnings impact to AI at all. Both numbers can be true at once — because the gap between them is not a technology gap. It is a measurement gap. For the first few years of generative AI, one number dominated the economics conversation: tokens. How many tokens did a model consume? What was the cost per million tokens? Could a smaller model perform the same task? Those questions mattered when enterprises were experimenting with AI. They are no longer enough as AI moves into production. An enterprise agent doesn't simply consume tokens. It reasons, retrieves context, invokes tools, calls APIs, verifies its work, retries unsuccessful actions and sometimes escalates exceptions to humans. The model call might cost pennies. The business outcome could cost considerably more. Which leads to an increasingly important question: What is the right economic unit for intelligence? From AI experimentation to economic accountability The first wave of enterprise AI was about possibility: Can AI do this? The next wave is about production, as AI becomes embedded in software engineering, customer service, finance, healthcare and supply chains. And production changes the question: Should AI do this and at what cost? Microsoft has moved decisively onto this ground. In August 2026, the Microsoft Foundry team launched its Economics of Agent Optimization series, arguing that "tokens have become the new unit of technology spend" and that AI should be run as a managed investment system. On the latest earnings call, Satya Nadella described Microsoft's objective as "advancing the frontier on the cost-to-outcome curve, ensuring every customer can turn tokens into business results." The discipline is going mainstream too: 98% of FinOps teams now manage AI spend, up from 31% two years ago. Microsoft's series is largely about the numerator of that curve - making every request, agent and dollar more efficient. This article is about the denominator: what an outcome is, what it truly costs, and what it is worth. The evolution of Microsoft Foundry reflects the same shift. At Build 2026, Microsoft expanded the conversation beyond building agents toward tracing behavior, evaluating quality, monitoring production performance, optimizing agents and connecting their operation to ROI. Think of the progression as: Trace → Evaluate → Monitor → Optimize → ROI This is more than a technology roadmap. It represents a shift from observing AI as technology to managing AI as an economic asset. Tokens became the unit of spend. They were never the unit of value. Consider two AI agents handling the same customer-service workflow. Agent A costs $0.08 per interaction. Agent B costs $0.20. Agent A appears cheaper. But suppose Agent A successfully resolves only 55% of cases, while Agent B resolves 90%. The remainder require retries, additional reasoning or human intervention. Which agent is actually cheaper? The inexpensive interaction may produce the expensive resolution. This illustrates a fundamental problem: We often measure AI where it is consumed rather than where value is created. Tokens are a unit of consumption. Businesses operate in outcomes. A customer-service leader cares about issues resolved. An engineering leader cares about high-quality software reaching production. A finance leader cares about reconciliations completed accurately. The economic denominator needs to move closer to the business. The AI Economic Ladder I think of this evolution as an AI Economic Ladder: Tokens → Interactions → Tasks → Outcomes → Value Each step moves measurement closer to what the enterprise actually cares about. At the token level: What intelligence did we consume? At the interaction level: What did each AI run cost? At the task level: What did it cost to complete the work? At the outcome level: What did a successful result cost? At the value level: Was the outcome worth creating? An AI system can become more efficient at every technical metric while creating little economic value. Conversely, an expensive AI workflow could be extraordinarily valuable if it prevents revenue leakage, reduces operational risk or accelerates a critical business process. The objective isn't cheaper AI. It is better economics. Not every completed task is a successful outcome There is another complication. If an agent completes a workflow, should we count it as a successful outcome? Not necessarily. A meaningful outcome needs three characteristics: Completed. Quality-gated. Attributable. It must reach its intended end state, meet an explicit standard for quality, accuracy, safety or business acceptability, and be attributable to the agent or workflow that produced it. That gives us a more meaningful measure: Cost per Successful Outcome = Fully Loaded AI Workflow Cost / Completed, Quality-Gated, Attributable Outcomes The denominator becomes real only when named in business language: cost per prior authorization resolved in healthcare, per pull request triaged and tested in engineering, per disputed invoice reconciled in finance operations. If you cannot name the outcome in a sentence the process owner recognizes, you are not ready to measure it. The quality gate matters. With AI, "the system ran successfully" and "the system produced a good outcome" are not the same thing. Microsoft Foundry's tracing and evaluation capabilities become economically important for precisely this reason. Evaluation isn't merely quality control. It helps determine what gets counted as value. What does an AI outcome really cost? The true economic footprint goes far beyond inference: Model + Reasoning + Grounding + Tools + Orchestration + Infrastructure + Retries + Evaluation + Governance + Human Intervention Human intervention is particularly easy to overlook. Every time someone must review, correct, approve or recover an AI-generated outcome, the economics change. The same applies to verification. An agent reaching an acceptable result in three steps has different economics from one requiring fifteen steps and multiple retries. And verification is not a rounding error — it is the bulk of the bill. McKinsey's 2026 analysis of production agentic workflows found roughly 60% of an agentic task's cost is tied to refining answers — checking, repairing, re-verifying — not generating the initial response. Most of what you pay for is not intelligence. It is assurance. This means quality and economics are connected. The quality bar you set influences the cost you pay. The challenge isn't simply minimizing consumption. It is finding the right balance between quality, cost, speed and risk. Cost per outcome is only half the equation Now imagine two agents. Both cost $5 per successful outcome. One saves an employee ten minutes of administrative work. The other prevents $500 in revenue leakage. Their cost efficiency is identical. Their economics clearly aren't. So we need to move another step up the ladder: from Cost per Outcome to Value per Outcome. The question isn't only how cheaply AI can complete the work. It is: How much economic value does this outcome create relative to the intelligence required to produce it? Now the CIO, CFO, CAIO and business leader have a common conversation. Give every outcome an Intelligence Budget Not every problem deserves the smartest model available. Classifying an email may require relatively little intelligence. Resolving a complicated customer complaint may justify more context and reasoning. Assessing the risks in a multimillion-dollar contract may justify sophisticated reasoning, multiple validations and human review. Every business outcome therefore has an economically rational amount of intelligence worth spending on it. Call it an Intelligence Budget. This changes the architecture question from which model should we standardize on, to: What combination of model, reasoning, context, tools and human judgment does this outcome deserve? This is where Microsoft Foundry's model router becomes interesting. Individual requests can be dynamically routed so simpler work doesn't consume the same model resources as complex reasoning. If the Intelligence Budget is the economic principle, intelligent routing is one way of operationalizing it. The future enterprise AI architecture won't be about one model doing everything. It will route intelligence according to the economics, quality and risk of the outcome. Making AI economics observable None of this works without visibility. An AI system can be technically healthy and economically unhealthy — responsive and error-free while repeatedly choosing inefficient reasoning paths, invoking unnecessary tools or producing outputs requiring expensive human correction. AI economics and AI observability are becoming inseparable. Microsoft Foundry increasingly connects these disciplines. Tracing shows what an agent did. Evaluation determines whether it met required criteria. Observability helps monitor production behavior. Agent optimizer can test improvements across prompts, skills and models. Microsoft's emerging ROI capabilities take the next step by connecting operating costs with measures such as task completion, time saved and cost efficiency. Attribution is the bridge to the finance conversation. Teams place Azure API Management in front of Foundry endpoints as an AI Gateway, stream token telemetry into Application Insights, and use Entra Agent ID to give every agent run a discrete identity that maps cost to its cost center. Microsoft Agent 365 extends the discipline tenant-wide — spending policies, budget caps and departmental chargeback across Microsoft and third-party agents. Together, they create something enterprises have historically lacked: A feedback loop between how intelligence is consumed and what that intelligence accomplishes. The paradox of cheaper intelligence There is another reason AI economics will become more important as models get cheaper. The Jevons paradox suggests that when technology makes a resource cheaper and more efficient, total consumption can actually increase. AI may experience the same effect. Cheaper intelligence enables more agents, more reasoning and more workflows that were previously uneconomic. So we could see cost per unit of intelligence fall while total intelligence consumed rises. Cheaper AI may therefore produce larger AI bills. That isn't necessarily bad — provided value grows faster than consumption. The objective isn't minimum AI consumption. It is maximum economic value from AI consumption. From workload economics to portfolio economics As AI scales, economics becomes a capital-allocation question. I see three levels. Workload Economics: Is this AI system running efficiently? Outcome Economics: Is it producing quality outcomes economically? Portfolio Economics: Where should we put our next AI dollar? That final question will become increasingly important. An enterprise with hundreds of AI initiatives shouldn't assume every one deserves continued investment. Some should scale. Some need optimization. Some should be redesigned or consolidated. And some should be stopped. The ability to experiment cheaply created the first explosion of enterprise AI. The discipline to allocate capital intelligently will determine what scales. Who owns AI economics? Once an agent becomes part of how work gets done, its economics cannot remain purely an IT metric. The business understands the value of the outcome. Technology understands the architecture and optimization levers. Finance brings economic discipline and comparability. That suggests a shared model: Business owns the outcome. Technology owns the optimization levers. Finance owns the economic discipline. AI economics ultimately isn't just a technology-cost conversation. It is a business-performance and capital-allocation conversation. From abundant intelligence to intelligent economics We are entering an era where intelligence is becoming an increasingly abundant, programmable and variable-cost resource. Microsoft Foundry and the broader Microsoft AI stack are making it easier to build, evaluate, observe, optimize and govern that intelligence. But abundant intelligence does not guarantee abundant value. Enterprises still need to decide where AI belongs, how much intelligence each problem deserves, what defines a successful outcome, when humans should remain involved and which AI investments deserve more capital. The winners won't necessarily use the cheapest models. They won't consume the fewest tokens. And they won't be the organizations that build the most agents. They will become exceptionally good at moving up the AI Economic Ladder: from consumption, to outcomes, to value. Because the next era of AI won't be won by organizations that buy intelligence most cheaply. It will be won by those that convert intelligence into value most efficiently. Where to start: the first 90 days Define the denominator for your top three agents — what counts as done, what quality gate applies, who signs off. Instrument attribution — Azure API Management as an AI Gateway, token telemetry to Application Insights, Entra Agent ID on every run. Wire evaluations into the cost pipeline so only quality-gated outcomes count. Set Intelligence Budgets — model router per request, agent optimizer against your evaluators, Agent 365 policies as circuit breakers. Stand up a joint monthly review — business, technology and finance on one dashboard: outcomes delivered, cost per outcome, value per outcome. Frequently asked questions What is Cost per Successful Outcome in enterprise AI? The fully loaded cost of an AI workload divided by outputs that were completed, quality-gated and attributable - for example, cost per prior authorization resolved or per pull request triaged. It turns token metrics into the unit economics of AI-performed work. What is an Intelligence Budget? The economically rational amount of intelligence - model capability, reasoning, context, tools and human review — worth spending on a given outcome, based on its value and risk. Model router in Microsoft Foundry is one way to operationalize it. Why do AI agents cost more than single model calls? One agent task can involve planning, tool calls, retries and verification - many model calls with compounding context. Research on production agentic workflows attributes roughly 60% of task cost to refining and verifying answers, not generating the first response. Will falling model prices make AI cost management unnecessary? No. By the Jevons paradox, cheaper intelligence expands consumption, so total AI spend typically rises as unit prices fall. The discipline that matters is maximizing value per unit of intelligence. Who should own AI economics? A shared model: the business owns the outcome and its value, technology owns the optimization levers, and finance owns the economic discipline and review cadence. #MicrosoftFoundry #Agent365 #AzureAI #FinOps #AgenticAI #AIAgents #Azure #MicrosoftCostManagement #AIEconomics #Tokens References Microsoft Azure Blog: "The Economics of Agent Optimization: From pilots to measurable returns" (August 12, 2026) Microsoft FY26 Q4 earnings call (Satya Nadella, July 2026) McKinsey — "Cost versus value: managing agentic AI system performance" (July 2026) FinOps Foundation — State of FinOps 2026; Microsoft Learn — Model router for Microsoft Foundry; Agent optimizer; Foundry Control Plane cost optimizationAdding a Fallback Model to Hermes with Microsoft Foundry
So the plan was simple. Leave the Bedrock configuration untouched, then wire Microsoft Foundry in behind it as a fallback, so Hermes always has somewhere else to go when the primary provider is not responding. A few other reasons pushed me towards Foundry in particular: Redundancy that does not need me. If Bedrock is throttled or out of quota, I want Hermes to fail over on its own rather than waiting for me to notice. A catalogue I already pay for. Foundry puts the latest GPT models next to open-weight and partner models in one place, so I can pick a model that suits the task instead of settling for whatever a single provider happens to offer. Enterprise controls out of the box. Region pinning, private networking, content filters and per-deployment quota all sit in the same portal, which makes the setup far easier to defend to a security reviewer. Learning the mechanics before I need them. Working out how Hermes handles a provider chain is much nicer on a quiet Tuesday than during a live outage. Here is the short version, if you are deciding whether to read on. Time: about thirty minutes if nothing goes wrong. Cost: pay-as-you-go tokens only, and none at all while the fallback sits idle. Result: an assistant that keeps answering when your primary provider stops. Before you start, you will need three things: a machine with Hermes already installed and a working primary provider configured, an Azure subscription with access to Microsoft Foundry in a region you can actually deploy into, and enough quota in that region to create a deployment. One thing that made this easy to justify: Foundry deployments bill per token on the standard pay-as-you-go tier. A fallback provider that never gets invoked costs nothing beyond the requests it actually serves, so the insurance is close to free until the day you need it. Chat surface (CLI, messaging) → Hermes Gateway → Primary Amazon Bedrock → Fallback Microsoft Foundry Figure 1: Where the fallback sits. Every request goes through the Hermes gateway to the primary provider; only when that provider is unavailable does the chain continue to Microsoft Foundry. Part 1: Deploying a Model on Microsoft Foundry The first half of this job happens entirely inside the Microsoft Foundry portal and has nothing to do with Hermes yet. All you are really doing here is making sure your Azure subscription can serve a model, and that you hold an endpoint and key Hermes can authenticate with later. 1. Deploy model in Foundry → 2. Copy endpoint + key → 3. hermes fallback add → 4. Authenticate → 5. Select models, test Figure 2: The whole setup in five moves. The first two happen in the Microsoft Foundry portal (orange); the rest happen on the Hermes machine (blue). Go to Microsoft Foundry > Build > Models > Deploy > Deploy a base model. You can deploy a fine-tuned model instead if you already have one, which works just as well with Hermes. Check the region shown at the top of the portal before you commit, because both model availability and deployment quota differ from one region to the next. Then deploy the model you have selected: In this case I deployed gpt-5.6-sol, which is the model Hermes will fall back to. The choice was deliberate rather than exciting. My primary model on Bedrock is a general-purpose chat model, and a fallback is only useful if the answers it gives feel like a continuation of the same conversation rather than a different assistant wearing the same name. The gpt-5.6-sol deployment matches that behaviour closely, it was available in the region I wanted to pin, and the quota I was granted comfortably covers a day of normal use. If a fallback surprises you the first time it fires, it is the wrong fallback. Once the deployment finishes, open it and take note of two values: the target endpoint URI and the API key. Copy both somewhere safe now, because you will be pasting them into Hermes in the next part. If your organisation rotates keys on a schedule, use a key with the longest life you are allowed, since a fallback secured with a credential that expires quietly stops being a fallback. What to copy Where it lives in the portal Where Hermes asks for it Target endpoint URI Deployment > Endpoint > Target URI "Endpoint" prompt in hermes fallback add API key Deployment > Endpoint > Key "API key" prompt, or choose Entra ID instead Deployment name Deployment > Details > Name Shown in the model list Hermes returns Region Top of the portal, next to the resource Must match the region you deployed into Figure 3: Everything Hermes will ask for, and where to find each value before you leave the portal. Part 2: Adding Foundry to Hermes as a Fallback With the Foundry side sorted, everything from here happens in the Hermes CLI. One thing worth knowing before you start: this is the fallback command, not the primary model command, so your existing Bedrock configuration is left completely alone. Nothing in this section can break what is already working, which makes it a good one to try on a live setup. Run the Hermes fallback command: When Hermes asks which provider to add, choose Azure Foundry. The picker still carries the old name; it is the same service that the portal now calls Microsoft Foundry. Paste the target endpoint URI from your deployment, then authenticate with the API key you copied earlier. Hermes also offers Microsoft Entra ID at this prompt, which is the better option if your organisation would rather not have a static key sitting on the machine. If authentication fails here, check the endpoint before you start suspecting the key. In my experience the endpoint is wrong far more often than the credential is, usually because the deployment name at the end of the URI does not match the deployment you actually created. Once authentication succeeds, Hermes lists the deployments your Foundry resource exposes and asks which ones you want to use. You can select more than one, and the order is not cosmetic: Hermes walks down the chain from top to bottom whenever the provider above is unavailable. Treat that list as a priority order, not a shopping basket. What happens to the primary What Hermes does What you see in the chat Responds normally Routes every request to the primary and never touches the chain Nothing. The fallback stays idle Throttled or out of quota Retries the next provider down the chain on the same request A reply, served by the fallback model Endpoint unreachable Keeps failing over on each new request until the primary recovers Slightly different tone and latency, but a working assistant Every provider fails Returns the error rather than hanging An error worth chasing with hermes status Figure 4: The chain in practice. The fallback only earns its keep in the middle two rows, which is exactly why it is easy to forget you configured it. Part 3: Promoting Foundry to the Primary Model At this stage Foundry is sitting in the back seat as a backup. I wanted to reverse the arrangement and make Foundry the primary while Bedrock slides down into the fallback slot, partly because I preferred keeping day-to-day traffic inside my Azure subscription, and partly because I wanted proof the chain works in both directions. Before promotion After promotion Primary: Amazon Bedrock → Primary: Microsoft Foundry Fallback: Microsoft Foundry → Fallback: Amazon Bedrock Figure 5: The promotion, in effect. Nothing is added or removed; the two providers simply trade places in the chain. There is no dedicated "promote" command in Hermes, so the manual route is a short sequence of steps rather than a single instruction: Select the fallback provider/model as the new primary: hermes model Remove the now-duplicate model from the fallback chain: hermes fallback remove Optionally add the old primary model as a fallback: hermes fallback add Restart the messaging gateway: hermes gateway restart Verify the result: hermes status / hermes fallback list That sequence works, and it is good to know what is happening underneath. But since I already had a working provider configured, I would rather just ask Hermes to rearrange itself. This is the part I genuinely enjoy about the tool: the configuration is something you can talk to, not only something you type commands at. Prompt: Okay now please make the model I configured on Microsoft Foundry into the main model, and make the Bedrock one the fallback model! Hermes rewrites the provider chain on its own and confirms the swap once it is done, which is a great deal less error-prone than running the five commands by hand. Part 4: Testing the Switch Configuration you have not tested is just an assumption with extra steps, so the next thing is to confirm Hermes really is talking to Foundry. Type /model when running Hermes to bring up the model picker. You will be prompted to select a provider first. Pick the Microsoft Foundry entry, then choose the specific deployment from the list underneath it. The active model should switch straight away. Send it a plain "Hello" to check that the deployment actually responds, rather than just looking correct in the menu. Appearing in a dropdown and serving a request are two very different things. A second test is worth the thirty seconds it costs: run hermes status to confirm which provider is live, then hermes fallback list to confirm the chain is ordered the way you intended. The picker tells you what you selected; those two commands tell you what Hermes will actually do at three in the morning. Part 5: The Obstacle, and What It Actually Taught Me Every walkthrough has the part the author quietly leaves out. Here is mine: the wrinkle was not the model, it was capacity. My first attempt deployed into the region closest to me out of habit, and the portal turned it down because there was no capacity left for that model at the tier I asked for. The model was clearly listed in the catalogue; being listed and being deployable in your region, on your subscription, at your quota, are three separate questions. Redeploying in a different region fixed it in a couple of minutes, but it meant the endpoint URI changed, which in turn meant the value I had already pasted into Hermes was stale. Re-running hermes fallback add against the new endpoint sorted it out. The lesson is cheap enough to hand over for free: check quota and regional capacity for your specific subscription before you design a walkthrough, a demo or a production fallback around one deployment. In the Foundry portal, Management then Quota shows exactly what you have been granted per region and per model family, which is the only list that matters. Symptom Likely cause Fix Deployment rejected in the portal No capacity for that model at the tier you asked for, in that region Deploy in another region, or drop to a smaller tier Hermes rejects the credential Endpoint URI does not match the deployment you created Re-copy the target URI from the deployment, not the resource Provider authenticates but lists nothing Key belongs to a different Foundry resource Check you are in the right resource, then re-run hermes fallback add Fallback never fires Chain ordered the wrong way round hermes fallback list, then reorder Figure 6: The four things that went wrong, or nearly did, and what fixed each one. There is a silver lining worth stating plainly. Because the fallback chain was already in place, a deployment that refused to come up did not take the assistant down with it. That is precisely the scenario this whole exercise was meant to cover, and it turned up on day one without me having to simulate it. Command Cheat Sheet Everything used in this walkthrough, collected in one place: hermes fallback add: attach a provider to the fallback chain hermes fallback remove: drop a provider from the chain hermes fallback list: show the chain in priority order hermes model: set the primary model hermes gateway restart: restart the messaging gateway after a change hermes status: confirm which provider is currently live /model: switch models from inside a running session Conclusion Adding Microsoft Foundry as a fallback behind my existing Bedrock setup took an afternoon, and most of that was spent recovering from a regional capacity limit I should have checked first. The work itself is small: deploy a model, copy the endpoint and key, run hermes fallback add, authenticate, pick your deployments. The payoff is that Hermes no longer depends on one provider staying healthy. Three things are worth carrying away from this: Check quota, not just the catalogue. The Foundry catalogue shows what Microsoft offers. It does not show what your subscription and region can actually deploy today. Confirm that first, before you build anything on top of a specific deployment. Order your fallback chain deliberately. Hermes works down the list from top to bottom, so the sequence you choose during setup is the failover policy you are going to live with. Put the model you actually trust at the top. Treat the endpoint as part of the credential. Redeploying in a new region changes the endpoint URI, and a fallback pointed at an endpoint that no longer exists is not a fallback. Re-run the setup whenever the deployment moves. The switch from Bedrock primary to Foundry primary also proved the chain runs in both directions, which is the real point. Provider redundancy is only useful if you have watched it work. Next on my list is deliberately breaking the primary provider to confirm the failover triggers on its own, without me typing a single command. If you run this against a different model, region or provider pairing, I would genuinely like to know how it went, particularly if your quota experience was better than mine. Drop it in the comments.Simplify AKS observability with Azure Native New Relic Service
An Azure Native path to New Relic Intelligent Observability for AKS AKS environments are dynamic by design. Applications can span clusters, namespaces, nodes, pods, and containers, while workloads scale and change continuously. Obtaining consistent visibility often requires platform teams to deploy and maintain monitoring components separately on every cluster. Azure Native New Relic Service simplifies this process by integrating New Relic onboarding and management into Azure. Customers can already use the service to: Create a new New Relic account or link an existing account from Azure. Configure the forwarding of Azure platform metrics and logs to New Relic. View the monitoring status of Azure resources. Consolidate procurement and eligible New Relic charges through Azure Marketplace. Monitor multiple Azure subscriptions through a single New Relic resource. With AKS extension support, customers can now extend this native management experience to their Kubernetes clusters. Install the New Relic integration from the Azure portal The new experience follows the same simple model used by Azure Native integrations for other compute resources. From an Azure Native New Relic Service resource, customers can navigate to New Relic account config > Azure Kubernetes Services, select an eligible AKS cluster, and choose Install Extension. Azure then deploys the New Relic Kubernetes integration by using the AKS cluster extensions framework. After deployment completes, the portal displays the installation status for the cluster. Customers can return to the same experience to review the status or select Uninstall Extension when monitoring is no longer required. AKS cluster extensions provide an Azure Resource Manager-based approach for installing and managing services on AKS. This gives customers a consistent Azure control plane experience for deployment and lifecycle operations instead of requiring a separate, manual Helm installation for each cluster. Gain deeper visibility into Kubernetes workloads The extension deploys the New Relic Kubernetes integration to the selected AKS cluster. The integration provides visibility across Kubernetes infrastructure and workloads, including cluster, node, namespace, deployment, pod, and container health and performance. Depending on the enabled New Relic configuration, customers can also bring together Kubernetes events, logs, and Prometheus-formatted metrics with application and Azure platform telemetry in New Relic. This helps application, platform, and site reliability engineering teams investigate issues across the stack without stitching together disconnected views. Teams can use New Relic to: Understand resource consumption and health across clusters, nodes, pods, and containers. Identify unhealthy workloads, container restarts, and capacity constraints. Correlate Kubernetes infrastructure signals with application performance data. Explore Kubernetes entities and relationships through New Relic's cluster experience. Create dashboards, alerts, and operational workflows using telemetry from Azure and AKS. The result is a more direct path from detecting an issue to understanding its impact on applications and users. Reduce operational toil with unified telemetry For organizations operating multiple AKS clusters, consistency is as important as visibility. Manual installation can lead to configuration drift, missed clusters, and additional work whenever monitoring components need to be changed. The Azure Native New Relic Service experience helps address these challenges by providing: Simplified onboarding: Install the integration from the Azure portal without building a separate deployment workflow. Centralized visibility: Review AKS extension status alongside other Azure resources connected to New Relic. Azure governance alignment: Use Azure Resource Manager and Azure role-based access control as part of the management experience. Lifecycle management: Install or remove the extension through a consistent Azure workflow. Unified, full-stack observability: Connect AKS telemetry with application, infrastructure, log, and Azure platform data in New Relic. This experience is particularly useful for platform teams that want to make observability available as a standardized service while allowing development teams to use New Relic for troubleshooting and performance optimization. Get started To begin monitoring AKS with Azure Native New Relic Service: Either browse to the Marketplace offer listing or in the Azure portal, create an Azure Native New Relic Service resource. Go to New Relic account config > Azure Kubernetes Services. Select the AKS cluster that you want to monitor. Select Install Extension and then confirm the installation. After the status changes to Installed, open New Relic to explore your Kubernetes data and configure the dashboards and alerts appropriate for your environment. If you face any technical challenges do raise a support ticket and share your feedback.Sizing Copilot Credits for Cowork? Let Your Users Use Cowork Investment Advisor Agent.
Right, let's talk about a headache. If you're helping a customer allocate Copilot Credits for M365 Copilot Cowork, you have to size it all up first — and doing that by hand is a fiddly, time-consuming job. Here's the problem, and a much easier way to solve it. Start with the Cowork Estimator Here's the good news: you don't have to start from scratch. Microsoft has published the Customer Cowork Estimator — a handy tool that turns personas, prompt complexity and expected usage into an estimated credit number. It's the perfect place to begin, and it does the core maths for you. To get the most out of it, you just need to feed it good inputs — and that's where a little groundwork comes in. For each part of the business, you'll want to: Work out who's who. There are four types of user — corporate knowledge workers, management and senior leaders, customer-facing folks, and technical staff. Count how many of each. Quick for a small team, a bit more involved for a big one. Pin down what they'd genuinely use Cowork for. The real multi-step workflows that hop across apps and actually do things — not just a quick chat or a summary. Judge how heavy each workflow is. Light, Medium or Heavy — since each level uses a different number of credits per run. Estimate how often it runs. Daily adds up to a lot of runs a month; weekly is far fewer. The estimator handles the sums beautifully once those inputs are in. Gathering the inputs themselves — persona by persona, workflow by workflow, across a few thousand people — is simply the part that takes time. And that's exactly where this agent lends a hand: it builds on the estimator by automating the groundwork that feeds it. How this agent helps This agent takes on that groundwork for you. Instead of sizing everyone from the outside, the admin simply switches the agent on for every user. Each person then sizes their own needs — and the agent does the clever bits for them, ready to drop into the estimator. Here's what it does behind the scenes: Spots the right persona. It works out which of the four personas each user fits. Finds the real workflows. It looks at the top Cowork scenarios the user would actually run. Keeps it honest. It checks each one is a genuine Cowork job — several steps, more than one app, real actions and a bit of decision-making — not something a Scheduled Prompt or plain Copilot Chat could do just as well. Grounds it in evidence. It reads the signals from recent work — emails, meetings, documents and Teams chats — so the estimate is based on what people actually do, not thin air. Does the sums. It maps each workflow's complexity to credits per run and totals it up — giving you numbers that line up neatly with the estimator. Shows the value. It gives a view of pay-as-you-go versus a pre-purchase plan, and a sense of the return on the spend. Stays transparent. It states its assumptions, flags how confident it is, and sticks to permitted data and the usual privacy and compliance rules. And here's the kind of report it hands back — persona, the top workflows, complexity, credits, cost and a clear recommendation, all in one place: Why this approach works better So why hand it to the users? A few good reasons: More accurate. The numbers come from each person's real workload, not a top-down guess. Consistent by design. Everyone follows the same method — same personas, same complexity bands, same maths — so the results line up and roll into one clean figure. No over-buying. Because it weeds out the workflows that don't really need Cowork, your customer only pays for credits they'll genuinely use. A proper business case. You get cost clarity and a feel for the return, so you're handing over more than just a number. It scales. Ten users or ten thousand — the effort on your side stays much the same. It saves you hours. You swap manual sizing for gathering and rolling up. Your time goes on advising, not tallying. Not ready to roll it out? Other ways in If an admin isn't quite ready to switch the agent on for everyone, that's OK— there's no need to. This agent is an M365 Copilot agent, and it has free access to Work IQ — the same engine that will eventually power Cowork. Because that access is free, the agent can read how people really work and take the guesswork right out of sizing. And there's more than one way to get at it: Build it with Agent Builder. Ready-made instructions, a description and a starter prompt are all sitting in the GitHub repo. Anyone with access to M365 Copilot can pop them into Agent Builder and stand the agent up in minutes. Prefer not to build an agent? Just use the prompt. There's a comprehensive prompt you can drop straight into the M365 Copilot Chat experience. Same sizing, same evidence — no agent to create. Ready to give it a go? Whichever route suits you best, getting started is dead simple. If you're rolling the agent out to your users: Have a quick word with your customer's admin about switching the agent on for their users. Let each person size their own Cowork needs. Gather it all up and roll it into a single estimate. Prefer to keep it in your own hands? Build the agent from the GitHub repo with Agent Builder, or drop the comprehensive prompt into M365 Copilot Chat — you'll get to the same place with even less setup. THE BOTTOM LINE That's the heavy lifting done for you. You'll save yourself hours, spare yourself the guesswork, and hand your customer a credit plan they can genuinely trust — with the numbers and the business case sitting right behind it. Give it a spin on your next engagement and see how much quicker it gets you there. Resources Everything you need is in one place: Customer Cowork Estimator — https://aka.ms/CustomerCoworkEstimator GitHub repo — agent instructions, description, starter prompt and the full Copilot Chat prompt: Cowork Investment Assessment
From insight to action: how Adobe and Microsoft are helping marketers move faster with AI
Today’s marketing leaders are under pressure to do more than ever—deliver meaningful personalization, accelerate execution, and prove measurable business impact. At the same time, teams are navigating increasing complexity: fragmented data, disconnected tools, and insights that arrive too late to act on. AI can change this—but only when it’s embedded directly into how people already work. That’s why Microsoft and Adobe are deepening our partnership: bringing customer experience intelligence, AI-powered workflows, and enterprise-grade AI directly into Microsoft 365 Copilot—so teams can move from insight to alignment to execution in one continuous workflow. The result is faster decisions, more coordinated execution, and clearer business outcomes—without breaking flow or context. Bringing customer experience intelligence into the flow of work Marketing teams don’t struggle because they lack data. They struggle because insights live in one place, collaboration in another, and execution somewhere else entirely. That disconnect slows teams down and creates unnecessary friction between analysis and action. Together, Adobe and Microsoft are changing that dynamic by connecting Adobe’s customer experience capabilities with Microsoft 365 Copilot and Copilot Cowork—so insight, collaboration, and next-best action can happen where work already happens: in Copilot Chat and in everyday apps like Teams, Word, and PowerPoint. Marketers can ask questions, explore insights, align with teammates, and take action without jumping between tools—turning intelligence into impact at the moment it matters. Adobe Marketing Agent for Microsoft 365 Copilot: now generally available A major milestone in this journey is the general availability of the Adobe Marketing Agent for Microsoft 365 Copilot, now available via Microsoft Commercial Marketplace. The Adobe Marketing Agent brings Adobe customer experience intelligence directly into Copilot, enabling marketing teams to: Accelerate time from insight to decision Move seamlessly from analysis to execution Keep humans firmly in control, with AI supporting—not replacing—decision‑making Importantly, the agent is enterprise-ready by design. IT administrators can deploy and manage the experience through the Microsoft 365 admin center, ensuring security, governance, and compliance at scale. Expanding executive experiences with Copilot Cowork Looking ahead, Adobe skills designed for customer experience orchestration will be accessible in Copilot Cowork—in a future release. This upcoming experience will enable customer experience leaders to engage with customer experience insights in a more direct, conversational way, bringing strategic visibility into the same Copilot environments where decisions are made and actions are coordinated. Built on Azure to scale securely and responsibly The technology foundation of this innovation is Azure. Adobe Experience Platform, Adobe Experience Platform Agent Orchestrator, and Adobe AI Agents are built on Azure and leverage Azure AI models, providing the scalability, security, and reliability enterprises require. By running on Azure, these agentic experiences benefit from Microsoft’s global infrastructure, enterprise‑grade security, and responsible AI commitments—supporting customer trust as organizations scale AI across their business. Designed for interoperability across agent ecosystems Modern enterprises don’t operate in a single ecosystem—and their agents shouldn’t either. Adobe agents are built to interoperate with agents created using Microsoft Azure AI Foundry or Copilot Studio, enabling customers to orchestrate richer, cross‑functional workflows across marketing, sales, service, and operations. This architecture is designed to enable organizations to compose agentic solutions that reflect how work actually happens—across systems, teams, and business processes. Moving from experimentation to execution This partnership reflects a broader shift in how organizations adopt AI—moving from experimentation to embedded, enterprise‑ready execution. By bringing the full power of Adobe Experience Platform together with Microsoft’s AI platform, cloud infrastructure, and Copilot experiences, we’re helping teams move faster with clarity, confidence, and control. This is how AI becomes not just powerful—but practical. Learn more Adobe + Microsoft partnership page Adobe Marketing Agent for Microsoft Copilot pageAzure Native Integrations: Public Preview of Napster Companion API on Azure
What is Napster Companion API? Napster Companion API is Napster's platform for building Omniagents: persistent, multi-channel AI agents with one identity, one memory, and one set of tools that show up across every channel an end user touches. The same Omniagent meets the customer on the website, in the mobile app, on video, and on the phone line with the same face, the same voice, and the same memory of the last conversation. The Omniagent as a digital worker The clearest way to think about an Omniagent is as a digital worker: It has a role (customer support specialist, sales advisor, internal IT assistant). It carries the memory of past shifts and prior conversations. It has the tools it needs to do the job which include APIs, knowledge bases, ticketing systems, CRMs. It shows up across every surface the end user touches, like a human worker who answers the door, the phone, and the inbox. When something is outside its scope, it hands off to a human colleague with the context already attached and picks the thread back up when the human is done. Use cases for the Companion API Teams are already exploring the Companion API across a wide range of scenarios: Agentic commerce. Agents that guide end users through discovery, recommendations, purchase, and post-sales support all in one continuous conversation across channels. Customer service. Agents that resolve issues end to end, escalate to humans with full context attached, and pick the thread back up across sessions. Internal operations and digital coworkers. Agents that orchestrate workflows, retrieve knowledge, and automate repetitive tasks for the workforce. Capabilities introduced by Napster Companion API The Companion API public preview brings the following capabilities to Azure customers: Persistent multi-channel agents that maintain identity, memory, and context across web, mobile, voice, video, and telephony. Real-time multimodal interactions across voice, video, and text for natural back-and-forth conversation. Tool and API orchestration that lets agents take real actions like opening tickets, updating records, retrieving documents, and triggering workflows. Persona-driven agents with configurable behavior, conversational style, and avatar-based interaction. Knowledge bases and deterministic question-and-answer pairs for grounded, accurate responses on topics where exactness matters. Developer SDKs and a no-code Dashboard for building, testing, deploying, and iterating on agents. Better together: Napster and Microsoft This integration is the result of a long-term Azure-native partnership between Napster and Microsoft. It is not an external service layered onto Azure infrastructure but it is a co-engineered offering designed to help enterprises operationalize persistent AI agents at scale. In practice, the Azure Native integration delivers: Benefit What it means for you Seamless development experience Provision and manage Companion API resources directly from the Azure portal, alongside your other Azure services. Build and operate Omniagents in the Napster Dashboard, reached through single sign-on. Bring your own model or use Napster Hosted Connect your Azure OpenAI realtime deployment on Microsoft Foundry so inference runs in your tenant. Or use the Napster Hosted tier where Napster manages the model for you. Simplified billing Manage Companion API spend through Azure Marketplace, on the same invoice as the rest of your Azure consumption which means no separate procurement, no separate billing relationship. Single sign-on with Microsoft Entra Switch between Azure resources and the Companion API Dashboard without re-entering credentials. Enterprise-ready foundation Built on Azure's compliance, security, and global infrastructure footprint. How it works?    If the player doesn’t load, open the video in a new window: Open video Get started in minutes Provisioning Napster Companion API on Azure takes just a few clicks: Open the Azure portal and search for *Napster Companion API*. Create a new resource and choose your subscription, resource group, region, and pricing tier. Link your Napster organization (or create one as part of resource provisioning). Launch the Companion API Dashboard from the resource overview page using single sign-on, and start building your first Omniagent in the Napster portal. Full step-by-step guidance is available in the Napster Companion API documentation on Microsoft Learn Resources Product documentation: Napster Companion API on Microsoft Learn Quickstart: Create a Napster Companion API resource Azure Marketplace listing: Napster Companion API Napster for partners: napster.com/partners Azure Native Integrations overview: Azure partner solutions What's next This public preview is the first milestone on a broader roadmap. We are eager to hear from early adopters. Try the public preview, build your first Omniagent, and let us know what you think as your feedback will shape what ships next. Get started today by searching for Napster Companion API in the Azure portal.Model Mondays S2E11: Exploring Speech AI in Azure AI Foundry
1. Weekly Highlights This week’s top news in the Azure AI ecosystem included: Lakuna — Copilot Studio Agent for Product Teams: A hackathon project built with Copilot Studio and Azure AI Foundry, Lakuna analyzes your requirements and docs to surface hidden assumptions, helping teams reflect, test, and reduce bias in product planning. Azure ND H200 v5 VMs for AI: Azure Machine Learning introduced ND H200 v5 VMs, featuring NVIDIA H200 GPUs (over 1TB GPU memory per VM!) for massive models, bigger context windows, and ultra-fast throughput. Agent Factory Blog Series: The next wave of agentic AI is about extensibility: plug your agents into hundreds of APIs and services using Model Connector Protocol (MCP) for portable, reusable tool integrations. GPT-5 Tool Calling on Azure AI Foundry: GPT-5 models now support free-form tool calling—no more rigid JSON! Output SQL, Python, configs, and more in your preferred format for natural, flexible workflows. Microsoft a Leader in 2025 Gartner Magic Quadrant: Azure was again named a leader for Cloud Native Application Platforms—validating its end-to-end runway for AI, microservices, DevOps, and more. 2. Spotlight On: Azure AI Foundry Speech Playground The main segment featured a live demo of the new Azure AI Speech Playground (now part of Foundry), showing how developers can experiment with and deploy cutting-edge voice, transcription, and avatar capabilities. Key Features & Demos: Speech Recognition (Speech-to-Text): Try real-time transcription directly in the playground—recognizing natural speech, pauses, accents, and domain terms. Batch and Fast transcription options for large files and blob storage. Custom Speech: Fine-tune models for your industry, vocabulary, and noise conditions. Text to Speech (TTS): Instantly convert text into natural, expressive audio in 150+ languages with 600+ neural voices. Demo: Listen to pre-built voices, explore whispering, cheerful, angry, and more styles. Custom Neural Voice: Clone and train your own professional or personal voice (with strict Responsible AI controls). Avatars & Video Translation: Bring your apps to life with prebuilt avatars and video translation, which syncs voice-overs to speakers in multilingual videos. Voice Live API: Voice Live API (Preview) integrates all premium speech capabilities with large language models, enabling real-time, proactive voice agents and chatbots. Demo: Language learning agent with voice, avatars, and proactive engagement. One-click code export for deployment in your IDE. 3. Customer Story: Hilo Health This week’s customer spotlight featured Helo Health—a healthcare technology company using Azure AI to boost efficiency for doctors, staff, and patients. How Hilo Uses Azure AI: Document Management: Automates fax/document filing, splits multi-page faxes by patient, reduces staff effort and errors using Azure Computer Vision and Document Intelligence. Ambient Listening: Ambient clinical note transcription captures doctor-patient conversations and summarizes them for easy EHR documentation. Genie AI Contact Center: Agentic voice assistants handle patient calls, book appointments, answer billing/refill questions, escalate to humans, and assist human agents—using Azure Communication Services, Azure Functions, FastAPI (community), and Azure OpenAI. Conversational Campaigns: Outbound reminders, procedure preps, and follow-ups all handled by voice AI—freeing up human staff. Impact: Hilo reaches 16,000+ physician practices and 180,000 providers, automates millions of communications, and processes $2B+ in payments annually—demonstrating how multimodal AI transforms patient journeys from first call to post-visit care. 4. Key Takeaways Here’s what you need to know from S2E11: Speech AI is Accessible: The Azure AI Foundry Speech Playground makes experimenting with voice recognition, TTS, and avatars easy for everyone. From Playground to Production: Fine-tune, export code, and deploy speech models in your own apps with Azure Speech Service. Responsible AI Built-In: Custom Neural Voice and avatars require application and approval, ensuring ethical, secure use. Agentic AI Everywhere: Voice Live API brings real-time, multimodal voice agents to any workflow. Healthcare Example: Hilo’s use of Azure AI shows the real-world impact of speech and agentic AI, from patient intake to after-visit care. Join the Community: Keep learning and building—join the Discord and Forum. Sharda's Tips: How I Wrote This Blog I organize key moments from each episode, highlight product demos and customer stories, and use GitHub Copilot for structure. For this recap, I tested the Speech Playground myself, explored the docs, and summarized answers to common developer questions on security, dialects, and deployment. Here’s my favorite Copilot prompt this week: "Generate a technical blog post for Model Mondays S2E11 based on the transcript and episode details. Focus on Azure Speech Playground, TTS, avatars, Voice Live API, and healthcare use cases. Add practical links for developers and students!" Coming Up Next Week Next week: Observability! Learn how to monitor, evaluate, and debug your AI models and workflows using Azure and OpenAI tools. Register For The Livestream – Sep 1, 2025 Register For The AMA – Sep 5, 2025 Ask Questions & View Recaps – Discussion Forum About Model Mondays Model Mondays is your weekly Azure AI learning series: 5-Minute Highlights: Latest AI news and product updates 15-Minute Spotlight: Demos and deep dives with product teams 30-Minute AMA Fridays: Ask anything in Discord or the forum Start building: Register For Livestreams Watch Past Replays Register For AMA Recap Past AMAs Join The Community Don’t build alone! The Azure AI Developer Community is here for real-time chats, events, and support: Join the Discord Explore the Forum About Me I'm Sharda, a Gold Microsoft Learn Student Ambassador focused on cloud and AI. Find me on GitHub, Dev.to, Tech Community, and LinkedIn. In this blog series, I share takeaways from each week’s Model Mondays livestream.Model Mondays S2E9: Models for AI Agents
1. Weekly Highlights This episode kicked off with the top news and updates in the Azure AI ecosystem: GPT-5 and GPT-OSS Models Now in Azure AI Foundry: Azure AI Foundry now supports OpenAI’s GPT-5 lineup (including GPT-5, GPT-5 Mini, and GPT-5 Nano) and the new open-weight GPT-OSS models (120B, 20B). These models offer powerful reasoning, real-time agent tasks, and ultra-low latency Q&A, all with massive context windows and flexible deployment via the Model Router. Flux 1 Context Pro & Flux 1.1 Pro from Black Forest Labs: These new vision models enable in-context image generation, editing, and style transfer, now available in the Image Playground in Azure AI Foundry. Browser Automation Tool (Preview): Agents can now perform real web tasks—search, navigation, form filling, and more—via natural language, accessible through API and SDK. GitHub Copilot Agent Mode + Playwright MCP Server: Debug UIs with AI: Copilot’s agent mode now pairs with Playwright MCP Server to analyze, identify, and fix UI bugs automatically. Discord Community: Join the conversation, share your feedback, and connect with the product team and other developers. 2. Spotlight On: Azure AI Agent Service & Agent Catalog This week’s spotlight was on building and orchestrating multi-agent workflows using the Azure AI Agent Service and the new Agent Catalog. What is the Azure AI Agent Service? A managed platform for building, deploying, and scaling agentic AI solutions. It supports modular, multi-agent workflows, secure authentication, and seamless integration with Azure Logic Apps, OpenAPI tools, and more. Agent Catalog: A collection of open-source, ready-to-use agent templates and workflow samples. These include orchestrator agents, connected agents, and specialized agents for tasks like customer support, research, and more. Demo Highlights: Connected Agents: Orchestrate workflows by delegating tasks to specialized sub-agents (e.g., mortgage application, market insights). Multi-Agent Workflows: Design complex, hierarchical agent graphs with triggers, events, and handoffs (e.g., customer support with escalation to human agents). Workflow Designer: Visualize and edit agent flows, transitions, and variables in a modular, no-code interface. Integration with Azure Logic Apps: Trigger workflows from 1400+ external services and apps. 3. Customer Story: Atomic Work Atomic Work showcased how agentic AI can revolutionize enterprise service management, making employees more productive and ops teams more efficient. Problem: Traditional IT service management is slow, manual, and frustrating for both employees and ops teams. Solution: Atomic Work’s “Atom” is a universal, multimodal agent that works across channels (Teams, browser, etc.), answers L1/L2 questions, automates requests, and proactively assists users. Technical Highlights: Multimodal & Cross-Channel: Atom can guide users through web interfaces, answer questions, and automate tasks without switching tools. Data Ingestion & Context: Regularly ingests up-to-date documentation and context, ensuring accurate, current answers. Security & Integration: Built on Azure for enterprise-grade security and seamless integration with existing systems. Demo: Resetting passwords, troubleshooting VPN, requesting GitHub repo access—all handled by Atom, with proactive suggestions and context-aware actions. Atom can even walk users through complex UI tasks (like generating GitHub tokens) by “seeing” the user’s screen and providing step-by-step guidance. 4. Key Takeaways Here are the key learnings from this episode: Agentic AI is Production-Ready: Azure AI Agent Service and the Agent Catalog make it easy to build, deploy, and scale multi-agent workflows for real-world business needs. Modular, No-Code Workflow Design: The workflow designer lets you visually create and edit agent graphs, triggers, and handoffs—no code required. Open-Source & Extensible: The Agent Catalog provides open-source templates and welcomes community contributions. Real-World Impact: Solutions like Atomic Work show how agentic AI can transform IT, HR, and customer support, making organizations more efficient and employees more empowered. Community & Support: Join the Discord and Forum to connect, ask questions, and share your own agentic AI projects. Sharda's Tips: How I Wrote This Blog Writing this blog is like sharing my own learning journey with friends. I start by thinking about why the topic matters and how it can help someone new to Azure or agentic AI. I use simple language, real examples from the episode, and organize my thoughts with GitHub Copilot to make sure I cover all the important points. Here’s the prompt I gave Copilot to help me draft this blog: Generate a technical blog post for Model Mondays S2E9 based on the transcript and episode details. Focus on Azure AI Agent Service, Agent Catalog, and real-world demos. Explain the concepts for students, add a section on practical applications, and share tips for writing technical blogs. Make it clear, engaging, and useful for developers and students. After watching the video, I felt inspired to try out these tools myself. The way the speakers explained and demonstrated everything made me believe that anyone can get started, no matter their background. My goal with this blog is to help you feel the same way—curious, confident, and ready to explore what AI and Azure can do for you. If you have questions or want to share your own experience, I’d love to hear from you. Coming Up Next Week Next week: Document Processing with AI! Join us as we explore how to automate document workflows using Azure AI Foundry, with live demos and expert guests. 1️⃣ | Register For The Livestream – Aug 18, 2025 2️⃣ | Register For The AMA – Aug 22, 2025 3️⃣ | Ask Questions & View Recaps – Discussion Forum About Model Mondays Model Mondays is a weekly series designed to help you build your Azure AI Foundry Model IQ with three elements: 5-Minute Highlights – Quick news and updates about Azure AI models and tools on Monday 15-Minute Spotlight – Deep dive into a key model, protocol, or feature on Monday 30-Minute AMA on Friday – Live Q&A with subject matter experts from Monday livestream Want to get started? Register For Livestreams – every Monday at 1:30pm ET Watch Past Replays to revisit other spotlight topics Register For AMA – to join the next AMA on the schedule Recap Past AMAs – check the AMA schedule for episode specific links Join The Community Great devs don't build alone! In a fast-paced developer ecosystem, there's no time to hunt for help. That's why we have the Azure AI Developer Community. Join us today and let's journey together! Join the Discord – for real-time chats, events & learning Explore the Forum – for AMA recaps, Q&A, and Discussion! About Me I'm Sharda, a Gold Microsoft Learn Student Ambassador interested in cloud and AI. Find me on GitHub, Dev.to, Tech Community, and LinkedIn. In this blog series, I summarize my takeaways from each week's Model Mondays livestream.The Cloud Foundation for Safe Agentic AI
Why enterprise agents need more than a working prototype Most AI conversations start with the model. Which model should we use? Which framework? Which agent platform? Which demo can we build quickly enough to make the idea feel real? Those questions are not wrong, but they are rarely the first questions that matter in an enterprise environment. In real projects, the hard part usually appears after the first prototype works. The demo can answer a question, call a tool, retrieve a document, or update a record. Then someone asks whether it can be connected to production data, used by more teams, or allowed to trigger real actions. That is where the conversation changes. In the first part of this series, I looked at why many companies are less ready for agentic AI than they think. The blockers were practical and familiar: unclear business problems, immature processes, weak data foundations, and no clear owner when an AI system makes a poor recommendation or takes a wrong action. The message was simple: Before a company asks what agents can do, it needs to understand what it is ready to delegate. But business readiness is only the first layer. Even when the use case is clear, the process is understood, and leadership is aligned, another question appears. Is the platform ready to support agents safely? This is where Part 2 begins. Agentic AI does not behave like a normal application workload. A traditional application usually follows predefined paths. It receives a request, processes logic, returns a response, writes to a database, or calls an API. Agents introduce a different pattern. They reason over context, retrieve information, choose tools, trigger actions, interact with other services, and sometimes operate across multiple systems at once. That makes the surrounding cloud platform much more important. There is also a shadow AI angle to this. In many organizations, agent-like capabilities are already entering through SaaS platforms, vendor copilots, browser extensions, and productivity tools. These systems may not run inside the organization’s governed Azure subscriptions, but they can still interact with enterprise data and business workflows. If the official platform is not ready, teams will often find less governed ways to experiment anyway. That is not always malicious. Sometimes it is just people trying to solve their work with the tools available to them. The marketing analyst pasting customer data into a public chatbot because the official AI platform is six months away. The support team using a browser extension that summarizes tickets, without anyone realizing those tickets are also being sent to a third-party service. From a governance point of view, the effect is the same. Cloud readiness for agentic AI is not defined by access to cloud services or model endpoints alone. The real question is whether the platform can support controlled autonomy. Before enterprises can trust agents to act, the platform must be able to identify them, observe their behavior, restrict their permissions, enforce policy, and contain failure. Without that, an organization is not really deploying an intelligent assistant in a controlled way. It is introducing a workload that can interact with enterprise systems without anyone clearly watching what it does or being able to stop it. From business readiness to cloud readiness After the business foundation is clear, the next layer is the cloud foundation. A company may have a strong use case, executive support, and even a working prototype. But that does not mean it is ready to deploy agents in production. A prototype can run with broad access, manual supervision, loose logging, and a small group of test users. Production requires more discipline. It requires clear identity, controlled access, traceable activity, enforceable policy, and operational ownership. Cloud readiness for agentic AI comes down to four pillars, in this order: Identity-first architecture Observability Policy controls Platform constraints The order matters. 1. Identity-first architecture Identity comes first because nothing can be governed properly if it cannot be identified. In traditional cloud systems, we already learned this lesson with users, applications, service principals, managed identities, and workloads. Agents add another layer of non-human actors into the enterprise environment. If an agent can retrieve data, call tools, trigger workflows, or interact with business systems, it needs a clear identity. Without that foundation, governance becomes fragile. Teams may struggle to control what the agent can access, understand what it did, or determine who is accountable when something goes wrong. I have seen agents running in production where nobody could clearly say who owned them. They worked. Until they did not. Identity-first architecture means each agent or agentic workload should have a defined identity, ownership model, permission scope, and lifecycle. It should be clear whether the agent is acting on behalf of a user, acting as a service, or operating within a delegated boundary. This matters because permissions are not an implementation detail. They define the blast radius and accountability model of the system. In Azure environments, this is where Microsoft Entra ID and newer agent identity capabilities become important. As agents become more common across Azure AI Foundry, Copilot Studio, Microsoft 365, and custom frameworks, organizations need a way to understand which agents exist, who owns them, what they can access, and how their lifecycle is managed. Identity is not only about authentication. It is also about visibility, traceability, ownership, permission boundaries, and accountability. Agents should not remain hidden inside application logic or operate through shared identities. If they can retrieve data, call tools, or trigger actions, they need to be managed with the same care as any other production workload. 2. Observability Once identity is established, observability becomes the next pillar. Knowing that an agent exists is not enough. The platform must be able to show what the agent did. For normal applications, observability often focuses on service health, latency, failures, and resource usage. For agents, those signals still matter, but they are incomplete. Agent observability also needs to capture the execution path across model calls, retrieved context, orchestration steps, tool calls, policy decisions, approvals, denials, and final actions. This changes how we think about monitoring. With agentic systems, the question is not only whether a request succeeded or failed. Teams also need to understand the path that led to the outcome, the context used, the tools called, the policies applied, and the point where behavior changed. Without that visibility, it is difficult to investigate failures and improve reliability. This is also where observability starts to support governance, not just troubleshooting. Once teams can measure how agents behave, they can move toward KPI-based governance. That may include reliability, escalation rates, policy denials, grounding quality, tool-call failures, cost per interaction, latency, and business outcome metrics. Without this measurement layer, maturity remains mostly opinion-based. With it, governance becomes evidence-based. In Azure, Azure Monitor is the obvious starting point. Together with services such as Application Insights and Log Analytics, it provides the telemetry foundation needed to understand how AI workloads behave in production. For agentic systems, this usually requires combining platform telemetry with application-level traces from orchestration, retrieval, model calls, policy decisions, and tool execution. This visibility is what makes continuous improvement possible. It is also what allows governance to mature from “we think the agent is behaving correctly” to “we can measure how the agent behaves over time.” Small difference. Large consequence. 3. Policy controls The third pillar is policy controls. This comes after identity and observability because policy needs both. Identity defines who or what the rule applies to. Observability helps teams understand whether the rule is effective, bypassed, misconfigured, or too restrictive. Policy controls define the boundaries for what agents are allowed to do. They determine how agents access data, which tools they can use, which environments are in scope, when approval is required, and when an action or response should be blocked. The key point is simple: Prompts can guide behavior, but they are not a reliable enforcement layer. For enterprise systems, policy needs to be external, testable, auditable, and enforceable. This becomes especially important because agents may operate across multiple systems. An agent may retrieve information from one source, reason over the result, call a tool, update a ticket, send a message, or trigger a workflow. Each step may appear safe in isolation, while the full chain creates risk. Policy controls provide boundaries around that chain. In Azure, this starts at the cloud governance layer. Azure landing zones, management group structures, and Azure Policy can help define where AI workloads are deployed, how environments are separated, and which rules apply consistently across subscriptions. At runtime, Azure AI Content Safety can help detect harmful content, prompt attacks, unsafe interactions, or outputs that drift away from the intended task. For tool and API access, Azure API Management can also be used as a controlled gateway between agents and downstream systems. This can support centralized authentication, throttling, mediation, logging, and policy enforcement. It is not mandatory in every design, but it is a useful option when agents need governed access to APIs instead of direct backend connectivity. The goal is not to create friction for the sake of control. The goal is to make sure the agent operates inside boundaries that are defined outside the prompt and outside the model response. 4. Platform constraints The fourth pillar is platform constraints. This area often receives less attention early in the project, but it strongly shapes whether an agentic system can operate safely and reliably in production. These constraints include network isolation, private connectivity, data residency, regional availability, quota limits, model throughput, latency, logging retention, integration boundaries, cost behavior, and operational ownership. They may seem like implementation details during early design discussions, but they often determine whether the system can actually run in production. For agentic workloads, these constraints also shape where experimentation happens. Sandboxed environments, isolated subscriptions, limited tool access, and controlled test data can help teams evaluate agent behavior before exposing it to production systems. This becomes even more important when agents are allowed to generate code, call external tools, or execute actions that may not be fully trusted at design time. Platform constraints are where the earlier pillars meet implementation reality. Identity affects how agents connect to services. Observability affects logging cost, retention, and investigation capability. Policy affects routing, network design, tool exposure, and user experience. By the time an agentic system reaches production, these constraints are no longer background details. They become design boundaries. In Azure, this is where landing zone design, private networking, regional planning, quota management, cost management, and operational runbooks matter. Azure landing zones, private endpoints, private DNS, Azure Firewall, NSGs, and controlled network paths all influence whether the agent architecture can move from prototype to production without being redesigned halfway through. And yes, that redesign usually happens at the least convenient moment. Architecture has a sense of humor. Not a kind one. From principles to Azure capabilities The four pillars are not only architectural principles. They need to be translated into platform capabilities, operating practices, and governance controls. In practice, controlled agent deployment is rarely achieved by a single product or service. It requires multiple layers working together. Identity, monitoring, policy, networking, runtime safety, API exposure, and operational controls all play a part. Azure provides several services and patterns that can help implement these controls, but there is no fixed blueprint that applies to every organization. The right combination depends on the use case, regulatory requirements, existing landing zone design, integration landscape, and the level of autonomy expected from the agent. The examples below should be seen as a practical toolset, not as a mandatory checklist. Pillar Goal Example Azure capabilities Identity-first architecture Make agents visible, owned, permissioned, and governable as enterprise workloads. Microsoft Entra ID, Microsoft Entra Agent ID, managed identities, service principals, workload identities, access reviews, Conditional Access, Privileged Identity Management Observability Understand runtime behavior, trace execution paths, investigate failures, and improve reliability. Azure Monitor, Application Insights, Log Analytics, Azure AI Foundry tracing, diagnostic settings, distributed tracing, correlation IDs, application-level telemetry Policy controls Enforce boundaries around access, actions, content safety, APIs, and governance. Azure landing zones, management groups, Azure Policy, Azure AI Content Safety, Prompt Shields, Microsoft Purview, Azure API Management, RBAC, approval flows Platform constraints Operate within real cloud boundaries such as networking, region, quota, compliance, and operations. Azure landing zones, private endpoints, private DNS, private networking, Azure Firewall, NSGs, quota planning, regional architecture, cost management The purpose of this mapping is not to suggest that Azure has one single service for each pillar. It does not. The practical goal is to combine the right services and patterns so the platform can identify agents, monitor their behavior, enforce boundaries, and operate within known cloud constraints. Conclusion Agentic AI does not become enterprise-ready simply because a model is available, a prototype works, or a business sponsor is excited. The real question is whether the surrounding cloud foundation can support agents that act within boundaries the platform actually enforces. Together, these pillars move the discussion from building an agent to preparing the environment in which the agent can operate responsibly. That distinction is important. A prototype can rely on broad access, limited logging, and close manual supervision. A production system needs clearer boundaries around ownership, access, traceability, and control. This is also where the series moves naturally into Part 3. Once the business foundation is clear and the cloud foundation is in place, the next challenge is the design of the agent itself. The cloud foundation matters here because it provides the controlled environment in which agents can be tested, limited, and observed before they are trusted with broader enterprise access. For more advanced scenarios, that also includes sandboxing patterns for generated code, tool execution, and untrusted actions. In Part 3, I will move closer to implementation and look at how to design an enterprise-ready agent. That means defining the agent’s scope, grounding it with reliable knowledge, deciding which tools it can use, designing safe execution loops, adding human oversight where it matters, and thinking carefully about when a single agent is enough versus when multi-agent coordination is justified. That is where agentic AI starts becoming more than an idea. And, as usual, that is also where the architecture starts to matter. This article is part of my Agentic AI readiness series and was also published on Medium.
