Quality and evaluation framework for successful AI apps and agents in Microsoft Marketplace
Why quality in AI is different — and why it matters for Marketplace Traditional software quality spans many dimensions — from performance and reliability to correctness and fault tolerance — but once those characteristics are specified and validated, system behavior is generally stable and repeatable. Quality is assessed through correctness, reliability, performance, and adherence to specifications. AI apps and agents change this equation. Their behavior is inherently non-deterministic and context‑dependent. The same prompt can produce different responses depending on model version, retrieval context, prior interactions, or environmental conditions, which is why non-deterministic AI testing is required to evaluate it reliably. For agentic systems, quality also depends on reasoning paths, tool selection, and how decisions unfold across multiple steps — not just on the final output. This means an AI app can appear functional while still falling short on quality: producing responses that are inconsistent, misleading, misaligned with intent, or unsafe in edge cases. Without a structured evaluation framework, these gaps often surface only in production — in customer environments, after trust has already been extended. For Microsoft Marketplace, this distinction matters. Buyers expect AI apps and agents to behave predictably, operate within clear boundaries, and remain fit for purpose as they scale. AI quality evaluation turns those expectations into something observable — and that visibility is what determines Marketplace readiness. You can always get a curated step-by-step guidance through building, publishing and selling apps for Marketplace through App Advisor. This post is part of a series on building and publishing well-architected AI apps and agents in Microsoft Marketplace. The series focuses on AI apps and agents that are architected, hosted, and operated on Azure, with guidance aligned to building and selling solutions through Microsoft Marketplace. How quality measurement shapes Marketplace readiness AI apps and agents that can demonstrate quality — with documented evaluation frameworks, defined release criteria, and evidence of ongoing measurement — are easier to evaluate, trust, and adopt. Quality evidence reduces friction during Marketplace review, clarifies expectations during customer onboarding, and supports long-term confidence in production. When quality is visible and traceable, the conversation shifts from "does this work?" to "how do we scale it?" — which is exactly where publishers want to be. Publishers who treat quality as a first-class discipline build the foundation for safe iteration, customer retention, and sustainable growth through Microsoft Marketplace. That foundation is built through the decisions, frameworks, and evaluation practices established long before a solution reaches review. What "quality" means for AI apps and agents Quality for AI apps and agents is not a single metric — it spans interconnected dimensions that together define whether a system is doing what it was built to do, for the people it was built to serve. The HAX Design Library — Microsoft's collection of human-AI interaction design patterns — offers practical guidance for each one. These dimensions must be defined before evaluation begins. You can only measure what you have first described. Accuracy and relevance — does the output reflect the right answer, grounded in the right context? HAX patterns Make clear what the system can do (G1) and notify users when the AI is uncertain (G10) help publishers design systems where accuracy is visible and outputs are understood in the right context — not treated as universally authoritative. Safety and alignment — does the output stay within intended use, without harmful, biased, or policy-violating content? HAX patterns Mitigate social biases (G6) and Support efficient correction (G9) help ensure outputs stay within acceptable boundaries — and that users can identify and address issues before they cause downstream harm. Consistency and reliability — does the system behave predictably across users, sessions, and environments? HAX patterns Remember recent interactions (G12) and notify users about changes (G18) keep behavior coherent within sessions and ensure updates to the model or prompts are never silently introduced. Fitness for purpose — does the system do what it was designed to do, for the people it was designed to serve, in the conditions it will actually operate in? HAX patterns make clear how well the system can do what it does (G2) and Act on the user's context and goals (G4) ensure the system responds to what users actually need — not just what they literally typed. These dimensions work together — and gaps in any one of them will surface in production, often in ways that are difficult to trace without a deliberate evaluation framework. Designing an evaluation framework before you ship Evaluation frameworks should be built alongside the solution. At the end, gaps are harder and costlier to close. The discipline mirrors the design-in approach that applies to security and governance: decisions made early shape what is measurable, what is improvable, and what is ready to ship. A well-structured evaluation framework defines five things: What to measure — the quality dimensions that matter most for this solution and its intended use cases. For AI apps and agents, this typically includes task adherence, response coherence, groundedness, and safety — alongside the fitness-for-purpose dimensions defined in the previous section. How to measure it — the methods, tools, and benchmarks used to assess quality consistently. Effective evaluation combines AI-assisted evaluators (which use a model as a judge to score outputs), rule-based evaluators (which apply deterministic logic), and human review for edge cases and safety-relevant responses that automated methods cannot fully capture. Who evaluates — the right combination of automated metrics, human review, and structured customer feedback. No single method is sufficient; the framework defines how each is applied and when human judgment takes precedence. When to evaluate — at defined milestones: during development to establish a baseline, pre-release to validate against acceptance thresholds, at rollout to catch regression, and continuously in production to detect drift as models, prompts, and data evolve. What triggers re-evaluation — model updates, prompt changes, new data sources, tool additions, or meaningful shifts in customer usage patterns. Re-evaluation should be a scheduled and triggered discipline, not an ad hoc response to visible failures. The framework becomes a shared artifact — used by the publisher to release safely, and by customers to understand what quality commitments they are adopting when they deploy the solution in their environment. Evaluate your AI agents - Microsoft Foundry | Microsoft Learn Evaluation methods for AI apps and agents Quality must be assessed across complementary AI evaluation methods — each designed to surface a different category of risk, at a different stage of the solution lifecycle. Automated metric evaluation — evaluators assess agent responses against defined criteria at scale. Some use AI models as judges to score outputs like task adherence, coherence, and groundedness; others apply deterministic rules or text similarity algorithms. Automated evaluation is most effective when acceptance thresholds are defined upfront — for example, a minimum task adherence pass rate before a release proceeds. Safety evaluation — a dedicated evaluation category that identifies potential content risks, policy violations, and harmful outputs in generated responses. Safety evaluators should run alongside quality evaluators, not as a separate afterthought. Human-in-the-loop evaluation — structured expert review of edge cases, borderline outputs, and safety-relevant responses that automated metrics cannot fully capture. Human judgment remains essential for interpreting context, intent, and impact. AI agent red-teaming and adversarial testing — probing the system with challenging, unexpected, or intentionally misused inputs (including prompt injection attempts and tool misuse) to surface failure modes before customers encounter them. Microsoft provides dedicated AI red teaming guidance for agent-based systems. Customer feedback loops — structured collection of real-world signals from users interacting with the system in production. Production feedback closes the gap between what was tested and what customers actually experience. Each method has a distinct role. The evaluation framework defines when and how each is applied — and which results are required before a release proceeds, a change is accepted, or a capability is expanded. Defining release criteria and ongoing quality gates Quality evaluation only drives improvement when it is connected to clear release criteria, often enforced through an LLMOps quality gate. In an LLMOps model, those criteria are automated gates embedded directly into the CI/CD pipeline, applied consistently at every stage of the release cycle. In continuous integration (CI), automated evaluations run with every change — whether that change is a prompt update, a model version, a new tool, or a data source modification. CI gates catch regressions early, before they reach customers, by validating outputs against predefined quality thresholds for task adherence, coherence, groundedness, and safety. In continuous deployment (CD), quality gates determine whether a build is eligible to proceed. Release criteria should define: Minimum acceptable thresholds for each quality dimension — a release does not proceed until those thresholds are met Known failure modes that block release outright versus those that are tracked, monitored, and accepted within defined risk tolerances Deployment constraints — conditions under which a release is paused, rolled back, or progressively expanded to a subset of users before full rollout Ongoing evaluation must be scheduled and triggered. As models, prompts, tools, and customer usage patterns evolve, the baseline shifts. LLMOps treats re-evaluation as a continuous discipline: run evaluations, identify weak areas, adjust, and re-evaluate before changes propagate. This connects directly to governance. Quality evidence — the record of what was measured, when, and against what criteria — is part of the audit trail that makes AI behavior accountable, explainable, and trustworthy over time. For more on the governance foundation this builds on, see Governing AI apps and agents for Marketplace readiness. Quality across the publisher-customer boundary Clear quality ownership reduces friction at onboarding, builds confidence during operation, and protects both parties when behavior deviates. In the Marketplace context, quality is a shared responsibility — but the boundaries are distinct. Publishers are responsible for: Designing and running the evaluation framework during development and release Defining quality dimensions and thresholds that reflect the solution's intended use Providing customers with transparency into what quality means for this solution — without exposing proprietary prompts or internal logic Customers are responsible for: Validating that the solution performs appropriately in their specific environment, with their data and their users Configuring feedback and monitoring mechanisms that surface quality signals in their tenant Treating quality evaluation as a shared ongoing responsibility, not a one-time publisher guarantee When both sides understand their role, quality stops being a handoff and becomes a foundation — one that supports adoption, sustains trust, and enables both parties to respond confidently when behavior shifts. What's next in the journey A strong quality framework sets the baseline — but keeping that quality visible as solutions scale is its own discipline. The next posts in this series explore what comes after the framework is in place: API resilience, performance optimization, and operational observability for AI apps and agents running in production environments. See the next post in the series: Designing a reliable environment strategy for Microsoft Marketplace AI apps and agents | Microsoft Community Hub. Key resources See curated, step-by-step guidance to help you build, publish, or sell your app or agent (no matter where you start) in App Advisor Quick-Start Development Toolkit can connect you with code templates for AI solution patterns Microsoft AI Envisioning Day Events How to build and publish AI apps and agents for Microsoft Marketplace Get over $126K USD in benefits and technical consultations to help you replicate and publish your app with ISV SuccessGoverning AI apps and agents for Marketplace
Governing AI apps and agents Governance is what turns powerful AI functionality into a solution that enterprises can confidently adopt, operate, and scale—an essential part of AI governance for agents. It establishes clear responsibility for actions taken by the system, defines explicit boundaries for acceptable behavior, and creates mechanisms to review, explain, and correct outcomes over time. Without this structure, AI systems can become difficult to manage as they grow more connected and autonomous. For publishers, governance is how trust is earned—and sustained—in enterprise environments, enabling responsible AI operations. It signals that AI behavior is intentional, accountable, and aligned with customer expectations, not left to inference or assumption. As AI apps and agents operate across users, data, and systems, risk shifts away from what a model can generate and toward how its behavior is governed in real‑world conditions. Marketplace readiness reflects this shift, defined less by raw capability and more by control, accountability, trust, and adherence to AI compliance standards for publishing. You can always get a curated step-by-step guidance through building, publishing and selling apps for Marketplace through App Advisor. This post is part of a series on building and publishing well-architected AI apps and agents in Microsoft Marketplace. The series focuses on AI apps and agents that are architected, hosted, and operated on Azure, with guidance aligned to building and selling solutions through Microsoft Marketplace. What governance means for AI apps and agents Governance in AI systems is operational and continuous. It is not limited to documentation, checklists, or periodic reviews — it shapes how an AI app or agent behaves while it is running in real customer environments. For AI apps and agents, governance spans three closely connected dimensions: Policy What the system is allowed to do, what data it is allowed to access, what is restricted, and what is explicitly prohibited. Enforcement How those policies are applied consistently in production, even as context, inputs, and conditions change. Evidence How decisions and actions are traced, reviewed, and audited over time. Governance works when intent, behavior, and proof move together — turning expectations into outcomes that can be trusted and examined. These dimensions are interdependent. Policy without enforcement is aspiration. Enforcement without evidence is unverifiable. Governance in action Governance becomes real when responsibility is explicit. For AI apps and agents, this starts with clarity around who is responsible for what: Who the agent acts for — and how its use protects business value Ensuring the agent is used for its intended purpose, produces measurable value, and is not misused, over‑extended, or operating outside approved business contexts. Who owns data access and data quality decisions Governing how the agent consumes and produces data, whether access is appropriate, and whether the data used or generated is reliable, accurate, and aligned with business and integrity expectations. Who is accountable for outcomes when behavior deviates Defining responsibility when the agent’s behavior creates risk, degrades value, or produces unexpected outcomes — so corrective action is timely, intentional, and owned. When governance is left vague or undefined, accountability gaps surface and agent actions become difficult to justify and explain across the publisher, the customer, and the solution itself. In this model, responsibility is shared but distinct. The publisher is responsible for designing and implementing the governance capabilities within the solution — defining boundaries, enforcement points, and evidence mechanisms that protect business value by default. Marketplace customers expect to understand who is accountable before they adopt an AI solution, not after an incident forces the question. The customer is responsible for configuring, operating, and applying those capabilities within their own environment, aligning them to internal policies, risk tolerance, and day‑to‑day use. Governance works when both roles are clear: the publisher provides the structure, and the customer brings it to life in practice. Data governance for AI: beyond storage and access For Marketplace‑ready AI apps and agents, data governance must account for where data moves, not just where it resides. Understanding how data flows across systems, tools, and tenants is essential to maintaining trust as solutions scale. Data governance for AI apps and agents extends beyond where data is stored. These systems introduce new artifacts that influence behavior and outcomes, including prompts and responses, retrieval context and embeddings, and agent‑initiated actions and tool outputs. Each of these elements can carry sensitive information and shape downstream decisions. Effective data governance for AI apps and agents requires clear structure: Explicit data ownership — defining who owns the data and under what conditions it can be accessed or used Access boundaries and context‑aware authorization — ensuring access decisions reflect identity, intent, and environment, not just static permissions Retention, auditability, and deletion strategies — so data use remains traceable and aligned with customer expectations over time Relying on prompts or inferred intent to determine access is a governance gap, not a shortcut. Without explicit controls, data exposure becomes difficult to predict or explain. Runtime policy enforcement in production Policies are stress tested when the agent is responding to real prompts, touching real data, and taking actions that carry real consequences. For software companies building AI apps and agents for Microsoft Marketplace, runtime policy enforcement is also how you keep the system fit for purpose: aligned to its intended use, supported by evidence, and constrained when conditions change. At runtime, governance becomes enforceable through three clear lanes of behavior: Decisions that require human approval Use approval gates for higher‑impact steps (for example: executing a write operation, sending an external request, or performing an irreversible workflow). This protects the business value of the agent by preventing “helpful” behavior from turning into misuse. Actions that can proceed automatically — within defined limits Automation is earned through clarity: define the agent’s intended uses and keep tool access, data access, and action scope anchored to those uses. Fit‑for‑purpose isn’t a feeling — it’s something you support with defined performance metrics, known error types, and release criteria that you measure and re‑measure as the system runs. Behaviors that are never permitted — regardless of context or intent Block classes of behavior that violate policy (including jailbreak attempts that try to override instructions, expand tool scope, or access disallowed data). When an intended use is not supported by evidence — or new evidence shows it no longer holds — treat that as a governance trigger: remove or revise the intended use in customer‑facing materials, notify customers as appropriate, and close the gap or discontinue the capability. To keep runtime enforcement meaningful over time, pair it with ongoing evaluation: document how you’ll measure performance and error patterns, run those evaluations pre‑release and continuously, and decide how often re‑evaluation is needed as models, prompts, tools, and data shift. This is what keeps autonomy intentional. It allows AI apps and agents to operate usefully and confidently, while ensuring behavior remains aligned with defined expectations — and backed by evidence — as systems evolve and scale. Auditability, explainability, and evidence Guardrails are the points in the system where governance becomes observable: where decisions are evaluated, actions are constrained, and outcomes are recorded. As described in Designing AI guardrails for apps and agents in Marketplace, guardrails shape how AI systems reason, access data, and take action — consistently and by default. Guardrails may be embedded within the agent itself or implemented as a separate supervisory layer — another agent or policy service — that evaluates actions before they proceed. Guardrail responses exist on a spectrum. Some enforce in the moment — blocking an action or requiring approval before it proceeds — while others generate evidence for post‑hoc review, supported by audit logging for AI agents. Marketplace‑ready AI apps and agents could implement both, with the response mode matched to the severity, reversibility, and business impact of the action in question. These expectations align with the governance and evidence requirements outlined in the Microsoft Responsible AI Standard v2 General Requirements. In practice, guardrails support auditability and explainability by: Constraining behavior at design time Establishing clear defaults around what the system can and cannot do, so intended use is enforced before the system ever reaches production. Evaluating actions at runtime Making decisions visible as they happen — which tools were invoked, which data was accessed, and why an action was allowed to proceed or blocked. When governance is unclear, even strong guardrails lose their effectiveness. Controls may exist, but without clear intent they become difficult to justify, unevenly applied across environments, or disconnected from customer expectations. Over time, teams lose confidence not because the system failed, but because they can’t clearly explain why it behaved the way it did. When governance and guardrails are aligned, the result is different. Behavior is intentional. Decisions are traceable. Outcomes can be explained without guesswork. Auditability stops being a reporting exercise and becomes a natural byproduct of how the system operates day to day. Aligning governance with Marketplace expectations Governance for AI apps and agents must operate continuously, across all in‑scope environments — in both the publisher’s and the customer’s tenants. Marketplace solutions don’t live in a single boundary, and governance cannot stop at deployment or certification. Runtime enforcement is what keeps governance active as systems run and evolve. In practice, this means: Blocking or constraining actions that violate policy — such as stopping jailbreak attempts that try to override system instructions, escalate tool access, or bypass safety constraints through crafted prompts Adapting controls based on identity, environment, and risk — applying stricter limits when an agent acts across tenants, accesses sensitive data, or operates with elevated permissions Aligning agent behavior with enterprise expectations in real time — ensuring actions taken on behalf of users remain within approved roles, scopes, and approval paths These controls matter because AI behavior is dynamic. The same agent may behave differently depending on context, inputs, and downstream integrations. Governance must be able to respond to those shifts as they happen. Runtime enforcement is distinct from monitoring. Enforcement determines what is allowed to continue. Monitoring explains what happened once it’s already done. Marketplace‑ready AI solutions need both, but governance depends on enforcement to keep behavior aligned while it matters most. Operational health through auditability and traceability Operational health is the combination of traceability (what happened) and intelligibility (how to use it responsibly). When both are present, governance becomes a quality signal customers can feel day to day — not because you promised it, but because the system consistently behaves in ways they can understand and trust. Healthy AI apps and agents are not only traceable — they are intelligible in the moments that matter. For Marketplace customers, operational trust comes from being able to understand what the system is intended to do, interpret its behavior well enough to make decisions, and avoid over‑relying on outputs simply because they are produced confidently. A practical way to ground this is to be explicit about who needs to understand the system: Decision makers — the people using agent outputs to choose an action or approve a step Impacted users — the people or teams affected by decisions informed by the system’s outputs Once those stakeholders are clear, governance shows up as three operational promises you can actually support: Clarity of intended use Customers can see what the agent is designed to do (and what it is not designed to do), so outputs are used in the right contexts. Interpretability of behavior When an agent produces an output or recommendation, stakeholders can interpret it effectively — not perfectly, but reasonably well — with the context they need to make informed decisions. Protection against automation bias Your UX, guidance, and operational cues help customers stay aware of the natural tendency to over‑trust AI output, especially in high‑tempo workflows. This is where auditability and traceability become more than logs. Well governed AI systems should still answer: Who initiated an action — a user, an agent acting on their behalf, or an automated workflow What data was accessed — under which identity, scope, and context What decision was made, and why — especially when downstream systems or people are affected The logs should show evidence that stakeholders can interpret those outputs in realistic conditions — and there is a method to evaluate this, with clear criteria for release and ongoing evaluation as the solution evolves. Explainability still needs balance. Customers deserve transparency into intended use, behavior boundaries, and how to interpret outcomes — without requiring you to expose proprietary prompts, internal logic, or implementation details. For more information on securing your AI apps and agents, visit Securing AI apps and agents on Microsoft Marketplace | Microsoft Community Hub. What's next in the journey Governance creates the conditions for AI apps and agents to operate with confidence over time. With clear policies, enforcement, and evidence in place, publishers are better prepared to focus on operational maturity — how solutions are observed, maintained, and evolved safely in production. The next post explores what it takes to keep AI apps and agents healthy as they run, change, and scale in real customer environments. See the next post in the series: Quality and evaluation framework for successful AI apps and agents in Microsoft Marketplace | Microsoft Community Hub. Key resources See curated, step-by-step guidance to help you build, publish, or sell your app or agent (no matter where you start) in App Advisor Quick-Start Development Toolkit can connect you with code templates for AI solution patterns Microsoft AI Envisioning Day Events How to build and publish AI apps and agents for Microsoft Marketplace Get over $126K USD in benefits and technical consultations to help you replicate and publish your app with ISV SuccessDesigning AI guardrails for apps and agents in Marketplace
Why guardrails are essential for AI apps and agents AI apps and agents introduce capabilities that go beyond traditional software. They reason over natural language, interact with data across boundaries, and—in the case of agents—can take autonomous actions using tools and APIs. Without clearly defined guardrails, these capabilities can unintentionally compromise confidentiality, integrity, and availability, the foundational pillars of information security. From a confidentiality perspective, AI systems often process sensitive prompts, contextual data, and outputs that may span customer tenants, subscriptions, or external systems. Guardrails ensure that data access is explicit, scoped, and enforced—rather than inferred through prompts or emergent model behavior. From an availability perspective, AI apps and agents can fail in ways traditional software does not — such as runaway executions, uncontrolled chains of tool calls, or usage spikes that drive up cost and degrade service. Guardrails address this by setting limits on how the system executes, how often it calls tools, and how it behaves when something goes wrong. For Marketplace-ready AI apps and agents, guardrails are foundational design elements that balance innovation with security, reliability, and responsible AI practices. By making behavioral boundaries explicit and enforceable, guardrails enable AI systems to operate safely at scale—meeting enterprise customer expectations and Marketplace requirements from day one. You can always get a curated step-by-step guidance through building, publishing and selling apps for Marketplace through App Advisor. This post is part of a series on building and publishing well-architected AI apps and agents in Microsoft Marketplace. The series focuses on AI apps and agents that are architected, hosted, and operated on Azure, with guidance aligned to building and selling solutions through Microsoft Marketplace. Using Open Worldwide Application Security Project (OWASP) GenAI Top 10 as a guardrail design lens The OWASP GenAI Top 10 provides a practical framework for reasoning about AI‑specific risks that are not fully addressed by traditional application security models. It helps teams identify where assumptions about trust, input handling, autonomy, and data access are most likely to break down in AI‑driven systems. However, not all OWASP risks apply equally to every AI app or agent. Their relevance depends on factors such as: Agent autonomy, including whether the system can take actions without human approval Data access patterns, especially cross‑tenant, cross‑subscription, or external data retrieval Integration surface area, meaning the number and type of tools, APIs, and external systems the agent connects to Because of this variability, OWASP should not be treated as a checklist to implement wholesale. Doing so can lead teams to over‑engineer controls in low‑risk areas while leaving critical gaps in places where autonomy, data movement, or tool execution create real exposure. Instead, OWASP is most effective when used as a design lens — to inform where guardrails are needed and what behaviors require explicit boundaries. Understanding risks and enforcing boundaries are two different things. OWASP tells you where to look; guardrails are what you actually build. The goal is not to eliminate all risk, but to use OWASP insights to design selective, intentional guardrails that align with the system's architecture, autonomy, and operating context. Translating AI risks into architectural guardrails OWASP GenAI Top 10 helps identify where AI systems are vulnerable, but guardrails are what make those risks enforceable in practice. Guardrails are most effective when they are implemented as architectural constraints—designed into the system—rather than as runtime patches added after risky behavior appears. In AI apps and agents, many risks emerge not from a single component, but from how prompts, tools, data, and actions interact. Architectural guardrails establish clear boundaries around these interactions, ensuring that risky behavior is prevented by design rather than detected too late. Common guardrail categories map naturally to the types of risks highlighted in OWASP: Input and prompt constraints Address risks such as prompt injection, system prompt leakage, and unintended instruction override by controlling how inputs are structured, validated, and combined with system context. Action and tool‑use boundaries Mitigate risks related to excessive agency and unintended actions by explicitly defining which tools an AI app or agent can invoke, under what conditions, and with what scope. Data access restrictions Reduce exposure to sensitive information disclosure and cross‑boundary leakage by enforcing identity‑aware, context‑aware access to data sources rather than relying on prompts to imply intent. AI Output validation and moderation Help contain risks such as misinformation, improper output handling, or policy violations by treating AI output as untrusted and subject to validation before it is acted on or returned to users. What matters most is where these guardrails live in the architecture. Effective guardrails sit at trust boundaries—between users and models, models and tools, agents and data sources, and control planes and data planes. When guardrails are embedded at these boundaries, they can be applied consistently across environments, updates, and evolving AI capabilities. By translating identified risks into architectural guardrails, teams move from risk awareness to behavioral enforcement that supports safe AI agent operation. This shift is foundational for building AI apps and agents that can operate safely, predictably, and at scale in Marketplace environments. Design‑time guardrails: shaping allowed behavior before deployment The OWASP GenAI Top 10 provides a practical framework for reasoning about AI specific risks that are not fully addressed by traditional application security models. It helps teams identify where assumptions about trust, input handling, autonomy, and data access are most likely to break down in AI driven systems. However, not all OWASP risks apply equally to every AI app or agent. Their relevance depends on factors such as: Agent autonomy, including whether the system can take actions without human approval Data access patterns, especially cross-tenant, cross subscription, or external data retrieval Integration surface area, meaning the number and type of tools, APIs, and external systems the agent connects to Because of this variability, OWASP should not be treated as a checklist to implement wholesale. Doing so can lead teams to over engineer controls in low risk areas while leaving critical gaps in places where autonomy, data movement, or tool execution create real exposure. Instead, OWASP is most effective when used as a design lens — to inform where guardrails are needed and what behaviors require explicit boundaries. Understanding risks and enforcing boundaries are two different things. OWASP tells you where to look; guardrails are what you actually build. The goal is not to eliminate all risk, but to use OWASP insights to design selective, intentional guardrails that align with the system's architecture, autonomy, and operating context. Runtime guardrails: enforcing boundaries as systems operate For Marketplace publishers, the key distinction between monitoring and runtime guardrails is simple: Monitoring tells you what happened after the fact. Runtime guardrails are inline controls—part of runtime AI safety controls—that can block, pause, throttle, or require approval before an action completes. If you want prevention, the control has to sit in the execution path. At runtime, guardrails should constrain three areas: Agent decision paths (prevent runaway autonomy) Cap planning and execution. Limit the agent to a maximum number of steps per request, enforce a maximum wall‑clock time, and stop repeated loops. Apply circuit breakers. Terminate execution after a specified number of tool failures or when downstream services return repeated throttling errors, reinforcing autonomous agent limits. Require explicit escalation. When the agent’s plan shifts from “read” to “write,” pause and require approval before continuing. Tool invocation patterns (control what gets called, how, and with what inputs) Enforce allowlists. Allow only approved tools and operations, and block any attempt to call unregistered endpoints. Validate parameters. Reject tool calls that include unexpected tenant identifiers, subscription scopes, or resource paths. Throttle and quota. Rate‑limit tool calls per tenant and per user, and cap token/tool usage to prevent cost spikes and degraded service. Cross‑system actions (constrain outbound impact at the boundary you control) Runtime guardrails cannot “reach into” external systems and stop independent agents operating elsewhere. What publishers can do is enforce policy at your solution’s outbound boundary: the tool adapter, connector, API gateway, or orchestration layer that your app or agent controls. Concrete examples include: Block high‑risk operations by default (delete, approve, transfer, send) unless a human approves. Restrict write operations to specific resources (only this resource group, only this SharePoint site, only these CRM entities). Require idempotency keys and safe retries so repeated calls do not duplicate side effects. Log every attempted cross‑system write with identity, scope, and outcome, and fail closed when policy checks cannot run. Done well, runtime guardrails produce evidence, not just intent. They show reviewers that your AI app or agent enforces least privilege, prevents runaway execution, and limits blast radius—even when the model output is unpredictable. Guardrails across data, identity, and autonomy boundaries Guardrails don't work in silos. They are only effective when they align across the three core boundaries that shape how an AI app or agent operates — identity, data, and autonomy. Guardrails must align across: Identity boundaries (who the agent acts for) — represent the credentials the agent uses, the roles it assumes, and the permissions that flow from those identities. Without clear identity boundaries, agent actions can appear legitimate while quietly exceeding the authority that was actually intended. Data boundaries (what the agent can see or retrieve) — ensuring access is governed by explicit authorization and context, not by what the model infers or assumes. A poorly scoped data boundary doesn't just create exposure — it creates exposure that is hard to detect until something goes wrong. Autonomy boundaries (what the agent can decide or execute) — defining which actions require human approval, which can proceed automatically, and which are never permitted regardless of context. Autonomy without defined limits is one of the fastest ways for behavior to drift beyond what was ever intended. When these boundaries are misaligned, the consequences are subtle but serious. An agent may act under the authority of one identity, access data scoped to another, and execute with broader autonomy than was ever granted — not because a single control failed, but because the boundaries were never reconciled with each other. This is how unintended privilege escalation happens in well-intentioned systems. Balancing safety, usefulness, and customer trust Getting guardrails right is less about adding controls and more about placing them well. Too restrictive, and legitimate workflows break down, safe autonomy shrinks, and the system becomes more burden than benefit. Too permissive, and the risks accumulate quietly — surfacing later as incidents, audit findings, or eroded customer trust. Effective guardrails share three characteristics that help strike that balance: Transparent — customers and operators understand what the system can and cannot do, and why those limits exist Context-aware — boundaries tighten or relax based on identity, environment, and risk, without blocking safe use Adjustable — guardrails evolve as models and integrations change, without compromising the protections that matter most When these characteristics are present, guardrails naturally reinforce the foundational principles of information security — protecting confidentiality through scoped data access, preserving integrity by constraining actions to authorized paths, and supporting availability by preventing runaway execution and cascading failures. How guardrails support Marketplace readiness For AI apps and agents in Microsoft Marketplace, guardrails are a practical enabler — not just of security, but of the entire Marketplace journey. They make complex AI systems easier to evaluate, certify, and operate at scale. Guardrails simplify three critical aspects of that journey: Security and compliance review — explicit, architectural guardrails give reviewers something concrete to assess. Rather than relying on documentation or promises, behavior is observable and boundaries are enforceable from day one. Customer onboarding and trust — when customers can see what an AI system can and cannot do, and how those limits are enforced, adoption decisions become easier and time to value shortens. Clarity is a competitive advantage. Long-term operation and scale — as AI apps evolve and integrate with more systems, guardrails keep the blast radius contained and prevent hidden privilege escalation paths from forming. They are what makes growth manageable. Marketplace-ready AI systems don't describe their guardrails — they demonstrate them. That shift, from assurance to evidence, is what accelerates approvals, builds lasting customer trust, and positions an AI app or agent to scale with confidence. What’s next in the journey Guardrails establish the foundation for safe, predictable AI behavior — but they are only the beginning. The next phase extends these boundaries into governance, compliance, and day‑to‑day operations through policy definition, auditing, and lifecycle controls. Together, these mechanisms ensure that guardrails remain effective as AI apps and agents evolve, scale, and operate within enterprise environments. See the next post in the series: Governing AI apps and agents for Marketplace | Microsoft Community Hub. Key resources See curated, step-by-step guidance to help you build, publish, or sell your app or agent (no matter where you start) in App Advisor, Quick-Start Development Toolkit can connect you with code templates for AI solution patterns Microsoft AI Envisioning Day Events How to build and publish AI apps and agents for Microsoft Marketplace Get over $126K USD in benefits and technical consultations to help you replicate and publish your app with ISV SuccessAccelerate your AI or agent build to sell on Marketplace with Quick-Start Development Toolkit
Want to skip right to coding in minutes? Start with the interactive wizard in App Advisor Building AI products quickly is becoming table stakes. Building them in a way that supports scalability, repeatability, and a path to commercialization is where software companies create advantage. The challenge now is reducing the time between identifying an opportunity and getting developers working inside a proven structure that supports real deployment outcomes. That’s where the AI, agentic, and Copilot branch of the Quick-Start Development Toolkit helps. Embedded directly within App Advisor, Quick-Start Development Toolkit helps software companies move from concept to implementation faster using guided development patterns, trusted architectures, deployable reference code, and practical resources designed to reduce friction across the development process. Build AI & agentic products faster without starting from scratch Development teams often know the customer scenario they want to solve. What slows momentum is deciding where to begin, selecting architecture patterns, and aligning implementation decisions across teams. The Quick-Start Development Toolkit helps remove that uncertainty. By answering a few focused questions about what you want to build, who it serves, and the products you’re building with, you’re matched with a development pattern designed to accelerate execution. Each development pattern includes: Self-serve, click-to-deploy reference code aligned to your scenario, Sample solution architecture to help visualize products and reduce guesswork, and Practical how-to resources and implementation guidance to overcome friction points, Everything is structured to support faster decision making and help teams move confidently into development. Accelerate development with purpose-built AI accelerators The AI and agent branch of Quick-Start Development Toolkit includes development accelerators designed around high-value scenarios, so your team can spend less time assembling foundations and more time building differentiated experiences. Each of these accelerators is built and fully maintained by Microsoft experts, so you can be confident your code template isn’t stale. Our most popular accelerators include: Multi-Agent Custom Automation Engine Accelerator: Delegate complex, repetitive tasks to AI agents that act on your behalf—executing work efficiently, reducing manual effort, and ensuring results align with your organization's standards. Conversation Knowledge Mining Accelerator: Improve contact center performance with AI-powered conversation intelligence—analyzing audio and text data on a large scale to show insights, improve service, and drive smarter decisions. Accelerate agentic applications for Unified Data Foundations (with Microsoft Fabric): Accelerate decision making at scale with secure, agentic AI built on a unified data foundation with two use cases for sales performance and customer insights. Each pattern includes common use cases, related resources, and pathways to adjacent scenarios so teams can continue progressing without losing momentum. The goal is to help your team move from experimentation to a product that can be packaged, deployed, and prepared for customers. You can see more of our accelerators here Coming this week: The Microsoft IQ solution accelerator leverages a shared intelligence layer to unify data, knowledge, and workflows, enabling AI-powered insights and coordinated actions for measurable business outcomes. Build with Microsoft Marketplace outcomes in mind Development choices shape commercial outcomes. Starting with trusted architecture and structured implementation guidance can help reduce redesign cycles later when preparing to package, publish, and scale. Quick-Start Development Toolkit helps software companies: Shorten time from idea to deployable AI product, Improve alignment across implementation decisions, Reduce development overhead through reusable foundations, and Create repeatable pathways toward publishing and selling. When development starts with clarity, commercialization becomes easier. Keep moving forward with App Advisor Quick-Start Development Toolkit is embedded within App Advisor because building is only one stage of the journey. App Advisor helps connect decisions across design, development, publishing, and growth so teams can continue moving forward with less context switching and more confidence. As your solution evolves, App Advisor provides curated, step-by-step guidance to help you prepare for Marketplace readiness and make the next decision faster. Ready to start? Explore Quick-Start Development Toolkit Start where you need help with App AdvisorAI agents on Microsoft Fabric for faster retail merchandising decisions
For our latest in the Partner Spotlight series, we’re highlighting a partner building business-ready AI agents on Microsoft Fabric, so organizations can turn governed enterprise data into faster decisions and automated workflows. I connected with the team at Lucid Data Hub to learn how Lucid Agents Hub brings agentic experiences directly to customers’ data in OneLake, helping retail teams move beyond manual reporting and into repeatable, insight-driven action. About Venu Amancha, Founder & CEO, Lucid Data Hub builds business-ready AI agents that run directly on enterprise data within Microsoft Fabric. Our platform, Lucid Agents Hub, enables organizations to move beyond reporting and into automated, insight-driven workflows without moving data outside their existing security boundaries. _______________________________________________________________________________________________________________________________________________________________ [JR] Who is your solution designed for, and what does it help them do? [VA] Lucid Agents Hub is designed for teams who need to make frequent, high-impact decisions from large volumes of operational data especially merchandising teams, buyers, and store operations leaders in retail. Instead of spending hours assembling recaps and interpreting dashboards, they can receive agent-generated insights and clear, actionable recommendations on a predictable cadence. The AI agent eliminated that manual cycle entirely. It now surfaces those insights automatically, every week, in minutes. One example is our Retail Sales Performance AI Agent, which automates the weekly sales insights cycle for merchandising teams and buyers by analyzing millions of rows of weekly sales, item, and store data across banners and store clusters. [JR] Can you give an example of how the Retail Sales Performance AI Agent solved a customer’s problem? [VA] At Heritage Grocers Group, merchandising teams spent 5+ hours every week manually building sales recaps. They could see what happened—but not why. Buyers lacked a clear view of category trends, item-level performance, quantity shifts, and store-cluster patterns. The Retail Sales Performance AI Agent eliminated that manual cycle. It now surfaces those insights automatically every week in minutes detecting item-level declines, identifying fast-moving margin-positive SKUs, flagging underperforming items by store cluster, and delivering recommendations directly to buyers and store managers. [JR] Which Microsoft technologies or services are foundational to what you’re building? [VA] The solution runs natively on Microsoft Fabric, using OneLake as the unified data layer. Our agents operate directly on enterprise data and inherit existing governance and access controls without additional configuration. Outputs flow into the dashboards, collaboration platforms, and reporting workflows customers already use, so insights show up where decisions get made. Microsoft Fabric was a deliberate choice, not just a default. Our enterprise customers especially in retail already have their critical data living in the Microsoft ecosystem. OneLake means there’s a single, governed copy of that data. No duplication, no movement, no additional risk surface. Our agents read directly from that layer, which means the security and compliance boundaries that customers have already invested in carry over automatically. The value of building on Fabric goes beyond the technical architecture. It fundamentally changes how enterprise buyers evaluate and procure a solution like ours. When IT and security teams see that agents operate entirely within their existing Fabric environment with role-based access controls, workspace permissions, and audit logs they already control. It removes the largest barrier to enterprise adoption: trust. Procurement conversations that used to require months of security review cycles are now dramatically faster. What we didn’t fully anticipate was how much Fabric’s native integration capabilities would simplify end-to-end delivery. Going in, we expected to spend significant engineering time on data pipeline infrastructure. What we found instead was that Fabric’s data ingestion, lakehouse, and compute layers fit together in a way that let our team focus almost entirely on agent logic and business outcomes, not infrastructure plumbing. That shift in where we spend our effort has meaningfully accelerated how quickly we can deploy for new customers and extend to new use cases. [JR] How are you using AI today in Lucid Agents Hub, and what business outcomes have customers seen? [VA] We use Microsoft Azure AI Foundry for core AI and language model capabilities, and Microsoft Fabric Copilot (Fabric IQ) as the data and compute backbone. Together, they power agents that analyze weekly sales data across banners and store clusters, generate narrative-quality insights at the category and SKU level, and deliver clear recommendations without human intervention in the analysis cycle. 5+ hours of weekly manual effort eliminated Item-level sales declines and fast-moving margin-positive SKUs surfaced automatically Top-growth categories and underperforming items identified by store cluster Recommendations delivered directly to buyers and store managers weekly This all leads to faster decisions, stronger merchandising actions, and measurable improvements in product mix, availability, and overall sales performance. [JR] Any architectural decisions or best practices you’d recommend to other partners building agents? How did you approach building securely? [VA] We designed the solution as a coordinated set of specialized agents one for data ingestion, one for validation, and one for insight generation and delivery. Each agent owns a focused task, and together they run as a connected, end-to-end workflow. This makes the system easier to maintain, consistent in its logic, and straightforward to extend to new banners, categories, or use cases. Agents run entirely within the customer’s Microsoft Fabric environment data never leaves the customer’s security perimeter. All access controls, role-based permissions, and governance policies are inherited directly from Fabric. [JR] What motivated you to publish on Microsoft Marketplace? And did you use any Microsoft tools or benefits to support your publishing process? [VA] Publishing on Microsoft Marketplace was a straightforward decision. It gives enterprise customers immediate confidence that they’re procuring from a trusted, Microsoft-validated source instead of navigating a separate vendor relationship. It also simplifies procurement transactions run through an established Microsoft channel; so, customers can move faster than in traditional sales cycles. And it expands our reach to buyers already operating in the Microsoft ecosystem who actively look to Marketplace for solutions. We actively use Marketplace Rewards, which has been valuable for amplifying go-to-market efforts and accessing Microsoft co-marketing resources. We also leverage AI-enabled Marketplace Listing Optimization and related Marketplace content guidance provided through Marketplace Rewards. We used this support primarily to improve our marketplace messaging, positioning, and listing content so it would better resonate with enterprise buyers evaluating solutions within the Microsoft ecosystem. [JR] What key takeaways would you share with other partners building and publishing agents? Any unexpected wins or challenges along the way? [VA] Building and publishing agents can be a complicated endeavor. To other partners, we’d say, start with workflows that are repetitive and directly tied to decisions weekly merchandising recaps are a perfect example. Think end-to-end, not task by task. And build on governed enterprise data from the start, because that’s what drives trust and adoption. An unexpected win was how quickly merchandising teams adapted. Receiving plain-language summaries broken down by banner, store cluster, category, and SKU was more accessible than navigating dashboards. Teams made faster, more confident decisions without needing to interpret raw data themselves. _______________________________________________________________________________________________________________________________________________________________ Closing reflection Lucid Data Hub shows how agents built on Microsoft Fabric can turn governed enterprise data into repeatable, decision-ready insight helping teams act faster while keeping security boundaries and access controls intact.Securing AI apps and agents on Microsoft Marketplace
Why security must be designed in—not validated later AI apps and agents expand the security surface beyond that of traditional applications. Prompt inputs, agent reasoning, tool execution, and downstream integrations introduce opportunities for misuse or unintended behavior when security assumptions are implicit. These risks surface quickly in production environments where AI systems interact with real users and data. Deferring security decisions until late in the lifecycle often exposes architectural limitations that restrict where controls can be enforced. Retrofitting security after deployment is costly and can force tradeoffs that affect reliability, performance, or customer trust. Designing security early establishes clear boundaries, enables consistent enforcement, and reduces friction during Marketplace review, onboarding, and long‑term operation. In the Marketplace context, security is a foundational requirement for trust and scale. You can always get a curated step-by-step guidance through building, publishing and selling apps for Marketplace through App Advisor. This post is part of a series on building and publishing well-architected AI apps and agents in Microsoft Marketplace. The series focuses on AI apps and agents that are architected, hosted, and operated on Azure, with guidance aligned to building and selling solutions through Microsoft Marketplace. How AI apps and agents expand the attack surface Without a clear view of where trust boundaries exist and how behavior propagates across systems, security controls risk being applied too narrowly or too late. AI apps and agents introduce security risks that extend beyond those of traditional applications. AI systems accept open‑ended prompts, reason dynamically, and often act autonomously across systems and data sources. These interaction patterns expand the attack surface in several important ways: New trust boundaries introduced by prompts and inputs, where unstructured user input can influence reasoning and downstream actions Autonomous behavior, which increases the blast radius when authentication or authorization gaps exist Tool and integration execution, where agents interact with external APIs, plugins, and services across security domains Dynamic model responses, which can unintentionally expose sensitive data or amplify errors if guardrails are incomplete Each API, plugin, or external dependency becomes a security choke point where identity validation, audit logging, and data handling must be enforced consistently as part of securing AI integrations—especially when AI systems span tenants, subscriptions, or ownership boundaries. Using OWASP GenAI Top 10 as a threat lens The OWASP GenAI Top 10 provides a practical, industry‑recognized lens for identifying and categorizing AI‑specific security threats that extend beyond traditional application risks. Rather than serving as a checklist, the OWASP GenAI Top 10 helps teams ask the right questions early in the design process. It highlights where assumptions about trust, input handling, autonomy, and data access can break down in AI‑driven systems—often in ways that are difficult to detect after deployment. Common risk categories highlighted by OWASP include: Prompt injection and manipulation, where malicious input influences agent behavior or downstream actions Sensitive data exposure, including leakage through prompts, responses, logs, or tool outputs Excessive agency, where agents are granted broader permissions or action scope than intended Insecure integrations, where tools, plugins, or external systems become unintended attack paths Highly regulated industries, sensitive data domains, or mission‑critical workloads may require additional risk assessment and security considerations that extend beyond the OWASP categories. The OWASP GenAI Top 10 allows teams to connect high‑level risks to architectural decisions by creating a shared vocabulary that sets the foundation for designing guardrails that are enforceable both at design time and at runtime. Designing security guardrails into the architecture Security guardrails must be designed into the architecture, shaping where and how policies are enforced, evaluated, and monitored throughout the solution lifecycle. Guardrails operate at two complementary layers: Design time, where architectural decisions determine what is possible, permitted, or blocked by default Runtime, where controls actively govern behavior as the AI app or agent interacts with users, data, and systems When architectural boundaries are not defined early, teams often discover that critical controls—such as input validation, authorization checks, or action constraints—cannot be applied consistently without redesign: Tenancy boundaries, defining how isolation is enforced between customers, environments, or subscriptions Identity boundaries, governing how users, agents, and services authenticate and what actions they can perform Environment separation, limiting the blast radius of experimentation, updates, or failures Control planes, where configuration, policy, and behavior can be adjusted without redeploying core logic Data planes, controlling how data is accessed, processed, and moved across trust boundaries Designing security guardrails into the architecture transforms security from reactive to preventative, while also reducing friction later in the Marketplace journey. Clear enforcement boundaries simplify review, clarify risk ownership, and enable AI apps and agents to evolve safely as capabilities and integrations expand. Identity as a security boundary for AI apps and agents Identity defines who can access the system, what actions can be taken, and which resources an AI app or agent is permitted to interact with across tenants, subscriptions, and environments. Agents often act on behalf of users, invoke tools, and access downstream systems autonomously. Without clear identity boundaries, these actions can unintentionally bypass least‑privilege controls or expand access beyond what users or customers expect. Strong identity design shapes security in several key ways: Authentication and authorization, determines how users, agents, and services establish trust and what operations they are allowed to perform Delegated access, constraints agents to act with permissions tied to user intent and context Service‑to‑service trust, ensures that all interactions between components are explicitly authenticated and authorized Auditability, traces actions taken by agents back to identities, roles, and decisions A zero‑trust AI agent architecture is essential in this context. is essential in this context. Every request—whether initiated by a user, an agent, or a backend service—should be treated as untrusted until proven otherwise. Identity becomes the primary control plane for enforcing least privilege, limiting blast radius, and reducing downstream integration risk. This foundation not only improves security posture, but also supports compliance, simplifies Marketplace review, and enables AI apps and agents to scale safely as integrations and capabilities evolve. Protecting data across boundaries Data may reside in customer‑owned tenants, subscriptions, or external systems, while the AI app or agent runs in a publisher‑managed environment or a separate customer environment. Protecting data across boundaries requires teams to reason about more than storage location. Several factors shape the security posture: Data ownership, including whether data is owned and controlled by the customer, the publisher, or a third party Boundary crossings, such as cross‑tenant, cross‑subscription, or cross‑environment access patterns Data sensitivity, particularly for regulated, proprietary, or personally identifiable information Access duration and scope, ensuring data access is limited to the minimum required context and time When these factors are implicit, AI systems can unintentionally broaden access through prompts, retrieval‑augmented generation, or agent‑initiated actions. This risk increases when agents autonomously select data sources or chain actions across multiple systems. To mitigate these risks, access patterns must be explicit, auditable, and revocable. Data access should be treated as a continuous security decision, evaluated on every interaction rather than trusted by default once a connection exists. This approach aligns with zero-trust principles, where no data access is implicitly trusted and every request is validated based on identity, context, and intent. Runtime protections and monitoring For AI apps and agents, security does not end at deployment. In customer environments, these systems interact continuously with users, data, and external services, making runtime visibility and control essential to a strong security posture. AI behavior is also dynamic: the same prompt, context, or integration can produce different outcomes over time as models, data sources, and agent logic evolve, so monitoring must extend beyond infrastructure health to include behavioral signals that indicate misuse, drift, or unintended actions. Effective runtime protections focus on five core capabilities: Vulnerability management, including regular scanning of the full solution to identify missing patches, insecure interfaces, and exposure points Observability, so agent decisions, actions, and outcomes can be traced and understood in production Behavioral monitoring, to detect abnormal patterns such as unexpected tool usage, unusual access paths, or excessive action frequency Containment and response, enabling rapid intervention when risky or unauthorized behavior is detected Forensics readiness, ensuring system-state replicability and chain-of-custody are retained to investigate what happened, why it happened, and what was impacted Monitoring that only tracks availability or performance is insufficient. Runtime signals must provide enough context to explain not just what happened, but why an AI app or agent behaved the way it did, and which identities, data sources, or integrations were involved. Equally important is integration with broader security event and incident management workflows. Runtime insights should flow into existing security operations so AI-related incidents can be triaged, investigated, and resolved alongside other enterprise security events—otherwise AI solutions risk becoming blind spots in a customer’s operating environment. Preparing for incidents and abuse scenarios No AI app or agent operates in a perfectly controlled environment. Once deployed, these systems are exposed to real users, unpredictable inputs, evolving data, and changing integrations. Preparing for incidents and abuse scenarios—including AI agent incident response—is therefore a core security requirement, not a contingency plan. AI apps and agents introduce unique incident patterns compared to traditional software. In addition to infrastructure failures, teams must be prepared for prompt abuse, unintended agent actions, data exposure, and misuse of delegated access. Because agents may act autonomously or continuously, incidents can propagate quickly if safeguards and response paths are unclear. Effective incident readiness starts with acknowledging that: Abuse is not always malicious, misuse can stem from ambiguous prompts, unexpected context, or misunderstood capabilities Agent autonomy may increase impact, especially when actions span multiple systems or data sources Security incidents may be behavioral, not just technical, requiring interpretation of intent and outcomes Preparing for these scenarios requires clearly defined response strategies that account for how AI systems behave in production. AI solutions should be designed to support pause, constrain, or revoke agent capabilities when risk is detected, and to do so without destabilizing the broader system or customer environment. Incident response must also align with customer expectations and regulatory obligations. Customers need confidence that AI‑related issues will be handled transparently, proportionately, and in accordance with applicable security and privacy standards. Clear boundaries around responsibility, communication, and remediation help preserve trust when issues arise. How security decisions shape Marketplace readiness From initial review to customer adoption and long‑term operation, security posture is a visible and consequential signal of readiness. AI apps and agents with clear boundaries—around identity, data access, autonomy, and runtime behavior—are easier to evaluate, onboard, and trust. When security assumptions are explicit, Marketplace review becomes more predictable, customer expectations are clearer, and operational risk is reduced. Ambiguous trust boundaries, implicit data access, or uncontrolled agent actions can introduce friction during review, delay onboarding, or undermine customer confidence after deployment. Marketplace‑ready security is therefore not about meeting a minimum bar. It is about enabling scale. Well-designed security allows AI apps and agents to integrate into enterprise environments, align with customer governance models, and evolve safely as capabilities expand. When security is treated as a first‑class architectural concern, it becomes an enabler rather than a blocker—supporting faster time to market, stronger customer trust, and sustainable growth through Microsoft Marketplace. What’s next in the journey Security for AI apps and agents is not a one‑time decision, but an ongoing design discipline that evolves as systems, data, and customer expectations change. By establishing clear boundaries, embedding guardrails into the architecture, and preparing for real‑world operation, publishers create a foundation that supports safe iteration, predictable behavior, and long‑term trust. This mindset enables AI apps and agents to scale confidently within enterprise environments while meeting the expectations of customers adopting solutions through Microsoft Marketplace. See the next post in the series: Designing AI guardrails for apps and agents in Marketplace | Microsoft Community Hub. Key resources See curated, step-by-step guidance to help you build, publish, or sell your app or agent (no matter where you start) in App Advisor, Quick-Start Development Toolkit Microsoft AI Envisioning Day Events How to build and publish AI apps and agents for Microsoft Marketplace Get over $126K USD in benefits and technical consultations to help you replicate and publish your app with ISV SuccessProduction ready architectures for AI apps and agents on Marketplace
Why “production‑ready” architecture matters for Marketplace AI apps and agents A working AI prototype is not the same as a production‑ready AI app in Microsoft Marketplace. Marketplace solutions are expected to operate reliably in real customer environments, alongside mission‑critical workloads and under enterprise constraints. As a result, AI apps published through Marketplace must meet a higher bar than “it works in a demo.” You can always get a curated step-by-step guidance through building, publishing and selling apps for Marketplace through App Advisor. Production‑ready Marketplace AI apps must assume: Alignment with enterprise expectations and Azure well‑architected AI principles, including cost optimization, security, reliability, operational excellence, and performance efficiency Architectural decisions made early are difficult to reverse, especially once customers, tenants, and billing relationships are in place A higher trust bar from customers, who expect Marketplace solutions to be Microsoft‑vetted, certified, and safe to run in production Customers come to Marketplace expecting solutions that are ready to run, ready to scale, and ready to be supported—not experiments. This post focuses on the architectural principles and patterns required to meet those expectations. Specific services and implementation details are covered later in the series. This post is part of a series on building and publishing well-architected AI apps and agents in Microsoft Marketplace. The series focuses on AI apps and agents that are architected, hosted, and operated on Azure, with guidance aligned to building and selling solutions through Microsoft Marketplace. Aligning offer type and architecture early sets you up for success A strong indicator of a smooth Marketplace journey is early alignment between offer type and solution architecture. Offer type defines more than how an AI app is listed—it establishes clear roles and responsibilities between publishers and customers, which in turn shape architectural boundaries. Across all other offer types, architecture must clearly answer three questions: Who owns the runtime? Where does the AI execute? Who controls updates and ongoing operations? These decisions will vary depending on whether the solution resides in the customer’s or publisher’s tenant based on the attributes associated with the following transactable marketplace offer types: SaaS offers, where the AI runtime lives in the publisher’s environment and architecture must support multitenant AI app design, strong isolation, and centralized operations Container offers, where workloads run in the customer’s Kubernetes environment and architecture emphasizes portability and clear operational assumptions Virtual Machine offers, where preconfigured environments run in the customer’s subscription and architecture is more tightly coupled to the OS and infrastructure footprint Azure Managed Applications, where the solution is deployed into the customer's subscription and architecture must balance customer control with defined lifecycle boundaries. What makes this model distinctive is its flexibility: an Azure Managed Application can package containers, virtual machines, or a combination of both — making it a natural fit for solutions that require customer-controlled infrastructure without sacrificing publisher-managed operations. The packaging choice shapes the underlying architecture, but the managed application wrapper is what defines how the solution is deployed, updated, and governed within the customer's environment. Architecture decisions naturally reinforce Marketplace requirements and reduce certification and operational friction later. Key factors that benefit from early alignment include: Roles and responsibilities, such as who operates the AI runtime and who is responsible for uptime, patching, scaling, and ongoing operations Proximity to data, particularly for AI solutions that rely on customer‑specific or proprietary data, where placement affects performance, data movement, and compliance Core architectural building blocks of AI apps Designing a production‑ready AI app starts with treating the solution as a system, not a single service. AI apps—especially agent‑based solutions—are composed of multiple cooperating layers that together enable reasoning, action, and safe operation at scale. At a high level, most production‑ready AI apps include the following building blocks: Interaction layer, which serves as the entry point for users or systems and is responsible for authentication, request shaping, and consistent responses Orchestration layer, which coordinates reasoning, tool selection, workflow execution, and retrieval‑augmented generation (RAG) flows across multi‑step interactions Model endpoints, which provide inference and generation capabilities and introduce distinct latency, cost, and dependency characteristics Data sources, including vector stores, operational data, documents, and logs that the AI system reasons over Control planes, such as identity, configuration, policy enforcement, feature flags, and secrets management, which govern behavior without redeploying core logic Observability, which enables tracing, monitoring, and diagnosis of agent decisions, actions, and outcomes Networking, which connects components using a zero‑trust posture where every call is authenticated and outbound access is explicitly controlled Together, these components form the foundation of most Marketplace‑ready AI architectures. How they are composed—and where boundaries are drawn—varies by offer type, tenancy model, and customer requirements. Specific services, patterns, and implementation guidance for each layer are explored later in the series. Tenancy design choices as an early architectural decision One of the earliest and most consequential architectural decisions is where the AI solution is hosted. Does it run in the publisher’s tenant, or is it deployed into the customer’s tenant? This choice establishes foundational boundaries and is difficult to change later without significant redesign. If the solution runs in the publisher’s tenant, it is inherently multi‑tenant and must be designed with strong logical isolation across customers. If it runs in the customer’s tenant, deployments are typically single‑tenant by default, with isolation provided through infrastructure boundaries. Many Marketplace AI apps fall between these extremes, making it essential to define the AI tenancy model early. Common tenancy approaches include: Publisher‑hosted, multi‑tenant solutions, where a shared AI runtime serves multiple customers and requires strict isolation of customer data, inference requests, identity, and cost attribution Customer‑hosted, single‑tenant deployments, where each customer operates an isolated instance within their own Azure subscription, often preferred for regulated or tightly controlled environments Hybrid models, which combine centralized AI services with customer‑hosted data or execution layers and require carefully defined trust and access boundaries Tenancy decisions influence several core architectural dimensions, including: Identity and access boundaries, which define how users and agents authenticate and act across tenants Data isolation, including how customer data is stored, processed, and protected Model usage patterns, such as shared models versus tenant‑specific models Cost allocation and scale, including how usage is tracked and attributed per customer These considerations are not implementation details—they shape how the AI system behaves, scales, and is governed in production. Reference architecture guidance for multi‑tenant AI and machine learning solutions in the Azure Architecture Center explores these tradeoffs in more detail. Understanding your customer’s needs Designing a production‑ready AI architecture starts with understanding the environment your customers expect your solution to operate in. Marketplace customers vary widely in their security posture, compliance obligations, operational practices, and tolerance for change. Architectures that reflect those realities reduce friction during onboarding, certification, and long‑term operation. Key customer considerations that shape architecture include: Security and compliance expectations, such as industry regulations, internal governance policies, or regional data requirements Target environments, including whether customers expect solutions to run in their own Azure subscription or are comfortable consuming centrally hosted services Change and outage windows, where operational constraints or seasonal restrictions require predictable and controlled updates Architectural alignment with customer needs is not about designing for every edge case. It is about making intentional tradeoffs that reflect how customers will deploy, operate, and depend on your AI solution in production. Specific security controls, compliance enforcement mechanisms, and operational policies are explored later in the series. This section establishes the architectural mindset required to support them. Separating environments for safe iteration Production AI systems must evolve continuously while remaining stable for customers. Separating environments is how publishers enable safe iteration without destabilizing live usage—and how customers maintain confidence when adopting and operating AI solutions in their own environments. From the publisher’s perspective, environment separation enables: Iteration on prompts, models, and orchestration logic without impacting production customers Validation of behavior changes before rollout, especially for AI‑driven systems where small changes can produce materially different outcomes Controlled release strategies that reduce operational risk From the customer’s perspective, environment separation shapes how the solution fits into their own development and operational practices: Where the solution is deployed across development, staging, and production environments How deployments are repeated or promoted, particularly when the solution runs in the customer’s tenant Whether environments can be recreated predictably, or whether customers are forced to manually reconfigure deployments with each iteration When AI solutions are deployed into the customer’s tenant, environment design becomes especially important. Customers should not be required to reverse‑engineer deployment logic, recreate environments from scratch, or re‑establish trust boundaries every time the solution evolves. These concerns should be addressed architecturally, not deferred to operational workarounds. Environment separation is therefore not just a DevOps choice—it is an architectural decision. It influences identity boundaries, deployment topology, validation strategies, and the shared operational contract between publisher and customer. Designing for AI‑specific scalability patterns AI workloads do not scale like traditional web or CRUD‑based applications. While front‑end and API layers may follow familiar scaling patterns, AI systems introduce behaviors that require different architectural assumptions. Production‑ready AI architectures must account for: Bursty inference demand, where usage can spike unpredictably based on user behavior or downstream automation Long‑running or multi‑step agent workflows, which may span tools, data sources, and time Model‑driven latency and cost characteristics, which influence throughput and responsiveness independently of application logic As a result, scalability decisions often vary by layer. Horizontal scaling is typically most effective in interaction, orchestration, and retrieval components, while model endpoints may require separate capacity planning, isolation, or throttling strategies. Treating identity as an architectural boundary Identity is foundational to Marketplace AI apps, but architecture must plan for it explicitly. Identity decisions define trust boundaries across users, agents, and services, and shape how the solution scales, secures access, and meets compliance requirements. Key architectural considerations include: Microsoft Entra ID as a foundation, where identity is treated as a core control plane rather than a late‑stage integration How users sign in, including: Their own corporate Microsoft Entra ID tenant B2B scenarios where one Entra ID tenant trusts another B2C identity providers for customer‑facing experiences How tenants authenticate, particularly in multi‑tenant or cross‑organization scenarios How AI agents act on behalf of users, including delegated access, authorization scope, and auditability How services communicate securely, using a zero‑trust posture where every call is authenticated and authorized Treating identity as an architectural boundary helps ensure that trust relationships remain explicit, enforceable, and consistent across tenants and environments. This foundation is critical for supporting secure operation, compliance enforcement, and future tenant‑linking scenarios. Designing for observability and auditability Production‑ready AI apps must be observable and auditable by design. Marketplace customers expect visibility into how systems behave in production, and publishers need clear insight to diagnose issues, operate reliably, and meet enterprise trust and compliance expectations. Key architectural considerations include: End‑to‑end observability, covering user interactions, agent reasoning steps, tool invocations, and downstream service calls Clear audit trails, capturing who initiated an action, what the AI system did, and how decisions were executed—especially when agents act on behalf of users Tenant‑aware visibility, ensuring logs, metrics, and traces are correctly attributed without exposing data across tenants Operational transparency, enabling effective troubleshooting, incident response, and continuous improvement without ad‑hoc instrumentation AI app observability design goes beyond infrastructure health. It must also account for AI‑specific behavior, such as prompt execution, model selection, retrieval outcomes, and tool usage. Without this visibility, diagnosing failures, validating changes, or explaining outcomes becomes difficult in real customer environments. Auditability is equally critical. Identity, access, and action histories must be traceable to support security reviews, regulatory obligations, and customer trust—particularly in regulated or enterprise settings. Common architectural pitfalls in Marketplace AI apps Even experienced teams run into similar challenges when moving from an AI prototype to a production‑ready Marketplace solution. The following pitfalls often surface when architectural decisions are deferred or made implicitly. Common pitfalls include: Treating AI as a single service instead of a system, where model inference is implemented without considering orchestration, data access, identity, observability, and operational boundaries Hard‑coding tenant assumptions, such as assuming a single tenant, identity model, or deployment topology, which becomes difficult to unwind as customer requirements diversify Not planning for a resilient model strategy, leaving the architecture fragile when model versions change, capabilities evolve, or providers introduce breaking behavior Assuming data lives within the same boundary as the solution, when in practice it may reside in a different tenant, subscription, or control plane Tightly coupling prompt logic to application code, making it harder to iterate on AI behavior, validate changes, or manage risk without full redeployments Assuming issues can be fixed after go‑live, which underestimates the cost and complexity of changing architecture once customers, subscriptions, and trust relationships are in place While these pitfalls may be caused by a lack of technical skill on the customer’s side, they could typically emerge when architectural decisions are postponed in favor of speed, or when AI behavior is treated as an isolated concern rather than part of a production system. What’s next in the journey The architectural decisions made early—around offer type, tenancy, identity, environments, and observability—establish the foundation on which everything else is built. When these choices are intentional, they reduce friction as the solution evolves, scales, and adapts to real customer needs. The next set of posts builds on this foundation, exploring different dimensions of operating, securing, and evolving Marketplace AI apps in production. See the next post in the series: Securing AI apps and agents on Microsoft Marketplace | Microsoft Community Hub. Key resources See curated, step-by-step guidance to help you build, publish, or sell your app or agent (no matter where you start) in App Advisor Quick-Start Development Toolkit can connect you with code templates for AI solution patterns Microsoft AI Envisioning Day Events How to build and publish AI apps and agents for Microsoft Marketplace Get over $126K USD in benefits and technical consultations to help you replicate and publish your app with ISV SuccessSecuring AI agents at scale: Identity, governance, and zero trust
AI agents are reshaping enterprise operations. They reason over data, call APIs, trigger workflows, and act on behalf of users — often without human intervention. IDC projects 1.3 billion AI agents in production by 2028, and Capgemini research shows four out of five organizations plan to integrate agents within the next three years. The race to build is already underway. The harder question — one most organizations are not yet equipped to answer — is: how do you operate AI agents safely at scale? Deploying a demo-ready agent is table stakes. Making that same agent production-grade, with proper identity, access controls, governance, and runtime protections, is an entirely different engineering and operational challenge. Why agentic AI changes the security equation Traditional software executes deterministic logic. An AI agent reasons, plans, and takes action — dynamically, across systems it was never explicitly programmed to touch. When connected to enterprise APIs and data, an agent stops being a chatbot and starts functioning as a digital worker. That shift has direct security implications: Agents can trigger business processes and access sensitive data autonomously Their capabilities expand as new tools and Model Context Protocol (MCP) servers are connected They behave non-deterministically, making static policy controls insufficient A compromised or misconfigured agent can have a large blast radius Most organizations are deploying into this environment without the governance infrastructure to match. The result is growing agent sprawl, unclear accountability, and unmanaged risk — exactly the conditions that security and compliance teams lose sleep over. A three-pillar framework for secure agent operations Securing AI agents at enterprise scale requires a structured approach. The following three-pillar framework provides a practical foundation: Manage, Govern, and Protect. Pillar 1: Manage — build an agent registry and enforce identity The first pillar focuses on visibility. You cannot secure what you cannot see, and you cannot control what you cannot identify. Every agent in your environment must have a unique, managed identity — a first-class identity type distinct from human accounts, service principals, and workload identities. Without it, agents are anonymous automation: untraceable, un-auditable, and uncontrollable. With agent identity in place, organizations gain: Traceability — a full record of what each agent accessed, when, and in what context Auditability — the clean audit trail that security and compliance teams require Granular control — the ability to restrict, block, or revoke access based on policy The operational target is an agent registry: a centralized fleet view of every agent across the organization, including those built on non-Microsoft platforms. Operators should be able to inspect agent metadata, validate ownership, review configuration, and block agents directly from this registry. This fleet management capability is what separates enterprise-grade deployments from uncontrolled agent sprawl. Pillar 2: Govern — automate agent lifecycle management The second pillar addresses the operational reality that emerges once agents number in the hundreds or thousands: manual governance doesn't scale. Three foundational practices make lifecycle governance work at scale. Assign a sponsor to every agent Each agent must have a designated owner — a person or team accountable for its activity and access decisions from creation through retirement. Orphaned agents, where no human is accountable, represent a high-risk surface area. Sponsorship creates a clear chain of accountability that also supports incident response when something goes wrong. Automate lifecycle policies Agents change hands, move between teams, and eventually retire. Access granted at creation may be inappropriate six months later. Lifecycle policies — covering creation, handoff, access review, and decommissioning — must be automated to remain effective at scale. The goal is to remove routine human intervention from the process while ensuring every action is logged and auditable. Enforce just-in-time, time-bound access Agents should never hold standing permissions. Access should be intentionally scoped, time-limited, and removed when no longer required. This directly addresses permission accumulation — one of the most common and underappreciated risks in agentic deployments, where an agent gradually gains access to resources far beyond its original purpose. The full lifecycle flow runs from creation (sponsor assigned, access requested and approved) through active operation (periodic access reviews) to retirement (identity deleted, access expired). Every stage should generate an auditable record. Pillar 3: Protect — secure agents at runtime The third pillar covers production runtime security — what happens once agents are live and actively accessing corporate resources. This is where risk is most dynamic and where a defense-in-depth approach is non-negotiable. Agents require access to file stores, APIs, calendars, MCP servers, and third-party SaaS systems to deliver value. Each of those connections is also an exposure point for prompt injection, data exfiltration, or access to malicious content. Three mechanisms form the runtime protection stack. Conditional access policies for agents The same conditional access framework used for human identities can be applied to agents — enforcing policy based on agent identity and operating context. This enables broad baseline protections across the entire agent fleet, with stricter controls for high-impact agents or sensitive scenarios. Access decisions happen continuously and automatically, not as one-off reviews. Risk signal detection and automated response Risky agent behaviors — sudden access spikes, repeated resource access failures, or attempts to reach unfamiliar systems — generate signals that can trigger automated responses. Policies can block the agent, limit its blast radius, and alert security operations for investigation, all without manual intervention. Network-layer controls Identity and access controls alone are not sufficient. Content inspection, threat-intelligence-based URL filtering, and file-level network filtering add additional layers of protection, particularly against agents that interact with external systems or user-supplied inputs. No single control point should be relied upon exclusively. Understanding agent blueprints and identity hierarchy One of the most important architectural concepts for enterprise agent governance is the distinction between an agent blueprint and an agent identity. An agent blueprint is a reusable, tenant-scoped template that defines the security, authentication, and governance characteristics for a class of agents. It sets the operational boundary: what the agent is allowed to do, which resources it can access, and which policies govern its behavior. An agent identity is a specific instance derived from a blueprint. One blueprint can generate many identities, each inheriting the governance constraints defined in the template. This parent-child model delivers concrete benefits across roles: Developers get a predictable, consistent identity model for every agent they build IT and security teams gain class-level audit trails and lifecycle control without per-instance overhead End users and customers can trust that every agent instance operates within the boundaries set at design time Governance intent — authentication, authorization, execution boundaries — is defined once and applied consistently across the entire agent family derived from that blueprint. MCP tool governance: Managing agent interoperability Modern agents rarely operate in isolation. They connect to external tools and services through protocols like MCP (Model Context Protocol), enabling tasks like retrieving documents from SharePoint, sending email, scheduling meetings, and coordinating with other agents. The power of this connectivity is real. A single agent instruction can simultaneously invoke multiple MCP servers — retrieving a file, generating a shareable link, sending it via email, and creating a calendar event — all in one natural-language-triggered flow. That same power is also a governance surface. Not every available MCP tool is appropriate for every agent, and not every tool is safe for production use without review. The right approach is to treat tool access like data access: intentional, reviewed, and tied to the agent's identity. Access to specific MCP servers should be formally requested, approved by IT or security, and documented as part of the agent onboarding process — not left to developer discretion at build time. Five practices for production-ready agent security The framework above translates into concrete practices for teams building and operating agents today. Build identity in, not on. Retrofitting governance onto an existing agent fleet is significantly harder than starting with identity baked into the build process. Every new agent should have an identity, a sponsor, and a defined access scope before it ships. Design for the fleet, not the agent. The practices and templates you establish for one agent should scale to hundreds. Reusable blueprints, automated lifecycle policies, and standardized onboarding patterns are investments that compound as agent counts grow. Threat-model before you deploy. For each production agent, work through what data it can access, what actions it can take, what happens under a malicious prompt, and what the blast radius looks like if it is compromised. Threat modeling surfaces risks that capability-focused development naturally misses. Automate governance alongside capability. The instinct in AI development is to automate what the agent does. Apply that same instinct to how the agent is governed. Lifecycle policies, access reviews, risk signal responses, and audit log generation should all run without manual orchestration. Apply zero trust to agents explicitly. Least privilege, continuous verification, and assume-breach principles apply to agents just as they do to humans and workloads. Agents that accumulate permissions over time, or that operate without continuous policy enforcement, are a liability as the threat landscape evolves. Distributing governed agents through Microsoft Marketplace The governance architecture described in this post — Entra agent identity, blueprints, lifecycle policies, and zero trust controls — applies equally to software development companies distributing agents through Microsoft Marketplace. These constructs are what determine whether a third-party agent can be trusted and managed by the enterprise customers who purchase it. With Agent 365 now generally available, enterprise customers have a control plane for managing every agent in their Microsoft 365 tenant, including agents procured from Marketplace. A software company-built agent that arrives without a registered Entra agent identity will surface as unmanaged in the customer's registry, requiring manual remediation before it can be approved for use — a friction point that directly delays adoption. For software company agents to pass enterprise IT and security review in an Agent 365-enabled tenant, three things must be in place from day one: Registered Entra agent identity — enables the customer to apply their own conditional access and lifecycle policies to the agent A well-defined agent blueprint — tells the customer's IT and security teams what the agent is permitted to do, what resources it accesses, and what governance characteristics it carries Declared and scoped MCP tool access — allows the customer's admin to review and approve tool permissions before the agent goes live A software company agent that ships with these elements in place signals it was built for enterprise operations from the start — and that trust signal is increasingly what separates agents that clear procurement review from those that stall in it. The competitive case for getting this right Organizations that build secure, governed agent infrastructure do not just reduce risk — they accelerate adoption. When security and operations teams can trust the agents in their environment, the internal friction that slows deployment decreases. Partners and software companies that deliver governed, auditable agent solutions create durable differentiation compared to those shipping ungoverned agents into customer environments. The companies winning in the agentic AI space are not just building agents. They are building the platform, practices, and governance infrastructure around them. That is what enterprise customers are increasingly requiring, and it is what separates a compelling demo from a production-grade solution. The principles have not changed — identity, least privilege, auditability, and defense in depth. What has changed is that those principles now apply to a new class of actor in the enterprise: the AI agent. *This blog is based on the Security for SDC Series Season 2 Episode 4: Securing and Governing AI Agents Before They Go Live presented by Ryan Nguyen and Paul Woo View past and upcoming sessions in the Security for software development company series: Security for SDC Series: Securing the Agentic EraDesign CI/CD for AI apps and agents selling through Microsoft Marketplace
In the previous post, Design observability for AI apps and agents selling through Microsoft Marketplace, we focused on observability—making AI app and agent behavior visible and explainable. Execution paths, retries, degradation patterns, and agent decisions can now be observed across environments and tenants. With that visibility in place, a new challenge emerges: how do you safely modify an AI system whose behavior you can now observe? You can always get curated step-by-step guidance through building, publishing and selling apps for Marketplace through App Advisor. This post is part of a series on building and publishing well-architected AI apps and agents in Microsoft Marketplace. The series focuses on AI apps and agents that are architected, hosted, and operated on Azure, with guidance aligned to building and selling solutions through Microsoft Marketplace. Using continuous integration/continuous delivery (CI/CD) to control AI system evolution AI apps and agents introduce numerous novel ways that production behavior can change. In addition to application code, updates to configuration, prompts, models, and guardrails, agent logic can alter execution, cost, and outcomes—often immediately and across tenants. CI/CD defines how these changes reach production. Without a structured delivery path, behavior‑shaping updates risk entering runtime without validation or recovery paths, making system behavior difficult to explain or reverse once customers encounter it. AI solutions are typically built and operated as cloud applications. Software delivery of cloud services, and the supporting components that enable it, remains part of the CI/CD pipeline, and any instability in these foundational components directly propagates into AI behavior. AI systems add two additional sources of change that require explicit control. MLOps governs model evolution. Agents introduce further variability, as agent logic and configuration evolve. CI/CD is what prevents these change vectors from interacting unpredictably across both publisher and customer environments. Core CI/CD requirements for AI apps and agents For AI apps and agents, CI/CD determines whether deployment strategies can be applied safely. Progressive rollouts, ring deployments, feature flags, and kill switches all rely on pipelines that isolate change, validate behavior, and support rollback. Observability provides insight into behavior; CI/CD controls when and how that behavior is allowed to change. CI/CD must reliably provision, configure, and promote cloud native infrastructure, including but not limited to front-end services, APIs, storage, identity, and networking across environments. Agent behavior depends directly on the stability of the platform it runs on. AI systems introduce additional CI/CD requirements through MLOps and agents. Model versions, routing logic, and evaluation configurations must move through pipelines as deployable artifacts, with isolation, validation, and rollback built in. Changes to models affect latency, cost, and outcomes even when application code remains unchanged, making promotion controls necessary at the model layer. A well-run CI/CD pipeline should positively impact AI models and agents in the following ways: Change isolation ensures code, prompts, models, and configuration evolve independently. Artifact versioning beyond code treats prompts, policies, tools, and models as release assets. Behavioral validation evaluates outcomes, constraints, and patterns rather than single responses. Safe promotion controls gate model and agent releases based on observed behavior. Rollback readiness allows fast reversion when model or agent behavior degrades. Building behavioral baselines for AI solutions using CI/CD Before an AI system is built by a pipeline, it is built by a team. CI/CD build pipelines are where these contributions are stitched together. Product managers define scope and constraints. UX designers shape how behavior is experienced. Full‑stack engineers assemble application logic. AI engineers wire reasoning and tools. Data engineers and data scientists curate data and models. In AI systems, a build does more than compile code. It captures a shared agreement across roles about what the system is expected to do. Application code, orchestration logic, prompts, configuration, guardrails, routing rules, and trained or fine‑tuned models are assembled into a single versioned artifact. That artifact represents a coordinated snapshot of intent, behavior, and constraints. This coordination must declare which models, prompts, policies, and tool definitions are included. Implicit dependencies—such as dynamically changing prompts or unpinned models—break shared understanding across teams and introduce behavior changes without acknowledgement. A successful build confirms that contributions from multiple roles are compatible and executable together. It does not decide when customers see the change. That decision belongs later, where behavior can be evaluated deliberately, enforced by build pipelines that are separating assembly from release. Testing AI solutions with CI/CD pipelines When an agent is updated, the first task is straightforward: the agent’s code changes. Logic is refined, tools are added, limits are adjusted. That change moves through the CI/CD pipeline, where it is built, packaged, and validated in isolation. At this point, the focus is narrow—does this agent compile, configure, and execute as expected? The second step widens the lens. The update now moves through testing aligned to the layers beneath it. For cloud solutions, tests confirm the platform still behaves as assumed: infrastructure provisions correctly, APIs and identity boundaries remain intact, and dependencies remain reachable. These tests ensure the environment can support execution before behavior is evaluated. Next, MLOps tests assess whether model behavior still aligns with system expectations. New model versions, routing logic, or provider changes are evaluated for cost, latency, and outcome consistency. The goal is not identical responses, but bounded behavior within known limits. Finally, testing shifts to the agentic system as a whole. Other agents need to be made aware of the new capabilities. When you go to update the agent the first job you have to do is update the agent code. The second job you have to do is to use your CI/CD pipeline to build, test and release that code. The third job is to test that the entire agentic system is running smoothly together. At this stage, testing answers a different question: not does the agent work, but does the system still work together. CI/CD release management as team coordination Once testing confirms that behavior remains within expected bounds, release management determines how changes are introduced and observed under real conditions. In AI systems, release management must reflect where change originates and how risk propagates across layers. Within the cloud services that support the AI solution, release management focuses on scope and blast‑radius control. Examples include staged rollout of infrastructure updates, controlled exposure of new API versions, and limiting configuration changes to specific environments or tenants before going global. These steps allow both publisher and customer teams to observe stability and dependency behavior under load. For MLOps, release management governs behavioral shifts introduced by model changes. Common patterns include routing a small percentage of requests to a new model version, limiting exposure to specific customer segments, or restricting usage to defined request types. This allows teams to compare cost, latency, and outcome patterns before expanding exposure. For agents, release management controls how new behaviors surface. Prompt updates, tool access changes, or guardrail adjustments may be released to specific workflows, tenants, or traffic slices. This makes it possible to observe planning depth, retry behavior, and termination patterns without affecting all users simultaneously. Rollback readiness remains essential. Release paths must allow fast reversion using version pinning or traffic shifting rather than full redeployment. Release management creates space to observe, adjust, and respond before changes reach full Marketplace scale. Deployment as a shared boundary Effective deployment pipelines ensure that software, models, and agent behavior enter production together, with changes explicitly acknowledged and observable. Versioning and rollback remain available, but deployment defines the moment when coordinated decisions become customer‑visible. Cloud service—For the software, deployment governs application code and supporting platform changes. These remain necessary foundations. Application binaries, infrastructure templates, runtime configuration, and orchestration must enter production in a known, versioned state so operational behavior can be correlated with specific changes. MLOps—Model version updates, routing rules, provider switches, and evaluation configurations can change system behavior without modifying application code. Deployment pipelines must therefore treat these artifacts as deployable units, subject to the same versioning, promotion, and rollback mechanics as software releases. Agent—Deployment includes behavior‑defining inputs such as prompts and system messages, tool definitions and permissions, guardrails, and execution limits. Changes directly affect how agents plan, execute, and terminate work. Allowing these inputs to change outside deployment pipelines breaks traceability and weakens accountability across teams. How CI/CD best practices positively impact marketplace readiness Customers expect updates to arrive in predictable ways. They expect that behavior changes can be explained, that issues can be reversed without prolonged disruption, and that outcomes remain consistent across trials and production use. CI/CD pipelines make these expectations achievable by ensuring changes are versioned, staged, and observable as they move through environments. Reliability depends on limiting how far unstable behavior propagates. Billing accuracy depends on knowing when changes alter execution paths, token usage, or metering logic. Compliance depends on being able to identify which versions of software, models, and agent configurations were active at a given time. Offer type shapes how CI/CD is applied. For transactable SaaS offers, CI/CD operates entirely within the publisher’s environment. For container offers and Azure Managed Applications, deployment boundaries extend to customer environments requiring a CI/CD hand-off between publisher and customer pipelines. Publisher CI/CD responsibilities for AI solutions Publishers must define what constitutes a deployable change. Updates to software, models, prompts, agent configuration, guardrails, or limits should not enter customer environments or generally available code implicitly. Each change that can influence behavior must flow through the publisher’s CI/CD pipelines so it can be versioned, observed, and reversed if necessary. Additionally, CI/CD pipelines require validation and approval before promotion, ensuring that behavior‑altering updates do not reach customers without visibility or control. Publishers are also responsible for communicating behavior changes. Customers should be able to understand when updates affect outcomes, performance, or cost profiles. Customers should never experience silent behavior shifts, undocumented updates, or releases that cannot be recovered cleanly. When those occur, trust erodes quickly. In this context, CI/CD is part of how publishers establish reliability, accountability, and trust with Marketplace customers. Customer’s responsibility: CI/CD across environments (Dev / Stage / Prod) While publishers own CI/CD pipelines, customers play an important role in how AI systems are evaluated and adopted across environments. AI behavior often manifests differently across Dev, Stage, and Prod because operating conditions change as systems move toward real usage. As environments scale, dependency interactions increase, traffic patterns diversify, and tenant behavior becomes less predictable—revealing execution paths and constraints that are not exercised earlier. These differences affect how behavior appears during evaluation and rollout. To keep behavior interpretable across environments, pipeline structure matters. CI/CD pipelines, validation steps, and promotion criteria should operate consistently so signals observed earlier can be understood later. When these mechanics diverge between environments, it becomes difficult to attribute changes in behavior to specific updates or conditions. Staging environments serve as a behavioral proving ground. They allow customers to observe retries, limits, degradation paths, and cost behavior under conditions that more closely resemble production. Trials often run against production‑like configurations, which means CI/CD gaps surface early. When behavior differs from expectations, the consistency of pipelines determines how quickly teams can diagnose and respond. What’s next in the journey With CI/CD establishing control over how AI systems change, the next focus is how those changes are introduced safely at runtime. The following posts cover deployment strategies, progressive rollouts, and operational patterns that allow AI apps and agents to evolve while remaining stable, observable, and ready for Marketplace scale. Key resources See curated, step-by-step guidance to help you build, publish, or sell your app or agent (no matter where you start) in App Advisor Quick-Start Development Toolkit can connect you with code templates for AI solution patterns Microsoft AI Envisioning Day Events How to build and publish AI apps and agents for Microsoft Marketplace Get over $126K USD in benefits and technical consultations to help you replicate and publish your app with ISV Success
