Design CI/CD for AI apps and agents selling through Microsoft Marketplace

Microsoft

May 11, 2026

In the previous post, Design observability for AI apps and agents selling through Microsoft Marketplace, we focused on observability—making AI app and agent behavior visible and explainable. Execution paths, retries, degradation patterns, and agent decisions can now be observed across environments and tenants. With that visibility in place, a new challenge emerges: how do you safely modify an AI system whose behavior you can now observe?

You can always get curated step-by-step guidance through building, publishing and selling apps for Marketplace through App Advisor.

This post is part of a series on building and publishing well-architected AI apps and agents in Microsoft Marketplace. The series focuses on AI apps and agents that are architected, hosted, and operated on Azure, with guidance aligned to building and selling solutions through Microsoft Marketplace.

Using continuous integration/continuous delivery (CI/CD) to control AI system evolution

AI apps and agents introduce numerous novel ways that production behavior can change. In addition to application code, updates to configuration, prompts, models, and guardrails, agent logic can alter execution, cost, and outcomes—often immediately and across tenants.

CI/CD defines how these changes reach production. Without a structured delivery path, behavior‑shaping updates risk entering runtime without validation or recovery paths, making system behavior difficult to explain or reverse once customers encounter it.

AI solutions are typically built and operated as cloud applications. Software delivery of cloud services, and the supporting components that enable it, remains part of the CI/CD pipeline, and any instability in these foundational components directly propagates into AI behavior. AI systems add two additional sources of change that require explicit control.

MLOps governs model evolution.
Agents introduce further variability, as agent logic and configuration evolve.

CI/CD is what prevents these change vectors from interacting unpredictably across both publisher and customer environments.

Core CI/CD requirements for AI apps and agents

For AI apps and agents, CI/CD determines whether deployment strategies can be applied safely. Progressive rollouts, ring deployments, feature flags, and kill switches all rely on pipelines that isolate change, validate behavior, and support rollback. Observability provides insight into behavior; CI/CD controls when and how that behavior is allowed to change.

CI/CD must reliably provision, configure, and promote cloud native infrastructure, including but not limited to front-end services, APIs, storage, identity, and networking across environments. Agent behavior depends directly on the stability of the platform it runs on.

AI systems introduce additional CI/CD requirements through MLOps and agents. Model versions, routing logic, and evaluation configurations must move through pipelines as deployable artifacts, with isolation, validation, and rollback built in. Changes to models affect latency, cost, and outcomes even when application code remains unchanged, making promotion controls necessary at the model layer.

A well-run CI/CD pipeline should positively impact AI models and agents in the following ways:

Change isolation ensures code, prompts, models, and configuration evolve independently.
Artifact versioning beyond code treats prompts, policies, tools, and models as release assets.
Behavioral validation evaluates outcomes, constraints, and patterns rather than single responses.
Safe promotion controls gate model and agent releases based on observed behavior.
Rollback readiness allows fast reversion when model or agent behavior degrades.

Building behavioral baselines for AI solutions using CI/CD

Before an AI system is built by a pipeline, it is built by a team. CI/CD build pipelines are where these contributions are stitched together. Product managers define scope and constraints. UX designers shape how behavior is experienced. Full‑stack engineers assemble application logic. AI engineers wire reasoning and tools. Data engineers and data scientists curate data and models.

In AI systems, a build does more than compile code. It captures a shared agreement across roles about what the system is expected to do. Application code, orchestration logic, prompts, configuration, guardrails, routing rules, and trained or fine‑tuned models are assembled into a single versioned artifact. That artifact represents a coordinated snapshot of intent, behavior, and constraints.

This coordination must declare which models, prompts, policies, and tool definitions are included. Implicit dependencies—such as dynamically changing prompts or unpinned models—break shared understanding across teams and introduce behavior changes without acknowledgement.

A successful build confirms that contributions from multiple roles are compatible and executable together. It does not decide when customers see the change. That decision belongs later, where behavior can be evaluated deliberately, enforced by build pipelines that are separating assembly from release.

Testing AI solutions with CI/CD pipelines

When an agent is updated, the first task is straightforward: the agent’s code changes. Logic is refined, tools are added, limits are adjusted. That change moves through the CI/CD pipeline, where it is built, packaged, and validated in isolation. At this point, the focus is narrow—does this agent compile, configure, and execute as expected?

The second step widens the lens. The update now moves through testing aligned to the layers beneath it. For cloud solutions, tests confirm the platform still behaves as assumed: infrastructure provisions correctly, APIs and identity boundaries remain intact, and dependencies remain reachable. These tests ensure the environment can support execution before behavior is evaluated.

Next, MLOps tests assess whether model behavior still aligns with system expectations. New model versions, routing logic, or provider changes are evaluated for cost, latency, and outcome consistency. The goal is not identical responses, but bounded behavior within known limits.

Finally, testing shifts to the agentic system as a whole. Other agents need to be made aware of the new capabilities. When you go to update the agent the first job you have to do is update the agent code. The second job you have to do is to use your CI/CD pipeline to build, test and release that code. The third job is to test that the entire agentic system is running smoothly together. At this stage, testing answers a different question: not does the agent work, but does the system still work together.

CI/CD release management as team coordination

Once testing confirms that behavior remains within expected bounds, release management determines how changes are introduced and observed under real conditions. In AI systems, release management must reflect where change originates and how risk propagates across layers.

Within the cloud services that support the AI solution, release management focuses on scope and blast‑radius control. Examples include staged rollout of infrastructure updates, controlled exposure of new API versions, and limiting configuration changes to specific environments or tenants before going global. These steps allow both publisher and customer teams to observe stability and dependency behavior under load.

For MLOps, release management governs behavioral shifts introduced by model changes. Common patterns include routing a small percentage of requests to a new model version, limiting exposure to specific customer segments, or restricting usage to defined request types. This allows teams to compare cost, latency, and outcome patterns before expanding exposure.

For agents, release management controls how new behaviors surface. Prompt updates, tool access changes, or guardrail adjustments may be released to specific workflows, tenants, or traffic slices. This makes it possible to observe planning depth, retry behavior, and termination patterns without affecting all users simultaneously.

Rollback readiness remains essential. Release paths must allow fast reversion using version pinning or traffic shifting rather than full redeployment. Release management creates space to observe, adjust, and respond before changes reach full Marketplace scale.

Deployment as a shared boundary

Effective deployment pipelines ensure that software, models, and agent behavior enter production together, with changes explicitly acknowledged and observable. Versioning and rollback remain available, but deployment defines the moment when coordinated decisions become customer‑visible.

Cloud service—For the software, deployment governs application code and supporting platform changes. These remain necessary foundations. Application binaries, infrastructure templates, runtime configuration, and orchestration must enter production in a known, versioned state so operational behavior can be correlated with specific changes.

MLOps—Model version updates, routing rules, provider switches, and evaluation configurations can change system behavior without modifying application code. Deployment pipelines must therefore treat these artifacts as deployable units, subject to the same versioning, promotion, and rollback mechanics as software releases.

Agent—Deployment includes behavior‑defining inputs such as prompts and system messages, tool definitions and permissions, guardrails, and execution limits. Changes directly affect how agents plan, execute, and terminate work. Allowing these inputs to change outside deployment pipelines breaks traceability and weakens accountability across teams.

How CI/CD best practices positively impact marketplace readiness

Customers expect updates to arrive in predictable ways. They expect that behavior changes can be explained, that issues can be reversed without prolonged disruption, and that outcomes remain consistent across trials and production use. CI/CD pipelines make these expectations achievable by ensuring changes are versioned, staged, and observable as they move through environments. Reliability depends on limiting how far unstable behavior propagates. Billing accuracy depends on knowing when changes alter execution paths, token usage, or metering logic. Compliance depends on being able to identify which versions of software, models, and agent configurations were active at a given time.

Offer type shapes how CI/CD is applied. For transactable SaaS offers, CI/CD operates entirely within the publisher’s environment. For container offers and Azure Managed Applications, deployment boundaries extend to customer environments requiring a CI/CD hand-off between publisher and customer pipelines.

Publisher CI/CD responsibilities for AI solutions

Publishers must define what constitutes a deployable change. Updates to software, models, prompts, agent configuration, guardrails, or limits should not enter customer environments or generally available code implicitly. Each change that can influence behavior must flow through the publisher’s CI/CD pipelines so it can be versioned, observed, and reversed if necessary.

Additionally, CI/CD pipelines require validation and approval before promotion, ensuring that behavior‑altering updates do not reach customers without visibility or control. Publishers are also responsible for communicating behavior changes. Customers should be able to understand when updates affect outcomes, performance, or cost profiles.

Customers should never experience silent behavior shifts, undocumented updates, or releases that cannot be recovered cleanly. When those occur, trust erodes quickly. In this context, CI/CD is part of how publishers establish reliability, accountability, and trust with Marketplace customers.

Customer’s responsibility: CI/CD across environments (Dev / Stage / Prod)

While publishers own CI/CD pipelines, customers play an important role in how AI systems are evaluated and adopted across environments. AI behavior often manifests differently across Dev, Stage, and Prod because operating conditions change as systems move toward real usage. As environments scale, dependency interactions increase, traffic patterns diversify, and tenant behavior becomes less predictable—revealing execution paths and constraints that are not exercised earlier. These differences affect how behavior appears during evaluation and rollout.

To keep behavior interpretable across environments, pipeline structure matters. CI/CD pipelines, validation steps, and promotion criteria should operate consistently so signals observed earlier can be understood later. When these mechanics diverge between environments, it becomes difficult to attribute changes in behavior to specific updates or conditions.

Staging environments serve as a behavioral proving ground. They allow customers to observe retries, limits, degradation paths, and cost behavior under conditions that more closely resemble production. Trials often run against production‑like configurations, which means CI/CD gaps surface early. When behavior differs from expectations, the consistency of pipelines determines how quickly teams can diagnose and respond.

What’s next in the journey

With CI/CD establishing control over how AI systems change, the next focus is how those changes are introduced safely at runtime. The following posts cover deployment strategies, progressive rollouts, and operational patterns that allow AI apps and agents to evolve while remaining stable, observable, and ready for Marketplace scale.