Bring Marketplace publishing into your agentic development workflow
Bringing Marketplace publishing into the development workflow can help teams connect product development and publishing activities through the tools and processes they already use. Teams build their product, then move into a different set of tools and processes to prepare, validate, and submit it. Marketplace Ingestion MCP creates an opportunity to bring those activities closer together. With Marketplace Ingestion MCP, publishers can use natural language and AI agents to create, validate, publish, and manage offers in Microsoft Marketplace. Instead of starting by manually constructing API payloads, publishers can describe the outcome they want. An agent can then help translate that intent into the information required by the underlying publishing APIs. Publishers can also build custom workflows using the MCP, connecting repeatable publishing activities with the development processes and tools their teams already use. The shift is straightforward: start with the publishing outcome, not the API payload. From publishing task to agentic workflow Marketplace Ingestion MCP is a Model Context Protocol (MCP) server that connects MCP-compatible agent tools with Partner Center APIs. For example, a publisher can ask an agent to create a SaaS offer and identify what is required before it can be published. The agent can retrieve the appropriate schema, build the payload, validate it, and create the offer as a draft. When the publisher is ready to continue, the agent can perform supported publishing actions. Publishers remain in control through the approval settings in their MCP client. This makes Marketplace publishing a more natural part of an agentic development workflow rather than a separate process that begins after development is complete. Four opportunities for agent-assisted publishing Teams exploring where Marketplace Ingestion MCP fits into their development lifecycle can start with four opportunities. 1. Start with intent Developers do not need to begin by manually assembling an API payload. They can describe the offer they want to create or the update they want to make. The agent can use Marketplace Ingestion MCP to retrieve the required schema, build the payload, and validate it before submission. This allows the publisher to focus first on the intended outcome while the agent handles more of the translation into API-specific structures. 2. Make repeated work part of the workflow Offer creation and updates often include steps that teams perform repeatedly. Those steps can become reusable agentic workflows and, where appropriate, be incorporated into continuous integration and continuous delivery (CI/CD) pipelines. The question for a development team now becomes: Which publishing activities do we repeat often enough that they should become part of our development workflow? The answer will differ across organizations, products, and release processes. Marketplace Ingestion MCP provides the building blocks teams can use to design a workflow that fits their needs. 3. Keep people in control Agent-assisted publishing does not require handing every decision to an agent. MCP client approval settings can require publishers to review and approve tool calls before changes are submitted. Teams can use those controls to determine where automation is useful and where human review should remain part of the process. 4. Work from familiar developer tools Marketplace Ingestion MCP works with supported MCP-compatible clients, including Visual Studio Code and GitHub Copilot CLI. This brings Marketplace publishing tasks closer to the environments where developers already build and operate their products. It also reduces the need to treat publishing as an isolated activity performed through a separate workflow. What does this look like in practice? Consider a fictional publisher, Contoso, that wants to bring a Microsoft 365 agent to customers. Contoso can provide its workflow integrated with the MCP with existing marketing material and the technical manifest for the agent. Marketplace Ingestion MCP can help translate that information into the structure required to create the product in Microsoft Marketplace. The workflow can then build on that information to create a linked SaaS offer for monetization, using the additional details required for the SaaS offer. The publisher describes what it wants to accomplish, and the agent uses Marketplace Ingestion MCP and the publishing APIs to turn that intent into supported publishing actions. Getting started Marketplace Ingestion MCP is included with access to Microsoft Marketplace publishing APIs. Before getting started, publishers need: A Partner Center account with the required role Access to Microsoft Marketplace publishing APIs A supported MCP-compatible client, such as Visual Studio Code or GitHub Copilot CLI The Microsoft Learn documentation covers prerequisites, installation and authentication, available tools, supported products, data handling, and an end-to-end SaaS example. Learn more: Marketplace Ingestion MCP server - Marketplace publisher Bringing publishing closer to development AI agents are becoming part of how developers build and operate software. Marketplace publishing can become part of those workflows as well. Marketplace Ingestion MCP connects natural-language intent with supported Marketplace publishing actions. It allows teams to create repeatable workflows while keeping publishers involved in reviewing and approving changes. The result is a closer connection between building a product and bringing it to market, helping developers focus on the cloud and AI solutions that differentiate their business. Thank you to the many teams whose collaboration brought Marketplace Ingestion MCP to publishers. Their work reflects a shared investment in helping publishers build, publish, and grow through Microsoft Marketplace.Explore App Advisor for details on how Frontier Accelerate for Marketplace can help you sell more
Explore the latest Frontier Accelerate for Marketplace updates available through App Advisor. Learn how software development companies can discover relevant Microsoft Marketplace resources, benefits, and commercialization opportunities that help accelerate solution development, reach new customers, and scale revenue growth through Marketplace.Microsoft Marketplace Partner Digest - August 2026
What's new in Marketplace Convert more buyers into customers Microsoft recently expanded Marketplace listing capabilities with enhancements to free trials and the introduction of an option to request private offers, giving partners more ways to engage buyers at different stages of the purchasing journey. Customers can now discover and evaluate solutions through expanded trial experiences and transition more easily into paid subscription discussions by requesting custom pricing and terms directly from your Marketplace listings. ✨ Help customers evaluate your solution with free trials What's new On July 23 we announced the release of Marketplace enhanced trial capabilities. Customers can try solutions with confidence with the free trials in Microsoft Marketplace. Trials are available for multiple offer types, including SaaS, Azure virtual machines, Dynamics 365, and Power BI offers. New capabilities include: Custom meter trials: Partners can configure trials for SaaS solutions with metered pricing. Flexible trial durations: Partners can now configure SaaS trial durations from 1 to 180 days. Partner analytics: Partners can access SaaS trial analytics - including free trials initiated, active free trials, free trials converted, free trials not converted, and average trial duration - all within Partner Center Insights workspace. Customer discovery and management: Customers can better discover and manage trials. Upgrade to paid subscription during trial: Customers can upgrade to paid subscription at public pricing any time during a SaaS trial. Why it matters With enhanced trial capabilities in Microsoft Marketplace, partners can run self-service trials that make it easier for customers to try before they buy — whether for metered, per-user, or flat-rate pricing offers. Marketplace gives partners one place to manage trials end to end — from provisioning and mid-trial changes to insights — while improving offer discoverability through improved search, filters, and dedicated trial badges. Recommended action Review your Marketplace offers and trial strategy. For eligible SaaS offers, evaluate whether free trials can help you engage with more qualified buyers, faster. 📖 Learn more about free trials for SaaS ⚡ Customers can now request a private offer on Marketplace What’s new As of July 20, you can add a private offer request option directly on your Marketplace product pages. Customer requests are captured as Marketplace leads in referrals workspace and connected customer relationship management (CRM) integrations you’ve setup. This setting is disabled by default. You can easily activate it for applicable public offers. Why it matters More and more customers are seeking customized contracts and subscription pricing and terms tailored to their individual needs. This simple call to action reduces friction and helps you start the conversation with prospective buyers. Recommended action In the Marketplace offers workspace of Partner Center, update each of your offers for which you provide customers private offers. Enable the Request Private Offer setting and republish the offer. Events Recent events Multiparty private offers expand channel growth opportunities in Australia, Japan and South Africa Multiparty private offers continue to help software companies accelerate channel-led sales through Microsoft Marketplace. This past month, we hosted two region-focused Marketplace office hours sessions designed to help partners better understand local market opportunities, engage distributors and resellers, and scale Marketplace transactions through channel ecosystems in Australia and Japan. Whether you're looking to expand an existing Marketplace motion or explore new geographic opportunities, these sessions will provide practical guidance and market-specific insights to help you grow. See a list of currently supported countries/regions. Check out the recorded sessions: • Channel growth in Australia (English) • Channel growth in Japan (Japanese) Helping developers build, monetize, and scale AI solutions with Marketplace Discover how Microsoft Marketplace helps software companies access AI models and developer tools, accelerate application development, and reach customers through Microsoft's global commercial ecosystem. Learn how to monetize applications and agents without building and maintaining your own commerce and distribution infrastructure. Watch the recording Upcoming events Build AI-powered solutions, monetize through Microsoft Marketplace, and scale revenue through Microsoft expertise, skilling, and investments. Frontier Accelerate for Marketplace is an upcoming unified offering for software development companies to bring AI-powered solutions to market, drive customer acquisition and revenue growth, and scale through Microsoft Marketplace with technical guidance, skilling, and investments aligned to every stage of growth. 📆 Join us on August 26th before general availability to learn about the new offering, benefit alignment, migration guidance, and enrollment requirements. Next steps Turn insights into FY27 success — starting today. Ready to carry the momentum forward from MCAPS Start for Partners? Get everything you need to refine and execute your FY27 strategy in the Partner Activation Zone, including session recordings, ready-to-use playbooks, and tools to start building pipeline. Leverage your Marketplace Rewards benefits Discover use cases and how to leverage your Azure sponsorship. With Marketplace Rewards Azure sponsorships, eligible software development companies can increase sales through Microsoft Marketplace by using the sponsorships for customer deployments tied to Marketplace deals—and to offset infrastructure costs from eligible free trial offers. 📍Review the Marketplace Rewards Azure sponsorship policy guide for the latestDecember 2025
Microsoft Ignite 2025 - Marketplace highlights Microsoft Ignite was packed with announcements and insights for Marketplace partners. From new commerce capabilities to AI-driven innovations, here are some key takeaways: Global expansion of Microsoft Marketplace - Microsoft announced that the reimagined Microsoft Marketplace, which launched in the U.S. earlier this year, is now globally available. This expansion includes new APIs for distribution partners, enabling them to link their own cloud marketplace with Microsoft’s, opening significant opportunities for software companies in SMB and mid-market segments. 🎬 Watch a recorded webinar with TD SYNNEX on the power of distribution to accelerate SMB marketplace sales. Global availability of Resale Enabled Offers - This capability allows software development companies to and channel partners to resell software solutions directly through Marketplace, simplifying transactions, expanding reach, and scaling revenue. 👉 Read more about this announcement and get started Introducing App Accelerate - A unified offer that brings together incentives, benefits, and co-sell support across the Microsoft Cloud. App Accelerate provides end-to-end technical guidance, developer tools, and go-to-market resources so software development companies can innovate and scale. Previews are beginning now, with full availability planned for 2026. ✅ Sign up to receive updates Enhanced Partner Marketing Center - Discover, customize, and launch campaigns faster with intelligent search and AI-powered tools—all on one connected platform. The current Partner Marketing Center will remain available as the new and enhanced Marketing Center platform launches in early 2026 with 24 campaigns-in-a-box, aligned to FY26 solution plays. ✨ Get ready for the new era of partner marketing Frontier Partner badge – New customer-facing badges recognize top services, channel, and software development company partners that are driving AI transformation with customers and offer them an opportunity to differentiate themselves from the competition. 🛡️Differentiate your AI-first leadership Catch up on Microsoft Ignite sessions Ignite 2025 delivered powerful insights and announcements for Marketplace partners, and now you can catch up on the sessions you missed. Explore these recorded keynotes to learn about new capabilities, partner programs, and strategies to accelerate growth through Microsoft’s ecosystem. Ignite opening keynote Ignite partner keynote: Powering Frontier Partnerships Additionally, we’ve compiled recordings of relevant Marketplace partner and customer sessions so you can watch on-demand. Revisit Marketplace-focused sessions and resources. Just look for the ✨ icon below. Partner sessions: PBRK415 Grow your business with Microsoft AI Cloud Partner Program Find out how the Microsoft AI Cloud Partner Program helps you grow with new benefits, designations, and skilling opportunities. This session covers updates like the Frontier Partner Badge, Copilot specialization, and streamlined Marketplace engagement—all designed to accelerate your AI transformation journey. PBRK416 Accelerate Growth through Partner Incentives Explore how Microsoft is boosting partner growth with streamlined incentives, AI-first strategies, and new designations like Frontier Distributor. This session covers expanded investments in Azure Accelerate, Copilot solutions, and security practices—plus insights on how to capitalize on evolving programs and co-sell opportunities. PBRK417 Partner: Connect, Plan, Win – Enhancing Co-sell Engagement Discover how to enhance collaboration, optimize joint efforts, and drive success in shared initiatives. Gain insights into improving interactions with Microsoft sellers and leveraging opportunities, along with guidance on proactive co-selling to align your goals with Microsoft's for sustained growth. PBRK418 Partner: Benefits for Accelerating Software Company Success Learn about the resources and benefits available for software development companies across all stages of the build, publish and grow journey in MAICPP. Whether you’re developing a new agent solution or working toward a certified software designation, there are targeted skilling opportunities, technical resources, and GTM benefits to help. Tap into new investments for AI apps and agents and hear from your peers on how they’ve used rewards such as customer propensity scores and Azure sponsorship. PBRK419 SI & Advisory Partner Readiness: Accelerating the Journey to Frontier Understand how Microsoft is empowering our SI and advisory partners to accelerate frontier firm readiness for our Enterprise customers by driving AI transformation with agentic solutions and services. ✨PBRK420 Executing on the channel-led marketplace opportunity for partners See how Microsoft’s unified Marketplace drives partner growth with resale-enabled offers, creating scalable channel sales and co-sell opportunities. This session shares practical steps to build a sustainable Marketplace practice and leverage the partner ecosystem for greater reach and profitability. PBRK421 Enabling a thriving partner ecosystem: New CSP Authorization Criteria Dive into what’s new for Cloud Solution Providers, including updated authorization requirements and designations that help you stand out. This session covers steps to choose the right tier, build trust as a customer advisor, and prepare for growth with AI-driven solutions and Copilot offerings. PBRK422 The Future of Partner Support: Customer + Partner + Microsoft Discover ‘Unified for Partners,’ Microsoft’s new support model designed for CSP partners to deliver customer success at scale. This session introduces the Support Services designation, offering faster response times, financial incentives, and integrated tools to strengthen your support capabilities. PBRK423 Partner Execution at Scale with SME&C Explore growth opportunities in the high-potential SME&C segment. This session highlights investments in co-selling, AI-first strategies, and what it means to become ‘customer zero,’ with examples of frontier firms driving innovation at scale. ✨PBRK424 Marketplace Success for Partners—from SMB to Enterprise Learn how to build, publish, and monetize AI-powered solutions through Microsoft Marketplace. This session shares a proven approach to align your Marketplace strategy with your sales motion and unlock new revenue opportunities. PBRK272 Accelerate Secure AI: Microsoft’s Security Advantage for Partners Explore Microsoft’s integrated security solutions and learn how to help customers strengthen their defenses in the AI era. This session highlights partner opportunities, resources to grow your security practice, and what it takes to lead as a next-generation security partner. Customer Sessions: ✨Microsoft Marketplace: Your trusted source for cloud solutions, AI apps, and agents | STUDIO47 Hear from Cyril Belikoff, VP of Commercial Cloud & AI Marketing, sharing the reimagined Microsoft Marketplace—the gateway to thousands of AI-powered apps, agents and cloud solutions—all built to accelerate innovation and drive business outcomes. Discover how customers benefit from faster deployment, seamless integration with Microsoft tools, and trusted solutions, and how partners can scale their reach, accelerate sales, and tap into Microsoft’s global ecosystem. Azure Accelerate in action: Confidently migrate, modernize, and build faster Join Cyril Belikoff for a rapid Q&A that spotlights real-world customer success and the transformative impact of Azure Accelerate. Hear how customers like Thomson Reuters achieved breakthrough results with our powerful offering that provides access to Microsoft experts and investments throughout your Azure and AI journey. ✨BRK213 Microsoft Marketplace: Your trusted source for cloud and AI solutions Discover how the reimagined Microsoft Marketplace is reshaping the future of cloud and AI innovation. In this session, we’ll explore how Microsoft Marketplace—unifying Azure Marketplace and Microsoft AppSource—empowers organizations to become Frontier Firms by streamlining the discovery, purchase, and deployment of tens of thousands of cloud solutions, AI apps, and agents. ✨BRK215 Boost cloud and AI ROI using Microsoft Marketplace As organizations embrace an AI-first future, cloud adoption is accelerating to drive innovation and efficiency. This session explores practical strategies to optimize cloud investments—balancing performance, scalability, and cost control. Learn how Microsoft Marketplace enables rapid solution deployment while maintaining governance, compliance, and budget discipline. Build a resilient, cost-effective cloud foundation that supports AI and beyond. Community Recap Partner of the Year Award Winners Congratulations to the winners and finalists of the 2025 Microsoft Partner of the Year Awards in the Marketplace category! 🏆 Explore all winners and finalists Fivetran earned the top honor as Marketplace Partner of the Year for its innovation in automating data movement on Microsoft Azure, enabling enterprises to accelerate AI and analytics initiatives. Varonis Systems Inc. and Bytes Software Services were recognized as finalists for delivering exceptional solutions and driving customer success through Marketplace. What’s Coming Up AI-powered acceleration: Scale faster in Microsoft Marketplace 📆 Thursday, December 04, 2025, at 9:00 AM PST Microsoft Marketplace is no longer just a procurement convenience; it’s a strategic revenue engine. Dive into operational readiness, CRM-native automation, seller engagement, trust signals, and AI-enabled acceleration. Whether you're just getting started or looking to optimize your Marketplace motion, this session will provide you with information that will turn your first sale into a repeatable growth engine. Scale smarter: Discover how resale enabled offers drive growth 📆 Friday, December 05, 2025, from 11:00 - 12:00 PM GTM+1 Discover how resale enabled offers help software development companies to scale through the Microsoft Marketplace by simplifying transactions, expanding reach and accelerating co-sell opportunities. Chart your AI app and agent strategy with Microsoft Marketplace 📆 Thursday, December 11, 2025, from 8:30 - 9:30 AM PST Organizations exploring AI apps and agents face a critical choice: build, buy, or blend. There’s no one-size-fits-all—each approach offers unique benefits and trade-offs. Tune in for insights into the pros and cons of each approach and explore how the Microsoft Marketplace simplifies adoption by providing a single source for trusted AI apps, agents, and models. Office hours for partners: Marketplace resale-enabled offers 📆 Thursday, December 18, 2025, at 8:30 AM PST Tune in to explore resale enabled offers through Microsoft Marketplace. This recently announced capability enables software companies to expand into new markets globally, at scale, and without additional operational overhead. Dive deep into the workflow and requirements for these deals. Learn about reporting and best practices from those that are already selling globally with resale enabled offers. Microsoft Ignite will return to San Francisco next year 📆 November 17-20, 2026 Sign up now to join the Microsoft Ignite early-access list and be eligible to receive limited‑edition swag at the event. 💬 Share Your Feedback! We truly appreciate your feedback and want to ensure these Partner Digests deliver the information you need to succeed in the marketplace. If you have any feedback or suggestions on how we can continue to improve the content to best support you, we’d love to hear from you in the comments below!Build a repeatable Marketplace growth engine with App Advisor to sell more
Publishing your offer is only the beginning. Learn how App Advisor helps software development companies build a repeatable Marketplace growth engine with curated go-to-market guidance, Marketplace Insights, AI-powered optimization, and proven best practices that drive long-term visibility, engagement, and revenue growth.Why some apps reach revenue faster than others—and how App Advisor helps close the gap
Your team is working hard to build powerful apps, agents, AI solutions, and industry-specific offerings designed to solve real customer problems. The challenge is getting those solutions to market—and generating revenue at scale—as quicklyClarity at every stage: App Advisor turns Marketplace complexity into action
For software development companies, the Microsoft Marketplace journey isn’t always linear. It’s a series of decisions: what to build, how to package it, how to publish, and how to sell it, each with real dependencies and real friction.Quality and evaluation framework for successful AI apps and agents in Microsoft Marketplace
Why quality in AI is different — and why it matters for Marketplace Traditional software quality spans many dimensions — from performance and reliability to correctness and fault tolerance — but once those characteristics are specified and validated, system behavior is generally stable and repeatable. Quality is assessed through correctness, reliability, performance, and adherence to specifications. AI apps and agents change this equation. Their behavior is inherently non-deterministic and context‑dependent. The same prompt can produce different responses depending on model version, retrieval context, prior interactions, or environmental conditions, which is why non-deterministic AI testing is required to evaluate it reliably. For agentic systems, quality also depends on reasoning paths, tool selection, and how decisions unfold across multiple steps — not just on the final output. This means an AI app can appear functional while still falling short on quality: producing responses that are inconsistent, misleading, misaligned with intent, or unsafe in edge cases. Without a structured evaluation framework, these gaps often surface only in production — in customer environments, after trust has already been extended. For Microsoft Marketplace, this distinction matters. Buyers expect AI apps and agents to behave predictably, operate within clear boundaries, and remain fit for purpose as they scale. AI quality evaluation turns those expectations into something observable — and that visibility is what determines Marketplace readiness. You can always get a curated step-by-step guidance through building, publishing and selling apps for Marketplace through App Advisor. This post is part of a series on building and publishing well-architected AI apps and agents in Microsoft Marketplace. The series focuses on AI apps and agents that are architected, hosted, and operated on Azure, with guidance aligned to building and selling solutions through Microsoft Marketplace. How quality measurement shapes Marketplace readiness AI apps and agents that can demonstrate quality — with documented evaluation frameworks, defined release criteria, and evidence of ongoing measurement — are easier to evaluate, trust, and adopt. Quality evidence reduces friction during Marketplace review, clarifies expectations during customer onboarding, and supports long-term confidence in production. When quality is visible and traceable, the conversation shifts from "does this work?" to "how do we scale it?" — which is exactly where publishers want to be. Publishers who treat quality as a first-class discipline build the foundation for safe iteration, customer retention, and sustainable growth through Microsoft Marketplace. That foundation is built through the decisions, frameworks, and evaluation practices established long before a solution reaches review. What "quality" means for AI apps and agents Quality for AI apps and agents is not a single metric — it spans interconnected dimensions that together define whether a system is doing what it was built to do, for the people it was built to serve. The HAX Design Library — Microsoft's collection of human-AI interaction design patterns — offers practical guidance for each one. These dimensions must be defined before evaluation begins. You can only measure what you have first described. Accuracy and relevance — does the output reflect the right answer, grounded in the right context? HAX patterns Make clear what the system can do (G1) and notify users when the AI is uncertain (G10) help publishers design systems where accuracy is visible and outputs are understood in the right context — not treated as universally authoritative. Safety and alignment — does the output stay within intended use, without harmful, biased, or policy-violating content? HAX patterns Mitigate social biases (G6) and Support efficient correction (G9) help ensure outputs stay within acceptable boundaries — and that users can identify and address issues before they cause downstream harm. Consistency and reliability — does the system behave predictably across users, sessions, and environments? HAX patterns Remember recent interactions (G12) and notify users about changes (G18) keep behavior coherent within sessions and ensure updates to the model or prompts are never silently introduced. Fitness for purpose — does the system do what it was designed to do, for the people it was designed to serve, in the conditions it will actually operate in? HAX patterns make clear how well the system can do what it does (G2) and Act on the user's context and goals (G4) ensure the system responds to what users actually need — not just what they literally typed. These dimensions work together — and gaps in any one of them will surface in production, often in ways that are difficult to trace without a deliberate evaluation framework. Designing an evaluation framework before you ship Evaluation frameworks should be built alongside the solution. At the end, gaps are harder and costlier to close. The discipline mirrors the design-in approach that applies to security and governance: decisions made early shape what is measurable, what is improvable, and what is ready to ship. A well-structured evaluation framework defines five things: What to measure — the quality dimensions that matter most for this solution and its intended use cases. For AI apps and agents, this typically includes task adherence, response coherence, groundedness, and safety — alongside the fitness-for-purpose dimensions defined in the previous section. How to measure it — the methods, tools, and benchmarks used to assess quality consistently. Effective evaluation combines AI-assisted evaluators (which use a model as a judge to score outputs), rule-based evaluators (which apply deterministic logic), and human review for edge cases and safety-relevant responses that automated methods cannot fully capture. Who evaluates — the right combination of automated metrics, human review, and structured customer feedback. No single method is sufficient; the framework defines how each is applied and when human judgment takes precedence. When to evaluate — at defined milestones: during development to establish a baseline, pre-release to validate against acceptance thresholds, at rollout to catch regression, and continuously in production to detect drift as models, prompts, and data evolve. What triggers re-evaluation — model updates, prompt changes, new data sources, tool additions, or meaningful shifts in customer usage patterns. Re-evaluation should be a scheduled and triggered discipline, not an ad hoc response to visible failures. The framework becomes a shared artifact — used by the publisher to release safely, and by customers to understand what quality commitments they are adopting when they deploy the solution in their environment. Evaluate your AI agents - Microsoft Foundry | Microsoft Learn Evaluation methods for AI apps and agents Quality must be assessed across complementary AI evaluation methods — each designed to surface a different category of risk, at a different stage of the solution lifecycle. Automated metric evaluation — evaluators assess agent responses against defined criteria at scale. Some use AI models as judges to score outputs like task adherence, coherence, and groundedness; others apply deterministic rules or text similarity algorithms. Automated evaluation is most effective when acceptance thresholds are defined upfront — for example, a minimum task adherence pass rate before a release proceeds. Safety evaluation — a dedicated evaluation category that identifies potential content risks, policy violations, and harmful outputs in generated responses. Safety evaluators should run alongside quality evaluators, not as a separate afterthought. Human-in-the-loop evaluation — structured expert review of edge cases, borderline outputs, and safety-relevant responses that automated metrics cannot fully capture. Human judgment remains essential for interpreting context, intent, and impact. AI agent red-teaming and adversarial testing — probing the system with challenging, unexpected, or intentionally misused inputs (including prompt injection attempts and tool misuse) to surface failure modes before customers encounter them. Microsoft provides dedicated AI red teaming guidance for agent-based systems. Customer feedback loops — structured collection of real-world signals from users interacting with the system in production. Production feedback closes the gap between what was tested and what customers actually experience. Each method has a distinct role. The evaluation framework defines when and how each is applied — and which results are required before a release proceeds, a change is accepted, or a capability is expanded. Defining release criteria and ongoing quality gates Quality evaluation only drives improvement when it is connected to clear release criteria, often enforced through an LLMOps quality gate. In an LLMOps model, those criteria are automated gates embedded directly into the CI/CD pipeline, applied consistently at every stage of the release cycle. In continuous integration (CI), automated evaluations run with every change — whether that change is a prompt update, a model version, a new tool, or a data source modification. CI gates catch regressions early, before they reach customers, by validating outputs against predefined quality thresholds for task adherence, coherence, groundedness, and safety. In continuous deployment (CD), quality gates determine whether a build is eligible to proceed. Release criteria should define: Minimum acceptable thresholds for each quality dimension — a release does not proceed until those thresholds are met Known failure modes that block release outright versus those that are tracked, monitored, and accepted within defined risk tolerances Deployment constraints — conditions under which a release is paused, rolled back, or progressively expanded to a subset of users before full rollout Ongoing evaluation must be scheduled and triggered. As models, prompts, tools, and customer usage patterns evolve, the baseline shifts. LLMOps treats re-evaluation as a continuous discipline: run evaluations, identify weak areas, adjust, and re-evaluate before changes propagate. This connects directly to governance. Quality evidence — the record of what was measured, when, and against what criteria — is part of the audit trail that makes AI behavior accountable, explainable, and trustworthy over time. For more on the governance foundation this builds on, see Governing AI apps and agents for Marketplace readiness. Quality across the publisher-customer boundary Clear quality ownership reduces friction at onboarding, builds confidence during operation, and protects both parties when behavior deviates. In the Marketplace context, quality is a shared responsibility — but the boundaries are distinct. Publishers are responsible for: Designing and running the evaluation framework during development and release Defining quality dimensions and thresholds that reflect the solution's intended use Providing customers with transparency into what quality means for this solution — without exposing proprietary prompts or internal logic Customers are responsible for: Validating that the solution performs appropriately in their specific environment, with their data and their users Configuring feedback and monitoring mechanisms that surface quality signals in their tenant Treating quality evaluation as a shared ongoing responsibility, not a one-time publisher guarantee When both sides understand their role, quality stops being a handoff and becomes a foundation — one that supports adoption, sustains trust, and enables both parties to respond confidently when behavior shifts. What's next in the journey A strong quality framework sets the baseline — but keeping that quality visible as solutions scale is its own discipline. The next posts in this series explore what comes after the framework is in place: API resilience, performance optimization, and operational observability for AI apps and agents running in production environments. See the next post in the series: Designing a reliable environment strategy for Microsoft Marketplace AI apps and agents | Microsoft Community Hub. Key resources See curated, step-by-step guidance to help you build, publish, or sell your app or agent (no matter where you start) in App Advisor Quick-Start Development Toolkit can connect you with code templates for AI solution patterns Microsoft AI Envisioning Day Events How to build and publish AI apps and agents for Microsoft Marketplace Get over $126K USD in benefits and technical consultations to help you replicate and publish your app with ISV Success
