Model Migration Process on Microsoft Foundry and Azure OpenAI
Every app built on an LLM will eventually move to a new model. The model you shipped may be retired, or a newer model may offer better quality, cost, or performance. Changing a model name in code from a retiring model such as gpt-4o to a newer one such as gpt-5.1 may take one line. That line hides a much larger migration. Model failures are often silent to the systems using the model and loud to users. Nothing crashes. Error rates stay flat. Every dashboard says the migration went fine. Meanwhile, responses change shape, summaries become longer and more hedged, JSON fields disappear, and tool calls fire in a different order. Users notice. Support queues grow. Downstream code that depended on the old behavior starts to break. A successful migration preserves the application's behavior or improves it in measurable ways. That requires a repeatable process to detect drift, adapt safely, and prove quality before broad rollout. The model migration process has six phases: Discover → Assess → Adapt → Validate → Roll out → Retire. This article explains what each phase looks like, which Microsoft Foundry tools support it today—including Azure OpenAI capabilities—and where teams still need to build around the platform. It then applies the process to a retail shopping assistant and points to additional resources in Go deeper at the end of this post! Why migrate now Every model has a retirement date. On Microsoft Foundry, generally available models typically ship with a retirement date about 18 months out, and older model families are actively replaced. For example, the model lifecycle and retirement schedule lists gpt-4o (2024-05-13) as retiring on October 1, 2026, with gpt-5.1 as its replacement. What happens at retirement depends on how you buy capacity: Standard, Global Standard, and Data Zone Standard (pay-as-you-go) deployments are auto-upgraded on a rolling, region-by-region schedule. You control the timing with versionUpgradeOption set to one of: OnceNewDefaultVersionAvailable, OnceCurrentVersionExpired, or NoAutoUpgrade. NoAutoUpgrade means the deployment stops working at retirement. Priority Processing follows the same path. Provisioned (PTU) deployments are not auto-upgraded. You migrate them yourself, either in-place (traffic moves over a 20–30 minute window with no downtime) or side-by-side (stand up the new deployment, test, shift traffic, delete the old one). Batch deployments follow the side-by-side path: deploy the new model, resubmit jobs, retire the old deployment. The developer problem is the same in every case: traffic eventually reaches a different model, but the platform cannot tell you whether the application still behaves as it did before. A responding endpoint does not prove that the app behaves correctly. A new model can change formatting, tone, tool-calling behavior, or JSON shape in ways that quietly break downstream code. When should I migrate? Start before the retirement date. Automatic upgrade handles the traffic transition for eligible deployments, but the team still owns behavioral validation. Provisioned deployments also require a manual migration. Microsoft typically makes a replacement available in Global Standard about 90 days before retirement, in provisioned regions about 30 days before retirement, and in standard regions about two weeks before retirement. That gives you time to evaluate the new model on your own terms. Retirement dates cannot be extended. You also do not need a deprecation notice to begin. If a newer model may improve quality, speed, or cost, run it through the process now. Waiting turns the switchover into a slow train wreck: responses drift, parsing becomes brittle, support tickets accumulate, and the team ends up debugging a model it did not choose on a date it did not pick. A deliberate migration makes the retirement date a formality and creates a process the team can reuse. Who this is for This process fits teams that own an LLM-powered feature inside a larger application and run migrations deliberately. It also applies to AI-native platform teams that centrally manage models for other application teams. The phases remain the same, though platform teams may run them faster and in parallel rather than in sequence. Fine-tuned workloads are out of scope here because they cannot be upgraded automatically, have separate training and deployment retirement schedules, and turn the Adapt phase into a distillation or retraining exercise rather than primarily prompt work. The six phases Phase Definition What success looks like Discover Learn that a model change is coming or needed. The team receives a timely, structured signal with the deprecation date, replacement model, and migration window. Assess Choose a target model and confirm that it is operationally available. The team understands the candidates and confirms capacity, region, and SKU before tuning starts. Adapt Replay the current workload on the new model, diagnose changes, and update prompts, parameters, tool definitions, output schemas, and calling code. The team runs side-by-side replay against real or representative traffic, can see the behavioral differences, and records every change. Validate Run the adapted workload against a quality rubric and decide whether it is safe to ship. The team has an evaluation suite that is affordable to run and trusted by application owners and reviewers. Roll out Promote the model through staged production exposure, monitor live behavior, and commit or roll back. Canary or weighted routing is in place, live quality is measured alongside latency and errors, and rollback remains possible. Retire Decommission the old deployment, free capacity, archive evaluation artifacts, and update internal documentation. The old SKU is gone, the deployment count falls, and the team carries what it learned into the next migration. Foundry tools at a glance Microsoft Foundry provide tools for each phase of the Model Migration Process. Phase Microsoft Foundry feature (including Azure OpenAI) Documentation Discover Model retirement schedule, lifecycle policy, Service Health alerts, and Models API lifecycleStatus Model retirement schedule Lifecycle policy Assess Model leaderboards and benchmarks for quality, safety, cost, throughput, and latency; trade-off charts; side-by-side comparison; suggested replacements Model leaderboards and benchmarks Side-by-side compare Adapt Prompt Optimizer in the Foundry Agent playground; agent optimization; simulator for synthetic data Prompt Optimizer Agent optimization Simulator Validate Azure AI Evaluation SDK with 30+ evaluators, LLM-as-judge, graders, and the portal evaluation wizard Azure AI Evaluation SDK Portal evaluation Roll out Automatic upgrade and versionUpgradeOption; provisioned in-place or side-by-side migration; continuous evaluation; Azure Monitor alerts Auto-upgrade with versionUpgradeOption Continuous evaluation Retire Models API to confirm 410 Gone; observability dashboard to track deployment count Models API Observability dashboard Breakdown of each phase 0. Prepare the test dataset Before starting the six phases, build a set of representative inputs, expected outputs, and agreed success criteria. This dataset gates the middle of the lifecycle: Adapt needs inputs for replay, and Validate needs ground truth and scoring criteria. Step 0 describes the workload rather than the candidate model, so it can begin during Discover, before the team selects a target. Build the dataset from captured production traffic or domain examples in .csv or .jsonl. If representative data is not available, use the simulator to generate synthetic inputs. Two practices determine whether this work pays off: Instrument capture before you need it. Production content capture is opt-in and never retroactive. Log prompts, responses, latency, and token counts now so the team has traffic to evaluate later. Freeze the dataset. Keep inputs, ground truths, and success criteria fixed throughout the migration. If they change, source and target results are no longer comparable. You also need an inventory of the model deployments your workload uses, including their deployment types (Standard, Provisioned, or Batch). For each source model, note its retirement date and suggested replacement from the Model retirement schedule. 1. Discover Discover begins when something forces the team to consider a model change: a deprecation notice, a new generally available model, a cost or latency problem, or a capability gap. The phase ends with a decision to begin migration or stay on the current model if it remains stable, performs well, and is not approaching retirement. Foundry tools. The model lifecycle and retirement schedule publishes retirement dates and suggested replacements. The Azure OpenAI model retirements documentation explains notification timing, including at least 60 days for generally available model retirements and at least 30 days for preview model retirements. It also explains how to configure Azure Service Health advisories and use the Models API for programmatic lifecycleStatus and deprecation checks. Those APIs provide the foundation for an internal discovery system. Where it breaks. Customers may learn about a retirement through email, a service health alert, or a production error. By the time the right team sees the signal, it may already be deep into the deprecation window and heading toward retirement. What your team provides. The schedule and Models API expose the data through a stable contract. Mature enterprises may add a thin notification layer that routes it to the right owners. 2. Assess The team chooses a candidate target model and confirms that it is usable: the correct region and SKU, enough quota, and availability alongside the current model so rollback remains possible. Assess also includes projecting monthly cost against historical traffic. Pricing structures change between model generations through reasoning tokens, cached input, structured-output overhead, and other factors. Those changes can move unit economics by 2x or more. For regulated workloads, compliance requirements such as BAA, FedRAMP, and regional Standard versus Global Standard availability may narrow the candidate list before quality testing begins. Foundry tools. Start with the replacement suggested in the retirement schedule, then build a shortlist with model benchmarks, which compare quality, safety, cost, throughput, and latency. Use trade-off charts such as quality versus cost and the side-by-side model comparison for up to three models. Compare context windows, feature support such as function calling, structured output, and vision, and available endpoints. Confirm SKU, region, quota, and upgrade mechanics in the model retirements documentation. Where it breaks. Teams face several plausible candidates, such as gpt-5.1, gpt-5.2, and a nano variant, without clear positioning between them. A selected model may be unavailable in the required region or SKU, a constraint that sometimes appears only after planning is underway. Historical traffic may also show that the new model costs substantially more, forcing an unplanned budget decision. What your team provides. Public benchmarks should filter the candidate list, not make the final decision. Confirm the shortlist against the team's own workload. Build the monthly cost view from token logs and current pricing. 3. Adapt Adapt is often the most time-consuming phase for embedded and product-facing workloads. Validate may take longer for regulated workloads. First, replay the existing workload on the new model without changing it. This isolates changes caused by the model. Diagnose shifts in verbosity, reasoning depth, structured-output adherence, tool-call shape, and latency. Then update the application until it recovers or improves on the previous behavior. Prompt editing is only one part of Adapt. A migration often changes four other surfaces: Parameters. temperature, top_p, max_tokens, and reasoning-effort controls may not map directly between generations. Some are unsupported by newer model families. Tool definitions. Argument names, descriptions, and required fields that reliably guided the old model may need clearer wording or tighter constraints. Output schemas. Structured-output behavior changes between models. A schema the old model followed loosely may need explicit constraints, or the new model may finally enforce it. Calling code. API and SDK differences, including Chat Completions versus Responses, streaming formats, and new or renamed request fields, can require code changes. Downstream parsers may also assume the old response shape. For agentic and workflow workloads, schema and tool-call changes can outweigh prompt changes. Foundry tools. Prompt Optimizer is available through the Optimize button below the system instructions field in the Agent playground. It restructures instructions, explains each change by paragraph, and supports iteration. For example, a team can add a constraint such as "keep the JSON schema exactly" and optimize again. It is a fast first pass for a prompt that would otherwise be rewritten by hand. For agent workloads, agent optimization tunes instructions, tools, and model selection together. Prompt Optimizer and agent optimization are available in Microsoft Foundry, not Azure OpenAI. When production data is unavailable, the simulator can generate synthetic and adversarial inputs. Where it breaks. Most migration time is spent in a manual diagnosis loop. Teams rerun prompts by hand, compare outputs by eye, and rarely record what changed or why. For agent builders, chat benchmarks may miss tool-call regressions such as extra fields, renamed arguments, or changed call sequences. Those problems appear only when the team replays real agent traces. Plan for three constraints: Start with the optimizers, then verify their output. They apply general practices in a single pass rather than fitting changes to the team's dataset. They tune instruction text, not tool definitions or output schemas. Copy the original prompt first because there is no version history, then evaluate the optimized prompt against the frozen dataset. Expect more manual work when moving between providers or model families. There is no "optimize for target model X" flow. Moving from one family to another, such as OpenAI to Claude, still requires deliberate prompt and schema translation. Record traffic before you need it. Replay is only as useful as the captured data. Existing traces are available as an evaluation source for agents today, while content capture is opt-in and never retroactive. Log prompts, responses, latency, and tokens now to prepare for the next Adapt phase. 4. Validate Run the adapted workload against a quality rubric on the frozen dataset. The rubric may combine rules, LLM-as-judge evaluation, human review, existing user-feedback signals, or a domain-specific scoring framework. Examples include a clinical summarization rubric for healthcare or a tool-call sequencing assertion for agents. Validation produces a pass-or-fail decision for production exposure. AI-native teams may run the same signal continuously on every commit rather than treating it as a one-time gate. The dataset is a dependency for both Adapt and Validate. Build and freeze it early, around Assess, even though its primary purpose belongs to this phase. Validation then has two touchpoints: Before Adapt, freeze the dataset and success criteria, then run the current model to establish the source baseline. After Adapt, run the target model against the same dataset and evaluators, compare it with the source baseline, and make the release decision. Prepare the evaluation runner early and apply the gate after Adapt. Both steps belong to Validate. Foundry tools. The Azure AI Evaluation SDK, installed with pip install azure-ai-evaluation, includes more than 30 evaluators. They cover grounding, relevance, retrieval, coherence, fluency, question answering, reference-based similarity, F1, BLEU, ROUGE, safety, agent behavior, and Azure OpenAI graders. Teams can also build custom LLM-as-judge evaluators for task-specific rubrics. The portal evaluation flow runs the same evaluators against model, agent, dataset, and trace targets. Run identical evaluators against source and target outputs on the frozen dataset so the results remain comparable. Measure the three dimensions used for sign-off: Quality: evaluator results Latency: leaderboard time to first token and throughput, plus operational latency from the workload Cost: (input tokens × input price) + (output tokens × output price) Where it breaks. Most teams do not have an evaluation suite. Teams that do often built it themselves and may not use platform evaluation tools. Regulated workloads add mandatory human review, which can become the bottleneck. For those teams, migrations often stall in Validate rather than Adapt. What your team provides. The evaluators are ready to run, but model workloads still require teams to curate a domain-relevant test set from production traffic. That is why Phase 0 pays for itself. 5. Roll out Promote the validated configuration in stages: non-production, then a canary or weighted percentage of production traffic, followed by broader exposure. Compare live latency, errors, and quality signals with the pre-migration baseline, then commit or roll back. Some workloads cannot expose a new model to customer traffic during testing, including flows involving protected health information or financial transactions. Use shadow or mirror mode instead: run the new model offline against production inputs and compare its outputs with the old model without affecting users. Foundry tools. Migration mechanics depend on the deployment SKU: Standard, Global Standard, and Data Zone Standard deployments upgrade automatically on a rolling schedule. Control timing with versionUpgradeOption: OnceNewDefaultVersionAvailable, OnceCurrentVersionExpired, or NoAutoUpgrade. Priority Processing follows the same path. Provisioned, Global Provisioned, and Data Zone Provisioned deployments migrate manually, either in place during a 20-to-30-minute Azure-managed traffic transition or through side-by-side deployments. Batch deployments migrate side by side. Deploy the new model, resubmit jobs, then retire the old deployment. Fine-tuned deployments do not upgrade automatically. They follow separate training and deployment retirement schedules, so plan retraining or distillation early. See the model retirements documentation for deployment-specific guidance. Use continuous evaluation to score a sample of production traffic in the Foundry Observability dashboard. Connect evaluation results to traces for root-cause analysis and configure Azure Monitor alerts for quality regressions. Where it breaks. Offline evaluation can miss production quality and latency regressions. Rollback decisions may also be forced by deprecation deadlines rather than evidence. What your team provides. Teams implement weighted routing between deployments in their application or gateway layer. They must also choose how long to keep the old deployment warm for rollback. Embedded copilot teams often target about 30 days. Design both mechanisms once and reuse them for future migrations. 6. Retire Retire is easy to forget. Decommission the old deployment, free its capacity, archive evaluation artifacts, update internal documentation, and communicate the change to downstream owners. That may include customer-facing documentation, marketing pages, support runbooks, and audit logs. Regulated workloads may need to retain artifacts for years. Retirement is also a governance step. Foundry tools. Use the Models API to confirm that the old version is retired through lifecycleStatus or 410 Gone. Use the observability dashboard to confirm that the active deployment count falls. Add useful production traces to the golden dataset so the next migration starts with better evidence. Where it breaks. Teams skip the phase. Zombie deployments accumulate, leaving teams with structural debris from migrations they never finished. What your team provides. The observability dashboard shows deployment count, but the team must decide which deployments still carry traffic. Create an explicit retirement ticket rather than relying on someone to remember. Embedded copilot teams also need to update public claims such as "powered by gpt-4o" after the model changes. Worked example: Zava's Shopping Assistant migrates from gpt-4o mini to gpt-5.x Zava is a fictional retailer used as a stand-in for a real customer story. The example reflects patterns observed in customer-facing embedded AI workloads. Zava's Shopping Assistant is one of the company's largest LLM workloads. It has two LLM stages: Per-review insight extraction identifies sentiment, attribute mentions, and defect signals across thousands of product reviews. Product-level summaries present those findings to shoppers on the product page. Together, the two stages account for a meaningful share of Zava's token volume. Discover Zava's central AI Platform team made gpt-5.x models available internally and notified feature teams. The Shopping Assistant team learned about the models through that channel and received a target migration window before gpt-4o mini's deprecation. Zava has an internal discovery layer built on the retirement schedule and Models API. Microsoft provides the underlying data, while Zava routes the signal to application owners. Assess The team compared gpt-5.4 nano, which offered lower latency and cost, with gpt-5.1 and gpt-5.2 using leaderboard trade-off charts. Selection remained difficult because of the rapid release cadence, unclear positioning between variants, and the lack of a behavioral benchmark for product question-and-answer workloads. Capacity planning required coordination with the AI Platform team. Both gpt-4o mini and the gpt-5.x candidate needed to remain available in the same regions so the team could roll back. Adapt The team ran its existing Shopping Assistant prompts against gpt-5.4 nano using a sanitized traffic sample. Customer queries were scrubbed of personally identifiable information before replay. The behavioral comparison found three problems: Summaries used more hedged language and sometimes contradicted the underlying review evidence, creating a shopper-trust risk. Insight counts varied across runs. The model sometimes extracted substantially more or fewer attribute mentions than gpt-4o mini, affecting downstream filtering. Latency varied more than expected on the synchronous product-page path. Prompt Optimizer helped restructure the summary prompt, but the team still diagnosed the differences manually. It built its own replay system and behavioral comparison on top of captured traffic. Reengineering took weeks and extended beyond prompts: parameters and downstream parsing for the insight-extraction output also changed. Validate The team scored outputs with Zava's Product Answer Quality (PAQ) rubric. Its nine criteria cover factual grounding, attribute accuracy, tone, and refusal behavior for questions outside the catalog. Zava implemented the rubric as custom evaluators in the Azure AI Evaluation SDK. Initial evaluations used unchanged prompts to isolate model behavior. The team reran them after each prompt change. Zava's QA team also completed a manual review, which the company requires for every new model used in a customer-facing workflow. The per-review insight extraction stage still has no automated evaluation, a gap the team has accepted for now. Roll out The validated configuration moved to non-production and then through staged exposure: employees first, followed by a small percentage of shoppers. The canary exposed latency regressions that offline evaluation had missed. The team rolled back the latency-sensitive synchronous product-page path while keeping the offline pregeneration path on the new model. Retire Retirement is not complete. The gpt-4o mini deployment remains warm for rollback. The team must resolve the synchronous-path latency regression before retiring it, which sends that code path back to Adapt. This split state is common. Retire can lag Roll out by weeks or months, and the old deployment remains visible in deployment-sprawl data. Lessons from the example Adapt consumed most of the schedule. Replay tooling, behavioral comparison, and prompt reengineering are the clearest opportunities for Microsoft to shorten migrations. Validate worked because Zava had invested in it. Most customers do not have an equivalent to PAQ. Making domain-specific evaluations cheaper to build would improve confidence in this phase. The migration is partially live and partially rolled back. The process must support split-state workloads rather than assuming a binary switch from old model to new. How to use this process The six phases can serve as a checklist and an interview script for teams creating or auditing a migration process. Documentation and process: Lead with the six phases. Most teams recognize them immediately. Investment priorities: Start with Adapt and Validate. Across customer stories, those phases consume the most time and confidence. Interviews and postmortems: For each phase, ask whether it happens, who owns it, which tool the team uses, where it failed last time, and what evidence would increase confidence. Metrics: Discover and Retire are the easiest phases to instrument through measures such as announcement reach and active deployment count. Adapt and Validate require purpose-built telemetry that most teams do not yet have. Go deeper Two companion resources turn the process into concrete implementation steps: Microsoft Learn guide: This article follows the six phases through identifying affected deployments, preparing a test dataset, adapting prompts, evaluating source and target models, and rolling out by deployment type. Foundry Models Accelerator: This community toolkit includes a deployment-inventory scanner, a feasibility and assessment playbook, code and API migration audit scripts, an A/B evaluation runner with golden datasets, and rollout guidance. It follows the same six-phase process. The Foundry Models Accelerator is a community-built toolkit provided as-is under the MIT License. It falls outside Microsoft Support. Always verify model availability and retirement dates against official documentation. Additionally, check out the Foundry Forgebook which hosts a plethora of recipes that walk through the required code changes to migrate from different source to target models, even across model families. The goal of a model migration is to change the model without changing your application’s behavior, or to change it measurably for the better. Lead with the six phases, invest first in Adapt and Validate phases, and treat the Retire phase as a governance step.Building Production-Ready AI Agents in Microsoft Foundry: 10 Lessons Learned
A Real-World Scenario Imagine a customer support agent that answers invoice questions. During testing, everything works perfectly. But in production: The Finance API occasionally times out. The knowledge base contains outdated information. Tool calls fail during peak traffic. Token consumption rises unexpectedly. The result isn't a broken AI model. It's an unreliable system. This is where Microsoft Foundry becomes critical. Building production-ready agents requires grounding, observability, resiliency, governance, and continuous monitoring. New challenges appear: Challenge Impact Hallucinated responses Reduced trust Missing citations Verification issues Tool failures Broken workflows High token consumption Increased cost Latency spikes Poor user experience Limited monitoring difficult troubleshooting Security concerns Compliance risks The lesson was clear: Production readiness is not about making the agent smarter. It is about making the agent reliable. My Observation While experimenting with AI agents in Microsoft Foundry, I found that model selection was rarely the primary challenge. Most production issues stemmed from grounding quality, tool reliability, observability, and security controls. Addressing these operational concerns often had a greater impact on user trust than changing the underlying model. Production AI Agent Reference Architecture A production-ready AI agent typically includes several components beyond the language model itself. Layer Technology User Interface Web App, Teams, Copilot Agent Runtime Microsoft Foundry Agent Service Knowledge Layer Azure AI Search Foundation Model GPT-4o Tool Integration APIs and Functions Monitoring Azure Monitor Security Managed Identity and Key Vault Request Flow Lesson 1: Start with the Use Case, Not the Model One of the most common mistakes is beginning with model selection. Many teams ask: Which model should I use? Should I choose GPT-4o? Should I use a larger context window? A more important question is: What business problem are we solving? Before evaluating models, define: Users Business goals Success metrics Compliance requirements Operational constraints An enterprise support agent and a financial compliance agent may use the same model but require completely different architectures. Key Takeaway Successful AI projects start with business outcomes, not model benchmarks. Lesson 2: Grounding Matters More Than Prompting Prompt engineering helps. Grounding drives trust. Without access to reliable enterprise data, even advanced models can generate confident but incorrect responses. Grounding Sources Azure AI Search SharePoint documents Internal knowledge bases Structured enterprise data Approved policy repositories Example from azure.ai.projects import AIProjectClient # Configure grounding with Azure AI Search agent = client.agents.create_agent( model="gpt-4o", name="support-agent", tools=[ { "type": "azure_ai_search", "azure_ai_search": { "index_name": "support-kb", "endpoint": "https://your-search.search.windows.net" } } ] ) In production environments, grounding through Azure AI Search helps agents retrieve trusted enterprise information rather than relying solely on model knowledge. This significantly improves response accuracy and trustworthiness. Try it yourself: Configure Azure AI Search as a knowledge source in Microsoft Foundry and compare grounded versus non-grounded responses. Key Takeaway Reliable retrieval is usually more valuable than sophisticated prompting. Lesson 3: Design Tool Usage Carefully AI agents become powerful when they can interact with external tools. Examples include: CRM systems Databases APIs Ticketing systems Business applications However, every tool increase complexity. Ask yourself: When should the agent call the tool? What happens if the tool is unavailable? How should failures be handled? Example Failure Scenario +-----------------------------+ | User asks for invoice status | +-------------+---------------+ | v +-----------------------------+ | Agent calls Finance API | +-------------+---------------+ | v +-----------------------------+ | API timeout detected | +-------------+---------------+ | v +-----------------------------+ | Fallback response returned | +-----------------------------+ Example Failure Scenario: A production-ready agent should gracefully handle external service failures and return a fallback response instead of failing completely. Key Takeaway Design for failure before designing for capability. Lesson 4: Evaluate Before Deployment Many teams test only for response quality. Production agents require broader evaluation. Evaluation Area Why It Matters Accuracy Correct answers Grounding Quality Faithful responses Latency User experience Safety Risk reduction Cost Sustainability Tool Success Rate Reliability Evaluation should become part of every deployment pipeline. Key Takeaway You cannot improve what you do not measure. Lesson 5: Make Observability a First-Class Feature Observability is often neglected until something breaks. Unfortunately, production systems always encounter unexpected behavior. Track metrics such as: Metric Purpose Request Volume Demand tracking Average Latency Performance Token Usage Cost visibility Grounding Success Rate Quality Tool Failure Rate Reliability User Satisfaction Business value Setting Up Tracing in Foundry from azure.ai.projects import AIProjectClient from azure.monitor.opentelemetry import configure_azure_monitor # Configure Azure Monitor for tracing configure_azure_monitor( connection_string="InstrumentationKey=xxx" ) # Run the agent response = agent.run( thread_id=thread.id, instructions="..." ) Tracing provides visibility into how an AI agent processes requests, invokes tools, and generates responses. By integrating Azure Monitor, teams can track latency, identify failed tool calls, analyze token usage, and troubleshoot unexpected agent behavior. This observability is essential for operating AI agents reliably in production environments. Try it yourself: Enabling tracing and monitoring for a Microsoft Foundry agent using Azure Monitor. Example Dashboard +-----------------------------+ | Production Monitoring Dashboard | +-----------------------------+ | Requests Today | 5,120 | | Average Latency | 3.2s | | Grounding Success | 97% | | Tool Failure Rate | 1% | | Average Tokens | 2,800 | +-----------------------------+ Monitoring production metrics helps teams understand agent performance, reliability, and cost efficiency. Key indicators such as latency, grounding success rate, tool failure rate, and token consumption provide valuable insights into the operational health of an AI agent and enable proactive troubleshooting before users are impacted. Key Takeaway What you can't observe, you can't effectively operate or improve. Tracing is a critical capability for maintaining production-ready AI agents. Lesson 6: Monitor Token Consumption Token usage directly impacts cost. A highly successful agent can quickly become expensive if token growth is unmanaged. Common optimization techniques: Optimization Benefit Prompt Compression Lower token usage Response Caching Reduced model calls RAG Filtering Focused context Context Trimming Smaller requests Model Selection Cost control Key Takeaway Cost optimization should be planned from day one. Lesson 7: Build Security into the Design Security should never be an afterthought. Enterprise AI systems must enforce the same security boundaries as traditional applications. Recommended Controls Control Purpose Managed Identity Secure authentication Azure Key Vault Secret management RBAC Authorization Audit Logs Compliance Content Filtering Safety Private Endpoints Network Security Guiding Principle An AI agent should never access data beyond a user's permissions. Key Takeaway Security is a design requirement, not a deployment task. Lesson 8: Expect Tool Failures External dependencies inevitably fail. Production-ready agents should anticipate: API downtime Authentication failures Network interruptions Rate limiting Unexpected responses Recommended Strategy Key Takeaway async def call_tool_with_resilience(tool_name, params): try: result = await tool_client.execute( tool_name, params, timeout=5.0 ) return result except TimeoutError: return await cache.get_fallback( tool_name, params ) Example Outcome: Introducing retry and fallback logic can significantly improve reliability. By handling temporary API failures gracefully and returning cached responses, when necessary, agents can reduce user-facing errors and provide a more consistent experience. Lesson 9: Evaluate the Process, Not Just the Output A correct answer does not always mean the process was correct. In production environments, teams should evaluate not only the final response but also the steps the agent took to generate that response. An answer may appear accurate even when it was based on incorrect retrieval results, unnecessary tool calls, or incomplete citations. Review: Retrieval quality Tool execution Citation accuracy Security compliance Reasoning path User Query | v Retrieval | v Tool Execution | v Reasoning | v Response For example, an agent might generate the correct answer by chance, even though it retrieved irrelevant documents or used an inefficient workflow. Without evaluating the process, these hidden issues may go unnoticed until they affect reliability, compliance, or user trust. Key Takeaway Inspect the entire workflow, not just the final answer. Lesson 10: Think Like a Production Engineer As adoption grows, operational excellence becomes the differentiator. Ask questions such as: Can we troubleshoot failures? Can we measure business impact? Can we control costs? Can we scale safely? Can we govern usage? These questions become more important than model selection over time. Key Takeaway Production success comes from engineering discipline, not model size. Microsoft Foundry Features That Helped Improve Reliability Foundry Capability Production Benefit Agent Service Agent orchestration Knowledge Sources Grounded responses Evaluations Quality measurement Tracing Workflow visibility Model Catalog Model flexibility Safety Systems Risk mitigation These capabilities help teams move beyond proofs of concept and build solutions that are ready for real-world adoption. Production Readiness Checklist Before releasing an AI agent, verify: ✅ Business goals defined ✅ Grounding strategy implemented ✅ Security controls enabled ✅ Evaluation pipeline established ✅ Monitoring configured ✅ Failure handling tested ✅ Cost optimization reviewed ✅ Governance process defined ✅ User feedback loop available ✅ Deployment rollback strategy prepared Get Started with Production-Ready Agents If you're building AI agents using Microsoft Foundry, start by focusing on grounding, observability, security, and evaluation from day one. Suggested next steps: Build your first agent in Microsoft Foundry Configure Azure AI Search grounding Enable tracing and monitoring Evaluate agent quality before deployment Add resilience and fallback strategies Which of these 10 lessons has been most valuable in your own AI agent journey? Share your experiences and insights in the comments.Model router updates: new regions, a refreshed model pool, and understanding the hill climb
Across Microsoft, "hill climbing" has become shorthand for how real AI progress happens: not in one dramatic leap, but through a disciplined loop. Microsoft AI defines the hill climb as an organization that continuously improves, cycle after cycle, through more compute, better data, and sharper evaluation. Reinforcement fine-tuning in Foundry defines it as improving the deployable model package one measured step at a time across quality, latency, and cost. Different altitudes, same premise: progress is not a one-shot decision. It's a loop. For most teams, the decision of what model to use when is made manually or with custom routing tools. A developer picks a model based on benchmarks, familiarity, or the last launch that made headlines, ships it, and revisits the choice only when something breaks. In an ecosystem where the frontier moves monthly, that decision goes stale fast. Model router in Foundry Models brings the hill climb to the selection layer. What's new: a bigger pool, in more places This release expands where teams can deploy model router, broaden the supported model pool, and delivers updates through a stable endpoint. Together, these changes help teams run production workloads in more locations, match a wider range of tasks to suitable models, and adopt supported updates without changing the application integration. A refreshed model pool. The supported model list now includes Anthropic Claude Opus 4.8 — a high-capability model built for complex reasoning and long-form generation, for scenarios that demand depth, structure, and quality — and the GPT-5.6 family. Just as importantly, the pool is pruned: gpt-5-chat, gpt-5.2-chat, gpt-5.3-chat, Deepseek-V3.1 have been removed from the model router as models reach the end of their lifecycle and are deprecated in Foundry. New region availability. The model router is now available in 28 regions for global standard and 21 data zone regions. For many organizations, inference requests must stay within specific geographic boundaries for regulatory, governance, or customer-trust reasons — and intelligent routing shouldn't force a compromise on that. Find the full list of regions here. The most important detail is what you don't have to do: these updates occur automatically*. The endpoint remains stable as the supported model pool is refreshed, so teams do not need to redeploy the model router to receive the update. Applications can continue using the same integration while the model router evaluates requests against the current supported pool. Teams should continue monitoring routing traces and application outcomes to confirm that quality, cost, latency, and governance requirements are met. *Models from Anthropic still need to be deployed separately before they can be routed to through the model router. Interested in hearing more about what's new to the model router? Tune in for the next episode of Model Mondays with Sanjeev Jagtap and Lee Stott, where they talk all things model router from evaluations to hill climbing. Sign up here to watch live or view the replay: Model Mondays - Spotlight On Model router in Microsoft Foundry | Microsoft Reactor The selection-layer hill climb At the selection layer, a step is a routing decision. Each one is a micro-optimization against your objective, and each one is instrumented: every response from the model router includes a model field showing which underlying model was selected, so the climb leaves a complete, auditable trail. Model router supports three parts of the optimization loop: A/B testing to compare two router configurations to understand quality, cost, and latency tradeoffs; model decomposition to use routing results to decompose a single-model application into a multi-model or multi-agent design, and continuous routing to keep the router in production for continuous per-request selection. Each pattern turns model choice into a measured, repeatable process rather than a fixed decision. 1. A/B Testing Question: Which model or routing strategy should I use in production? A/B testing helps teams compare candidate models, model families, or router configurations against the same workload. Representative traffic is sent to competing deployments, and teams compare quality, cost, latency, and governance outcomes. The goal is to understand tradeoffs and identify the model or routing strategy that best meets workload requirements before promoting it to production. 2. Model Decomposition Question: What work is my application actually doing? Model decomposition uses model router as a diagnostic tool. By deploying the model router against a representative workload and examining routing telemetry, teams can see how requests naturally separate into different task classes. Simple retrieval, classification, and summarization requests may route to smaller models, while reasoning, planning, and agentic workflows may require more capable models. The goal is not to choose a winner, but to understand the structure of the workload and uncover opportunities for optimization, specialization, or architectural improvements. 3. Route continuously Question: Why choose a single model at all? Route continuously is the pattern model router was designed for but is not limited to. Rather than treating model selection as a one-time decision, teams leave the model router in production and allow the best-fit model to be selected for each request. As the supported model pool, regional availability, and platform capabilities evolve, teams can continue using the same endpoint while evaluating whether updates improve workload outcomes. Model selection becomes an ongoing optimization process rather than a project that must be repeated every time the model landscape changes. Together, these patterns illustrate a broader shift: the model router is more than a model. It is a tool for the optimization loop itself, helping teams evaluate tradeoffs, understand workload behavior, test hypotheses, and continuously refine model selection as requirements evolve. Whether used to compare candidate models, decompose applications into specialized tasks, or automate per-request routing in production, model router turns model selection into an observable, measurable, and repeatable process. As the model landscape continues to change, that optimization loop becomes a durable advantage. Getting Started Ready to start your own hill climb? Whether you're exploring the model router for the first time, evaluating routing strategies against your workload, or building a long-term optimization practice, these resources can help you move from experimentation to production with Microsoft Foundry. What's new in model router? Sign up for the next Model Mondays episode for a deep dive into new features, optimization patterns, and the latest model router updates. How do I build agents with model router? Check out the Model Router Agents Lab and build agent experiences with routing, retrieval, web search, tool calling, and multi-agent patterns. How do I evaluate model router? Compare model router against baseline models using your own prompts, then review quality, cost, latency, and routing decisions with the Auto Evaluation Toolkit. How do I optimize model router for my workload? Start your hill-climbing journey with the Model Mastery workshop, where you'll test one optimization lever at a time and measure how each change impacts workload outcomes. How do I build a model router optimization playbook? Explore the Model Releases repository to track new capabilities, understand the optimization question behind each release, and try focused notebooks that demonstrate one optimization lever at a time.Introducing GPT-transcribe and GPT-live-transcribe in Microsoft Foundry
A transcription model hears “account number 8-4-7-2” but returns “account number eighty-four seventy-two.” A single error can break a downstream automation workflow. Developers building voice applications need transcription models that can handle real-world audio conditions, natural speech patterns, and business-critical details, including codes, dates, addresses, account numbers, mixed-language conversations, specialized terminology, and quiet or low-volume speech. GPT-transcribe and GPT-live-transcribe do just that and are available in Microsoft Foundry today. Two updates to the audio model family designed to improve automatic speech recognition across asynchronous transcription and live streaming scenarios. Built for More Accurate Transcription in Real-World Audio GPT-transcribe is the highest accuracy ASR model from Open AI, designed for asynchronous speech-to-text transcription of completed audio files and batch workloads. It accepts audio input and returns text output, making it a strong fit for workflows that process recorded, uploaded, or submitted audio, including meeting recordings, voicemails, and media files. GPT-live-transcribe is designed for low-latency streaming transcription through the Realtime API. It supports real-time audio input and text output, helping developers build live experiences where speech needs to be transcribed continuously as audio arrives. This model also introduces “tunable latency” where developers can adjust the latency/accuracy trade-off for streaming. It is a strong fit for live captions, voice assistants, contact center workflows, accessibility experiences, field service applications, real-time intake, and monitoring systems. Together, these models give developers transcription options in Microsoft Foundry for stored audio and live voice interactions. Their text output can support downstream workflows such as search, summarization, routing, analytics, automation, and quality review. What’s New in Both Models The features of the new transcription models focus on improving transcription quality in real-world audio environments where speech can be brief, noisy, accented, quiet, domain-specific, or mixed across languages. Key capabilities include: Background noise: Helps isolate speech in noisy environments so transcription quality can remain more reliable when audio conditions are not controlled. Short utterances: Improves recognition of brief commands, confirmations, interruptions, and clipped speech that can be difficult to capture accurately. Alphanumeric perception: Strengthens transcription of IDs, codes, phone numbers, dates, addresses, account numbers, and mixed letter-number sequences. Domain terminology understanding: Improves recognition of specialized vocabulary used in product, workflow, industry, and business-process contexts. Codemix: Improves understanding when speakers switch between languages within a conversation or utterance. Context awareness: Uses topic hints and past conversation context to improve transcription accuracy and help maintain consistency. Accent robustness: Improves handling of regional accents, non-native accents, dialects, and varied speaking styles. Whispering: Improves recognition of quiet or low-volume speech, including whispered commands and private dictation. Live captioning and accessibility experiences: Generate real-time captions for meetings, events, media experiences, and assistive applications. Contact center and voice workflows: Capture spoken details as conversations happen, supporting routing, quality review, summarization, and downstream automation. Monitoring, analytics, and compliance workflows: Provide text visibility into ongoing spoken input so teams can analyze, review, and act on conversation data. Also Available: GPT-realtime-2.1 and GPT-realtime-mini-2.1 gpt-realtime-2.1 and gpt-realtime-mini-2.1 are also available in Microsoft Foundry for developers building speech-to-speech applications. Unlike GPT-transcribe and GPT-live-transcribe, which return text, these models accept audio and generate audio for low-latency conversational experiences over the Realtime API. gpt-realtime-2.1 focuses on interaction quality and robustness, while gpt-realtime-mini-2.1 provides a smaller, faster, and more cost-efficient option for high-volume deployments. Together with GPT-transcribe and GPT-live-transcribe, these realtime audio updates give developers more flexibility to build voice applications that need both accurate transcription and responsive spoken interaction, whether the experience is centered on capturing speech as text, responding with audio, or combining both patterns in a single workflow. Use Cases by Model GPT-transcribe Use GPT-transcribe when the application needs accurate text transcripts from recorded, uploaded, or submitted audio. It is a strong fit for meeting and call transcription, media transcription, customer support intake, voicemail and message processing, quality review, compliance workflows, and domain-specific transcription where short utterances, structured alphanumeric details, specialized terminology, accents, background noise, code-mixed speech, or quiet audio can affect downstream accuracy. GPT-live-transcribe Use GPT-live-transcribe when the application needs live streaming transcription with low latency. It is designed for real-time captions, accessibility experiences, contact center transcription, voice-enabled workflows, live monitoring, operational dashboards, and agent-assist scenarios where spoken input needs to become text continuously as the interaction unfolds. Pricing The following pricing example shows Global Standard rates by model and modality. Rates for GPT-realtime-2.1 and GPT-realtime-mini-2.1 are listed per 1 million tokens. GPT-transcribe and GPT-live-transcribe are listed per audio hour. Model Deployment Modality Input Cached Input Output GPT-realtime-2.1 Global Standard Audio $32.00 $0.40 $64.00 Text $4.00 $0.40 $24.00 Image $5.00 $0.50 -- GPT-realtime-mini-2.1 Global Standard Audio $10.00 $0.30 $20.00 Text $0.60 $0.06 $2.40 Image $0.80 $0.08 -- GPT-live-transcribe Global Standard Audio -- -- $1.02/hour GPT-transcribe Global Standard Audio -- -- $0.27/hour Getting Started Choose GPT-transcribe when your application processes complete audio files asynchronously, or GPT-live-transcribe when it needs text continuously as speech arrives. Try the models in Microsoft Foundry, then use the resources below to explore the Realtime API, follow the audio quickstart, compare available models, and review Azure OpenAI in Foundry Models documentation. For asynchronous transcription, submit a complete audio file to GPT-transcribe and process the returned transcript after the request completes. This pattern works well for recordings, voicemails, and uploaded media. For streaming transcription, open a Realtime API session with GPT-live-transcribe, send audio as it is captured, and handle incremental transcript events. This pattern supports live captioning and agent-assist experiences that need text during an active interaction. Refer to the linked quickstart and Realtime API documentation for current SDK setup, authentication, request schemas, and supported audio formats. Explore Microsoft Learn documentation to learn more: Use GPT Realtime API for speech and audio with Azure OpenAI in Foundry Models GPT Realtime audio quickstart Azure OpenAI in Foundry Models overviewFor the first time, real-time transcription goes multilingual
When we introduced Post-Stream Refinement earlier this year, it closed the oldest gap in real-time speech: you could finally get instant streaming results and a highly accurate final transcript, with no latency penalty. But it kept one hard requirement — you had to tell the service, up front, which single language to expect. Real-world speech does not work that way. People code-switch mid-sentence, product and brand names cross languages, and a global app serves users who simply speak differently from one session to the next. Today we remove that requirement. Multilingual Post-Stream Refinement enters public preview for Azure AI Speech in Microsoft Foundry, and for the first time ever a single real-time stream can transcribe multiple languages in one session — the spoken language is detected automatically, no locale is declared in advance, and the final transcript is refined for accuracy. Everything you already know about Post-Stream Refinement still applies; what changes is that the refinement pass itself is now multilingual. 📖 Read the Documentation What's New in This Release If you have already used Post-Stream Refinement, here is exactly what changes with the multilingual preview — and what stays the same: Quality Impact In internal testing and partner evaluations across Tier-1 locales, multilingual Post-Stream Refinement reduced word error rate (WER) by approximately 10% relative on average, with double-digit relative reductions on the hardest cases — long utterances, proper nouns, and multilingual or code-switched speech. Partial-result latency is unchanged; only the final transcript is refined. Gains are relative reductions versus the standard real-time model and vary by language, acoustic conditions, and content type. The refined final result may add a small amount of latency to the final segment; partial results are unaffected. Supported Languages and Regions The public preview supports 15 Tier-1 locales. Because language is detected automatically, a single stream can contain any mix of them: Available in these Azure regions: Real-World Impact Preview customers across industries — including travel, consumer electronics, automotive, aviation, and media — have reported positive gains in transcription quality. Customers testing multilingual and domain-specific audio have observed the clearest improvements on the hardest content: proper nouns, code-switching, and long-form speech. Several are actively validating the feature on their own audio ahead of general availability. Get Started Enabling multilingual Post-Stream Refinement is a small configuration change on your existing SpeechConfig. You will need: Speech SDK 1.50 or later. Earlier versions do not support the multilingual path. A Speech resource in one of the supported regions listed above. Auto-detect language configuration (open range) so the service identifies the language from the audio — no candidate list required. Set the post-processing option to PostRefinement and pass an open-range AutoDetectSourceLanguageConfig when you create the recognizer. Here is a complete, copy-paste Python example, including the optional end-of-utterance detection line: import azure.cognitiveservices.speech as speechsdk speech_config = speechsdk.SpeechConfig( subscription="YourSpeechKey", region="YourSpeechRegion") # 1) Refine the final transcript (Post-Stream Refinement) speech_config.set_property( speechsdk.PropertyId.SpeechServiceResponse_PostProcessingOption, "PostRefinement") # 2) Multilingual auto-detect - no candidate language list needed auto_detect_config = speechsdk.languageconfig.AutoDetectSourceLanguageConfig() audio_config = speechsdk.AudioConfig(use_default_microphone=True) recognizer = speechsdk.SpeechRecognizer( speech_config=speech_config, auto_detect_source_language_config=auto_detect_config, audio_config=audio_config) 💡 Tip: Refinement matters most for applications that store or process the final transcript — meeting notes, call analytics, compliance archives, AI summarization. If you only use partial results for a live display and discard them, your real-time UX (already fast) is unchanged, while any final transcript you keep improves. Try Multilingual Post-Stream Refinement Today Turn on higher-accuracy, language-aware transcription in your Azure AI Speech applications with a single configuration change. Available now in public preview in Microsoft Foundry. 📖 Read the Documentation We would love your feedback. Try Post-Stream Refinement in your applications and tell us how it improves your transcription quality.
Grounding Copilot Studio Agents with Azure AI Search and Foundry IQ
An employee opens the HR agent and asks, "How much PTO do I accrue each month?" A few minutes later, someone else asks, "Where is the official code of ethics policy?" Those sound like the same problem. They are not. The first person needs a grounded answer they can understand. The second person needs a link to the right document quickly, without interpretation. If you design for one experience, the other one feels broken. That is usually where knowledge-agent projects start to get messy. “Grounding” can sound like one switch you turn on, but in practice it is a spectrum: from zero-code classic search, to agentic retrieval over a knowledge base, to a forced-grounding agent that synthesizes answers when synthesis is required. The easier way to think about it is this: who is doing the retrieval work, and what does the user need back? This post walks through five working retrieval patterns for an “Ask HR” agent built on Copilot Studio, Azure AI Search, and Foundry IQ. Each one is running code in the companion sample repo: foundry-copilot-hr-policy-knowledge. Each has a clear “use this when,” and the five patterns share the same reusable knowledge base so you can layer them on without re-indexing. By the end, you should have a decision tree you can reuse for your own knowledge source, whether that is HR policy, product docs, or support runbooks. Scope: companion sample for learning and experimentation, not production-ready deployment. Review the Azure Well-Architected Framework for reliability, security, cost, and operational hardening before you ship. The scenario: one index, many front doors Here is the setup. The sample answers employee questions from a small corpus of internal HR policy documents: PTO accrual, hiring rules, code of ethics, blood-borne pathogen procedures, and dozens more. Underneath every pattern is one foundation: an Azure AI Search index named hr-policy-index, populated by an indexer and skillset that chunk and vectorize the documents. Patterns A, C, and the Hosted Agent query that index directly. Patterns A2 and B add a Foundry IQ knowledge base named hr-knowledge-base on top of the same index for agentic retrieval. That layering is the part to pay attention to. The retrieval assets stay separate from the orchestration layer, so you can start with the simplest pattern, prove value quickly, and move to a more capable one later without re-indexing. Two questions that decide everything Before we get into the patterns, it helps to define the two retrieval terms I use throughout the rest of the post: Classic search, index-first retrieval: one hybrid (keyword + vector) query against an Azure AI Search index, ranked and returned. Fast and predictable. Agentic retrieval, the knowledge base plans multiple sub-queries from the user's question, runs them in parallel, re-ranks, and merges the results before the agent composes an answer. Higher quality on complex, multi-part questions. If you want the fuller picture of how these two approaches map to retrieval-augmented generation, the Azure AI Search team's RAG and generative AI overview walks through the trade-offs and uses a similar HR/PTO example. Once those terms are clear, the decision tree comes down to three practical questions: Q1: Do users need an answer or are they really trying to find the right document? If they just need the document, stay on the locator path. If they need the policy explained or summarized, move into the answer-synthesis path. QL: Is the content in a citation-friendly knowledge base? For example, SharePoint content or Azure AI Search content with a reliable blob_url. If yes, Copilot Studio can usually handle this with native citation cards in Pattern A. If not, use Pattern C with the dual tool /api/lookup path so the agent can return the exact document link. Q2: Do you actually need an LLM agent in the middle? If the answer is no, keep it simple: use classic search or agentic retrieval over the knowledge base. If the answer is yes, move into the agent path. QK: For that non-agent path, is classic index search enough, or do you need agentic KB retrieval? Classic search points to Pattern A. Agentic retrieval over the knowledge base points to Pattern A2. Q3: If you need an agent, do you want Foundry to run the request loop, or do you need to self-host it? If Foundry can manage the runtime, use Pattern B. If you need the request loop in your own container, use the Hosted Agent. That is the decision tree in plain terms: Q1 decides whether this is a document-locator experience or an answer-synthesis experience. Q2 decides whether you need an LLM agent at all. Q3 is only about where the agent runs, either Foundry or your container. It does not change the front door; Copilot Studio can still be the user-facing experience. How the sample repo is organized The repo follows the same flow as the post. Start with docs/DataPipelineAndTesting.md to understand how the HR policy corpus is indexed, tested, and validated. Use docs/RetrievalPatterns.md as the decision model for choosing between classic search, agentic retrieval, forced grounding, and hosted runtime options. Then use the pattern-specific docs when you are ready to wire each path. For Copilot Studio patterns, docs/CopilotStudioIntegration.md maps to Pattern A, while docs/CopilotStudioHybridExample.md maps to Pattern C and the dual-tool locator flow. For the more advanced agent paths, docs/FoundryAgentArchitecture.md covers Pattern B and the hosted agent architecture. docs/Distribution-M365-Teams.md shows how the agent can be distributed through Microsoft 365 and Teams once the retrieval pattern is working. The rest of the post is that tree, one branch at a time. Pattern A: Direct index (classic search, zero agent code) Start here. Copilot Studio queries hr-policy-index directly through its built-in Knowledge action. No custom agent code runs in the answer path. The sample only owns the index, skillset, and indexing pipeline. Populate the index (server-side indexer + skillset handles chunking and vectorization): uv run python scripts/index_knowledge_base_integrated_vectorization.py # Builds hr-policy-index; a client-side alternative exists for dev/test What you get: very low latency in the sample, roughly 1-2 seconds, no LLM cost in the retrieval path, and native citation cards. When the source documents carry a blob_url or metadata_storage_path, Copilot Studio can render a click-through card straight to the document. For many "where is the policy?" questions, that may be enough. The honest limitation: Pattern A is still classic search. It does not force synthesis. If Copilot Studio generates an answer from retrieved snippets, it may paraphrase a policy in a way that is close, but not precise enough. For HR policy, that matters. If exact wording matters, that is your sign to step up to Pattern B. Pattern A2: Copilot Studio meets Foundry IQ (agentic retrieval, no prompt agent) This is the pattern I would look at when you want better retrieval quality without taking on the overhead of a full prompt agent. In the Copilot Studio new agent experience preview, an agent connects directly to a Foundry IQ knowledge base through Microsoft IQ, with no Foundry prompt agent in between. You reuse the same hr-knowledge-base on top of the same hr-policy-index (one command: python -m src.agents.create_foundry_agent), but retrieval is now agentic: the knowledge base plans sub-queries, retrieves in parallel, reranks, and hands merged results to the agent. Wiring it takes a few clicks in Copilot Studio (step-by-step on Microsoft Learn): Build → Microsoft IQ → Foundry IQ → Create new connection Choose Microsoft Entra ID Integrated authentication Select hr-knowledge-base Add to agent A2 is worth the upgrade from A for two reasons. First, you get agentic-retrieval quality without having to build, deploy, or maintain a prompt agent. The knowledge base becomes the reusable asset you improve in Microsoft Foundry, not something you keep reworking inside each Copilot Studio agent. Second, when configured with Microsoft Entra ID Integrated authentication, retrieval can return ACL-trimmed results per user. Each person sees content based on their access. Foundry IQ knowledge bases can also inherit enterprise-readiness controls such as customer-managed keys, network isolation, and Entra ID. A single knowledge base can also federate across multiple knowledge sources in parallel. Use A2 when you want stronger hybrid retrieval quality without taking on the overhead of operating a full agent. Pattern B: Foundry Agent Service with forced grounding When answers need to be synthesized and grounded, publish a prompt agent to Microsoft Foundry with Foundry Agent Service. In the sample, the agent uses an MCPTool pointing at the knowledge-base endpoint, with tool_choice="required" so the model retrieves policy chunks before answering. # src/agents/hr_policy_agent.py (excerpt) agent = PromptAgentDefinition( model=model_deployment_name, # e.g. gpt-5-mini instructions=HR_POLICY_INSTRUCTIONS, tools=[mcp_tool], # KB MCP endpoint tool_choice="required", # require retrieval before answering ) Invoke it through the OpenAI client the project hands you: client = project.get_openai_client() response = client.responses.create( input="How does PTO accrue for a new hire?", extra_body={"agent_reference": {"name": agent_name}}, ) What you get: synthesized answers with grounding and inline [Policy XXXX - Title] citations, all from a single SDK call on a managed runtime. The trade-off: synthesis takes longer. In the sample, answers take roughly 10-14 seconds versus 1-2 seconds for classic search. For policy explanations, that extra time can be worth it because the user gets a composed, grounded answer instead of a list of snippets. Pattern C: Dual-tool routing for deterministic document locators Some questions do not need an essay; they just need the right URL, fast. Pattern C lets Copilot Studio route per turn: "Where is the PTO policy?" → POST /api/lookup, a deterministic endpoint with no LLM, roughly 1-2 seconds, returning the document URL verbatim in the answer body. "How many PTO hours do I accrue?" → hand off to Pattern A or B for a synthesized answer. POST /api/lookup { "query": "PTO policy" } → 200 OK { "policy_id": "12345", "title": "Types of Leave: Paid Time Off (PTO)", "blob_url": "https://.../12345-pto.pdf" } Reach for Pattern C when native citations are not enough. For example, use it when you need fast locator responses, the URL printed directly in the answer body, deterministic and auditable output, or support for a source that is not citation-friendly. The endpoint lives at src/backend/main.py:/api/lookup, with its contract in copilot/openapi-lookup-v2.json. Hosted Agent: the same agent on your own runtime If you need to own the request loop, custom authentication, side-car services, or infrastructure that stays inside your boundary, run the agent yourself. The Hosted Agent is the self-hosted version of the same idea: a container built on Microsoft Agent Framework with FoundryChatClient. It supports both classic and agentic retrieval through one environment variable: RETRIEVAL_MODE Strategy Retrieval type tool (default) Custom @tool search_hr_policies (hybrid + semantic) Classic search context-semantic Built-in AzureAISearchContextProvider before each turn Classic search context-agentic AzureAISearchContextProvider over hr-knowledge-base Agentic retrieval The context-* modes use Agent Framework’s out-of-the-box RAG context provider. Retrieval runs automatically before each model call with standardized context and citation prompts, so the agent does not have to call a search tool explicitly. That gives the self-hosted path parity with the managed Foundry path across both retrieval types. Copilot Studio can still be the front door. Q3 in the decision tree is really about where the request loop runs, not who greets the user. Choosing a pattern Pattern Orchestrator Retrieval Latency (sample) Best for A Copilot Studio Classic ~1-2 s Start here, native citations, no agent code A2 Copilot Studio → Foundry IQ Agentic ~2-4 s Agentic quality, no agent to maintain B Foundry Agent Service Classic/agentic via MCP ~10-14 s Forced-grounding synthesis in Foundry C Copilot Studio (router) None for lookup ~1-2 s Deterministic, verbatim document locators Hosted Agent Framework container Classic + agentic ~10-14 s Self-hosted runtime, custom auth A simple way to read the table: start at A, move to A2 when you want agentic retrieval without operating an agent, choose B when each answer needs to be synthesized and grounded in Foundry, add C for high-volume locator traffic, and pick the Hosted Agent when you need the runtime on your own infrastructure. These are not mutually exclusive. A mature agent often routes locator queries to C and content questions to A2 or B. What's next? Try it: clone the sample and follow Steps 1-3 of the walkthrough to stand up Pattern A, provision hr-knowledge-base, connect Copilot Studio, and ask a question in minutes. Go agentic: wire the same knowledge base into the Copilot Studio new agent experience via Foundry IQ (Pattern A2) and compare answer quality side by side. Learn more: explore agentic retrieval in Azure AI Search, Foundry IQ, and Microsoft Agent Framework. Adapt it: swap the HR policy corpus for your own product docs, support runbooks, or internal knowledge source, then compare Pattern A, A2, and B against the same user questions. Use the repo-doc map: start with docs/RetrievalPatterns.md for the decision model, docs/CopilotStudioIntegration.md for Pattern A, docs/CopilotStudioHybridExample.md for Pattern C, docs/FoundryAgentArchitecture.md for Pattern B and Hosted Agent, and docs/DataPipelineAndTesting.md for ingestion and validation. My recommendation: start simple, prove the index works, and move up the stack only when the use case needs it. Some questions need a trusted link. Others need a grounded explanation. A strong architecture supports both without forcing every request through the same path. References Copilot Studio + Foundry IQ Connect to Foundry IQ from an agent Foundry IQ FAQ Foundry IQ / knowledge layer What is Foundry IQ? Connect a Foundry IQ knowledge base to Foundry Agent Service Azure AI Search, retrieval Agentic retrieval overview RAG and generative AI in Azure AI Search Classic vs agentic search Create a knowledge base Create a knowledge source Hybrid search Semantic ranking Quickstart: agentic retrieval Tutorial: end-to-end agentic retrieval solution Microsoft Foundry Agent Service (Pattern B) Foundry Agent Service overview Microsoft Agent Framework (Hosted Agent) Microsoft Agent Framework overview Hosted MCP tools Governance Azure Well-Architected Framework Related Microsoft Foundry blog posts Foundry IQ is now in Copilot Studio Answers You Can Trust: Grounding Enterprise Agents with Foundry IQ Foundry IQ: Unlocking ubiquitous knowledge for agents
o3-mini not returning reasoning tokens
Hi, I work on a service that leverages o3-mini via Microsoft Foundry. In the past few days, I've observed that when calling o3-mini via Microsoft Foundry, that completion_token_details always has the reasoning_tokens value set to 0, regardless of the reasoning setting being used. In my testing, it seems that the reasoning is still occurring, as increasing reasoning value causes the completion_tokens field to increase by a good amount, but none of the reasoning levels cause the reasoning_tokens value to be anything other than 0. Has anyone else encountered this issue? Thanks! TomMigrating to GPT-5.x Without Breaking GPT-4: A Practical, Backward-Compatible Playbook
The first request your service sends after swapping gpt-4o for gpt-5.1 in production will return HTTP 400. Not in two weeks. On the first call. And the parameter the error points to isn't one you set anywhere in your code - it's bound onto the request by a LangChain helper you've used for two years. This post walks through every breaking change between the GPT-4 and GPT-5 families on Azure OpenAI in Microsoft Foundry, the integration cliffs nobody warns you about, and the small set of files you need so the same call sites work against both model families without branching. Who this is for: engineers maintaining an existing production codebase that calls Azure OpenAI / OpenAI - directly or through LangChain - and needs to onboard GPT-5.x while keeping the GPT-4 deployments alive during rollout. What you'll leave with: one copy-paste compatibility module, a tiny LangChain subclass, a prompt-audit harness, and a 10-step rollout checklist. 1. Why this migration is different Every previous Azure OpenAI bump - 3.5 → 4, 4 → 4o, 4o → 4o-mini - was additive. You changed engine="gpt-4o" and everything kept working. GPT-5.x is the first generation that is subtractive: parameters you used to send now return 400 Unsupported parameter. The wire protocol itself changed because GPT-5 is a reasoning model - it spends tokens thinking internally before it answers, so the parameters that controlled the old sampling pipeline (temperature, top_p, presence_penalty, frequency_penalty) no longer exist on the request schema. What this means for production code: A passing test suite against gpt-4o will fail on the first call against gpt-5.1 with HTTP 400. A passing test suite against gpt-5.1 will fail on every legacy gpt-4* deployment because the new reasoning controls (reasoning_effort, verbosity) are not recognised there. LangChain helpers that worked unmodified for two years (notably create_sql_query_chain) silently bind stop=[...] onto your LLM and trigger the same 400. Source-grep won't find the offending line because it lives inside the library. The good news: the divergence is mechanical. With one detection helper, one parameter-builder, and one tiny LangChain subclass you can run the same code against both families. 2. The breaking-changes matrix Concern GPT-4 / GPT-4o (legacy) GPT-5.x / o1 / o3 (reasoning) Output budget max_tokens max_completion_tokens (rejects max_tokens) temperature 0.0–1.0 Only the default (1) is accepted - omit it top_p Supported Rejected presence_penalty, frequency_penalty Supported Rejected logprobs, logit_bias Supported Rejected stop sequences Supported Rejected on most reasoning deployments reasoning_effort Rejected New: minimal | low | medium | high verbosity Rejected New: low | medium | high (sometimes via extra_body) System instruction role system developer recommended; system still works as alias Output token cost Output tokens only Output + reasoning tokens count against your cap Recommended API version 2024-12-01-preview or earlier 2025-03-01-preview or later Two consequences are easy to miss: max_completion_tokens is a shared budget. GPT-5.1 can burn 2–4× more tokens internally before emitting the first response token. A cap of 4096 that comfortably held a SQL query on GPT-4o now silently truncates the answer mid-token on GPT-5.1. Multiply your legacy budgets by ~2.5× and add a floor (e.g. 4096) before sending. The stop parameter is the silent killer. Any helper that calls llm.bind(stop=[...]) - and there are several in langchain - will turn a working code path into a 400 the moment you swap deployments. 3. Compatibility strategy: detect, don't fork The temptation is to fork: one branch for GPT-4, one for GPT-5. Don't. The right unit of abstraction is one function that classifies the deployment into a family, and one function that builds a kwargs dict the SDK will accept for that family. Every call site - SDK, LangChain, raw HTTP - drains into the same kwargs builder. When you eventually retire GPT-4 you delete the legacy branch in one file, not in fifty. 4. The industry-agnostic compatibility module Drop the following file into your project. It has no Azure / OpenAI / LangChain imports at module load time, so the same file works from a web service, a serverless function, a notebook, or a CLI tool. 4.1 model_compat.py """ Model compatibility helper for GPT-5.x with GPT-4 backward compatibility. This module centralises the parameter translation needed to talk to the "reasoning" generation of OpenAI / Azure OpenAI models (GPT-5, GPT-5.1, o1, o3, o4) while keeping older deployments (gpt-4, gpt-4o, gpt-4-32k, gpt-3.5-turbo, etc.) working unchanged. """ from __future__ import annotations import logging import os import re from typing import Any, Dict, Iterable, Mapping, Optional # --------------------------------------------------------------------------- # Family detection # --------------------------------------------------------------------------- _REASONING_PATTERNS = ( # gpt-5, gpt5, gpt-5.1, gpt_5, GPT 5, gpt5mini-prod-eu, ... re.compile(r"(?i)(^|[^a-z0-9])gpt[-_ ]?5(\.\d+)?([^0-9]|$)"), # o1, o3, o4, o1-mini, o3-preview ... re.compile(r"(?i)(^|[^a-z0-9])o[134](-mini|-preview)?([^a-z0-9]|$)"), ) _LEGACY_PATTERNS = ( re.compile(r"(?i)gpt[-_ ]?4o"), re.compile(r"(?i)gpt[-_ ]?4(?!\d)"), re.compile(r"(?i)gpt[-_ ]?4[-_ ]?32k"), re.compile(r"(?i)gpt[-_ ]?3\.?5"), re.compile(r"(?i)gpt[-_ ]?35"), ) def get_model_family(model_or_deployment: Optional[str]) -> str: """Return ``"reasoning"`` for GPT-5.x / o-series, ``"legacy"`` otherwise. Honours an ``OPENAI_MODEL_FAMILY`` env-var override for deployments whose user-defined name does not embed the model family (e.g. ``prod-default``). """ override = (os.getenv("OPENAI_MODEL_FAMILY") or "").strip().lower() if override in {"reasoning", "gpt-5", "gpt5", "gpt-5.1", "o-series", "o1", "o3"}: return "reasoning" if override in {"legacy", "gpt-4", "gpt4", "gpt-3.5", "gpt35", "chat"}: return "legacy" name = (model_or_deployment or "").strip() if not name: # Fail closed: when we don't know, assume legacy so old code keeps # working. Misclassifying a reasoning deployment as legacy fails fast # with a clear "Unsupported parameter" 400; the reverse silently # drops parameters the caller expected. return "legacy" for pat in _REASONING_PATTERNS: if pat.search(name): return "reasoning" for pat in _LEGACY_PATTERNS: if pat.search(name): return "legacy" return "legacy" def is_reasoning_model(model_or_deployment: Optional[str]) -> bool: return get_model_family(model_or_deployment) == "reasoning" # --------------------------------------------------------------------------- # Reasoning controls # --------------------------------------------------------------------------- _VALID_REASONING_EFFORT = {"minimal", "low", "medium", "high"} _VALID_VERBOSITY = {"low", "medium", "high"} def _coerce_choice(raw: Optional[str], valid: Iterable[str]) -> Optional[str]: if raw is None: return None value = str(raw).strip().lower() if not value: return None if value not in set(valid): logging.warning( "Ignoring unsupported value '%s'; expected one of %s", raw, sorted(valid), ) return None return value def get_reasoning_effort(override: Optional[str] = None) -> Optional[str]: return _coerce_choice( override if override is not None else os.getenv("OPENAI_REASONING_EFFORT"), _VALID_REASONING_EFFORT, ) def get_verbosity(override: Optional[str] = None) -> Optional[str]: return _coerce_choice( override if override is not None else os.getenv("OPENAI_VERBOSITY"), _VALID_VERBOSITY, ) # --------------------------------------------------------------------------- # max_completion_tokens scaling # --------------------------------------------------------------------------- def _reasoning_token_scale() -> float: """Multiplier applied to legacy ``max_tokens`` when targeting a reasoning model.""" try: scale = float(os.getenv("OPENAI_REASONING_TOKEN_SCALE", "2.5")) except (TypeError, ValueError): scale = 2.5 return scale if scale > 0 else 1.0 def _reasoning_token_floor() -> int: try: floor = int(os.getenv("OPENAI_REASONING_TOKEN_FLOOR", "4096")) except (TypeError, ValueError): floor = 4096 return floor if floor > 0 else 4096 def scale_max_tokens_for_reasoning(max_tokens: Optional[int]) -> Optional[int]: """Scale a legacy ``max_tokens`` budget up for reasoning models. ``None`` and ``-1`` ("no explicit cap") are passed through. """ if max_tokens is None: return None if max_tokens == -1: return -1 return max(int(round(max_tokens * _reasoning_token_scale())), _reasoning_token_floor()) # --------------------------------------------------------------------------- # Kwargs builders # --------------------------------------------------------------------------- _SAMPLING_KEYS = ("temperature", "top_p", "presence_penalty", "frequency_penalty") def _drop_none(mapping: Mapping[str, Any]) -> Dict[str, Any]: return {k: v for k, v in mapping.items() if v is not None} def build_openai_chat_kwargs( model: str, *, max_tokens: Optional[int] = None, temperature: Optional[float] = None, top_p: Optional[float] = None, presence_penalty: Optional[float] = None, frequency_penalty: Optional[float] = None, reasoning_effort: Optional[str] = None, verbosity: Optional[str] = None, extra: Optional[Mapping[str, Any]] = None, ) -> Dict[str, Any]: """Build kwargs for ``openai.OpenAI / AzureOpenAI .chat.completions.create``. Splat the result directly: ``client.chat.completions.create(**kwargs)``. Unsupported parameters are silently omitted for reasoning models; legacy deployments retain the historical behaviour. """ family = get_model_family(model) kwargs: Dict[str, Any] = {"model": model} # ---- output budget ---- if max_tokens is not None and max_tokens != -1: if family == "reasoning": kwargs["max_completion_tokens"] = scale_max_tokens_for_reasoning(int(max_tokens)) else: kwargs["max_tokens"] = int(max_tokens) # ---- sampling ---- if family == "legacy": kwargs.update(_drop_none({ "temperature": temperature, "top_p": top_p, "presence_penalty": presence_penalty, "frequency_penalty": frequency_penalty, })) else: for key, value in ( ("temperature", temperature), ("top_p", top_p), ("presence_penalty", presence_penalty), ("frequency_penalty", frequency_penalty), ): if value is not None: logging.debug( "Dropping unsupported parameter '%s' for reasoning model '%s'", key, model, ) # ---- reasoning controls ---- if family == "reasoning": effort = get_reasoning_effort(reasoning_effort) if effort is not None: kwargs["reasoning_effort"] = effort verb = get_verbosity(verbosity) if verb is not None: # ``verbosity`` is not a top-level kwarg in openai-python <= 1.65.x; # route it via ``extra_body`` so it lands in the JSON without a # TypeError from the SDK. kwargs.setdefault("extra_body", {})["verbosity"] = verb # ---- caller-supplied extras (already filtered) ---- if extra: for key, value in extra.items(): if value is None: continue if family == "reasoning" and key in _SAMPLING_KEYS: continue kwargs[key] = value return kwargs def build_langchain_chat_kwargs( deployment_name: str, *, max_tokens: Optional[int] = None, temperature: Optional[float] = None, top_p: Optional[float] = None, reasoning_effort: Optional[str] = None, verbosity: Optional[str] = None, ) -> Dict[str, Any]: """Build kwargs for ``langchain_openai.AzureChatOpenAI`` / ``ChatOpenAI``. Older ``langchain-openai`` releases don't expose ``max_completion_tokens`` as a top-level kwarg, so we forward it through ``model_kwargs`` (which langchain passes straight to the SDK). """ family = get_model_family(deployment_name) kwargs: Dict[str, Any] = {} model_kwargs: Dict[str, Any] = {} if max_tokens is not None and max_tokens != -1: if family == "reasoning": model_kwargs["max_completion_tokens"] = scale_max_tokens_for_reasoning(int(max_tokens)) else: kwargs["max_tokens"] = int(max_tokens) if family == "reasoning": effort = get_reasoning_effort(reasoning_effort) if effort is not None: model_kwargs["reasoning_effort"] = effort verb = get_verbosity(verbosity) if verb is not None: model_kwargs.setdefault("extra_body", {})["verbosity"] = verb else: if temperature is not None: kwargs["temperature"] = temperature if top_p is not None: kwargs["top_p"] = top_p if model_kwargs: kwargs["model_kwargs"] = model_kwargs return kwargs def get_system_role(model_or_deployment: Optional[str] = None) -> str: """Return ``"developer"`` for reasoning models when opted in, ``"system"`` otherwise. Defaulting to ``"system"`` preserves compatibility with LangChain prompt templates and SDK helpers that don't yet recognise the new role. Opt in with ``OPENAI_USE_DEVELOPER_ROLE=1`` once your stack supports it. """ if not is_reasoning_model(model_or_deployment): return "system" raw = os.getenv("OPENAI_USE_DEVELOPER_ROLE", "") return "developer" if raw.strip().lower() in {"1", "true", "yes", "on"} else "system" 4.2 What this buys you Every direct-SDK call collapses to two lines: from openai import AzureOpenAI from model_compat import build_openai_chat_kwargs client = AzureOpenAI( azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], api_version=os.environ["OPENAI_API_VERSION"], api_key=os.environ["AZURE_OPENAI_API_KEY"], ) kwargs = build_openai_chat_kwargs( model=os.environ["OPENAI_ENGINE"], max_tokens=4096, # automatically becomes max_completion_tokens for GPT-5 temperature=0.2, # automatically dropped for GPT-5 reasoning_effort="low", # automatically dropped for GPT-4 ) response = client.chat.completions.create( messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": user_input}, ], **kwargs, ) The same call site now correctly targets gpt-5.1, gpt-4o, gpt-4-32k, o3-mini, or any future deployment whose name embeds the family - and you can override with the OPENAI_MODEL_FAMILY env var when the deployment alias is opaque. 4.3 Raw HTTP call sites Some legacy code paths bypass the SDK and POST JSON directly. The same builder works there: import json import requests from model_compat import build_openai_chat_kwargs, get_system_role deployment = os.environ["OPENAI_ENGINE"] api_version = os.environ["OPENAI_API_VERSION"] endpoint = ( f"{os.environ['AZURE_OPENAI_ENDPOINT']}/openai/deployments/{deployment}" f"/chat/completions?api-version={api_version}" ) payload = { "messages": [ {"role": get_system_role(deployment), "content": system_prompt}, {"role": "user", "content": user_prompt}, ], } # Splat the kwargs into the payload, then strip the SDK-only ``model`` key. payload.update(build_openai_chat_kwargs( model=deployment, max_tokens=800, temperature=0.7, top_p=0.95, reasoning_effort="low", )) payload.pop("model", None) # ``model`` is encoded in the URL for Azure payload.pop("extra_body", None) # already on the payload root resp = requests.post( endpoint, headers={"Content-Type": "application/json", "api-key": api_key}, data=json.dumps(payload), timeout=60, ) resp.raise_for_status() 5. LangChain: the hidden stop parameter langchain.chains.sql_database.query.create_sql_query_chain calls llm.bind(stop=["\nSQLResult:"]) internally to terminate the model's output before the example block in its prompt. That stop value is forwarded to the SDK on every invocation. GPT-5.1 rejects it: openai.BadRequestError: Error code: 400 - {'error': { 'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', }} You can't reach into the chain to disable it. The clean fix is a thin AzureChatOpenAI subclass that drops stop for reasoning models only: 5.1 langchain_compat.py """LangChain-side compatibility shim for reasoning-class deployments.""" from __future__ import annotations from typing import Any, List, Optional from langchain_core.callbacks.manager import ( AsyncCallbackManagerForLLMRun, CallbackManagerForLLMRun, ) from langchain_core.messages import BaseMessage from langchain_core.outputs import ChatResult from langchain_openai import AzureChatOpenAI # use ChatOpenAI for non-Azure from model_compat import is_reasoning_model class ReasoningSafeAzureChatOpenAI(AzureChatOpenAI): """``AzureChatOpenAI`` variant that hides parameters reasoning models reject. Reasoning models (GPT-5.x, o1/o3/o4) return HTTP 400 when a request payload carries ``stop``. LangChain's SQL helpers unconditionally bind it, so the unsupported parameter reaches the SDK regardless of how the caller configured the LLM. This subclass strips ``stop`` for reasoning deployments while forwarding it unchanged for legacy GPT-4 / GPT-3.5 deployments - the behaviour is byte-identical to upstream LangChain for those models. """ def _deployment_id(self) -> str: # ``langchain-openai`` >= 0.2 exposes ``azure_deployment``; older # releases use ``deployment_name``. Either may be set by the caller. return ( getattr(self, "azure_deployment", None) or getattr(self, "deployment_name", None) or "" ) def _generate( self, messages: List[BaseMessage], stop: Optional[List[str]] = None, run_manager: Optional[CallbackManagerForLLMRun] = None, **kwargs: Any, ) -> ChatResult: if is_reasoning_model(self._deployment_id()): stop = None return super()._generate(messages, stop=stop, run_manager=run_manager, **kwargs) async def _agenerate( self, messages: List[BaseMessage], stop: Optional[List[str]] = None, run_manager: Optional[AsyncCallbackManagerForLLMRun] = None, **kwargs: Any, ) -> ChatResult: if is_reasoning_model(self._deployment_id()): stop = None return await super()._agenerate(messages, stop=stop, run_manager=run_manager, **kwargs) Use it as a drop-in replacement: from langchain_compat import ReasoningSafeAzureChatOpenAI from model_compat import build_langchain_chat_kwargs llm_kwargs = build_langchain_chat_kwargs( deployment_name=os.environ["OPENAI_ENGINE"], max_tokens=6000, temperature=0, reasoning_effort="low", ) llm = ReasoningSafeAzureChatOpenAI( azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], azure_deployment=os.environ["OPENAI_ENGINE"], openai_api_version=os.environ["OPENAI_API_VERSION"], api_key=os.environ["AZURE_OPENAI_API_KEY"], **llm_kwargs, ) That single substitution makes create_sql_query_chain, SQLDatabaseChain, and the ChatOpenAI-based RAG helpers all work against GPT-5.1 without any other changes. 6. The second LangChain gotcha: prose where SQL should be create_sql_query_chain is documented to return the literal string "I don't know" (or a similar fallback) when the LLM cannot form a query. The default code path takes the chain output and runs it against the database: sql = chain.invoke({...}) # -> "I don't know" result = db.run(sql) # -> sends "I don't know" to pyodbc The database faithfully returns: [42000] Unclosed quotation mark after the character string 't know'. (105) Which surfaces to the end user as a misleading "SQL syntax error". The mitigation is a one-line guard that validates the chain output looks like SQL before execution: import re _SQL_START_RE = re.compile( r"^\s*(?:WITH|SELECT|INSERT|UPDATE|DELETE|CREATE|DROP|ALTER|MERGE|EXEC|EXECUTE|TRUNCATE)\b", re.IGNORECASE, ) def looks_like_sql(text: str) -> bool: """True only if ``text`` starts with a recognised SQL DML/DDL keyword.""" if not text or not text.strip(): return False return bool(_SQL_START_RE.match(text)) sql = extract_sql_query(chain.invoke({...})) if not looks_like_sql(sql): logging.warning("SQL chain returned a non-SQL response: %r", sql[:200]) return ( "I couldn't form a SQL query for that question. " "Please rephrase or add more context." ) result = db.run(sql) This isn't specific to GPT-5.1 - it's good hygiene for any LLM that backs a SQL agent - but the failure mode becomes much more frequent on reasoning models because they're better at refusing. 7. Cleaning Markdown out of create_sql_query_chain output Reasoning models like to wrap their answer in a markdown fence and append a "Note:" or "Explanation:" paragraph. None of that survives db.run(). A defensive extract_sql_query handles all the variants: import re def extract_sql_query(text: str) -> str: """Strip markdown fences, leading prose, and trailing explanations.""" # 1) Prefer SQL inside a markdown code fence. m = re.search(r"```(?:sql|SQL|Sql)?\s*\n(.*?)\n```", text, re.DOTALL) if m: text = m.group(1) text = text.strip() # 2) Drop any prose *before* the SQL by jumping to the first SQL keyword. m = re.search( r"(?im)^\s*(WITH|SELECT|INSERT|UPDATE|DELETE|CREATE|DROP|ALTER|MERGE|EXEC|EXECUTE|TRUNCATE)\b", text, ) if m: text = text[m.start(1):] # 3) Cut at the first "Explanation:" / "Note:" / "This query..." marker. m = re.compile( r"(?im)^\s*(?:Explanation|Note|Notes|Here(?:'|\u2019)?s|" r"This\s+(?:query|SQL|statement|returns|counts|selects|will|gets|finds)|" r"The\s+(?:query|SQL|above|result|statement)|" r"Result|Results|Description|Output|Answer)\b[^\n]*" ).search(text) if m: text = text[: m.start()].rstrip() # 4) Drop any trailing fence that survived step 1. if text.endswith("```"): text = text[:-3].rstrip() return text.strip() 8. Package versioning The bare minimum your requirements.txt / environment.yml needs: Package Last GPT-4-only version First GPT-5.x-safe version Notes openai 1.55.x 1.65.x (recommend 1.65.4+) Earlier versions reject max_completion_tokens and reasoning_effort as unknown kwargs langchain-openai 0.2.14 0.3.7+ 0.3.x line exposes azure_deployment and forwards model_kwargs correctly to the new SDK langchain 0.3.14 0.3.21+ Pin together with langchain-openai and langchain-core langchain-core 0.3.29 0.3.49+ Update in lockstep with the others langchain-community 0.3.14 0.3.20+ Mostly transitive; needed for SQLDatabase helpers tiktoken 0.7.x 0.8.0+ Encodings for GPT-5.1 ship in 0.8.0; older versions fall back to cl100k_base for unknown models tokencost (optional) 0.1.16 0.1.20+ Update for GPT-5.x price tables Azure OpenAI API version 2024-12-01-preview 2025-03-01-preview First version that ships reasoning_effort and the GPT-5.x routing Pin exact versions after testing - LangChain has a habit of moving public re-exports between minor releases. requirements.txt snippet: openai==1.65.4 langchain==0.3.21 langchain-core==0.3.49 langchain-openai==0.3.7 langchain-community==0.3.20 tiktoken==0.8.0 9. New GPT-5.x knobs worth using Once you're on a reasoning deployment, two new parameters become available. Both are optional, both default to a sensible value, and both are stripped by the kwargs builder above when the target is a legacy model. reasoning_effort minimal - one-shot lookups, classification. low - deterministic structured output (SQL, JSON-schema extraction, rule-based rewrites). Lowest cost overhead. medium (default) - RAG, summarisation, normal Q&A. high - multi-step analytical reasoning, complex code synthesis. A useful pattern is to choose the level by task profile rather than at the call site: TASK_EFFORT = { "sql": "low", "structured_extract": "low", "kg_cleaning": "low", "rag_qa": "medium", "vision": "medium", "analytical": "high", } verbosity low | medium | high. Controls the length of the response, not its substance. Useful for grounding chat UIs where you want crisp answers - set low for /answer endpoints and high for "explain like a senior engineer" panels. Note: in openai-python <= 1.65.x, verbosity is not yet a top-level keyword argument; pass it through extra_body (the builder above already does this). developer role GPT-5.x prefers {"role": "developer", "content": "..."} for instructions that previously used system. The change is non-breaking on the Azure side - system is still accepted as an alias - but some downstream LangChain prompt templates predate the role and will reject it on construction. Treat developer as opt-in (OPENAI_USE_DEVELOPER_ROLE=1) for now; flip the default after your prompt-template version is known good. 10. Auditing your existing prompts When the wire-level migration is done your service will talk to GPT-5.x - but that doesn't mean it says the right thing. Reasoning models read prompts differently in ways that won't show up as 400s: They take instructions more literally. A prompt that worked when GPT-4o rounded the corners may surface every edge case verbatim. They refuse more often. "I don't know" / "I cannot help with that" are more frequent because reasoning models are less willing to confabulate. They ignore "be concise" / "be terse". Use the new verbosity knob. Step-by-step / chain-of-thought instructions become redundant. The model already reasons internally; extra "think before you answer" prose competes with its own chain of thought and often hurts output quality. Negative-only instructions can backfire. "Never output X" prompts occasionally cause refusals where you'd rather have a workaround. 10.1 Build a prompt regression harness Capture every system+user prompt your service emits in a CSV, then replay each one against both deployments and diff the output. The diff is the single most useful artefact you can produce before the cutover: # prompt_audit.py - minimal differential tester import csv from openai import AzureOpenAI from model_compat import build_openai_chat_kwargs LEGACY = "gpt-4o" REASONING = "gpt-5.1" client = AzureOpenAI( azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], api_version=os.environ["OPENAI_API_VERSION"], api_key=os.environ["AZURE_OPENAI_API_KEY"], ) def run(model: str, system: str, user: str) -> str: kw = build_openai_chat_kwargs( model=model, max_tokens=4096, temperature=0.2, # auto-dropped for reasoning reasoning_effort="medium", # auto-dropped for legacy ) resp = client.chat.completions.create( messages=[ {"role": "system", "content": system}, {"role": "user", "content": user}, ], **kw, ) return resp.choices[0].message.content or "" with open("prompts.csv") as f_in, open("diff.tsv", "w", newline="") as f_out: writer = csv.writer(f_out, delimiter="\t") writer.writerow(["id", "legacy_first80", "reasoning_first80", "len_legacy", "len_new", "identical"]) for row in csv.DictReader(f_in): legacy = run(LEGACY, row["system"], row["user"]) new = run(REASONING, row["system"], row["user"]) writer.writerow([ row["id"], legacy[:80].replace("\n", " "), new[:80].replace("\n", " "), len(legacy), len(new), legacy.strip() == new.strip(), ]) Capture three signals per prompt - they're enough to triage 95% of drift: Format compliance. Did the output still parse as the expected JSON / YAML / Markdown / SQL? Run your existing downstream parser on both columns. Token cost delta. Reasoning models tend to be more verbose by default. Anything beyond +20% is a candidate for the verbosity="low" knob. Semantic drift. Spot-check 5–10% of rows by hand. You're looking for changes in intent, not changes in wording. 10.2 Common rewrites to make prompts model-agnostic The goal isn't to write two prompts. It's to write one prompt that produces correct output on both families by moving constraints out of the natural-language body and into the request shape. 10.2a. Format constraints belong in response_format, not the prose Don't: Output ONLY a JSON object with keys `name` and `score`. Do not include any explanation. Do not wrap in markdown. Do not say anything else. Do: resp = client.chat.completions.create( messages=[...], response_format={ "type": "json_schema", "json_schema": { "name": "scored_entity", "schema": { "type": "object", "properties": { "name": {"type": "string"}, "score": {"type": "number"}, }, "required": ["name", "score"], "additionalProperties": False, }, "strict": True, }, }, **kw, ) response_format is honoured by both gpt-4o (>= 2024-08-06) and the entire GPT-5.x line. The prompt loses three lines of brittle natural-language constraints and you get schema-validated output for free. 10.2b. Replace "think step by step" with reasoning_effort Don't: Let's think step by step. First identify the entity. Then find the category. Then compute the score. Then format the answer. Do: delete the prose and pass reasoning_effort="medium" (or "high") for reasoning deployments. The kwargs builder drops the parameter automatically for GPT-4 models, so the same prompt now produces: step-by-step reasoning internally on GPT-5.x (lower output token cost), the same final answer on GPT-4o that the verbose prompt used to elicit. 10.2c. Replace temperature-based variety with n sampling If your code relied on temperature=0.9 to get diverse completions, GPT-5.x will return roughly the same answer every time. Generate variety the explicit way: resp = client.chat.completions.create(messages=[...], n=5, **kw) candidates = [c.message.content for c in resp.choices] Or call the model N times with slightly different framings. Both patterns work against either family with no further code changes. 10.2d. Move procedural instructions to the developer role For multi-step workflows, the new developer role gives clearer separation between what the system enforces and what the user is asking: messages = [ {"role": get_system_role(deployment), "content": role_card_for_assistant}, {"role": "developer", "content": procedural_instructions}, {"role": "user", "content": user_question}, ] get_system_role returns "system" for legacy models and "developer" for reasoning models opted in via OPENAI_USE_DEVELOPER_ROLE=1. Once your LangChain templates support the new role you can flip the default. 10.2e. Add a literal-execution header for strict formats For prompts where the exact output shape matters (table generation, SQL with a fixed column order, structured incident reports), prepend an explicit literal-execution header so reasoning models don't drift into "helpful improvements": LITERAL_EXECUTION_HEADER = ( "Execution mode: follow the instructions below literally and in order. " "Do not infer intent, skip, reorder, merge, or add steps. Honour the " "exact formatting, tone, and verbosity specified. If a step is " "ambiguous, respond with the literal interpretation and flag the " "ambiguity instead of guessing." ) def apply_literal_execution(prompt: str) -> str: if LITERAL_EXECUTION_HEADER in prompt: return prompt return f"{LITERAL_EXECUTION_HEADER}\n\n{prompt}" It's a no-op on GPT-4o (the older models already follow instructions literally enough) and a meaningful guard rail on GPT-5.1. Wire it behind an OPENAI_LITERAL_EXECUTION flag so you can disable it without redeploying. 10.3 A prompt-shaped checklist Run every prompt your service emits past these questions: Question Action Does it specify output format in prose? Move to response_format (10.2a) Does it include "think step by step"? Remove; set reasoning_effort (10.2b) Does it set tone constraints ("be concise")? Use verbosity Does it use negative-only instructions ("never X")? Add positive alternative ("do Y instead") Does it embed example outputs with values that would change? Replace concrete values with placeholder tokens (<VALUE>) Does it rely on temperature > 0 for variety? Use n=K sampling (10.2c) Is the system prompt > 2k tokens? Split into role-card (system) + procedure (developer) Does output ordering matter? Add the literal-execution header (10.2e) 10.4 Score before you ship Don't approve a rewritten prompt by eyeballing one example. Score it: Format compliance rate. Percentage of N=50 outputs that pass your existing downstream parser / JSON schema validation. Token cost delta. Cap regression at +20% versus the legacy baseline. Beyond that, dial verbosity="low" or tighten the prompt. Latency p50 / p95 delta. Reasoning models add tail latency. If your SLA is tight, set reasoning_effort="low" for the path or move it to a background queue. A prompt that regresses on any of those by more than your tolerance window ships behind a feature flag with rollback wired in. 11. Testing strategy Two test layers catch >90% of regressions: Family-classification tests import pytest from model_compat import get_model_family, build_openai_chat_kwargs @pytest.mark.parametrize("name,expected", [ ("gpt-5.1", "reasoning"), ("gpt5", "reasoning"), ("gpt-5-prod-eu", "reasoning"), ("o3-mini", "reasoning"), ("o1", "reasoning"), ("gpt-4o", "legacy"), ("gpt-4", "legacy"), ("gpt-4-32k", "legacy"), ("gpt-35-turbo", "legacy"), ("", "legacy"), # unknown -> fail closed to legacy (None, "legacy"), ]) def test_family(name, expected): assert get_model_family(name) == expected def test_kwargs_for_reasoning_drops_temperature(): kw = build_openai_chat_kwargs( model="gpt-5.1", max_tokens=1000, temperature=0.2, top_p=0.9, reasoning_effort="low", ) assert "temperature" not in kw assert "top_p" not in kw assert kw["max_completion_tokens"] >= 4096 # floor applied assert kw["reasoning_effort"] == "low" def test_kwargs_for_legacy_keeps_temperature(): kw = build_openai_chat_kwargs( model="gpt-4o", max_tokens=1000, temperature=0.2, top_p=0.9, ) assert kw["max_tokens"] == 1000 assert kw["temperature"] == 0.2 assert kw["top_p"] == 0.9 assert "reasoning_effort" not in kw Wire-level smoke tests For each LLM call site you maintain, write a single integration test that exercises the chain against a real (or mocked) endpoint and asserts: HTTP 200, non-empty content, finish_reason != "length" (so you catch silent truncation), (optional) classifier-style assertions against a golden output. Run those tests once against the legacy deployment and once against the new one - same test code, two OPENAI_ENGINE values. 12. Things that don't change It's easy to over-correct. Several pieces of plumbing keep working without modification: Authentication. AAD token providers, managed identity, and API keys are unchanged. Embeddings. text-embedding-3-small, text-embedding-3-large, and text-embedding-ada-002 are not part of the reasoning generation; the embeddings call shape is identical. Function calling / tool use. Same JSON schema, same response shape. Streaming. SSE format is unchanged. Token counters. tiktoken still works, but bump to 0.8.0+ so the new model name resolves to the right encoding instead of silently falling back to cl100k_base. 13. Next steps If you only do four things from this post, do these - in order: Deploy a GPT-5.1 model side-by-side with your current GPT-4 deployment in Microsoft Foundry. Keep the GPT-4 deployment live; you'll need both for the parallel-run period. Drop model_compat.py and langchain_compat.py into your project (Sections 4 and 5). Replace every AzureChatOpenAI(...) construction with ReasoningSafeAzureChatOpenAI and route every kwargs literal through the builders. Run the prompt-audit harness (Section 10.1) against your top 50 most frequently invoked prompts. Triage the diff with the checklist in 10.3. Roll out behind a percentage-based flag. Start at 5% of traffic for 24 hours, compare quality and cost telemetry against the GPT-4o baseline, then ramp. Reference material Azure OpenAI in Microsoft Foundry - model overview Azure OpenAI model retirements and deprecations Reasoning models in Azure OpenAI Structured Outputs in Azure OpenAI openai-python SDK changelog langchain-openai release notes Talk to us Open an issue on the Microsoft Foundry GitHub samples repository if you hit a gap this post didn't cover. Share your migration story or numbers in the comments below - field data is the fastest way to make this guide better for the next team. If you operate a regulated workload (finance, health, public sector) and need help sequencing the rollout with your model retirement deadlines, reach out to your Microsoft account team or a Microsoft Foundry partner. GPT-5.x is the first major model bump in two years that requires code changes - but the changes collapse into one small compatibility module and a one-line LangChain subclass. With those in place your code is forwards-compatible (works on reasoning models today) and backwards- compatible (still works on every GPT-4 deployment you haven't migrated yet). The investment pays a recurring dividend: when the next reasoning bump ships, the only file that needs updating is model_compat.py. Appendix A - Minimal .env template # Endpoint and auth (unchanged between families) AZURE_OPENAI_ENDPOINT=https://<resource>.openai.azure.com AZURE_OPENAI_API_KEY=<key> # The deployment name decides the family. The classifier reads it. OPENAI_ENGINE=gpt-5.1 OPENAI_API_VERSION=2025-03-01-preview # Optional override for opaque deployment names # OPENAI_MODEL_FAMILY=reasoning # or "legacy" # Optional reasoning controls (ignored for legacy deployments) OPENAI_REASONING_EFFORT=medium OPENAI_VERBOSITY=medium OPENAI_REASONING_TOKEN_SCALE=2.5 OPENAI_REASONING_TOKEN_FLOOR=4096 # Flip when your LangChain templates support it # OPENAI_USE_DEVELOPER_ROLE=1 Appendix B - One-liner sanity checks # Does a deployment name classify correctly? python -c "from model_compat import get_model_family; print(get_model_family('gpt-5.1'))" # -> reasoning # Does the LangChain LLM strip ``stop`` when the deployment is GPT-5.1? python -c " from langchain_compat import ReasoningSafeAzureChatOpenAI import inspect; print(inspect.getsource(ReasoningSafeAzureChatOpenAI._generate)) " Companion repository: drop model_compat.py and langchain_compat.py next to each other in your utils/ package. They are zero-dependency on import, so you can vendor them into any service - web, function, batch job - without dragging Azure SDK or LangChain into module-load.Foundry IQ: Improve recall by up to 54% with knowledge bases
Foundry IQ: Improve recall by up to 54% with knowledge bases. Foundry IQ (Azure AI Search) has improved its agentic retrieval engine resulting in better answer quality and improved token cost savings. We compared standalone retrieval tools to knowledge bases using the challenging BrowseComp-Plus benchmark and found: Replacing single-shot RAG with a knowledge base improves evidence recall by up to 46%. Combining a smaller agent model with agentic retrieval improves evidence recall by up to 54% while controlling costs and increasing agent responsiveness. In both cases, the amount of retrieval tool calls your agent makes is reduced, resulting in 34% token cost savings.Introducing MAI-Transcribe-1, MAI-Voice-1, and MAI-Image-2 in Microsoft Foundry
Another Step Towards a Complete AI Platform Since inception, our goal with Microsoft Foundry has been to deliver the most complete AI and app agent factory; giving developers access to the latest frontier models, tools, infrastructure, security, and reliability to confidently build and scale their AI solutions. Today, we're taking another step towards that vision by announcing the public preview of three new models from Microsoft AI in Microsoft Foundry: MAI-Transcribe-1: Our first-generation speech recognition model, delivering enterprise-grade accuracy across 25 languages at approximately 50% lower GPU cost than leading alternatives. MAI-Voice-1: A high-fidelity speech generation model capable of producing 60 seconds of expressive audio in under one second on a single GPU. MAI-Image-2: Our highest-capability text-to-image model, which debuted on #3 on the Arena.ai leaderboard for image model families. These are the same models already powering our own products such as Copilot, Bing, PowerPoint, and Azure Speech, and now they're available exclusively on Foundry for developers to use. We can't wait to see what you create with these new multimedia AI models in public preview. Read on for a deeper look at each model's capabilities and how to start building with them in Foundry! MAI-Transcribe-1 & Voice-1: End-To-End Voice Experiences Voice and speech are rapidly becoming the primary interface for the next generation of AI agents, and building great voice experiences requires models that can both speak and listen with precision. With MAI-Voice-1 and MAI-Transcribe-1, Microsoft is delivering exactly that: a comprehensive, first-party audio AI stack purpose-built for developers. MAI-Voice-1 is a lightning-fast speech generation model capable of producing a full minute of audio in under a second on a single GPU; making it one of the most efficient speech systems available today. On the listening side, MAI-Transcribe-1 supports up to 25 languages and is engineered for enterprise-grade reliability across accents, languages, and real-world audio conditions. But what truly sets it apart is its efficiency: when benchmarked against leading transcription models, MAI-Transcribe-1 delivers competitive accuracy at nearly half the GPU cost; an advantage that translates directly into more predictable, scalable pricing for enterprises 1 . Use cases for MAI-Transcribe-1 and MAI-Voice-1 MAI-Voice-1 and MAI-Transcribe-1 are designed for production use across a broad set of real-world scenarios: Conversational AI & Agent Assist: Enable real‑time transcription for IVR systems, virtual assistants, and call‑center workflows to power voice‑driven interfaces, live agent assist, and post‑call summarization. Live Captioning & Accessibility: Deliver real‑time captions for large events, enterprise meetings, and digital communications to improve accessibility and inclusivity across spoken experiences. Media, Subtitling & Archiving: Automate video subtitling, dialogue indexing, and transcription to support scalable content production, searchability, and long‑term media archiving. Education & Training Platforms: Transcribe lectures, learning modules, and certification programs to enhance discoverability, reviewability, and knowledge retention in e‑learning environments. Customer & Market Insights: Convert spoken interactions across research interviews, focus groups, and support channels into structured data for downstream analytics and business intelligence. We're also applying these model capabilities inside Microsoft's own products. MAI-Voice-1 powers the expressive voice experiences in Copilot's Audio Expressions and podcast features. MAI-Transcribe-1 drives Copilot's Voice Mode transcriptions and the new dictation feature, connecting natural voice input with the generative power of Copilot's language models. Both models are available through Azure Speech, where developers can tap into first-party MAI model quality alongside the enterprise-grade reliability, scalability, and 700+ voice gallery of the Azure Speech ecosystem. Try MAI-Transcribe-1 & Voice-1 Today MAI-Transcribe-1 and Voice-1 are available now through Azure Speech. Here's how to get started: Experiment in MAI Playground: Speak, record, or upload audio to see the models in action at the MAI playground. Build in Foundry: deploy MAI-Transcribe-1 and MAI-Voice-1 in Azure Speech. MAI-Transcribe-1 starts at $0.36 USD per hour, while MAI-Voice-1 pricing starts at $22 USD per 1M characters. Developers looking to create custom voices using MAI-Voice-1 can do so through the Personal Voice feature in Azure Speech — including the ability to clone a voice from a short 10-second audio sample. Note that custom voice creation requires an approval process consistent with Microsoft's responsible AI policies. MAI-Image-2: Limitless Creativity For Every Builder Images are at the center of how developers build compelling AI-powered creative experiences; from marketing tools to content platforms to multimodal agents. MAI-Image-2 is Microsoft's answer to that demand. This model has been developed in close collaboration with photographers, designers, and visual storytellers and debuted in the top-3 text-to-image model families on the Arena.ai leaderboard. It raises the bar across the capabilities that matter most in real creative workflows; more natural, photorealistic image generation, stronger in-image text rendering for infographics and diagrams, and greater precision on complex layouts, detailed scenes, and cinematic visuals. Use cases for MAI-Image-2 Developers can integrate MAI-Image-2 across a range of high-impact workflows: Media & Creative Ideation: Designers, illustrators, and creative teams use text‑to‑image generation to explore visual directions, styles, and compositions early in the creative process—moving from concept to exploration faster. Enterprise Communications & Internal Branding: Organizations create custom visuals for internal campaigns, training materials, and executive communications directly from text, ensuring clarity, polish, and brand alignment without relying on stock imagery. UX & Product Concept Visualization: Product teams visualize interfaces, workflows, environments, and conceptual product scenarios from text descriptions, helping teams communicate ideas and align early—before engineering or design resources are engaged. WPP, one of the world's largest marketing and communications groups, is among the first enterprise partners building with MAI-Image-2 at scale, using it to power creative production workflows that previously required significant manual effort. "MAI-Image-2 is a genuine game-changer. It's a platform that not only responds to the intricate nuance of creative direction, but deeply respects the sheer craft involved in generating real-world, campaign-ready images. WPP has some of the best creative talent in the world and MAI-Image-2 is making them even better." -Rob Reilly, Global Chief Creative Officer, WPP We’re also implementing MAI-Image-2 to power image generation within Microsoft’s own products, including Copilot, Bing Image Creator, and PowerPoint, and now you have access to this powerful, cost effective model for your own apps. Try MAI-Image-2 Today Experiment in the MAI Playground: Preview MAI-Image-2 at MAI playground and share feedback directly with the team. Build in Foundry: deploy MAI-Image-2 via the API and start building your apps and agents! MAI-Image-2 starts at $5 USD per 1M tokens for text input and $33 USD per 1M tokens for image output. We look forward to your feedback on these models in Foundry. References: 1 1 st on overall WER on the FLEURS benchmark. Out of the top 25 global languages, MAI-Transcribe-1 ranks 1st by FLEURS in 11 core languages. It wins against Whisper-large-v3 on the remaining 14 and Gemini 3.1 Flash on 11 of those 14.
