Blog Post

Microsoft Mission Critical Blog
16 MIN READ

Azure OpenAI Architecture: The Decisions That Actually Matter (Part 2)

luciacasucci's avatar
luciacasucci
Icon for Microsoft rankMicrosoft
Jun 08, 2026

Introduction

In Part 1 of this series, we walked through the architectural decisions that shape any Azure OpenAI / Microsoft Foundry Models workload: capacity model, deployment location, governance layer, grounding strategy, and quota engineering. Part 2 moves from decisions to discipline. Once you have made those choices, how do you make sure your design holds up to the Azure Well-Architected Framework (WAF)?

The five WAF pillars — Cost Optimization, Security, Reliability, Performance Efficiency, and Operational Excellence — apply just as strongly to GenAI systems as they do to traditional cloud workloads. In fact, they matter more, because GenAI systems are not static: models are upgraded and deprecated, quotas shift, usage patterns grow unpredictably, and new capacity tiers (such as Priority Processing) are introduced while you are running in production.

This post walks through each pillar in the context of Azure OpenAI in Microsoft Foundry, with best practices, common pitfalls, and the trade-offs Cloud Solution Architects (CSAs) tend to hit in real engagements. Where details are time-sensitive — pricing percentages, SLA windows, model retirement timelines, regional rollout delays — they are flagged with "At the time of writing". Always confirm current behavior against Microsoft Learn before committing to a design.

Who is this series for?

  • Cloud and Solution Architects
  • Platform and product owners
  • Senior developers responsible for operating Azure OpenAI workloads in production

What you’ll learn in Part 2: 

  • How each WAF pillar maps to concrete Azure OpenAI design choices.
  • Where Priority Processing fits across cost and performance trade-offs (and what its eligibility constraints are).
  • How to plan for model lifecycle events — upgrades, deprecations, retirements — without firefighting.
  • Which signals to monitor day-to-day, and how to bake them into a GenAIOps loop.
  • A WAF Decision Matrix at the end of the article, to use as a reusable checklist.

In Part 3, we will look at the part that makes GenAI architecture genuinely different from a traditional service: the platform itself never stops moving.

We’ve also included a summary decision matrix at the end of this post for quick reference.

1. Cost Optimization: Designing for Sustainable Scale

Cost optimization in GenAI is primarily a capacity strategy problem, not just a token-pricing problem. The first question is whether to use pay-as-you-go capacity, reserved capacity, or one of the newer tiers in between.

Reserved capacity (Provisioned Throughput Units, PTUs)

If your workload is steady or growing predictably, you can significantly reduce costs by reserving capacity up front for 1 or 3 years. At the time of writing, reservations typically yield in the range of 30–50% savings compared to hourly pay-as-you-go rates — but the exact discount depends on term length, region, and the model family, so always confirm against the current Azure pricing page.

Fully utilizing a reserved (provisioned) deployment turns cloud spend into a predictable infrastructure investment, much like allocating VM or database capacity. This requires a mindset shift: treat a provisioned Azure OpenAI deployment as always-on infrastructure sized for peak demand, not as on-demand burst capacity.

Importantly, PTU quota is now model-agnostic within a region. You purchase generic throughput units that can be applied to any supported model in that region, so you do not risk stranded capacity when upgrading (say, from one GPT-4 family version to a newer one) or changing model versions. Your investment follows your architecture, not a specific model endpoint.

Avoid dynamic PTU "auto-scaling"

Unlike VM-based infrastructure, dynamically scaling Azure OpenAI capacity up and down to chase cost savings is not recommended. Additional capacity is not guaranteed to be instantly available when you need to scale up, especially if other tenants are consuming the region's resources. Frequent resizing can also negate the benefits of reservations and introduce performance variability. Unused PTUs are not waste — they are headroom that absorbs burst traffic. In practice, design for the peak load and optimize through reservations rather than trying to constantly dial capacity up and down.

Batch tier

Use Batch deployments for asynchronous, non-user-facing jobs (large-scale document processing, nightly data enrichment, evals, embeddings backfills). At the time of writing, Batch can reduce cost per token by up to around 50% compared to Standard pay-as-you-go calls, in exchange for a 24-hour completion window. It also takes pressure off your real-time deployments.

Priority Processing

For workloads that need prompt responses but do not yet warrant a full dedicated PTU deployment, Azure OpenAI offers Priority Processing. Functionally, it is pay-per-token like Standard, but with SLA-backed lower and more consistent latency on the shared infrastructure.

  • Activation: set the service_tier attribute on the request to "priority" (other values are "default" and "auto").
  • Model eligibility: at the time of writing, requires model versions released on or after 2025-12-01.
  • Deployment eligibility: only available on Global Standard or Data Zone Standard (US) deployments.
  • Pricing: at the time of writing, roughly 20–40% higher per-token cost than Standard, but still meaningfully cheaper than reserving PTU for a low-volume latency-critical path.

Treat Priority Processing as the natural in-between rung: more predictable than Standard for latency-sensitive production traffic, but without the commitment and capacity-planning effort of PTU.

Putting it together

Segment your workloads by interaction pattern and performance need, then assign the most cost-efficient capacity model to each. A common anti-pattern is over-provisioning expensive real-time capacity for jobs that could run asynchronously. Evaluate whether each use case truly requires sub-second latency, or whether a longer batch window (minutes or hours) is acceptable. Use real-time capacity for customer-facing queries and time-sensitive tasks; use Batch or Priority Processing for everything else, depending on tolerance for latency.

 

                                                                                  [Diagram 1 — Cost Strategy Layering]

2. Security: Compliance, Isolation, and Data Protection

Security in Azure OpenAI begins with deciding where your inference runs and how data is handled. This is often a compliance-driven decision before it is an architectural one.

Deployment scope

  • Global deployments — Maximize model availability and capacity by allowing Azure to route inference across regions. Pro: broad elasticity and access to the latest models. Con: data is not confined to a single geography, which may violate strict data residency requirements. Global also adds slight troubleshooting complexity, since requests can be served from various regions.
  • Data Zone deployments — Constrain inference to a specific zone or political boundary (for example, EU-only or US-only Data Zones). Pro: a compliance-friendly middle ground — data processing stays within a defined region set (for example, entirely within the EU to satisfy GDPR), while retaining more elasticity than a single region. Con: slightly reduced model availability and capacity headroom compared to Global.
  • Regional deployments — Confine inference to one Azure region. Pro: meets the most stringent data sovereignty requirements and can minimize latency for users in that region. Con: limited to the capacity and models available in one region, with no automatic overflow if the region is saturated. New model versions may also roll out to some regions later than others — at the time of writing we have observed delays of roughly 2–6 months for certain releases in specific regions; check Microsoft Learn for the current rollout schedule.

Choosing among these is a regulatory risk-management decision, not just an infrastructure preference. Engage your compliance and data governance teams early to determine the minimum scope of data movement that satisfies requirements. Many teams initially over-constrain this choice out of caution; it is often better to start with a broader deployment (Global or multi-region Data Zone) where permissible, and tighten the scope later if needed. Conversely, if your organization mandates that all data stay in-country, you might go straight to Regional and invest in architectural mitigation for its limitations (capacity planning, multi-region backup plans).

Baseline protections + defense in depth

Regardless of deployment type, Azure OpenAI provides baseline protections: it does not use your prompts or completions to train the underlying models, and all data is encrypted in transit (TLS 1.2+) and at rest (AES-256). Defense in depth is still essential — implement compensating controls at multiple layers:

  • Redact sensitive data from prompts (or prevent it from being entered) at the client or gateway layer. Use Azure API Management policies or custom middleware to strip out PII or secrets before requests reach the model.
  • Apply content filtering to both prompts and responses. Use the built-in content filters and/or Azure AI Content Safety to detect and block sensitive or undesirable input and output.
  • Use strong authentication and role-based access control. Front your Azure OpenAI endpoint with Microsoft Entra ID; scope tokens with least privilege (for example, the Cognitive Services OpenAI User role or managed identity access) instead of distributing master API keys. If a credential is compromised, the blast radius is limited.

Additional best practices

  • Managed Identities — use them for any internal communication between your application and Azure OpenAI (or other Azure services like storage and databases) instead of embedding API keys. This eliminates the risk of leaking secrets and simplifies credential rotation.
  • Private endpoints — enable Azure Private Link to keep traffic between your application and the Azure OpenAI service inside your virtual network and the Azure backbone, off the public internet.
  • Content Safety tooling — integrate Azure AI Content Safety or custom validation functions to scan prompts and completions for policy violations or confidential data. This extra inspection layer can catch issues the base filter misses, and lets you log or modify disallowed content before it reaches the user.

In short, security for GenAI is not just about encryption or API keys — it is about reducing the blast radius of any potential breach or misuse. Confine inference to approved locations, strip sensitive data before it reaches the model, and strictly limit which identities and networks can call your endpoints.

 

 

                                                                                  [Diagram 2 — Data Boundary Visualization]

3. Reliability: Designing for Change, Not Just Stability

Reliability in Azure OpenAI is as much about managing model evolution as it is about traditional uptime. Unlike static services, GenAI models are periodically updated and improved by the provider. New versions are released, older versions are deprecated and eventually retired — so a truly reliable system must plan for these changes just as carefully as it plans for hardware failures.

Model lifecycle

At the time of writing, Generally Available (GA) models are typically supported for at least 12 months after release, followed by a deprecation phase of roughly 6 months before retirement. Always confirm the current support windows on Microsoft Learn before locking in a design — these timelines have shifted in the past and may shift again as new model families ship.

When retirement hits:

  • Standard deployments still pinned to a retired model (with "No Auto-Upgrade" set) stop responding to requests entirely — the API typically returns HTTP 404 (or a similar error) for that model name.
  • Provisioned deployments using a retired model return HTTP 410 (Gone) errors until you manually switch them to a supported model.

In short, every model version you deploy has a built-in expiration date. Good reliability planning means never being caught unprepared by a model retirement.

Auto-upgrade modes for Standard deployments

Three modes are available:

  • Auto-upgrade to the latest version — the deployment moves to the new default model version as soon as Azure makes it available. Always on a supported version, but you have no control over timing. Generally not recommended for mission-critical production workloads, since new versions can have different behavior.
  • Upgrade only on retirement — the deployment stays on its current version until that version is about to be retired, then automatically switches to the latest. Recommended for most production Standard deployments: stability during the model's supported lifespan, with continuity guaranteed at retirement. You still need to test and adjust to the new version, but at least you do not face an outage if you miss the date.
  • No auto-upgrade — the deployment stays pinned to a specific version unless you change it manually. Not recommended for production: it puts the entire burden on you to track retirement timelines.

Most teams choose option 2 ("upgrade on retirement") for Standard. It allows controlled change during the model's supported period and provides a safety net at retirement. Proactively evaluate new versions for quality, performance, and cost before the forced swap, but the setting greatly reduces the risk of surprise outages.

Provisioned (PTU) migrations

Provisioned deployments do not support auto-upgrade — you must manage these migrations yourself. Azure sends retirement announcements via Azure Service Health alerts and emails, at the time of writing typically 60 days or more in advance. Have a runbook ready. Two approaches are common:

  • In-place migration — upgrade the deployment's model version through the portal or CLI. The endpoint stays the same and the model is updated behind it. Fast, no new connection string, but expect a brief disruption during the switch and rollback is not straightforward (you may need to contact support to re-enable the old version, if at all possible).
  • Side-by-side (blue/green) — create a new deployment with the new model version in parallel. Gradually shift traffic (for example, 10% via APIM routing rules), monitor, and roll back instantly if needed. Maximum control and safety, at the cost of running two deployments in parallel for the migration window.

Before any model migration, verify you have sufficient PTU quota in the region for the new model. More advanced models may require more throughput units for the same workload than smaller predecessors — at the time of writing, plan for the possibility that a new generation needs roughly two times (or more) the PTUs to deliver similar throughput. Request quota increases before you hit the upgrade window, not during it.

Multi-region strategy

Consider a multi-region strategy to improve reliability during model rollouts and deprecations. New model versions do not always appear in all regions simultaneously — at the time of writing, Microsoft often launches a model in one region (frequently East US or West Europe) first.

  • Maintain a secondary deployment in a "first-wave" region to evaluate new versions early.
  • Use a traffic manager (Azure Front Door, Traffic Manager) to fail over to a region where the model is still available if your primary region lags behind.
  • Multi-region active-active designs also protect you against single-region outages.

In essence, reliability for GenAI means designing for change. A highly reliable platform is not one that never changes; it is one that changes gracefully. Model upgrades, deprecations, and capacity adjustments should be routine, well-rehearsed events — not fire drills. Achieving this typically requires automation for detecting and applying updates, redundant deployments or regions for flexibility, and ongoing testing of new models well before your current ones retire.

 

                                                                                      [Diagram 3 — Model Upgrade Strategy]

4. Performance Efficiency: Predictability Over Raw Speed

Performance in GenAI is multi-dimensional. It is not just about raw throughput or the fastest response on an empty system — it is about consistent, predictable latency at scale. Users care that responses are reliably snappy under load, not just fast in ideal conditions.

Performance profiles by capacity model

  • Standard (shared infrastructure) — multi-tenant, no guaranteed latency SLA. Performance fluctuates with regional demand; you may see throttling (HTTP 429) at peak. Best-effort: great for development, testing, and non-critical workloads, but not a fit for consistent low latency under spikes.
  • Priority Processing — also shared infrastructure, but your requests jump the queue ahead of Standard traffic. At the time of writing, this is the only pay-per-token tier with an SLA on latency. Activated by setting the service_tier attribute to "priority" on each request (other values: "default", "auto"). Requires model versions released on or after 2025-12-01 and is only available on Global Standard or Data Zone Standard (US) deployments. Pricing premium is roughly 20–40% over Standard. The natural fit for latency-sensitive workloads at intermediate scale — better than Standard, without committing to PTU.
  • Provisioned Throughput (dedicated capacity) — reserved capacity with isolation from other tenants. The most consistent performance and the strongest Azure SLA on latency (typically bounded p50 and p99 within your provisioned capacity). If your application has strict response-time requirements or user-facing SLAs and the volume justifies it, PTU is the right answer.

A practical pattern: Standard for early-stage and non-critical scenarios; Priority Processing for latency-sensitive paths that have not yet earned a PTU reservation; PTU for steady, high-volume, latency-critical production traffic.

Model selection and configuration

  • Model size — smaller models generally respond faster than larger ones. Do not automatically pick the biggest model if a smaller one meets your quality bar.
  • max_tokens — capping response length caps worst-case latency and cost. A 500-token cap finishes sooner than 2000 tokens, even when users ask open-ended questions.
  • Sampling parameters — low temperature (more deterministic) and a high top_p can shave a small amount of processing overhead versus highly creative or multi-sample setups. Minor compared to model size and length, but real.
  • Streaming responses — enable streaming wherever possible. The first tokens arrive immediately while the model is still generating; perceived latency drops dramatically even when total time is unchanged.

Treat performance as an explicit design goal. Choose the right capacity model for the job, tune model settings to avoid unnecessary slowdowns, and do not over-engineer with a larger model than needed. A common mistake is defaulting to the biggest model "just in case". Benchmark — a smaller model with good prompt engineering often delivers a fraction of the latency at a fraction of the cost, with negligible quality loss.

5. Operational Excellence: Running GenAI as a Living System

Operational excellence in GenAI means treating your platform as a continuously evolving product. Models change, user behavior shifts, new features ship. Success requires ongoing monitoring, maintenance, and improvement processes — often called GenAIOps (or MLOps for generative AI).

Proactive monitoring

Set up Azure Service Health alerts for your Azure OpenAI / Foundry resource to be notified about service incidents and, importantly, upcoming model deprecations or retirements. At the time of writing, Microsoft typically gives around 60 days of notice for retirement events — but it is easy to miss those notifications if no one is watching. Early awareness lets you test new models and plan migrations calmly instead of reacting at the last minute.

Continuously track key metrics in Azure Monitor or Application Insights, with alerts on anomalies:

  • Latency percentiles — monitor p50, p95, and p99. A trend up in tail latency is an early warning of saturation or regression.
  • Error rates — watch HTTP 429 (throttling) and HTTP 503 (server) error trends. Spikes signal capacity limits or service-side issues.
  • Capacity utilization — for PTU, watch utilization continuously. Sustained operation near 100% means no headroom for bursts. On Standard, watch token usage against subscription limits and quotas.
  • Token consumption trends — track growth over time. Helps with cost forecasting and reveals runaway usage (unexpectedly popular features, looping clients).

Useful alerting practices: alert on p99 latency breaching a threshold, on any sustained increase in 429s, or when PTU utilization regularly exceeds around 80%. Early warning lets you scale up, optimize, adjust prompts, or throttle specific users before user experience suffers.

Evaluation and reproducibility

Use the evaluation tooling in Azure AI Foundry to compare outputs from two models side by side on a fixed set of test prompts. Re-evaluate periodically — slowly degrading quality often goes unnoticed without a structured comparison.

Implement Infrastructure-as-Code (IaC) and GitOps practices for your Azure OpenAI and supporting resources (APIM, storage, key vault, monitoring). Bicep, ARM, or Terraform templates checked into source control make environments reproducible across dev/test/prod, simplify recovery, and enable change tracking. If something breaks, you can roll back to a known-good configuration quickly.

In summary, operational excellence for GenAI is about continuous learning and improvement. Embrace an AI DevOps culture: invest in monitoring, train your team on model changes, keep optimizing prompts and configurations, and refine processes after each lesson learned. The effort pays off by preventing fire-drills and keeping the platform robust as it evolves.

Final Perspective and Key Takeaways

Applying the Well-Architected Framework to Azure OpenAI forces a higher level of architectural rigor — exactly what GenAI projects in production need. Each pillar drives concrete decisions.

Key takeaways from Part 2:

  • Cost Optimization — align capacity to workload patterns. Reserve for steady, predictable load; use Batch for offline jobs; use Priority Processing for latency-sensitive paths that do not yet justify PTU; do not pay for ultra-low latency you do not need.
  • Security — match deployment scope (Global, Data Zone, Regional) to compliance requirements, then layer controls (network isolation, identity and access, data sanitization, content filtering) to minimize blast radius.
  • Reliability — anticipate continuous model evolution. Use upgrade-on-retirement for Standard, run parallel deployments for PTU migrations, and design for multi-region failover where it matters. Reliability is about avoiding surprises, not just outages.
  • Performance Efficiency — choose the right capacity model, right-size models and responses, and use streaming. A smaller model with good prompt engineering often beats a bigger one on user experience.
  • Operational Excellence — treat the platform as a living product. Monitor, alert, automate, evaluate, and version everything as code. The discipline keeps the platform improving instead of decaying.

The organizations that succeed with Azure OpenAI / Microsoft Foundry Models are those that treat capacity planning, security and compliance, model lifecycle management, and governance as first-class design concerns — not afterthoughts. Generative AI architecture is not about deploying a model and walking away; it is about building a resilient, adaptable platform that gracefully evolves as models change and usage grows.

In Part 3, we bring everything together into a comprehensive reference architecture for an enterprise-grade Azure OpenAI platform — combining scalable capacity strategies, layered security and governance, proactive lifecycle (GenAIOps) practices, and multi-region resiliency into a cohesive blueprint ready for production.

 

                                                                                                [Diagram 4 — WAF pillars summary]

 

WAF Decision Matrix : Quick Reference

Use this as a checklist when reviewing or sign-off-ing an Azure OpenAI / Microsoft Foundry Models design. One row per decision; one rule of thumb per row.

Pillar

Decision

Rule of thumb

Watch out for

 

Capacity tier mix

Variable load → Standard. Latency-critical → Priority Processing. Offline bulk → Batch. Steady high-volume → PTU.

Single-tier platforms over-pay for elasticity or under-deliver on latency.

Cost

Reservation term (PTU)

1- or 3-year terms for predictable workloads; size for peak, not for average.

Dynamic resizing of PTU to chase savings; capacity not guaranteed at scale-up time.

Cost

Priority Processing eligibility

Requires service_tier="priority", model 2025-12-01+, Global Standard or Data Zone Standard (US).

Assuming it works on every region/model — confirm eligibility before committing the design.

Security

Deployment scope

No residency rule → Global. Multi-region zone OK → Data Zone. Strict residency → Regional.

Over-constraining out of caution; or under-constraining and missing a compliance requirement.

Security

Identity and access

Microsoft Entra ID + Managed Identity + scoped roles. No master keys in apps.

Long-lived API keys distributed across teams.

Security

Network and data

Private Link for in-network traffic; PII redaction at the gateway; content filtering on prompts and responses.

Public endpoints, raw PII in prompts, only relying on the built-in filter.

Reliability

Auto-upgrade strategy

Standard → "upgrade on retirement". PTU → planned blue/green migration with sufficient quota in advance.

Pinning Standard with no auto-upgrade and forgetting; in-place PTU migration with no rollback path.

Reliability

Multi-region

Active-active (or first-wave secondary) for critical paths; traffic manager for failover.

Single-region production with no plan for capacity or model-rollout lag.

Performance

Capacity match

Match capacity tier to latency target: Standard for non-critical; Priority for latency-sensitive; PTU for SLA-bound.

Expecting Standard to deliver consistent low latency under spike load.

Performance

Model and response sizing

Pick the smallest model that meets quality. Cap max_tokens. Stream responses.

Defaulting to the largest model "just in case"; long uncapped responses; no streaming.

Operations

Monitoring and alerting

Track p50/p95/p99, 429/503 rates, PTU utilization, and token trends. Alert on tail latency and sustained throttling.

Average-only dashboards; missed Service Health notifications for model retirements.

Operations

IaC and GitOps

Bicep/ARM/Terraform under source control; reproducible dev/test/prod; pipeline-driven changes.

Click-ops in the portal; environment drift between dev and prod.

Disclaimer

I am a Microsoft employee. The views and opinions expressed in this article are my own and do not necessarily reflect those of Microsoft. This content is informational and educational; it is not an official Microsoft statement, recommendation, or commitment. Service tiers, model availability, pricing, SLAs, and feature eligibility evolve — always validate against the latest Microsoft Learn documentation before making architectural decisions.

References

Updated Jun 05, 2026
Version 1.0
No CommentsBe the first to comment